HttpClient、Azure和System.Net.Sockets.Sockets.SocketException:怎么办?

人气:391 发布:2022-10-16 标签: azure c# httpclient .net-core-2.2

问题描述

我们在Azure中的Web应用程序上遇到很多问题,该应用程序使用HttpClient远程调用终结点。

我意识到Azure中的出站连接限制存在问题,因此我仅使用单一模式的一个HttpClient实例重构了代码。

这是我的帮助者:

public class HttpClientHelper
{
    string ApiKey = "";
    static HttpClient httpClient;
    
    public HttpClientHelper(IConfiguration config)
    {            
        var appsettings = config.GetSection("AppSettings");
        ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
        
        if (httpClient == null)
        {
            httpClient = new HttpClient();
            httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
        }
    }

    public async Task<T> SingleRead<T>(string baseAddress, string url)
    {
        T returnValue = default(T);
        
        try
        {
            var response = await httpClient.GetAsync($"{baseAddress}/{url}").ConfigureAwait(false);
            if (response.IsSuccessStatusCode)
            {
                var r = await response.Content.ReadAsStringAsync();
                returnValue = JsonConvert.DeserializeObject<T>(r);
            }
            else
            {
                Log($"SingleRead failed, error: {JsonConvert.SerializeObject(response)}");
            }
        }
        catch (Exception ex)
        {
            Log(ex, $"SingleRead ex: {baseAddress}/{url}");
            throw ex;
        }

        return returnValue;
    }

    public async Task<T> PostRead<T>(string baseAddress, string url, object entity = null)
    {
        T returnValue = default(T);            

        try
        {
            string json = JsonConvert.SerializeObject(entity);
            var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
            var response = await httpClient.PostAsync($"{baseAddress}/{url}", body).ConfigureAwait(false);
            if (response.IsSuccessStatusCode)
            {
                var r = await response.Content.ReadAsStringAsync();
                returnValue = JsonConvert.DeserializeObject<T>(r);
            }
            else
            {
                Log($"PostRead failed, error: {JsonConvert.SerializeObject(response)}");
            }
        }
        catch (Exception ex)
        {
            Log(ex, $"PostRead url: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
            throw ex;
        }

        return returnValue;
    }
}

然后我执行基本请求,调用SingleRead()(用于GET)或PostRead()(用于POST)。

问题是,有时我仍然会收到SocketException:

下面是完整堆栈跟踪:

System.Net.Http.HttpRequestException:
   at MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:MyProjectWebAppHelpersHttpClientHelper.cs:77)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at MobilityServer.Helpers.MigrationHelper+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:MyProjectWebAppHelpersMigrationHelper.cs:54)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at MobilityServer.Controllers.AndroidController+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:MyProjectWebAppControllersBasicController.cs:104)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+TaskOfIActionResultExecutor+<Execute>d__0.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextResourceFilter>d__23.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__18.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__16.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware+<Invoke>d__3.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__7.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Eccezione interna System.Net.Sockets.SocketException gestita in MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

这是异常堆栈跟踪:

   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:MyProjectWebAppHelpersHttpClientHelper.cs:line 63

我不知道问题出在哪里。 出站连接现在低于服务计划(";s3:2";)。

SNAT似乎远远低于限制:

出站连接相同:

(注意:我已在昨天21:30将3个实例的大小调整为2个,这就是为什么那些绿色实例放弃了&p>)。

我真的搞不懂发生了什么事。环境设置正确。

有什么线索吗?

编辑 以下是添加建议的IHttpClientFactory的版本(即编辑):

// Startup.cs
public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);            
    services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);
    services.AddHttpClient<MigrationHelper>();
}

public class MigrationHelper
{
    HttpClientHelper httpc;

    public MigrationHelper(IConfiguration config, HttpClient httpClient)
    {
        // ...
        
        httpc = new HttpClientHelper(config, httpClient);
    }
}

public class HttpClientHelper
{
    string ApiKey = "";
    HttpClient httpClient;
    
    public HttpClientHelper(IConfiguration config, HttpClient _httpClient)
    {
        var appsettings = config.GetSection("AppSettings");
        ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
        
        httpClient = _httpClient;
        httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
    }
    
    // ...
}

仍然不起作用,我收到相同的错误套接字消息。这里有一个完整的示例:

{
    "Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
    "Data": {},
    "InnerException": {
        "ClassName": "System.Net.Sockets.SocketException",
        "Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
        "Data": null,
        "InnerException": null,
        "HelpURL": null,
        "StackTraceString": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)",
        "RemoteStackTraceString": null,
        "RemoteStackIndex": 0,
        "ExceptionMethod": null,
        "HResult": -2147467259,
        "Source": "System.Private.CoreLib",
        "WatsonBuckets": null,
        "NativeErrorCode": 10060
    },
    "StackTrace": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
 at System.Threading.Tasks.ValueTask`1.get_Result()
 at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 at System.Threading.Tasks.ValueTask`1.get_Result()
 at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
 at System.Threading.Tasks.ValueTask`1.get_Result()
 at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
 at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
 at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:line 60",
    "HelpLink": null,
    "Source": "System.Net.Http",
    "HResult": -2147467259
}

IHttpClientFactory

尝试1:您在您的应用程序中最多只能使用推荐答案实现弹性Http请求(请参阅:https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests)。

尝试2:当您向外扩展端点(如果它在您的控制之下)时,所有实例上都有一个负载均衡器,因此,流量在它们之间自动负载平衡。可能您需要更好的负载均衡解决方案(请参阅:https://docs.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview)。

尝试3:如果您的应用程序在单机上运行得更好,您应该向上扩展并使用按应用程序缩放(请参阅:https://docs.microsoft.com/en-us/azure/app-service/manage-scale-per-app)

尝试4:解决Azure应用程序服务中的间歇性出站连接错误(请参阅:https://docs.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors)

附注:一个胡乱猜测是端点无法处理来自您的3个实例的不断增加的请求(一个实例在使用通过HttpClientHandler.MaxConnectionsPerServer属性设置的HttpClient对象发出请求时,允许256个并发连接(每个服务器端点)的限制);这就是为什么您减少实例数量时,它的行为会更好

131