问题描述
我正在尝试将一个动态构建的大型JSON文件传输到客户端(可能超过500MB)。我尝试禁用响应缓冲的原因有很多,但主要是为了提高内存效率。
我尝试直接写入HttpContext.Response.BodyWriter
,但响应似乎在写入输出之前缓冲在内存中。此方法的返回类型为Task
。
HttpContext.Response.ContentType = "application/json";
HttpContext.Response.ContentLength = null;
await HttpContext.Response.StartAsync(cancellationToken);
var bodyStream = HttpContext.Response.BodyWriter.AsStream(true);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("["), cancellationToken);
await foreach (var item in cursor.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
await bodyStream.WriteAsync(JsonSerializer.SerializeToUtf8Bytes(item, DefaultSettings.JsonSerializerOptions), cancellationToken);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes(","), cancellationToken);
await bodyStream.FlushAsync(cancellationToken);
await Task.Delay(100,cancellationToken);
}
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("]"), cancellationToken);
bodyStream.Close();
await HttpContext.Response.CompleteAsync().ConfigureAwait(false);
注意:我意识到此代码非常繁琐,试图使其正常工作,然后将其清除
我使用Task.Delay
验证本地测试时没有缓冲响应,因为我没有完整的生产数据。我还尝试了IAsyncEnumerable
和yield return
,但失败了,因为响应太大了,以至于Kestrel认为可枚举数是无限的。
我已尝试
设置KestrelServerLimits.MaxResponseBufferSize为小数字,甚至为0; 使用HttpContext.Response.WriteAsync
编写
使用HttpContext.Response.BodyWriter.AsStream()
编写
使用烟斗写手模式和HttpContext.Response.BodyWriter
删除所有中间件
正在删除对IApplicationBuilder.UseResponseCompression
的调用
更新
在设置ContentType
之前尝试禁用响应缓冲(因此在对响应进行任何写入之前),但没有效果
var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();
已更新示例代码
这非常简单地再现了这个问题。在调用response.CompleteAsync()
之前,客户端不会收到任何数据。
[HttpGet]
[Route("stream")]
public async Task<EmptyResult> FileStream(CancellationToken cancellationToken)
{
var response = DisableResponseBuffering(HttpContext);
HttpContext.Response.Headers.Add("Content-Type", "application/gzip");
HttpContext.Response.Headers.Add("Content-Disposition", $"attachment; filename="player-data.csv.gz"");
await response.StartAsync().ConfigureAwait(false);
var memory = response.Writer.GetMemory(1024*1024*10);
response.Writer.Advance(1024*1024*10);
await response.Writer.FlushAsync(cancellationToken).ConfigureAwait(false);
await Task.Delay(5000).ConfigureAwait(false);
var str2 = Encoding.UTF8.GetBytes("Bar!
");
memory = response.Writer.GetMemory(str2.Length);
str2.CopyTo(memory);
response.Writer.Advance(str2.Length);
await response.CompleteAsync().ConfigureAwait(false);
return new EmptyResult();
}
private IHttpResponseBodyFeature DisableResponseBuffering(HttpContext context)
{
var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();
return responseBufferingFeature;
}
推荐答案
对于仍感兴趣的用户,此代码在使用cURL时立即发送数据:
public async Task Invoke(HttpContext context)
{
var g = context.Features.Get<IHttpResponseBodyFeature>();
g.DisableBuffering(); // doesn't seem to make a difference
context.Response.StatusCode = 200;
context.Response.ContentType = "text/plain; charset=utf-8";
//context.Response.ContentLength = null;
await g.StartAsync();
for (int i = 0; i < 10; ++i)
{
var line = $"this is line {i}
";
var bytes = utf8.GetBytes(line);
// it seems context.Response.Body.WriteAsync() and
// context.Response.BodyWriter.WriteAsync() work exactly the same
await g.Writer.WriteAsync(new ReadOnlyMemory<byte>(bytes));
await g.Writer.FlushAsync();
await Task.Delay(1000);
}
await g.CompleteAsync();
}
我尝试使用和不使用DisableBufering()
以及写入管道(IHttpResponseBodyFeature.Writer
vsHttpContext.Response.Body
)的变化似乎没有什么不同。
在cURL中,它会立即显示消息,但在Chrome和一些REST客户端中,它会等待整个流显示。
因此,我建议使用不等待整个流呈现代码的客户端来测试代码行为。我仍在检查的另一个选项是,如果客户端请求压缩,即使管道中没有配置压缩,aspnet core是否会自动获取压缩可能性。 所以我建议