The idea is to use HttpContext.Response.OnStarting callback, as this is the last event that is fired before the headers are sent. Add the following to your middleware Invoke method.
public async Task Invoke(HttpContext context)
{
context.Response.OnStarting((state) =>
{
if ...