Skip to content

Latest commit

 

History

History
365 lines (284 loc) · 16.8 KB

http-passthrough.md

File metadata and controls

365 lines (284 loc) · 16.8 KB

SQL HTTP Passthrough

SQL HTTP Passthrough provides a bridge between an HTTP stream (via JavaScript on a web page) and the SQL Server transport. It leverages SQL Transport - Native and SQL Attachments.

Design

Server side hosting

SQL HTTP Passthrough is designed to be consumed by any web application built on ASP.NET Core. For example, it can be used to send a message from a Controller, a BaseController, a Filter or a Middleware.

Deduplication

To handle intermittent connectivity issues it is desirable to have a web client leverage a retry mechanism so if a request fails, the same request can be immediately re-sent. To prevent this resulting in duplicate message being placed on the queue, message deduplication has to occur. SQL HTTP Passthrough leverages the deduplication feature of SQL Transport - Native.

Data and attachments

To send both message content and associated binary data (attachments) a multipart form is used.

Usage

Server-side

ASP.NET Core startup

At ASP.NET Core startup several actions are taken:

  • AddSqlHttpPassthrough is called on IServiceCollection which makes the ISqlPassthrough interface available via dependency injection.
  • AddSqlHttpPassthroughBadRequestMiddleware is called on IApplicationBuilder, which adds Middleware to the pipeline. This means that if the request parsing code of the SQL HTTP Passthrough throws a BadRequestException, that exception can be gracefully handled and an HTTP BadRequest (400) can be sent as a response. This is optional, and a Controller can choose to explicitly catch and handle BadRequestException in a different way.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        var configuration = new PassthroughConfiguration(
            connectionFunc: OpenConnection,
            callback: Callback,
            dedupCriticalError: exception =>
            {
                Environment.FailFast("Dedup cleanup failure", exception);
            });
        services.AddSqlHttpPassthrough(configuration);
        services.AddMvcCore();
        // other ASP.MVC config
    }

    static Task<Table> Callback(HttpContext http, PassthroughMessage message)
    {
        //TODO: validate that the message type is allowed
        //TODO: validate that the destination is allowed
        if (message.Destination == null)
        {
            var customDestination = new Table("Custom");
            return Task.FromResult(customDestination);
        }

        var destination = new Table(message.Destination);
        return Task.FromResult(destination);
    }

    public void Configure(IApplicationBuilder builder)
    {
        builder.AddSqlHttpPassthroughBadRequestMiddleware();
        builder.UseMvc();
        // other ASP.MVC config
    }

    static Task<SqlConnection> OpenConnection(Cancel cancel) =>
        //TODO open and return a SqlConnection
        null!;
}

snippet source | anchor

Append Claims

Append the Claims of the ClaimsPrincipal from HttpContext.User to the headers of the outgoing message.

By default each header will get a prefix of SqlHttpPassthrough.Claim.

var configuration = new PassthroughConfiguration(
    connectionFunc: OpenConnection,
    callback: Callback,
    dedupCriticalError: exception =>
    {
        Environment.FailFast("Dedup cleanup failure", exception);
    });
configuration.AppendClaimsToMessageHeaders();
services.AddSqlHttpPassthrough(configuration);

snippet source | anchor

A custom prefix can also be defined.

var configuration = new PassthroughConfiguration(
    connectionFunc: OpenConnection,
    callback: Callback,
    dedupCriticalError: exception =>
        Environment.FailFast("Dedup cleanup failure", exception));
configuration.AppendClaimsToMessageHeaders(headerPrefix: "Claim.");
services.AddSqlHttpPassthrough(configuration);

snippet source | anchor

For unit testing and integration purposes it may be useful to manipulate a raw Dictionary<string, string>. This can be done using ClaimsAppender.

To append claims:

var claims = new List<Claim>
{
    new(ClaimTypes.Email, "[email protected]"),
    new(ClaimTypes.NameIdentifier, "User1"),
    new(ClaimTypes.NameIdentifier, "User2")
};
ClaimsAppender.Append(claims, headerDictionary, "prefix.");

snippet source | anchor

To extract claims:

var claimsList = ClaimsAppender.Extract(headerDictionary, "prefix.");

snippet source | anchor

It may also be necessary to process claims with no reference to NServiceBus.SqlServer.HttpPassthrough. This can be done using the following utility methods. Note that these methods use JsonConvert from Json.NET.

public static void Append(
    IEnumerable<Claim> claims,
    IDictionary<string, string> headers, string prefix)
{
    foreach (var claim in claims.GroupBy(_ => _.Type))
    {
        var items = claim.Select(_ => _.Value);
        headers.Add(prefix + claim.Key, JsonConvert.SerializeObject(items));
    }
}

public static IEnumerable<Claim> Extract(
    IDictionary<string, string> headers,
    string prefix)
{
    foreach (var header in headers)
    {
        var key = header.Key;
        if (!key.StartsWith(prefix))
        {
            continue;
        }

        key = key.Substring(prefix.Length, key.Length - prefix.Length);
        var list = JsonConvert.DeserializeObject<List<string>>(header.Value)!;
        foreach (var value in list)
        {
            yield return new(key, value);
        }
    }
}

snippet source | anchor

Message callback

AddSqlHttpPassthrough takes a required parameter callback with the signature Func<HttpContext, PassthroughMessage, Task<Table>>. This delegate will be called during each request-to-message execution. This occurs after the HTTP request has been parsed, and before the outgoing message is placed on the SQL table. The return value is a Table that dictates the SQL table and schema that the message will be written to.

While callback supports async, via returning a Task<Table>, any required async action should have its result cached so as not to slow down subsequent requests. For example, it may be necessary to perform authorization in a callback. The result of this authorization should be cached for some period of time, and the cached result should be purged when permissions are changed.

The message callback can be used for several purposes:

  • Validate that the message type and destination are allowed.
  • Add extra headers to the outgoing message.
  • Manipulate any other properties of the outgoing message

WARNING: Note that a "trust but verify" approach should be taken in regards to the HTTP client. The combination of message type/namespace and destination should be verified against a known allowed list.

PassthroughMessage contains the following properties:

  • Id: Contains the MessageId value from HttpRequest.Headers
  • CorrelationId: Contains the MessageId value from HttpRequest.Headers
  • Type: Contains the MessageType value from HttpRequest.Headers. Will be combined with Namespace and used for the NServiceBus.EnclosedMessageTypes header.
  • Namespace: Contains the MessageNamespace value from HttpRequest.Headers. Will be combined with Type and used for the NServiceBus.EnclosedMessageTypes header.
  • Body: Contains the Message value from the IFormCollection.
  • Destination: Contains the 'Destination' value from HttpRequest.Headers. Primarily used to convert to a Table as a return value for the passthrough callback.
  • ClientUrl: The URL of the submitting page. Contains the HeaderNames.Referer value from HttpRequest.Headers. Will be written to a header MessagePassthrough.ClientUrl in the outgoing NServiceBus message.
  • Attachments: Contains all binaries extracted from IFormCollection.Files
  • ExtraHeaders: Any extra headers to add to the outgoing NServiceBus message.

Usage in a controller

Usage in a controller consists of several parts.

  • ISqlPassthrough injected through dependency injection.
  • The Controller handling the HTTP Post and passing that information to ISqlPassthrough.Send.

[Route("SendMessage")]
public class PassthroughController(ISqlPassthrough sender) : ControllerBase
{
    [HttpPost]
    public Task Post(Cancel cancel) =>
        sender.Send(HttpContext, cancel);
}

snippet source | anchor

WARNING: In a production application the controller would be performing any authorization and authentication on the incoming request.

Exception behavior

If ISqlPassthrough fails to send, a SendFailureException will be thrown containing all context in a PassthroughMessage property.

If the incoming HTTP request fails to be parsed, a BadRequestException will be thrown with the message containing the reason for the failure.

Client - JavaScript

Form submission

The JavaScript that submits the data does so through by building up a FormData object and POSTing that via the Fetch API.

function PostToBus() {
    var message = new Object();
    message.Property1 = document.getElementById("property1").value;
    message.Property2 = document.getElementById("property2").value;
    var jsonString = JSON.stringify(message);

    var data = new FormData();
    var files = document.getElementById("files").files;
    for (var i = 0; i < files.length; i++) {
        data.append('files[]', files[i], files[i].name);
    }
    data.append("message", jsonString);

    var postSettings = {
        method: 'POST',
        credentials: 'include',
        mode: 'cors',
        headers: {
            'MessageType': 'SampleMessage',
            'MessageNamespace': 'SampleNamespace',
            'MessageId': newGuid(),
            'Destination': 'Endpoint'
        },
        body: data
    };

    return fetch('SendMessage', postSettings);
}

snippet source | anchor

MessageId generation

For deduplication to operate, the client must generate a MessageId, so that any retries can be ignored. JavaScript does not contain native functionality to generate a GUID, so a helper method is used.

function newGuid() {
    return ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, c =>
        (c ^ crypto.getRandomValues(new Uint8Array(1))[0] & 15 >> c / 4)
            .toString(16)
            .toUpperCase()
    )
}

snippet source | anchor

Client .NET

Creating and posting a multipart form can be done using a combination of MultipartFormDataContent and HttpClient.PostAsync. To simplify this action the ClientFormSender class can be used:

var clientFormSender = new ClientFormSender(httpClient);
await clientFormSender.Send(
    route: "/SendMessage",
    message: "{\"Property\": \"Value\"}",
    typeName: "TheMessageType",
    typeNamespace: "TheMessageNamespace",
    destination: "TheDestination",
    attachments: new()
    {
        {"fileName", "fileContents"u8.ToArray()}
    });

snippet source | anchor

This can be useful when performing Integration testing in ASP.NET Core.

var hostBuilder = new WebHostBuilder();
hostBuilder.UseStartup<Startup>();
using var testServer = new TestServer(hostBuilder);
using var httpClient = testServer.CreateClient();
var clientFormSender = new ClientFormSender(httpClient);
await clientFormSender.Send(
    route: "/SendMessage",
    message: "{\"Property\": \"Value\"}",
    typeName: "TheMessageType",
    typeNamespace: "TheMessageNamespace",
    destination: "TheDestination",
    attachments: new()
    {
        {"fileName", "fileContents"u8.ToArray()}
    });

snippet source | anchor