Rate Limited Async Loop

A recent project included some modest load testing. For this we created a small console application to hit our API over HTTPS. A key metric in load testing is the number of requests an endpoint can handle per second, so it’s useful to be able to control and configure the rate at which requests are made.

This in itself is not difficult: a basic sleep wait of duration 1/requests-per-sec will achieve this. However we had an additional constraint that called for a slightly more complex solution.

The application uses Auth0, an authentication-as-a-service provider, and it rate limits use of its API. Exceeding the rate results in failed HTTP requests, and if frequent enough, can result in users being blocked. Furthermore, it is a remote and relatively slow API, with round-trip times in the order of 3 seconds (i.e. fetching 100 users serially would take 5 minutes), so it’s important that we access it concurrently, up to our limit. Additionally, the token received from calling it is cachable until its expiry, and if we can get the token from our cache then we want to skip any sleep-wait in order to minimize running time.

This leads to the goal: to maximize the number of concurrent requests made to an API up to a fixed number of requests per second; and to use cached data (and therefore not use a request) where possible. To solve this I want a rate-limited concurrent loop.

Implementation

A little searching on the internet resulted in either extensive libraries that implemented a different paradigm, like Reactive, or things that didn’t quite meet my requirements. I therefore – having taking the appropriate remedies to treat potential Not-Invented-Here Syndrome – went ahead and put something together myself.

public class RateLimitedTaskProperties
{
    public bool IgnoreRateLimit { get; set; }
}

public static async Task RateLimitedLoop(int perSec, IEnumerable enumerable, Func<T, Task> action)
{
    int periodMs = 1000 / perSec;
    var tasks = new List();
    foreach(T item in enumerable)
    {
        T capture = item;
        Task task = action(capture);
        tasks.Add(task);

        if (task.IsCompleted && task.Result.IgnoreRateLimit)
            continue;

        System.Threading.Thread.Sleep(periodMs);
    }

    await Task.WhenAll(tasks);
}

The loop starts a new task every periodMs. Concurrency is achieve by using tasks, which are non-blocking, and waiting for their completion outside the loop with await Task.WhenAll(tasks). The case where something has been retrieved from a cache is handled by the task returning synchronously and setting the IgnoreRateLimit flag. This combination causes the loop to skip the sleep and move straight onto triggering the next task.

The following is an example of its use, where MyOperation() is a method that returns a flag indicating whether or not it performed a fetch from the rate-limited API.

const int tokenReqsPerSec = 5;
await RateLimitedLoop(tokenReqsPerSec, items, async(item) =>
{
    bool requiredFetch = await item.MyOperation();
    // don't rate limit if I got it from the cache (fetch wasn't required)
    return new RateLimitedTaskProperties { IgnoreRateLimit = !requiredFetch };
});

Auth0 Mock

Auth0 is a well-known authentication-as-a-service provider. Its database connection storage option allows organizations to reference a custom database, which is very useful if you want to store your user information with your business data and maintain integrity between those using foreign key constraints. You can do this in Auth0 by setting up a connection that accesses your hosted database (with appropriate firewall restrictions!) to add, update, and remove users.

A challenge with this is that each new environment requires a new database and Auth0 setup. This is particularly difficult if that environment is a developer’s machine and isn’t accessible to a connection string from the internet (due to Firewalls/NAT). One option is for each developer to have their own cloud database, but that gets expensive quickly, and adds unrealistic latency to database calls from their machine, making development more difficult.

I was faced with this problem while building integration tests using Auth0 and .NET Core, and opted to create a mock object.

Implementation

The top level interface for Auth0 in C# is IManagementApiClient. This consists of a number of client interface properties, and it’s these that I found most appropriate to mock using Moq. This leads to a basic structure as follows:

using Auth0.Core;
using Auth0.Core.Collections;
using Auth0.Core.Http;
using Auth0.ManagementApi;
using Auth0.ManagementApi.Clients;
using Auth0.ManagementApi.Models;
using Moq;

public class Auth0Mock : IManagementApiClient
{
  Mock _usersClient = new Mock();
  Mock _ticketsClient = new Mock();

  public Auth0Mock()
  {
    // setup for _usersClient and _ticketsClient methods
  }

  public IUsersClient Users => _usersClient.Object;
  public ITicketsClient Tickets => _ticketsClient.Object;

  public IBlacklistedTokensClient BlacklistedTokens => throw new NotImplementedException();
  // etc. for ClientGrants, Clients, Connections, DeviceCredentials,  EmailProvider, Jobs, Logs, ResourceServers, Rules, Stats, TenantSettings, UserBlocks
  public ApiInfo GetLastApiInfo()
  {
    throw new NotImplementedException();
  }
}

In this project only a small number of Auth0 methods were used (something I expect would be true for most projects), so only a few Auth0 client methods actually needed to be mocked. However it is quite important, for integration testing, that these methods replicate the key behaviours of Auth0, including writing to a database, and storing user metadata (which isn’t always in the database). To support these, the mock class includes some custom SQL, and a small cache, which are used by the mocked methods. The following code illustrates this using two methods. They are set up in the constructor, and implemented in separate methods.

using System.Collections.Generic;
using System.Data.SqlClient;
using Dapper;

private string _sql;

// local cache storing information that our sql table doesn't
private Dictionary _users = new Dictionary();

public Auth0Mock(/* injection for _sql connection string */)
{
  _usersClient.Setup(s => s.CreateAsync(It.IsAny())).Returns((req) => CreateAsync(req));
  _usersClient.Setup(s => s.DeleteAsync(It.IsAny())).Returns((id) => DeleteAsync(id));
}

private async Task CreateAsync(UserCreateRequest request)
{
  int userId = 0;
  using (var conn = new SqlConnection(_sql))
  {
    var rows = await conn.QueryAsync(@"INSERT INTO [MyUserTable] ...", new { ... });
    userId = (int)rows.Single().userId;
  }

  var user = new Auth0.Core.User
  {
    AppMetadata = request.AppMetadata,
    Email = request.Email,
    FirstName = request.FirstName,
    LastName = request.LastName,
    UserId = "auth0|" + userId
  };
  _users[user.UserId] = user;
  return user;
}

private async Task DeleteAsync(string id)
{
  var match = Regex.Match(id, @"auth0\|(.+)");
  string userId = match.Groups.Last().Value;

  using (var conn = new SqlConnection(_connStr))
    await conn.ExecuteAsync(@"DELETE FROM [MyUserTable] ...", new { userId });

  if(_users.ContainsKey(id))
    _users.Remove(id);
}

Being a mock object there are limitations. For instance, in this example the cache only includes users added via CreateAsync, not all the users in the test database. However where these limitations lie depends entirely your testing priorities, as the sophistication of the mock is up to you.

One downside to this approach is that Moq doesn’t support optional parameters, so the signatures for some methods can get quite onerous:

_usersClient.Setup(s => s.GetAllAsync(0, 100, null, null, null, null, null, It.IsAny(), "v2"))
  .Returns((i1, i2, b3, s4, s5, s6, b7, q, s9) => GetAllAsync(i1, i2, b3, s4, s5, s6, b7, q, s9));

private Task<IPagedList> GetAllAsync(int? page, int? perPage, bool? includeTotals, string sort, string connection, string fields, bool? includeFields, string query, string searchEngine)
{
  // regex to match query and fetch from SQL and/or _users cache
}

Authorization

The Auth0 mock class provides authentication, but not authorization, and it would be nice if any integration tests could also check authorization policies. The run-time system is expecting to process a cookie or token on each request and turn that into a UserPrincipal with a set of claims. Therefore our tests must also populate the UserPrincipal, and do so before authorization is checked.

For this we need a piece of middleware that goes into the pipeline before authorization (which is part of UseMvc()). My approach was to place the call to UseAuthentication() into a virtual method in Startup and override that method in the test’s Startup:

public class TestStartup : Startup
{
  protected override void SetAuthenticationMiddleware(IApplicationBuilder app)
  {
    app.UseMiddleware();
  }
  
  protected override void SetAuthenticationService(IServiceCollection services)
  {
    // This is here to get expected responses on Authorize failures.
    // Authentication outcomes (user /claims) will be set via TestAuthentication middleware,
    // hence there are no token settings.
    services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme).AddJwtBearer();
  }
}

The middleware, TestAuthentication, remembers the last user that was set. It must be registered as a singleton with the dependency-injection framework so that the user is remembered between service calls. Testing code can set the user at any time by calling SetUser().

When a request is made TestAuthentication‘s InvokeAsync method applies claims based on that user. These claims will be processed as policies in the normal way so that Authorize attributes work as intended.

public class TestAuthentication : IMiddleware
{
  private string _userId;
  private string _roleName;

  public async Task InvokeAsync(HttpContext context, RequestDelegate next)
  {
    if (_userId > 0)
    {
      var identity = new ClaimsIdentity(new List
      {
        new Claim("http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier", "auth0|" + _userId),
        new Claim("http://myuri/", $"Role:{_roleName}")
      });

      var principal = new ClaimsPrincipal(identity);
      context.User = principal;
    }
    await next(context);
  }

  public void SetUser(string userId, string roleName)
  {
    _userId = userId;
    _roleName = roleName;
  }
}

With this combination we are able to successfully mock Auth0 while retaining our ability to work with our database, test non-Auth0 functionality, and test authorization.

Sharing Test Dependencies with Startup

An issue I’ve had while developing integration tests in .NET Core is sharing information between my TestContext and the Startup class.

The documented approach looks something like this:

var hostBuilder = new WebHostBuilder().UseStartup()
_server = new TestServer(hostBuilder);

The problem is that Startup is called from deep within new TestServer making it impossible to pass a reference from the calling context. This is particularly a problem with integration tests on an API, where we need the an HttpClient to be made from the TestServer instance in order to call API methods.

_client = _server.CreateClient();

Dependency Injection into Startup

What I hadn’t originally appreciated is that Startup class accepts dependencies defined by the host. Therefore anything already configured in the services, which is the container for ASP.NET’s dependency injection system, is available for injection into Startup.

For instance, to pass a reference to the current TestContext we register the current instance as a singleton before calling UseStartup:

var hostBuilder = new WebHostBuilder()
  .ConfigureServices(s => { s.AddSingleton(this); })
  .UseStartup()

Now, a the TestContext in the following Startup class will be populated:

public class Startup {
  private TestContext _ctx;
  public Startup(IConfiguration config, TestContext ctx) {
     _ctx = ctx;
  }
...

Passing a Shared Object

A more cohesive approach is to place mutual dependencies in another class and make it available via much the same approach. The following is an example allowing any class access to the TestServer’s client.

public interface ITestDependencies {
  public TestContext Context {get;}
  // also various Mock objects...
}

public class TestDependencies : ITestDependencies {
  public TestContext Context {get; private set;}

  public TestDependencies(TestContext ctx) {
    Context = ctx;
  }
}

public class Startup {
  private readonly ITestDependencies _testDependencies;
  public Startup(IConfiguration configuration, ITestDependencies testDependencies) {
    _testDependencies = testDependencies;
  }
  // other methods - use _testDependencies.Context.Client
}

public class TestContext {
  public HttpClient Client {get; private set;}
  private readonly TestServer _server;

  public TestContext() {
    var builder = new WebHostBuilder()
      .ConfigureServices((IServiceCollection services) => {
        services.AddSingleton(typeof(ITestDependencies), new TestDependencies(this));
      })
      .UseStartup();
    _server = new TestServer(builder);
    Client = _server.CreateClient();
  }
}

Hangfire

As part of my application I wanted to run a background service. In some fantasy future this might run as a separate process on another machine, scaling independently of the API server, so the service would naturally be isolated in its own class. For now I just needed something that would run scheduled jobs and be initialized during the Startup methods. The most popular solution for this problem seems to be a library called Hangfire which has had ASP.NET Core support since v1.6.0 (v1.6.16 at the time of writing).

Hangfire is backed by a database, so part of the setup involves selecting a database connector. There are two options for MySql, but the link for Hangfire.MySql goes 404, so I opted for Hangfire.MySqlStorage. I was able to get the basics of Hangfire working with this connector, although I did encounter some problems, notably that the Recurring Jobs dashboard page causes MySql exceptions and doesn’t load. One factor in this may be that, with Hangfire.Mysql as well as Pomelo.EntityFrameworkCore.MySql, I have references to different definitions of various MySql.Data.* classes in multiple assemblies. But as it currently works for my purposes, I haven’t pursued those errors further.

The other decision around the database is whether to share with the application database or use a separate schema. I opted for the latter to avoid any complications with my migration and test data code.

With that, we present the code. Firstly the .proj file:

<PackageReference Include="Hangfire" Version="1.6.*" />
<PackageReference Include="Hangfire.Autofac" Version="2.3.*" />
<PackageReference Include="Hangfire.MySqlStorage" Version="1.1.0-alpha" />

And then the startup functions. The first is called from ConfigureServices:

protected virtual void AddHangfireService(IServiceCollection services)
{
    services.AddHangfire(options =>
    {
        options.UseStorage(new Hangfire.MySql.MySqlStorage(
            Configuration["ConnectionStrings:HangfireMySql"],
            new Hangfire.MySql.MySqlStorageOptions
            {
                TransactionIsolationLevel = System.Data.IsolationLevel.ReadCommitted,
                QueuePollInterval = TimeSpan.FromSeconds(60),
                JobExpirationCheckInterval = TimeSpan.FromHours(1),
                CountersAggregateInterval = TimeSpan.FromMinutes(5),
                PrepareSchemaIfNecessary = true,
                DashboardJobListLimit = 50000,
                TransactionTimeout = TimeSpan.FromMinutes(1),
            }));
        options.UseAutofacActivator(this.IocContainer);
    });
}

and the second from Configure:

protected virtual void ConfigureHangfire(IApplicationBuilder app)
{
    app.UseHangfireDashboard();
    app.UseHangfireServer();

    RecurringJob.AddOrUpdate<Domain.Notification.INotifier>(
        "cbe-api-notification",
        notifier => notifier.Rollup(DateTime.UtcNow.AddDays(-1)),
        Cron.Daily(15) // 15:00 UTC - i.e. 3am NZST, 1am AEST
    );
}

This runs my job daily at 1500 UTC, which is the middle of the night from my perspective.

One aspect that Hangfire does very well is integrate with dependency injection frameworks. I have used Autofac, and you can see in the code above that nowhere have I had to construct the class for the notifier variable, instead the interface parameter INotifier suffices. The integration with Autofac is established in options.UseAutofacActivator(this.IocContainer); in the first code block. At the time UseAutofacActivator is called this.IocContainer is still null, but it doesn’t appear to be used until after Autofac is setup, which happens very soon thereafter.

Mocking MySqlException

As pleased as I am about Entity Framework Core’s InMemoryDatabase for testing, some failures cannot be easily simulated because InMemory doesn’t enforce database integrity, like constraint validation.

One of my operations uses foreign key constraints to validate data on insert and I wanted to mock this functionality to ensure a correct return value. This proved a little challenging because the type which triggers the failure, MySqlException, has no public constructors. Therefore a little reflection magic was required:

using System.Reflection;

public class MockSaveChangesAsyncSqlContext : SqlContext
{
    public MockSaveChangesAsyncSqlContext() { }
    public MockSaveChangesAsyncSqlContext(DbContextOptions<SqlContext> options) : base(options) { }

    public override Task<int> SaveChangesAsync(CancellationToken cancellationToken = default(CancellationToken))
    {
        var mySqlExceptionType = typeof(MySql.Data.MySqlClient.MySqlException).GetTypeInfo();
        var internalConstructor = (from consInfo in mySqlExceptionType.DeclaredConstructors
                                    let paramInfos = consInfo.GetParameters()
                                    where paramInfos.Length == 1 && paramInfos[0].ParameterType == typeof(string)
                                    select consInfo).Single();

        var innerException = internalConstructor.Invoke(new[] { "foreign key constraint fails" }) as Exception;
        throw new Exception("", innerException);
    }
}

It is important to note this is testing code. I would absolutely advise against using reflection to dig out hidden constructors in application code for many reasons, not the least of which is that a change to the library could suddenly break your application!

.Net Core Serializing File and Objects

For one of my API methods I wanted to send a file as well as object data. This is straight-forward enough when the object data consists of value types: the front end adds key-value-pairs to a FormData object, including the File object as one of the values; and the .NET Core back-end model object includes an IFormFile. e.g.

// JavaScript client
let data = new FormData();       
data.append("file", file);
data.append("id", "44b6...");
return this.httpClient.fetch(`...`, { method: 'post', body: data });
// C# Model
public class MyObj {
    public Microsoft.AspNetCore.Http.IFormFile File { get; set; }
    public Guid Id { get; set; }
}
// C# Controller Method
[HttpPost]
public async Task<IActionResult> Post(MyObj request) { ... }

However this approach fails if the model includes objects as in the following case where Numbers will be null.

public class MyObj {
    public Microsoft.AspNetCore.Http.IFormFile File { get; set; }
    public Guid Id { get; set; }
    public List<int> Numbers { get; set; }
}

At this point the model deserialization in .NET Core and the serialization done in JavaScript don’t match. However I found trying to use the suggested techniques to be somewhat over-complicated. My impression is the ‘right’ approach is to use a custom Model Binder. This seemed nice enough, but then got into details of needing to create and configure value binders, when I really just wanted to use some built-in ones for handling lists.

In the end I went with a different, perhaps less flexible or DRY, but vastly simpler approach: creating objects that shadowed the real object and whose get/set did the serialization.

public class ControllerMyObj : MyObj {
    public new string Numbers {
        get {
            return base.Numbers == null ? null : Newtonsoft.Json.JsonConvert.SerializeObject(base.Numbers);
        }
        set {
            base.Numbers = Newtonsoft.Json.JsonConvert.DeserializeObject<List<int>>(Numbers);
        }
    }
}

// Controller Method
[HttpPost]
public async Task<IActionResult> Post(ControllerMyObj request) { 
   MyObj myObj = request;
   ...
}

And now the front-end needs to be changed to send JSON serialized objects. That can be done specifically by key or using a more generic approach as follows.

let body = new FormData();
Object.keys(data).forEach(key => {
    let value = data[key];
    if (typeof (value) === 'object')
        body.append(key, JSON.stringify(value));
    else
        body.append(key, value);
});
body.append("file", file);
// fetch ...

.NET Core Secrets

Securing sensitive configuration information is one of those things that we know as developers is important, but so often gets deferred for more pressing commercial concerns, usually because of our confidence in the security infrastructure of our environments e.g. firewalls, VPNs. However in the field I’m heading into, security of personal information is important and expected to be audited, so with that commercial consideration in mind today I decided to tackle the challenge of securing configuration.

Secret Manager

There is a lot of chatter around the .NET Core Secret Manager but there appears to be two problems with it. Firstly, it is not a trusted store: “The Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store. It is for development purposes only. The keys and values are stored in a JSON configuration file in the user profile directory.”. Secondly, and more significantly I believe, it is user specific. That means that each user has their own credentials.

When I set up a development environment for a team I want it to be as uniform as possible for the whole team. A uniform environment makes it easier for team members to help each other, and makes it easier to script tools for automation. And many development resources will be shared, such as an AWS test instance.

Finally, this doesn’t help with production. For production the above website suggests using environment variables. Such variables are almost certainly stored in plaintext somewhere – in my case in Elastic Beanstalk configurations. Storing in plain-text is insecure and if nothing else is going to be a black mark on a security audit.

Extending .NET Core Configuration

What I want is sensitive information to be stored in an encrypted file where the encrypted file and the key are stored separately i.e. at least one of those is not checked into the source repository. I also still want to have different configurations available for different environments. It is also important that the file is relatively easy to modify.

What I propose is a custom configuration provider that is inserted into the ConfigurationBuilder which processes the other settings file when it is instantiated. The concept is outlined in this extension method:

public static IConfigurationBuilder AddEncryptedAndJsonFiles(this IConfigurationBuilder builder, string fileName, string basePath, bool optional, bool reloadOnChange = false)
{
    string jsonFilePath = builder.GetFileProvider().GetFileInfo(fileName).PhysicalPath;
    var encryptedConfiguration = new EncryptedConfigurationSource(jsonFilePath, basePath);
    encryptedConfiguration.UpdateStoredSettings();

    return builder
        .AddJsonFile(fileName, optional, reloadOnChange)
        .Add(encryptedConfiguration);
}

UpdateStoredSettings() will look through the appsettings file for keys starting with SENSITIVE_name. It will then add the name and corresponding value to the encrypted file and remove it from the appsettings file. The ConfigurationProvider returned by the IConfigurationSource.Build method will read the encrypted file and return a data dictionary of keys and values. The location of the key file will be set in the appsettings and read by both the source method and provider.

The extension method above will allow a simple replacement of AddJsonFile with AddEncryptedAndJsonFiles leaving Startup like this:

var builder = new ConfigurationBuilder()
    .SetBasePath(configBasePath)
    .AddEncryptedAndJsonFiles("appsettings.json", configBasePath, optional: true, reloadOnChange: true)
    .AddEncryptedAndJsonFiles($"appsettings.{env.EnvironmentName}.json", configBasePath, optional: true)
    .AddEnvironmentVariables();
Configuration = builder.Build();

Implementation

The implementation requires three classes as is standard for configuration providers:

  • a ConfigurationProvider which writes the properties into the dictionary used by consumers of configuration;
  • an IConfigurationSource which is the factory for ConfigurationProvider and where I opted to put the pre-processing method; and
  • an extension method for convenience.

The implementation uses AES for encryption. I considered deliberately using a slower method, but had trouble finding documentation and examples specific to .NET Core for symmetric encryption (as opposed to password hashing which is where those algorithms tend to be used).

Unlike the appsettings.json, the encrypted settings are stored in a single flat object, with the key being that used for configuration lookups e.g. configuration["parent:child"]. If a matching setting is found then it will overwrite the old one, allowing settings to be repeatedly updated.

One delightful problem I had was that the default IFileProvider refused to resolve paths above the base path, making it impossible to use for a relative path pointing outside the repository. As a result I had to pass in the base path, which feels like something of a hack.

A gist with the full source code can be found here