Hangfire

As part of my application I wanted to run a background service. In some fantasy future this might run as a separate process on another machine, scaling independently of the API server, so the service would naturally be isolated in its own class. For now I just needed something that would run scheduled jobs and be initialized during the Startup methods. The most popular solution for this problem seems to be a library called Hangfire which has had ASP.NET Core support since v1.6.0 (v1.6.16 at the time of writing).

Hangfire is backed by a database, so part of the setup involves selecting a database connector. There are two options for MySql, but the link for Hangfire.MySql goes 404, so I opted for Hangfire.MySqlStorage. I was able to get the basics of Hangfire working with this connector, although I did encounter some problems, notably that the Recurring Jobs dashboard page causes MySql exceptions and doesn’t load. One factor in this may be that, with Hangfire.Mysql as well as Pomelo.EntityFrameworkCore.MySql, I have references to different definitions of various MySql.Data.* classes in multiple assemblies. But as it currently works for my purposes, I haven’t pursued those errors further.

The other decision around the database is whether to share with the application database or use a separate schema. I opted for the latter to avoid any complications with my migration and test data code.

With that, we present the code. Firstly the .proj file:

<PackageReference Include="Hangfire" Version="1.6.*" />
<PackageReference Include="Hangfire.Autofac" Version="2.3.*" />
<PackageReference Include="Hangfire.MySqlStorage" Version="1.1.0-alpha" />

And then the startup functions. The first is called from ConfigureServices:

protected virtual void AddHangfireService(IServiceCollection services)
{
    services.AddHangfire(options =>
    {
        options.UseStorage(new Hangfire.MySql.MySqlStorage(
            Configuration["ConnectionStrings:HangfireMySql"],
            new Hangfire.MySql.MySqlStorageOptions
            {
                TransactionIsolationLevel = System.Data.IsolationLevel.ReadCommitted,
                QueuePollInterval = TimeSpan.FromSeconds(60),
                JobExpirationCheckInterval = TimeSpan.FromHours(1),
                CountersAggregateInterval = TimeSpan.FromMinutes(5),
                PrepareSchemaIfNecessary = true,
                DashboardJobListLimit = 50000,
                TransactionTimeout = TimeSpan.FromMinutes(1),
            }));
        options.UseAutofacActivator(this.IocContainer);
    });
}

and the second from Configure:

protected virtual void ConfigureHangfire(IApplicationBuilder app)
{
    app.UseHangfireDashboard();
    app.UseHangfireServer();

    RecurringJob.AddOrUpdate<Domain.Notification.INotifier>(
        "cbe-api-notification",
        notifier => notifier.Rollup(DateTime.UtcNow.AddDays(-1)),
        Cron.Daily(15) // 15:00 UTC - i.e. 3am NZST, 1am AEST
    );
}

This runs my job daily at 1500 UTC, which is the middle of the night from my perspective.

One aspect that Hangfire does very well is integrate with dependency injection frameworks. I have used Autofac, and you can see in the code above that nowhere have I had to construct the class for the notifier variable, instead the interface parameter INotifier suffices. The integration with Autofac is established in options.UseAutofacActivator(this.IocContainer); in the first code block. At the time UseAutofacActivator is called this.IocContainer is still null, but it doesn’t appear to be used until after Autofac is setup, which happens very soon thereafter.

Profiling .NET Core

Some of my application requests are running slowly and I need an overview of what is taking so long, so I turned to the internet to find a profiler. With .NET Core still relatively new I expected that finding mature profilers would be challenging. In addition .NET (in general) has thrown a curve-ball in the direction of profilers in the last few years with the use of async.

Calling await in a method causes dotnet to generate a state-machine that splits the function up into many different parts, and fills stack traces with MoveNext() functions. To be useful, a profiler needs to link these pieces of state-machine – which I believe could be running on different threads – back-together so the developer can understand what it is waiting for.

The Field

The only profiler that seemed to handle async was ANTS Performance 9.6. I initially found it’s results quite counter-intuitive until I changed the timing options drop-down to wall-clock time. Then it became much clearer from the call tree where the delays were. However it didn’t seem to load the source code despite PDB files being and place, and it was also the most expensive tool I evaluated.

The best free tool, in my opinion, was CodeTrack which provides a reasonable timeline view to enable navigation of the calls, but doesn’t have any in-built async handling.

A similar function was provided by dotTrace 2017.2 (EAP3). dotTrace also seems to be able to handle a few async cases, combining calls from the same source with async or cont, but for most cases it didn’t link them together.

There are also light-profilers, intended more for monitoring. MiniProfiler seems tailored for full MVC apps, and I couldn’t get it to produce output in my view-less API project. Prefix didn’t seem to work at all, as noted by other commented on their website, which may be related to using Core 1.1.

Finally, I should not I do not have Visual Studio 2017 so I don’t know what its profiler is like.

.Net Core Serializing File and Objects

For one of my API methods I wanted to send a file as well as object data. This is straight-forward enough when the object data consists of value types: the front end adds key-value-pairs to a FormData object, including the File object as one of the values; and the .NET Core back-end model object includes an IFormFile. e.g.

// JavaScript client
let data = new FormData();       
data.append("file", file);
data.append("id", "44b6...");
return this.httpClient.fetch(`...`, { method: 'post', body: data });
// C# Model
public class MyObj {
    public Microsoft.AspNetCore.Http.IFormFile File { get; set; }
    public Guid Id { get; set; }
}
// C# Controller Method
[HttpPost]
public async Task<IActionResult> Post(MyObj request) { ... }

However this approach fails if the model includes objects as in the following case where Numbers will be null.

public class MyObj {
    public Microsoft.AspNetCore.Http.IFormFile File { get; set; }
    public Guid Id { get; set; }
    public List<int> Numbers { get; set; }
}

At this point the model deserialization in .NET Core and the serialization done in JavaScript don’t match. However I found trying to use the suggested techniques to be somewhat over-complicated. My impression is the ‘right’ approach is to use a custom Model Binder. This seemed nice enough, but then got into details of needing to create and configure value binders, when I really just wanted to use some built-in ones for handling lists.

In the end I went with a different, perhaps less flexible or DRY, but vastly simpler approach: creating objects that shadowed the real object and whose get/set did the serialization.

public class ControllerMyObj : MyObj {
    public new string Numbers {
        get {
            return base.Numbers == null ? null : Newtonsoft.Json.JsonConvert.SerializeObject(base.Numbers);
        }
        set {
            base.Numbers = Newtonsoft.Json.JsonConvert.DeserializeObject<List<int>>(Numbers);
        }
    }
}

// Controller Method
[HttpPost]
public async Task<IActionResult> Post(ControllerMyObj request) { 
   MyObj myObj = request;
   ...
}

And now the front-end needs to be changed to send JSON serialized objects. That can be done specifically by key or using a more generic approach as follows.

let body = new FormData();
Object.keys(data).forEach(key => {
    let value = data[key];
    if (typeof (value) === 'object')
        body.append(key, JSON.stringify(value));
    else
        body.append(key, value);
});
body.append("file", file);
// fetch ...

.NET Core Code Coverage

I haven’t written anything for a while because, frankly, I’m passed the platform R&D stages of my current application and just churning out features, and so far I haven’t found anything much to inspire me to write about. After digging through my code looking at things I’d done, one area I thought may be interesting to readers is getting (free) code coverage in .NET Core.

OpenCover

The open source tool of choice for code coverage seems to be OpenCover. This comes as a nice zip file which can be extracted anywhere. Getting set up for .NET Core was mostly a case of following various instructions online, and there was just one gotcha: the MSBuild DebugType must be full, which is typically not the case for .NET Core where the goal is deployment to multiple operating systems. To get around this my coverage script overwrites the .proj file before running and puts portable back when it is done.

The script runs the dotnet executable from the assembly folder, meaning the assemblies aren’t directly specified in the script. The graphical output of the coverage is put together using ReportGenerator, which I have deployed inside my report output folder.

Here is a cut-down version of my Powershell script:

Push-Location

# change portable to full for projects
$csprojFiles = gci [Repository-Path] -Recurse -Include *.csproj
$csprojFiles | %{
     (Get-Content $_ | ForEach  { $_ -replace 'portable', 'full' }) | 
     Set-Content $_
}

# Setup filter to exclude classes with no methods
$domainNsToInclude = @("MyNamespace.Auth.*", "MyNamespace.Data.*")
# Combine [assembly] with namespaces
$domainFilter = '+[AssemblyPrefix.Domain]' + ($domainNsToInclude -join ' +[AssemblyPrefix.Domain]')
$filter = "+[AssemblyPrefix.Api]* $domainFilter"

# Integration Test Project
$integrationOutput = "[output-path]\Integration.xml"
cd "D:\Code\RepositoryRoot\test\integration"
dotnet build
[open-cover-path]\OpenCover.Console.exe `
    -register:user `
    -oldStyle `
    "-target:C:\Program Files\dotnet\dotnet.exe" `
    "-targetargs:test" `
    "-filter:$filter" `
    "-output:$integrationOutput" `
    -skipautoprops

# Generate Report
$reportFolder = "[output-path]\ReportFolder"
[report-generator-path]\ReportGenerator.exe `
    "-reports:$integrationOutput" `
    "-targetdir:$reportFolder"

# restore portable in projects
$csprojFiles | %{
     (Get-Content $_ | ForEach  { $_ -replace 'full', 'portable' }) |
     Set-Content $_
}

Pop-Location

The end result after opening the index.html is something like this (looks like I need to work on that branch coverage!):
coverage

ASP.NET Core Authentication

One of the challenges I had early in developing my current project was getting authentication set up nicely. My back-end is an API running in .NET Core, and my general impression is that ASP.NET Core’s support for API use cases is somewhat weaker than for MVC applications.

ASP.NET Core’s default transport for authentication context still seems to be via cookies. This was quite surprising as my impression of the industry is that, between their complexity (from which it is easy to make security mistakes) and recent EU rules, cookies were on their way out. ASP.NET Core also introduced Identity for authentication, but the use of ViewModel in examples indicates that is targeted towards an MVC application.

My preference was to use JSON Web Tokens (JWTs) sent as bearer tokens in the authorization header of an HTTP request. I also wanted to use authorization attributes, like [Authorize("PolicyName")], to enforce security policy on the API controllers.

Validation and Authorization

.NET Core has support for validating JWTs via the System.IdentityModel.Tokens.Jwt package. Applying this requires something like the following in the Startup.Configure method:

JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear();
app.UseJwtBearerAuthentication(new JwtBearerOptions()
{
    Authority = Configuration["AuthorityUrl"],
    TokenValidationParameters = new TokenValidationParameters() { ValidateAudience = false },
    RequireHttpsMetadata = true,
    AutomaticAuthenticate = true,
    Events = new JwtBearerEvents { OnTokenValidated = IocContainer.Resolve<Auth.IValidatedTokenHandling>().AddUserClaimsToContext },
};

The recommended approach to authorization in ASP.NET Core is to use claims and policies. To that end the code above responds to the OnTokenValidated event and sends it to a method that queries the user and adds claims based on information about the user.

public async Task AddUserClaimsToContext(TokenValidatedContext context) 
{
    var claims = new List<Claim>();

    // JWT subject is the userid
    var sub = context.Ticket.Principal.FindFirst("sub")?.Value;
    if(sub != null)
    {
        var user = await _users.FindById(Guid.Parse(sub));
        if(user != null)
        {
            if(user.UserVerification > 0)
                claims.Add(new Claim("MustBeValidatedUser", "true", ClaimValueTypes.Boolean));
        }
    }
    var claimsIdentity = context.Ticket.Principal.Identity as ClaimsIdentity;
    claimsIdentity.AddClaims(claims);
}

Finally the policies themselves must be defined, typically in the Startup.ConfigureServices method:

mvc.AddAuthorization(options => {
    options.AddPolicy("MustBeValidatedUser", policy => policy.RequireClaim(Auth.ClaimDefinitions.MustBeValidatedUser, "true"));                   
});

Generating Tokens

.NET Core does not have support for generating JWTs. For this it recommends IdentityServer4.

IdentityServer4 is intended to be a fully fledged authentication server supporting the many flows of OAuth2 and Open ID Connect. For my purposes I only required username and password validation, so in many respects IdentityServer4 was overkill, but given lack of alternatives for generating JWTs, I forged ahead with it anyway.

It is worth noting my solution deviates from the norm. IdentityServer seems predicated on the idea that the authentication service is a standalone server, microservice style. Given the early stage of development I was at, having another server seemed like an annoyance, so I opted to have the authentication service as part of the API server. Really the only problem with this was it obscured the distinction between the ‘client’ (the JWT validation and authorization) and the ‘server’ (IdentityServer4) meaning it perhaps took a little longer than I’d have preferred to understand my authentication and authorization solution.

Using identity server is trivial – one line in the Startup.Configure: app.UseIdentityServer();. Set up, even for a basic solution, is a little more complex and will admit that to this day I do not fully understand scopes and the implications of them.

Supporting the server involves defining various resources in Startup. The scopes referenced in the Configure method end up in the scopes field in the JWT payload.

using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using System.IdentityModel.Tokens.Jwt;
using Microsoft​.AspNetCore​.Authentication​.JwtBearer;
using Microsoft.IdentityModel.Tokens;

public virtual IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddIdentityServer()
        .AddInMemoryIdentityResources(Auth.IdentityServerConfig.GetIdentityResources())
        .AddInMemoryApiResources(Auth.IdentityServerConfig.GetApiResources())
        .AddInMemoryClients(Auth.IdentityServerConfig.GetClients())
        .AddTemporarySigningCredential();
}
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IApplicationLifetime appLifetime)
{
    app.UseIdentityServer();
    // Configure authorization in the API to parse and validate JWT bearer tokens
    JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear();
    app.UseJwtBearerAuthentication(GetJwtBearerOptions());
    app.AllowScopes(new[] {
        IdentityServer4.IdentityServerConstants.StandardScopes.OpenId,
        IdentityServer4.IdentityServerConstants.StandardScopes.Profile,
        Auth.IdentityServerConfig.MY_API_SCOPE
    });
}

The configurations referenced in ConfigureServices link to a static class with a similar structure to that from the quick starts.

Testing

The final challenge with this set up was running integration tests with ASP.NET Core’s TestServer. The difficultly was that the authentication process would try to make a web request to the authentication server URL (e.g. http://localhost:5000). However because TestServer is not a real server listening on a port, then no authentication response would be received.

To resolve this an additional option was added to the JwtBearerOptions during Startup only for the integration tests. This class intercepts the authentication request and copies it to the TestServer’s client instance (using a static, which I’m not proud of). This is all illustrated below.

options.BackchannelHttpHandler = new RedirectToTestServerHandler();

public class RedirectToTestServerHandler : System.Net.Http.HttpClientHandler
{
    ///<summary>Change URL requests made to the server to use the TestServer.HttpClient rather than a custom one</summary>
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        HttpRequestMessage copy = new HttpRequestMessage(request.Method, request.RequestUri);
        foreach (var header in request.Headers)
            copy.Headers.Add(header.Key, header.Value);
        copy.Content = request.Content;

        Serilog.Log.Information("Intercepted request to {uri}", request.RequestUri);
        HttpResponseMessage result = TestContext.Instance.Client.SendAsync(copy, cancellationToken).GetAwaiter().GetResult();
        return Task.FromResult(result);
    }
}

Musings on Unit Testing and Architecture in .NET Core

One of the challenges I’ve found in architecture has been how to effectively mock the data layer for unit testing domain classes. I’ve worked with various combinations of tiers and repositories, and what I consider the optimum approach is to take a domain-first approach. In a domain-first approach we construct operations in terms of the models and interfaces needed to fulfill the operation, then rely on an overseer, the ‘dependency injector’ or ‘composite root’, to serve up objects that implement those interfaces. The nice thing about this approach is it allows for very granular operations which at their extreme can be single operation classes in the style used by the command pattern. This granularity fits well with SOLID design principles because a single operation should have clear responsibilities, we are injecting the dependencies, and we can define highly specific interfaces giving us excellent interface segmentation.

Typically a good chunk of these interfaces will be for accessing data, and the result of this approach would be a repository class something like

public class SomethingRepository : IGetSomethingByName, IGetSomethingByGuid, IGetSomethingByNameAndType, ...

This is often somewhat confusing because we’re encouraged to create repository classes as generic as possible into order to avoid repetition.

// A classic generic repository interface
public interface IRepository<Something> {       
  IEnumerable<Something> Get();
  Something GetById(int id);
  IEnumerable<Something> Find(Expression<Func<Something, bool>> predicate);
  void Add(Something something);
  void Delete(Something something);
  void Edit(Something something);
  void Save();
}

Already there is a mismatch. The domain behaviour expressed by the interfaces acts in terms of targeted methods like IGetSomethingByName.Get(string name) while the generic repository uses a more general Find(predicate). Some comprise must be made – either we let the domain know more about the data layer by getting it to specify predicates, thus reducing our domain-first approach and interface segmentation; or we extend the generic repository for Something with the tailored methods.

Then we get to more complex operations that involve multiple data-sources and we either have to get units of work involved, which now means sharing context between repositories which in turn makes creation (by injection) awkward; or we create wider scoped repositories more suitable for the whole bounded context which tends to reduce cohesion. And then we have to consider how to deal with transactions.

The point is, that after all this we’ve created a very extensive plumbing layer to fulfil two purposes: to get a gold star for architectural design; and to allow the domain to be effectively tested.

How do we implement the repository behemoth layer? If we’re dealing with a database then the default today is to get out Entity Framework because writing raw SQL comes with maintenance penalties. And here is where it all goes a little wrong…
Here is the opening paragraph on the Repository pattern from P of EAA:

A system with a complex domain model often benefits from a layer, such as the one provided by Data Mapper, that isolates domain objects from details of the database access code. In such systems it can be worthwhile to build another layer of abstraction over the mapping layer where query construction code is concentrated.

This is what Entity Framework is. When we use Entity Framework (and I’m thinking code-first here) we define a domain model and then we tell EF how to map that data to a schema e.g. how to map inheritance, keys, constraints, etc. The repositories are each DbSet in the DbContext, and the DbContext itself is a unit of work.

So if we create a custom repository layer that calls EF we’re basically trying to re-implement EF using EF, which is not a very good use of time. If instead we expressed our domain behavior in terms that EF understands, like IQueryable, then we could just use EF.

At this point you could argue that using DbContext as a dependency is not a well segregated interface at all, and overall I’d agree as EF doesn’t map to our domain interfaces. But the granularity of its methods allows us to express domain behavior in terms of domain objects and limited tools for manipulating those, so I feel satisfied it is a good clean boundary. And of course, we’re in business, so let’s not waste valuable time and mental resources on extra layers whose only purpose is to earn an architecture award.

But this lack of a concise interface is a problem for testing, because adequately mocking something of the scope of EF is an enormous challenge. And historically this is where having that extra layer wrapping EF was beneficial; even necessary.

Finally we’ve reached the tool that inspired this post.

In Entity Framework 7 there is a new feature, an in-memory database provider. To quote the docs:

InMemory is designed to be a general purpose database for testing, and is not designed to mimic a relational database.

With this tool our testing problem has gone. We can now effectively mock a DbContext by setting it up with pre-canned data, just as we would have via manual repository mocks, and then inject it into the test. It’s that simple: the same DbContext class used for production can be used in test by giving it a different database provider.

Here is the builder my unit tests use. NewGuid() is used to give the instance a unique name because, by default, the same in-memory database instance will be shared by all instances of a given context type.

var dbContextOptions = new DbContextOptionsBuilder<SqlContext>()
  .UseInMemoryDatabase(databaseName: "UnitTest" + Guid.NewGuid()).Options;

var ctx = new SqlContext(dbContextOptions);

.NET Core Secrets

Securing sensitive configuration information is one of those things that we know as developers is important, but so often gets deferred for more pressing commercial concerns, usually because of our confidence in the security infrastructure of our environments e.g. firewalls, VPNs. However in the field I’m heading into, security of personal information is important and expected to be audited, so with that commercial consideration in mind today I decided to tackle the challenge of securing configuration.

Secret Manager

There is a lot of chatter around the .NET Core Secret Manager but there appears to be two problems with it. Firstly, it is not a trusted store: “The Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store. It is for development purposes only. The keys and values are stored in a JSON configuration file in the user profile directory.”. Secondly, and more significantly I believe, it is user specific. That means that each user has their own credentials.

When I set up a development environment for a team I want it to be as uniform as possible for the whole team. A uniform environment makes it easier for team members to help each other, and makes it easier to script tools for automation. And many development resources will be shared, such as an AWS test instance.

Finally, this doesn’t help with production. For production the above website suggests using environment variables. Such variables are almost certainly stored in plaintext somewhere – in my case in Elastic Beanstalk configurations. Storing in plain-text is insecure and if nothing else is going to be a black mark on a security audit.

Extending .NET Core Configuration

What I want is sensitive information to be stored in an encrypted file where the encrypted file and the key are stored separately i.e. at least one of those is not checked into the source repository. I also still want to have different configurations available for different environments. It is also important that the file is relatively easy to modify.

What I propose is a custom configuration provider that is inserted into the ConfigurationBuilder which processes the other settings file when it is instantiated. The concept is outlined in this extension method:

public static IConfigurationBuilder AddEncryptedAndJsonFiles(this IConfigurationBuilder builder, string fileName, string basePath, bool optional, bool reloadOnChange = false)
{
    string jsonFilePath = builder.GetFileProvider().GetFileInfo(fileName).PhysicalPath;
    var encryptedConfiguration = new EncryptedConfigurationSource(jsonFilePath, basePath);
    encryptedConfiguration.UpdateStoredSettings();

    return builder
        .AddJsonFile(fileName, optional, reloadOnChange)
        .Add(encryptedConfiguration);
}

UpdateStoredSettings() will look through the appsettings file for keys starting with SENSITIVE_name. It will then add the name and corresponding value to the encrypted file and remove it from the appsettings file. The ConfigurationProvider returned by the IConfigurationSource.Build method will read the encrypted file and return a data dictionary of keys and values. The location of the key file will be set in the appsettings and read by both the source method and provider.

The extension method above will allow a simple replacement of AddJsonFile with AddEncryptedAndJsonFiles leaving Startup like this:

var builder = new ConfigurationBuilder()
    .SetBasePath(configBasePath)
    .AddEncryptedAndJsonFiles("appsettings.json", configBasePath, optional: true, reloadOnChange: true)
    .AddEncryptedAndJsonFiles($"appsettings.{env.EnvironmentName}.json", configBasePath, optional: true)
    .AddEnvironmentVariables();
Configuration = builder.Build();

Implementation

The implementation requires three classes as is standard for configuration providers:

  • a ConfigurationProvider which writes the properties into the dictionary used by consumers of configuration;
  • an IConfigurationSource which is the factory for ConfigurationProvider and where I opted to put the pre-processing method; and
  • an extension method for convenience.

The implementation uses AES for encryption. I considered deliberately using a slower method, but had trouble finding documentation and examples specific to .NET Core for symmetric encryption (as opposed to password hashing which is where those algorithms tend to be used).

Unlike the appsettings.json, the encrypted settings are stored in a single flat object, with the key being that used for configuration lookups e.g. configuration["parent:child"]. If a matching setting is found then it will overwrite the old one, allowing settings to be repeatedly updated.

One delightful problem I had was that the default IFileProvider refused to resolve paths above the base path, making it impossible to use for a relative path pointing outside the repository. As a result I had to pass in the base path, which feels like something of a hack.

A gist with the full source code can be found here