.NET Core Secrets

Securing sensitive configuration information is one of those things that we know as developers is important, but so often gets deferred for more pressing commercial concerns, usually because of our confidence in the security infrastructure of our environments e.g. firewalls, VPNs. However in the field I’m heading into, security of personal information is important and expected to be audited, so with that commercial consideration in mind today I decided to tackle the challenge of securing configuration.

Secret Manager

There is a lot of chatter around the .NET Core Secret Manager but there appears to be two problems with it. Firstly, it is not a trusted store: “The Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store. It is for development purposes only. The keys and values are stored in a JSON configuration file in the user profile directory.”. Secondly, and more significantly I believe, it is user specific. That means that each user has their own credentials.

When I set up a development environment for a team I want it to be as uniform as possible for the whole team. A uniform environment makes it easier for team members to help each other, and makes it easier to script tools for automation. And many development resources will be shared, such as an AWS test instance.

Finally, this doesn’t help with production. For production the above website suggests using environment variables. Such variables are almost certainly stored in plaintext somewhere – in my case in Elastic Beanstalk configurations. Storing in plain-text is insecure and if nothing else is going to be a black mark on a security audit.

Extending .NET Core Configuration

What I want is sensitive information to be stored in an encrypted file where the encrypted file and the key are stored separately i.e. at least one of those is not checked into the source repository. I also still want to have different configurations available for different environments. It is also important that the file is relatively easy to modify.

What I propose is a custom configuration provider that is inserted into the ConfigurationBuilder which processes the other settings file when it is instantiated. The concept is outlined in this extension method:

public static IConfigurationBuilder AddEncryptedAndJsonFiles(this IConfigurationBuilder builder, string fileName, string basePath, bool optional, bool reloadOnChange = false)
{
    string jsonFilePath = builder.GetFileProvider().GetFileInfo(fileName).PhysicalPath;
    var encryptedConfiguration = new EncryptedConfigurationSource(jsonFilePath, basePath);
    encryptedConfiguration.UpdateStoredSettings();

    return builder
        .AddJsonFile(fileName, optional, reloadOnChange)
        .Add(encryptedConfiguration);
}

UpdateStoredSettings() will look through the appsettings file for keys starting with SENSITIVE_name. It will then add the name and corresponding value to the encrypted file and remove it from the appsettings file. The ConfigurationProvider returned by the IConfigurationSource.Build method will read the encrypted file and return a data dictionary of keys and values. The location of the key file will be set in the appsettings and read by both the source method and provider.

The extension method above will allow a simple replacement of AddJsonFile with AddEncryptedAndJsonFiles leaving Startup like this:

var builder = new ConfigurationBuilder()
    .SetBasePath(configBasePath)
    .AddEncryptedAndJsonFiles("appsettings.json", configBasePath, optional: true, reloadOnChange: true)
    .AddEncryptedAndJsonFiles($"appsettings.{env.EnvironmentName}.json", configBasePath, optional: true)
    .AddEnvironmentVariables();
Configuration = builder.Build();

Implementation

The implementation requires three classes as is standard for configuration providers:

  • a ConfigurationProvider which writes the properties into the dictionary used by consumers of configuration;
  • an IConfigurationSource which is the factory for ConfigurationProvider and where I opted to put the pre-processing method; and
  • an extension method for convenience.

The implementation uses AES for encryption. I considered deliberately using a slower method, but had trouble finding documentation and examples specific to .NET Core for symmetric encryption (as opposed to password hashing which is where those algorithms tend to be used).

Unlike the appsettings.json, the encrypted settings are stored in a single flat object, with the key being that used for configuration lookups e.g. configuration["parent:child"]. If a matching setting is found then it will overwrite the old one, allowing settings to be repeatedly updated.

One delightful problem I had was that the default IFileProvider refused to resolve paths above the base path, making it impossible to use for a relative path pointing outside the repository. As a result I had to pass in the base path, which feels like something of a hack.

A gist with the full source code can be found here

Typeahead and Aurelia

The pace of my blogging has dropped somewhat now I’m on a full-time engagement, and so far there haven’t been too many stories to share from the engagement. But the one that was perhaps most frustrating and in need of sharing was getting Twitter typeahead working with Aurelia.
I will prefix this with a disclaimer – these are steps as I best recollect them and I haven’t re-done them all as I’m not in the mood for breaking something that is (finally) working.

No longer maintained?

The first obstacle was finding the source code for typeahead. The sourced linked from the official looking page (linked earlier) points to this GitHub repository which, at the time of writing, had no commits for almost two years. However various google searches intimated more recent changes, and eventually it became clear that most people were referring to this which, in its own words, “is a maintained fork of twitter.com’s autocomplete search library, typeahead.js.”

Getting the library

The working typeahead repository can be obtained through npm:
npm install corejs-typeahead --save

This does not come with an index.d.ts so if you’re like me and using TypeScript then you’ll need a typings file. Despite the strong possibility it is for the wrong typeahead repository, the typings file returned by typings search typeahead seemed to work fine once installed by typings install dt~typeahead -GS

Typeahead is dependent on jQuery v2, so after fetching via npm, in my package.json I fixed the version as follows "jquery": "^2". Finding a typings for jQuery was straightforward but getting TypeScript to be happy with it was harder. In the end I found simply import 'jquery'; to be the most effective way of including it in a TypeScript file. Then there was the challenge with a collision on the $ variable (with angular-protractor) which I could only resolve by editing the typings file for jQuery:

declare module 'jquery' {
    export = jQuery;
}
declare var jQuery: JQueryStatic;
//declare var $: JQueryStatic;

This does mean jQuery must be referenced as jQuery(...) rather than $(...) in source code, but personally I don’t mind – as good as jQuery is, my sensitivities find there to be something a tiny bit wrong with a library deciding it can own a single letter global variable.

Aurelia Bundles

Typeahead isn’t really a module and thus the TypeScript declaration file doesn’t export one, but it does for Bloodhound which is hard to avoid if you want to do anything useful with typeahead. To import these they have to be included in the aurelia bundle, and best as I can tell, they have to be separate. So my aurelia.json looks like this:

"jquery",{
    "name": "Bloodhound",
    "path": "../node_modules/corejs-typeahead",
    "main": "dist/bloodhound",
    "deps": ["jquery"]
},{
    "name": "typeahead",
    "path": "../node_modules/corejs-typeahead",
    "main": "dist/typeahead.jquery",
    "deps": ["jquery", "Bloodhound"]
},

To actually use them, they can be fully imported as follows:

import 'jquery';
import 'Bloodhound';
import 'typeahead';

Bootstrap Integration

At this stage typeahead was working, but didn’t fit in with the UI which was otherwise styled via Bootstrap 3. The advice from the internet was to use Bootstrap-3-Typeahead which seemed fine initially, but I simply couldn’t get working with remote data sources. After a lot of stepping through library source code and periodic googling I discovered this bridge simply didn’t support remote (according to that ticket it may now have been fixed).

So instead of using the bridge, I got the styles from here and applied a couple of modifications:

.tt-hint {
    border: 0
}
/* Support inline use */
.form-inline span.twitter-typeahead {
  width: auto;
  vertical-align: middle;
}

The Code

The end result is an aurelia element that does typeahead. The code for that is in a gist here

The element requires a prepare-query function which sets the remote url and applies any necessary headers. This is an example of one:

prepareQueryFn(query, settings: JQueryAjaxSettings) {
    settings.url = this.apiUrl + 'search?query=' + encodeURIComponent(query);
    settings.headers = { 
        'Accept': 'application/json',
        'Authorization': this.getAuthToken()
    }
    return settings;
}

Perils of AddDbContext

The following snippet is the suggested approach to injecting an Entity Framework context in ASP.NET Core, taken from the docs page on Dependency Injection in ASP.NET Core:

services.AddDbContext<SqlContext>(options => 
    options.UseMySql(Configuration.GetConnectionString("DefaultConnection")));

This means that whenever you have a constructor that includes SqlContext, the runtime will provide an instance of SqlContext without the developer having to type new anywhere.

Dependency injection in ASP.NET Core comes in three flavours: Transient, meaning a new object (e.g. SqlContext) is called each time a constructor requires the dependency; Scoped, meaning the same object is used for all dependencies of that type during the current web request; and Singleton, where only one instance of the object is ever created. The default for AddDbContext is Scoped meaning that during a web request all classes will be accessing the same SqlContext.

When Entity Framework fetches a record from the database it caches it so if the application requests it again it returns the results from its cache. If someone else updated the database during the request, their change won’t be seen by the first caller. For short and stateless web requests this seems like a reasonable optimization. While technically the data you’re seeing is inconsistent, it was valid a few milliseconds ago, so any permissions that might have been revoked were valid then, and if you try and write anything then optimistic concurrency will alert you to a problem.

Problems

However care must be taken as it is very easy to end up getting cached values when that wasn’t intended.

If the dependent object is a singleton, then it will receive a context instance when the object is first created, and hold onto that context until the server recycles. During that time anything it reads will be cached in the context and any changes to the database ignored. This is particularly problematic where there are balanced web servers where the singletons on two different servers will be unaware of the changes the other made.

This approach is also quite opaque. It is not obvious from the code, compared to say a using statement, the lifetime of the context and therefore what side-effects might occur among the many classes in a given web request using this context. This is true of any object with scoped lifetime and is why I tend to avoid using scoped.

While not really a problem with its usage, that we injecting a concrete class rather than an interface doesn’t really follow the basic pattern of dependency injection. It is not terribly difficult to create an interface for the context, but as noted earlier, it is not the standard being documented. I assume the reason for this is that the testability problem with injecting a concrete context has been removed by the introduction of DbContextOptionsBuilder.UseInMemoryDatabase().

A Solution

Frankly I prefer more control and transparency so I’ve returned to the time-honored tradition of having a data factory so that my classes can create a database context when they want to. To do this the factory uses the IoC container to create contexts on demand.

public interface ISqlContextFactory { SqlContext NewContext { get; }}

public class SqlContextFactory : ISqlContextFactory
{
    private Func<SqlContext> _getSqlContext;
    public SqlContextFactory(Func<SqlContext> getSqlContext)
    {
        _getSqlContext = getSqlContext;
    }
    public SqlContext NewContext { get { return _getSqlContext(); } }
}

This could be done by passing the IoC container into the factory, but I chose to pass the factory a function which generates the contexts, leaving options configuration in Startup.

Edit 16-March-2017: The original code source caused problems with the mysql connector similar to those discussed on Connection reuse here. The following code has been updated so the options are set for every request, rather than just once at startup, and this appears to have fixed the issue.

services.AddSingleton<ISqlContextFactory, SqlContextFactory>(provider => {

    string mysqlConnStr = Configuration.GetConnectionString("Mysql");

    return new Data.SqlContextFactory(() => {
        var options = new DbContextOptionsBuilder<Data.SqlContext>();
        options.UseMySql(mysqlConnStr);
        return new Data.SqlContext(options.Options);
    });
});

By removing the AddDbContext we have broken migrations and fixing this requires a little unsavory use of statics. The dotnet ef command line seems to run the Startup class but does not use the dependency injector, so instead I created a little class just for the migrations to use, and set the static connection string in Startup.ConfigureServices() so that the migration class doesn’t have to repeat the configuration loading code.

using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;

/// This is ONLY for use by dotnet command line tool
public class SqlContextMigrationsTarget : IDbContextFactory<SqlContext>
{
    public static string ConnectionString;
    private DbContextOptionsBuilder<SqlContext> _builder = new DbContextOptionsBuilder<SqlContext>();

    public SqlContextMigrationsTarget() {}

    public SqlContext Create(DbContextFactoryOptions options)
    {
        _builder.UseMySql(ConnectionString);
        return new SqlContext(_builder.Options);
    }
}

// and in Startup.ConfigureServices()
SqlContextMigrationsTarget.ConnectionString = Configuration.GetConnectionString("Mysql");

End of the Functional Road (for now)

After three months hiatus from ‘work’, I’m back focusing on a specific project. While I tried to do it work with F# and Suave I found that my current (lack of) knowledge with F# meant I was spending more time trying to understand the framework than interact with it. So, out of a sense of pragmatism, I’ve returned to C# and am now expanding my horizons through the use of .NET Core, Aurelia, Docker, developing in VS Code, and targeting Linux.

Sad as it is, this is the end of the functional domain project for now. I really would like to use F# in a commercial sense at some stage, but for now the key limitation has been a desire to use .NET Core and the F# tooling not quite being ready.

The source code remains available here: https://github.com/winterlimelight/FunctionalDomainProject