Automated Testing Priorities

There’s a common theme in automated software testing that the greatest effort should go into unit tests, lesser into integration tests, and least into UI tests. This is known as the testing pyramid. However I’m not convinced this is the best use of automated test developers for web applications, and I believe this is because the nature of web standards and a commonly used web application architecture invalidate some of the assumptions behind the pyramid.

Testing Goals

Before we continue we need to state the goals of automated testing. In my mind they are:

  1. Validate the correctness of the thing under test
  2. Allow refactoring of the thing under test

Note that correctness includes ‘non-functional’ things, like authorization security.

From a business perspective, we want to know that the product works as intended. Working as intended means that the application performs correctly when used through its interfaces. This would suggest that UI tests are the most important, which is the opposite of conventional wisdom.

The reason often given for placing less focus on UI tests is that they have been considered notoriously fragile. However I posit that this has been due to the nature of the interfaces being tested, which have tended to make identifying and interacting with UI elements automatically very hard; e.g. having to use control ids with WinForms applications. I’m also suspicious that less focus on UI tests is a consequence of Agile methodologies that insist on jamming all testing into the same cycle as development, resulting in automation trying to be written against a UI in an extreme state of flux.

Unit Test Fragility

One problem I have with unit testing is that developers are encouraged to test the internals of the unit. This happens when mock objects are checked to see if certain methods were called on the mock.

The purpose of functions and classes are that they expose some contract and hide the details of how that contract is fulfilled. Testing how a unit is doing its work means examining inside the black box, which defeats the purpose of using testing to support refactoring because now we can’t make a change to the implementation of a unit without breaking its tests.

UI Test Fragility

In his 2012 article Fowler says:

An enhancement to the system can easily end up breaking lots of such tests, which then have to be re-recorded. You can reduce this problem by abandoning record-playback tools, but that makes the tests harder to write. Even with good practices on writing them, end-to-end tests are more prone to non-determinism problems, which can undermine trust in them. In short, tests that run end-to-end through the UI are: brittle, expensive to write, and time consuming to run.

I believe that some of these assumptions are less valid in modern web test automation.

Automated web testing tends to be hand-written because (in my experience) the recording tools can create quite fragile paths, usually because they don’t know what the least variant information is. It is straight-forward to hand-write UI tests thanks to CSS selectors which are easy to use, well-supported, and when done simply (i.e. via id and class selectors rather than paths) aren’t hugely prone to change. These selectors are usually wrapped into page objects that further insulate the tests from changes.

The HTML DOM also exposes an event model which allows tests to mimic the vast the majority of UI actions, removing the complexity of older style tools which involved a lot of mouse-coordinates and button states.

And finally, in web development, UI testing has the added benefit of enabling testing across multiple browsers – something less applicable to downloaded applications.

However I agree that they remain time-consuming to run, and if there are lots of developers committing to the repository then having your continuous integration run on every commit may not be possible, reducing the benefit of the tests for quickly catching problems.

Middle-Ground – Integration Testing the API

It is increasingly common for web applications to be built as a web API and a web (JavaScript) client. This is my personal preference over server-side rendering as it nicely decouples the presentation from the logic and allows the application to more easily integrate with other applications. There is some development overhead in this approach, but given most web pages perform some dynamic interaction with the server thus requiring some level of client richness, this overhead is quite minimal.

Having an API provides an excellent place for automated testing. An API is a contract and will express most, if not all, of the business rules through its inputs and outputs. It also requires basic security, and allows validation and authorization to be checked. It can be easily extended to run more extensive security testing (i.e. by manipulating HTTP headers and sending malicious data) and performance tests.

Integration testing the API doesn’t mean a full environmental setup is required. It is still reasonable to use mocks for calls that are slow or resources that aren’t available. For instance my integration tests use .NET Core’s TestServer rather than running a web server, EF’s UseInMemoryDatabase rather than instantiating a database, and stub out AWS service calls. These are reasonable compromises because I’m confident those areas will perform to their contracts.

Conclusion

This is my ‘testing pyramid’ from highest to lowest priority:

  1. API integration tests
  2. Integration or unit tests for things that can’t be reached from the API
  3. UI tests for success scenarios

In my current application I have 98% code coverage and 90% branch coverage (largely ruined by not throwing exceptions inside all the C# using statements) of my business layer using the first two items on my list, and it has enabled considerable evolution and refactoring of the code-base over the last six months.

Musings on Unit Testing and Architecture in .NET Core

One of the challenges I’ve found in architecture has been how to effectively mock the data layer for unit testing domain classes. I’ve worked with various combinations of tiers and repositories, and what I consider the optimum approach is to take a domain-first approach. In a domain-first approach we construct operations in terms of the models and interfaces needed to fulfill the operation, then rely on an overseer, the ‘dependency injector’ or ‘composite root’, to serve up objects that implement those interfaces. The nice thing about this approach is it allows for very granular operations which at their extreme can be single operation classes in the style used by the command pattern. This granularity fits well with SOLID design principles because a single operation should have clear responsibilities, we are injecting the dependencies, and we can define highly specific interfaces giving us excellent interface segmentation.

Typically a good chunk of these interfaces will be for accessing data, and the result of this approach would be a repository class something like

public class SomethingRepository : IGetSomethingByName, IGetSomethingByGuid, IGetSomethingByNameAndType, ...

This is often somewhat confusing because we’re encouraged to create repository classes as generic as possible into order to avoid repetition.

// A classic generic repository interface
public interface IRepository<Something> {       
  IEnumerable<Something> Get();
  Something GetById(int id);
  IEnumerable<Something> Find(Expression<Func<Something, bool>> predicate);
  void Add(Something something);
  void Delete(Something something);
  void Edit(Something something);
  void Save();
}

Already there is a mismatch. The domain behaviour expressed by the interfaces acts in terms of targeted methods like IGetSomethingByName.Get(string name) while the generic repository uses a more general Find(predicate). Some comprise must be made – either we let the domain know more about the data layer by getting it to specify predicates, thus reducing our domain-first approach and interface segmentation; or we extend the generic repository for Something with the tailored methods.

Then we get to more complex operations that involve multiple data-sources and we either have to get units of work involved, which now means sharing context between repositories which in turn makes creation (by injection) awkward; or we create wider scoped repositories more suitable for the whole bounded context which tends to reduce cohesion. And then we have to consider how to deal with transactions.

The point is, that after all this we’ve created a very extensive plumbing layer to fulfil two purposes: to get a gold star for architectural design; and to allow the domain to be effectively tested.

How do we implement the repository behemoth layer? If we’re dealing with a database then the default today is to get out Entity Framework because writing raw SQL comes with maintenance penalties. And here is where it all goes a little wrong…
Here is the opening paragraph on the Repository pattern from P of EAA:

A system with a complex domain model often benefits from a layer, such as the one provided by Data Mapper, that isolates domain objects from details of the database access code. In such systems it can be worthwhile to build another layer of abstraction over the mapping layer where query construction code is concentrated.

This is what Entity Framework is. When we use Entity Framework (and I’m thinking code-first here) we define a domain model and then we tell EF how to map that data to a schema e.g. how to map inheritance, keys, constraints, etc. The repositories are each DbSet in the DbContext, and the DbContext itself is a unit of work.

So if we create a custom repository layer that calls EF we’re basically trying to re-implement EF using EF, which is not a very good use of time. If instead we expressed our domain behavior in terms that EF understands, like IQueryable, then we could just use EF.

At this point you could argue that using DbContext as a dependency is not a well segregated interface at all, and overall I’d agree as EF doesn’t map to our domain interfaces. But the granularity of its methods allows us to express domain behavior in terms of domain objects and limited tools for manipulating those, so I feel satisfied it is a good clean boundary. And of course, we’re in business, so let’s not waste valuable time and mental resources on extra layers whose only purpose is to earn an architecture award.

But this lack of a concise interface is a problem for testing, because adequately mocking something of the scope of EF is an enormous challenge. And historically this is where having that extra layer wrapping EF was beneficial; even necessary.

Finally we’ve reached the tool that inspired this post.

In Entity Framework 7 there is a new feature, an in-memory database provider. To quote the docs:

InMemory is designed to be a general purpose database for testing, and is not designed to mimic a relational database.

With this tool our testing problem has gone. We can now effectively mock a DbContext by setting it up with pre-canned data, just as we would have via manual repository mocks, and then inject it into the test. It’s that simple: the same DbContext class used for production can be used in test by giving it a different database provider.

Here is the builder my unit tests use. NewGuid() is used to give the instance a unique name because, by default, the same in-memory database instance will be shared by all instances of a given context type.

var dbContextOptions = new DbContextOptionsBuilder<SqlContext>()
  .UseInMemoryDatabase(databaseName: "UnitTest" + Guid.NewGuid()).Options;

var ctx = new SqlContext(dbContextOptions);

Dependency Injection

My project architecture has been setting up to allow dependency injection. For instance, the commands take repository instances as arguments. But the approach I’ve been preparing is very object-oriented, and in my notes I had mused on thoughts about how partial application would be a more functional way of doing this. However exactly how to structure this has eluded me. Thankfully the ever understandable F# for Fun and Profit just created a post Functional approaches to dependency injection that bridges this gap, so now I’m going to walk through my conversion from interfaces to function types.

Interfaces to Function Types

The repositories the simplest place to start. At present the interface for the template repositories is:

type ITemplateReadRepository =
    abstract member FindById: System.Guid -> Template option

type ITemplateWriteRepository =
    abstract member FindById: System.Guid -> Template option
    abstract member Save: Template -> unit

Changing these to function types means replacing each member with a function type.

I firmly believe in read-write separation so it’s important that there is a distinction made between finds made by the write system for the purpose of identity and validation, and finds made by a read system for querying. So despite having identical signatures, I like the concept of different types for FindById.

type FindTemplateById = System.Guid -> Template option
type FindTemplateByIdForValidation = System.Guid -> Template option
type SaveTemplate = Template -> unit

In a large project these would likely be separated into different modules purely for the purpose of code organization.

Passing partials

The current implementations of these methods directly instantiate a data context, which means they create a dependency, which is what we’re trying to avoid.

member this.FindById (id:Guid) = 
    let dc = Sql.GetDataContext()
    // use dc to find template

In object-oriented dependency injection the domain class would have a dependency on some IRepository and the IoC container would create a repository instance passing in the connection information. In functional programming this option is not available, so the dependencies have to be passed as function arguments meaning we need a method with this signature:

DbContext -> Guid -> Template option
// for example, the persistence method:
let findTemplateById (dc:DbContext) (id:Guid) = ...

However this means the caller has to know how to create the DbContext dependency. That is likely not the responsibility of the caller, so we need another abstraction that manages the dependencies and only requires caller to provide variables that it is responsible for. We can do this by providing a function which can convert between the signature understood by the caller and the signature of the target method.

// Persistence method: DbContext -> Guid -> Template option
let FindTemplateById dc id = ...  

// Domain method: (Guid -> Template option) -> Guid -> Template option 
let GetTemplate findById id = 
    findById id

// Caller
let dc = Sql.GetDataContext()
let findTemplateByIdWithDc = FindTemplateById dc // Signature Converter
let res = GetTemplate findTemplateByIdWithDc id

The converting function, findTemplateByIdWithDc, is a partially applied function of FindTemplateById because we have not specified all of the arguments, leaving the second (id) to be set when findById is called.

In my project the DbContext instance is created in the program.fs which is a layer higher than the caller function (my Api level) above. This same pattern can be applied so that the DbContext is passed transparently through the Api level as well as the Domain. For the sake of organization, all of these ‘signature converters’ are placed into a file given the name CompositionRoot. That file is defined immediately before the first file that uses it, in this case before program.fs. The end result looks something like the following, which is a snapshot of the full stack used for the GET template request.

type FindTemplateById = System.Guid -> Template option // domain/persistence 
type IGetTemplate = System.Guid -> Template option // api/domain

// Persistence.fs
module Persistence.TemplateReadRepo =
    let findById dc id = 
        // use dc ...

// (Domain)Operations/Template.fs
module Operations.Template =
    let GetTemplate (findById:FindTemplateById) id = 
        findById id

// Api/TemplateController.fs
module Api.Template =
    let getTemplate (getTemplateById:IGetTemplate) (id:Guid) : WebPart =
        match (getTemplateById id) with ...

// CompositionRoot.fs
module Operations =
    let getTemplate dc = AmApi.Operations.Template.GetTemplate (Persistence.TemplateReadRepo.findById dc)

module ApiMethods =
    let getTemplate dc = Api.Template.getTemplate (Operations.getTemplate dc)

// Program.fs
let route dc =
    choose [
        pathScan ... (ApiMethods.getTemplate dc)
        ...

The composition root creates partial functions like Operations.getTemplate dc which mean that the argument given to Api.Template.getTemplate still conforms to the signature it requires while the information about the context travels to the domain, and in a similar fashion to the persistence call where it is finally used.

SqlProvider

Having been some time with an in-memory mutable dictionary serving as a data store, the time seemed right to introduce a permanent data store.

Selecting a data store

Choosing a store for a solution is a key architectural decision and requires evaluating a huge set of factors including requirements, domain-store match, cost, scale, read vs write counts, eventual consistency vs availability, transactional integrity, etc.

There are a number of different stores that are reasonable for this project. Ideally I’d try implementing them all, however I don’t expect that to happen in the short-term. These are some options:

  • Document store e.g. MongoDB. This may be appropriate given our aggregates (template and asset) are quite complex objects.
  • Search index e.g. ElasticSearch. If there are many orders of magnitudes more reads than writes then optimizing for querying using a search index may be appropriate.
  • Event store. An event store, typically implemented manually on flat files, records all the business events triggered by API or system actions. It is often used in a CQRS system in concert with a read-specific store. Having read/write separation in our stack allows for this option.
  • Relational database e.g. SQL Server. Provides transactions, is the most mature, and the best supported by libraries.

Choosing between these relies on criteria that don’t exist because this isn’t a commercial project, so because I’m learning and trying to reduce the number of learning variables I chose the tried-and-true, a relational database.

As an aside, as an architect we also want to allow for future contingencies where it is practical. By providing a clear persistence layer in the code and comprehensive integration tests, we allow the choice of store to be re-evaluated in the future (providing the organization is prepared for the cost).

SQLProvider

F# has an excellent feature, Type Providers, that generate strongly typed code at compile time for interacting with data sources. This page has a nice comparison of some type providers for F#. For no particular reason I selected SQLProvider

As mentioned earlier, the domain aggregates are relatively deep, so they map to multiple tables, which is why in a production solution I’d lean towards a non-relational store. Here is a comparison of the domain type template, and the tables that store it in SQL (constraints excluded):

type FieldValue = 
    | StringField of string
    | DateField of System.DateTime
    | NumericField of float

type FieldDefinition = {
    Id: System.Guid
    Name: string
    Field: FieldValue
}

type Template = {
    Id: System.Guid
    Name: string
    Fields: FieldDefinition list
    MaintenanceProgramId: System.Guid option
}
CREATE TABLE [dbo].[FieldValue](
    [FieldValueId] [uniqueidentifier] NOT NULL,
    [FieldDefinitionId] [uniqueidentifier] NOT NULL,
    [AssetId] [uniqueidentifier] NULL,
    [ValueType] [tinyint] NOT NULL,
    [StringValue] [nvarchar](max) NULL,
    [DateValue] [datetime] NULL,
    [NumericValue] [real] NULL)

CREATE TABLE [dbo].[FieldDefinition](
    [FieldDefinitionId] [uniqueidentifier] NOT NULL,
    [Name] [nvarchar](255) NOT NULL,
    [TemplateId] [uniqueidentifier] NOT NULL)

CREATE TABLE [dbo].[Template](
    [TemplateId] [uniqueidentifier] NOT NULL,
    [Name] [nvarchar](255) NOT NULL,
    [MaintenanceProgramId] [uniqueidentifier] NULL)

The end result is that we end up with some fairly complex queries and mapping logic. One of the sources of complexity is that the FieldValue table is used both for default template field values (when AssetId is null) and asset field values.

The query contains two left outer joins because a template may exist without any fields. This is shown using the rather cryptic (!!) operator. It would be nice if a more pleasant name could be used for this operator.

The mapping between discriminated unions and the underlying store can be handled by joining to a different table for each case, or by the approach used here which is to have a different column for each case. Using different columns makes the query easier, but results in a sparse table. Given we only have three cases the sparseness of the table shouldn’t be a big space penalty.

The read mapping is shown below. The full code is available here.

let private mapSingleTemplate (rows:TemplateQueryResultSet list) : Template =
    let fields = [ for row in rows do
                    let (_, defn, value) = row
                    if defn.FieldDefinitionId <> System.Guid.Empty then // empty guid means template has no fields
                        yield {
                            Id = defn.FieldDefinitionId
                            Name = defn.Name
                            Field = match value.ValueType with
                                    | 1uy -> StringField(value.StringValue.Value)
                                    | 2uy -> DateField(value.DateValue.Value)
                                    | 3uy -> NumericField(float value.NumericValue.Value)
                                    | _ -> failwith "Unknown field type"
                        }]
    let (templateCols, _, _) = rows.Head
    {
        Id = templateCols.TemplateId
        Name = templateCols.Name
        Fields = fields
        MaintenanceProgramId = templateCols.MaintenanceProgramId
    }

let private templateByIdQuery (dc:DbContext) id : System.Linq.IQueryable =
    query { 
        for template in dc.Dbo.Template do
        // (!!) means left outer join
        for fieldDef in (!!) template.``dbo.FieldDefinition by TemplateId`` do
        for fieldVal in (!!) fieldDef.``dbo.FieldValue by FieldDefinitionId`` do
        where (template.TemplateId = id && fieldVal.AssetId.IsNone)
        select (template, fieldDef, fieldVal)
    }

let private templateById id : Template option =
    let dc:DbContext = Sql.GetDataContext()
    let rows = templateByIdQuery dc id |> Seq.toList
    if [] = rows then None else Some (mapSingleTemplate rows)

Considering Architecture

I’m getting through Expert F#, and was starting to consider how a domain model might be implemented. But I think before that it is important to consider where that model fits into the solution, which requires us to think about architecture.

I must say that my disorganized thoughts were put into a much clearer perspective by the F# for fun and profit post Organizing modules in a project and you will likely find that a more practical and clearer explanation than I’m about to make.

Architecture in general

Any solution beyond a few hundred lines of code needs an architecture to organize the code. An architecture provides hard rules and soft guidelines on how different parts of a solution should interact, with the primary goal of making the solution maintainable. A maintainable solution is one that is understandable, and therefore easier to work on and more secure; is slower to accumulate debt; and is more testable and therefore of higher quality.

An architecture creates organization by separating unrelated concerns, controlling message paths, and hiding information. The goal of this project is to create an API for an asset management solution so we are implementing a web service, and the most common organization for a web service architecture is to split the solution into layers. The previous post, ASP.NET Core with F#, briefly considered an implementation for the top layer – the web interface. Beyond that we must also consider:

  • domain – the business rules and (for an API) the core business value;
  • persistence – how data is stored and retrieved;
  • infrastructure – cross-cutting concerns such as user context, configuration, logging.

In the most naive implementation, this creates the following architecture, where each layer depends on the one above:

archstack1

Domain Layer

I believe an architecture should be designed from the domain outwards because the domain is where the core business value resides. The domain comprises groups of data, which I will refer to as data objects, and domain operations which create or act upon the data. In an OO domain model the relevant operations are stored with the relevant data objects, although this tends to get messy.

Coming from C#, F# adds a new constraint: that types must be declared before they are used. This is not as different as it first seems since a domain focused solution (even in an OO stack) should define its data and behavior separate from its implementation to allow for unit tests (not to mention all the other benefits of clear architectural layers). In C# this means defining domain dependencies as interfaces, then injecting the concrete implementations once they are defined.

As F# is strongly typed, in order to have unit tests we must be able to provide at least two different objects (the concrete and a mock) that share a type. The end result is identical to C#: we must create interfaces to express behavior.

So the domain operations depend on:
1. data objects;
2. interfaces defining operations/services that must be provided. Some of those behaviors will be other domain operations and for the sake of consistency it might be desirable to include all the domain operation interfaces in this dependency.

Other Layers

In defining operations/services interfaces, the domain layer has essentially created the persistence and infrastructure interfaces.

With the domain nicely isolated, the remainder of the architecture must now provide what it needs. We expect that the concrete classes for these dependencies will be largely in the persistence and a little in the infrastructure. This results in a network as follows, with the addition of a startup layer which creates key context classes, such as the dependency injector.

archnetwork2

At this point nothing here is F# specific – and that is a good thing! A good architecture should allow implementation decisions to be made as late as practical.

The end result is we now have a place for our domain data objects and operations, and we can begin to define those.

Functional Domain Project Intro

The first project I plan to undertake is creating an architecture with a functional domain model. By ‘functional’ I mean using the functional programming paradigm, i.e. focusing on expressions rather than instructions.

The (long) rationale

In my career, where there has been an architecture at all, it has tended to be some form of object oriented domain model. In such a model each domain concept (I’m avoiding saying entity due to it’s Domain Driven Design implications) is represented by a class that exposes all its properties, a collection of get/find methods, save method, and some advanced manipulation that reflects more complex business rules. Eventually someone realizes that a given class is getting too complex and creates a separate service for that behavior. And so we end up with a set of heavily intertwined classes representing some combination of data and operations that is appropriate to the business domain.

I find this approach troubling for many reasons.

Firstly, regardless of the original intentions of the architect, these classes always end up getting big. By big I mean that the combination of possible states of the class’s fields makes it hard to reason about what will happen when a change is made in the class. This means that developers are left relying on unit tests (should they even cover the area of concern) to ensure the integrity of changes. I’m not disputing the value of tests, but I believe that better outcomes can be achieved by understanding the code and therefore being able to simplify it, rather that just painting on another method. Simplification means less code and that is better for maintenance, while code that can’t be understood can’t be maintained and is therefore technical debt.
Basically the class fields start to exhibit all the failings of global variables.

Secondly, it isn’t cohesive. A class with high cohesion is one with as few responsibilities as possible. These classes are responsible for reading data, writing data, and performing many different operations, which makes them very non-cohesive. It strikes a nerve with me that people are happy to violate cohesion by putting reading and writing code into the same class when they are such different operations: writing deals with validation, consistency, error handling; while reading deals with filtering, paging, caching; and beyond that we still have business rules and operations that act across classes.
So I believe that reading and writing classes should be separate, despite the vitriolic sentiments of the OO community calling this approach ‘anemic’.

A third issue is one I mentioned earlier: when should operations be moved from the concept classes into their own one? There simply isn’t a good line, and this is a problem because although it might seem obvious to the architect by virtue of their experience, it won’t to an intermediate developer, and so the code will go into the concept class it make it more unwieldy and less cohesive. Additionally, if the operation is moved into another class, can it access all the data it needs, or is that hidden within concept class?

Finally there is testability. Establishing all the states of the object necessary to thoroughly test is hard, especially when some of the fields are private.

I’m told it’s important to break-up lots of text to help people pause and regroup while reading, so here is a random picture of a plane that I took years ago.
p1000491

So what’s my suggestion?

If we take the concept class and split the reading and writing functionality into two classes, then all the data needs to be accessible to both classes, necessitating the creation of a data class, a.k.a. the DTO.

At this point I imagine OO proponents yelling “information hiding” at me (largely because some have), but the reality of the systems I’ve worked with is there is seldom any information that is hidden (let alone needs to be hidden) behind complex methods, and where the data needs manipulation to match the repository format, simple property accessors are fine.

Given we are in the world of web development and objects are usually created for a specific request, there seems very little value in keeping state because it probably won’t change anyway. This raises the question: why allow the state to be changed at all? If the state is probably going to be fixed from the moment the object is created, either from data submitted by the user or via repository access, then let’s enforce that by making the state immutable. Now we get all the benefits of immutability, like thread safety, and a clear ability to understand the object because anytime we perform an operation to it or with it we will get the same result.

The end result is a functional domain model, where data is held in an immutable object, and operations are expressed by functions acting on that data.

Implementation

Given my background is with .NET I’ve decided to do this using F#. To that end I’m presently making my way through Expert F# 4.0 by Syme, Granicz and Cisternino. I’m also hoping to using ASP.NET Core for this, to become more familiar with that.

I’m going to create a basic asset management system to illustrate this. One of the products on worked on around 9 years ago was an asset management system, so the domain for this will draw on what I can vaguely remember from then. The next post will introduce this problem domain.

It will likely be some time before I’m ready to tackle the meat of the project, but before then I’ll try to go into more detail on the problem domain, and get setup with ASP.NET Core in F#.