Measurement and Agile Software Development

Introduction

I’m going to start this politically, but I promise it’ll get to software development. The trigger for this scribbling of thoughts was an article discussing the under-funding of many areas of the public sector and the quote from the finance spokesperson for New Zealand’s recently-ousted opposition party: “… the government should be thanking [the] National [party] for inheriting such a strong economy.” And it struck me that economic performance was the sole benchmark by which they gauged success. In reality, the country is vastly more complex than one set of economic indicators, and different people have very different perspectives on what constitutes success.

The ‘duh’ disclaimer

As I’ve said in some previous articles, none of this will be new to anyone who has spent, studied, or even thought about management. And it certainly isn’t the first time I’ve thought about it, but the above article engaged some dormant mental spirit to write things down 🙂

You are what you measure

Different people’s values mean that what they consider important and unimportant will vary and that is fine and healthy. The challenges with measurement are the consequences of measuring and how people’s behavior changes in response to the measure.

To take a non-software example, the New Zealand education system places strong emphasis on success at NCEA achievement, which has translated into students being encouraged to take easier courses or teachers being encourage to teach towards the tests. In this case the goal of giving students the best high school education has been subverted by a measurement which effectively demands certain pass rates.

The classic example in software development is measuring lines of code. Lines of code is a basic metric for measuring the overall size and therefore likely cost of learning and maintaining a code base. It is an appalling measure of programmer productivity: good programmers will write less code through reuse; refactoring may end up removing code altogether; and on the other hand, readability is far more important than concision.

Thankfully I believe the industry is well past measuring productivity by LoC, or even the highly amorphous function points. However the beast is far from slain, for instead we have story points and velocity.

Agile Software Development

Agile Software Development, according to Dave Thomas, author of The Pragmatic Programmer and co-author of The Manifesto for Agile Software Development, can be summarized by this process:

  • find out where you are
  • take a small step towards your goal
  • adjust your understanding based on what you’ve learned
  • repeat

And when faced with alternatives that deliver similar value, take the path that makes future changes easier.

This is very idealistic and quickly crashes into commercial reality where managers, usually on behalf of customers, want to know: when will it be ‘done’ and what will it cost? Of course, this ignores all the benefits of learning-as-we-go, Lean style (which is essentially the same thing as agile software development but applied to business), and that you get much better, albeit far less predictable-at-the-outset, outcomes than any upfront planning based process. But we can’t really ask everyone to be rational can we?

Nevertheless, marketing release dates and the like meant we had to invent ways to measure progress and estimate ‘completion’ (I keep using inverted commas because I think we all know that done or complete are very subjective terms). And so Agile (sorry Dave T, I’m going to be using it as a noun) planning evolved from concepts of managing risk and uncertainty via loose estimation in Agile Estimating and Planning to full blown methodologies that are so militaristic they require specialized commanders like Scrum Masters.

A plague of story points

And here’s where I feel agile software development goes wrong. The people involved are so invested in the process they forget the actual goals of their organization or of agile software development. Having the ‘right’ ceremonies and getting the points right become the focus. More significantly, people become concerned with the consequences of their measurement, so they will avoid having a high-scoring sprint because it’ll increase expectations on their future performance (and by this stage the team probably isn’t feeling all that empowered, but that’s another story).

So now the process is about having accurate estimates, and consistent or slightly growing measurements, regardless of the impact on the delivered product. Because although it might be possible to explain to your manager that your productivity (as measured by story points) has bombed in the last month because you decided to refactor X in order to speed up lots of expected future work, by the time it’s aggregated to their manager and so on, that nuance is lost. And now that manager is getting shafted based on that measurement which doesn’t actually reflect whether or not your team is doing a good job.

My favorite Agile

The first time I ‘did agile’ was almost by accident. We had a three person development team working on a product and a product manager who had a three page Word table with a prioritized list of well broken-down features. And every fortnight, we wrote down on a whiteboard what, from the list, each of us was going to work on and how many days we thought it would take. If something needed re-prioritized the product manager would come in (any time) and we’d change what we were doing and update the whiteboard.

The point is that we were focused on delivering the outcomes that the business wanted almost as soon as it knew it wanted them. Sometimes we’d be asked to have a bit of a guess at how long half a page of priorities might take, leading to a 6-8 week kind of estimate. But all parties also understood that estimates were exactly that and things might change, both in terms of time taken, and in terms of what was critical to get done. Unfortunately I don’t believe this approach really scales, and it requires serious buy-in from stakeholders (despite all the evidence of the value of Agile/Lean approaches).

Conclusion

As is normal for these drawn out discussion posts, I can’t conclude with ‘the answer’ – and there are a lot of people out there who’ve spent a lot of time trying to find ‘the answer’ and haven’t found one.

What I am confident of is that measurements can’t show nuance and they subvert the behavior of what they intend to measure. So it’s incredibly important to continually reflect on whether your measurements, and their driving processes, are serving you well or whether people are now just optimizing for that measurement at the expense of actually achieving things.

I understand that an organization needs to gauge how it’s performing – whether it can be more productive, achieve different goals, eliminate waste. To do this it needs concise explanations of whether it is meeting relevant sub-goals. But the consequence of this concision is a loss of nuance that sands off the random edges that create effectiveness.

Workplace Flexibility

There’s a risk with this post that I’m going to shoot myself in the foot, but I believe in being as open and honest as possible, so I’m going to share my thoughts on good working environments, particularly remote ones.

This comes about because I’m trying to find some flexible, part-time, and remote work. I’ll come to each of these in turn during this post, but to start with, why am I looking?

I’ve been working as the technical partner in a small self-funded team developing a new product, and the level of product development required fluctuates. At present we’re going through a cycle of taking what we’ve learned, realizing our existing strategy isn’t going to work, and changing direction to suit. This is healthy – it takes time to: learn from the market what is really needed, especially with large prospective customers where it’s often a month from scheduling to having a meeting; and understand the costs, risks, and rewards of different commercial options. While I think it’s fair to say we haven’t done a good job of ‘failing fast’ so far, we continue to figure out what we can sell to the market and target the product to suit.

The end result is that I need to periodically attend meetings in another city and push forward with product development, but ultimately have time on my hands that I’d like to put to use somewhere. So I’ve been looking for flexible, part-time, remote work.

Silicon Valley Culture

Nothing turns me off in a job ad more than the term “Silicon Valley Culture”. Silicon Valley is very exciting in terms of getting VC money, but I remain unconvinced it’s a good place to work. If there is one word I’d use to describe what I’ve seen and heard about working in Silicon Valley as a developer, it’s insular.

Insular

What! How could it be insular to be working with so many great technical minds!

When birds of a feather are working together there’s a strong chance they will create a nest to suit their flock. The Silicon Valley stereotype is the cloistered geek. How can someone so insular understand real-world-problems, since being at the office 12 hours a day means they hardly experience the real world? Where do they: cross paths with tradespeople, nurses, children, (non-IT) engineers; or have experiences requiring empathy; or have a multi-faceted political discussion that doesn’t end up with people not speaking to each other (i.e. the real world equivalent of unfriending)?

Face Time

Which brings me to my next point: Silicon Valley Culture values hours in the office – free lunches, and dinners, and we’ll bring a masseuse on site, and… – basically we’ll do anything cheap (relative to your salary) to keep you in the office because (we assume) that if you’re in the office then you’re making progress, and we don’t pay any extra for that.

If you’re a twenty-something with no partner, family, or life, that’s great. If you’re anything else, forget it. Silicon Valley Culture is a big red flag that says “you may not have a family”, “you may not have a life (outside work)”. I’ve already addressed how this makes you an insular person, but it’s also terrible for productivity.

Productivity

Anyone who has done 6 hours of serious development in a day – by which I mean uninterrupted, building or debugging of some significant chunk of code twisting its way through layers of the application – knows that afterwards your brain will be dead. Development was once described to me as sitting two three-hour exams a day, and there have been plenty of days where I’d agree with that. Encouraging (or worse, measuring) time in the office leads means that hours beyond that six are a waste of my time and the organizations, and we both resent having our time wasted. And the more overtime goes on, the more it flows into people’s personal life, and into the next day, and so on until the developer is just burnt-out. I’ve been there – multiple times, and it’s not always easy to swing back from.

Don’t believe me? Then go and read chapter nine of Slack, or search the index of Peopleware for overtime. We’ve known for generations the hours people can repeatedly handle without it being detrimental. I’m not sure why Silicon Valley Culture hasn’t figured it out.

Consequent Demographics

Developers have an average age of 30-32. Assuming a constant working age population between 20 and 70, the average should be 45. Certainly some developers will become managers or analysts, and this is still a new industry so we’d expect to undershoot the average, but by 15 years (60%)? Is it possible that Silicon Valley Culture makes being an ‘old’ developer a problem? The culture is certainly incompatible with having a family. It’s also incompatible with people with more life experience – people who have reached that point where their bottom four levels of Maslow’s Hierarchy are met and realize there’s an awful more to life than work, especially if work isn’t able to allow them to fulfill their potential.

Silicon Valley Culture also implies a boys’ club. Look at the developers in the Silicon Valley TV program: young single guys. There have also been long-standing issues with sexism. So it’s not surprising that the earlier-referenced survey put the percentage of female developers at around 10%. Given the stereotypes, the sexism, the family-unfriendliness (at the risk of being lambasted, mums are still more likely to stay at home with the kids than dads), we shouldn’t be surprised at that figure.

In short, Silicon Valley Culture is a terrible culture. If that’s how you describe your organization you are not going to get wise or mature developers.

P1020378
Here’s a picture of a baby to break up the text. It in no way reflects how Silicon Valley Culture treats developers.

Part-Time

Why does everyone want full-time staff?

My gut reply to thinking as an employer is that I want commitment. But I’m going to step deeper into this and ask: why do I want commitment? What does being committed mean in an organization?

Commitment

The first commitment given by an employee comes when they sign an agreement saying “I’m committing this much time into your organization in return for compensation which reflects that commitment”.

Hiring someone comes with an on-boarding cost and to maximize their return the organization wants the employee to be useful as quickly as possible, which can only be achieved through time experience. In that scenario being full-time will reach this goal faster (in terms of calendar days), although the cost to reach a certain experience level won’t change. Depending on the role this ‘calendar time to usefulness’ may or may not be a factor. For instance, domain-knowledge intensive roles like architecture and product management often have much longer on-boarding periods than development, making the additional time delay of part-time too big an opportunity cost for the organization.

From another perspective, most new hires describe their early weeks as “drinking from the fire-hose”. I imagine if less is drunk per day, because a part-time day is shorter, more of it will be retained.

The second commitment is level of energy or zeal the employee chooses to bring to their job. This commitment is a function of finding a personality which can engage with the organization and then providing them the environment that makes them want to engage (rather than just turn-up). I imagine that once hours drop considerably the employee may find it hard to really engage; but conversely, working fewer hours they may have more energy to engage with. So I conclude that being part-time (say in the 20+ hours/week range) wouldn’t have a significant impact on this kind of commitment.

Focus Time

Earlier I mentioned that development can be brain-intensive, and that beyond a certain level the productivity of each extra hour diminishes quickly. So why pay for those? My experience has been that people working shorter days tend to plan better and be more focused, and I estimate that someone working 5-hour days probably gets 7 hours worth of work done i.e. you’re paying 25 hours worth of time per week for 35 hours worth of productivity (all else being equal).

At this point hiring full-time by default seems like a tradition more than anything, a tradition that is worth re-examining.

Flexibility

The concept of a two-parent family where only one parent works is history. The rise of day-care and after-school-care have made that quite clear (I’m not entirely comfortable with this concept from a sociological perspective, but each to their own, and I digress…). This means there are an awful lot of experienced workers out there who have to juggle family and work. If you make that juggling difficult for people then they can’t work for you because, like it or not, family ultimately comes first.

For my part, I have pre- and school age children, so between my wife and me someone needs to be home by 1430 to pick them up and look after them. Once we’re home and they’re fed, I can typically resume whatever I was doing earlier. Sometimes they’re sick and someone needs to be home with them, but usually they’re just quietly sleeping and there is little to impede working. A strict “X-hours a week in the office” contract doesn’t cope with these scenarios. It is, like default-to-full-time, a hangover from bygone days. The 21st century workplace requires flexibility, and full credit to New Zealand which does enshrine this concept in law. But regardless of legislation, flexibility is something organizations should do because it’s better for them.

Being flexible with hours:

  • increases the talent pool available to you;
  • tends to result in breaking up the workday, which makes for fresher and more productive minds;
  • allows staff to manage their creativity, meaning you’re not paying for mental downtime;
  • creates a feeling of mutual respect and reciprocity, which means asking for a little urgent work outside of hours is a fair exchange, rather that leaving the employee feeling that they gave their time for free.

When combined with remote work it opens up opportunities to access the global talent pool. For instance I’ve applied for several jobs in the EU, despite being in New Zealand. This would work out nicely for me because my wife is home in the evenings and I can be available from 7pm-midnight three nights a week. That’s 15 hours of overlap a week (give or take daylight saving). Equally if I applied in the Americas then I can work early mornings or Saturday (American Friday) because it’s outside standard NZ working hours.

P1010486
Babies are flexible. Is your organization better than a baby?

Remote

Have you ever worked in an open plan office that nobody complained about? Depending on who you ask they are too hot, cold, noisy, distracting, constrained, or impersonal. What they are is cheap, and they allow poor managers to sit watch upon their domain (micromanagement). It has been clear for years that working environment affects productivity, a result which continues to be reinforced.

If you’re like me and need quiet and a means to control distractions then working remotely is bliss. I recall when I first started working remotely, my productivity immediately doubled. Thankfully that was in an organization with a good remote culture, because having remote staff requires a level of organizational discipline. Having remote staff requires inclusive communication and decision making processes, something that makes everyone happier. For this to work leaders must ensure communication is only happening in public forums and decisions are being reached by an inclusive process. This is good business practice, but with remote people it is more important because it is harder for them to see when they are being left out.

Essentially a remote organization must revolve around a text/audio/video chat application like Slack or Hipchat. People share their thoughts in writing for everyone to think about and provide feedback on. In this shared space, all voices can be heard so people are informed decisions are made inclusively. These tools can also be controlled so that people are not disturbed when they don’t need to be and can thus focus on the task at hand.

Agile Software Development talks about the importance of information ‘convection currents’ i.e. the information accidentally shared by people in proximity hearing each other. This is something that is lost with remote workers. They also talk about ‘drafts’ – the information that wafts about which is completely irrelevant or distracting, and in my experience the drafts tend to outweigh useful information. The beauty of a remote work culture is that the information sharing is recorded in text and available for anyone who is interested, and crucially is searchable. As a result that information is available to everybody, not just those who happened to be in earshot and paying attention at that time.

One concern with a remote team is that remote workers might be slacking around on company time. I don’t buy this excuse: it’s usually pretty obvious if a developer is contributing to the level expected simply by looking at their commits and documentation.

So yes, remote workers require a culture shift, but it’s a positive one, and it opens up huge benefits of being able to access talent pools well beyond your current city.

Conclusion

Despite acknowledging that the nature of work is changing our workplaces seem very slow to catch up, especially given the benefits of wider talent pools and increased staff happiness and productivity that part-time, flexible, and remote work (both independently and together) create.

So if you are interested… I’m flexible if you’re flexible. I can legally work in New Zealand, Australia, United Kingdom, and Europe (at least until Brexit goes through, if it goes through), and I’m sure contract terms can be worked out elsewhere. You can get an approximation of my expertise from this blog and the about page, and I will say I’m a generalist and proud of it! 🙂

You can contact me through the contact page.

Value of Data

“Big data is the future” or so we are told. With enough data we can create models that provide us with good outputs for unknown inputs using an array of techniques like: using probabilities to estimate likely relationships; regression to find trends and interpolate answers; or by training general purpose learning algorithms.

In particular, Machine Learning (ML) is in vogue, and although the underpinning concepts aren’t new (I had assignments combining computer vision and artificial neural networks at university back in 2001), the capabilities of machines and easy access to massive levels of computing power now allow much more practical application of these concepts.

Regardless of the technology or the hype, there are universal concepts that are paramount to the successful application of a technology. For instance it is important to understand what the technology can and can’t do, and what properties are intrinsic and what are variable. Continuing with ML as an example, it is more effective to pre-process an image and extract key attributes and feed those into a neural network than to give it a million pixels per data entry.

One universal concept is that the technology needs to solve a real problem, or to use business terms, needs to ‘add value’. There is a cost to using a technology – for big data, collecting data can be expensive, notably in mitigating the risk of failing to manage the data i.e. ensuring it is secure and compliant. To offset this cost we need to establish value, which means asking:

  • How does having this give us a competitive advantage?
  • How can I monetize this?

For some of the big and famous organizations the answers to these are fairly clear: Amazon wants shopping data to provide better choices than competitors, drawing more customers and therefore more sales; Google and Facebook want information that targets their adverts to more of the right people, resulting in more buying per advert, incentivizing customers to buy more adverts.

One strategy for answering these questions is to create data which is so much better than competitors’ data, that customers will pay to access the data. This is not a new concept as software products have been up-selling reporting since time immemorial, but recently there seems to be more inclination to answer modelling questions rather than just provide charts. This is where the business questions need to be applied. For instance, if it is possible to mine data to answer questions like “what impact does doing X have on Y”, then ask yourself whether these answers are something that customers will pay for and competitors don’t have. If so, then you’re onto an excellent strategy. If not, then is having that data valuable?

Automated Testing Priorities

There’s a common theme in automated software testing that the greatest effort should go into unit tests, lesser into integration tests, and least into UI tests. This is known as the testing pyramid. However I’m not convinced this is the best use of automated test developers for web applications, and I believe this is because the nature of web standards and a commonly used web application architecture invalidate some of the assumptions behind the pyramid.

Testing Goals

Before we continue we need to state the goals of automated testing. In my mind they are:

  1. Validate the correctness of the thing under test
  2. Allow refactoring of the thing under test

Note that correctness includes ‘non-functional’ things, like authorization security.

From a business perspective, we want to know that the product works as intended. Working as intended means that the application performs correctly when used through its interfaces. This would suggest that UI tests are the most important, which is the opposite of conventional wisdom.

The reason often given for placing less focus on UI tests is that they have been considered notoriously fragile. However I posit that this has been due to the nature of the interfaces being tested, which have tended to make identifying and interacting with UI elements automatically very hard; e.g. having to use control ids with WinForms applications. I’m also suspicious that less focus on UI tests is a consequence of Agile methodologies that insist on jamming all testing into the same cycle as development, resulting in automation trying to be written against a UI in an extreme state of flux.

Unit Test Fragility

One problem I have with unit testing is that developers are encouraged to test the internals of the unit. This happens when mock objects are checked to see if certain methods were called on the mock.

The purpose of functions and classes are that they expose some contract and hide the details of how that contract is fulfilled. Testing how a unit is doing its work means examining inside the black box, which defeats the purpose of using testing to support refactoring because now we can’t make a change to the implementation of a unit without breaking its tests.

UI Test Fragility

In his 2012 article Fowler says:

An enhancement to the system can easily end up breaking lots of such tests, which then have to be re-recorded. You can reduce this problem by abandoning record-playback tools, but that makes the tests harder to write. Even with good practices on writing them, end-to-end tests are more prone to non-determinism problems, which can undermine trust in them. In short, tests that run end-to-end through the UI are: brittle, expensive to write, and time consuming to run.

I believe that some of these assumptions are less valid in modern web test automation.

Automated web testing tends to be hand-written because (in my experience) the recording tools can create quite fragile paths, usually because they don’t know what the least variant information is. It is straight-forward to hand-write UI tests thanks to CSS selectors which are easy to use, well-supported, and when done simply (i.e. via id and class selectors rather than paths) aren’t hugely prone to change. These selectors are usually wrapped into page objects that further insulate the tests from changes.

The HTML DOM also exposes an event model which allows tests to mimic the vast the majority of UI actions, removing the complexity of older style tools which involved a lot of mouse-coordinates and button states.

And finally, in web development, UI testing has the added benefit of enabling testing across multiple browsers – something less applicable to downloaded applications.

However I agree that they remain time-consuming to run, and if there are lots of developers committing to the repository then having your continuous integration run on every commit may not be possible, reducing the benefit of the tests for quickly catching problems.

Middle-Ground – Integration Testing the API

It is increasingly common for web applications to be built as a web API and a web (JavaScript) client. This is my personal preference over server-side rendering as it nicely decouples the presentation from the logic and allows the application to more easily integrate with other applications. There is some development overhead in this approach, but given most web pages perform some dynamic interaction with the server thus requiring some level of client richness, this overhead is quite minimal.

Having an API provides an excellent place for automated testing. An API is a contract and will express most, if not all, of the business rules through its inputs and outputs. It also requires basic security, and allows validation and authorization to be checked. It can be easily extended to run more extensive security testing (i.e. by manipulating HTTP headers and sending malicious data) and performance tests.

Integration testing the API doesn’t mean a full environmental setup is required. It is still reasonable to use mocks for calls that are slow or resources that aren’t available. For instance my integration tests use .NET Core’s TestServer rather than running a web server, EF’s UseInMemoryDatabase rather than instantiating a database, and stub out AWS service calls. These are reasonable compromises because I’m confident those areas will perform to their contracts.

Conclusion

This is my ‘testing pyramid’ from highest to lowest priority:

  1. API integration tests
  2. Integration or unit tests for things that can’t be reached from the API
  3. UI tests for success scenarios

In my current application I have 98% code coverage and 90% branch coverage (largely ruined by not throwing exceptions inside all the C# using statements) of my business layer using the first two items on my list, and it has enabled considerable evolution and refactoring of the code-base over the last six months.

Generalists and Specialists

While I’m on my theme of people value, there is a group of technology professionals who are often quite undervalued – the generalists. Until the last couple of years there had been a trend of increasing specialization of technology development roles, notably the back-end/front-end split which is now being replaced again by that great generalist role, the full-stack developer. And here’s the thing – overall, the cost of generalists and specialists doesn’t vary heavily, with subject specialists – e.g. InfoSec, DBAs, Authentication Architects (I saw that one advertised in Wellington) – costing a bit more, and platform specialists – e.g. Java dev, UI dev – a little less. In this continuum of generalists to specialists, generalists represent an absolute bargain.

The Generalist Developer

An experienced developer can do 90% of what a specialist can do in their field.

Need DevOps? Why not get your developer to do it? They can research potential solutions, read and understand API documentation, pickup Bash or Powershell pretty quickly, and setup basic configurations based on recommended best-practice from vendors. Plus when they’re done, they can go back to development rather than twiddling their thumbs.

Need QA automation? Need requirements analysis? Need basic network setup? Need Project management? Need customer support? Need internal IT? Need architecture? Need a DBA? These are all things I’ve done to a production level in my 14 years primarily as a ‘developer’.

The vast majority of software out there is about understanding, automating, and transforming processes, and generalists are amply qualified to solve these problems. And where they can’t solve a problem from experience, they are expected to go out and research the plethora of alternatives, running a huge gamut of potential technologies (and therefore specializations), and pick a solution.
Sure, they may not create the algorithm that founds the next Google, but those companies represent a minuscule segment of the field and require a level of specialization more likely found in academia than in industry anyway.

In software product development you want generalists. These are people who know that, for instance, information security is important so they pick technologies, tools, and solutions that promote good security practice. And because they’re not as sure of themselves they are more likely to test and verify their assumptions and designs. They also have a wide view of the world, so can much more effectively evaluate trade-offs between different solutions and solve a wider range of problems than a specialist can. And typically, for little difference in price!

The Specialist

I’m not suggesting we don’t need specialists at all. I’ve found their value to be acting in advisory or consultancy roles where they are the checks-and-balances that warn people about traps and pitfalls and educate the generalists on best practices. I freely acknowledge I wouldn’t have some of the knowledge I have today without the support specialists have been able to provide.

However this very act of education decreases the relative value of the specialist because, by receiving more knowledge the generalists ‘level-up’ and reduce their knowledge gap in the specialist’s field. That makes the need for the specialist more tenuous, and some people find it challenging to overcome the instinct to protect one’s space. This assumes that specialists are static creatures, and I would expect they too are continually learning and trying to level-up, but within one organization the usefulness of their knowledge may be limited.

Another problem with specialists in a small organization, is that they effectively constrain your solutions. The company thinks, “well, I’ve got an Oracle DBA so we’d better use Oracle and stored procedures” even if it’s not the best solution. Whereas, a generalist will evaluate a range of solutions based on their experience, the company’s accumulated knowledge and environment, industry trends, future hiring needs, relative costs, etc. etc. to inform the solution choice.

Conclusion

If you’re a five person development shop, a specialist doesn’t make sense. If you need that expertise you should hire a consultant. If you’re a five hundred or five thousand person development enterprise, then those specialists should have plenty to do to make it worth having them on the payroll.

Perverse Incentives

Setting

It is well established that CEO pay has climbed astronomically in the last decades, and that over a similar period inequality has grown throughout the Western world. This period has also been dominated by laissez faire economics, and corporatism to the extent of corporate welfare.

There are some indications that we are swinging away from this setting, with even the IMF saying, “Empirical evidence suggests that it may be possible to increase [personal income tax] progressivity without adversely affecting economic growth” (IMF Fiscal Monitor, October 2017), but it will be some time, if ever, before the mind-set of actors in this economic dance changes.

Right now, you might be wondering whether this is a software blog or some economic philosophy one, but be assured, this relates back to management, and particularly how it impacts high-skill industries such as software development.

Introduction

Our current economic setting has created a system of incentives that is at odds with the goals of good management.

Good management in knowledge industries emphasises an ‘upside-down-pyramid’ of management which supports, rather than directs, the activities of skilled front-line executors. Put another way, it genuinely recognizes that people are the most important asset in a business and the role of management is to create an environment where those people with the skills can excel.

It is also clear that managers can add value in ways that others can’t. They can use their outright authority to connect otherwise separate groups, resolve conflicts, and use their bird’s-eye-view of the business to sponsor change and ensure strategic alignment, a.k.a. ensuring the right work is being done (HBR, 1995).

In our society we equate value with money, and given the greater value managers can add, pay managers more. We also expect more return from more money, so we expect increased responsibility and accountability from said managers. But here we reach the crux of problem: to support skilled staff it is important to empower them with responsibility, so how can the manager be held accountable whilst giving away the responsibility? To readers who (like myself) have been managers this is the “no kidding” moment, so I would ask the question, how have you tried to change that? This is a systemic problem and as far as I can tell our approach has been grudging acceptance of the status-quo.

How strong is that responsibility?

Good managers empower people, and people make mistakes, and it is unfair to hold the manager responsible for those mistakes, otherwise everyone would be too afraid to make mistakes and we’d destroy effectiveness and any hope of innovation. We also read in the media stories of obfuscating CEOs who (quite reasonably) admit they couldn’t have known that X was happening in their organization and so (will make some knee-jerk changed to resolve it, and) can’t really be held responsible.

By highlighting that they’re ‘not really that responsible’ the premise that with increased value comes increased responsibility has been completely undermined. Now this isn’t the only reason managers are paid more: The other commonly held notion is that managers add more value because their decisions have greater (financial) consequences. This I dispute largely because those decisions are never made in a vacuum, and there are a lot of advisors and analysts that effectively determine the decisions in the first place, with the executive being closer to a rubber stamp than a judge. But that would be a digression from the point of this post, which is to focus on the consequences of ‘having responsibility’.

Micromanagement

Responsibility encourages micromanagement. When your head is on the line then your instinct is to get as involved as possible to ensure the outcome you want. There are plenty of good managers out there who manage to overcome this instinct, but the system is very much setup to encourage micromanagement, and that destroys empowerment.

Under micromanagement, instead of having a team who are comfortable to pursue their own direction to achieve the goals, you’ve now got a sort-of team member (the manager) with a much more limited view of the details (after all, the higher up an organization you are the wider but shallower view you’re expected to have) who ironically requires more energy from the team-members to ‘manage’. This also makes the team feel less ownership for their work because accountability becomes blurred between the manager and the team. And instead of being measured by outcomes the team are judged on details; details that the manager is often not as qualified (as they think they are) to judge.

Micromanagers can also become the de-facto communication channel instead of the more effective approach of organizing teams to communicate with each other. This creates a knowledge and communication bottle-neck which is inefficient.

What does an effective manager do? They set a vision and goals, empower people to execute on them, provide cover so they can achieve their goals, resolve conflicts, and then put their feet up or have strategy off-sites or something like that. They should not be required to answer all the questions other people in the organization have – the team is the most capable to do that – but they can filter the questions to ensure the team can focus on the work.

But if your organization insists on you knowing everything because you are ‘responsible’ because you are expensive, then how can you be an effective manager?

Solution

So how do we fix this?

Essentially, be realistic with managers roles. If their roles are seen to be closer to text-book roles of planning, leadership, organization, and control rather than a know-it-all of everything in their fiefdom, then responsibility is going to fall closer to where it is held, with the executing team.

With this reduction in perceived responsibility and expectation there should be a reduction in compensation.

This will also improve empowerment among the teams which will give them a greater sense of ownership and satisfaction, meaning they’re less likely to turn-over.

Leadership

This is something of a digression from previous ‘how-to’ posts. Instead I’ve felt motivated to share my perspective on leadership, which is an issue in society that impacts organizations of all sizes and kinds, from parenting through to corporations and government.

What is Leadership

I’ve often been struck by the distance between leadership as it is defined in management texts and how it is executed. My trusty management textbook places leadership as the fourth pillar of management in the section “Leading – To Inspire Effort”, and defines leadership as “the process of inspiring others to work hard to accomplish important tasks” (Schermerhorn 2001, p262). This is a fairly open definition that could include anything from managing an empowered self-managing team through to slavery. To be fair to the author, it is followed by six chapters expanding on the subject.

The contents of that text are based on the outcome of decades of research and analysis. In general, research seeks to simplify the thing under study as much as possible – to be the crucible that burns away insignificance and leaves us with the key factors that impact something. I believe concise leadership contingency models like Hersey-Blanchard and it’s three-dimensional matrix of relationship and task behavior and follower readiness illustrate how complicated systems can be abstracted to their significant details, and that such models are critical for illuminating various facets of management and preparing managers to handle the many different people and situations they may encounter (and to be clear: leaders are managers. If you are inspiring people then you are managing them).

What scientific management seems to cover less (or at least less so in introductory textbooks) is that people are.. well.. people. They aren’t ‘rational agents’ conforming to the box neatly defined by research, and they have – insert-deity-here forbid – feelings! People are squishy and unpredictable, and frankly if you’re in a management position and don’t think that I’m stating the bleeding obvious, then you need to find another job. The literature on this side of management tends to be more anecdotal, but also easy to empathize with regardless of which side of the managing/managed fence you fall on.

Theory vs Practice

So now I will add my anecdotes through some hypothesizing. I’m told (by Wikipedia) that around 1% of the population suffers from the most extreme form of narcissism, and it is my contention that these people tend to cluster in leadership roles. The very nature of “knowing you’re right” and projecting that confidence (however un-examined it may be) creates the vision that management theory looks for. It also creates an environment that followers need to have a sense of fulfillment – after all in our comfortable post-Maslow worlds we need to make a difference to find satisfaction, and what better way than fulfilling a vision to ‘achieve great change/improvement/innovation/etc’. The people who espouse this confidence are also lauded by their superiors who naturally prefer supposed simplicity over the complex reality of the situation, and thus these people tend to elevate into positions of power. Unfortunately people who ‘know’ they’re right also tend to be extremely resistant to anything that challenges their perspective. Such a conflict can be very personal and highly destructive given that any challenge is perceived as a threat to that person’s self-image or core values.

In practice the leader’s vision tends to be skewed towards their own goals, and while organizational alignment is usually covered by at least lip service, the goals tend to be angled towards their individual needs, whether for career progression (who hires the manager who thought the status quo was working great and opted not to change anything?) or a psychological need (e.g. admiration, entitlement).

This is the point where I start to struggle with these people. I believe I’m experienced enough to be positive about people and work with them to foster the goals of the relevant organization, but my natural desire for analysis means that over time I tend to find concerning dissonance in their positions. Where I’m not experienced enough, or perhaps just disinclined to submit to this aspect of culture, is that I will point out that dissonance, and in doing so create the conflict.

The world of management theory tells us to be transparent with problems because organizations can’t fix problems they can’t see, and it tells us managers that a moderate level of conflict is good (too little means people have stopped caring and are probably looking for other jobs). What it doesn’t tell us is that some of the time the manager is going to see that as a personal threat, or they’re going to ignore it and place you in the ‘whiner’ box, because these managers aren’t entirely cut out for their jobs, but there is seldom any way to observe this problem and correct it in an organization. Studies strongly indicate that the most significant factor in employee retention is their immediate manager, and yet dysfunction in that relationship is often invisible to the organization until it is too late.

We know what’s good for us, but…

Perhaps the most scientific expression of this I’ve run into is in the book Good to Great by Jim Collins. Chapter 3 very clearly summarizes that the best leaders have the opposite traits to narcissists. They are modest and under-stated, are diligent and workman-like, and they give credit where it is due but shield others from failure (Collins 2001, p39). This doesn’t stop them having a firm vision and a strong will to achieve it, but they do so by getting the right people and getting them to buy into the vision and steer the organization toward it, and expecting they’ll do the same at the next level of the organization. It is a positive and virtuous cycle if achieved.

The literature also highlights how salary and performance of top leaders correlate negatively. And yet this need for ‘leadership’ for the self-fulfilment and simplicity reasons I highlighted earlier mean these leaders, who are by all accounts bad at their jobs, continue to be highly rewarded – and probably more so that than less confident peers given their heightened sense of self-worth likely translates into salary expectations.

I doubt any of this is new. Much has been written about how “ignorance more frequently begets confidence than does knowledge” (Darwin). What remains surprising or perhaps depressing, is that for all the things we’ve learned about scientific management and about people and behaviors, is that we still reward sub-optimal behavior. Put another way, society seems to revere leaders that overestimate and under-deliver, and who are comfortable treating us as disposable minions to be crushed on the path to their own glory. And that doesn’t seem like progress.