Ian Wold

90% of my Homepage was Useless

Wed, 06 Dec 2023 00:00:00 Z

One week ago, my homepage was around 550k uncompressed and took about as many ms to load. I thought this was light, though I'd have liked it to be faster - I just attributed this to a side-effect of hosting with GitHub pages.

This weekend I discovered the 512kb Club which ranks websites by how big they are, the catch being that you need to be below 512k uncompressed. I looked at some of the sites on the high end of that list, and I felt pretty disappointed. There were sites that had much more content, some with many large images, while staying below 512k. I have an SVG and a bunch of black-and-white text.

Well, I made some small optimizations and got it below the threshold, but then I caught the bug and presto changeo before I knew it I had eliminated 90% of my website and it is pixel-perfect to how it was before. I'm now at 62.8k, a member of the "green team" on the 512kb Club site, and equally important I reduced the load time of the site (by any time metric you like) by around 50% just with the size optimizations.

Check Your Fonts

The single most impactful thing I did was I checked my fonts. I use Google WebFonts, and their API gives you a few opportunities for improvement. First, make sure you're not loading any fonts you're not using.

Then eliminate any weights you're not using. I use Rubik for my text, and the homepage only uses weights 300 and 500, but I had been loading everything from 100 to 900:

<link href="https://fonts.googleapis.com/css2?family=Rubik:wght@100;200;300;400;500;600;700;800;900&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css2?family=Rubik:wght@300;500&display=swap" rel="stylesheet">

I use Montserrat for my name at the top and nothing else, and Google provides an API for you to specify the exact text you need to load in a font. This took Montserrat from 30k to 2k or so:

<link href="https://fonts.googleapis.com/css2?family=Montserrat:wght@100;200;300;400;500;600;700;800;900&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css2?family=Montserrat:wght@700&display=swap&text=IAN%20WOLD" rel="stylesheet">

Optimize Your SVGs

SVGOMG is a great tool for this. The SVG at the top of my page was 64k, now it's about 8k. That's huge!

I had also previously been using Feather Icon's JS client to swap out i tags for their icon SVGs. However, by directly embedding the SVGs (which is easy and maintainable because they've got a great site) I was able to eliminate 30k and several ms from my page load time.

Torchlight is Pretty Cool

I had been using Highlight.js for syntax highlighting, and it was easy enough to set up (I highly recommend it if you need to get up and going fast) but I wasn't too much a fan of the highlighting for my cases, and it was putting a 30k dependency on my site. Yes, I was able to take it off my homepage - no code there - but I figured I could also help my post pages out.

Torchlight is absolutely awesome. They run a server with the same language servers that VSCode uses with an API that will perform syntax highlighting for your snippets. Best part is they've got a Node package that you can run on a local directory to detect any <pre><code></code></pre> blocks in your HTML, post the code to their server, and swap out your code blocks with their highlighted blocks. It's free for non-revenue-generating sites, works like charm in the build step, and generates the best syntax highlighting with the most features.

Maybe Minify Your Pages

I only saved a few kb doing this. What's most important I think is to consider bundling assets that can be bundled to save on the chore of loading them from a server. If your files are sufficiently small (as they tend to be on a statically-generated site) you're not going to save too much by minifying. However, if it's easy to add, then go for it.

Next Optimizations

SEO
Accessibility
Performance
The SVG at the top of the page still doesn't turn white in dark mode

Adding a Database to our Railway App

Wed, 08 Nov 2023 00:00:00 Z

Last time I looked at Railway, I got it up and running with a Blazor WASM app. Now, I'll look at adding a PostgreSQL database to it. As with the app I got working, Railway's interface will make it incredibly simple to provision the database, and we'll need to do minimal work to get the connection info to our app.

Provisioning the Database

From the Railway dashboard, you can click into your project, and there should be a New button at the top (as in a buttno that says "New", not a button whose appearance would not heretofore be expected):

This brings up the familiar dialog to provision resources, from which we'll select Database:

And then I'm going to choose PostgreSQL here. Note that if Railway doesn't have an option here for the database you want, you can always create a plain Docker image from an image of your preferred database.

And look at that, it deploys a new PostgreSQL database! This is one of the things I love about Railway - their interface to set up resources has eliminated all of the steps to setup that aren't 100% necessary, and the default settings they choose are logical and easily overwritten later if we need.

Notice also the attached pgdata volume. As discussed before, Railway just stores all of your resources in Docker images. This doesn't require that you create a dockerfile for each of your projects and resources, they can do that for you if you don't want to. It is to say that they give you all the tools you might want to use to be able to manage your resources as docker containers and volumes, so you can more or less choose the level of control you need over your resources.

The way I use Railway, I let it manage all my resources for me, and I don't bog myself down in the weeds of Docker as much. I've got plenty of time in my day job to get frustrated with Docker!

Connecting to our Database

If you click on your new database resource and then the Variables tab, you should see a bunch of variables:

This contains all the information that our app will need to connect to it. Railway allows the resources within a project to reference each others' variables, so we just need to know which ones to reference to build a connection string.

Now, they've all got incredibly obvious names so you can certainly accomplish this by guessing in as much time as it takes you to read this article, but for thoroughness' sake I've got a connection string here:

User Id=${{Postgres.PGUSER}};Password=${{Postgres.PGPASSWORD}};Host=${{Postgres.PGHOST}};Port=${{Postgres.PGPORT}};Database=${{Postgres.PGDATABASE}}

If you copy that as the value of a new variable in your application, it will fill in each of the references with the values from your database's variables, and that should be a sufficient connectino string to test that it works:

Note that I'm using the variable name ConnectionStrings__Database, with two underscores, which behaves as though I'm inserting the following into my appsettings (in a .NET context):

"ConnectionStrings": {
    "Database": "..."
}

And that's all the configuration needed for your app to be able to consume your new database!

A Scrum Odyssey

Fri, 20 Oct 2023 00:00:00 Z

I had an unexpected and not entirely unpleasant experience in a recent retrospective meeting: all of the engineers on the team moved to stop having daily standup meetings altogether. Although I'm no stranger to working without standup meetings, and in many ways I prefer that way of working, I don't think it's unreasonable to suggest that the majority of engineers are fine with well-structured daily standups.

Even with a team that enjoys standup meetings however, there's value in breaking the norm, becoming more flexible, and trying things out. Maybe such a team would settle back into daily standups, maybe they'd find something else that works for them. Teams which don't enjoy their standups though will almost certaily settle on a new scheme.

I sat down to type out a guide on how and why a team might experiment with alternative approaches to the daily standup. But that came out so incredibly dry I weaved those points into a story demonstrating a hypothetical team undergoing that process. That resulted in something which I'm not entirely sure got the point across. But then I remembered that ChatGPT (insert eye roll here) is actually pretty good at generating Shakespearean sonnets (insert double eye roll here). So, I ran my story through the ringer and I present to you A Scrum Odyssey: A journey away from daily scrum meetings, as a cycle of eight Shakespearean sonnets:

The Rite of Dawn's Assembly

In days of yore, where standups held their sway,
Religious teams to this rite did cling tight.
Believing in its might, they'd all display
Their tasks and toils, at morning's first light.

“'Tis the only path!” they’d loudly decree,
To boost our voice and our work's potency.
Yet, in this sea of daily scrutiny,
Did they e'er question its necessity?

For in their hearts, joy and productivity,
Seemed hand in hand with scrum's consistency.
But having not trod another pathway,
How sure were they, in their daily ballet?

To question not is to be but a drone,
Yet change may show more ways than they've known.

Whispers 'Midst the Court of Teams

As time doth pass, objections loud do cry,
Standups draw long, and patience wears so thin.
Some feel the gaze of oversight too nigh,
Whilst others sigh at tales they're lost within.

The daily rite, once held in high esteem,
Now burdens hearts, and feels not quite so lean.
"Maintain the stand!" the managers exclaim,
"For knowledge's spread, 'tis the only regime."

Another quips, "'Tis how we stay aligned!"
Yet in their hearts, a restlessness they find.
The status quo, now feels a heavy chain,
With every voice, there grows a silent pain.

Unease doth spread, as waters dark and deep,
The team in hope, for change they might yet reap.

Into Uncharted Councils Steered

In growing dissent, the teams make their stand,
To question the rites of daily discourse.
With keen eyes they search, across the vast land,
For new methods, their spirit to endorse.

The captains do doubt, and their concerns voice,
Yet vibrant ideas do from teams arise.
"Demos!" cries one, "Let our works make their noise!",
Another suggests, one-on-one 'neath the skies.

Pair programming, reviews, new tactics bloom,
As weary traditions begin to wane.
The tides of change, they sweep away the gloom,
Old ways dissolve, as new paths become plain.

Thus in the dance of time and adaptability,
Teams find their stride, and new possibility.

Quests in the Halls of Converse

In search of ways, the teams diverge their path,
Asynchronous rites, emails that they pen,
Fewer meets they seek, to avoid the wrath,
Of daily drudgery, again and again.

More intimate chats, one-on-one they hold,
In pair programming, they sharpen their skill.
Exploring each method, both new and old,
They weigh every boon, every bitter pill.

Managers, once stern, now their gaze doth shift,
Witnessing changes, benefits unfold.
Yielding their ground, as sands of time do sift,
Embracing the new, releasing the old.

In the grand ballet of work's ebb and flow,
Teams evolve, and brighter futures they sow.

Where Two Worlds in Concert Meet

From all the trials, some choices do emerge,
Asynchronous standups, the globe's teams cheer.
For visual minds, digital boards surge,
As preferred tools, their thoughts to make clear.

Meetings grow few, much to all's delight,
Those that remain, are purposeful and keen.
Gone are the days of endless oversight,
In place, moments of value are seen.

"Support!" cry the captains, in newfound glee,
Feeling the strength of their teams' tethered core.
From the throes of old, they now are set free,
Bonded and strong, as never before.

From trials and tests, the best paths are clear,
Teams and their leaders, in harmony steer.

Musing on Fortune's Fickle Hand

With shifts in tide, come trials unforeseen,
Some souls do long for morning meetings past.
An anchor lost, in routines once serene,
Others claim that commitment's not so vast.

Managers note, their gaze oft on boards dwell,
More than before, in this changed paradigm.
For in each change, some benefits dispel,
Replaced by costs, in this evolving time.

"No method’s perfect," the team concedes true,
Every path has its toll, its weight to bear.
But in pursuit of what’s best, they pursue,
Balancing gains, with costs they must wear.

In life’s grand dance of loss and of gain,
Teams learn to thrive, through pleasure and pain.

Melding Yesteryears with the Morrow

From experiments vast, wisdom teams glean,
Blending the best from methods they've tried.
A hybrid emerges, both fresh and seen,
Balancing old and new, side by side.

Asynchronous notes, for days in-between,
With face-to-face meets, weekly they decide.
Though returning to paths once before seen,
They come renewed, with broader stride.

Most vital of all, in this new phase,
Teams claim their process, its reins they hold.
Each member engaged, in collective praise,
For a system by them, shaped and molded.

From trials diverse, a way they have found,
Where every voice in harmony does sound.

Ascendance Beyond Time's Norms

In newfound grace, the teams now flex and bend,
Having learnt the worth of norms held askew.
Witnessed have they, to what change can tend,
The boon of steps, both retraced and anew.

The culture's hue, to feedback does incline,
Embracing shifts, with open arms and heart.
Where once was rigidity, now they entwine
Flexibility, as their foremost art.

Managers see, that control's grip too tight,
Does not always lead to the best of ways.
Productivity, in this newfound light,
Reaches peaks unseen, to everyone's praise.

From challenging norms, to heights unforeseen,
They've journeyed forth, with purpose evergreen.

Bandwidth is Infinite ... ly Troublesome

Fri, 26 Apr 2024 00:00:00 Z

In my last post on latency I cited an interesting definition given by David Boike: TotalTime = Latency + (Size / Bandwidth). This suggests that latency and bandwidth are related, and that's intuitively true. At a very simplified level, one deals with packet count and the other with packet size; these add together into a concept of packet volume, and so issues related to each do compound on each other. In that article, I was able to demonstrate some techniques which, depending on how aggressive you might be able to be with your system's requirements - can very largely mitigate issues related to latency.

Bandwidth is not, I think, so simple. Indeed, the total bandwidth of the world has increased markedly year over year, but then our demand for it has as well. In some cases our demand is outpacing our ability to increase bandwidth. Theoretically we could increase bandwidth enough to meet way more than we could ever demand, but a 1-mile-diameter cable between New York and London sounds slightly infeasible. I'm not a cable engineer so I won't get too deep into that debate, but until they give us that cable we'll need to deal with bandwidth.

This is the third fallacy of distributed computing, that bandwidth is infinite. Of course there's a physical limit to how much data we can send, and there's not really any clever tricks here that can mitigate that terribly well. The first step, maybe the most difficult, would be to not transmit erroneous data. From there, we can compress data, tier it by importance, and maybe shift around where it physically gets moved. Nonetheless, we can't adopt an attitude which ignores it.

Bandwidth Happens

I'll end up sounding quite like a broken record in this article, but it's for good cause. If you're sending a message over the wire, you're consuming bandwidth. The best strategy to prevent this from becoming an issue is to not send data over the wire. Don't architect a distributed system. That's not entirely satisfactory - some systems do need to be distributed. Some systems have to serve their users over a website.

A Cost of Doing Business

It's apt in a couple ways to suggest that bandwidth is a cost of doing business. Of course it costs money to consume; distributed systems need to pay for this, and large systems are going to have to pay more. Cloud providers know this, and depending on your case you might face an eye-watering bill. Each unit of bandwidth can be exponentially more expensive at higher levels, and this is a more immediate limitation on our systems than that absolute bandwidth is capped.

Describing it as a cost of doing business is apt also because any distributed system requires some consumption of bandwidth; this is a cost in itself. While we have some control over whether messages in our system incur latency, we have no control over that they use up bandwidth. This potentially makes bandwidth-related problems more severe; there are fewer tools at my disposal to mitigate these problems.

Induced Demand

Induced demand is a term used in urban planning, it's a phenomenon whereby increasing the number of lanes in a highway causes traffic problems to worsen as it causes more people to choose the highway to commute, thinking that the added capacity will benefit their travel. I think the same thing happens to us with bandwidth. As our bandwidth capacity has increased, users expect to be able to consume more data, businesses expect to be able to ingest and work with more data, and as engineers we architect and build systems which fulfil these requests. We see they're constantly adding more cables and we see other systems working with much more data, we feel more and more comfortable transporting more of our own data.

Media and streaming has become more ubiquitous and has come to incur a higher cost. Ten years ago I was happy with 720p video from YouTube, now I can stream 4k! I stream 4k video from Netflix and haven't touched a DVD in years, I stream music all day instead of listening through my CD catalog. Videos play instantly on my social media apps now.

IoT has caused a huge proliferation of devices connected to the internet, and if these devices aren't streaming data somewhere they're phoning home with some frequency. Cloud computing has made it cheaper and easier than ever to set up systems that add to the problem, and lately even some small sized companies are interested in what they can do with big data.

How much of the data we're sending over the wire is erroneous? I don't know the exact number, but I know it's not 0% - I've written software that's piped unnecessary data between services! I apologize for my part in this. For what it's worth, I do try - more now than earlier in my career - to keep the amount of data to a minimum. We should all be proud to build a system that is as light on the internet infrastructure - and our private infrastructure - as possible.

Mitigating Patterns

Once we've eliminated erroneous data from being transmitted around our system, we can take some steps to reduce the strain we put on the cables. The smaller we can make our messages the better, and we have a couple of patterns and practices to follow here.

If we've reduced our packets as much as we can, then we need to deal with the physicality of our traffic. Monitoring and prioritizing critical traffic or geolocating where the bulk of that traffic happens can give us more fine-grained control over keeping our bandwidth usage in check.

And if those strategies fail, we can always throw our hands up, despair, and yell at the cloud. Haha, get it? The cloud!

Serialization and Compression

The first and most obvious tool to reduce the amount of data sent over the wire is ... reducing the amount of data sent over the wire. Maybe that seems obvious but a lot of folks miss this first step, especially serialization!

JSON has become the ubiquitous way to serialize data, but it has plenty of inefficiencies, other formats use fewer characters to represent the structure of the data and can be parsed much faster. Indeed, I would expect that there's a correlation between how many characters a format requires to represent a structure and the speed of (de)serialization.

For those of us in the .NET world, MemoryPack is an attractive option, being up to 200 times faster than the first-party System.Text.Json and coming in at a smaller serialized output than JSON. The catch of course that prevents this from being used in a lot of distributed applications is that the client needs to be able to deserialize it - you'll need to either use .NET or their TypeScript code generator. At least if I use JSON then anyone will be able to deserialize it.

An attractive option then is protobuf, a fast and small format developed by Google. This format is relatively ubiquitous and has demonstrated real-world effectiveness: LinkedIn adopted it and reduced latency by 60%! Though they measured latency, these gains were due to smaller message sizes resulting in a reduced impact on bandwidth.

Compression is the other factor here, and it's a saving grace that there are a fair number of compression algorithms available to us; we need to worry less about language-specific tools when we consider compression. Algorithms like gzip and brotli have many implementations in every language, and MemoryPack even comes with a brotli compressor built in. However, speed is crucial here. Plenty of compression algorithms are fast enough but this is important to consider.

Claim Check

Simply put, this pattern suggests using a URL to the location of a resource rather than the resource itself as the payload across the wire. You've surely encountered systems where images or other files are stored in a CDN separate to the application's persistence, and URLs are used to represent these resources. This is sort of the same idea.

This is a good pattern to use with asynchronous communication if events contain enough data - you can store the payload in a persistence system and just pass the URL as the event content. Naturally though, you might need to include some subsection of the data with the URL so that consumers can identify the data and choose whether and how to react; you wouldn't be saving much bandwidth if each consumer needed to fetch the data at the URL to check it.

Naturally, this is much more effective for larger payloads, and might be harmful for very small amounts of data. I'd expect though that you're not running into bandwidth issues with small events though. I would also suspect that geography plays a part in deciding at which size of resource this pattern might be a benefit; penalties for long-distance data transfer are not incurred solely due to bandwidth limitations!

As an aside, I will say that I've never actually seen this pattern used except insofar as keeping images on a separate CDN could be considered related. However, I was speaking with a colleague the other day, inquiring about his firm's approach to mitigating bandwidth concerns, and he said something to the effect of "If we have bandwidth issues we just 'S3' it." I suppose this pattern is quite natural then!

Traffic Management

In a lot of cases, bandwidth becomes an issue when it affects a particular type of resource, especially content that we need to deliver to a user or data that's otherwise in the "hot path". It makes sense that if we're going to have issues with bandwidth then we're going to at least want to be able to keep those issues away from certain paths.

Managing traffic to shape bandwidth concerns can be as simple as setting bandwidth or rate limits on different parts of your network. By throttling non-critical systems you're naturally freeing up resources to be able to be consumed by other paths. This can require some monitoring and fine-tuning, but it's a simple solution that can be effective.

Depending on your use case, you might either want to buffer or drop packets when bandwidth limits are hit. I think the default option would be buffering for obvious reasons - we generally want to preserve data. Dropping packets is generally preferable though in scenarios like streaming audio or video to a client (where buffering can compound on itself) or for non-critical data like logs or telemetry data, if the occasional loss doesn't have a significant impact. IoT comes to mind here - systems which stream large amounts of sensor data will probably benefit more from dropping packets.

Setting up a priority queue is only slightly more involved but a more elegant and controllable solution. Message queues typically support this feature, as do gateways like firewalls and load balancers. From an architecture perspective, it's probably good practice anyway to intentionally identify and appropriately segregate the critical parts of the system anyway.

Edge Computing

Redesigning your application to run at the edge is probably (read: certainly) overkill if you're just trying to solve bandwidth issues, but it should be clear that edge computing gives you more control over how and where you send messages, which allows you to shape bandwidth use at different points in the network.

By handling local traffic locally, you're not just keeping data away from your central system, you're also potentially shortening the wires over which the bulk of your data is transferred; bandwidth issues are more issuesome at larger distances. This is almost a requirement anymore for streaming applications - I can't imagine that Netflix could service most users streaming 4k data from New York to Hong Kong. The centralized systems typically get the lion's share of bandwidth allocation, but interestingly edge computing challenges this because the nodes of the network become bandwidth-heavy. Again, this is an advantage for systems which have bandwidth concerns that are heavily geographically-influenced.

Edge computing adds a fair bit of complexity and a heap of cost though, and that's why I suggest that it's not a solution exclusively for solving bandwidth concerns. I think it's fair to be skeptical of edge computing except for cases that truly require it. If you are computing on the edge though, you're probably dealing with a fair amount of data and one of the multitude of concerns which should motivate the architecture of your system should be these bandwidth considerations.

Conclusion

Our key first strategy to minimizing latency concerns is to send fewer requests, and our key first strategy to mitigating bandwidth concerns is to send less data. This seems like we're trapped then, like there's a hard, physical limit to the amount of data we can send at any one time. Indeed, this is the case. It seems like we're naturally pushed towards not distributing our system. Again, this is not just a reasonable conclusion but probably the correct one. Distributed systems are more complex, more error prone, and more costly. By being careful about what we distribute we can manage that complexity when we have to.

In those cases where distribution is required, approaching their design with all of these fallacies in mind will only get us so far too. Large streaming systems still fail with some frequency and can only fall back on managing their users' expectations with a thoughtful error message when there simply isn't enough bandwidth to deliver their video.

The Best Business Brunch in Minneapolis (So Far)

Fri, 02 May 2025 00:00:00 Z

I've been working as a software engineer in the Twin Cities for quite some time, and I value keeping up with my colleagues here. The best way to meet up with folks is a well-timed Business Brunch™, and Minneapolis offers many great options for a successful business brunch. What I look for in a place is fast service, food that's good and familiar, an atmosphere that allows conversation, and well-priced mimosas (most importantly). Over the last year and a half I've been branching out and trying many places for business brunches, and I'm happy to share this list of the best spots so far! If you're in Minneapolis you probably already know many of these spots, and I can recommend them for a business brunch. If you find yourself visiting and need to meet a colleague, then look no further:

Book Club 2/2024: Recovering from TDD and Unit Tests

Sat, 24 Feb 2024 00:00:00 Z

I'm on the record as having labeled myself a unit test hater. This is perhaps a bit extreme taken at face value; I like to advocate for approaches which consider the pros and cons of all solutions, and matching the right solutions to the problems they best solve. More often than not, unit tests are not the right solution; largely I find they corrupt our codebases, encouraging us to twist our architectures and use poor engineering practices. To be sure though, there's instances that unit testing is appropriate.

But what kind of "unit testing" is good anyway? Forgive my diversion into definitions here, but this is important: are we talking about the same thing? One engineer might take "unit testing" to mean any form of testing where I'm only testing a single function or method, while another might take the same term to mean the specific practice of abusing mocks to white box test the various code paths of each method containing business logic. Can you see my bias? The former is quite agreeable, the latter is the source of much consternation. Insofar as the latter definition is a subset of the first, then I might say that I prefer the difference of the two.

Given the general kind of unit testing which is good then, we can perhaps more specifically define the proper sort. A coupling of the test itself to the implementation of the code it tests should, in all but extreme cases, be off the table. Further, example testing whereby we match outputs to known inputs leads to fragile tests, missed cases, and generally avoids addressing the purpose of the method in question; the proper sort of unit tests test the relationship between the input and output, not the specific cases. Given these two properties of proper unit tests, I could specifically cite forms of parameter-based testing such as property testing and fuzz testing as being ideals. Some others on the edge can be interesting too: mutation testing and contract tests both have their uses but should be strictly used within the confines of our testing rules.

Now, let me be quite clear here: drawing a line in the sand saying a unit test can't be coupled to the implementation it tests means we can't use mocks or fakes to instantiate a class to test its methods. Indeed, these testing methods are probably only proper for "pure" functions. I think this derivation is a sign I'm on the right path here; I take that indication from two more fundamental principles I hold. First, our testing is in place to test the behavior of our system, not to double or triple our implementation. Second, maximizing the amount of logic in "pure" functions is good. Perhaps you disagree with me on these; that may well lead you to a very different conclusion.

Understanding why we are testing - what specifically we want to gain with the tests - is perhaps the most important thing we need to sort out. As I stated, I'm interested in automated tests which verify that the system I've built satisfies the requirements it was given. Realistically, this should mean a full decoupling between the suite of tests and the code it tests. Here's the catch then: if I'm testing individual methods, I don't have total decoupling. I couldn't (or it would be quite difficult to) write my tests in one language and the system itself in another language, for a more extreme example. This is where I come in with functional testing strategies like integration tests - not that I'd necessarily want to use separate languages, but I want to achieve that sort of decoupling.

This doesn't work in all cases. Can I test a mathematics class library with integration tests? Almost certainly not; testing for an example like that belongs to the domain of parameterized testing and the like. A microservice API though? I'm not sure that there's a lot here for which integration tests aren't the best and obvious choice. At least insofar as I want to test the business requirements of my system, integration tests are ideal for most of our CRUD systems. These are the tests that best let me define the constraints of the system, and that guarantee that it conforms.

This brings me to the topic of BDD and TDD. You might guess that I dislike TDD, and you'd be right. Well, a bit. I dislike the idea that TDD should be prescribed. If you want to write unit tests coupled to the implementation before you write that implementation, you do you. Throw those away before you commit your changes, but bring whatever tools to the table make you the most effective engineer you can be. BDD - Behavior-Driven Development - on the other hand is something I can get behind. By my reading, BDD is just giving a name to the way I'd want to develop software anyway: that everyone developing the system should be aligned on how it's supposed to behave before we implement it.

It's on this subject - TDD and BDD - that I focused most of my research this month. As I outlined, it seems to me that most issues we have with our automated testing strategies is that we seem to have a tendency to stray from the fundamentals here. Before even considering automated tests, are we aware of all of the testing strategies and patterns, and the scenarios in which each works the best? Have we fully thought through why we need tests and what they provide, not just in general but for the specific codebase we want to test? As with any other coding tasks, we first need to approach testing from the right orientation before we can start engineering. BDD and TDD are overarching process ideas which seek to orient our approaches to testing, so I think that these and other processes are the interesting thing to consider when studying testing strategies.

Finally, I feel I should touch on code coverage. If my goals for my automated tests are to ensure that it meets all of its requirements, then it seems to me that I'd want some kind of metric of "requirements coverage" or the like. If my code is meeting all of its requirements - that this is genuinely a big if - then I don't really care about code coverage, do I? Take an API for example - if I have integration tests set up which cover 100% of my business cases - both happy and sad path - then code coverage is just a metric of how much code I have in my codebase that isn't getting hit. It seems to me that the utility of code coverage is not as a primary indicator of the quality of my software, but rather an incidental heuristic which I could choose to use from time to time to help refactor bits of code.

Videos

Writing

The last two posts are referenced (and elaborated on) by a series of conversations by Kent Beck, David Heinemeier Hansson, and Martin Fowler: Is TDD Dead?

Book Club 1/2024: What is a Software Architect?

Wed, 24 Jan 2024 00:00:00 Z

This question - what is a software architect? - is a bit of an expansive one; I wouldn't think that I could ever answer it thoroughly, much less succinctly in a few article links. I do however think it's worth some time considering some of the different perspectives we encounter. I'd venture that a poll of 10 software architects would yield 11 answers to this question.

Is software architecture decidedly different from software engineering? The analogy with which these titles are made - that of architects, engineers, and builders collaborating over the construction of a building - would seem to suggest so. However, a software project is decidedly different (in almost every single aspect) from a construction project. This is my opinion, but I would put it to you that while "software architecture" is probably a definable thing, a software architect is not really so different from an engineer. Indeed, I would also assert that a proper software engineer must be engaging with and actively doing software architecture as a part of their job.

Should a software architect be doing anything different from an engineer? Well, perhaps not on the surface. I'd expect that an architect would be writing code about as much as someone with an engineer title, and I'd expect that an engineer would be as engaged with diagrams and ephemeral conversations about the nature of software as someone with an architect title. The separate titles then are probably most useful as a way to distinguish the primary focus that one has within a software team, and an architect would be working differently from an engineer insofar as the architecture would be their primary focus, or area of responsibility.

So is it useful then to separate ourselves out with titles of "engineer" and "architect"? There are advantages to having the separate architecture title though: to motivate the team to keep architecture as a primary consideration, to be a figure of wisdom or authority with respect to the architecture, or to be the "point person" to communicate architectural concerns to other teams or other departments within a company. Perhaps it's not a useful title to have, but a separate role or "hat" which a member of a software team wears; in fact this was the way I was introduced to the concept of architecture.

Having both had the title of "architect" and having been an engineer at firms which had this title, I can see its utility. To caution though: just as every piece of software, every product, and every team is different - sometimes radically different - so too are the roles of those we call architects. This is perhaps why there are so many different ideas of what architecture is, and what a proper software architect ought be. Perhaps there can't be a single, specific definition to encapsulate the idea, as the types of architecture and architectural concerns are necessarily so varied between software efforts, domains, and technologies?

To try to pin some universals down, I think I could say a few definitive things about architects. Architects should be coding, and actively involved in the engineering process. Architects should primarily focus on assisting all of the members of a development team - helping to ensure alignment and stimulating everyone to think architecturally. Architects shouldn't be dictators or cudgels which make demands - a somewhat common perception which is unfortunately sometimes earned. Architects should strive to be the most approachable and collaborative members of a team.

While architects should be very technically knowledgeable, I don't know that they should have all the answers when it comes to architecture itself - curiously, I think the best architects are ones which are very creative and can help brainstorm architectural ideas, the idea being that they act as an enabler of a team to define a good architecture, rather than dictating an architecture from the start. In a difficult twist, this does mean that sometimes the architect will need to furnish a specific recommendation upon request, for example for a team in a tight bind which requires a quick resolution to a difficult problem.

I can hear a colleague of mine saying now: "This answer is so nonspecific as to be only slightly more helpful than a software architect theirself!"; unkind but not unfounded. I put it to you though that the composition of any software team is a jigsaw puzzle balancing who can do what with which areas of expertise are required of the technology and domain, and from team to team the jigsaw piece labeled "architect" will be a very different shape and in a very different place in the overall puzzle.

Links

I was recently recommended Mark Richard's series "Software Architecture Monday", which is a part of his site dedicated to developing more architects. This series is well-thought-out, and contain a few excellent articles/videos which contain insights directly related to this topic:

Book Club 1/25: Results, Railways, and Decisions

Mon, 27 Jan 2025 00:00:00 Z

I've written before on my enthusiasm for the result pattern, but I'm going to expand on that a bit here and how it relates to testing. I've spent a fair chunk of my past work-month standing up a new distributed service at work, and it's such a simple project as to almost have completely written itself, except for one opportunity I took: I wanted to see how many ways I could approach black-box testing the API. If you know me, you know I'm a big fan of writing automated tests ... in one way - I really really like integration-level tests, and even then just covering successful vs failing input and checking state change is robust enough for the vast majority of our systems. I usually do this and then move on to actually implementing the darn thing, and that's perfectly fine. I'm poorly-versed in other methods, and so that's what I've been reading this past month.

I promise that the result and railway design patterns do fit into my black-box testing ideas here; let me take a step back to motivate that. Last month, I spoke with a number of folks about the requirements of this API. It's only got a few endpoints, but there was some complexity around external systems it would need to consult - who has what data - so there were a number of models of the API developed before I coded anything. Goodness knows there are dozens of ways to model systems, but these (advantageously) took the form of control flow diagrams. Here's a simplified version of one for an endpoint to get a resource, which attempts to create the resource if it doesn't yet exist:

Get the resource from the database
- If it exists, go to step 4
- If it does not exist, proceed
Consult a cache holding some data about the resource
- If there is no data in the cache, respond not found
- If there is data in the cache, proceed
Create the resource in the database with the data from the cache
- If this fails, respond unexpected error
- If this succeeds, proceed
Consult another cache to get more data about the resource
- If this succeeds, merge the datem and respond successfully
- If this fails, respond with the partial content from the database

This is a real treat to have! For all purposes, each step represents a different result-returning method - and the results are right there, in easy-to-read markdown no less! Not only does this neatly model my results, it also neatly models the railway of the system. If you're new to the railway concept, it's quite straightforward. Almost all of our business logic can be thought of as a railway in that it has two tracks. There's the success track and the error track. The logic progresses through several steps along the success track, but any step might be unsuccessful and divert the control flow onto the other track. My model above is a bit more complicated because of the branching control flow and there being multiple success and failure paths, but this demonstrates the utility of this model: it keeps the extra complication straightforward and comprehensible. If I were to draw a railway diagram I might have (again, simplified):

This might be a stylistic preference of mine, but there are two things at the top of my mind when I develop business logic. The first is what the railway of the logic looks like - I encounter incredibly little business logic that is not made more clear by railwaying. The second thing at the top of my mind are the results of the operations that create decision points where I might switch tracks. Those two things wholly constitute the structure of most any business logic: what tracks I have and the events that move between them. The business logic queues operations in order; those operations operate and report result; the business logic decides which results, if any, change tracks; repeat. I hope I can motivate my enthusiasm at being able to have these models as an output of my engagement with the stakeholders in the system - the code really does darn-near write itself.

Side note: "railway" is more of a way of thinking, not really a pattern and not really an orientation. Maybe I could say "the rail...way of thinking!" Please laugh.

This brings me to testing, and a hypothesis: I figure that just as these models were able to prefigure the structure of the system's code, so too they would prefigure its tests. This seems somewhat obvious: the model is a graph and by traversing each possible path through the graph I can generate all of the test scenarios necessary to cover all of the branches in control flow and ensure proper responses. For the graph I've given you it's quite easy to do this manually, however larger graphs whose nodes have more results (e.g. different types of successes or failures) will become unwieldy. This is the main thrust of my research this month, with which I hacked together a tool to generate test scenarios for me from these shared models.

Owing to the large number of different ways we can diagram our systems, there are an equally large number of ways to generate test scenarios from these models. A collection of knowledge on this sort of practice can be found under the term model-based testing. This sort of approach certainly has limits though. In my case the models were a product of the kind of collaboration I ended up doing for this project, and surely the utility of the specific testing approach I chose here is also influenced by the architecture of the broader system and the culture at my firm. Indeed, the approach might have to be quite different (or not used at all) on other projects.

It is a very interesting consideration though, along with the rest of the resources available for test generation. I'm quite contented by the results of my exploration here; it's given me a more robust and flexible framework to approach testing!

My related writing:

On results and railways:

A final note on the result pattern (on which I should probably elaborate in a future article) is its utility lies in modeling expected failure cases which have associated business logic. This is a distinction between bug, error, failure, or the other sorts of paths the code might take.

Software Testing: Defect, Bug, Error, and Failure - Rahmat Ullah Orakzai gives a good overview of this
Bug, Fault, Error, or Weakness: Demystifying Software Security Vulnerabilities - Irena Bojanova and Carlos Eduardo C. Galhardo
The Error Model - Joe Duffy gives my favorite explanation and prescribed solutions here

I don't know of a better overview of the railway concept than that from Scott Wlaschin. He's also written a bit on property testing:

On testing

Geeks for Geeks gives an acceptable overview of concepts
Control Flow Software Testing - Geeks for Geeks
Introduction to Model-Based Testing by TorXakis begins a large repository of knowledge on the topic
Automated Combinatorial Test Methods – Beyond Pairwise Testing - D. Richard Kuhn, Dr. Raghu Kacker, and Dr. Yu Lei
Combinatorial Methods for Trust and Assurance - NIST

Book Club 1/2026: C#, Without Allocation

Sun, 25 Jan 2026 00:00:00 Z

I'm not very good at coding for performance. I can do some rudimentary profiling, and if you give me a couple metrics to optimize for I can do a pretty good job getting those numbers down. When I sit down to write a piece of software though, performance is maybe my second or third concern, after ensuring the software works properly, it's well-formed (for readability and extensibility), it's well-documented, I can deploy it easily, and the like. Intentionally, I write software both in a language/runtime and in the sorts of domains for which this approach works.

I don't think that every engineer on a team needs to be extremely performance-focused. We have colleagues who are fantastic at performance, and I hold their contributions to a high esteem. At the same time though, it's no good allowing anyone to think it's alright for an engineer to neglect developing a decent knowledge of engineering for performance, and it should especially be discouraged for anyone to not consider performance altogether. The worst offense would be to develop a habit of slopping performance concerns on our performance colleague; often code written without an eye towards performance is code that can't be made more performant.

I know plenty of folks will disagree with my enumerating "performance" among the sorts of skills that software engineers can have - for the above reason no less - but the fact is that there are many different sorts of considerations that need to be held in the head of a software engineer, and it's natural that we will each be good at some considerations but not all of them. Again, this is not to give anyone a reason to neglect performance. Instead, this presents an opportunity to ask: how frequently do you review your own code through a performance lens?

My answer is "not as frequently as I'd like," which I think is a good opportunity to practice more performant development. One aspect of writing performant code, particularly in a garbage-collected language like C#, is to consider how the runtime allocates data. Reducing allocations in languages like C# or Java isn't just about making sure less memory is used - more memory use also increases garbage collector time, which can result in its own set of difficult-to-debug issues if left unchecked. Using the wrong kind of object (class/struct) depending on your use case can cause copying where there doesn't need to be any.

Fortunately, we don't need to go looking through IL to be able to learn some fundamentals that can significantly help us here! I've collected some resources on writing allocation-free C#. Notice: there's a lot of repeated information between the sources isn't there?

Before listing these links though, I want to point everyone to one of my favorite C# engineers, though I have never spoken with them. Yoshifumi Kawai (@neuecc) and their company Cysharp have published a wealth of open-source projects that provide a master class in practical, performant C#. From a zero-allocation LINQ implementation to maybe the fastest binary (de)serializer for .NET, I am a huge fan.

Reading

Writing

And a preview of more performance things...

High-performance code design patterns in C# - Konrad Kokosa

Book Club 10&11/2025: .NET 10, C# 14

Thu, 27 Nov 2025 00:00:00 Z

Happy Turkey Day! Last year when I published my newsletter on Thanksgiving, I was thankful for my colleagues who make a habit of saying "no.". This year I'm thankful for my colleagues who keep .NET applications updated to the latest versions; it's not just a burden to have to maintain many codebases at different versions, but each new .NET is a significant improvement over the last. .NET Framework had cultivated a well-earned reputation as a bogged-down runtime reached for by large enterprises who care more about their relationship with Microsoft than the quality of their line-of-business software. It could be said that .NET was born as MS-flavored Java and stayed there.

.NET (just net, not "framework" or "core") has, I think, completely lifted away from this problem. Not just multi-platform and AOT support, but performance optimizations and the inclusion of libraries that significantly improve development capabilities in networking, security, and performance, combined with an explosion of features in C# and F#, have put the ecosystem on-par with the best-optimized runtimes/languages. It seems more that Java is playing catch-up to .NET anymore. Great success!

The latest release this month, .NET 10, continues all of that but at a manageable pace. Microsoft has been sprinting with recent releases that have dramatically altered the .NET world, but .NET 10 is not shaking anything up that much ... on the surface.

A couple important developments though. First, they're calling Aspire just "Aspire" now, not "Aspire.NET"; it appears that they're attempting to develop this into a multi-language tool with utility for folks outside the .NET ecosystem. Reading the tea leaves from many miles away I might guess that Microsoft wants to create a platform for any engineer to consider for distributed application development, and this would make sense with my understanding of Aspire being a plot to sell more Azure. Helpfully though, with .NET 10 allowing us to run single-file C# applications, Aspire can be segregated to some single file instead of needing a whole CSPROJ and separate directory and the like.

Speaking of running single files, yes! This is important for my Metalsharp tool that generates my blog; I will be moving this forward to version 1.0.0 soon! You can see an (admittedly somewhat complex) example of a single file app on my blog's repo. The fun bits are at the top.

C# got a new field keyword exposing the compiler-generated field on each auto property. Finally. This should have been included along with auto properties back in the day but what can you do. More interesting though, C# has an extension keyword now:

public static class Extensions {
    // New extension keyword groups a bunch of extensions together for a type
    extension(SomeType thing) {
        // "static" and "this" are not needed!
        public string GetHello() => "Hi";

        // Whoa look it's an extension property!
        public string SayHello => "Hello";
    }

    // But wait there's more! What if I don't name the extension argument?
    extension(SomeType) {
        // Whoa look, static extensions!
        public static string Hello => "Howdy";
    }
}

// Outputs:
// Howdy
// Hello
// Hi
public void TestExtensions() {
    Console.WriteLine(SomeType.Hello);

    var thing = new SomeType();
    Console.WriteLine(thing.SayHello);
    Console.WriteLine(thing.GetHello());
}

This feature is really cool and seems innocuous. Well, in fact this is an important step towards what the C# team has called "extension everything." The idea is that everything should be able to be extended at any scope, and it's an idea that goes back to the beginning of .NET Core. The language team has been very slow and deliberate in considering this feature, and I expect it to be roled out bit-by-bit in many versions. This is the first step to that.

Also absent from this release: discriminated unions! Not my favorite feature but my most-needed feature for so long. There's been important developments from the language team on this front though and I expect to see the first iteration of these roll out next year. I am impatient though so I wrote a source generator and analyzer as a proof-of-concept so I can have actual unions now, which I've aptly named UnionizeNow. Full explanation and kind of a blog post in that repo. It's very cool! I might not make this into a full Nuget package since unions are, in theory, around the corner, but two important things for me with it:

It frees me to start developing union-dependent ideas in a way that will be mostly compatible with the unions that the language team will roll out (their thinking is far enough along to give confidence here), and
If the first iteration of unions are lacking some small features which I need, I hope I could easily retool this generator to graft those on.

Here's more reading on new .NET:

Watching:

The Coolest Feature of .NET 10 is Here - Nick Chapsas

Book Club 10/2023: Functional Patterns in C#

Fri, 27 Oct 2023 00:00:00 Z

Happy spooky season!

This month I've focused on functional domain modeling and related patterns. We're just a few weeks away from the release of the next version of C#, and like each previous version it'll introduce even more functional features. We still aren't getting discriminated unions, but as C# becomes more functional, these patterns are becoming increasingly more attractive. Traditionally, C# is written in OO or procedural styles, and from my perspective there doesn't seem to be a great deal of discussion among C# engineers about incorporating functional patterns. Maybe you run in different circles, but I think there's room for improvement across the board here. Even the result monad, which can be used within an entirely OO context, is infrequently implemented.

I think it's important to be discussing functional patterns in C#, for a few practical reasons:

Eventually we're getting DU, and that's going to change a lot of things
Our F# colleagues are doing great work in this area, we should engage them more
Using functional patterns is cool

But above all, if C# supports these patterns and they can help us write better code, why would we neglect them? To explore this topic, I've curated a set of talks by Scott Wlaschin and Mark Seemann who both do a great job explaining functional programming from a conceptual perspective, demonstrating its power in F#, and ultimately demonstrating C# equivalents.

I hope these talks make you all as excited for discriminated unions in C# as I am!

Book Club 10/2024: Fallacies of Distributed Computing

Mon, 28 Oct 2024 00:00:00 Z

Happy Halloween! Boo! I'm celebrating now for two reasons. First, the weather is very nice where I'm at; I prefer wearing more layers and Fall is the perfect month for that. Second, I just finished my series on the fallacies of distributed computing - perhaps the world didn't really need it but I was here to provide it.

Much earlier this year I figured that writing this would be a good way to exercise the writey muscles and allow me to express some thoughts on a topic I surely have some thoughts on. I think I achieved those goals, though perhaps such endeavors in the future could focus on being more interesting. This is a relatively dry topic as it stands, and I don't know that I put a lot of creativity into it. Nonetheless, while this is a frequently-commented-on topic, I feel that I was able to go more in-depth on a number of points which are lacking in other writing.

In particular, a lot of writing on this subject doesn't emphasize the difficulty of distributing. It's not just non-trivial; I'm firmly committed to the position that a distributed system has 10x as many things as monolithic systems - 10x the number of bugs, 10x the amount of development effort, 10x the number of meetings (both to plan and to figure out what's going on), 10x the maintenance cost, and so on. To me, this is the implication of the fallacies: there's a ton of work you can't forget to do. Why would anyone ever distribute their system?

Of course, there are problems that are unsolvable but by distribution. It's the only option if you need to independently scale some component, which perhaps any system at a sufficient size will need to do. Before we can ask how to distribute a system, we need to ask whether we should. Once we've identified a need for a discrete component to be distributed, then we need to go about the how. That's going to be different in different cases, and the fallacies do a great job, in my mind, of guiding our conversation around that question. I didn't prescribe a lot of "do this and that" in my series because I don't see that as the point. It's the mindset that's the point, the architectural mindset: how do I consider all the tradeoffs there are?

Being clear, there are major tradeoffs. You can't solve for all of the potential issues. Look back at latency and bandwidth: these two are related and trying to reduce one will (almost certainly) increase the other, unless we can consume more resources. We can only consume more resources to a certain point, so what gives? Well, nothing gives. You're going to be vulnerable somewhere, and you can choose where you want that. This is, of course, the point to reiterate that the going advice ought be that we shouldn't distribute in the first place unless it's absolutely necessary.

So the best default we have in doing development is to create a well-architected monolith. One with really good principles and domain segregation. Yeah that's tough, but you're not decreasing anything by distributing - especially if you look to it as your savior. With load there'll come components which require distribution, and we separate those out at this point. This architectural prescription will (probably) not ever result in a microservices system. Rather, we'll end up with a very dapper trunk-and-branch architecture, where we have a monolithic trunk surrounded by individually-distributed branches. You might have a "leaf" layer of common services, and that's it! Keep the possible paths through the system shallow. Easy monitoring, easy debugging, you can keep everything under control.

The distribution-skeptical monolith-to-trunk-and-branch engineer will typically architect a more resilient and adaptable system than the distribution-zealous microservices engineer. I wonder if coming years will be characterized by an equal zeal for the "modular monolith", which would end up being - to my estimation - hilarious. Ours is an industry continuously held hostage by trends.

Today I'll share a bunch of related links, particularly ones which came up in research for this series. It's helpful to see what others are writing, both to get a sense of the area and to avoid repeating what others have offered. Particular Software, which maintains the popular NServiceBus, keeps a blog that's worth reading. Naturally they also have a series on the Fallacies, and while sparse it's a good resource. I was particularly influenced by their final post on the eighth fallacy, which enlightened me to the idea that the eighth fallacy can look at every aspect of heterogeneity between system components.

Particular is run by Udi Dahan, who has given a number of talks on distributed computing in the .NET world, and has a series focusing on the fallacies.

Other writing on the fallacies:

Fallacies of Distributed Computing Explained - Arnon Rotem-Gal-Oz, a paper from the University of Wisconsin that seems to be frequently shared.
The Fallacies of Distributed Computing — Latency is Zero - Ryan French seems to be part of an unfinished series.
The Network Is Reliable - Peter Bailis and Kyle Kingsbury
A Guide to Managing the First Fallacy of Distributed Computing - Anadi Misra
What are the main Cloud Design Patterns? - Milan Milanović

I think Renegade Otter's article Death by a Thousand Microservices is one of those articles we'll still be sharing in ten years. The ugliness created by the zealous pursuit of distribution is described well here. My favorite observation from that article is that Shopify is a Rails monolith (presumably a trunk with branches) and has a revenue in the several billions.

Finally, Mark Richards is an architect who has made a fair career out of being good at explaining fundamental concepts in architecture. Two recent talks of his which I think are particularly good and pertinent to this topic are Elements of Distributed Architecture and Decomposition Patterns.

Book Club 11/2023: New .NET, New C#

Wed, 22 Nov 2023 00:00:00 Z

I'm looking forward to turkey day tomorrow gobble gobble! This year I'm thankful that I work in ecommerce so I get to have a peaceful extended weekend because nobody visits ecommerce sites on Thanksgiving weekend. At least that's what they told me in the interview before they hired me. Insert joke about how it's better to be working in ecommerce than at OpenAI this weekend regardless...

Anyway, last week we got the new release of .NET, which brings langauge and tooling updates across the board, so I want to focus on some of those fun things.

First, Blazor has taken the third (or is it fourth now) of the 1,000 steps it needs to take to become a viable platform for SAAS, with hybrid client/server rendering. I don't have a lot to say there, but I use Blazor for a number of personal projects when I need to quickly draw up a UI to look into some .NET backend scenario or another.

Since Microsoft started down the path of .NET Core, the whole ecosystem has been embracing OSS and free software in a way that's completely rewritten the whole modus operandi of Microsoft under Nadella. Indeed, it seems like at this point in time, you can use .NET without a single worry about vendor lock-in. Well, Microsoft is here to save you from that horrible wasteland of unrestricted freedom with .NET Aspire. Nevermind that you probably don't need a distributed system, and even if you did you almost certainly don't need microservices, they have cloud computes to sell you! Aspire makes it easy to avoid footgunning yourself as you begin your next project distributed from the start by skipping you right to the step where you blow your foot off with a bazooka - all hail the mighty Azure! Or, you know, if like 99.99% of all apps out there you'd be fine with it deployed in a Docker container with a couple of related services, you could just use Railway. Note that I'm definitely queuing up an article on using Aspire with Railway despite my skepticism that Aspire is a good idea.

C# hasn't gotten too many updates, but we have two syntactical updates that are essential and should have been included much earlier: collection expressions and primary constructors for classes.

Collection expressions, or perhaps "enumerable literals", should have been a part of the language from the start, and you should convert all of your code over to using these.

var list = [1, 2, 3, 4, 5];

Records have had primary constructors since they were introduced, and I think that was largely related to the desire to have tuple interop (is interop the right word here?), but now classes have them too except quite different. The parameters in a primary constructor for a class are, more logically than records, private members of that class, significantly reducing the amount of boilerplate if you're still using dependency injection. It's been how many years since Scala came out, but now we can be one of the cool kids on the block too! Right?

Anyway, I'll just leave you with a few talks from the .NET conference with some of the other tidbits that should be used in .NET going forward:

And then a couple of talks from the lead designers of C# and F# regarding the history and direction of each language:

Book Club 11/2024: No

Thu, 28 Nov 2024 00:00:00 Z

Happy Turkey Day gobble gobble! Those of us in the US are celebrating Thanksgiving Day, and today I'm thankful for all my colleagues who make a habit of saying "no". Does the period go inside the " in this case? I've always been unclear.

Anyway, I'm not a big fan of "yes" - "yes, we can add all those features", "sure you can add those extra layers to the architecture", "Decompose our microservices into nanoservices in spite of only having 12 users? You betcha!" These are kind of silly examples admittedly but it's a common picture that's been painted by plenty of opinion-havers in our industry before me. This isn't a problem unique to ours too, the so-called yes men are everywhere.

I spend a lot of time speaking with my colleagues about the importance of qualifying knowledge, learnings, and opinions with conditions; too many of us are too absolute in our thinking. Instead of holding the opinion "we should always use onion architecture" it's better to assert "if we have a large monolith that requires robust domain segregation, then onion architecture is preferred." Whether you hold either opinion or not, I hope it's obvious that we would all prefer to work with the engineer holding the second. When we learn new things or develop new strategies or whathaveyou, the success always happened in the context of some environment, and the facts of that context are as much a part of the success as the actual good thing that was implemented. Our work is more robust when we incorporate those facts into our learnings in these qualifying conditions.

This is a way of not saying "yes" at each opportunity. In my example, we change from "yes, let's use onion architecture" to "maybe, let's see if we should use onion architecture." The same comes from the other direction with "no" statements. Think of all the software engineering opinions you hold, and let me ask you how many are "no" opinions? I mean propositions like "don't use onion architecture" or "don't put opening curlies on the same line" or the like. Now of those "no" opinions you have, how many of them are just the opposite of a concrete "yes" opinion you hold? I mean, for example, if you hold the opinion "don't put opening curlies on the same line" you also think "always put opening curlies on the next line." When I did this exercise I found that a lot of my "no" opinions are just mirrors of other "yeses."

These are equally unproductive, and transforming them into conditionals will do good. It's clear then that offering a "no" opinion, or saying "no" in any way, is always best served by making sure it's the right "no" for the context, and being open to the possibility of a "yes" instead. That said, there's infinitely more "no" than "yes" to be had at any given time. Think about the last trivial bug you had to solve at work - there's a million choices you could take that are obviously wrong. You didn't need to rewrite the whole application, you didn't need to distribute it out into a separate service, you didn't need to introduce a cache. Well, probably not for all of those, but you get the point: there are very few good paths and an infinite number of bad paths.

So "no" positions are significantly more common, and you're going to be in a better position defaulting to "no." This comes back around to our starting point being annoyed by our yes-colleagues: if most paths before us are bad, they get us into trouble indiscriminately saying yes. If only a few paths are good paths forward, skepticism and carefulness behoove us. Being good at saying "no" helps to keep us on the right course.

On a final note, I've heard from a few folks that their interest in the topic of "no" is more inclined towards a largely unexplored field of ethics within software engineering. I think there's a growing sense that we need some manner of organization in this area; our industry has kind of been flying by the seat of its pants for decades, and almost every other profession is better organized with a proper code of ethics. I can see how the "no" topic fits in here: what are our ethical obligations as to when to say "no?" I'll confess that I'm not terribly interested in this question at the moment, I was more interested in exploring how "no" can shape our outlook on the problem space, and the space itself. Certainly though I'll link some resources below.

Would it help if I spoke to your management? - Adam Ralph
No one can teach you to have conviction - Ben Kuhn
Grug on Saying No
Saying "NO" - A Superpower - Anand Safi
I'm David Heinemeier Hansson, Basecamp CTO, and This Is How I Work
Not Having Problems - Lucas F. Costa (loosely related)
Finally a general link to Renegade Otter, where each blog post is a "no"

Video/Podcast:

On Ethics:

Book Club 12/2023: Workflow, Process, and Agile

Sat, 16 Dec 2023 00:00:00 Z

Ho ho ho merry book club day - I'm sending this one out a bit early; tomorrow I'm headed on vacation and won't be back until the new year. Preemptively thinking about the new year and resolutions, I'm thinking about changes in workflow and process. How can we develop software on our own, on our teams, and in our organizations?

Several years ago a group of smart engineers drafted the Agile Manifesto to take a stab at figuring out the best way to develop software. I've yet to see a better summation of best practices than the 12 Agile Principles, however they end up being kind of vague and non-clear, almost like a fortune cookie or today's horoscope.

A lot of folks since have tried to come up with more concrete principles built on top of Agile. The most famous (or perhaps infamous) one is Scrum. That one is so common that today "Agile" is used interchangeably with "Scrum". However, I challenge you to read the Agile manifesto and principles and the Scrum guide and tell me that these look like they're actually trying to do the same thing. They're not. Scrum, at it's best, is a horrible corruption of Agile to try to shoehorn a better quality of life for software engineers into big corporate environments. At it's worst Scrum is a malicious attempt to keep a whole cadre of "scrum masters", "agile coaches", and other similar folks employed in cushy corporate jobs.

Agile, in spite of its generality, is the best starting point I know of to try to solve how to develop software, and Scrum ain't it. At the core, what I've always tried to stick to from the Agile thinking is to deliver value early and often, change the actual process itself as the team and product evolve, and eliminate any blockers to being able to do those two. While I think highest of Agile, I have several other bits and pieces in my toolbelt here. I like a lot of things about the Kanban process, and I start my day with a Grug quote, among other smaller bullet points I've picked up in my days.

What I've taken away from my readings this month are two new tools I'm going to add to my belt: #NoEstimates and "hypothesis over requirements", two concepts explained in the Allen Holub videos I share. Individually, either might be fine, but I think the real value of these are when used together. I doubt that these practices could just be adopted willy nilly by a regular scrum team tomorrow - these describe a way for business to work, as much as they describe a way for engineering teams to work. If all the stakeholders can be aligned on this though, and if the constraints on the project allow it, there's a potential for a huge benefit here.

But what is there for those of us stuck in regular Scrum teams that might not be able to adopt these right away, or ever? I've become very interested in this question, and I think I've found a few practices that could meaningfully help any engineering team. I'm certainly going to be bringing these practices up at my day job, and in future I may write on their success.

Videos

Articles to Ponder

The True Capacity of Your Engineering Team - Ant Weiss
How to replace estimations and guesses with a Monte Carlo simulation - Lucas F Costa
Actually, read just about anything from Lucas F Costa: lucasfcosta.com
Science and Software Development - Dave Farley

Things you can do Today

And to close out, I wrote a few articles recently that are on-topic:

Reclaim Your Agile: The One Clever Trick Agile Coaches Don't Want You to Know
Clean Meetings: A Software Engineer's Guide
My (Continuing) Descent Into Madness
A Scrum Odyssey (Okay that one's a lot of GPT but it was translating my writing)

Book Club 12/2024: Team Metrics and KPIs

Mon, 30 Dec 2024 00:00:00 Z

Happy holiday season and new year to you all! In the week between the holidays here I'm thinking about metrics and KPIs and the whatnot that we use in our team processes. Are they useful? My typical answer for any such question on this blog: yes of course, if they're being used well for a particular situation and so on and so forth. Measuring things can be good or bad.

Is there a feeling that metrics are largely useless? I don't know. There's plenty of opinions to be found on good and bad metrics, and good and bad use of metrics. I don't have a lot of patience to entertain a whole host of disparate, two-bit opinions, so I might suggest cutting through everything by asking: Are the metrics measuring something of value which are used to address a specific problem, and is the whole deal well-defined?

Most scrum teams measuring velocity are doing so out of habit, and that's probably useless (this is where the term "cargo cult" is often misapplied in blog posts like these), but other teams might have noticed some issue in their process and solved it by measuring a KPI. Specific measurement, specific problem, specific definition. The KPI for KPIs seems to be degree of specificity. Are meta-KPIs a thing? I did a Google search after writing that last question and can happily report that they are not a thing.

I've got a couple stories about metrics then, one bad and one good.

The bad one relates to code coverage. This, like velocity, is one of the typically useless metrics. Not because of some inherent uselessness, but it happens more often than not that they're misused in some way, typically they're there just for the sake of it. But my story about code coverage:

I now test all of my distributed services almost exclusively via integration testing of some sort. All the code of the service is a black box to the tests, which are able to cover all of the business and technical cases the service might fall into. I use Gherkin so that the tests are readable to all stakeholders, which allows a BDD process that greatly simplifies the process of quality assurance.

Is a code coverage metric useful here? Unequivocally, no. What would be useful here is some way to measure what percentage of the business cases the tests cover. That's probably not achievable, but code coverage is useless. If I have written tests to prove that the service satisfies all of the business cases, the code itself is quite beside the point. Indeed, these services end up with very a very high degree of code coverage, but that metric gives me nothing of value except the percent of code not contributing to satisfy the business constraints. Maybe I have added some code to output to a non-critical or temporary admin report. Maybe I'm testing a new feature, or any host of things. Even having code coverage suggests the code doesn't mean to be there. Even worse, the build gates on the repositories for these services have coverage requirements. That's even more useless - now I can't check in these extra bits of code without some manual exclusion from the coverage report! Alas, there's an organizational momentum behind the damned thing and there's only so many demons you can stomp on at once.

My second metrics story is, on the other hand, a story about a good use of metrics. Many years ago (how time flies, it feels like it was yesterday) I was leading a team developing a new SaaS product, and we noticed that work would go through periods of stagnation owing to PRs being open too long, and once several of them accrue bad things tend to happen. Indeed we discussed whether we needing a PR process and decided that we did want it, so we sought to address the issue by measuring PR open time and asking five whys to find all the root causes of this stagnation.

I crafted a spreadsheet as a sort of dashboard to show open PRs, their state, how long they've been open, and the like. I was able to regularly touch base with PR authors and reviewers when we noticed some slippage. I mean this to be a success story about a metric, but I figure the real star here is the "five whys" method (or, perhaps, any method to get deeper than the surface level of a problem) because indeed the real causes of stagnation were varied and not all what you'd have expected. I suppose then the success of the metric is it allowed those conversations to happen efficiently.

At any rate, in very short order we had addressed a number of underlying issues with the development process, and after just a couple of months we settled into a pace and process that allowed a phenomenal amount of code to be written over quite a bit of time. Great success!

I don't write on metrics or KPIs, but I do write a fair bit about process. Here's my reading/watching from the past month:

Reading

Listening

Mastering Observability: Unlocking Customer Insights with Gojko Adzic - InfoQ

Watching

Book Club 2/2026: Constraints

Sat, 28 Feb 2026 00:00:00 Z

Constraints, in a very broad sense, have been on my mind this last month. I have spent, as I imagine we all have, a lot of my career wrangling systems suffering from constraint problems. I mean "constraint" in the technical sense; the data or rules of the system have not been appropriately constrained. Interestingly, the age of a system doesn't seem to correlate to systems having poorly-implemented constraints; it seems to be a skill issue mostly.

Constraints are a first-class tool we have for defining our systems. It might not seem the most intuitive at first: if we're told "the system needs to be able to do X, Y, and Z," the first thought will surely be how we craft our systems for X, Y, and Z. However, by saying I want the system to behave some way, I'm necessarily also saying I want it to not behave in contradictory ways: if A, B, or C inhibit X, Y, or Z, then I have to also constrain my system to not A, B, or C. Constraints aren't enabling requirements, instead they are preventing errors and error states.

There's a lot of ways we can consider programming with constraints; constraint programming is a topic in its own right. I'm going to list a number of areas of focus I've had this last month, starting with the more esoteric.

Constraint Logic Programming

Or CLP, this is not something with which most of us are regularly engaged. Logic programming, is its own paradigm of programming, separate to the rest, concerned with crafting programs as repositories of logical facts. A program executes in this context when a question is asked about these facts; programs are essentially interpreted by logic solving algorithms. Take this example lifted from the Wikipedia:

is_parent(bob, sue).
is_parent(bob, john).
is_sibling(X, Y) :-
    is_parent(Z, X),
    is_parent(Z, Y).

And then a question to run a program:

?- is_sibling(sue, A).
A = john

This is already a form of programming by constraints; I'm constraining the definitions of parentage and siblingness. However, we can go more constrainey with better constraint-aware algorithms to do the solving. Hence we get constraint logic programming, which is indeed much more powerful and has practical applications. If you want to learn to program with constraints, this is the holy grail: programs are (I'll wave my hand a bit here) composed entirely of constraints.

Most problems - much more than you might think - can be very well-defined by their constraints; well-defined in the sense that the code I write makes sense for the problem. Remember that to programming constraints is to program out error conditions; writing programs with CLP then is probably the best-class solution for a whole set of problems. Programming a user interface, though, is not necessarily one of those problems; HTTP clients are probably not best served by CLP also. That leaves CLP in a bit of a bind: if I want some other language bookending my whole system, why keep a separate CLP in the middle?

Aha! Here is exactly the great practicality of logic programming: because the programs are just a particular application of proof-solving algorithms, these languages can be expressed perfectly naturally as libraries within other languages. MiniKanren is a lightweight DSL library with sibling implementations in most languages; C# has uKanren.NET. There's plenty other libraries in most languages, and frankly it's trivially simple to implement a proof solver. Decider is an actively maintained non-Kanren system. This gives you an engine in whichever language you fancy to express deserving logic using CLP. It's too bad that this is so often overlooked; really a huge amount of logic can be more elegantly expressed with these tools.

Reading

Watching

Dependent Typing

This is some crazy voodoo magic. Most programmers (especially if you're reading my blog) are familiar with type theory - maybe not at the theoretical level, but you intuit that this won't compile:

string variable = 12

string is a type, variable cannot have values that aren't strings, and 12 is not a string. This is incredibly simple: a string is a string is a string. A more complicated type might be an array:

var ints = new int[2];
ints = [1, 2, 3];

C# allows the above, but it is a curious bit of code. Why would I specify a length of 2 and then overwrite the array with a length of 3? C# allows this because the type of ints is just int[], but it seems like the author might have intended something else. Alas, C# does not allow its types to rely on literal values. This is what dependent typing is: allowing types themselves to have values. In a dependently-typed language, the above would not compile, as ints would be able to have type int[2].

This can be an extremely powerful way of programming, which puts constraints directly in the type system. In the array example, I constrain my array to only ever have length 2. Indeed, dependently-typed languages allow the programmer to apply any constraints that might be needed. So frequently we write programs expecting data to have certain shapes or properties or values; dependent typing allows us to write programs that make misrepresentations impossible, eliminating a whole class of errors.

This knowledge has less practical application. To start, dependent typing is a fundamentally functional concept; the type checking algorithms that can validate these sorts of systems rely on functional constructs. Being a more esoteric branch in the functional tradition, then, the syntaxes that implementors are drawn to are, could I say, difficult to read. Agda will let you use any unicode character in a variable name. From the Wikipedia:

data _≤_ : ℕ → ℕ → Set where
  z≤n : {n : ℕ} → zero ≤ n
  s≤s : {n m : ℕ} → n ≤ m → suc n ≤ suc m

Studying this form is still hugely beneficial, as it teaches a mindset of loading assumptions into the type system rather than runtime. This allows problems to be caught at compile time, and is the main tool for those looking to make illegal states unrepresentable.

Reading

Watching

Refinement Typing

While fewer folks yet are looking at refinement typing, I think there is vastly more potential practical benefit here than in dependent typing. Actually, the two dovetail. I'm incredibly excited about this area. I would love to see refinement typing become the next big thing; most languages could probably implement this with some ease.

While dependent types allow types to depend on values, refinement typing allows types to depend on some pure constraint about them - not a value that could be some program variable, but some predicate function. Though, I might try to explain it another way: most languages anymore have pattern matching syntax. C# has a robust one:

if (value is >= 0 and <= 100)
{
    //...
}

Although this is a large wave of the hand, I think of refinement typing as attaching some pattern matching statement onto a base type; the resulting refined type is guaranteed that this pattern will always match the value of the variable with that type. The above example matches some int with a pattern, perhaps we could imagine a C# with refinement types (I am not proposing a syntax, mind you):

int{>= 0 and <= 100} value = 12;

So long as we're not allowing runtime variables into this pattern statement, this is not a difficult task for the type checker. The power here, though, is immense: think of how many methods your code has with lots of guard clauses at the front? Example:

public User CreateUser(
    string username,
    int age,
    string email,
    decimal accountBalance
)
{
    if (username is null or { Length: 0 })
        throw new ArgumentException("Username must be provided.", nameof(username));

    if (age is < 18 or > 120)
        throw new ArgumentOutOfRangeException(nameof(age), "Age must be between 18 and 120.");

    if (email is null or { Length: <= 3 })
        throw new ArgumentException("A valid email address must be provided.", nameof(email));

    if (accountBalance is < 0m)
        throw new ArgumentOutOfRangeException(nameof(accountBalance), "Account balance cannot be negative.");

    // Implementation
}

If I could move these constraints into the types, I can eliminate the guards, catch errors at compile time, and ensure that callers are guarding data appropriately:

public User CreateUser(
    string{not null and {Length: > 0}} username,
    int{> 18 and < 120} age,
    string{not null and {Length: > 3}} email,
    decimal{> 0} accountBalance
)
{
    // Implementation
}

That's a pretty neat deal to me.

Reading

Note these are all academic; there isn't a great into article I've found. Future post idea?

Watching

Book Club 2&3/2025: Slack: Communication, Organization, Teams

Mon, 24 Mar 2025 00:00:00 Z

So a bit ago I got shuffled around onto a new position, I am now the leader of the team responsible for PLPs, PDPs, and such item-related things at crateandbarrel.com. New team = new ideas, and being a remote worker throughout my whole career ideas about communication are particularly at the fore. My firm uses Slack to communicate, and I think Teams and other tools present a similar experience.

I've been reading a lot of disparate pieces about the functioning of teams, so my book club (for two months!) is a bit of a disjointed mess, but I think communication is the concept that ties it all together.

There's a lot of different ways to do communication, particularly over messaging platforms, and different situations call for different sorts of communication. How to create, manage, and structure various Slack channels seems like a critical skill; identifying communication patterns, shortcomings, and requirements can make or break a project. Here's a couple articles to do with Slack "architecture":

I've implemented Friday Slack demos to mixed success. I think colleagues enjoyed it but participation was spotty. When year-end reviews came around, those who participated more got more out of it. That practice is probably better for some teams and more mixed for others; definitely worth a try!

No doubt different practices work for different situations, teams, individuals. I've got a number of ideas having worked remotely for so long, but I haven't published them because my remote working practices seem so ... personal? Inapplicable to others. I might do yet though; the following have given me some interesting thoughts even though I don't think they apply universally:

And of course there's different styles of communication depending on the scenario or situation -

Some writing on general team practices -

And some writing on learning -

Last but not least, here are a few articles I've published generally on this mater:

That's really all I've got here; usually I use this book club as a way of developing, rounding out, and reinforcing ideas and practices that I've developed over my career, but I took a step back here to absorb. This is altogether a lighter, less set-in-stone sort of topic anywho. For those readers who live in the northern midwest US, I hope you're all enjoying seeing all the birds migrate back; I spotted my first egret of the year yesterday!

Book Club 3/2024: Simplicity

Mon, 25 Mar 2024 00:00:00 Z

Everything is too complicated. Not just in software, everything! This isn't a novel concept, Teddy Roosevelt would have us believe that anything worth doing involves difficulty. I think this is a widely-adopted notion; simplicity in software seems to be a whole genre of software blogs. That makes my job here of sharing links quite easy though!

To spare all of the usual writing, simplicity is software is valuable because it makes all the things better. Simple code is easier to read! It's easier to maintain! Your code will perform 10x faster if it's simple! Simple code is more extensible, testable, and debuggable! Then we might consider the software itself - the UI, UX, and all the user-considerations. Simple software is easier to use! Users are less likely to reach an unintended state with simple software! You'll have fewer support calls and lower burden of feature growth with simple software!

These are all true, but a trap a lot of us fall into is neglecting complexity. Sure, our software could be extremely simple, but it has to solve real user requirements. This is the main source of complexity. Sometimes a very simple piece of software can solve most requirements of most users mostly well. Sometimes a more complicated piece of software can solve those requirements very well. We can't let ourselves want software more simple than the use case requires. I could just as easily rattle off plenty of pro-complexity platitudes: Complexity is necessary for real-world scenarios! Complexity is required to adequately address security and performance concerns! Architectures that maximize extension and innovation are necessarily complex!

Both simplicity and complexity are necessary. The difference here is that the necessity of complexity always comes with that asterisk - the kind and level of complexity that is required is the minimal amount required to satisfy the requirements you have for your system. Any complexity too far beyond that is detrimental; all of the listed benefits of simplicity are, in fact, true.

Understanding requirements is essential then. User requirements only capture so much. What are the environmental requirements? What quality attributes does the code need to support? These can introduce necessary complexity but often go overlooked. A complete understanding of the requirements of the code will allow you to develop a holistic view of the complexity that you need to introduce. This is a sort of focused complexity - this is not just complexity that is necessary but that is also understood; you know why it's there. Unfocused complexity is surely just as bad as excessive complexity. No doubt we all have codebases that have the wrong level of complexity, but ask also whether they have the wrong kind of complexity. Maybe indeed the level of complexity is right but it's unfocused.

Watching

And surely Jonathan Blow has something to say on the topic.

Reading

These two are kind of related:

From Me

I've had a couple articles recently that more or less relate to "simplicity":

Book Club 4/2024: I Don't Like ORMs

Wed, 24 Apr 2024 00:00:00 Z

Indeed, Object-Relational Mappers (ORMs) are a negative thing to me. For the uninitiated, these are tools which automagically provide ways of mapping data between the object (class) representations in our application software and the corresponding relational representations in databases (specifically, of course, relational databases). They present an alternative to writing SQL and deserializing the data manually. Popular ORMs include Entity Framework for .NET or Prisma for Node/TS.

ORMs typically have two main components: some way to map between properties in a class and table/columns in the database, and some way to not have to write SQL in your application. In the case of Entity Framework and similar libraries for .NET, this takes the form of attributes on class properties for the former, and using LINQ to write the queries which then "intelligently" get translated to SQL at runtime and sent down the wire. It provides type safety and absolves the engineer from having to write SQL in their applications. Many take this to be a good thing.

The problem which I have with ORMs is twofold, but it mainly boils down to the second. First, I perceive ORMs as polluting my objects - attributes and configuration code are annoying, sure, but I have concerns with vendor lock-in and more greatly that the query abstractions tend to hide what they're doing. Taking Entity Framework specifically, you need to be quite good at knowing what it's doing in the LINQ there to avoid any number of gotchas, which increases the incidence rate of footguns. This leads into my second and important gripe: ORMs offer less performance. Even if you're really good at them you're never going to get an overall performance benefit.

The performance problem does frequently come from a misunderstanding of how the SQL abstraction of a particular ORM maps to SQL, yes, but it's also just less performant to insert an abstraction over something. A lot will push back on me at this point that performance isn't everything - readability and maintainability are important, and my straw man here will feel that the type-safe abstraction of the ORM is more of both. I think this is misguided. Unless you know SQL quite well and you know how your ORM generates SQL from its abstraction you're going to be setting up some potential very volatile code, so you need to read the code you wrote in this abstraction through this lens. Which seems easier then - writing and reading this abstracted code while considering the SQL translation, or just writing and reading SQL?

One can claim "skill issue" here all they want, but the point is that I cannot see how using the ORM reduces any burden on me as I'm writing code. It only adds to the considerations I have to make. To be clear, I do believe you need to learn and know SQL well in order to be able to effectively use a relational database ever, whether you're using an ORM or not. For an engineer to desire to avoid learning and understanding SQL is lazy. Perhaps someone has indeed invested the time in learning it well and decided that they really would rather have the abstraction over it; this is an informed opinion which I can respect, and I'm sure such an engineer would be grateful for what they did learn from SQL (my favorite straw men are the respectable sort). I don't know that you can ultimately use any tool to make up for not understanding the fundamental working of databases and their query engines - just as in sports, it's not the tools that make one successful.

So all that considered, my recommendation is to just use SQL. That, to me, has the lowest burden across the board. Since I need to learn SQL I know it, when I write or read SQL I don't have to jump through translations, and when it comes to mapping or deserializing the response from the database I have all the control I need. But wait! Doesn't that lead to so much more boilerplate code? No, and I'm disinclined to elaborate for I've never seen a proof of this and I believe the burden of proof is on the individual making this assertion. Please leave your proof in the comments! To anticipate a response, the boilerplate which I use over ADO.NET and copy/paste/modify for any particular project clocks in at under 100 lines, I and everyone else in the codebase know exactly what it does, and it outperforms your ORM.

As a final note, there are a category of ORMs which assist you in writing actual SQL then handling just the mapping; they don't abstract over the SQL, which is my main objection. In .NET the most popular one by far is Dapper. Considering the low cost of entry to rolling my own mapping, I prefer and recommend that approach. However, there are projects or environments where Dapper can be useful, and I would recommend that everyone look to this library for these cases, or as an inspiration for what you can achieve on your own.

Watching

Reading

I know that dev.to and Medium articles are a bit of a mixed bag, but these are well-considered with good, practical examples:

Book Club 4&5/2025: Incidents and Resiliency

Sat, 31 May 2025 00:00:00 Z

In March I moved to a different role at my current firm, and the thing that's changed the most for me is incidents. In the past quarter I have had to respond to more incidents than I had in the two and a half years prior. Comes with the territory sometimes, I suppose, but that puts the topic into my mind!

I'm very happy to have been subscribed to Lorin Hochstein's blog Surfing Complexity, which has given me a lot of links to share with colleagues. Lorin's approach to rigidity in complex systems is to center the interactions of the components of the system as the potential, and sometimes actualized, causes of failure. This is opposed to the traditional model of there being a single "cause" or "actor" within the system from which the failure arose. Stated plainly this seems like an obvious thing to think, but turning that thinking into something operational is a different thing altogether.

Following up on an incident, how do we update our system to exclude failure-causing interactions? How do we do this in a way that avoids thinking about "root causes"? Most fundamentally, how do we even identify all the interactions in the first place? I figure that the complexity of a system will directly correlate with the complexity of incident-causing interactions. I also figure that complexity of any sort directly correlates with lack of understanding.

So, as the complexity of the system increases, the complexities of the causes of failure increase, the ignorance of the workings of the system increases, and the inabilities to comprehend the causes of failures increase. In short, the probability that the holistic understanding of the causes of an incident cannot be had increases with time. The greater the possibility that I can't understand the full picture as to why an incident occurred, the less useful it is for me to engage in root cause analysis. In such a system, the root cause analysis will at best not fix the issue, but at worst will introduce new issues.

What's to be done then? Resiliency, of course! Also easier said than done. The implication is that failure can never be ameliorated, but can only be managed. A system that better manages errors is more resilient and will function properly a greater percentage of the time. That might be an unfulfilling answer, but I'll ask you whether that maps better onto your experiences with complex systems?

When we follow up from incidents, we can identify individual failures in components of our system, and we can probably identify several points of interaction failure. The question to answer is how we can update the components to succeed in the face of the errors they encountered, not how we can fix the problem. Assume you can't fix the problem, because you can't comprehend the problem. You can only control the limited view you have over the system, and you can only make that slice of the system behave better in its error-laden environment.

I was convinced to write about this topic for the book club this month by this recent post from Lorin : Not causal chains, but interactions and adaptations.

Links from folks other than Lorin, in alphabetical order of their names:

Book Club 5/2024: SOLID

Wed, 29 May 2024 00:00:00 Z

Happy Memorial Day to those in the US, I haven't had time to post a lot this month but hopefully this one is controversial enough to make up for it! SOLID, as I imagine you know (having subscribed to this newsletter), is a set of principles for developing software introduced a little over 20 years ago by - look at that - Uncle Bob. It's five guidelines which seek to constrain how we write software in such a way as to make it more likely that "clean", maintainable code is written.

I have some problems with each of these principles specifically, but in the general sense I can say a couple things. It's my (admittedly evidentially unsubstantiated) observation that we as an industry tend to latch on to dogmas very quickly. Various buzzwords, libraries, architectures, and patterns all have their five minutes of fame, so to speak, finding themselves the focus of momentary and ill-considered obsession. Some of these come and go very quickly (As an example, I don't know what's popular on NPM these days but ... probably that), and some of them last a bit longer (microservices). Some of them last far too long, and SOLID is one of those.

By hyper-focusing on fad concepts, we are doing ourselves and an industry a disservice in several ways. Obviously, saddling future engineers with the bad ideas we didn't think through now is bad, but the more interesting aspect to me is that by focusing our individual attentions on these fads we're stealing our own time away from ourselves to develop thorough and rigorous skills in actually foundational skills. Taken to the extreme, an obsession with silver bullets blinds some among us from the understanding that there's nothing but hard work and careful consideration at the center of this industry.

So I tend to avoid the fads as they come and go, and looking around it seems to me like SOLID is on the way out, but I'd like to contribute a drop in the bucket of the anti-SOLID sentiment. Really I'd like to be writing in favor of diligence and developing an understanding of which tools fit which jobs, but SOLID serves as a great contrary example. I'll briefly touch on each of the five principles -

Single Responsibility Principle - this causes more arguments and mangled code than it helps anything. At the surface it is absolutely correct to say that any given module should have a single purpose, but the trouble comes not just in defining what a "single responsibility" is but also defining how to define what a "single responsibility" is. Great, now I need a whole ontology of orders for definitions to resolve a PR! Should "single responsibility" mean that each iota of code is doing the bare minimum of things? Should it mean what Uncle Bob suggests, that it only has one "business reason" to change? Good luck defining that one too. And what is a module anyway - do we apply this to methods, classes, and namespaces? Oh, not namespaces - why? There's another fight.

Open-Closed Principle - This gets into the difficulties with inheritance in general, that being the complexity incurred by using it. This principle doesn't exacerbate the troubles with inheritance, but in my opinion it doesn't really do anything to constrain inheritance problems. Superficially sure - it says I shouldn't modify the base class! Okay, fair enough. In practice I don't really see that ever being an actual issue that comes up, rather that any use of inheritance adds complexity. There are limited cases that inheritance is useful when developing boilerplate logic, but it's never necessary to use. I think this principle just disappears by properly limiting or eliminating your use of inheritance.

Liskov Substitution Principle - Again runs into the problems with inheritance, but this principle also applies to interfaces, which I like when used appropriately. I think that overall the use-case for interfaces is less than they tend to be used for - certainly so in the C# world - but when they're used I actually think the Liskov principle is a good one to keep in mind, generally. That said, when's the last time you saw some code checked in that might have actually violated this principle? A lot of the times this might come in from one interface implementation violating the not-documented contract for a particular method (i.e. "return null in this case to signify such and such"). While Liskov could be said to capture this, I don't think it does so explicitly enough, and so a better solution might be to adopt some principle which ameliorates the contract violation concern, specifically.

Interface Segregation Principle - It's fair enough that a consumer of a module shouldn't be required to consider a non-relevant interface. This principle is good in that sense, but it runs afoul of being too nonspecific - we start getting arguments again. "This interface should really be broken into three interfaces" when the interface declared two methods is a frustrating conversation I've had in some form twice. Being clear, this isn't nearly as much a problem as SRP is, and this principle has its utility. My issue with it is similar to Liskov, I think there are other practices out there which more appropriately constrain our manner of development with respect to interfaces. That would be a good blog for me to blog...

Dependency Inversion Principle - the way this typically gets implemented, at least common to C# codebases, really grinds my gears. I wonder if this one ends up being misunderstood a lot, but I can't really say, it does seem to imply that every class should have an interface. This is a ridiculous notion, as I've discussed previously and in my article I linked above for Liskov. Utility classes with few methods and only one implementation definitely don't need an interface, and anything with only one implementation doesn't. An often-used objection here is that I might want an interface if I ever need to refactor by creating a parallel implementation, but the response is obvious: create the interface at that point in time. Maybe this was more difficult back in the day, but with modern development tools it's not difficult to do that.

Upon reflection, it's quite funny how SOLID is one bad argument-generating principle followed by four milquetoast suggestions regarding interfaces. It seems like SOLID might do well to be replaced by a set of principles on the appropriate handling of abstractions and interfaces, but to my previous point I think it's a disservice to focus on these catchy - for lack of a better word - memes. Properly, it would be better for the industry to discuss principles in an if-this-then-that format: given situation X, principles [A1, A2, ..., An] are beneficial because [R1, ... Rn]. I hope you all concur with me that the use of quasi-mathematical notation immediately makes my point more rigorous.

Reading and Watching

David Bryant Copeland has a whole series on SOLID: SOLID Is Not Solid
Inheritance Is Evil. Stop Using It. - Nicolò Pignatelli
Is SOLID Still Relevant in Modern Software Architecture? - Vasco Veloso
Solid Programming - No Thanks - The Primeagen - this just happened to come out this month
Where Does Bad Code Come From - Casey Muratori
Ranking the SOLID Principles - Nick Chapsas

Book Club 6&7/2024: Postgres

Mon, 29 Jul 2024 00:00:00 Z

I hope you all in the northern hemisphere are enjoying summer; I've been trying to take some extra time here and there to enjoy things before Minnesota inevitably plunges back into the depths of winter, so I've been writing a bit less (and combining two months' book clubs) but I think that's okay, all things considered!

A few months ago I wrote a short piece recommending that we just use PostgreSQL, and I've spent several months becoming more entrenched in this position. This might be me jumping on the bandwagon - Postgres has become increasingly popular in recent years - but its genuine, proven capabilities leave me wondering what else could be the best de facto choice? You might argue that a default database option isn't needed, though I see a lot of value in it; at least as a standards bearer, like Git is anymore for version control, it provides a baseline for the whole industry.

I've become very serious about insisting that any new projects must use Postgres unless there's a clear argument in favor of another database, and that's a difficult hill to climb out of. The vast majority of professional applications are not just adequately served by a Postgres database, rather Postgres is typically one of the better options. The kicker though is that Postgres usually allows you to do a lot more than just data storage, keeping the solution architecture very minimal for a whole host of apps.

I didn't touch on the ancillary capabilities that Postgres has in my original article, but from the articles below we can collate a heck of a lot. It can be your cache, message queue, job queue, or cron daemon. It can act as a document database, graph database, geospatial database, or search database. It supports any data model and easily handles complicated data patterns like event sourcing. Logging, metrics, and analytics can all live within Postgres; Timescale allows you to make Postgres your data warehouse.

Most importantly, Postgres is used, and it's used a lot. All of these solutions are first-class and have been in use in major production systems for quite a while. There's great documentation all over for Postgres and all of these capabilities.

The other month I used Redis for a project, as SignalR doesn't have the ability to use PostgreSQL as a backplane, but I'm going to see about writing that. I've been exploring using Postgres for most of the uses above, and I'm feeling quite settled that Postgres is the standard bearer.

Next time you get the inkling to use something other than Postgres on a professional project, ask why not Postgres? Insist on a proper, comprehensive answer that accounts for the flexibility, security, simplicity, documentation, and resilience that you get with Postgres. Yes, it's a tall hill to climb; yes, our professional work requires it.

And some videos:

And now for something completely different:

DHH discusses SQLite (and Stoicism) - Aaron Francis with DHH

Book Club 6&7/2025: OOP

Wed, 30 Jul 2025 00:00:00 Z

Happy summer! It's been a couple months since I checked in on my newsletter; I've been busy enjoying sweltering heat, lots of rain, and wildfire smoke. In truth, I've been focusing more on a new role at work than my blog, but I promise I have found plenty of gripes there that I will resume my regular posting cadence here shortly! These last two months though, I've been thinking about object-orientation, so today you get some ramblings at a high level.

As ideas ebb and flow in popularity in the software world, OOP is perhaps a bit on the ebb nowadays. Certainly it's not difficult to understand a contemporary distaste with OOP. At the risk of gross oversimplification, Java and C# engineers (and others I'm sure) have over the last several decades developed a style of programming which emphasizes overengineering and overarchitecting, developing massive codebases of passthrough layers, pointless interfaces, useless abstractions over sometimes useful abstractions; no doubt all reflections of the sometimes bizarre bureaucratic structures of the larger firms that typically employ these languages. This is the monster a lot of our colleagues envision when hearing "object-oriented programming," and they're not entirely off-base.

While this is what OOP means these days - at least colloquially - any rigorous definition of OOP isn't going to make it obvious that such code necessarily results from the paradigm. Indeed, surely in a decade there will be a revival of "classic OOP" or some nonsense as the popularity pendulum overcorrects far to the other side. These "classic OOP"ers will insist that OOP is solely about objects, or encapsulation+polmorphism, or message passing. I feel confident in my prediction here as this sort of defense has already begun in some circles, and again it's not entirely off base. These are all suitable definitions.

On that last one, message passing: if you spend a sufficient time reading about OOP you'll uncover the group of colleagues who insist that Smalltalk is the only real OOP language. Avoiding the lower-level debate about what it means for a word to mean some thing (in this case, "OOP" to definitionally include "Java"), such a claim doesn't quite pass the sniff test to anybody with a decent understanding of the history of programming languages; the OOP ideas all came through in various stages and together form a consistent thread of ideas for approaching software architecture and engineering. Nonetheless, I'd be hard-pressed to defend the idea that the present understanding of OOP is better than the more limited Smalltalk-esque understanding; the more focused principle (or, maybe, more principled focus) of the messaging notion of OOP might well provide the necessary constraints for software to be developed better. Perhaps that's why so much new software is being developed in Smalltalk these days! Well...

Perhaps it's unfair to conclude from Smalltalk's failure that its particular paradigm is also destined to failure, but I wouldn't hold my breath. So too for other paradigms that claim some sort of purity; there are no silver bullets. Today's OOP has developed as a hodgepodge of many different ideas, forms, and learnings; this makes it as undefinable as unfocused, or maybe un-specific. There's sometimes many valid approaches to the same problem within the present OOP canon. Yet, it would be a mistake to conclude that this is a bug instead of a feature, for the very reason that OOP has developed in this way is precisely because it has been forced to solve just about every programming task under the sun. Recognizing that different problems will require (sometimes very) different paradigms to optimally solve, we must conclude that our languages and architectures must have the flexibility to accommodate many different approaches at once, thus making the hodgepodgeness of OOP a practical necessity.

So the contemporary, undefined, colloquially-understood OOP is not such a bad thing, from a certain point of view. What though when we come upon a situation that objects are insufficient bags for our logic, or that a more pure functional language compiles much more efficiently for some or another problem? Today's software is more varied and diverse than ever before, so our incidence rate of stubbing our toes on these situations is quite high now. Hence the unpopularity of OOP: the traditional languages within the paradigm have not been flexible enough for some of the most basic procedural or functional styles best-suited to many of these tasks. On the other hand, many traditionally-OO languages have been incorporating support for other paradigms to quite a great extent; anecdotally, much of the C# world (including myself) has moved on to quite functional-looking styles as that language has (not insignificantly) morphed to accommodate it.

The ideal future of OOP, should we all continue to agree this is the desirable path, is maybe a sort of fading into oblivion in which its constituent ideas are considered coequal with those from the other paradigms, and our languages and runtimes support a fluid combination of whichever structures we might discover are the most efficient solutions to our problems. There's certainly plenty of momentum in this direction, but making our tooling more permissive won't stave off any potential future hype for some "pure" form of OOP or functional or what-have-you programming; nothing can replace solid principles. OOP, by any definition, isn't good or bad, nor is it any one specific thing.

Watch

Book Club 8/2024: Labor

Mon, 02 Sep 2024 00:00:00 Z

Happy Labor Day to those in the US! This is the August Book Club but I'm sending it out a couple days late to coincide with Labor Day. I don't have a central thesis to focus on, but I want to take a look into how our industry is doing. This involves matters of labor, hence the connection.

The industry feels like it's in a very interesting state since COVID, and there's some data to support this. The overarching sentiment I get - what I think the bulk of the focus is on - is that the industry has contracted slightly after the pandemic. I think something of the sort would be inevitable, as the spending patterns that fueled the huge intake in personnel during the pandemic wouldn't have been sustainable. Another possible inevitability is the fall in demand in lower-skilled engineers - particularly with the pool of jobs contracting it seems obvious that corporations would want "the most bang for their buck" with the engineers they retain and later hire on.

This will reverse again at some point. It was true that before COVID we needed to add a huge number of software engineering jobs; the demand was very high. That demand has not disappeared. Rather, it's more likely that the spending patterns of the larger corporations have changed that we see this result. That said, I have a worry here. One way that so many engineers were hired during the pandemic was the onboarding of a record number of junior engineers, and it's this group which is disproportionately laid off. Are these folks landing on their feet within the industry or are they abandoning it? I have no data here.

Something else which has contracted, much to my disappointment, is conferences. I know plenty of us dislike conferences; I am not one. I have always seen a huge value in independent conferences, particularly local ones. Earlier this year I lamented the end of the Twin Cities Code Camp, and one year ago I attended the final Strange Loop. There are (some) others to fill the voids left by the closure of these and other conferences, but the landscape is lesser than it was a decade ago.

I'm a strong advocate for working from home and allowing flexible hours. I've worked from home for my entire career (well before the pandemic) and I've resolved that I always will. There's no doubt that working from home does reduce the amount of and opportunities for contact with others, so I also feel strongly that a healthy home work routine involves taking advantage of frequent opportunities like conferences and meetups.

To continue a somewhat bleak outlook, the recent StackOverflow developer survey reports that only 20% of us are "happy" at work. I do think this is the first time they measured this, so I'm skeptical to read into it too much, but my off-the-cuff reaction would be that this number is short of where I would hope we would all want it to be. I don't think that a 20% happy rate today portends a happy future in the industry.

This is all not in service of trying to paint a narrative that the industry is in a bad way. It certainly is in some respects, but that's not new and it's not unique to our industry. There are always better days ahead if we want there to be, and the progress that the industry has made to this point is incredible. Calling out the negative aspects and working to improve those is the best way to move forward.

Here's a dump of a bunch of Primeagen videos and some others:

Book Club 8&9/2025: Async

Sat, 27 Sep 2025 00:00:00 Z

These last two months, the interesting thing that I'm researching is asynchronous communication patterns. Last year I wrote a whole series on the Fallacies of Distributed Computing exploring some common dos and don'ts for systems that rely on network communication. Asynchronous communication (which, I'm just going to type "async" here on out) is a common solution to reach for as the number of nodes in the distributed network increases. It solves the problem of huge latency in requests that need to chain through several services, though at the expense of system complexity and comprehension.

In an ideal world, systems would be quite "flat", with user-level requests being able to be serviced in only a single network call to the server. Plenty of systems can't accommodate this for one reason or another, a backend might need to consult other services to fulfill a request; here though, plenty of systems can be designed so that those bottom-level services don't need to make blocking calls to each other to fulfill their own requirements. Commonly, there might be a backend-for-fronend that owns all of the synchronous calls, making it easier to manage temporal dependencies by isolating them all to a particular area.

Suppose the lower-level services do need to exchange data? Alternately, suppose we end up with too many temporal dependencies, resulting in long-running calls even though we've made them architecturally understandable within a single layer? That's where we should be inclined to reach for async. I'm supposing here that we're imagining a relatively well-designed system; that the call patterns are well-considered to fulfill actual business needs.

How does async work? At the top level, when transformations happen to any data or state in the system, the service doing the changing emits an event through a shared bus, which is then read by systems that care about that transformation. Those systems will store their own representations of the data they care about, and are free to service requests on the "hot path" without having to introduce blocking calls. This has several implications: data is duplicated in many points throughout the system, the data is eventually consistent, the code for each service becomes more complex, considerations need to be made to prevent data drift, and plenty more there.

Given the set of problems solved by async and the implications of adopting it, there are many patterns and ideas around them. I haven't been terribly organized or disciplined in this research; this is a grab-bag of things that have caught my eye these last two months:

Kind of related:

Watching:

This has all taken me down a rabbit hole about AsyncAPI, but that may be for another time...

Book Club 9/2023: Papers I Love

Thu, 28 Sep 2023 00:00:00 Z

Author's Note: this first "Book Club" post is not included in the newsletter; I hadn't yet set up the newsletter.

Last week I attended the final Strange Loop Conference. This conference has been very influential on my career and my academic interests. In fact, this conference began a short time before I started making money in software. Having been able to watch the talks on the YouTubes over the years, I credit it with having a significant impact on how I've approached my career as well as my studies in college, where I got bachelors' in both philosophy and computer science.

However, I had never been to a Strange Loop before! It's bitersweet then that I was able to attend the final one. Perhaps unurprisingly, I attended several presentations sponsored by (or should I say "presented by"? unsure) Papers We Love. In that spirit, I'm going to share five papers here that cut across my philosophical and computer science inclinations that I have very much enjoyed over the years:

Book Club 9/2024: Blogroll

Sat, 28 Sep 2024 00:00:00 Z

As I posted a couple weeks ago, this month is the anniversary of restarting my blog! As though trapped in a Norm MacDonald interview, I'm frequently asked by my colleagues where I get my ideas from. Thus, for today's book club I'm just going to share a list of blogs which I tune into frequently. Some of them are folks I've bumped into at a conference, others are folks who just came across my radar at some point. Some blogs haven't gotten an update in a bit, some are quite active. All of them are somehow related to software engineering.

If you have other blogs that you like and want me to check out (even your own), don’t ever hesitate to connect with me , or leave a comment right here 👇🏻

Building a Documentation Habit

Wed, 19 Mar 2025 00:00:00 Z

Two weeks ago I started working with a new team at my company, and the first (real) meeting I attended with the team was a manager's attempt to start increasing the documentation the team kept. This being the manager's goal (and, being clear, I'm talking about a good manager with a good goal), this is my goal to some extent. How do we take a longstanding team and build a habit of writing documentation?

Sometimes it's not really that necessary to keep up large amounts of documentation. Like code, documentation needs to be kept up-to-date, but what's more dangerous than code is that documentation can lie to us if it becomes too stale; the code always tells me exactly how it's running. The most important documentation, then, is to give a history of why certain decisions were made, to explain why the code looks the way it does at any point. The best time to document why being when the decision happens, a longstanding team just starting to compile documentation is never going to recreate this.

I'm not claiming to be some documentation guru, but we should all understand it as one of the important outputs of our professional work, so it makes sense to instill a habit for writing documenation on any team. Building any habit on a team takes two things:

Constantly doing; and
Constantly reminding

Don't think you can call a meeting, say "let's all document now," and expect any change. With the team having decided it wants to build a habit, someone gets appointed the habit czar and keeps an eagle eye on opportunities to exercise it. Constant reminders by the habit czar, constant doing by the whole team. That is a bit too neat and tidy though, isn't it? If the habit is too burdensome it won't be adopted, if nobody likes the czar the team will probably start doing the opposite. Habits are people processes, and it takes some compassion and empathy on the team to be able to achieve.

Back to documentation, there's a couple things to insist on:

Foundational Points

The ultimate goal is simplicity at all levels so adoption becomes effortless. Any decisions along the way should reinforce these two:

Reduce the burden to document

Don't worry about where or how documentation is happening. The team should pick one repository for documentation to go, and that's enough decisions to start. Nobody should feel restricted about where particular bits of documentation are put, how they're formatted, or the like. If you're starting from zero, then any documentation is a step up.

Constantly refactor

Revisit documentation you've made, visit (and revisit) documents made by colleagues. Move files around, create or delete folders, move chunks from one file to another; any opportunity to add necessary context, remove cruft, and keep the whole structure tidy is good. Remember the boyscout rule: always leave the documentation campsite cleaner than you found it.

Practical Points

A good search feature is necessary

Because building the habit is our primary focus, organization takes a bit of a backseat. Yes, we refactor to improve the organization of the documentation, but that ends up moving the cheese a bit. Those are good things because they are the tools to reduce the barrier to adopting the habit, and that is the goal! Do not lose focus!

But how do we navigate the documentation? Shouldn't we impose a directory hierarchy at least to make it simpler? We could say that dev docs go in this folder, and architectural docs go in this other folder... NO! Grug reach for club! You're trying to optimize prematurely and that will hurt everyone.

Instead, rely on a great search tool. I really hate Atlassian products, but they have produced one good thing: the search bar in Confluence. I've heard from colleagues that the search in Notion is also quite good. Rely on that for navigation! A singular, comprehensible organization pattern will emerge over time with colleagues engaging with each other over refactoring, let that develop naturally. Even once such a pattern is established, you're never going to know perfectly where everything is. Search is friend!

Document at all the levels

Documentation that only focuses on technical details, only on 10000-foot details, or only on why details is incomplete. Maybe you need or want incomplete documentation; that's perfectly fine! If there are techincal details that have been around a while - say, tribal knowledge like "don't alter this sproc otherwise Bob gets really angry" - that's good knowledge to have. Equally, basic architectural diagrams of what systems depend on which other systems is really great to have. There's so many points of good-to-have knowledge that it's impossible to enumerate here; I would say I'm barely scraping the iceberg but this is more like looking at the iceberg!

Remember that if the documentation is bunk it can be deleted later; because you're constinuously refactoring any documentation will be discussed by the team and can be moved/removed. If you're unsure about the utility, put a note at the top: "Ian is unsure this is long-term useful but put it here because it looked really strange as he was looking at some issue."

Each team having different requirements in different environments, you won't know what bits of information are really the most valueable long-term but by practice. Onboarding new engineers requires lower-level documentation, while being able to communicate the cause of an issue to a VP requires higher-level documentation. Experienced practitioners will surely have developed a sixth sense about what kinds of documentation are more valuable in different sorts of contexts; this guide is for the inexperienced.

Get outside feedback

Which bits are useful, which aren't? Which bits aren't going to be able to be maintained, which bits are concrete? What document structures make the information clearest, which are overloaded or burdensome or the like? Any questions about the quality of the documentation or the quality of the experience of reading the documentation can be best helped by a fresh pair of eyes.

I wrote a while about using guerilla testing for DevEx, and you can do the same here for ... DocEx? Surely we can call it something better than that...

Documentation for its own sake isn't terribly useful, documentation should serve a function. When we throw up documentation that might be useful for such and such reason, we'll need some way to later understand what is actually useful; often interactions with the world outside the team are the best determiner of that. Did this documentation help resolve an issue raised by another team? Did Bob object to being called "angry" in the other doc? (And just to note, best practice is to not blast other colleagues in docs...)

CFWeaver Version 1.0.0 Published

Sat, 12 Apr 2025 00:00:00 Z

I've just published verion 1.0.0 of [CFWeaver, my test generation project]! This tool has helped me on a few complicated projects at work, sorting out how systems behave. In fact, while I developed it to help generate integration test scenarios, I've used it more to help me map the functionality of some legacy bits of code.

You can read my previous writing about CFWeaver or more generally about this way of thinking that the tool supports. The latest version is available for download on GitHub now!

Variables

In this release I added the ability to define "variables" along with each step to make it easier to track certain properties of the system through to the table of results. I had previously decided on a syntax for conditions which had the condition specified after a ?. This turned out to be conveniently similar to the URL query string syntax, so adding variable definitions was as simple as allowing & after this to append variables to the result.

* Step1: Result ? some condition & myVar = value

Multi-Line Results

This was a big one since it was getting tedious to append multiple results on the same line with |. Take:

* AuthenticateOnExternalSystem: Success ? DidError = no | Timeout = 500 ? auth server timed out & VerifyLog = auth server timeout & DidError = yes | NotAuthenticated = 403 ? not authenticated | OtherError ? non-timeout error from auth server & VerifyLog = auth server unknown error & DidError = yes

That's not great to read. Much better is to have separate lines, and I had supposed early on that I might achieve that with nested lists. However, I really don't like whitespace dependence, even for a simple modeling language that's coopted markdown. Instead, I opted to use - as list bullets to distinguish a result list lien from a step line, which isn't the greatest either since it requires specific knowledge, but it strikes me as the least-bad option.

* AuthenticateOnExternalSystem
    - Success ? DidError = no
    - Timeout = 500 ? auth server timed out & VerifyLog = auth server timeout & DidError = yes
    - NotAuthenticated = 403 ? not authenticated
    - OtherError ? non-timeout error from auth server & VerifyLog = auth server unknown error & DidError = yes

Tests

The last thing I've done in preparation for this release is I've added an integration testing apparatus. I don't have everything tested since I'm still considering the best way to test HTML and markdown output, but I do have all the errror conditions tested! This was an interesting process that I haven't done for a command line app yet.

Crucially, I had to set up the application to inject the console and file I/O functionality; you can see that in this commit. This allows the tests to inject fakes to capture these operations and allows me to run integration tests without needing to build the application and run it in Docker or some other squirrely setup for testing.

Clean Meetings: A Software Engineer's Guide

Sun, 03 Dec 2023 00:00:00 Z

If being in meetings all day isn't bad enough, spending more time thinking about them seems horrible. However, meetings are going to continue to be inflicted upon us, and there will come a time (perhaps more than a few) that we'll need to inflict meetings upon our colleagues in turn. Meetings should be short, concise, and mutually beneficial to everyone involved, and in order to ensure their utility it's necessary to be mindful and considerate when facilitating or participating in a meeting.

I've had to hold a lot of meetings in my time. I've spent many years in the roles of engineer, team leader, and architect, and I have to conduct meetings outside of work besides. If you want to go full-on meeting nerd I recommend that you pick up a copy of Robert's Rules, but I wanted to distill my experience and thinking on the matter into an easy-to-follow checklist for the 99.99% of us that don't want to have to spend more time than necessary on one of the admittedly more onerous parts of our profession. So, here's a simple guide on making sure you're getting the most out of your meetings.

Before we get started though, I want to give the single most important advice regarding meetings. If you read no further, take this away: Meetings are an unfortunate tool of last resort. Do you actually need to have this meeting? Would an email chain suffice? How about a Slack message, thread, or channel? Uncle Bob says (paraphrasing) that we should strive to write as few comments in our code as we can, and when we do we should acknowledge it as a failure. Take a similar approach to meetings: While maybe not a failure, we should rarely be having meetings and should, let's say, acknowledge that there might be a potential for a better way of working here. (Is that appropriately diplomatic?)

Note that meetings can take any level of formality between ad-hoc meetings between junior engineers and a senior-level presentation to the CEO. Some meetings require you to write some or all of these down, such as formulating and distributing an agenda, but some do not. Irrespective the level of formality and preparation, it's good to keep these points at least in mind as you conduct meetings at different levels.

Meeting Structure

The most often overlooked aspect to keeping meetings clean is the meeting structure. Meetings have a purpose and an outcome that should be specified in an agenda (whether it is explicitly written or not) that is understood and agreed to by all participants. Some participants play different roles in the meeting, and they will take different actions (what Robert would call "motions" but I'm not going to use that language lest I might confuse those of you who use Vim BTW) during the meeting. This might seem low-level, but it's often the details that can keep a meeting productive.

Purpose and Outcome

A meeting is scheduled with a specific set of participants agreed on a purpose and outcome: Why is it necessary to gather this group of people and what do they need to achieve? You might need a brainstorming session, you might need to reach a decision or consensus, or you might need to share information. Whatever the case, be clear about each of these.

✔️ Do define a purpose

✔️ Do specify an outcome

✔️ Do ensure all participants understand the purpose and outcome

❌ Do not neglect this step! Don't neglect the other steps but especially not this one!

❌ Do not assume all participants understand or agree with the purpose and outcome - reach out and be sure

Agenda

Simply put, an agenda is a set of topics to discuss at the meeting. This can be as simple as a single bullet point (i.e. * Brainstorm a name for the new microservice) or a detailed breakdown of gaps in technical knowledge.

Side note: if your agenda is "Brainstorm a name for the new microservice" that's just a Slack thread. I've been in that exact meeting too many times and I'm here to tell you - your microservice will be replaced by 5 others and deleted in 2 years anyway, just call it "Craig" or something.

Some agendas are written, some aren't, but each meeting has an agenda. There are few absolute truths in the world, but I can tell you that meetings which follow the agenda are good, and meetings that don't aren't.

Another often-overlooked point is that the agenda needs to be agreed on by all participants. It is not the job of the individual calling the meeting to carve the agenda in stone, it's their responsibility to make sure the agenda fulfils the purposes intended by the participants and the outcomes desired by them. Are the intents and desires of one participant contradictory with another? Sounds like multiple meetings. Maybe multiple Slack threads, but what do I know?

✔️ Do be specific about the purpose and outcome in the agenda

✔️ Do communicate the agenda to participants ahead of time

✔️ Do ask participants to contribute to the agenda, both before and at the beginning of each meeting

❌ Do not require all potential participants to attend, especially if they feel they aren't interested in the agenda

❌ Do not write the agenda in one pass. Instead, write a first pass and give ownership to the team

Roles

Each meeting has roles. At the least, you need to acknowledge:

The facilitator leads the meeting, ensures the agenda is followed, and facilitates discussion,
The secretary records the minutes and important decisions, and
The participants engage in the discussion, provide input, and carry out assigned action items post-meeting.

✔️ Do be clear about which role is fulfilled by which participant

❌ Do not assign multiple roles to a single person in larger meetings

Before the Meeting

There are a few steps to take before holding a meeting, and they can greatly help to set the meeting up for success. The most important thing before the meeting is to remember the most important advice: Is this meeting actually necessary? Abort if not.

Prepare the Agenda

I covered all the dos and don'ts in the previous section, but this is the most important step before the meeting. Write the agenda down and distribute it as early as you can. Some meetings are regular rituals, and it still helps to write the agenda for these down and distribute them. Sometimes there is requisite technical or business information for the meeting - provide these as resources for participants. If they suggest changes, make them!

✔️ Do include knowledge perrequisite for the meeting in the agenda (links and short descriptions, please)

✔️ Do take feedback - if necessary, send a revised agenda to participants

✔️ Do include any necessary discription or goals for points needing clarification

❌ Do not write a novel for the agenda - stick to bullet points

Sometimes the objective of the meeting is reached while distributing the agenda. If it is: abort the meeting. Your objective is reached. As Sun Tzu says in The Art of War, "The wisest general is the general who never fights."

Pre-Meeting Preparation

While the facilitator has the most work to understand the agenda and work towards the objective, participants need to prepare as well. Read the agenda, suggest feedback if needed, and study the prerequisite information.

✔️ Do review the agenda beforehand

✔️ Do suggest agenda changes. If you think you might have solved the problem or resolved the outcome of the meeting, even if only minutes before, say so and abort the meeting

❌ Do not attend the meeting if you feel you don't have to. Best to ask the facilitator to drop, explaining why

During the Meeting

In a meeting, everyone should be engaged and it should be kept as short as possible. If you're not engaged, drop. I've worked at places where I was told explicitly to not do this, but I would drop anyway and never heard any complaints. Your mileage might vary. If you're facilitating a meeting, let your participants drop.

Adhrere to the Agenda

The agenda specifies the purpose and outcome of the meeting, and the topics worthy of consideration in furtherance of the objective. At the start of the meeting, the first order of business is to approve the agenda and make any last-minute changes needed by the participants. Once everyone agrees on the agenda, stick to it. Topics not on the agenda are to be tabled, either for another meeting or preferrably for a Slack thread.

✔️ Do ensure everyone agrees to and understands the agenda

✔️ Do be clear when moving from one agenda item to the next

✔️ Do cut participants or yourself off if non-agenda topics start being discussed: "Let's table that thought"

✔️ Do allow for a brief period at the end of the meeting for any additional topics if it's a wide-ranging meeting, but be eager to move individual conversations to later meetings

❌ Do not amend the agenda mid-meeting. It was agreed to, and if we find we're not focused that signifies that we can abort the meeting - it's not too late

❌ Do not "afterparty" - these are either separate conversations or separate meetings (or better yet, Slack threads)

Time Management

The meeting should have a specific amount of time. My favorite, and I am very serious when I say favorite - historical fact is that Winston Churchill would limit all meetings during the war to 20 minutes. YOUR MICROSERVICE'S RAM CONSUMPTION IS NOT MORE IMPORTANT THAN THE ALLIES WINNING THE SECOND WORLD WAR.

I don't know how accurate that fact is but I quote it more than once a week on average and in my experience nobody ever checks me on it, so either it's true or you can also use it to keep your meetings below 20 minutes. Whatever the historical accuracy, my experience has taught me that 20 minutes is enough for almost all meetings I've had to attend or conduct. Meetings that go over 20 minutes deliver exponentially less value to participants per minute of runtime. The ideal length of a meeting is the amount time it takes to compose a Slack message. Oh look at that, your meeting can be a Slack message!

✔️ Do adhere to a strict time limit. Some objectives need to be reached and the meeting extended if not, but the vast majority don't. Set the expectation with everyone that you'll hold them to a timeframe and, as if by magic, the meeting will resolve in the right amount of time

✔️ Do keep conversation flexible but insert yourself when it needs to move along

✔️ Do set approximate time limits for top-level items

❌ Do not tell participants about those time limits - just convey how long the meeting will be

❌ Don't harp on participants about time, they're doing their best to navigate the meeting; it's always OK to politely interrupt with a brief reminder about time when necessary

Take Notes

Liek the heading says, take notes. This is why I enumerated "secretary" as one of the roles in the meeting. Someone other than the facilitator should be writing down notes. "Even for daily standup meetings?" you might ask. Well, you should consider whether those are necessary, but yeah you probably should be taking notes if you're having that meeting. If you counter that nobody cares about those daily standup meetings enough to take notes, then you might consider a second time whether those meetings really are necessary!

Notes will serve as important documentation after the fact for participants about what's been discussed and agreed to. What's more is that not everyone in your company can be in every meeting, but you need to make sure that information is available to everyone in your organization. Keep minutes to allow everyone to revisit them when necessary.

✔️ Do appoint a "secretary" to take notes

✔️ Do share the notes with everyone during the meeting

✔️ Do take note of important questions, information, and decisions

❌ Do not record every word of every participant

❌ In fact, do not record the actions of specific participants unless it's absolutely necessary

❌ Do not let everyone amend the minutes during the meeting - have one person dedicated to this task (brainstorming sessions maybe aside)

Everyone Must Participate

If you have a participant who isn't participating, then you don't have a "participant" - you have a voyeur. I'm creeped out by voyeurs, and you probably are too. Excuse them.

Everyone in a meeting should be participating, either actively listening or contributing to the present conversation. If you're facilitating and notice someone isn't participating, ask their opinion. Create an atmosphere where they're welcome to share their opinion, no matter their seniority (either rank-wise or domain-wise). Participants asking clarifying questions, no matter how basic they may be, must be encouraged.

Side note on the topic of conversations: if a participant gives a hostile answer to a question, call them out on it. This is difficult but it's a win-win - the participant at the receiving end of the hostility knows you have their back, and the hostile participant (and everyone else) knows this behavior isn't acceptable.

✔️ Do encourage a diversity in opinions and voices

✔️ Do ask everyone their thoughts regularly

✔️ Do actively engage in the conduct of the meeting and encourage a positive atmosphere

❌ Do not let participants be passive in a meeting

❌ Do not overlook non-verbal cues in virtual meetings; they can be indicators of agreement, confusion, or the desire to speak

❌ Do not avoid tough conversations. About that...

Engage Difficult Conversations

Meetings are called for specific purposes - sometimes to reach decisions, sometimes to explore new ideas, or any multitude of reasons that might touch on the passions of several participants. Meetings can bring up difficult, complicated, or heated conversations or even arguments. Make no mistake - this is good, and it's a good sign that a meeting was needed if this happens. A team that is passionately engaged is infinitely more preferrable than a team that always agrees with itself. There will always be a difference in opinion, and the best thing for the team(s) involved is to engage these points head-on.

✔️ Do allow heated conversations and respectful arguments

✔️ Do ensure the topic is stuck to during these moments, and ensure everyone maintains the overall focus of the meeting

✔️ Do act to keep participants in-line when necessary

✔️ Do suggest a breather if necessary. Do reprimand participants if they cross a line

✔️ Do create a safe space for dissent and disagreement. Do emphasize the focus on ideas and not individuals

❌ Do not shut down difficult conversations or honest arguments

❌ Do not disengage if an argument comes up

After the Meeting

There's still a bit more work to do after the meeting - you're not done yet! All the more reason to respect the time of the meeting, or to try to avoid it altogether. Remember that what was discussed needs to be appropriately documented and easy to reference for participants. Remember also that not everyone in your organization was able to attend this meeting, but goodness knows when the topics discussed will impact them.

I recommend keeping your meeting minutes in a single place that's easy to reference for everyone in your organization. Products like Notion, Monday, or Confluence allow you to add tags and @ members as necessary, making them searchable too.

Finalize the Minutes

After the meeting, ask all participants to spend a minute or two to add anything to the minutes that might have been missed. You asked them to refrain from adding anything to the minutes during the meeting so they could stay focused on the conversation, now that the conversation is over they can add any extra context they need.

✔️ Do leave the minutes editable, at least for a period of time. Preferrably store them in a system with history tracking.

✔️ Do follow up and ensure everyone agrees on the final minutes

✔️ Do broadcast to your organization that your meeting is done and the minutes are available

✔️ Do set a deadline for when minutes must be finalized and shared post-meeting

❌ Do not fail to publish the minutes

❌ Do not allow a "he said, she said" fight in the minutes after the meeting

Follow-Up Actions

A lot of meetings will result in participants being assigned specific tasks. You should either follow up with them in the appropriate timeframe, or ensure their managers do, that they've completed these tasks. It's good to update the minutes at this time to reflect that this was done, and if possible link to the result.

✔️ Do ensure all participants are clear on follow-up items

✔️ Do record follow-up work in the minutes

❌ Do not assign follow up work after the meeting

Keep in Mind Always

Meeting Etiquette

✔️ Do be mindful of your role, participation, and conduct during meetings

✔️ Do respect everyone's time

✔️ Do respect everyone's individual contributions

✔️ Do adhere to virtual meeting norms, such as muting when not speaking

✔️ Do leave, not attend, or abort the meeting if you can

❌ Do not dominate a meeting, either as a participant or a facilitator

❌ Do not underestimate the impact of your physical environment in virtual meetings (e.g., background, lighting)

Regular Review

✔️ Do regularly assess the necessity and effectiveness of meetings, especially regular rituals

✔️ Do be open to feedback and make adjustments as needed

✔️ Do make adjustments or abort the meeting if and when necessary

✔️ Do periodically ask if the frequency of regular meetings is still appropriate or if adjustments are needed

❌ Do not ignore patterns of unproductive meetings. If certain meetings consistently fail to achieve their objectives, it's a sign that they need to be reevaluated

❌ Do not continue meetings just because they are a routine

Diverse Perspectives

✔️ Do value and seek a range of opinions and ideas, and encourage participation from everyone

✔️ Be mindful of inclusive participation, especially in diverse teams

❌ Do not allow the same individuals to dominate the conversation in every meeting

❌ Do not dismiss ideas without proper consideration, and do not create an environment where only certain opinions are valued over others

Conclusion

Yeah that's a lot of bullet points, but I have a lot of ideas on this topic. I feel strongly that meetings are very productive tools, but there's a lot of awkwardness in our industry around how any why we have meetings. Be considerate and hold your meetings with intention and purpose, drive towards your outcome, and your meetings will work for you and all of your colleagues.

Above all, remember: you probably don't need a meeting for it.

Windows Users: Consider a Tiling Window Manager

Mon, 12 Feb 2024 00:00:00 Z

Note of warning to the reader: the words "productivity" and "efficiency" occur in abundance in the following text.

Tiling window managers, or TWMs, are an often-hyped feature of true Linux nerddom and a permanent staple in the toolbelt of any honest around-the-clock workflow optimizer. In short, it's what the cool kids are doing (and have been doing for about 40 years). The concept isn't new - using keyboard shortcuts and clever defaults to keep windows arranged optimally on the desktop - but they're almost entirely overlooked on Windows.

This is unfortunate though, because those of us working on Windows stand to benefit just as much from the TWM as our Linux-using colleagues. Though ultimately coming down to a matter of personal preference, I would suggest that it is worthwhile for Windows users to consider a TWM. The benefits of using a TWM are essentially productivity and customizability. I've been using one on Windows for about half a year now, and the control it's given me over my workflow has had a noticeable impact on my productivity. To the point of customizability, how else could I be taken seriously when I pull out my laptop at a conference but by opening it to a thoroughly riced-out RGB setup that reports my CPU temperature in Kelvin?

Keyboard Shortcuts

There's no question (in my mind at least, but hopefully in yours too) that ALT+(number) is a much more efficient way to navigate than ALT+TAB+TAB+TAB+TAB+TAB+(SHIFT+TAB)+(stare for a bit)+(release ALT). I find myself switching between windows and contexts a lot, and being able to have a single keystroke take me to the same window context every time has done so much to help keep me focused, and that directly feeds into productivity. My IDE is always at ALT+2 no matter where I am. I can do all my communication (Slack, Discord, email, Zoom) at ALT+3. I don't really need to set anything up each time I start my machine; everything is in its place for me.

Using and learning the keyboard shortcuts isn't difficult, either. I haven't really been a keyboard shortcut person until recently, and I was able to become really proficient with the shortcuts after a couple days' use. This is a case where the investment in learning the tool is so small that it offers excellently outsized returns with the productivity gain.

Workspace Navigation

A lot of the time when we're working with windows we need to context switch twice to fully be able to reorient ourselves: once to change up our window arrangement and another to switch into the next context. I find that the workspace customization offered by TWMs eliminates the middle context switch, saving me a lot of time not just in the process of switching the windows around, but reducing the mental load I need to bear. This workspace customization extends to my multiple-monitor setup. I've already set up which tools I want on which monitors, and I have a couple of simple keystrokes to switch them between monitors when I need to switch that. This extends the predictability and efficiency of my setup, and almost entirely removed any need for me to drag windows back and forth or spend extra time trying to get my windows lining up correctly.

Windows 11 has the ability to sort windows into panels, but that is lightyears behind the capabilities offered here by TWMs: with the latter I can set up any windowing arrangement without needing to take the time to manually move the windows around each time. The real estate of each screen is always perfectly optimized with a TWM. Not just that the windows are all maximized by default, but that they're arranged in the specific manager that you expect them. The same way each time, and when you need to change the window arrangement you'll be changing it in consistent and predictable ways. I've already set up how I want my windows tiled in each workspace, and that gives me a framework to work in when I need to rearrange windows as my work progresses. The less I need to focus on the minutae of where my windows are and why, the more I can focus on the work I have at hand (like typing a mediocre article on tiling window managers).

Customization is Fun

Yes, having a well-customized system is also a feedback in efficiency and productivity and all the good things. However, if I'm being honest with myself, customization is a bit of flair for flair's sake. To me, that's fun. I enjoy being able to show off a super-nerdy desktop and make things flash around with a couple keystrokes. I have a VS Code extension that puts space images in the background of my code, and while I might be able to make a case that having different colored code panes better helps me find my way around my code, really it's just fun to have.

TWMs offer customization in all of the workspace and productivity ways, but they also offer aesthetic customization, and you can tinker pretty fine-grained with it. Yes efficiency is important but the personalized tweaks can be a huge benefit just for their own sake. I think this is an important point about TWMs, and it's something that we sometimes lack on Windows.

Conclusion

If you use Windows, consider using a TWM. Try it out for a week! It's not everyone's cup of tea, but if you're open to the idea you might find that you like it. I tried it out half a year ago just to see what the hype was and I ended up loving it. The same could be true for you.

Two great ones to consider are GlazeWM and Komorebi. A couple others which I have not tried, but look interesting, are HashTWM and FancyWM.

Happy tiling!

Console2048

Sat, 26 Apr 2014 00:00:00 Z

Jumping on the bandwagon, here's a C# implementation of Console 2048. Of course, 2048 has had a few console implementations, and most better done than this, but here it is anyway, because sometimes life is a mixed bag of apples and grapes.

I've got it on the GitHub here.

Daily Grug

Thu, 26 Oct 2023 00:00:00 Z

One of favorite article is The Grug Brained Developer.

Much wisdom, big article, hard read. Need daily small grug.

So made API for daily grug and made home page.

Deploying ASP.NET 7 Projects with Railway

Tue, 05 Sep 2023 00:00:00 Z

Nevermind that I haven't posted in more than 6 years, Railway is a startup cloud infrastructure provider that has gained a fair amount of traction for being easy to use and very cost effective to get started with. It's pretty barebones right now, but that makes it especially great for hobbyist projects. They have a free introductory tier, but then the next tier is $5/month plus a small resource usage fee. Really, their pricing is fantastic.

When you deploy with Railway, they'll shove your app into a Docker container and handle the management/scaling/etc. behind the scenes. In addition, they have the ability to stand up a database for you - as of the time of this writing, you can choose PostgreSQL, MySQL, Mongo, and Redis. That said, they of course allow you to deploy any docker image or volume, so if you're willing to put in a little more work I imagine you can make any stack work for you. That all means of course that Railway probably isn't the best solution if you need control over container orchestration, but for CRUD projects and startups it seems quite promising to me.

What's especially great is their integration with GitHub - it takes just a couple minutes to sign up for Railway, authenticate with GitHub, point it at a repo, and Railway takes care of the deployment from there. It has some magic to sense what kind of project your repo is and attempt to construct a build pipeline for your project right away. This doesn't work too well in .NET, but their interface is very sparse and easy to use to get up and going with it.

In the remainder of this article, I'm going to be demonstrating how to get a .NET 7 app deployed with Railway. I'll start with a simple ASP.NET API, and then I'll demonstrate getting Blazor working. I'd encourage you to follow along with me - it's no cost to you (you don't even need to type in a credit card) and I think you'll be impressed with how easy it is to get a little hobby app deployed with Railway.

Setting Up

We'll just need three things to get started:

An ASP.NET API
A GitHub repo for that API
A Railway account

Let's go top to bottom there

Setting up a new .NET API

To begin with, I'll assume you have .NET installed, and you have a GitHub account. We can create a barebones API from the console:

dotnet new web -n RailwayAspApiDemo

This starts us off with the following, which will output "Hello, World!" at /:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

In the MyApi directory, we can create a new repo. I'd recommend adding the VS .gitignore first, too.

git add .
git commit -m "Getting Started"

Once you've created a repo in GitHub, we can push it:

git remote add origin https://github.com/{username}/RailwayAspApiDemo.git
git push -u origin master

For reference, you can see this repo here.

Setting up Railway

This is real simple - just go to railway.app, click login at the top, and then you can login with GitHub.

That's all you need in order to set up Railway. Seriously! Of course we're going to push ahead and click that shiny New Project button though...

Deploying our first API

If we had containerized our API with Docker, Railway would have been perfectly happy for us to give it a dockerfile, and it would deploy that no problem. However, Railway also supports building and deploying .NET apps without needing to containerize them. Let's do that first, since we're trying to keep things barebones to get started.

Deploying From a GitHub Repo

One of Railway's coolest features is that you can start a project off by pointing it at a GitHub repo, and it'll automatically (ish) deploy the repo, and set up hooks to listen to any changes on master and deploy then.

Configuring the Repository for Deployment

After logging in, we should be faced with a big New Project button

Here we'll select Deploy from GitHub repo

And then we can select the repo we just pushed

And why not try deploying right off the bat, so long as it's giving us the option?

We should see the deployment fail in just a few seconds.

Debugging the First Errors

Let's click on the deployment and inspect the deploy logs. The first thing to notice is that Railway actually did a really good job guessing what our build config should be. At the top of the logs, we can see:

 setup      │ dotnet-sdk                                    
────────────────────────────────────────────────────────────
 install    │ dotnet restore                                
────────────────────────────────────────────────────────────
 build      │ dotnet publish --no-restore -c Release -o out 
────────────────────────────────────────────────────────────
 start      │ ./out/RailwayAspApiDemo

That's really spectacular! Just because we had a .csproj file, it was able to fill this all out. But it's not all peaches and pringles, we've got a build error. And indeed we're able to see a failure just a few lines down:

#10 1.426 /nix/store/832ihvqk3vxgqqs5hvcyvg6bxqybky14-dotnet-sdk-6.0.403/sdk/6.0.403/Sdks
          /Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(144,5):
          error NETSDK1045: The current .NET SDK does not support targeting .NET 7.0.
          Either target .NET 6.0 or lower, or use a version of the .NET SDK
          that supports .NET 7.0. [/app/RailwayAspApiDemo.csproj]

Classic cloud moment - we need to know how to configure the .NET SDK version. Thankfully, Railway's docs, though sparse, do give us exactly what we need, an environment variable:

NIXPACKS_CSHARP_SDK_VERSION="7.0"

This can be set on the Variables tab on the UI for the service:

Adding that variable should reschedule the deployment. Indeed, it works!

Just one thing - how do we see it? We'll need to generate a domain ourselves in the Settings tab in the UI for the service:

That will generate a slightly random .up.railway.app domain for you to get started with. Of course, you can add a custom domain here if you've purchased one, but I'm going to roll with this because somehow I managed to snag railwayaspapidemo-production.up.railway.app. Luky me!

Now we can navigate to that link and see "Hello, World!" right?

Right?

Well, the build deployed, so let's look at our deploy logs. I imagine yours will look similar to mine:

info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://0.0.0.0:3000
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: /app

It says it's listening on port 3000, so it seems like the app is running, but why can't we see it? That's because when Railway deploys our app, it deploys it in a Docker container and generates a port for us. That means we either need to wire our app up to listen on the port that Railway dictates, or we need to tell Railway to use our nice, pretty port 3000. Luckily, Railway allows us to do both; the port number lives in the environment variable PORT, so we can either override that in Railway or consume the environment variable from our API.

Overriding Railway's Port Assignment

To override Railway's port assignment, we can just set the environment variable in the variables tab, just like how we set the NIXPACKS_CSHARP_SDK_VERSION variable earlier:

PORT="3000"

This will trigger a redeploy, and then we'll cross our fingers, refresh the app, and...

Nice!

Using Railway's Port in our API

Alternatively, if you want to use the Railway-generated port, we can add just a bit of code to do that. Go ahead and delete the PORT environment variable if you added that.

We can update our Program.cs:

var builder = WebApplication.CreateBuilder(args);

if (Environment.GetEnvironmentVariable("PORT") is not null and string environmentPort
    && int.TryParse(environmentPort, out int port))
{
    builder.WebHost.ConfigureKestrel(o => o.ListenAnyIP(port));
}

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Aside: I hate the syntax is not null and string but I'm not going to complain. Too much.

Push that code to master and you should see Railway start deploying your API right away. Once that's up, you should see "Hello, World!" in the browser at your app.

Deploy From Docker or CLI

I could type out a whole section here, but honestly I would just be copying Mark Rendle's excelent explanation. His tutorial was quite helpful for me getting started, and I'd like to give some credit where it's due. So, if you want to containerize your app and use the dockerfile instead of Railway's build steps, or if you want to deploy using Railway's CLI, please give his article a visit!

Deploying a Blazor App

We've got our barebones API up and running, but it's missing a number of things yet. Frankly, that's a trivial example that's hiding a number of problems that still exist. I think Blazor's a good way to demonstrate these, especially since an ASP.NET-hosted Blazor WASM project requires sharing static files.

Setting up a Blazor Project

Similar to the API we created above, we can get the default Blazor project initialized with

dotnet new blazorwasm --hosted -n RailwayBlazorDemo

This will create an "ASP.NET-hosted" Blazor app, which means we'll get a separate client and server project. Go ahead and run this locally - it will spin up the server, and the server will serve you the Blazor WASM client at root:

Go ahead and push this to a new repository (for reference you can see mine here) and create a new project in Railway, linking to this new repository.

That should start a deploy like before, and just like before you'll get a failed build. Remember to set the NIXPACKS_CSHARP_SDK_VERSION environment variable, and resolve the port issue however you choose. I'll choose to resolve it in my code. In order to do that, I'll edit the /Server/Program.cs file with the same lines we added to the barebones API. While that deploys, we also need to generate a domain for this app like we did before. And, don't you know it, I got lucky again: railwayblazordemo-production.up.railway.app! Neat.

Configuring Railway to Deploy the Server

At this point, we might expect it to work. However, you'll notice after building Railway attempts to start the service several times, but fails with the same message:

/bin/bash: line 1: ./out/RailwayBlazorDemo.Client: No such file or directory

Uh oh - we don't want to deploy the client, we want to deploy the server, because the server is configured to serve the client. The cause of this can be seen in the build logs like we'd expect - the automagic build figurer-outer guessed that we wanted to deploy the client:

setup      │ dotnet-sdk_7                                  
───────────────────────────────────────────────────────────
install    │ dotnet restore                                
───────────────────────────────────────────────────────────
build      │ dotnet publish --no-restore -c Release -o out 
───────────────────────────────────────────────────────────
start      │ ./out/RailwayBlazorDemo.Client

This, like the other environment configuration issues in Railway, is simple to resolve. If you navigate back to the Settings tab on the UI for the service, scroll down and you'll see Deploy settings, with a helpful place to override the start command. In fact you can override any of the build steps in these settings, although you'll notice that they're pretty sparse. For our needs here though, those settings are all fine, so we'll just update the start command to ./out/RailwayBlazorDemo.Server:

This will trigger a rebuild, and that should succeed! Our app should now be at the address we generated earlier, right?

Right?

Configuring the ContentRootPath

Well, that's interesting, because it's a different error than we got when first deploying the barebones API earlier. In that case, we got a nice error displayed with Railway's UI. This tells us that the server is up and running - of course though we verified that when the server started logging after it deployed a minute ago. Thus, we know that the problem is with the server being able to serve up the client app.

What's going on here isn't entirely obvious and it relies on a bit of knowledge about Docker to be able to intuit what's going on. There are two key lines in the logs. The key line is in the deploy logs:

Content root path: /app

What's going on is that the client is stored as a static file on the server, and the server needs access to that file to be able to serve it, of course. ASP calls the root directory for static files the "content root path", and this one is a bit bunked.

This has a code solution. Replace the first line of Server/Program.cs with the following:

var builder = WebApplication.CreateBuilder(new WebApplicationOptions() {
    Args = args,
    ContentRootPath = "./"
});

Yes, ContentRootPath is supposed to default to its root directory, but there's some weirdness that got introduced ... *checks notes* ... somewhere? Honestly, I'm not sure how this solves it - I've just debugged enough weird directory issues with Docker in my professional career that it triggered my spidey senses.

Making that change and pushing to master will trigger a rebuild. Then, as if by magic, our app is working at the link:

Conclusion

Despite a few bumps in the road (which, apart from the content root path issue, make sense in context) we were able to get two .NET apps deployed in no time and with no money! In future, I think we'll explore adding a database with Railway and hooking our Blazor app up to it.

I think this is the sort of case where Railway really excells. If you don't have an overly complicated backend system, deploying with Railway is extremely fast, simple, and cheap. Their focus on this area significantly reduces the barrier to entry to get a hobby app out the door. And, it seems that Railway does have enough capability to scale if you do attract users - at least through the first phase or two. Because of these factors, I'll be using Railway to deploy all my hobby apps in the future here! I'm very excited to discover a cloud provider this capable at this price.

Railway's limitations are very apparent though - such is the tradeoff with the simplicity they've achieved. While you certainly can deploy any container with Railway, an overly complicated backend system could potentially become more burdensome to maintain than not. Really though, I don't know where that boundary is, but I suspect it's decently high enough that even if I do take some pet projects into production properly, I can rely on Railway to be able to serve them adequately.

Resources

You can see my GitHub repos used in this article here:

Railway's documentation is pretty good, but their Discord server is an excellent and lively place to get help when you need it.

Deploying Your Prolog API with Docker

Fri, 28 Jun 2024 00:00:00 Z

Prolog is truly a leader at the forefront of modern technology, and if you're anything like me then you're convinced that this langauge is the way forward for our microservice APIs. Part of our evidence for this (as though it isn't inherently obvious) is how easy it is to containerize and deploy with Docker.

To demonstrate this, let's set up a simple Hello World API with SWI Prolog:

% Import SWI modules for HTTP servers
:- use_module(library(http/thread_httpd)).
:- use_module(library(http/http_dispatch)).
:- use_module(library(http/html_write)).

% Set handle_request as the handler for URL /
:- http_handler('/', handle_request, []).

% Respond "Hello, World!" to requests
handle_request(_Request) :-
	format('Content-type: text/plain~n~n'),
	format('Hello, World!').

% Start the server listening to localhost:Port
server(Port) :-
	http_server(http_dispatch, [port(Port)]),
    
    % Spin while waiting for the next message
	thread_get_message(_).

There's a couple things to point out here. First, accepting a Port variable for the server predicate allows us to keep the port flexible as an environment variable, best to not hardcode that. Second, invoking the thread_get_message predicate isn't necessary when we're debugging locally but it is going to be necessary when we run it in Docker to prevent the server from halting immediately.

The dockerfile isn't terribly difficult at all. We can base off of ubuntu:latest, install SWI Prolog, and then run the command to start the prolog interpreter with the call to the server predicate:

# Use an official Ubuntu runtime as a parent image
FROM ubuntu:latest

# Set the working directory
WORKDIR /usr/src/app

# Install SWI-Prolog
RUN apt-get update && \
    apt-get install -y software-properties-common && \
    apt-add-repository ppa:swi-prolog/stable && \
    apt-get update && \
    apt-get install -y swi-prolog

# Copy the current directory contents into the container at /usr/src/app
COPY . .

# Make port 5000 available to the world outside this container
EXPOSE 5000

# Run swipl when the container launches
CMD ["swipl", "-g", "server(5000)", "-t", "halt", "server.pl"]

I'm hardcoding 5000 as the HTTP port here but it's just as easy to use an environment variable at this point. Note too I'm assuming the source is in server.pl.

With this we can build the image:

docker build -t prolog-server .

And deploy:

docker run -p 5000:5000 prolog-server

And that's it, we're off to the races! With dev ex this smooth, Prolog will concur the world. Any day now. Surely.

Deploy Your Own NocoDB on Railway

Mon, 12 May 2025 00:00:00 Z

NocoDB is an interesting open source project I came across some weeks ago; it aims to be an OSS alternative to Airtable. Airtable is a "nocode" solution for line-of-business applications, focusing on business process automation. It's like an enterprisey IFTTT (although; IFTTT would probably want you to think of them as the enterprisey IFTTT). NocoDB is in its early stages but seems to want to occupy the same niche, but there's a couple differences that make it useful to me for other purposes.

Crucially, it's built as a UI wrapper around a database of your choice. Their documentation is a bit sparse right now, but I think I'm right in saying they support Postgres, MySQL, and SQLite at the time of this writing. If you've read my work before, you know that I like Postgres. There's a lot of UIs you can use with Postgres, even some that have fancy visualizations like this NocoDB, but the difference with NocoDB that's impressed me is in their support for automation: by developing their software into the no/low-code niche, it becomes more useful for those cases where I just need to set up some trigger here or interaction there.

I've been wanting to replace all my spreadsheets with Postgres for a while now, and NocoDB might just be the UI that lets me do that. Being OSS, I can deploy it myself with my own Postgres database for free! Here I'm going to outline how that's done in Railway, my preferred cloud host. The relevant docs from NocoDB are their Docker install and environment variables.

The setup will be quite simple: we'll just need a Postgres database and a Docker container (with a volume) for NocoDB. We'll start by creating an empty project:

To which we can add a Postgres database:

I was fine with the default Postgres settings, but feel free to change yours how you need.

Creating the NocoDB container is easy enough; the image nocodb/nocodb:latest is available from docker hub. We can create the container by selecting Docker Image when we go to create the container:

and entering the image name

You can deploy this now if you like, but we're going to want to attach a volume and connect it to Postgres to work. To attach the volume, right-click on the NocoDB container:

And mount the volume to /usr/app/data/

In order to connect to the postgres database, we can set the NC_DB variable on the NocoDB container. You can click on the container and then the Variables tab:

Railway has support for referencing other services in its environment variables, which we can take advantage of in constructing our connection string. When I added my variable, I used the following to get a connection string in the format that matches NocoDB's docs:

pg://${{Postgres.RAILWAY_PRIVATE_DOMAIN}}:${{Postgres.PGPORT}}?u=${{Postgres.PGUSER}}&p=${{Postgres.PGPASSWORD}}&d=${{Postgres.PGDATABASE}}

That's all we need now in order to deploy and check that we're successful. Once that's verified, you can configure a public URL for your NocoDB container and start using it!

Develop Effective Coding Standards

Wed, 14 Feb 2024 00:00:00 Z

Coding standards can be a blessing or a curse: a practical resource that helps the team consistently maintain their codebases or an overbearing cudgel of impractical formatting prescriptions. Indeed, ineffective coding standards are worse than no coding standards: they are frustrating and can exacerbate any maladies on the team. I'd go so far as to suggest that even effective coding standards are not a necessity in the industry; many teams might naturally have no need for standards.

There are several situations where it's beneficial to have a solid set of standards - if the team has a large number of codebases (say, microservices) in various states of disrepair, if the team needs to onboard engineers frequently (say, for intern/junior rotations), or if your team is embarking on a 1+ year refactor of a large, legacy system with too many different styles. These are all situations where good coding standards can help ensure that a whole group of people is aligned over a period of time.

I'll admit failure on my part here. I have created some godawful coding standards in the past - ones which have an exclusive focus on style that explain nits for twenty pages. I hope I've learned from my younger self, that proper standards are a very different beast. I like to think that since then I've been able to establish some standards which have been quite good. If you take nothing else away from this article, remember that the practice of creating and maintaining standards is an active, full team activity. The whole team should be engaged in developing the standards, and they should be reviewed by the whole team regularly.

Less is More

Bad coding standards are meandering swamps of useless bloviation. It's not useful to anyone to have fifty pages listing out "Do this, not that" steps. I should be able to skim the standards quite quickly and come away with a solid understanding, if just a feeling, of the way this team wants its code developed.

Focus on what's truly important. You'll have plenty of "Do this, not that" examples, but use those only to disambiguate where necessary. Explain key, high-level concepts. Avoid getting too far into the weeds of specifics. Your goal should be to lay a solid foundation on top of which any number of beautiful houses can be built.

Focus on "Why"

Prescribing rules for coding is generally useless. They are annoying and can get out of date quickly, so they end up ignored in the long run. They get overlooked on review because they fade into the background when the document is in active use.

What's more important - 1000 times more important - is to document why your team feels a certain way. Suppose your team prefers having a strong emphasis on being able to read your code more vertically than horizontally. You might want one rule to limit the width of the document, one rule to format ternaries on multiple lines, one rule to break conjunctions in conditionals on multiple lines, etc. Rather than stating each of these individually, they can be examples in the broader context of verticality.

## Prefer Verticality

This team prefers writing code in a way that makes it more _vertical_ than _horizontal_. Run-on lines tend to be more difficult to read given our domain and architecture. As a consequence, the team is able to read code faster when it's consistently formatted vertically. We like to limit the horizonal width to 80 characters, and break code into multiple lines in logical places. For example:

**Use multi-line ternaries**

Do not prefer:

var myThing = first >= 50 ? first : second;

Do prefer:

var myThing = 
    first >= 50
    ? first
    : second;

**Break long conditions into multiple lines**

Do not prefer:

if (thingToCheck is SomeThing someThingToCheck && service.CheckSomeThing(someThingToCheck) && businessConditionForNextLogic && featureFlagConditionForNextLogic)

Do prefer:

if (
    thingToCheck is SomeThing someThingToCheck
    && service.CheckSomeThing(someThingToCheck)
    && businessConditionForNextLogic
    && featureFlagConditionForNextLogic
)

By explaining why, you're giving everyone an insight as to what the team feels, not just the result of their feelings. You don't need to enumerate every single rule that follows from that feeling since you've (hopefully) articulated the why. It's easier to refactor these statements as your team's feelings shift since you've written those feelings down - the consequences might be harder to grasp.

Continuous Learning, Continuous Improvement

I hope that you and your teammates are learning constantly, and always becoming better engineers. As your team changes, its opinions on how code should be developed will change too. This should be reflected by continuous reviews of the coding standards. First though, you need some way to capture that you're learning and what you've learned.

Retros are the obvious candidate - a "What did you learn in the last two weeks?" question is great. It's less useful for capturing changes in your coding disposition over time though. Monthly or quarterly learning check-ins can be good for these, but whether that works for your team to adequately capture those and translate them into changes in this document are an individual question.

Your team should establish some periodicity by which it reviews all of its documentation, and your coding standards should be part of that. This can be monthly, quarterly, or yearly - just what works for you. Importantly though, consider starting these off by asking everyone on the team what they learned in that time. Did they experience anything cool or interesting, out of their usual development practice? Are they excited by something new they saw they wanted to try?

However you collect it, document learnings and use them to drive the review of the coding standards. This feeds back on explaining the why and not the what in the document itself.

Coding Standards Affect Culture

If the standards are too nitpicky, you'll start seeing too many nits in code reviews. If the standards are too prescriptive, you'll see fewer creative ideas in the codebase. If the standards don't embrace new features, libraries, or architectures, your codebase will be stuck in the past. Maybe these are actually effects that you want - perhaps you're maintaining a large COBOL platform, and modernization will halve your paycheck. Fair enough.

For the rest of us, I'd venture that we want to work on exciting, collaborative teams that encourage using new ideas to continuously refine the code. If you've got effective standards that the team is actively using, these standards need to be an example of the kind of culture you want to work in.

By developing the standards themselves collaboratively, this will cause the team to develop more collaboratively. By keeping standards focused on the why, PR reviews and pair programming sessions will have a focus on why. If your standards allow for variance within the parameters your team must work (business requirements, etc) then your team's solution space will be as wide as possible. If you review and improve your standards at intervals where you can incorporate the latest new features of your language or libraries - for example if you're a .NET team and you review standards every November after each .NET/C# release - your team too will incorporate more of the latest features.

Conclusion

Are coding standards necessary? No, at least not for everyone. If your team isn't going to use the standards, don't create any. If the team isn't engaged in developing the standards, then you'll probably get subpar standards. Remember, unless you've got effective standards it's better to not have any. If your team is in a pinch where they decide they need to align on coding practice, then having concise standards to document its feelings can be beneficial. Develop the standards and review them constantly from a collaborative perspective, and focus on documenting your feelings with a small number of key examples.

But how do we use these standards once we've written them? Well, curiously, you might not. If you're onboarding a new member of the team or explaining yourselves to another engineer in your organization, you should break out these standards to show off how the team feels. But if your team is really aligned on development practice - as it should be after developing a set of standards - then you and your teammates won't need them, not day-to-day. Standards can't be used as a cudgel to enforce conformity, that creates resentment and disfunction on the team.

Don't take this to mean they're not useful at this point - indeed it's the very process of continuously reviewing this document that keeps your team aligned and allows everyone to update and refresh their ideas on the same page. It's curious - the primary utility of the document is not the document itself or the regular use of the document, but the regular and ongoing development of the document as a team.

Don't Retro the Same Twice

Mon, 22 Jul 2024 00:00:00 Z

Retrospective meetings are a cornerstone for most modern software engineering teams. They're the only regular practice mentioned in the agile manifesto, they've been written about ad nauseum, and many colleagues seem quite excited on the days their teams have retrospectives. Well-conducted, the retrospective can offer a lot of benefits to a team, their projects, and their organization on the whole, so everyone really wants to get the retro meeting right.

There's maybe a billion (I could be underselling it) formats for retro meetings, and in their care and consideration some folks have very strong opinions on the "right" structure for retro meetings. As (I hope) my writing on this blog can attest, I'm not a fan of that way of thinking about process. Retrospectives are simple - as long as there is a way to identify and discuss problems and successes then you're good to go; specific formats give some icing on the cake.

That icing can be quite important at times. While the primary utility of the retro is to identify and expedite the solution to problems, there's a host of ancillary benefits that can be gained from different retrospective formats. Indeed, different retro formats are going to be better suited to expressing different benefits. Maybe your team has a project which requires something more specific out of your retrospectives, and a particular format is obviously good for your team. That's great! Just remember it might change.

For most teams, one format isn't going to be clearly superior, but they can benefit - and benefit differently - from any format. So this is the advice I'll give: Don't have the same retro meeting twice in a row. This allows your team to get the benefits of all the formats, and there's a few ways to go about it. If you've got some persnickety folks on the team, you can rotate between different preferred formats to keep them all happy. You could, ahead of the meeting, briefly describe a couple of obvious challenges to Chat GPT and ask it to recommend a retro format to suit. My preferred way is to pick a format out of a hat and run with it.

Whatever you choose, don't retro the same twice! No format can do all the things for you and your team, and keeping the retrospectives fresh allows your team to approach their problems from different angles. For short-term problems (problems within a sprint) any format is going to be sufficient - as I said earlier all that is really needed is the ability to identify and discuss these. However, a lot of teams have long-running or more complicated problems, and the different angles approach might be better at unblocking these.

For the sake of being (too) analytical about it, here are some of the ancillary benefits (beyond identifying problems, improving process, and adapting to change) that retrospectives offer:

Increase collaboration between team members
Opportunity for distributed teams to build culture
Celebrating individual and team successes makes everyone feel good
Builds trust between members
Helps individual members learn from mistakes
Ensures team members have ownership over process
Ensures members are aligned on team goals
Allows individuals to "vent" when needed

And then here are some common formats, and how they express different benefits. You can find endless blogs about retro formats, so this isn't an exhaustive list but an exercise to point out how different formats can yield different benefits.

Four Ls

"Loved, Loathed, Longed For, and Learned". The "Loathed" column is particularly beneficial for members to see the results of mistakes and learn from them, and the "Learned" column specifically helps to stimulate collaboration.

Sailboat

"Wind (what's helping us), Anchor (what's holding us back), Rocks/Icebergs (potential future blockers), Island (what's the destination)". The "Rocks" and "Island" columns ensure the retro is balanced on looking to the future as much as the past, which enhances understanding and ownership over goals.

Mad, Sad, Glad

Self-descriptive, I hope! This breaks down the barriers to conversation, helping to build culture, celebrate successes, and allowing individuals to vent.

Six Hats

Each member "wears" one of the hats - Facts, Emotions, Creativity, Process, Positives, Negatives - and each issue is looked at from that perspective. Forcing these perspectives to be accounted for helps to stimulate collaboration and trust between individuals.

Five Whys

For each problem, ask "Why?" five times. By hyper-focusing on root causes, this can give the team a lot of control over its process by gaining a better understanding of how problems arise from situations or facts.

STAR

Each problem is described by the Situtation that happened, the Task that was being done, the Actions that were taken, and the Result. This is a hyper-focus on the relationship between goals, actions, and results, which is beneficial for helping individuals learn from mistakes. In the right situation it can help members vent in a productive way.

.NET 10's Single File Feature is Perfect ... Almost

Sun, 22 Jun 2025 00:00:00 Z

.NET 10 is finally introducing a feature that is some 20 years (give or take a few) too late, allowing a single CS file to be ran as a standalone program, not requiring any solution, project, or Nuget files. This is something I've been needing for some time - yes, needing, not wanting - so I just installed the preview version for .NET 10 and I'm glad to see that it's very nearly working perfectly out of the box.

For background, my use case is using C# to script builds in GitHub actions, particularly the static generator for this blog. For some time I've been (more or less) keeping up my project Metalsharp, a C# version of the JS static site generator Metalsmith. I'm really happy with the project and I and other colleagues have used it successfully in a pretty wide set of applications. The only problem is that C# doesn't really work as a scripting language; it's compiled. It is a bit ridiculous to suggest that a build agent using my tool should include a build step to build the script that builds the site, so I turned to the dotnet-script, a dotnet tool that allows a single CS-esque file to be executed with a single command.

There were a few troubles with that workflow: The build agent needed to include a separate step to install the dotnet-script tool, I generally prefer not to rely on third party things for very foundational components of any project, and the builds took a long time to execute. What would take a fraction of a second on my local machine debugging with a CSPROJ file took 60 seconds in GitHub actinos with dotnet-script. The biggest problem though was the local development experience: I've never been able to get any intellisense for dotnet-script in any IDE, so I'd have to build the project to see errors. Alas, there was no other way to run a single CS file without a build step.

This all prevented me from considering Metalsharp to have reached v1, and I haven't yet published or publicized the project. I hope this sets up how excited I am that .NET 10 is going to be including this feature: I'll be able to "officially" release this years-long project if it's a stable foundation for development and production.

As I mentioned, I did get it up and going and the initial indication is a great success. Importantly, it works out of the box exactly how I'd expect; there are no surprises or hidden configurations or the like to get it to work locally or in my GitHub Actions. It does indeed solve most of the problems I've had with dotnet-script: I need no extra step to include the feature with dotnet, the feature is a first-party part of .NET, and what had been taking 60 seconds to run in GitHub is now taking 8. To me, this is a runaway success, except for this one issue: it still doesn't have intellisense support, at least not that I can find.

This is a preview feature of .NET so I'm expecting that this will be fixed when it is released in November, but for the time being it is still quite annoying. Nonetheless, in the next couple of months I'm going to be preparing my Metalsharp project to bump to v1 with .NET 10, and I'm very happy about that!

You can read more about the feature on Microsoft's post or look at the build configuration for this blog to learn more about this feature.

Hot off the Press: .NET 9, C# 13

Wed, 13 Nov 2024 00:00:00 Z

The new .NET and C# are out as of yesterday, and I'm underwhelmed. Well, maybe since we've known for some time what was in this release I should just be whelmed.

AI being the current hype, Microsoft is placing front and center what it can find related to AI. On the positive side, this means some updates to Tensor<> and vectors, though these don't impact me on the day-to-day. Most of what they've got seems to just be a version bump for AI clients from OpenAI and such. In fact, I'm a bit hazy on what real problems the set of "AI" features are solving other than giving them the ability to do presentations about how ".NET 9 enhances support for AI." If you've got more information please comment below.

Then they're really pushing .NET Aspire, which I continue to skeptically watch; is Aspire not a conspiracy by big Azure to sell more cloud? I can't imagine that's not part of the calculus, but Aspire has a couple cool things, the local dashboard is neat, as is the container orchestration. It still seems like a quite opinionated sort of boilerplate I need to write to get it up and going when existing tools (docker compose and the tooling around that) take just as much code, have been around longer, and give me most of the same advantages.

Aspire has some neat logging improvements now on their nifty dashboard, but you get this with other standard otel tools like Honeycomb. Even the local development experience doesn't seem to be necessarily better than the experience with existing tools, just different. Aspire does make it very easy to throw on distributed components once I've got an app up and going with it, but not easier than my existing setup. I feel like I have to be missing something here, but if I'm not then this is just a more architecturally-constraining way to distribute your application. Am I becoming the old man yelling at the cloud? If you have an answer, please let me know.

Then there's a decent smattering of improvements across the board for ASP, SignalR, and Blazor. I'm excited to update a few applications for these benefits, but they're just the standard sort of regular improvements. Which is good - don't get me wrong - just whelming.

The coolest such improvemnt is that SignalR now allows covariant types, with the caveats that you're using System.Text.Json and are comfortable throwing some attributes on your models. I'm going to see about setting up covariant result types with this, possibly as a follow up to my article on properly using the result pattern in C# for use in SignalR.

C# doesn't have a lot going on, which is maybe a welcome change of pace considering each version since 7 has introduced very new things. It does leave a conspicuously discriminated-union-shaped hole in these release notes, though. To be fair, it's my impression that such an addition has a fair enough amount of complexity that a slower and more deliberate approach is needed. Fair.

The annual performance article by Stephen Toub is excellent as always, and excitingly (for me at least) seems to reveal a lot of work put into compilation in this release. That's the best news - it suggests improvements across the board.

.NET: From Framework to 10

Fri, 21 Nov 2025 00:00:00 Z

It's been more than six years since the last major version of .NET Framework (4.8) was launched along with C# 7.3, and the .NET world has come a huge way in that time. Last week Microsoft released .NET 10 and C# 14, continuing a trend of releases that significantly rework how software is written on the .NET platform. While Microsoft, and most brand-new .NET development along with them, have been making these strides, there's still plenty of older applications still on .NET Framework; as a result, there are plenty of engineers focused on maintaining these codebases that feel increasingly disconnected with modern .NET practices.

Each year now brings a new release of .NET (that is ".NET" and not ".NET Framework" or ".NET Core") along with a new release of C#, and each of these releases is a significant lift to the libraries, language, tooling, best practices, and standards. C# has become much more accommodating (and in some cases requiring) functional programming styles, while .NET has introduced sweeping new features across all of its workloads. This huge amount of change is reflected in the ways engineers think about, architect, and productionalize applications in .NET.

I can't give a perfectly thorough explanation of all the ways that .NET is different now than it was in 2019, but I do want to type up some thoughts on what's going on. Each year it becomes more and more difficult for .NET Framework engineers to make the hop over to the latest .NET with the same level of productivity and familiarity. It's not just the new .NET APIs or the new C# keywords but the styles and paradigms of the development and the "why"s motivating the new changes. 10 seems like a good version number for a catch-up, so I want some writing here that can serve as a jumping-off point for .NET Framework engineers wanting to get up-to-speed with the latest toys.

It is difficult to, for any individual update, explain the precise way that the new features have resulted in a different style of development. This is particularly so since many of these changes have been made in order to accommodate development practices that had already emerged; the language and framework evolved to meet the requests of engineers that have been influenced from other sources. Thus, by giving an explanation of modern .NET practices by starting from these updates, I'm missing out on the other force that is moving .NET development along, and that will need to be a future post from me. Nonetheless, I think that the majority of .NET engineers are not on this vanguard but are instead wanting to adopt these new features into their existing workflow, and this is where I see the utility of tying all these into an explanation of "modern .NET."

As a fair forewarning though, I'm largely going off of memory and some incomplete historic documentation from Microsoft to figure everything out historically so there might be some inaccuracies around the edges here. However, everything I say you can do in C# and .NET now is accurate, maybe the history is a bit wonky. Furthermore, I'm also not giong to say anything about F#. I enjoy using F# but I use C# professionally, so I want to focus on how changes to .NET and C# have affected how we do development with those two. Finally, I will not be explaining any improvements to EntityFramework because a polished dung is still a dung.

.NET History: Framwork, Standard, and Core

I think a (very small) bit of history can help understand most of the reasons that have motivated the lion's share of changes in .NET and help disambiguate terms. .NET Framework came about ages ago to be the standard library for the (then) new C# language, and expanded and evolved quite rapidly alongside the language to support all the different kinds of applications Microsoft wanted to support. ASP was added as a web framework, WinForms (then WPF then a mess of other things) supported desktop GUIs, WCF supported SOAP, Silverlight allowed engineers to frustrate everyone who wanted to watch the 2008 Olympics, and so on.

As software engineers we understand very well that if requirements expand and change too rapidly and haphazardly, the sooner a rewrite of the system is going to be needed, but we usually try to stave off a rewrite until something happens to make it necessary. What happened to .NET a bit over ten years ago is it needed to run on Linux among a host of other factors (I'm boiling it down significantly). It came to be that a rewrite and some breaking changes needed to be made.

So, Microsoft embarked on creating an alternative standard library for C#, which they called .NET Core, and released it in 2016. They also created a separate deal called ".NET Standard" to act as a standard between .NET Framework and .NET Core, guaranteeing certain similar APIs in both standard libraries to ease transitioning from one to another, or writing code that could be guaranteed to work in both libraries.

.NET Core progressed through a few versions before Microsoft quickly decided that it would be best to only develop updates for the new .NET Core going forward. In 2019 they released the last major version of .NET Framework, but also to demark the change they decided to be rid of the "Core" name as well. The last version of .NET Core, 3.1, was developed into the first version of ".NET", which they began at version 5 as the lowest major version number not common to either of its predecessors. Each year since .NET 5 was released, Microsoft has released a new major version of .NET along with a new major version of the C# language, and this period has seen the most significant updates to the framework and language.

.NET Core: 1.0 to 3.1

The various versions of .NET Core essentially ended up serving as the ramp-up period for the new .NET development, and did not very significantly diverge from .NET Framemwork but for having full support for Linux and MacOS.

The first super duper thing introduced with it was the dotnet CLI tool exposing actions like creating a new app, building it, restoring packages, and the like. This acts in a similar way to the compilers of most programming languages, facilitating doing C# development in a similar manner to any other language. Go has commands like go run ., and now .NET has dotnet run as well.

.NET Core 3.0 is the first interesting .NET version to me. It introduced the ability to publish an app as a single file; instead of having to ship a bunch of DLLs and support files, the C# app gets contained in one .exe or whatever format it is for the target OS you select.

.NET Core 3.0 shipped along with C# 8, which had a huge number of updates to the language. The most significant feature was the nullable reference types. Null was, previously, very tricky with reference types, since they always had the option to be null, requiring huge amount of null checks and lots of time lost to reference-type variables unexpectedly becoming null in prod. The solution was to allow the nullable ? operator to be added for reference types, and a compiler directive to tell C# that it should throw warnings or errors if null checks weren't explicitly added around these.

string? myString = ...;
PrintLength(myString); // Error: myString can be null

public void PrintLength(string s) =>
    Console.WriteLine(s.Length());

In the csproj, the Nullable property needs to be set, and now is by default:

<Nullable>enable</Nullable>

I'm going to dump a bunch of little things here:

One of the cooler things in C# 8 is default interface members:

public interface IThing
{
    string GetName() => "Unnamed";
}

public class Thing : IThing {}

Console.WriteLine((new Thing()).GetName()); // Prints "Unnamed"

As well as "static abstract" members:

public interface IThing
{
    static abstract string Name { get; }
}

public class Thing : IThing
{
    public static string Name => "I'm Thing"; // Required to be present to compile
}

public void PrintName<T>() where T : IThing =>
    Console.WriteLine(T.Name);

Switch expressions were also introduced, allowing you to treat a switch as a value:

var myThing = myIntThing switch
{
    1 => "one",
    2 => "two",
    _ => "larger than two"
};

This supports very functional-looking development doing pattern matching on the shape of arguments in a method, though this example is admittedly very contrived:

public bool IsEmpty<T>(IEnumerable<T> list) => list switch
{
    var l when l.Count > 0 => true,
    _ => false
}

Local functions are pretty significant and just support keeping appropriate scope for methods:

public void DoThing()
{
    string GetString() =>
        "hi";

    Console.WriteLine(GetString());
}

I make huge use of null-coalescing assignment for lazy loading properties:

public class Thing
{
    private MyLoadableThing? _prop;
    public MyLoadableThing Prop => _prop ??= new MyLoadableThing();
}

A huge quality of life improvement came with .. as a range operator like many other languages have (this example taken from MS's docs):

int[] numbers = [0, 10, 20, 30, 40, 50];
int start = 1;
int amountToTake = 3;
int[] subset = numbers[start..(start + amountToTake)];

One of the most interesting additions to the .NET world with .NET Core 3.0 was Blazor, which takes a bit of explaining for the uninitiated. For some time, there's been an amount of effort to create an assembly language for the web - Web Assembly or WASM - understood by all browsers, that can be as ubiquitously implemented as JavaScript but that is, well, assembly, instead of JS. This has resulted in a number of languages being able to target WASM as a platform separate to Windows, Linux, and so on.

Microsoft figured that with .NET Core intending to support multiple platforms that WASM should be no different, but of course the use case for WASM is web apps, so Microsoft developed Blazor as both the .NET runtime for WASM as well as the framework for developing a web UI in C# for WASM.

I won't dig too much into Blazor here, that can be a whole separate thing. However, I'll point you to an open source app I have: FreePlanningPoker.io, which is a simple Blazor + SignalR app that you can dive into for a taste of Blazor!

.NET 5 and C# 9

Being the first version to "consolidate" the standard library post-Framework, there wasn't too much added to the APIs here. Most significantly is probably that ASP introduced the option to self-host instead of needing to rely on IIS, making it much easier to deploy containerized.

C# 9 was released in tandem, and just like the previous release had a huge number of things included. I think the most significantly new feature in C# in the last five years was introducted in this version: records. A record is a type of class in C# that is primarily designed for immutable data; that is, a class where no properties change. These are used extensively in functional languages and the compiler can make a lot of optimizations for immutable data when you use them.

public record Person(string Name, int Age);

This is kind of equivalent to:

public class Person
{
    public string Name { get; private set; }

    public int Age { get; private set; }

    public Person(string name, int Age) { /* assign props */ }
}

Except the Name and Age properties can never be modified (either from inside or outside the record). Further, records have deep value equality, so:

var first = new Person("John", 30);
var second = new Person("John", 30);
Console.WriteLine(first == second); // Outputs true

Functional styles with immutable records will typically assign new values when record data needs to change, and C# supports this pattern now with the with keyword:

Person Birthday(Person person) =>
    person with { Age = person.Age + 1 };

The other really awesome change is a quality of life improvement that Program.cs does not need a class, Main method, or even a namespace now. The following will compile as a valid C# program:

Console.WriteLine("Hello, World!");

.NET 6 and C# 10

.NET 6 started seeing some new things introduced in the API level. Most notably though is MAUI, "modern application UI", which was created to replace Xamarin and WPF as the cross-platform way to develop UIs in C#. This is a huge disappointment: it was buggy and incomplete on its release and it has not improved very much since.

In the ASP level, .NET 6 introduced "minimal APIs," which were developed to be able to have as easy a proof-of-concept story as other languages. For example, in Python, setting up a hello world endpoint is trivially simple:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"

In ASP this had previously required creating a controller class, and while that's not difficult, it's not super speedy feels like the future, so now we've got this as an option alongside controllers:

app.MapGet("/", () => "Hello, World");

Which is actually astonishing for rapid development. Kudos here. Other great thing: lots of Open Telemetry support. I just use the regular OTEL added with the templates nowadays so I'm kind of dumb as to what specifically was done here, but it was done.

On the C# side, the super cool deal everyone was talking about was file-scoped namespaces. Now, instead of:

namespace MyApp
{
    class MyClass
    {
        // ...
    }
}

Do this:

namespace MyApp;

class MyClass
{
    // ...
}

I refuse to read code if it is >= version 10 and does not do this. Another cool namespace thing you can do:

global using WhateverNamespace; // now no other file needs "using WhateverNamespace;"

And then the functional support is extended through pattern matching with "property patterns", this from MS's docs:

if (e is MethodCallExpression { Method.Name: "MethodName" })

This can be a bit unintuitive at first, but it fits right in line with how a lot of languages with better pattern matching support do. Pattern matching is hugely important for functional patterns, and can be kind of fun:

public abstract record Shape;

public record Circle(double Radius) : Shape;
public record Rectangle(double Width, double Height) : Shape;

public static double Area(Shape shape) => shape switch
{
    Circle { Radius: var r } => Math.PI * r * r,
    Rectangle { Width: var w, Height: var h } => w * h,
    _ => throw new ArgumentException("Unknown shape") // required for exhaustiveness in the switch expression
};

.NET 7 and C# 11

I remember .NET 7 being very cool, but looking back at the historical documentation I can't really remember this happening. If I'm remembering correctly, .NET 7 came out a little over a year after I started at Crate & Barrel, and I had upgraded some brand new microservices from .NET 5 to .NET 7 right then because I could. What a deal!

The really cool thing here was you can now choose to have .NET generate a Dockerfile when you publish your app, instead of just compiling it. This makes it so you don't need to hand-write the Dockerfile, which is a really nifty feature and has supported being able to debug directly in Docker from Visual Studio. Maybe Code also. I don't use it very much but no doubt it's nifty.

This was also the year that Stephen Toub's annual performance improvement report became the talk of the industry. He'd been writing these for some time, but this one was really overwhelming at the time. To say these articles are "huge" would be an understatement as large as these articles are; they're incredibly thorough, meticulous, and technical, but a "TLDR" on the .NET 7 article sums up the cause of its impact in this particular release:

TL;DR: .NET 7 is fast. Really fast. A thousand performance-impacting PRs went into runtime and core libraries this release, never mind all the improvements in ASP.NET Core and Windows Forms and Entity Framework and beyond. It’s the fastest .NET ever. If your manager asks you why your project should upgrade to .NET 7, you can say “in addition to all the new functionality in the release, .NET 7 is super fast.”

.NET 7's focus on performance continued past this release into each of the subsequent releases, and each year since I've observed improvements from each version bump we do. .NET gets a lot of flack for being bloaty; .NET 7 is Microsoft righting that course.

Apart from supporting dark mode for developer exception pages, ASP had a lot of small quality of life features, but it's tough to parse out a single one that's changed how development happens. By this point too, development on WinForms and WPF had largely stalled, while MAUI and Blazor made a couple steps forward but not enough for me to really enjoy them. MAUI remained half-baked and Blazor remained difficult to debug.

C# 11 brought a bunch of things that have really influenced how I do development.

The feature they called "generic math" had a huge amount of discussion on the C# language design repository, and launched to much acclaim. Essentially they had all the numeric types in C# implement INumber<self>, so Integer implements INumber<Integer> for example. This allows math to be done indiscriminately of whether the numeric type is double or int or whatever. From MS's docs:

public static TResult Sum<T, TResult>(IEnumerable<T> values)
    where T : INumber<T>
    where TResult : INumber<TResult>
{
    TResult result = TResult.Zero;

    foreach (var value in values)
    {
        result += TResult.Create(value);
    }

    return result;
}

Very cool! I never use it, but indeed there are many applications for it.

What I do use all the time, particularly for SQL scripts, is raw string literals:

var myString = """
    Hello,
    World
    """;

And you can specify how many curlies are needed for interpolation by adding $s:

var hi = "Hello";
var myString = $$$$$"""
    {{{{{hi}}}}},
    World
    """;

File types are also something I make a lot of use of. Instead of private nested classes, file will limit the scope of a type to a file:

// No longer do:
public class MyClass
{
    private record MyRecord(../);

    // Use MyRecord
}

// Now do:
file record MyRecord(...);

public class MyClass
{
    // Use MyRecord
}

Saves nesting, makes it easier to navigate files.

Continuing the support for pattern matching (and functional patterns by extension), this version introduced list pattern matching. You can do some wild things:

x switch
{
    [1, 2, 3] => //matches if x is int[] { 1, 2, 3 }
    [1, 2, 3, ..] => // matches if x is int[] { 1, 2, 3, 4, etc }
    [_, _, ..] => // matches if x has at least 2 items (_ is a throwaway identifier, the values are not kept)
    [.., 2] => // matches if x ends with 2
    [ >1, ..] => // matches if the first element of x is greater than 1
};

.NET 8 and C# 12

This is the first .NET release I did a blog about!

I remember this release being particularly good for ahead-of-time (AOT) compilation. .NET Core and its successor have focused on providing support for AOT compilation, which differs from the typical .NET compilation strategy of just-in-time (JIT) by compiling the entire application into native machine code, instead of compiling into IL and relying on a pre-installed .NET runtime on the target machine.

Compiling AOT obviously has more limitations as .NET is primarily designed for JIT, but it is an important feature to support the portability and, in some cases, performance requirements of many applications. After .NET 8 I became quite comfortable using AOT.

I also remember that the System.Text.Json library became very good in this release, as it was after .NET 8 that I replaced a lot of code that had used Newtonsoft with the new one.

This release also had a significant upgrade for the standard dependency injection framework with keyed services, letting you "key" a service implementation with a string:

builder.Services.AddKeyedScoped<IService, ServiceOne>("one");
builder.Services.AddKeyedScoped<IService, ServiceTwo>("two");

app.MapGet("/", ([FromKeyedService("two")] IService service) => ...);

I don't say that keyed services are significant because I use them, but because I now have to be cognizant of them whenever I implement some DI thing!

In the web development world, there were a lot of small things that, looking back on them now, I do not use. There are a couple neat things though.

SignalR, for one, got a new WithStatefulReconnect feature that a client connection can specify to automatically reconnect with the same client id when it's disconnected. This, however, relies on the same SignalR server staying alive (as it remembers the client id in its memory). This makes it more difficult to deploy SignalR servers unless you have access to some fancy deployment management tooling to keep servers with outstanding connections alive after new versions are deployed. I stopped using it after a bit.

Blazor got a bit of a major rejigger on this release, allowing us to blend server- and client- side Blazor. You can now have a server-rendered Blazor application that has some components that execute in WASM on the client, or you can even wriet a SPA that initially is server-rendered but then seamlessly transitions itself into a full WASM client once the client is fully downloaded, eliminating long startup times. They also introduced "Blazor Hybrid," allowing an electron-like experience with Blazor and MAUI, but MAUI is still bad.

Blazor applications are still 10MB by default, so very large, and the whole server-to-client transition thing is a bit of a workaround, but it's not uncommon for SPA libraries to support this and Blazor makes it quite easy.

C# 12 introduced two huge quality of life features: primary constructors and collection expressions.

When records were introduced in C# 9, they had "primary constructors":

public record Person(string Name, int Age);

These arguments are then compiled into public, read-only properties in the record. C# 12 made primary constructors available for classes, but they behave differently:

public class PersonService(PersonRepository repository, ILogger<PersonService> logger)
{

}

Notice the argument identifiers are lower-case: for classes, primary constructor arguments compile into private, read-only fields. In fact, the above is exactly equivalent to the following in previous versions:

public class PersonService
{
    private readonly PersonRepository repository;
    private readonly ILogger<PersonService> logger;

    public PersonService(PersonRepository repository, ILogger<PersonService> logger)
    {
        this.repository = repository;
        this.logger = logger;
    }
}

The primary constructor, then, significantly reduces the amount of things we need to write, particularly for services with many dependencies. One thing to note also is the primary constructor gets rid of the need to preface private fields with an underscore (_). In previous versions, most C# engineers would have preferred to write the following:

public class PersonService
{
    private readonly PersonRepository _repository;
    private readonly ILogger<PersonService> _logger;

    public PersonService(PersonRepository repository, ILogger<PersonService> logger)
    {
        _repository = repository;
        _logger = logger;
    }

    public Person? GetPerson(int id)
    {
        var personResult = _repository.GetPersonById(id);

        if (personResult is not PersonDao p)
        {
            _logger.LogInfo("Unable to find person {Id}", id);
            return null;
        }

        return new(p.Name, p.Age);
    }
}

However, it's now typically preferred to eschew the underscore entirely:

public class PersonService(PersonRepository repository, ILogger<PersonService> logger)
{
    public Person? GetPerson(int id)
    {
        var personResult = repository.GetPersonById(id);

        if (personResult is not PersonDao p)
        {
            logger.LogInfo("Unable to find person {Id}", id);
            return null;
        }

        return new(p.Name, p.Age);
    }
}

Collection expressions are the other great feature in this release. What this feature does is it makes the following a literal value in C#:

[1, 2, 3]

So, I can do the following:

public int[] GetInts() =>
    [1, 2, 3];

This is huge! Previously I'd need to do:

public int[] GetInts() =>
    new int[] {1, 2, 3};

With the spread operator this saves a lot of code:

public static T[] Concat<T>(this T[] first, T[] second) => [..first, ..second];

.NET 9 and C# 13

Having done such a significant amount of development to this point, .NET 9 slowed down a bit and, like .NET 7, had a huge focus on performance. Not that .NET 8 didn't have major performance improvements, but .NET 9 really did. When I wrote about this release I mostly complained about Microsoft pushing AI and Aspire, and I continue to hold this opinion.

There were some fun goodies though. I enjoy my LINQ and we got new CountBy and AggregateBy methods, which do what they sound like they do.

In this version the compiler started recommending the following when doing any kind of regex matching (example from MS's docs):

[GeneratedRegex(@"\b\w{5}\b")]
private static partial Regex FiveCharWordProperty { get; }

This uses source generators for regex, which does seem to be a whole lot faster in parsing regex. This is good news all around, I know plenty of applications with a little bit of regex here and there (mostly in a validation layer) that benefit from this.

A big improvement to async work came with await foreach and Task.WhenEach (example also from MS's docs):

await foreach (Task<string> t in Task.WhenEach(first, second, third))
{
    Console.WriteLine(t.Result);
}

The other great async improvement is the new System.Threading.Lock object, which provides a better interface and language integration for locking assets in async operations (instead of locking on an object, use the Lock API).

On the web side, ASP got app.MapStaticAssets(), which is a great builtin feature for shipping static assets, like a React SPA, with compression and etags built in.

Blazor got constructor injection for Razor components! Yes, before this you had to use property injection. No joke!

SignalR received a number of improvements. Polymorphic type support was big for me when it came out, but it also received a lot of updates to the telemetry it omits. This was really key for a lot of applications; having a complicated websockets app with insufficient telemetry was probably not great for a lot of folks.

Finally, ASP also introduced a HybridCache type that is capable of being both in-memory and distributed, handling a lot of the complexity of having those two tiers of cache. Very handy! I constantly insist that caches are difficult and hazardous; having a well-constructed first-party implementation with an appropriately restrictive API allows a wider adoption of safer caches.

On the C# side, there's a bunch of small things. With how impactful previous features were, it is good they slowed down a bit.

My favorite is being able to use any collection type for params:

public void DoThing(params IEnumerable<string> args) => ...

The language also introduced ^ as an operator you can use when indexing arrays, meaning "from the end":

var myArray = new int[5];
myArray[^1] = 12;

myArray then has 12 at the 1th position from the end. Interesting!

.NET 10 and C# 14

As of my writing this, .NET 10 and C# 14 have just been released a few days ago. Exciting!

This release is similar to the last in that the development has settled into a more manageable pace; it keeps the focus on broad performance improvements, incremental nice-to-haves, and rounding out C#'s support for modern development patterns.

One very nice addition is WebSocketStream, which essentially lets you treat a web sockets connection like a stream. I think I may be using this a lot!

My favorite nice-to-have is dotnet run program.cs, which completes the scripting capabilities in C#. With this feature, I do not need a SLN or CSPROJ in order to run a C# script. If I have a file sayhi.cs:

Console.WriteLine("Hello, World!");

With nothing else in the directory, I can dotnet run sayhi.cs and it runs. Super cool! This isn't just great for local scripting, I use this to generate my site!

My site has a single build.cs
Which I then run to generate this static site in a GitHub action

On the web development side, Blazor got a huge number of quality-of-life improvements that make developing with it - even a full WASM app - actually doable. I have a few Blazor projects and since upgrading to the .NET 10 previews it's been actually very nice. I won't go into any particular features but to say that if you've not enjoyed Blazor in the past, this one might make you more comfortable with it.

Another thing I'm eager to dig into is an overhaul of JSON Patch. PATCH is still a difficult thing to deal with, but there's a lot of applications for it and a new, robust, standards-compliant implementation looks good.

C# 14 has two features I'm very excited for that have been a long time in the making, maybe more than 10 years each.

First is the field keyword. For too long, we've had to manually create backing fields for properties:

public class MyClass
{
    private LazyService? _service;
    public LazyService Service =>
        _service ??= new(...);
}

This is unfortunate because auto properties compile to include their own scoped field anyway, but the programmer has never had access to this. Having to do the above exposes the backing field internally in the class. Should instance methods use Service or _service? The answer is definitely Service but that gets ignored sometimes. Now, we can use field instead to ensure the field is properly scoped to the property:

public class MyClass
{
    public LazyService Service =>
        field ??= new(...);
}

The next super-cool feature is "extension members." Not really sure where they got that name but that's what we're going with. For a very long time now, the C# language design team has dreamed of what they call "extension everything," extending the langauge to support that any kind of extensions could be made for types. Many languages, not just functional languages, support a much more robust extension system than C#. In fact, C#'s extensions are kind of hacky, essentially just giving a way to turn a static method call into a .-looking call. Not really extensions.

This version of C# lets you do the following wizardry, in a step towards "extension everything":

public record Person(string FirstName, string LastName, int Age);

public static class Extensions
{
    extension (Person)
    {
        public static string Species => "Homo Sapiens";
    }
}

What can I do with that?

Console.WriteLine(Person.Species); // Outputs "Homo Sapiens"

Whoa!! Super cool! It's an actual extension! But wait, there's more - if I name that type then I can put extensions into the type, similar to how I do with the this keyword today:

public static class Extensions
{
    extension (Person person)
    {
        public string FullName => $"{person.FirstName} {Person.LastName}";
    }
}

But ... notice it's a property and not a method! Wooooo!!! You can still do methods too.

Bringing Everything Together

So, given all of these, what do modern development practices look like today?

I've mentioned "functional" styles a fair amount. There's three features in particular that support functional-style development: records, pattern matching, and switch expressions. Switch expressions might seem an odd inclusion on this list, but let me demonstrate that it's necessary to include these to really be able to harness functional patterns in C#. Most functional languages rely on function definitions using pattern matching, like this Haskell Fibonacci function:

fib :: Integer -> Integer
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)

The C# equivalent now looks extremely similar:

public static int Fib(int n) => n switch {
    0 => 0,
    1 => 1,
    _ => Fib(n - 1) + Fib(n - 2)
};

And it's not just an aesthetic similarity at that. Functional patterns will assume functions can be written in this form, and the switch expression is the thing that allows C# to be able to adopt the patterns 1-to-1.

A more fun comparison could be made with list patterns, here the classic list sum function:

sum :: [Int] -> Int
sum [] = 0
sum (x:xs) = x + sum xs

Is also equivalent in C#:

public static int Sum(IEnumerable<int> l) => l switch {
    [] => 0,
    [var x, .. var xs] => x + Sum(xs),
};

To bring records into the fold, these are equivalent to data types that typically exist in functional languages; immutable types are the norm there, and their patterns require these kinds of structures. In particular, functional languages have algebraic data types (ADTs), which are discrete (closed) unions of other types. Take this example:

data Expression
    = Const Int
    | Add Expr Expr

What is this saying? There is a type Expression that can be either a Const (which will itself contain a single integer), or an Add (which will contain two Expression objects). This is modeled like so today with records in C#:

public abstract record Expression;
public record Const(int Value) : Expression;
public record Add(Expression Left, Expression Right) : Expression;

In this case, C# uses inheritance to signify that an Expression can be a Const or an Add, and this is a typical arrangement when mapping functional ADTs into C#. It is not, however, as precise a translation as the previous examples, because the Haskell structure is saying something kind of different. Where in my C# code I could create a record in some other part of the code and inherit Expression, I cannot do this in the Haskell example. Technically, we would say that the C# Expression is not closed. This presents a difficulty now in how we handle these in functions. Take this Haskell:

evaluate :: Expression -> Int
evaluate (Const n) = n
evaluate (Add a b) = evaluate a + evaluate b

Since Expression is closed in Haskell we will have no problems detected by the compiler, but C# needs an extra line:

public static int Evaluate(Expression e) => e switch {
    Const c => c.Value,
    Add(var l, var r) => Evaluate(l) + Evaluate(r),
    _ => throw new ArgumentException("Unknown expression")
};

This last line is required to complete the switch, since C# currently has no mechanism by which it can close the Expression type while allowing the inheritance by Const and Add. Nonetheless, this is the current proper translation of the functional functions, and is commonly seen in C# today. In fact, this is a great example of what I mentioned in the introduction to this article, that the langauge and framework also evolve to meet the demands of engineers who are using C# in new ways owing to external influence! In this case, the C# langauge team has been pursuing including ADTs fully into the language, a feature which they call "discriminated unions." The language design team's latest writings on the feature seem to indicate we should see initial support for these in the next version.

I've also written a bit about some functional patterns that have become particularly popular in the .NET world:

Eight Maxims

Wed, 17 Jan 2024 00:00:00 Z

Less is More Write less code, use fewer abstractions, be simple and concise. Complexity is easy, simplicity is hard.

Diff Your Commits Read your own code before you commit it, before you PR it, and after you merge it. Keep your code tidy and clear.

Document Why, not How I can (usually) figure out how to use your code. I can (usually) never figure out why you wrote it that way.

Collaborating over Coding Software Engineering is 9 parts collaboration and 1 part coding. Be approachable and helpful, reach out to others often.

Respect Legacy Code There is much wisdom in legacy code. Understand before judging, embrace before extinguishing.

Continuous Learning/Continuous Improvement Always learn, always try new things, always be open-minded.

Embrace Change Change is the only constant. Expect it, embrace it, and code for it.

Lift Everyone Up Share knowledge openly, celebrate accomplishments, support and mentor freely. You grow the most when you help others grow.

Roll Your Own End-to-End Encryption in Blazor WASM

Wed, 17 Apr 2024 00:00:00 Z

Just the other day I released my free planning poker app and there was one thing missing that I really wanted to have: end-to-end encryption. Sure, folks aren't going to be entering sensitive information into a planning poker tool, right? Well, probably not at least, right? Maybe it's more the principle of the matter, but I like that this application's server will never be in a position to even accidentally have that info.

Collaborative applications which support users interacting in real-time are pretty common, and Blazor WASM seems to want to support this use case pretty well. Microsoft's documentation walks you through a real-time chat application, and there's no end of tutorials showing how ot use it with SignalR. I'll probably add to that noise in future. One thing that's unfortunately absent is encryption.

Microsoft's own libraries are complicated and/or don't work on WASM (I'm still very unclear on the state of this - please comment if you know something), and I did not have any success with third-party libraries for Blazor. Do I really want to import a third party package for this though? These "small" Nuget packages are a dime a dozen around Blazor, and most of them are poorly maintained.

After some tinkering this was an area I decided to roll my own, and I'm glad I did. The SubtleCrypto API supported by most browsers (all the important ones) is actually quite a good encryption provider, and it's easy to set up JS interop with Blazor. This solution clocks in at 58 lines of JS and 11 lines of C#, so I can copy and paste between projects and tailor however I need for each one.

The Simple Case

In my case, I only need to encrypt a few short strings. A title and a name - that's it! In fact, the solution I need is going to be almost exactly the same as Excalidraw's encryption. In the case of my planning poker app, a user who creates a planning session will share their URL with their other participants, who can then join the session at that URL. That URL contains the session ID, and also the encryption key exactly as Excalidraw does.

A note on the key, then: it is passed in the URL in the hash, so a URL for my application looks like https://freeplanningpoker.io/session/2ae188d8#key=9aac428b962ead5a678b13f7. This is important, as the hash never makes it to the server.

Because the key is stored in the URL, I can access it from the page's JS, so I don't really need to know about it from C# except when I first generate it, so that I can redirect the user to the correct session page. This can simplify my C# API down quite a bit:

async Task<string> GetEncryptionKeyAsync();
async Task<string> EncryptAsync(string value);
async Task<string> DecryptAsync(string value);

I don't want to make things excessively complicated, especially for this case. Here then I want to write a bit of JavaScript with which these two methods can interop, and let the JavaScript worry about parsing the key from the URL and storing that.

Handling the Key

I encourage you to read the documentation on the SubtleCrypto API, it's quite straightforward. I'm going to use the AES-GCM algorithm, and we can use window.crypto.subtle.generateKey for this:

window.encryptionKey = await window.crypto.subtle.generateKey(
    { name: "AES-GCM", length: 128 }, // The algorithm
    true,                             // The key is extractable - important!
    ["encrypt", "decrypt"]            // This key is used for both directions
);

Note that I'm storing the key at window.encryptionKey - this is how I'm sorting the key on the JS side rather than the C# side. Not sure if putting it in window is ideal, I tend to avoid JavaScript when I can. insert skill issue meme here...

This creates a key object with a bunch of properties that SubtleCrypto uses, but we need to get a simple string to be able to pass through in our URL. For this purpose we can call exportKey and encode it with jwk:

return (await window.crypto.subtle.exportKey("jwk", window.encryptionKey)).k

We can wrap these up into a getEncryptionKey function and call this from C#, assuming we have injected IJSRuntime jsRuntime into our class:

public async Task<string> GetEncryptionKeyAsync() =>
    await jsRuntime.InvokeAsync<string>("getEncryptionKey") ?? /* Do something for null case */;

Getting the Key from the URL

When a new participant visits the link they were given by the session organizer, I need to get the key out of the URL and store it in window.encryptionKey, where my encrypt and decrypt functions are going to expect it. When the window loads, it's simple to get the key out of the hash:

const objectKey = window.location.hash.slice("#key=".length);

And while there's a fair amount of fluff here, it's also simple to reconstruct the key (this comes right from Excalidraw's implementation):

window.encryptionKey = await window.crypto.subtle.importKey(
    "jwk",
    {
        k: objectKey,
        alg: "A128GCM",                     // The algorithm
        ext: true,                          // Extractable
        key_ops: ["encrypt", "decrypt"],    // For both operations
        kty: "oct",                         // Not entirely sure...
    },
    { name: "AES-GCM", length: 128 },       // Still the same algorithm
    true,                                   // Still extractable
    ["encrypt", "decrypt"]                  // Still for both operations
);

And this can all be stuffed nicely into a DOMContentLoaded event handler.

Encrypting

Now the fun part! This is almost as simple as calling window.crypto.subtle.encrypt. As it happens, this encrypt function expects to deal with ArrayBuffer objects intead of strings. Fair enough, but we have to do a bit of wrapping around the whole deal. The first thing is to transform the string to an ArrayBuffer, and that's easy enough with TextEncoder:

const encodedValue = new TextEncoder().encode(stringValue);

We can use this value and the encryption key that we've saved in order to encrypt the value:

const encrypted = await window.crypto.subtle.encrypt(
    { name: "AES-GCM", iv: new Uint8Array(12) },        // Use empty IV
    window.encryptionKey,                               // The key we saved
    encodedValue
);

We now have another ArrayBuffer in encrypted, but we want to translate that into a string before we send it up. For this, I'm going to use a method described by David Myers to get a base64 string from the ArrayBuffer:

return window.btoa(String.fromCharCode.apply(null, new Uint8Array(encrypted)));

In all, this makes for a very tidy, very simple encrypt function.

And then another simple call from C#:

public async Task<string> EncryptAsync(string value) =>
    await jsRuntime.InvokeAsync<string>("encrypt", value) ?? /* Do something for null case */;

Decrypting

In this case now, I need to turn my base64 string back into an array buffer, decrypt it, and then turn the resulting ArrayBuffer back into a string to send up to the C#. The case of parsing the base64 string comes straight from David Myers again:

const bValue = window.atob(base64Value)
const buffer = new ArrayBuffer(bValue.length)
const bufferView = new Uint8Array(buffer)

for (let i = 0; i < bValue.length; i++) {
    bufferView[i] = bValue.charCodeAt(i)
}

That's a little unfortunate but 6 lines isn't so terrible. The buffer variable can now be passed into window.crypto.subtle.decrypt:

const decrypted = await window.crypto.subtle.decrypt(
    { name: "AES-GCM", iv: new Uint8Array(12) },        // Same algorithm, empty IV
    window.encryptionKey,                               // The key we saved
    buffer
);

And just as we initially used TextEncoder to get the string from C# into an ArrayBuffer, we'll use TextDecoder to go the opposite direction:

return new TextDecoder().decode(new Uint8Array(decrypted));

Instantiating a new Uint8Array turns the ArrayBuffer into an array, which is what TextDecoder wants.

This is an equally simple invocation from C#:

public async Task<string> DecryptAsync(string value) =>
    await jsRuntime.InvokeAsync<string>("decrypt", value) ?? /* Do something for null case */;

The More Complicated Cases

I'm really quite happy with the solution I have here for simple strings. More importantly, I'm a fan of small-code roll-your-own solutions because they give flexibility and customization. When I carry this code over to other projects, there might be (surely will be) different requirements that will require changes here. I want to demonstrate a couple of cases for which this code can easily be extended.

Encrypting Whole Objects

I struggle to think of a case where JSON serialization isn't the answer here. The good deal is that this can be achieved either in C# or JS.

If you're handling serialization in C#, you can leave the JS as-is and add all the logic to EncryptAsync and DecryptAsync:

using System.Text.Json;

public async Task<string> EncryptAsync<TValue>(TValue value)
{
    var serialized = JsonSerializer.Serialize(value);
    return await jsRuntime.InvokeAsync<string>("encrypt", serialized) ?? /* Do something for null case */;
}

public async Task<TValue> DecryptAsync<TValue>(string value)
{
    var serialized = await jsRuntime.InvokeAsync<string>("decrypt", value) ?? /* Do something for null case */;

    try
    {
        return await JsonSerializer.Deserialize<TValue>(serialized) ?? /* Do something for null case */;
    }
    catch
    {
        /* Do something for error case */
    }
}

This seems like the best case to me as I'm only chaging one part of the code, but depending on your use case you might want to do the serialization in JS. In this case you still need some more error handling in C#.

async function encrypt(value) {
    const stringValue = JSON.stringify(value);
    // Same as previous implementation
}

async function decrypt(value) {
    // Same as previous implementation

    return JSON.parse(decryptedStringValue);
}

IJSRuntime.InvokeAsync is generic, so we can cast right to the type we want, but we do need to handle errors. This does a JSON parse itself, so I'm not certain if this is more performant in most cases. That said, when we're doing the JSON serialization in JS, we don't need to modify our EncryptAsync method from the first pass.

public async Task<TValue> DecryptAsync<TValue>(string value)
{
    try
    {
        return await jsRuntime.InvokeAsync<TValue>("decrypt", value) ?? /* Do something for null case */;
    }
    catch
    {
        /* Do something for error case */
    }
}

Using an IV

In my simple case, I did not use an IV (initialization vector). Well, I should say that I used an IV with all zeroes. There are plenty of use cases for which you do want this though.

Luckily, SubtleCrypto does have an easy way to generate these, but the catch is that we want to create a different IV per encryption, so it's no good to create a static IV at the same time that we create the encryption key. Therefore, the IV needs to ride alongside the encrypted value. An easy way to achieve this would be to generate the IV before encryption, the encrypt the value with this IV and create the base64 string, then create an object holding both this value and the IV, then serialize that object and base64 encode it. This would then allow the deserialization to extract the IV before deserializing the value.

This is relatively straightforward to implement just in JS with window.crypto.getRandomValues, assuming you don't care about the IV in C#:

async function encrypt(value) {
    const iv = window.crypto.getRandomValues(new Uint8Array(12)); // Generate the IV
    const encrypted = window.btoa(String.fromCharCode.apply(null, new Uint8Array(
        await window.crypto.subtle.encrypt(
            { name: "AES-GCM", iv: iv }, // Use the IV
            window.encryptionKey,
            new TextEncoder().encode(value)
        )
    )));

    const values = {
        value: encrypted,
        iv: iv
    };

    return window.btoa(JSON.stringify(values));
}

async function decrypt(value) {
    const values = JSON.parse(window.atob(value)); // Reverse the process

    const bValue = window.atob(values.value)       // Get the encrypted value
    const buffer = new ArrayBuffer(bValue.length)
    const bufferView = new Uint8Array(buffer)

    for (let i = 0; i < bValue.length; i++) {
        bufferView[i] = bValue.charCodeAt(i)
    }
    
    return new TextDecoder().decode(new Uint8Array(
        await window.crypto.subtle.decrypt(
            { name: "AES-GCM", iv: values.iv },    // Use the new IV
            window.encryptionKey,
            buffer
        )
    ));
}

Farewell, Twin Cities Code Camp

Wed, 27 Mar 2024 00:00:00 Z

The Twin Cities Code Camp was the first programming conference I attended. I was just a freshman in high school, yet it was invigorating to get to share time with such a large group of folks in this industry. I attended for several years, then stopped as I moved to Iowa City for university, then I attended again until COVID. At its peak they had hundreds of attendees, the conference spanned the entire weekend, and they were organizing both spring and fall events. It was as though the entire software engineering industry in the Twin Cities would turn up for these events.

I learned an incredible amount at these conferences, and I give them a fair amount of credit for a lot of foundational ideas I hold. It was through a simple talk on the Sprache library that I ended up developing an interest in programming language theory. A simple talk on the then-new Windows Phone led me to write a lot of apps for the platform when it launched (famously Windows Phone now has a near-monopoly on mobile devices). It was my primary in-person source for learning about all the brand new emerging platforms, like GitHub!

Above all I enjoyed the time I got to spend with people there, and the folks I was introduced to at those events. But, all good things come to an end, and understandably the organizers of the conference have decided to discontinue it. They have my gratitude for having organized it for as long as they did.

So what now? Since COVID the feeling of the industry seems to have gone through an entire shift; longstanding conferences seem to be falling by the wayside. I'm going to be looking into starting a local meetup group in downtown Minneapolis. Get in touch with me if that sounds interesting. If you don't know what a "code camp" is - or if you are interested in organizing one - read the code camp manifesto.

Farewell, Twin Cities Code Camp!

Four Deeply-Ingrained C# Cliches

Sun, 04 Feb 2024 00:00:00 Z

There's a lot to love about C# and .NET, and there are some things that I don't love as much. Then there are four bad habbits that are so deeply ingrained they've become cliches within our codebases. Despite being obviously bad practices, their ubiquity seems to force them into our codebases in spite of our knowing better. When fighting the battle against complexity spirit demon it's easy to tire and allow these to slip through, perhaps justified by their disarming commonality.

It's not as though these subjects haven't been written about before, but that they are still practiced in spite of our best efforts to craft well-written code. I think it's useful to call these four out, and I want to propose alternative solutions with which I have found success in the past. Take my solutions with a grain of salt if you will, but do have a think about a better alternative yourself. These cliches have plagued too many codebases, creating spaghettiable (is that a word?) code and exposing too many bug vectors.

Interfaces

Interfaces are critically overused and misused. When I open a C# codebase I can guarantee I'll be treated to a whole host of interfaces that have exactly one implementation. Maybe they'll also be mocked in 200 different unit tests. I know I'll be treated to files that have classes so simple they haven't changed in five years, yet there's an interface right at the top for this class exposing the single public method. I know I'll find at least one (usually many) empty interfaces, and I'm always on the edge of my seat to find how much runtime reflection there is referencing the empty IModel.

What's going wrong here? I think we've forgotten the utility that interfaces have for us, yet interfaces are almost entirely necessary when developing even relatively simple production applications. Interfaces should be used to represent meaningful, coherent abstractions or capabilities relevant to your domain, and only used when there is a clear need for abstraction and/or polymorphism. One benefit gained by interfaces is to allow for multiple class implementations, facilitating class swapping, refactor testing, decorator pattern, or testing fakes. A (maybe better) benefit comes from the ability to assign multiple abstract concepts to a single concrete class, a la the Liskov principle.

Often though we find a one-interface-per-class rule in C# codebases, which eliminates or outright ignores a fair set of benefits. How can any of us claim our C# code is SOLID if we restrict ourselves away from even being able to use the Liskov principle? Sure, we don't need to adhere to Liskov religiously, but it's a good heuristic to help judge when to use an interface. If they're only mirroring the class structure, they're unnecessary. In these cases the code is better without these interfaces. If you do need to swap the class in a future refactor, you can create an interface then; it's dead weight to live in the code in the meantime. If you need to rely on the interface for mocks or fakes for unit tests, there's a good chance you're testing wrong and you should think about that too.

As an example, consider the following example which we have each encountered several thousand times:

public interface IMailChimpFacadeEmailService { ... }

public class MailChimpFacadeEmailService : IMailChimpFacadeEmailService { ... }

And there is almost never an IEmailService to be found! In this case we're usually better just deleting IMailChimpFacadeEmailService, but in the cases where the interface abstraction is necessary then replacing IMailChimpFacadeEmailService with IEmailService is entirely appropriate.

Empty interfaces should be deleted outright. But how will we reflect on our models without an empty IModel interface? Well, we probably shouldn't reflect on them, especially not after startup time. A lot of legitimate uses can be replaced by source generators here. These empty interfaces really grind my gears; they pollute my code and make me sad.

Abstractions

Overusing abstractions isn't a sin that's endemic to C# codebases specifically; it's across the entire industry. There's an interesting observation to be made that the incidence rate of overused abstractions tends to directly correspond with the number of software engineers involved in the team or company which owns that code - when we have time to kill at work, we tend to work. Abstractions are inherently complex, and we should strive to create simple code. The trouble is that abstractions can increase the flexibility of code, so they're also essential tools. The prevailing wisdom is that abstractions should be used sparingly and when necessary, and nobody (not even me) can define when it's necessary to abstract.

One common find in C# codebases is the passthrough layer - often the repository layer. Indeed, gone are the days of the Data Access Layer, we now have the Repository Layer, named after the pattern. This layer is often a passthrough to Entity Framework (EF, like most ORMs, already implements the repository pattern), to service layers which contact external services, or both. Whether it's a repository layer or not, passthrough layers are, I think universally agreed, incredibly annoying.

These layers add no value, they don't provide modularity, and often hinder scalability. Demonstrably, they add to development time. Kill these layers, even if they cause you to have to rethink your architectural approach.

Overall, I would say that an abstraction should clearly and demonstrably add distinct value or functionality. If you can't explain the value-add in one simple sentence, you've probably missed the mark. Architectural abstractions should specifically enhance modularity, scalability, or separation of concerns in a beneficial and articulable way. When refactoring (which ought be a constant practice), pay particular attention to abstractions. Celebrate when you can remove an abstraction, and always prefer clear, readable, imperative or procedural code over abstractions where the benefit of the latter is tenuous.

Exceptions

There's a lot that's been written about the ills of exceptions, and I think the negativity associated with them is maybe a bit exaggerated. One concept which I think is overlooked is the two-pronged error model; that there are some errors which need to be handled immediately, and then others which require an interruption to the control flow. The fact of the matter is that exceptions satisfy the use cases for the latter - the truly exceptional scenarios. We get into trouble however when we start relying on exceptions for all of our error handling.

The blame for this one definitely falls first on the C# language - it really only natively supports exceptions and there's still not a great way to create and consume a robust result or option monad (discriminated unions when). That said, one of the variants of the result pattern are better for the former sort of error. If you can encapsulate the result of your operations - particularly your data access and business operations - in such a way that the resulting value can't be accessed without a success check, then you'll be guarding the consuming code against abusing your method - you'll be making your contract more explicit.

Here's a simple result object for C# that uses the try pattern to do this:

public class Result<T>
{
    private bool isSuccess;
    private T? value;

    private Result(bool isSuccess, T? value)
    {
        _isSuccess = isSuccess
        _value = value;
    }

    public bool IsSuccess([NotNullWhen(true)]out T? result)
    {
        result = value;
        return isSuccess;
    }

    public static Result<T> Success(T value) => new(true, value);

    public static Result<T> Failure() => new(false, default);
}

I can't consume the value unless I make a check now:

var itemResult = GetItem();
if (!itemResult.IsSuccess(out var item))
{
    // handle error case
    return;
}

item.DoSomething();

The other bad aspect of exceptions comes when we start using them for situations that aren't even errors. It takes very little ingenuity - disappointingly little, even - to adapt exceptions as Malbolge's new goto. It isn't uncommon to see this in codebases, especially in library or utility logic, where some non-happy-path, yet non-error, scenario will throw, expecting to be caught somewhere. Sometimes this is code that merely lacks the most basic consideration (say, an email validator that throws if the string is empty instead of returning the validation error).

Other times this is entirely intentional, given very careful consideration in the entirely wrong direction. A great example is the IDataReader from Microsoft.SqlClient - if I want to access a column that doesn't exist (i.e. dataReader["some_column"]) it throws! Maybe that would made sense if they implemented a TryGetValue or ContainsColumn, but they did neither. The problem is so onerous because the contract for this object is that I am supposed to use try/catch here to poll for maybe-extant columns.

Unit Tests

I'll admit up front here that I'm a unit test hater. Always have been, and aggressively moreso every single time that I have to rewrite 20 unit tests because I just did a minor, non-behavior-altering refactor. Unit tests, as they're currently practiced, are in their ideal form a way to isolate a single method (or "unit") of code to ensure its contract is respected. There's a lot wrong with this.

First of all, our unit tests (almost) never live up to this. And it's not a question here of letting perfect be the enemy of good enough, they're so completely far away from this supposed ideal that I don't think it's fair to say they're even trying to aspire to it. We find unit tests heavily reliant on mocks (or fakes if we're lucky) where methods that don't even make sense to test in isolation are stood up in an almost Frankensteinian manner to prod it with ill-figured test scenarios.

On top of that, it's rare to find a test that even tests the properties of these methods appropriately. Often times, these tests don't even seem to know what the properties of the methods are they need to test for (a stark contrast to the much more focused discipline of property testing).

More troubling than either of those is that isolating methods is usually not the best way to test an application. The majority of methods in a business application do not need isolated testing. Rather, the system as a whole needs testing. This ideal of isolating methods to ensure that they adhere to the contracts they establish loses the forest for the trees; contracts can and ought be enforced in the code itself.

This all culminates in the extremely common deception that our codebases are robust because we have 90% code coverage with thousands of tests that are severely abusing mocks and checking arbitrary input and output values, or call patterns worse yet! This leads to our codebases being so fragile that even small changes require several unnecessary changes across dozens of useless unit tests. And it's entirely missing the point to suggest that this is because unit tests aren't being used correctly. The definition I provided earlier is inherently flawed and leads towards code being developed this way. Mocks or fakes are necessary to set up most classes for testing, and this practice leads to huge swathes of tests which are coupled to the implementations they test.

You'll be interested to learn then that this definition of a unit test that is so ubiquitous across our industry is entirely wrong. It's a great misunderstanding of the original intent of the "unit test". I'll link this brilliant talk by Ian Cooper as an in-depth explanation, but it suffices to say that the original intent of the "unit" test was that a "unit" represented some portion of the system which was behaviorally isolated. Perhaps we should be easy on ourselves for having mistaken that to mean "method" but after enduring how many unit test refactors, my charitability is thin.

Proper behavior tests are sorely underused, often not at all. It's quite typical that codebases will have maybe 95% unit testing and 5% proper behavior testing. This is backwards - when we're developing LOB or product software, the behavior is the most important, not the implementation. The tests need to be divorced from the implementation in order to properly isolate and test behavior.

Conclusion

tl;dr:

Interfaces are overused and have become almost like header files, mindlessley mirroring the class structures in our codebases. Consider the Liskov principle when creating interfaces, and best yet only use interfaces when they're helping you fulfil a pattern or a meaningful abstraction.
On the topic of abstractions, they tend to be used too frequently for cases which don't require abstraction. Sometimes it's difficult to see these abstractions enter the codebase over several PRs, so refactor your code frequently and aim to reduce abstractions every time. Prefer writing straightforward imperative or procedural code when possible.
Exceptions are often used for any kind of error handling and sometimes as extra special gotos. Only use exceptions for exceptional conditions that require you to break control flow, and consider using some form of the result pattern for error handling.
Unit tests are the work of the devil; they tend to lead to tests coupled to the implementations they test. Most applications can (and should) be entirely tested with behavioral (integration, e2e) tests instead of unit tests. When you do need to isolate discrete methods or classes for testing, consider whether patterns like property testing would better test the behavior of the method.

These cliches lead to code of a lower quality. Sometimes they just make it a bit more tedious to add a feature or change a bit of business logic. Other times, often over time, they create mangled codebases where making progress is like swimming through molasses. Though these are common practices in C# codebases, they don't need to be. I've suggested several alternatives that I've found to be very well worth considering, but you might have your own alternatives instead. What's most important is that we can stop letting these four cliches into our code, and start doing something more robust.

Thing I Made: FreePlanningPoker.io

Sat, 13 Apr 2024 00:00:00 Z

For the last week and a bit I've been working on FreePlanningPoker.io and I think it's ready enough to show off. It's a Blazor app that uses SignalR and Redis to maintain state across several connected clients.

I'm not an advocate for using points, and I'm certainly no fan of scrum but the fact of the matter is that a lot of us get stuck in plenty of card pointing meetings. There's various tools to facilitate this online, and none of them had all the features I wanted, or they weren't just right. Some of them incentivise practices that I think are detrimental. As many of us are prone to, I asked myself "How hard could it be to make one?"

Very easy. Actually, extremely easy. Sure there's probalby a minor race condition or two but let's not let perfect be the enemy of good enough here. (obligatory "/s")

The Code

If you've worked with SignalR before, this is very straightforward. If you haven't, you're going to be up-to-speed in a few sentences. SignalR is Microsoft's websockets implementation for .NET. It works reasonably well, especially with ASP and Blazor. You're able to define interfaces for the server methods and the notifications the client will receive for a hub, then you can use source generators on the client to hook it up to the server. Though it's not quiiiite as performant as I might appreciate at scale, it does the trick handily with a small amount of code.

SignalR can use Redis as a backplane, and that's advantageous because this app is a sort of state machine without much data, so Redis is a good way to keep state on the server. Alas, there doesn't seem to be a good PostgreSQL backplane for SignalR otherwise I probably would have used that, but Redis is fast and lightweight for this sort of application.

The code is open source, of course, and I've got much finer-grained descriptions of it in the readme.

Goals and Principles

My intial goal was met in about a week: I wanted a cheap app that solved my problem the way I wanted it solved. I'm very happy with that result! I want to keep this around though, I think it can be a benefit in a few ways. First, I think it's a good example of using the strongly-typed client source generators for SignalR in Blazor, which are to my knowledge not documented apart from one post from Kristoffer Strube. In fact I think it's generally a good sample Blazor application.

I think an application like this should be very minimal - not just in the UI and user experience, but it should take almost nothing to run the application. Sure, I chose Blazor which isn't exactly the most lightweight SPA framework, but overall the footprint is quite light. The server costs very little to run, being only an ASP server with a single SignalR hub and a Redis instance. It's got a low memory footprint and doesn't consume a lot of computing resources. Apart from resulting in a fast and maintainable application, this is a greener way to approach software.

There should also be more software which encourages users to make it their own. I don't mean from a UI styling perspective, but from the code itself. If you want to take my code and deploy it yourself for you or your team to use on your own infrastructure, that should be easy, well-documented, and the code should provide all the environment variables you might need to tweak your setup. This application only really needs a Redis connection string but you get the idea. If you want to fork the code and make modifications on top of that, you should be well-supported by the code and its author in doing so.

I chose the Unlicense for this project for all those reasons too - more free code is more good.

Finally, I think users' data should be respected. How much sensitive information is being transacted over a planning poker app? I hope none. Nonetheless, it's your information and that should be respected. This application deletes all the data from each session as soon as the session is left by the final participant. I don't log or retain any information from the sessions, and I'm going to be adding client-side encryption to completely remove that as an issue.

Soliciting Feedback

If you use this, please do let me know if there's anything to improve, or if there's something you like about it. You can comment or webmention on this post, you can open an issue on the repository or you can connect with me through several other avenues.

If you'd like to add something to the application, I'm happy to entertain PRs! I'd like to add some fun features to help distract the participants that they're in a planning meeting; that might do some more good for the world.

Giscus Is Awesome

Thu, 14 Sep 2023 00:00:00 Z

giscus.app is really awesome!

Last week I posted for the first time in six years and I figured I wanted to see about adding comments to this site. A Google search got me to Giscus really quick, and I was able to wire it up in just ten minutes. The thing that's still blowing my mind is that it works.

Behind the scenes, it syncs up with the GitHub Discussions tab on the repo that hosts this website, and it matches a discussion to a page based on the the page's title.

When somebody adds the first comment to a page, it creates a corresponding discussion thread for the page. When a page loads, it checks to see if there is a corresponding discussion and it loads the conversations from that discussion thread.

I can do comment moderation and whatnot on GitHub discussions, and if somebody stumbles upon my website on GitHub they can see the conversation right there. If the tool stops working, the conversations still exist in GitHub, living right alongside the source for this site.

And - I can't emphasis this enough - it just works. I see so few tools that just work and this one does.

Check it out!

Guerrila DevEx Testing

Fri, 11 Oct 2024 00:00:00 Z

I first became aware of hallway usability testing (also referred to as "guerrila" testing) at the beginning of my career when I was encouraged to read The Joel Test, a now-famous article on what makes a good team. It's a brilliant article, and at more than two decades on I think it's still foundational as to how I think about the health of a team or a project. It contains the following description of hallway usability test:

A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.

Here, the phrase "use the code you just wrote" means you're going to execute the code and ask the participant to use your actual software, navigating your UI and simulating a user. I love this phrasing though because that isn't where my mind went the first time I read that sentence. No, I instead thought it might be more interesting to yoink other engineers from the hallway and ask them to navigate my code. Not a code review, not some full architectural scrutiny, but a real-world test of whether my code actually works as code.

Nowadays we have the term "developer experience" to refer to this concept of the developer-usability of a piece of code or a development environment or the like, though this is a notoriously hand-wavey term. Especially considering my code's quality, I'm not sure there's ever going to be a great objective way to measure that. At the broad level, different groups of engineers are going to be biased in different ways towards one pattern or practice or another. Different styles are better suited for different problems or types of applications, and even throughout the lifetime of a codebase we might find that changing styles might work better with a changing environment around the codebase.

And yet it isn't nonsensical to talk about code quality or developer experience more broadly. While disagreeing on specifics, surely we can all say we've worked on codebases that were easy to modify or extend, as well as codebases that aren't. A few colleagues on the fringe might suggest - intending to argue that "developer experience" is either meaningless or not worth attention - that instead of good or bad codebases there's just varying levels of bad. Beware of this rhetorical play: it's still an admission that there do exist codebases of a higher quality than others! Developer experience is a finicky thing to capture but worth our consideration.

So that raises the question as to how to consider developer experience. Can I measure it? Well, I can't quantify it - please comment below if you can! However, when I write bad code, or at least code which my colleagues consider bad, they're usually quite ready to let me know. Giddy, sometimes, to point out my faults. Being fair, I do have a blog which allows me to anonymously gripe to the world - literally tens of people on a good week - about their bad code.

This is how I gauge developer experience, and it's the basis for a lot of practices I've developed over the years. This is a terribly imprecise art, considering not just the subjectivity of the matter but also the large number of variables at play that can alter how we think about quality in code. That said, I defy you to come up with a better method than this: watch your colleagues "use" your code. Sit them down and watch them debug an issue or add a feature.

This is the hallway devex test, and as I mentioned it is higher-level than a code review and lower-level that some fine-grained walk through a whole codebase. Code reviews are done frequently and focus narrowly on the bit of code being added or removed, while the latter exercise is so effortsome that I've only ever heard of such practice in theory. This test is the in-between: given a functioning codebase (one that's in production), how well does it align with our quality attribute requirements?

Running this test is very easy. If your code is in production then you surely have a set of bugs or simple features you've identified which could be worked on. From your firm's hallway (or some sort of Slack roulette if you're remote) yoink a colleague and promise to give them coffee and/or doughnuts in 30 minutes (or an hour or whatever amount of time makes sense) under one condition: they have to work on a card you give them on your codebase. For what it's worth, the promise of a doughnut is typically unsatisfactory over Slack; in this case you can just promise to subscribe to your colleague's book club.

The crucial step is to watch your colleague. Look for where the pain points are, look for the assumptions they make, and look at the expression on their face. If you ask them how they feel or what they think, you'll get absolutely nowhere. Humans are funny beings; we're bad at knowing how we feel and sometimes we're bad at knowing what we think. There's frequently a disconnect between a person's actions, beliefs, and perceptions. Crucially, there's often a disconnect about what we think or say about our own actions, beliefs, and perceptions, so you can't trust me to describe myself to you. Further, you can't directly observe my beliefs or perceptions. However, you can directly observe my actions and you don't need to take my word for it!

How I go about doing software engineering on your code is what reveals my developer experience with your code. How your other colleague works reveals their developer experience with your code. In a way this cuts down to exactly what "developer experience" is: you're observing the experience of your colleagues doing developering with your code. As Joel said of usability testing, just 5 hallway passersby will teach you 95% of what you need to know - I find this largely true for devex as well. This test is easy enough to run periodically, and from there you can develop a feedback loop of collecting pain points, addressing them, letting the codebase evolve a bit further, and repeat.

What specifically you're looking to address is tough to universally quantify - the types of experiences working on an embedded system will be quite different from a CRUD webpage or a video game. You might even find your colleagues reveal whole new categories of pain points you hadn't previously considered! I don't think about this too much beforehand, but I can think of a few high-level questions I typically look to resolve:

Was it easy for my colleague to understand what the codebase is doing and how? Do they understand the architecture, patterns, conventions, and structures in the code?
Was my colleague ultimately able to identify where to make the change or insertion?
Did my colleague feel or become confident with the codebase? Did their confusion increase or decrease over time? Did unexpected side effects occur with their changes?
What role did the development environment play - did they have difficulty working with the code in their IDE? More fundamentally, were they able to get it building right away?
Where did they become confused? Where did they become frustrated? Where did they have to reach out to me? Can I identify styles, patterns, or such which they dislike?

In conclusion, I submit that "developer experience" is a useful subject for consideration, and in spite of its subjective nature I feel that I can say, definitively, that developer experience can be observed by observing the experience of developers. The hallway or "guerrila" test works for me to gauge this quality, and I'm sure it can work for you as well.

How do you Enforce Architectural Decisions?

Tue, 20 Jan 2026 00:00:00 Z

"My best-architected projects were the ones that never had an Architect." I was recently told this by a former project manager who retired feeling unimpressed by colleagues over-eager to adopt the title architect. I can't blame him, on this blog I've documented a number of observations I've made of flawed attitudes towards architecture. That is, attitudes about the nature of software architecture and the work that should be expected of architects, not ideas about this-or-that architectural pattern. Indeed, it seems intuitive that a project could be seriously derailed by an architect with a particularly nonproductive attitude about their job.

This is the sort of architect my project manager friend seems to have found, and he told me how he stopped hiring architects: During interviews, he started asking one question to which he never received a satisfactory answer. With that, he refused to hire any more architects, and his projects were all the better for it. I happen to be very comfortable, though, professionally describing myself with the architect title, so I'm curious to answer this question. How do you enforce architectural decisions?

Imagine you've applied for an architectural position and you're asked this - what is your answer? It's really not straightforward; the answer depends a great deal on your conception of the practice of architecture. I can imagine a lot of potential answers, and it's not hard to figure how one could go years without hearing a satisfactory one.

For me, this is a trick question; at least, I'll answer it like it's a trick question. It's certainly a leading question by my reading, supposing that it's in the architects' nature to make and enforce decisions. This may be so, but I'd venture that most folks in our industry are going to adopt the model of the architect-as-superior; that architectural decisions are apart from and supercede engineering decisions. This is the trap.

I think it sometimes does our industry a disservice that we lean on a construction analogy to describe our roles. In building a house, the to-be homeowner consults with an architect over floor plans and renders which result in a final blueprint passed to builders who assemble the final, tangible house. It's right to forgive anyone who mistakenly thinks that the analogy with our industry maps the homeowners to the business, builders to software engineers, and the architect to software architects. This is completely wrong though, and that this mistake has tangible implications for millions of software professionals and the projects we're meant to steward is a serious fault. In fact, the analogy would be to map the homeowner to everyone with a stake in the software's quality, the building architect to the software engineer, and the house builder to our compilers.

Except for a very small number of cases, software is not developed by making a complete, abstract definition of the final product before writing all the code to fulfil the definition. Rather, software tends to be constantly created and recreated in an interactive process; typing is significantly less resource-intensive than constantly erecting and destroying walls and rooms and roofs. We iterate a bit, deploy that iteration for all stakeholders (clients/users are stakeholders too), they give us feedback, and we rejigger what we need for that new feedback. What part of the house-building process does that sound like? That's right, it's the consultation between the homeowner and the architect. Remember that from the analogy, the homeowner maps to the stakeholders, and the house architect maps to the software engineers. The code is the blueprint we're all consulting on, and it's trivially cheap to ask the house builders to build a brand new version of our blueprints - after all, the house builder maps to the compiler (which takes seconds to produce the built software).

I will reiterate - it's a grave mistake to think that the software engineering process is the analogy-equivalent of the house builders building the blueprint. If you will, close your eyes and imagine a small office with a house architect and a couple commissioning her to build them a new home. There's a blueprint on the desk and a big, red button that says "BUILD." The couple asks the house architect to make changes to the blueprint to accommodate some requirement they have, and the architect obliges and updates the blueprint. Let's imagine this office sits at the head of a long road, which we can look down through a window in the office. The road is lined with houses, each next house a bit different from the first, with addresses like "v1.0.101", "v1.0.102", and so on. The house architect hits her BUILD button, and within seconds a new house appears at the end of the row of houses - address "v1.0.103." This process repeats ad infinitum. This is what software engineering is, the process inside the office. Let's not let ourselves be distracted by any misdirections from the imperfection of the construction analogy.

So where does that leave Mr. Software Architect? It might look like we've squeezed him out of the picture, but that's not true. Well, not entirely; I want to very intentionally exclude anyone who draws diagrams before any code is written and berates colleagues for not implementing his diagrams. Software architecture is not brought into existence by diagrams or charts or whitepapers or anything like that; these are just tools which can help us to explain and teach an architecture after it comes into existence. Software architecture is made by the very process of writing the software; the architecture emerges from the code that is actually written (and for aggregate systems it is begat by the components that are deployed and the interactions that emerge between them). Software architecture is an artifact of the engineering process - remember the office before the row of houses.

If we're trying to think about software architecture as something abstract, separate from and superior to the code or the engineering process, we're completely missing the fundamentals. Software architecture might be better understood as subordinate to the engineering process, but really it's best understood as being one of the essential components of what goes into doing software engineering. Every software engineer is making architectural decisions, every day. Every software engineer is, to a greater or lesser extent, a software architect. Not because they've developed a separate skill, but because software architecture is an aspect of doing software engineering.

Where that leaves Mr. Software Architect then is that he is a sofware engineer all the same as his colleagues on his team. If we have somebody with the architect title, what we ideally have is an engineer who has a particularly good understanding of the architecture aspect of engineering, somebody who strengthens the whole team by setting a quality example of thinking and engineering architecturally. Other folks on the team will have other strengths like design or security or performance, which are all themselves aspects of software engineering. How does our colleague who's good at performance lead the team to deliver a performant product? This seems like a silly question because the answer is obvious: if they're a professional then they'll act professionally with the team on performance. All the same with our colleague who's good at architecture.

There's no doubt that architectural decisions come to be made during the engineering process, and likewise there's no doubt that such decisions should inform future development. Nonetheless, I object to the question how do you enforce architectural decisions because this question conjures an image of an individual architect making decisions and authoritatively holding them over an engineering team. If instead we want to have a conversation about how an architecturally-inclined member of the team can best contribute their skills, helping to find and bolster good architecture that comes out of the engineering process, then we can talk!

How I am Super Techy in my Gherkin

Sat, 08 Mar 2025 00:00:00 Z

A few weeks ago I wrote about how it's okay to be a bit techy in your Gherkin, by which I mean that it's okay to include technical sorts of language in your Gherkin. Gherkin is designed to abstract away the specific technical details of automated tests to make it easy for engineers to collaborate with non-engineer stakeholders over automated tests for a codebase. The general guideline goes that Gherkin should be written in a "business-friendly" manner, but this is actually not quite right; rather, our Gherkin should be written in a "stakeholder-friendly" manner.

If your stakeholders include very-technically-illiterate folks, then you'll define your Gherkin steps to account for that. If, on the other hand, your stakeholders all understand what it means to send an HTTP request into the service you're testing, the following is (almost certainly) going to be easier to collaborate over:

Scenario: Create a resource when myFlag is on
    Given the feature flag "myFlag" is ON
    When I send a POST request to "/my/resource" with the body
        """
        {
            "name": "Bob",
            "age": 42
        }
        """
    Then the response status code should be 200

Since I wrote that last blog post, I've had a lot of successful collaboration over a couple projects at work, and I want to share some of my findings here. Ultimately, I'm impressed by the very tight testing loop which this "low-level" form of Gherkin supports for my team's backend services. If we observe a bug or find one while doing exploratory testing, then we'll already have the request body - this Gherkin format makes it very easy for anyone - not just engineers - to write a test which can be immediately executed on the codebase.

So in a matter of minutes I have the issue reproducible on my local machine, and I can then reach out to other stakeholders to confirm the expected behavior. Again, everyone understands what an HTTP request is, so I can send them the Gherkin, and they can respond to me with updated, valid Gherkin in no time. This all shows the value in ensuring that you set up your Gherkin so that it works in the environment it's supposed to, and that you're using it to actually facilitate collaboration.

Steps to Arrange Data

In my environment (this could be entirely different for you) anyone who understands what an HTTP request is knows, generally, how SQL or Redis or the like work. Setting up data for the test can be extremely simple that way. It's even easier for a document database. Take the following:

Given the "items" table in the Postgres database has the following records:
    | id | name  | age |
    | 1  | 'Bob' | 42  |
    | 2  | 'Sue' | 50  |
And the "some" key in the Redis cache has the value "hello"
And the external endpoint "/my/service/other/resource/1" will respond with status code 200 and body
    """
    {
        "id": 1,
        "favorites": [ "Bob", "Sue" ]
    }
    """

Database tables are set up with - surprise - tables, Redis values are set directly, and responses from external services can have their bodies written out in full. Redis is an interesting case actually, since it can have lists and hashes and so on. It's relatively simple to add those:

Given the "some" key in the Redis cache has the list [ 1, 2, 3 ]
And the "other" key in the Redis cache has the hash
    | name   | value |
    | first  | 1     |
    | second | 2     |

To reiterate, this works so long as all collaborators understand those sorts of systems. When they do, this straightforward approach gives the path of least resistance in moving from an observed scenario into the Gherkin language which allows collaborating over what the business requirements (outputs from the service) are in these conditions. Speaking of which:

Steps to Test Responses

One key point is that a service (an HTTP API, in this case) does not just output responses to requests. Yes, that's the primary thing to test, but you might want to ensure that events are enqueued in a Kafka or that specific logs are output in certain scenarios to drive alerting, or the like.

These extra scenarios get a bit tricky and probably require that your tests have some way to read all of these output channels. That solution might vary wildly for your project, but the Gherkin steps can be very simple. Remember to keep the focus on what kinds of steps are both most understandable to all your stakeholders and produce the least amount of friction when creating Gherkin scenarios from real-world observed scenarios.

Then an error level log is written containing "SuperImportantService failed to respond"
And the following event was sent to the Kafka topic "topic"
    """
    {
        "id": 1,
        "name": "Bob"
    }
    """

Testing the actual response from the service is just as easy, though in this case we might want to validate more than just the body - response status code is quite meaningful over HTTP:

Then the response status code is 200
And the response body is
    """
    {
        "id": 1,
        "name": "Bob",
        "favorites": [ "Bob", "Sue" ]
    }
    """

Now, there is a catch here testing the bodies of both the event and the API response: how is the code written to match the bodies? If I write a multi-line string in my test but the server actually responds with a single-line JSON, I can't do a string compare. Even if we could solve perfectly for whitespace (I don't recommend trying) property rearranging is an issue. But then I also don't necessarily want to test every property; maybe for some tests I just want to make sure one property is coming back altered.

What's wanted then is to test that the response JSON is a superset (for lack of a better word - is there something else we call this?) of what we expect. Luckily, this is quite easy to define:

bool IsJsonSuperset(JsonElement targetJson, JsonElement expectedJson) =>
    expectedJson.ValueKind switch
    {
        JsonValueKind.Object =>
            expectedJson.EnumerateObject().All(expectedProperty =>
                targetJson.TryGetProperty(expectedProperty.Name, out var targetValue)
                && IsJsonSuperset(targetValue, expectedProperty.Value)
            ),
        JsonValueKind.Array =>
            targetJson.GetArrayLength() >= expectedJson.GetArrayLength()
            && expectedJson.EnumerateArray().All(expectedArrayElement =>
                targetJson.EnumerateArray().Any(targetArrayElement => IsJsonSuperset(targetArrayElement, expectedArrayElement))
            ),
        _ => targetJson.GetRawText() == expectedJson.GetRawText()
    }

Accounting for HTTP Headers

Ah, headers are just a table - surely I could just include them along with the body when I send the request:

When I send a POST request to "/my/resource" with header and body
    | name          | value            |
    | Content-Type  | application/json |
    | Authorization | Bearer hello     |

    """
    {
        "name": "Bob",
        "age": 42
    }
    """

Alas, Gherkin does not support such an idea. In a traditional setup, we'd break this down into multiple steps:

Given the request has header "Content-Type" with value "application/json"
And the request has header "Authorization" with value "Bearer hello"
When I send the request ...

But barrier! My goal is to make it as easy and seamless as possible to move from scenarios we observe in test and prod into our Gherkin - and I want anyone to be able to do that. We need more simple.

How this shakes out in any particular circumstance is going to be affected by tooling and preferences and the usual factors, but what I've found works is to adopt the frontmatter concept to the JSON bodies. With my specific projects (and I can't stress enough that this may be drastically different depending on a lot of factors) the easiest thing has been to include the headers as YAML frontmatter atop the JSON in the body:

When I send a POST request to "/my/resource" with the body
    """
    ---
      headers:
        - Content-Type: application/json
          Authorization: Bearer hello
    ---
    {
        "name": "Bob",
        "age": 42
    }
    """

It's easy then to parse out --- and get the information from the YAML. That "frontmatter" could just as easily be JSON, or you could even abandon the pseudo-frontmatter idea and write the whole body using http file syntax.

How to Write an App in One Day

Sat, 25 Oct 2025 00:00:00 Z

I've been pretty short on time in the last few months; my recent rate of output on this blog might attest to that. Between learning the ropes at a new role at work and a large number of personal engagements, I haven't had a lot of time to get a lot of coding in.

Nonetheless, I've gotten a number of projects up and going. I previously wrote about what makes a successful personal project, but I'm thinking now I left a lot out: what is there to be done in the face of serious time constraints? Or, potentially more interesting, are these time constraints a benefit? I think I can give some good thoughts on the former question, at least.

I've found that you can get a lot done in a single day. You can even get a lot done in a few hours in the morning in a cafe; I'm writing this now from a quiet cafe on a rainy day (this is a productivity cheat code, at least for me). Building something meaningful in this time frame just reduces down to being organized: set a reasonable goal, know what that goal is, implement realistic steps to getting it done, pay for your coffee, and you're now the proud owner of a whatever you just made!

I guess this is just a more practical elaboration on my personal projects article...

Rome wasn't built in a day but "todo" apps can be built in a couple hours

Most of the projects I have in incomplete states are there not because I didn't set a realistic goal for something to build but because I didn't understand well-enough what I wanted to build. I don't think it's reasonable to expect specific definitions to rise out of an active coding session; if you don't know what you want to code then writing code is putting the cart before the horse.

This is maybe homework to do before sitting down then, but the advantage is it can be done over several days during the free moments we get for thinking. It pays to be more thorough in this thinking than less - what are all the models I need to handle, what user actions are there, what's the service architecture going to look like, how do the user actions inform the code architecture, what technologies do I know that will enable me to implement this, what am I lacking in knowledge that's going to be a blocker?

To boot, these questions don't just need answers but reasonable answers. You're not going to build your cool app if you start in a language you've never seen before. It's perfectly reasonable to have a project in a brand new language, but this article is focusing on trying to maximize how much thing can be built in a short time.

If you know your service architecture, user actions, and models pretty well then you're 80% of the way there, and that's usually where I start. It's good to know your knowns and unknowns, and it's good to know the specific technologies you'll need to employ to get the thing off the ground. To be very thorough, it's great to think through all the extra tidbits that tempt us on the way: does the project really need a full OTEL dashboard, do I really need a revproxy, does a side project really need automated tests, and so on (the answer should probably be "no" for most of these).

Just use the thing you know, and use it well

To reiterate, this is from the mindset of wanting to focus on delivering the thing; I don't want to be taken as advocating for never learning new bits. I have a grab bag of technologies, providers, and libraries that I reach for when I need something that isn't going to be a blocker. When I need a database I just use Postgres, I almost exclusively use C# or Go, I do exclusively use GitHub and Railway to build and deploy. I like Clerk for auth, Stripe for payments, and NATS as a message bus.

You'll have a different grab bag than I do, to be sure. The point is that these are things that I know, I don't have to spend time wondering how to do this or that as I'm building an application. If I want to deliver the thing, I want to minimize the number of things I don't know, and focus my energy on tackling those things. If you think about it, the amount of time required to deliver a project is a sum of two numbers: the time required to get through the setup for all the things I know, and the time required to think through, experiment, and debug the things I don't know. The size and complexity of your app influence the first number, and the number of unknowns influences the second number; both potentially exponentially.

What's just as important as choosing the right tools is using them correctly. If you're reading my blog, you should know the software development loop: 1. make it work, 2. make it work right, 3. make it pretty. This loop should be as tight as possible, you should not skip step 3, and you should start with the most difficult and necessary things.

Let me elaborate on that. When we start a project, we're probably going to have some kind of startup template for whatever language/library we've chosen. Maybe too frequently, we see that run locally and then we dive into sketching out the UI or server calls. This is not just disrespecting the loop, it's also not tackling the hardest problem first! We've done step 1, but no doubt the standard app templates aren't quite right for you; you'll need to change some low-level things and clean up the code to your preference. Do that first!

Then, what's the next hardest thing in setup? Yes, setup is generally easy, but what's the next hardest thing? I find most bugs come from deploying the app on the cloud if you're doing a website, or setting up the installer if it's a desktop app, or whatever deployment deal for whatever other thing. If I'm setting up a web app my next step is to get a GitHub repo and deploy it on Railway, and I'll even just start with the server at that. Invariably there's something that's a different version or been improved since I did this last month that needs to be ironed out. Get it working right, then go over it all again to make sure it's pretty. Keep the work area clean and you can go much faster.

Then when I add my Postgres or Clerk auth or whatever else I need to set up, I can see it work locally and right away I'll validate that it works when deployed. Sooner than later I'll also deploy a production version (unless that deployment is the production version; that's a great time-saver). Particularly if you have third-party auth, separate environments can be a little hairy. After we're set up, what are the unknowns we've identified? These are the next hardest things.

By always tackling the difficult things up front you're always keeping your time use most efficient. Practicing the development loop tightly means that at every step you're not just marching towards the finish line, but you're resolving error states, edge cases, code quality, and keeping the UI looking right. You're not deferring little things to the end of the project because those little things add up (note here: this is how you avoid the 80/20 trap). The development loop means we're not exploding our backlog, our app will be done when we're finished working through our user actions.

And don't forget to pay for your coffee

I like working from the cafe near my house, that's a productive spot for me. Environment can be such a huge factor in our ability to function, not just at work or when writing an app but always. If you understand the environment that makes you happy you'll find it's much easier to sort through a project.

I suppose "do what makes you happy" is just obvious life advice though?

Intuiting Jevon's Paradox

Fri, 13 Dec 2024 00:00:00 Z

I wrote in a recent post about Jevon's Paradox, and I received some feedback on that; more than I usually do. As always, I'm grateful when you all contact me though continue to encourage use of the excellent comment feature or, if you want to be one of the cool kids, webmentions, both of which I recently included on this blog! Anywho, I want to expand on this topic.

Jevon's Paradox is one of those very unintuitive but easily observable facts of life. It was originally descried by economists (so I'm lead to believe) but has applications across the board, economics being the strage confluence of so many different social studies as it is. There are many different staements of the paradox, but I like this one:

As the efficiency of consuming a resource is increased, the net consumption of that resource increases.

That's a bit of a compact definition, so it's worth giving a more plain-language explanation. It's not a paradox in the logical sense (i.e. Russell's Paradox), but in that its implication subverts our expectations. By way of example, I can explain it using the well-known concept of induced demand in urban planning. Suppose we live in a city with a highway going through it and we've been tasked by the mayor with finding a solution to that it always seems to be in a traffic jam. The most straightforward and intuitive response would be to add more lanes to the highway, right?

In fact, if you clicked that link describing induced demand, you'd know that adding more lanes is not the right answer, as it will actually cause an even greater demand on the highway than there was before. What's more, this is not a linear demand: if the current demand for the highway is x cars per lane per hour, after adding more lanes it's actually xy cars per lane per hour, where y is some constant greater than one such that the resulting demand is even more intensive after adding more lanes. This is the "paradox" bit - it's entirely unintuitive that increasing the units of highway resource (lanes in this case) should result in an even higher demand per resource unit. We expect demand to remain the same, and that increasing the efficiency of use will resolve issues in congestion caused by that use. Because of Jevon's Paradox, however, not only have we failed to resolve the congestion issue but we've even made it worse.

What of these implications then? There are a lot of conclusions to draw. First, and I think most importantly, we can say that when there are congestion issues with the consumption of a resource, making that consumption easier will exacerbate the problem and should not be considered as a solution. What solutions there are to the problem are probably context-specific, but if it involves increasing consumption efficiency then that's not one.

The next conclusion we can draw is that the inverse of Jevon's is probably true:

As the efficiency of consuming a resource is decreased, the net consumption of that resource decreases.

If I remove lanes from the highway, then of course there won't be as many cars on the highway. To extend this example, I might ask if removing lanes has resolved any issues though? There is some level of need to consume road resources, so unless I can find the root cause of that need I'm probably just shifting the problem around by changing the highways. This is another important conclusion from Jevon's, albeit more indirect than the others: address problems at their root. It seems that the five whys end up in a lot of places!

Jevon's Paradox is so called because it's unintuitive, and indeed without a proper intuition for its implications you might find yourself amid the problem it describes. Certainly we're not all urban planners dictating the adding or removing of highway lanes, but indeed it has plenty of applications for us software engineers. Any software is necessarily a resource user - there's computational, storage, and networking resources which are easy for us to understand. The ease of consuming each has only increased over time, but is our software any better? In a lot of ways software has become much less efficient, bloated, worse for the users, and slower. Jonathan Blow has documented this quite well I think. Are we Jevon's-ing ourselves and our software?

Maybe. I'm certain a collective failure to intuit Jevon's is not the only problem with our industy, and I'm doubtful that it might even be a major factor. However, I find it quite compelling to think in terms of resource consumption, and Jevon's is one of the prime gotchas for the ways in which software could be said to be a resource consumer.

How to intuit it? The answer is common to many problems in software engineering: I think you need to think about it really hard. This is also my answer when colleagues ask how I know my code works. Being infallible as I am, this of course proves to be correct every time and I have never had an instance of being wrong on this count! To be serious though, this is one to keep in the back of the mind. When approaching architectural questions, ask whether you're solving a resource problem in the wrong way, or if you're creating a potential one. Consider whether Jevon's applies, and if you're diagnosing an existing resource problem consider whether it had applied.

It's Better to be Consistently Incorrect than Inconsistently Correct

Thu, 15 Feb 2024 00:00:00 Z

Anyone who has worked with me for any period of time is probably tired of hearing me say this. "It's better to be consistently incorrect than inconsistently correct." I don't say this to mean that being inconsistent is a good thing; rather, I say this whenever I need to underscore the importance of consistency, almost always with respect to a piece of code. Codebases with established conventions - which consistently adhere to these conventions - are, in my opinion and experience, easier to comprehend and maintain.

If I'm working in an app that uses the pipeline pattern to chain behaviors everywhere but then throws a random decorator pattern at me for one slice, I'm going to be confused. Maybe scared. Probably both. Either way, I'm paying a lot of attention to that area, and usually for naught. Suppose this hypothetical app has a 2-tier architecture which might genuinely be served better by decorators, as opposed to pipelines, for its use case: the engineer who checked in the decorator for the one slice probably had good reasons for doing so; it very well could be the "correct" solution for this app.

But this is where I come in with my annoying catchphrase. The app has been established as using pipelines, and pipelines are the consistent answer here. Throwing decorators in instead breaks this consistency, irrespective of the correctness of that solution. Indeed, in a lot of cases these two patterns accomplish the same thing, and the benefits gained by using one over the other are typically minimal. This is where it's more important to adhere to consistency. When I'm in this codebase, I know that I'm going to be treated to a big helping of pipeline. It's better to be consistently incorrect than inconsistently correct.

What makes code - especially code style - correct or incorrect? The more fine-grained of thing we want to call "correct" or "incorrect", the faster this distinction becomes a matter of personal preference. Pipeline vs decorator pattern is, in simple codebases, almost always a matter of preference. Maybe your debugging tools favor pipelines, maybe you'll write less code with decorators. If we're talking about a simple enough app, the distinction does not become so important as to rise above the level of consistency - pick one and be consistent. If the weight is so far in the favor of one vs the other and the weight comes down to the opposite one used, then you know that what you need is a refactor, not a fractured codebase.

Maybe instead I should say "it's better to consistently adhere to established principles in a codebase in spite of your subjective feelings towards them than inconsistently applying your subjective feelings across the codebase," but that's really a mouthful. Maybe I should say "I don't care about your feelings, consistent patterns are measurable and predictable," but that comes off as uncaring. I do care about your ideas, I just don't want them making my codebase more difficult to navigate. It's better to be consistently incorrect than inconsistently correct.

This conversation inevitably brings up the debate over the utility of consistency. I think there's strong examples of universal agreement for situations where consistency is highly valued - we tend to hold naming conventions as being important for a codebase, and using a single build system for projects (in the same language) within the same repo tends to be important. Are there examples of (near) universal agreement among software engineers of consistency being unwelcome? I can think of a few examples that aren't universally held - such as individuals who feel stifled in their creativity in a certain codebase, or when it's annoying that a codebase uses a style or pattern which is perceived as "incorrect". Ah!

These are the situations though that I suggest consistency is more valuable; these are the situations where the value judgement is subjective. What is "incorrect" is subjective, but what is "consistent" is usually measurable. If it's not perceivable, measurable, and obvious then it isn't consistency!

Now here's one situation that's important to consider: when the consistent thing to do is objectively incapable of satisfying the requirement of the piece of code. There's another saying, not my own, which goes nicely along with the one that I'm explaining here: "Things that behave the same should look the same, and things that behave different should look different." I think this is hand-in-hand with my idea about consistency. If one piece of code behaves fundamentally different from another piece of code, it shouldn't look similar. The requirement for consistency does not cross this boundary. If we have a layer in our application where each class persists some data on a single database, those should all probably look and act pretty consistently. If we add a layer of classes that contact external services to perform side-effects in the system, those classes should probably look and act a bit different to the persistence layer!

To take it back to the center though, we all perceive different things as being "correct" or "incorrect" in our code. Objectively incorrect things are bugs and unserved requirements; we fix those. Most of our disagreements are over subjective ideas. We can be tempted to check in code - one class here, one method here - that imposes our subjective ideas of correctness on an unwilling codebase. We should avoid doing this; consistency in style and pattern is an important quality of our codebases.

It's better to be consistently incorrect than inconsistently correct.

It's Okay to be a Bit Techy in Your Gherkin

Tue, 31 Dec 2024 00:00:00 Z

No, the title is not an innuendo, I mean tests written in Gherkin, the language developed to facilitate better BDD practices. Gherkin provides an abstraction between the definition of a test and the code which executes the test, allowing non-coding stakeholders to collaborate with engineers over automated tests for a project. While it's not the only way to shift testing left in the development cycle, I've used it to great effect on professional projects over the last few years that I've been at Crate and Barrel.

Gherkin is meant to be written in as human-readable a style (read: non-engineer style) as possible, this being what can facilitate collaboration between all the various sorts of stakeholders. If you've used Gherkin for a period of time as I have, you'll have encountered the suggestion that Gherkin tests should have no technical ontology, keeping its abstractions entirely separate from its test subject. Indeed, a proper distinction between a bit of code and its tests is necessary, but I want to push back on the idea that there shouldn't be any techyness in the Gherkin tests.

Indeed, for plenty of cases it does make more sense to allow the Gherkin tests to be more techy. I don't know that I can provide a formal definition of the word "techy" for my purposes here, but I think I can give a fair example: suppose I want to test a login page. The techy way might make use of identifiers and tech-focused concepts:

Given the user navigates to "http://example.com/login" using a Chrome browser
When the user inputs "john_doe" into the element with id "username-field"
And the user enters "password123" into the element with id "password-field"
And clicks the button with id "login-button"
Then the element with id "welcome-banner" should display "Welcome, John Doe!"

While the non-techy way would abstract those away to only describe user behavior:

Given John Doe is on the login page
When he enters his username and password
And clicks the "Login" button
Then he should see a welcome message saying "Welcome, John Doe!"

These styles are two extremes. Identifiers really don't need to be used in such a case, but on the other hand I could feasibly have different business rules for different browsers. Certainly it should be no trouble to explicitly state user input in the tests. Maybe a better middle ground would be:

Given John Doe is on the login page using a Chrome browser
When he enters username "john_doe" and password "password123"
And clicks the "Login" button
Then he should see a welcome message saying "Welcome, John Doe!"

If you were to take a dogmatic, anti-techy view regarding your Gherkin, you might feel that this is a bit too much techyness. I don't think most would see it that way - certainly not most stakeholders. This is exactly the point I mean to make: the stakeholders with whom we are collaborating are stakeholders on a software project; they are techy to some level. Lesser than the engineers one would indeed hope, but clearly not incapable of navigating the tech world. In fact, I'll suggest that software tests do in fact need to be technical in plenty of ways, no matter the level of abstraction. They are, after all, automated tests for software!

To develop the idea a step further, I think I can give a good example of a time to have very-techy Gherkin. There's no doubt that the way stakeholders collaborate over a project is greatly - maybe mostly - influenced by who the stakeholders are and what the project is. The same would be true for any Gherkin tests used for the project - their nature will be mostly influenced by the nature of the stakeholders and project. It follows that if the stakeholders are themselves very techincally-minded, or if the project is of a sufficiently technical level, that the Gherkin tests themselves would be written in a very techsome manner.

Suppose I have a project of standing up a new microservice to aggregate various events into doodads for other internal services. The few stakeholders that there are know what a doodad is, what a microservice is, and so on. I should have no problem with the following:

When a "create-doodad" even is published with the body
    """
    {
        "DoodadId": 12,
        "Name": "Bob"
    }
    """
And I send a GET request to "doodad-api/v1/doodads/12"
Then the response should have status code 200 with body
    """
    {
        "doodadId": 12,
        "name": "Bob"
    }
    """

If everyone understands concepts like events, event and response bodies, GET requests, and the whole shebang, then I suggest there's no issue with the above. In fact, being able to reduce the number of Gherkin steps to a small number of more generic, more technical steps can greatly simplify the testing. On a smaller project that's advantageous!

Now, this example is so technical as to bring into question why Gherkin is used at all - the tests read almost just as the code does. Truly, if there's no need for the Gherkin abstraction, then by all means don't use Gherkin. I did recently find myself in this position where the non-engineering stakeholders (such as a project manager, business analyst, and the like) knew perfectly well what HTTP requests and the like were - and preferred to define business rules in those terms - but would have found themselves too slowed reading actual test code. The uber-techy Gherkin provided exactly the right level of abstraction to facilitate collaboration and a successful BDD process.

So, don't be too dogmatic! There'll be plenty of times that various degrees of tech-speak need to make it into your Gherkin, and when done with respect to the expectations and capabilities of the project's stakeholders it can be a distinct benefit.

I've IndieWebbed My Site

Mon, 01 Apr 2024 00:00:00 Z

The IndieWeb is a small and loose collection of formats and protocols which allow those of us with our own sites to identify each other and their content across the web. Even that description is a bit much - there's only the webmention protocol and a couple so-called microformats for identifying people and content.

Obviously I own my own site, and I had to do very little work to ensure that the markup of this site conformed to the flexible specifications of the microformats. With this all in place, I can authenticate myself using my site, others can identify me by my domain, y'all can scrape my site for posts, and more. This gives a common foundation for individuals to be able to control their own content and exposure to the wider web. By using the h-card microformat to link to each other across blogs, it also decentralizes the relationships between this content, obivating the utility of scoail media platforms as providers of these relationships.

The glue that holds it together is the webmention protocol. This is a simple way for someone who publishes content to notify another when an author mentions someone on their site, the idea being that each person maitains a listener at a certain endpoint (identified by a <link rel="webmention"> header tag) that will listen for and save pings from others, and then display them on their site. My webmention URL is https://webmention.io/ian.wold.guru/webmention, and if you webmention one of my posts it wil lshow up at the bottom of the post! The other half is publishing of course - I'm working on a clean way for this site's build pipeline to send these pings. The tricky part is a ping should only be sent once, when the mention is first published. I used the excellent site webmention.io In order to set up receiving and querying webmentions. A simple script queries webmentions for each article and displays them.

The IndieWeb wiki was an excellent resource that got me quite excited about the potential for this idea. The intent seems to be that this is a supplement to social media, but I see potential in it as an alternative to social media. I'm not a fan of any of the social platforms nowadays, and attempts at federated platforms like Mastodon don't seem to provide all that much for me. The notion that I might be able to replace the conversation aspect of social media with something as simple as the IndieWeb is attractive.

I'd encourage everyone with their own site to implement at least the IndieWeb formats. IndieWebify.me is great for testing your site, and there's a lot more resources in the wiki. If you're not so keen on coding your own site and want to use a hosting provider instead, it happens that WordPress will have no problem IndieWebifying you, and there's several other hosts you can consider as well. Finally, there are chats set up in Slack, IRC, and Discord to help get going.

Is there some IndieWeb component which I'm missing on this site but I should include? Webmention me and I'll set it up; this is an ongoing project for me!

I've Stopped Using Visual Studio

Mon, 30 Sep 2024 00:00:00 Z

One year ago, Microsoft released the C# Dev Kit extension for VS Code, promising to bring the solution explorer to the editor along with quality of life updates around building, packages, testing, and other niceties. I was doing a fair amount of JavaScript at work at the time this was released, so I thought I'd install it in VS Code and use that editor for all my work instead of switching between editors several times a day.

Over the last year, I've opened Visual Studio less and less. There are some legacy .NET Framework projects I need to use it for, otherwise all of my work is done in VS Code. This is due to a few factors: I can actually develop in several langauges, I can customize it better, its extensions allow me to use it for more workflows than just development, and I find I'm actually more productive in the simpler environment. To be clear, I'm not and never have been a Visual Studio hater - I don't like Rider or ReSharper, I've always been very happy with Visual Studio. I just think the simpler editor leads to a better workflow.

Customizations

To start with, I mapped all of the panels on the activity bar to key commands. CTRL+SHIFT+_ gets me around all the panels, and I only mapped keys on the left of the keyboard for quick, one-hand navigation. x opens the _e_xplorer, c opens version _c_ontrol, space opens or hides the bottom panel, a closes the side panel entirely, and so on. I can keep the activity bar closed and get around everywhere I need quickly.

Then I made some UI updates so that the tab bars at the top of the editor are much narrower, and I cleared up the clutter on the status bar. I installed Studio Icons because after the last 14+ years of using Visual Studio I'm still used to its icons. Finally, I imported by Monokai Gray theme, which I originally made for Sublime but now use on all my editors.

Background

The best idea I had though was to install the Background extension. This extension allows you to specify background images for your code panels. I know it sounds like a distraction or a pain when you first consider the idea, but in practice it's anything but.

I installed it and loaded it with several different-colored space wallpaper images (one is red, one is blue, one is green, and so on). This has made a huge improvement in my ability to keep groups of tabs distinct when working. I use an ultrawide monitor most of the time, so it's not rare that I have six tab groups all open side-by-side. Now they're all (gently) colored! Not to mention, i makes me look like one of the cool kids 😎

Other Extensions

The WSL extension has made working in the WSL so much simpler. It wasn't necessarily difficult before, but if you d oany work in the WSL this one will sell VS Code to you (if you're not using it already - it's been well-known for some time that VS Code works well with it).

The Postman extension is so good that I barely need to open the Postman application anymore! It's really excellent to be able to keep Postman open in the same context I'm using to edit code. I don't use Postman to execute local requests for debugging, but when you need to test calls to services you're connecting to in your code it keeps the workflow entirely inside VS Code.

GitHub Pull Requests and other extensions from GitHub have allowed me to do the majority of my GitHub work right from th editor. Anyone who's reviewed PRs from their editor will (I'm assuming) be quick to describe how much better the experience is to see code changes in the context of the whole codebase instead of through the small window on the online interface.

Finally, the Excalidraw extension is really excellent. Im' a big fan of Excalidraw not just as a product, but I'm impressed by the development they've put into the product. With this exension I keep all my diagrams as local .excalidraw files, and I edit them all from the editor.

The best part about the last three extensions is they all use my space backgrounds too!

Workflow

Using VS Code for the last year has had an improvement, I think, on my workflow. For the most part, the editor stays out of the way. At first configuring building and testing took a bit of doing, but not more than a couple days' worth of reading docs online. I'm friends with the editor now, and its simplicity (or maybe its ability to be simple) is its strongest suit.

That does mean, of course, that there are a lot of features it doesn't support through the UI. Of course, anything can be done through the terminal, and the editor makes that very easy as well. I have CTRL+SHIFT+SPACE mapped to open the panel, and it opens right to the terminal now. I make a lot of use of that anymore; I barely did when using .NET for C# projects. For complicated operations - particularly some of the more finicky Git things - the terminal is a much better tool anyway.

Almost every small problem I have with the editor either has a workaround or different flow to avoid it, or gets fixed in the not-too-distant future. The one major gipe I have left though is that it is an Electron app. Microsoft has gone a fair way to address some of the performance issues resulting from that, but they are still there. F12 (go to definition) takes noticeably longer in VS Code than Visual Studio. Even button clicks seem to take an extra split second to register. Even opening a file is faster in the other.

At the end of the day though, I'm incredibly happy with the editor and the workflow I've found with it here. I'm going to be keeping it going forward, and I'd recommend anyone on Visual Studio give it a try.

I've Switched to Linux

Thu, 18 Dec 2025 00:00:00 Z

I guess this is more of a short announcement post, but I would encourage other .NET engineers to consider doing the same. As .NET has progressed, its support on Linux has gotten better and better. The VS Code C# Dev Kit has all of the features from VS that I ever need to develop, and I don't find myself needing to develop a whole lot of super-Windows-specific desktop applications anymore.

With the .NET 10 release and a round of tooling updates, I've found that all of my development requirements are completely supported on Linux. Windows is increasingly bloated (and increasingly without the ability to unbloat it), the update experiences are extremely frustrating, and the sharp increase - even for Microsoft - in privacy violations as of late makes Windows more of a burden than a help, even though I primarily use Microsoft technologies in my day-to-day development.

The reality is that I can do all of the .NET development I want to from a much more lightweight system, and I can close the book on worrying about (an amount of) Microsoft's spying. This is a step I've been hoping to take for a long time, and just as well it comes at this time of year that I can call it a Christmas present to myself!

I Have a Blogroll Now!

Fri, 01 Nov 2024 00:00:00 Z

A bit ago I collected a list of links to blogs that I enjoy tuning into. I think this is a good list of blogs; these are the ones I enjoy coming back to with some regularity - even the ones which haven't posted in a while have great articles that are worth revisiting.

There's a couple problems here though. First, I'm generally averse to blog posts with lots of updates; I think a post is a relatively static thing, so updates should be limited to correcting spelling/grammar errors and the occasional clarification. Thus, as I collect more blogs I'm disinclined to keep that page updated. The second problem is reading them: they're links in my browser and it's quite sporadic how I go about reading them. I've got most of them saved in an RSS reader on my phone but I tend to go months without remembering that that exists.

So I want a couple things: I want a dynamic page that I can update with new RSS feeds I want to tune into, and I want an easy way to be able to catch updates from these blogs. The obvious step is to add a page at the top of this site that contains the list; this isn't a post so I can update it as frequently as I want (like my now page), but then this still requires that I go onto this page and click through to read the blogs. That's not much better than having a favorites bar, though it does make the blogroll public.

It's obviously advantageous that these are RSS feeds; I should be able to set up some sort of scheme to get the latest posts from them. To me, the best place for this would be my browser homepage; I like to start my day a bit lighter before an onslaught of morning meetings, so if I could have a reading list right when I get started I'll be more inclined to tune in more frequently.

As it happens, I do have a project that I haven't finished - a static site generator which I use to generate this blog. I already know how to hook that up with GitHub Actions and Pages to get a static site really easy, so it really shouldn't be any work to set up a site that can host my bogroll and pull the latest posts from their feeds!

Indeed, it wasn't difficult to set up a first pass. I just have a JSON with a list of RSS links, and a C# script to download these RSSes, parse them, and build the site. It's just two pages: a list of blogs and a list of their latest articles (the latter page now being my browser homepage). What a deal! I didn't want to get bogged down in UI design or any super-fancy coding, so I stuck to the C# script setup that I know works, and I used MVP.css to give an attractive style to plain, old HTML. Super simple and out the door in no time!

This did replace Daily Grug as my homepage, so I'm going to need to get the daily grug quote inserted into the page.

I'll encourage you all to check out my blogroll! If you've got suggestions for blogs I should follow (or your own blog), comment below or reach out to me on any of my other channels, I'd love to find more.

In future I'll be putting in a bit of work to set up a template repo in GitHub to make it easy for anyone to use this scheme to publish a blogroll - stay tuned!

I Like Petite-Vue

Fri, 27 Jun 2025 00:00:00 Z

I like simple things in software: simple engineering, simple architecture, simple tools. The primary, almost entire, focus of our jobs in software is to wrangle the demon spirit complexity, the best tool for which is a focus on necessity: what is necessary to build a thing, and what is necessary about that thing?

If you need a personal site (hello!) then a statically generated page is much simpler than writing your own server. Server not necessary! If you are generating a static site, you might find that 90% of it isn't necessary - and that's bytes, not content! If you need a page with interactivity, React is probably overkill for you; Alpine might be all you really need.

I like Alpine a lot! In order to get all of the interactivity you'll ever really need for most simple applications, it's got the right level of learning at a low package size. It's easy to recommend for a lot of smaller applications, but there's some limitations. Very small pages make use of query selectors just fine, and applications on even the slightly larger side might, depending on the domain, be a bit wary of adopting a tool which might not be able to satisfy changing requirements long-term. A lot of applications don't have vastly changing requirements over their lives. Some do. Products almost certainly do.

These are two problems that are solved by Petite Vue (in addition to there being, or maybe having been, a "minimal" memory leak in Alpine). As its name suggests, the project is a subset of Vue that provides a similar functionality as Alpine. After resolving dependencies it has on the main Vue project it still comes in well under the size that Alpine does and it runs faster! It lacks support for extensions like Alpine has, but being a subset of Vue, you can swap Petite Vue out for regular Vue and none of your code will need to change. This makes it a lot easier to recommend for very small projects as well as those which need to accommodate changing requirements. It satisfies more needs.

The neatest part is that it's complete; you'll see no commits in three years (as of this writing) on their repo! So much software can't ever be completed, too frequently the product is not small or concise enough. We often look to the commit frequency on a project to be a determiner of status, that a product in "active" development is more worth investing in, but this seems entirely backwards to me. If they're always fixing bugs, isn't this an indicator that the project is unstable? If they're always introducing new required features, isn't this an indicator that the project is underbaked? If they're always introducing new unnecessary features, isn't this an indicator that the project is bloating?

So I like Petite Vue. It's a role model, I think, for the best kind of software development: it exemplifies the focus on necessity.

Just Use PostgreSQL

Sat, 13 Jan 2024 00:00:00 Z

There are a lot - and I mean a lot - of options when you're considering a database to use. Every application has different data, making new, specialized databases interesting to us as potential options; it's often desirable to work with tools that support our specific use cases. More often than not though (heck, the vast majority of the time) storing data is just that - storing data. Specialization is good but if it's not really needed it's easy to become too constrained or too complex. Not to mention that as requirements change our systems often evolve beyond the constraints we start with, making it a bit tougher to choose a very specialized database.

I think that for the vast majority of systems, PostgreSQL is the best choice. It is open-source, performant, secure, and supports any data model or pattern you need. It's well-documented for all of its use cases, the tooling ecosystem around it is excellently mature; heck, darn near everything about it is excellently mature. I think it's the easiest database to get started with for just about any use case, and it's able to extend with your requirements more than any other database.

This is entering opinion territory, but I think that unless you have truly one-of-a-kind requirements, chosing PostgreSQL is the best way to set your new project up for success.

PostgreSQL Supports Your Data Model

As requirements change, your data model can change for part or all of your application. I don't like being constrained by my database to a single data model or a set of models, and I don't like being shut out from implementing any patterns I need. Sometimes different data needs to be represented differently, or I might need to use different patterns for different domain contexts in my data, while still needing to maintain references between them. Only a system as widely capable and documented as PostgreSQL can allow me to do that.

Obviously PostgreSQL is a relational database, but it can support any data model you need. Built-in support for JSON data gives you everything you need for a document store. You can code your own solutions or use widely popular extensions for graph or column-family models. You can get quite far with native PostgreSQL, and well-maintained, well-documented, and well-used extensions can get you the rest of the way if you have more complicated requirements.

Any patterns you can think of are supported as well. Event sourcing, which can particularly complicated, has a lot of documentation. A lot of patterns like versioned schemas or CQRS aren't related to the specific database you choose, but PostgreSQL and its excellent community documentation makes using these patterns effortless.

Your PostgreSQL Database can Perform and Scale

Especially with reent versions, PostgreSQL offers excellent support for performance tuning, and tools like Citus can make scaling to any size quite easy. That's relatively speaking of course; optimizing any database is necessarily complicated, and distribution is never an easy problem. But PostgreSQL has the tooling, documentation, and maturity to ensure you can get there with your use case.

Being largely open-source, there's a good chance you can set your PostgreSQL environment up without spending anything on the database system itself - ideally all of your costs can be on your infrastructure providers. Being as popular and long-lived as it is, PostgreSQL can run on just about any infrastructure you might be interested in provisioning for it.

Don't Use a Document Database

Document databases are excellent for non-relational data. And being fair, there's a huge amount of non-relational data out there, particularly across the maybe millions of microservices we've created. If you ever end up needing to persist relational data though - and this isn't an unlikely change in your future requirements - you'll be in a bit of a pickle. Yes, most document databases can support relational data to a greater or lesser extent, but if you've got relational data you want a relational database.

I think for the vast majority of use cases, PostgreSQL is as capable a document database as any other popular choice. There's even libraries for most languages that allow you to use PostgreSQL with an interface more akin to popular document database drivers. But then when you need to start storing relational data, you can start using PostgreSQL in that capacity without needing to migrate your data to a different system, and without using a database that doesn't naturally support it.

When to Not Use PostgreSQL

The other databases exist for reasons though, and I don't want to be taken as advocating an exclusively PostgreSQL approach. There are plenty of edge scenarios where you'll want a different database, but I caution that these scenarios are very much the exception.

If you need to serve extremely high volumes of high-performant real-time analytics, BigQuery probably makes sense over PostgreSQL. If your application absolutely requires the highest write throughput that humanity's capabilities can muster, then a document database or KV database optimized for writes will outperform PostgreSQL. Similarly, if you're handling truly planet-scale amounts of data (many petabytes) then you should probably be reading other blogs altogether, actually.

The case you're more likely to encounter (note I did not say you're likely to encounter, just that it's more likely) is when your application - say, a microservice - is uniquely, wholy (not partly), and inextricably reliant upon the specific data structures, patterns, or capabilities offered by an alternate, specialized database. I caution you to re-read the qualifiers I put on that statement: uniquely, wholy, and inextricably. If your data is mixed model or paradigm, it probably makes sense to have it all in a PostgreSQL database. If you need referential integrity between the data in the different models, then it definitely makes sense to keep it in PostgreSQL. But Elasticsearch is spectacular if you've got a microservice exclusively for complex searches, and EventStoreDb is great if you've got a state machine microservice (do those exist?).

But Really, Just Use PostgreSQL

Those edge cases truly are at the edge of what we need to support. The overwhelming set of persitence needs for our systems is well-handled by PostgreSQL. The majority of scenarios that are supported are extremely well supported, and PostgreSQL will be very competent for everything else.

Latency is Zero and the Speed of Light is Getting Faster

Fri, 29 Mar 2024 00:00:00 Z

Did you know that the speed of light is getting faster? It is! 10 years ago it was 10 mm per clock cycle. 5 years ago it was 5 mm per clock cycle. Today it's 3 mm per clock cycle! This is good news to those of us attempting to solve latency issues in distributed systems because the speed of light is a natural barrier that cannot be overcome; this imposes a limit on all travel, including information travel, so the universe dictates a natural level of latency in all of our applications. A distributed system at scale will be particularly affected by this. "Latency is Zero" is the second Fallacy of Distributed Computing because of this - we can't ignore latency or assume it away because it is always a baseline effect on every operation.

David Boike, writing for Particular, gives a succinct definition of Latency in terms of transmission time, size, and bandwidth: TotalTime = Latency + (Size / Bandwidth). Latency is an additional cost incurred by every operation. Even local, CPU-bound operations still need to travel over a short wire within the client machine to resolve. This is low latency. A request from a client to a server is going to incur a significantly higher amount of latency - this must be accounted for. In a distributed system, every single inter-service operation incurs this latency. That makes this a big problem at scale.

Resolving Latency We Control

Unlike network reliability issues, the latency issue actually can be mostly overcome. Well, in theory it can; in practice probably not. And it depends on what kind of latency you care about. And only within the confines of your server components. Do you have total control over all of the latency in your system, and have you designed your system in such a way that no component is temporally dependent on another? I'm not sure it's possible for any real-world system to match that, but no doubt this is an area where theory can inform practice.

Maximize Utility-per-Call

Latency occurs on each operation we perform, but it's most egregious - and most impactful to us - on network calls. If we have more network calls we have more latency, and less latency with fewer calls. This doesn't eliminate latency itself from being a problem, but it lessens the effects.

Consider a microservices system where a gateway API might call into an orders service, which itself needs to call into an items service to get item information to return with an order, which itself might need to call into an availability service to return that data with the item. Assuming a baseline 50ms latency per network call (sometimes this is a lot more) our total latency in a single call is 200ms from the client. If we could shave two calls off that reduces our total latency to 100ms. Keep in mind though latency is usually more than 50ms!

Eliminating calls works to a point, obviously we still need to make network calls at some point. When we do make network calls, we want to make sure we need to make that call, and we need to make sure they're returning exactly the data we need. On top of this, sometimes calls can be combined - if we observe a pattern where service A regularly makes two calls to service B, that's an indication that service B should make a new endpoint available that condenses the data from both calls. This halves the latency incurred between the two services.

Remember that maximizing call utility does not mean "cram as much crap as possible in each request". Bandwidth still matters, and each temporally-coupling call must be absolutely necessary.

An important pattern to consider in maximizing the utility of each request is database connection pooling. Opening a new database connection can be costly, so this pattern suggests that we should maintain a "pool" of open database connections that can be reused for different requests. For applications which frequently interact with their database, this technique can significantly reduce latency in this specific scenario. This is especially important since most services will maintain some form of persistence, so database interactions are a key target to improve metrics in this respect.

Asynchronous Communication and Eventual Consistency

As I've covered before in my series on the Fallacies, asynchronous communication can help resolve a lot of network issues. To review, "synchronous" communication is when we need to communicate between distributed components in real-time to satisfy a request. When a system need to receive a response to a request before it can proceed with its task, it is said to have a temporal coupling here.

With respect to latency, these temporal couplings are a primary concern. Moving to a pattern of asynchronous communication - where communication between two distributed components is not required to satisfy a request - removes latency as a concern with respect to satisfying these requests. Asynchronous communication is achieved by having the components of the system publish and subscribe to events on state change. Instead of requiring Service A to request a resource from Service B when it needs it, we would rather have Service B publish an event whenever its resources are modified and to have Service A listen to these events and update its own persistence with the information it requires from that resource.

To be clear, this shifts where the latency is in our system. When I make a request from Service A, I will not experience the latency that would otherwise occur if that call was temporally coupled to Service B. However, there is an observable period of time between when Service B's resource is updated and when Service A is able to reflect that update. This resource is said to be eventually consistent between the two services; there are periods of time where they have different understandings of the resource. Sometimes this is OK; other times not.

If you have a system which is 100% eventually consistent and communicating entirely asynchronously, then you've eliminated request latency as a concern. But I'm betting your client still needs to contact its server, and I'm betting you'll need a temporally-coupled call here or there.

To be clear also, this strategy introduces its own concerns. You will end up with inconsistent data, for example - how do you resolve this? Can you?

Beware the Cache Demon

Caching is a tempting solution for a lot of network issues since it eliminates the need to make network calls under some circumstances. This is the catch though, our caches need to be particularly aware of what those circumstances are. When to cache, what to cache, and when to clear what from the cache are difficult questions, and misapplying a cache can lead to confusing and difficult errors. As the saying goes, there's only two hard things in Computer Science: naming things, cache invalidation, and off-by-one errors. Cache invalidation is difficult enough that I do a double take whenever the word is used in a meeting.

Understanding cache patterns will help to determine where, if anywhere, to introduce one. A caution though, fewer caches is always better - if you're able to avoid them entirely that might be just as well so as to eliminate a potential headache. Ryan French has an excellent overview of some related cache patterns on his article on this Fallacy.

External Factors

Truly though, we don't have control over all of our latency. Most systems of an appropriate size or utility need to connect to some external services. Cloud providers are a primary culprit here since most of our server computing uses them nowadays. Are you using AWS lambdas or Azure's identity provider? You might own the application logic within the Lambda or the data within the ID provider, but that's wrapped in their own system; you're interfacing with a component that you don't necessarily control.

So while you can't control all of your latency, you still do get to control your network topology. By carefully designing and managing the configuration and traffic patterns between the system components you can "box in" and reduce reliance on potentially volatile areas. I'd suggest that understanding network topology is more essential for identifying and addressing latency issues within distributed environments.

Topology Management

I considered naming this section "alphabet soup" - there's lots of patterns with three-letter acronyms in this domain. This can be its own article entirely, and topology will be related to the other Fallacies as well, so I'll just touch on a couple of things to consider for further research.

Software-defined networking (SDN) is the commonly-cited approach here; this suggests that software should be employed in defining the network topology and intelligently routing communication across the network. By allowing software to intelligently control network traffic, your system can more intelligently react to various network conditions. This is overkill if you've got a client-server setup, but necessary if you're operating at any scale.

It's a good idea to couple this with network function virtualization (NFV), which suggests that various vital components of the network can be run on virtualized environments to allow for better management. Resources like firewalls and load balancers that are necessary across the entire system can be scaled horizontally as needed. If all of your components are operating in one or several container clusers, then this is probably already done for you.

Both of these strategies allow us to use existing systems that can react to latency in the network and aids the system at scale. They don't eliminate latency as a problem, they mitigate the effect of the problem when it happens.

Geography

The physical distance between server and client is a fundamental factor affecting latency—data packets don't travel instantaneously. This isn't just theory; it's a practical concern that affects the responsiveness of distributed systems on a global scale. A request sent from Minneapolis to St. Paul is naturally going to complete faster than one sent from New York to Hong Kong, mostly because the data has less distance to travel but also because there is much more bandwidth between Minneapolis and St. Paul - the wires between the two cities support more traffic than they need, while the wires between New York and Hong Kong are strained with much more traffic.

We might be attracted to edge computing to alleviate this, and intelligently-applied this can be quite good. We do want to keep resources geographically close to their consumers. This is quite the double-edged sword though. Moving too many processes to the edge can decentralize system management to a problematic degree, leading to difficulties in maintaining consistency, security, and operational oversight. There's a balance to be struck between leveraging the edge to improve latency and ensuring that the system remains manageable and secure. Theo-T3 has an interesting perspective moving away from edge computing recently.

External Systems

In my mind, the most concerning latency issues is that we (probably) can't control all of our own sources of latency. As I mentioned, you're almost certainly communicating with an external system, and if you're not then you're probably on a cloud provider that's giving you some form of abstraction.

Suppose one of these systems changes their network topology, or some other information. Suppose that system moves its Kubernetes cluster from a fast cloud provider to a slow one. You've suddenly got more latency in your request.

To mitigate this, it might be necessary to adopt more aggressive strategies towards these external services. Implementing timeout policies, circuit breakers, or message queues (ahem did somebody say "Enterprise Service Bus"?) can safeguard your system against the unpredictability of external dependencies. These measures give you some control back, but there's no approach that keeps your system perfectly safe.

Catching Latency When it Happens

Because we can have latency thrust upon our poor systems, part of our job is to be able to react to it when it happens. We can proactively guard against latency enough to be reasonably sure that we probably aren't going to hurt ourselves, but we need a reactive posture against that latency we don't control. This means testing and monitoring, of course.

While the specific, technical considerations in setting up monitoring are out of the scope of this article, it's really not difficult nowadays to set up a basic, sufficient level of latency monitoring - there are tools which can watch and log the times of network calls, and these systems can usually be configured to ping you (or, ideally, someone else who gets paid more than you) if latency spikes at 2 AM. You might not need to react immediately in these cases, but it's vital here for the team owning a distributed system to set up a monitoring feedback loop, and maybe to include performance targets (latency being key here) as one of their performance indicators.

Here we Load Test

Because of the relationship between bandwidth and latency (they're defined in terms of each other) issues with the latter are going to arise in environments with issues with the former. That means that latency issues become more apparent as bandwidth issues do, and that means we need to understand the system under load.

Indeed, load testing systems can capture problems in all the areas we've discussed, giving important insights. We're bound to have bottlenecks somewhere in our systems, so the task is to identify where those bottlenecks are and whether they're beyond the acceptable parameters of the system. Use incremental load testing to ramp up and identify these areas with more precision. Having a solid grasp of the topology of your network will allow you to design precise load tests that can simulate load across particular geographical or logical areas.

Conclusion

As with each of the Fallacies - heck, the whole point - is that they need to be kept in mind designing distributed systems. As far as we're concerned about latency, understand what latency you can control and what you can't. Understand your network topology and use monitoring to your advantage to be reactive where you need to be.

Alas, maybe this is all for naught. After all, the speed of light is getting faster!

Learn the Old Languages

Fri, 20 Sep 2024 00:00:00 Z

There's a lot of interest in programming languages anymore. I think there always was, but it's become easier over the years for one to make their own language, and it seems like they have. Some of these languages address real-world (if often niche) problems, others are more frivolous. Most probably don't have a great reason for existing other than as a fun project for those involved. Which ones are worth learning? Well, probably not a lot of them. Most of them are interesting projects for small groups of people, though impractical outside an academic interest. A lot of them feel very samey.

Some of them, to be sure, are worth learning. However, even the new languages in this category don't have a lot of staying power for me. Rust is probably the most useful new language to learn, not just because there's a fair amount done with it now but because it does require that you express your ideas in a different way. Rust is getting well-mature and has a fair community, but it might be one of only two or three from this crop of languages that will stick.

I think that learning and using a variety of languages is an important practice. Knowledge for its own sake is good, but in this case it offers the very tangible benefit that it improves your thinking and enlightens new ways to consider problems. Knowing multiple (useful) languages makes you more valuable to more companies too - given two otherwise similar candidates, if I'm hiring for a firm which has legacy code in a more esoteric language I'm inclined to hire the candidate with a longer list of languages under their belt, even if the particular language in question isn't listed - it shows an increased engagement with programming.

So if you haven't yet, I would encourage looking back at some of the older languages. Sure, you might have opened up the Wikipedia article for Smalltalk and gawked at its syntax a couple of times, but have you built any OO-suited systems with it? Maybe you had to submit a few Discrete Structures assignments in Prolog in college, but did you ever try to implement real-world algorithms with it? These older, more forgotten languages offer the same benefit in expanding your programming skills while having the distinct advantage of practicality. To be clear, Delphi is nowhere near as practical as C#, but there's sure still legacy code written in it, and as it happens those languages were created by the same guy - just maybe there's some insights to be gained there.

Here's a small set of languages I'd encourage you learn. Not just to gloss over the syntax, but to sit down and build with. Personal side projects of course - though there are some proper professional uses for these yet, it's probably most advisable to the majority of folks to leave these for fun. The skills you gain by going deep with these languages can translate in real, practical, and sometimes surprising ways into the languages we typically use professionally.

Smalltalk

Smalltalk is a neat language, though if you've spoken with anyone who has actually worked in Smalltalk before you might get the sense that these folks are just about inclined to split the years on the calendar into "Before Smalltalk" and "After Smalltalk". So many of them love this language. This is decidedly a language firmly situated in its own time, so you might not find you're able to gain quite that affinity for it today, but it nonetheless has a surprising ability to win over its programmers.

Smalltalk was, as you might well know, the first object-oriented language. Everything in the language is an object, and you pass messages instead of making function calls. As Smalltalk initiated the paradigm, that's technically how we're supposed to think about the Java or C# world, but in those languages it ends up being a distinction without a difference - particularly in languages which attempt to blend functional and object-oriented styles like Scala or C#. Not so in Smalltalk: everything reinforces and builds on the paradigm.

A number of those folks with an affinity for Smalltalk insist that to this day it is still the only object-oriented language. Whereas Smalltalk has an extremely light syntax laser-focused on its paradigm, the other languages have more robust - maybe "bloated" - syntaxes which, if not encouraging you to program against the paradigm, do at least make it more difficult to stay focused. If you spend a reasonable amount of time with the language, you might run into a similar feeling I did. It takes a little bit of effort to get to the point that you "click" with the style of program that Smalltalk encourages, and when I got there I remember thinking "golly, this is so different to object-oriented programming!" The kicker of course is that this paradigm is (arguably) the One True object-oriented one, and divergences I make in other languages are maybe something a bit different.

So I do think it will give you a great foundation of OOP in understanding how to craft a "pure" OO implementation, and I think that goes a long way to teaching what the fundamental goals of the paradigm are. Sure, we all had to read some paragraph about the paradigms in college, but that doesn't mean a whole lot in the face of finger-on-keyboard experience.

The other envy you'll come away with is the live debugging ability - you're able to change programs as they're running and see those effects in real-time. If that's news to you then you might be surprised that that's true for an object-oriented language - isn't that supposed to be the thing that functional languages are good at? This is another brick in the "Smalltalk is the only object-oriented language" wall - its message-sending focus allows the various components of the code to be decoupled so as to allow this. It's a great programming experience, though seeing as Smalltalk isn't used anymore I will confess it did serve to bias me towards functional languages which offer this capability today.

Speaking of its use today, it's not used! At least, I can't really find a lot of documentation towards industry use. In fact, with as loved and useful as the language was, it's surprising that it didn't last that long. I don't have firsthand experience with this, but my understanding is that a fair amount of the legacy Smalltalk has already been rewritten. If you have knowledge about this, please do leave a comment! I think if you're pining for a high-paying job at a bank rewriting a legacy system (as every child dreams for when they grow up) you're probably still better off elsewhere. However, I put the following to you: if you go into an interview there's a nonzero chance yet you'll find yourself across the table from an experienced colleague with an affinity for Smalltalk, and I guarantee you'll be benefited there!

Prolog

Prolog is one of my favorites! I've not hidden that one of my two degrees is Philosophy, and when I was a student I focused quite a bit on logic. In fact, I was able to do a lot of my philosophy with prolog, which might be a bit of a head-scratcher if you haven't encountered logic programming before. This is a separate paradigm where you write programs by declaring logical statements of facts about your program. The computation is then done by an interpreter applying problem solving algorithms to resolve the relations of facts in the program.

This is a decidedly different way of thinking about a program - one which you are unlikely to use developing most professional software. However, this skill of writing programs by description translates directly to a lot of tasks. Any database work maps 1-to-1, and it's the same part of your brain you need to engage when writing tests to reason about the system. Modern pattern matching algorithms are essentially doing the same thing as a prolog interpreter, just on a different level, so if nothing else you might be able to break out some slick pattern matching moves!

This style of programming is quite influential in the field of AI, where it was largely employed back in the first AI boom in the eighties. It was also used, and to an extent still is, in natural language processing, data querying, and theorem proving. Is there a lot of that sort of work being done at our normal 9-to-5 jobs? Unlikely. Is it the cool thing to be programming though? Absolutely, as it has been for some time. The style of programming supported by Prolog cuts through a lot of busy work and can get you going quite fast if you're looking for projects in these areas.

One of the coolest ways to engage with Prolog though is to write your own interpreter for it. Heck, if you're so inclined it's somewhat trivial to write a parser for it. I know I introduced this article suggesting something other than making your own language, but I've had to sit through enough so-called experts tell me that Prolog isn't a real language so arguably this doesn't count. Being serious though, a parser and interpreter for it took me about 2 weeks of on-my-own-time effort, and that's really not bad. It was even fun ... towards the end!

Now while that project might teach you the finer details of SLD resolution, the primary emphasis I want to make is on how logic programming is beneficial, and for that you'll need to roll up your sleeves and dive into Prolog. However, there is an alternative: if you're more database or just data-inclined, then Datalog might be more your speed. It's a subset of Prolog that acts as a query language for databases. This use itself is a distinct paradigm for working with data, and its applications to more traditional SQL are more directly obvious.

Prolog is one of the primary recommendations I make when I'm asked for "different" languages to learn, not just because it's so different from our normal bread and butter, but because the skills you gain from it apply in a lot of smaller but consequential ways. I think this gives Prologgers (I have no clue if that's what we're supposed to call ourselves) a leg up all around. Maybe it's the single best "old" language to learn to really round out a programmer. That is, in my humble opinion.

Fortran

Yes, I do mean the Fortran created in 1958 that originally used punchcards, though you needn't use that old of a version! Would it surprise you to learn that its most recent version was released in 2023? Would it surprise you even more to learn that it still has a roadmap? You can load it up today in VS Code, intall dependencies from a package manager, and develop some really serious programs with it. Indeed, it's not just a use_able_ language, it is still used (I think more than the previous two).

The applications for this language are sciency and mathy, to use technical terms. Being the low-level language it is, its uses are in that sort of computing that researchers run on supercomputers and the like. To support this, Fortran has strong support for array operations, efficient memory usage, and parallel computing. As such, today it's running practical computations in weather prediction, fluid dynamics, biology and chemistry simulations, and engineering modeling.

Being one of (maybe the) first science-focused language, a lot of its concepts provide the foundation for our newer languages in the same field, such as Python or R. As far as Fortran's practicability (i.e. "can I build a simple web API with it?") I do genuinely get the sense that it is a lower-level R: they both have packages and fair documentation for doing modern application development. I'm unsure if either is able to be used, easily, for mobile application development. However, there are frameworks for each to support desktop and web application development, which helps greatly in eliminating any gap between the research and productizing (for lack of a better term) work that might need to be done on such projects.

The skills I think you're most likely to pick up with this language are varied, and all at a low level. High-performance computing (HPC) is an obvious answer, but unless you've got easy access to a supercomputer that part might remain theoretical. Instead, practically, I'd suggest memory management and parallel computing. Sure, you can get a really good foundation in memory management with C, but I would suggest that Fortran's straightforward syntax is a better instructor on this subject. Parallel computing tends to become a more abstract concept the higher-level your language is, while I think Fortran would give you a relatively easy tool to explore that subject at the lowest level.

I started toying with Fortran a year ago thinking it would be funny to be able to suggest creating microservices in the language at work, and I was surprised that I enjoyed the language so much. Today it's quite fashionable to use functional languages for the more mathy services that we need to write at our jobs. That used to be the domain of C or C++, but those have slightly fallen out of favor due both to the popularity of functional languages as well as concerns about the seemingly high presence of footguns in those languages. I'd suggest here that Fortran is a viable candidate to consider as well for these tasks - some computation is very well-served by the functional paradigm, but plenty of algorithms are best represented imperatively.

COBOL

Do not learn this.

Forth

Forth seems to have a perpetual, ebbing popularity - I see a bit of interest in it rise maybe every 5 years. That could just be me, but I think there's a good casual appetite for the language out there. Just like each of the previous languages, Forth operates in a separate paradigm to the rest. All programs in Forth run on a stack which holds all of the program data. Data is manipulated by popping it from and pushing results onto the stack, chaining stack operations together into a program.

This has a couple of distinct advantages beyond solving the problem that we sometimes fight over what name a variable should have. The paradigm necessarily restricts programs to expressing concepts that map quite well directly onto the CPU operations while allowing higher levels of expression in the program code than other low-level languages. As such, it naturally rns quite fast with efficient memory use. The stack paradigm is particularly well-suited to state machines and expresses mathematical operations very neatly and composably. These facts all make it a great choice for firmware, robotics, and other sorts of embedded systems.

Unsurprisingly, it is still used for these tasks but quite rarely. There are plenty of newer stack-based languages as well, and those are maybe a bit more common in the toolbelt of the engineers working on these systems. That said, I have found (with admittedly limited experience) that all of these stack-based languages have the same core feeling, and Forth has excellent extensibility and pretty fair documentation, so I've stuck with this one.

Like the logic programming paradigm of Prolog, this stack paradigm is one that can be replicated in any other programming environment. If your application has discrete, one-off requirements that can be served best with a stack-based system, you can surely find a good number of libraries in your language providing these capabilities without needing to include a Forth executable alongside your application. The best way to learn how to use this paradigm though, I suggest, is to properly learn a stack language.

Being able to think primarily in terms of the stack has a lot of benefits on the side as well. Notably, I think most engineers use or have encountered Java or C# before, and the virtual machines which run each of these languages operate on a stack-based instruction set. There are plenty of ways that stacks are used creatively to boot, so I think it's safe to suggest that a robust understanding of them tends to pay off in the long term regardless.

As a final note here, I'll mention also that Forth is particularly well-regarded for its extensibility - it's very easy to add new words to the language and develop domain-specific languages with them. There was a major push in favor of DSLs in the last decade which doesn't seem to have taken off, but in the abstract their concept applies to all the programming we do. When we create a library or implement a standard for a layer in any application, we're defining a DSL of sorts which is defined by the constraints and standards we develop for that codebase. There are a few languages which are particularly biased towards this DSL-style thinking, of which Forth is one. This is a completely separate sort of thinking than stack thinking, though nonetheless hugely beneficial for any programmer. Though, I wonder if that's a separate article in the future!

Looking Beyond GitHub

Sat, 21 Feb 2026 00:00:00 Z

Like most folks in our industry, I've been using GitHub extensively for a long time, and I've become very familiar with it (and, as a consequence, Git). For the most part, I do like doing version control in Git. There's a number of efforts afoot to try to propose alternate VC systems that solve common issues with Git, but I don't really see these gaining a huge traction. Then again, there's a lot of folks reevaluating a lot of common practices these days.

GitHub is a huge advantage gained for using Git: the tool makes it incredibly easy for me to host public and private projects, with all the CI bells and pages whistles I need. Nonetheless, I'm going to be working on getting myself off of GitHub over the next several months. Instead, I'll be spinning up my own Gitea server on Railway. Why? Ownership, cost, and control.

I have a very significant amount of code in GitHub: some public, mostly private, some in between. GitHub has fine features to facilitate collaboration on code, but nothing so granular as what I can do with my own server. Controlling code access is one thing, but being able to have easier control over my CI/CD infrastructure is also appealing. I might be getting some cost saving in some areas, but more importantly it's easier to set up custom infrastructure with my own server than fooling around with GitHub.

At an even higher level though, being able to have ownership of your own things is good these days. It's devastatingly simple to spin up a Gitea, committing me to not much more time than I normally would put into maintaining my GitHub while gaining a bit more platform independence. Lately I've been a huge advocate for owning your own tools; I think this mirrors my preference to roll your own code where practical. Increasingly, everything about our lives is becoming a subscription service requiring us to pay for the privilege of owning less - how exponentially impoverished we're becoming! Ownership isn't now about wealth; it's a matter of freedom.

So I'll be spinning up my own Gitea server, migrating off GitHub, and gaining a huge benefit - by my eyes. There remain issues:

If I delete my GitHub, how do I find others' repos? How do they find mine?
Gitea doesn't have an equivalent to Pages, but obviously this can be done through Railway. Need to figure a good architectural approach.
Railway, funny enough, is built entirely around "give me a GitHub repo and we deploy that with no effort." There will be more effort now.

These problems will be ironed out, and I'll document on my blog how I get through these. The first point is a sticky one though: the industry is so used to being on GitHub. That's not as hard an expectation now as it was five years ago, but lacking any separate protocol for repo discovery this may be interesting. For public projects I might set up some kind of GitHub replication. Goodness knows!

Many Dimensions of Heterogeneity

Fri, 25 Oct 2024 00:00:00 Z

The eighth and final fallacy of distributed computing is "the network is homogenous," originally referring to network hardware. The caution at the time was to not overlook the challenges in dealing with different physical routers, switches, servers, and the like when dispatching requests across a network.

Over time software protocols have been able to alleviate most hardware concerns for developers of modern distributed systems, so the fallacy is more often used now to caution that the software dimension is heterogeneous. Changing topologies, different protocols or message formats, and evolving network configurations and policies are some examples in this dimension. As components of distributed systems are increasingly tended to by greater numbers of administrators, the degree of network heterogeneity at the software level increases.

The next dimension above this is maybe the most interesting one - architectural heterogeneity. This would encompass everything from differences in the semantics of contracts and business objects to persistence and consistency models to SLAs (service level agreements). This dimension is not frequently touched on in discussions regarding the fallacies of distributed computing, and I think that's a shame. I typically think of a distributed system being more difficult and/or complex than a monolith by a factor of 10 in every aspect - 10x more surface area for bugs, 10x as many considerations to guard against, 10x as much development time required, etc. It occurs to me then that there would be 10x as much architectural burden from these systems, and the heterogeneity of the constituent parts of the system is directly contributing to that.

Having already discussed most technical aspects of the fallacies in my other articles on the matter, here I want to focus on the various dimensions in which distributed systems are heterogeneous. Just as standardized abstractions and a defensive attitude are able to stave off the complications which were initially observed stemming from hardware heterogeneity, I think we'll find the same approach will adapt well to these other dimensions.

The Software Dimension

If we review my previous articles on distributed computing, we'll quickly find that distributed systems often do not - nay, typically do not - have homogenous latency, bandwidth, security, topologies, nor administration. We've well-explored the software causes and solutions to these in the past. There are plenty of software-level concerns apart from these networking ones to explore.

Protocols and Formats

These are, of course, the fundamental building blocks of any kind of distribution scheme. I need a protocol for the client and server to understand to transact packets, and those packets need to be formatted in a mutually intelligible manner. I think it's fair to say that today we have a healthy set of each to choose from which are well-defined and well-established. IP, DNS, HTTP(S), FTP, TCP, Websockets; XML, JSON, Protobuf; I needn't list them all but you get the picture. If we're building a brand new system today we're probably going to fall back on the good 'ol DNS, HTTPS, JSON, UTF-8, and the like - you know the drill.

All of our languages, frameworks, and tooling supports this stack natively and of course also supports the other common protocols and formats. But we'll use the familiar defaults and then we won't need to worry about differences right? Well, just a decade ago (maybe a decade and a half? Surely not two decades yet? Time flies...) a lot of us were thinking that SOAP and WSDLs and XML was the greatest thing since sliced bread and it would be great into the future. Visual Studio could generate a client so easily! So that didn't pan out, and now a lot of us need to write code to integrate with these systems we now call "legacy" because we don't like WSDLs today.

Popular protocols and formats move on, and this introduces multiple protocols into our systems. We can also inherit them if we absorb another company's system into our own (takeover, merger, etc). Sometimes our services even have technical requirements to differ! It's the same with formats, where XML gave way to JSON and then for a few months everyone wanted to switch to Protobuf. Versioning is especially fun too - HTTP/1.1 differs from HTTP/2 in a lot of ways.

Nowadays we tend to do a pretty good job, when using HTTP, to pass necessary information along with the headers - this data is JSON, it's UTF-8, and so on. I've yet to see a header though detail what format the dates are though, and in spite of my best efforts I still run into date (de)serialization issues every now and again.

There's a whole lot of detail in these, and assuming two services are 100% in-line with each other - even if they agree HTTP, JSON, and the like - is to commit this fallacy. Indeed, this is perhaps the area of the greatest heterogeneity in our systems, even though we might not notice all of it. In a few paragraphs I'll discuss the semantics of contracts, and it's in these contract negotiations that this can all be hashed out. While you're arguing whether the API needs to return the created date for an object, you should also be having the conversation of what that (or any) date looks like. The increasing standardization of our protocols and formats is welcome, as is the tendency for our tools to become more resilient with each other as a result. None of this saves us from the fact that these differences exist. At that, they don't just possibly exist, they very well do in each of our systems!

Software Stacks

It's been said quite a bit about microservices - less and less lately now that the trend is subsiding - that one of their great advantages is the ability for separate teams to use different technology stacks - even different languages - in building out their individual services. Certainly this is a significant side effect of the architecture, thogh depending on what kind of organization you're working for this can be significantly good or significantly bad. The idea of watching the ripple effects across a department of 50 C# engineers slowly coming to the realization of the implications of one team having gone rogue in deploying a Clojure gRPC service is an entertaining one.

Not that I've ever observed such a phenomenon.

But here you might say: "Well Ian, this is simple! We can have some organizational coding standards, enforce C# and some common Nuget libraries, and avoid the concern!" If this is you, then it is you for whom I am writing this article, for this line of reasoning is to fall victim to one of the classic blunders: it is this fallacy!

Separate teams deploying separate services will diverge and have separate tech stacks. Separate tech stacks communicating with each other will have inconsistencies. Tautologies are always true. Being clear, this will always be a bug vector, but to be aware of this fact is to minimize the surface area of the vector. Do vectors have surfaces? I'm bad at geometry.

This usually crops up in small things. Do the two separate systems have the same ability to parse dates? Do the email validations follow the same rules? Do you have addresses? Do you have international addresses? Do you have a button on your site that says "I know you weren't able to parse this address but keep the one I entered"? My disdain for handling addresses aside, just in data interpretation alone there are so many points for slight divergence between different software systems - different libraries, languages, configuration defaults thereof, and so on.

Let's throw some load balancers, firewalls, and caches on top. What kinds of request policies do these expect of your client? Is the client service capable of acting in a manner friendly with this? You'd expect so, but the point is to not assume so. All of these concerns are, in theory, wrapped up into the contract negotiated between the two entities. And how many contract negotiations have happened that excluded, uh, most of the things it should include? Haha, this article doesn't seem so pointless all of a sudden!

This bleeds nicely into the next dimension: if we can't trust the components of a distributed system to be architecturally homogenous with each other, can we at least trust the network itself to be architecturally homogenous?

The Architecture Dimension

To immediately answer the above question: no, of course not!

For a long time I disregarded the eighth fallacy - the network is homogenous - as its original sense doesn't really apply as much any more. It gained a new life for me in this more abstract sense having read David Boike's article for Particular on this same fallacy. It's unfortunate that article is so short because there's plenty more to be said regarding architectural heterogeneity.

Semantics: Objects, Actions, and Contracts

Now you might be tempted to dismiss this as just a difference in semantics (hahaha ... please laugh) but this really does matter. In fact, semantics is a lot of what we do in modeling the business, whether we know it or not. Ensuring consistency between our semantics is paramount to ensuring a correct mapping of the business domain. Misunderstandings - even at a fine-grained semantic level, is an ever-present vector for bugs.

I currently work in ecommerce, so I can take the business object "item" to demonstrate semantic difference. Items exist, obviously, all over the place on a shop website. You can view an item, maybe customize it with some options, and you can add it to a cart or a registry. This involves at least three teams: an items team, a cart team, and a registry team. Does item mean the same thing to all these teams?

Let's say the items team is tasked with maintaining a service that stores all the items; this is the source of truth for what the "Luxury Dinner Plate" and other items are. Their service lets you query the details for this item and produces events when it becomes no longer available and the like. To the items service, that's what an item is: It has details (name, price, personalization options) and can go in and out of availability, and it only ever has a single "Luxury Dinner Plate".

But now take the cart or the registry. Those can have multiple "Luxury Dinner Plate"s in them, maybe with different personalization options. Therefore you'd expect the cart and registry services to have different identifiers for their items than the item service; an item for these other services is actually an item in cart or item in registry. Hopefully it's clear then the importance that when these teams work together (and consequently when the services interact with each other, directly or indirectly) that everyone understands the semantic difference. For quick example, it would be insufficient for an "item shipped" event to only include the identifying item information used by the item service but exclude information required by the registry service.

That example might or might not be a recent problem I had to tackle.

This naturally extends beyond the objects and their properties to the sorts of actions taken on an object. Of course you can't purchase an item stored in the item service, you can only purchase an item stored in the cart service. If you feel compelled to interject here and try to correct me with "but you can also purchase an item from the registry service!" you'd be wrong! Haha, got you! You cannot, but you can add to cart an item from the registry service. "Oh but come on," I can hear you objecting, "that's just a semantic difference!" Exactly! And if it's left unconsidered you're going to end up with a real fun issue in production.

These two aspects - what an object are and what its actions are - are the basis of contracts between components in the system, and their semantics form part of the semantics of the contracts. I'm being very careful to not accidentally assert that they are all of what the contracts are. No, unfortunately the real world is more complicated, but they're the two halves of domain modeling and consequently I think that object and action semantics are the two main bug vectors from the contract/semantic side. The hope is that this impresses the importance of thoroughness in negotiating and defining these contracts. If you encounter a colleague accusing you of getting too granular or "semantic" about it, remember that that's the whole point!

Persistence and Consistency Models

That a distributed system would have a mixture of persistence and consistency models is, I think, an absolute certainty. If you're not dealing with enough data or transactions to have to soften data requirements somewhere then your application is probably a poor candidate for distribution.

Persistence models mostly affect individual services while consistency models apply across the system. Ideally components of a distributed system are isolated and one needn't know about the internal function of another (persistence being an "internal function"), though these two do affect each other; different persistence models can allow a service to support a given consistency model more or less easily. A hard persistence model such as ACID ensures a service can maintain data integrity and best supports a hard consistency model, while at the other end a soft soft models such as BASE are entirely contiguous with the softer consistency models.

ACID and BASE, when applied to database transactions, are typically only describing persistence within a discrete service - a single actor against the database. When we move up a level to consider separate pieces of related data in a system, we might find that we need to maintain data integrity across distributed systems, and we might find ourselves tempted to consider support for single transactions distributed across these components. My advice is to avoid distributed transactions at all costs; it is typically an antipattern. If you encounter the need for them this is a smell which indicates that you should either combine the components which share the related data, or you should instead rearchitect the system to accept softer consistency in that data. If you really do need distributed transactions though then use a single database which supports such transactions.

The overall trend though is that distributed systems will result in components with softer persistence models, which in turn results in softer consistency models across the system. I think it would be difficult for a distributed system to avoid having any eventually-consistent components. Considering the technical constraints which we explored with the other fallacies, asynchronous communication is an essential tool in solving some of the more difficult technical issues. Not only does that require accepting eventual consistency, but it circles back to having to accept a soft presistence: packets might get dropped without the downstream system being any the wiser. State drift!

For an example that demonstrates these effects, consider the ecommerce system from before with the items, cart, and registry services. Suppose that to save having too many calls into the items service, we update the items service to send out an event when data about the item changes (let's say name and price), and asking consuming systems to listen to these events and record what they need themselves. It's easy to see a potential issue here: suppose an event goes out and it is received by the cart service but not the registry service: you might get a situation where a customer sees the "Luxury Dinner Plate" for $10 in a registry, adds it to their cart, and then sees it as $12 in the cart because the registry service never got the event that updated the price.

This seems like a glaring issue with event-driven systems, but the key is on data requirements. Perhaps an item price is a poor choice for such a setup, since many systems have a hard requirement to be aligned on that data. How about the name of an item though? That's probably only going to change if there's a typo; this is an excellent candidate to be updated via an asynchronous channel. We can say that item name can accept softer persistence and consistency models, while the item price does not have that luxury.

We do have other options with the item price though. While we might need to accept a very hard persistence model for the item price data, we might be able to identify some components in the system that do not need to have as hard a consistency with item price data than other components. That is to say, some components need real-time item prices, while others would be satisfied with only near-real-time prices. For those latter components, we might be tempted to replicate the item price data geographically so that requests to the item data can still have high throughput with minimal strain on the broader of the network. In such a scheme, the item service would expose two separate interfaces for the real-time and near-real-time flows; the former requests would query against the main dataset (the one which is always written to from which the other replicas copy) guaranteeing real-time data, while the latter requests would query against the closer, faster replica data. Feasibly the items service could be broken out so that the item price data is managed by a separate service, considering the different architectural requirements for this data.

To bring it back around, we can see that different components of the sytem will require different persistence models, and this has a knockon effect of supporting many different (though potentially similar) consistency models. That becomes a cycle where consistency models will influence individual persistence models, with the asterisk in the whole cycle being that business requirements will set lines in the sand as to how far in any direction the system can move here. Components with different models will interact with each other in interesting ways, which makes this a crucial area requiring communication and understanding. Even between two services which each describe their consistency model in the same way, to assume they really are entirely the same is a potential flaw.

SLAs and QoS

One of my favorite observations to come out of the microservices milieu is that most microservices, in most microservice setups, don't implement proper failover techniques. Or, they don't implement them at all. This is to say that if Service A depends on Service B then Service A really should be able to serve requests with some kind of meaningful response (other than 502) when Service B is down, but this seems to be implemented infrequently. If we expand this to a system containing many services which are probably not isolating themselves from downstream failures, it's easy to see the potential for cascading failures; these might be incurred by network errors or low fault tolerance. Indeed, when my service depends on another it's difficult to figure how to get around that dependency when it is down. This is often where it's beneficial to be able to grasp the wider scope of the system and ensure its architecture supports the kinds of services which are able to do that.

We can see there's two aspects (beyond architecting the system and coding the services correctly) which influence the fault tolerance architecture of a service. The first is how these aspects are included in the contract(s) the service has with its client(s): what uptime is guaranteed, average response time targets, and so on. This is the SLA, or Service-Level Agreement. The second aspect is the relationship the service has with the network: different services have different performance and network-usage characteristics requiring different provisions of network resources. This is the QoS, or Quality of Service. Think of SLAs as keeping up the business functionality and QoS supporting the overall network functionality. These are two aspects which should be analyzed, have standards agreed, and monitored. To be clear though, this doesn't need to be an overly formal thing if you're supporting a simpler system - but because you are monitoring your services (right?) then you should be setting metrics for what you expect to see at the least.

The key point to be made here, of course, is that the SLA and QoS goals are going to be different for each component in the system. Having the targets for each defined are what allow me to plan for those differences; actually implementing strategies for this heterogeneity is the next step. The implementation is a continuous action of monitoring and adjusting. You'll adjust your architecture to support different techniques for failover and fault tolerance and you'll adjust your network to accommodate the many different requirements. Just as well that we haven't fallen on the assumption of homogeneity here - look at what we can do!

Managing the QoS is about shaping the network traffic - something that we're well-familiar with having reached the end of our series on the fallacies. This involves identifying where we can apply traffic shaping mechanisms, rate limiting, and the like. There's lot of ways of going about finding these; it suffices to say that you should be logging information from your services which help identify traffic patterns. Relying on the OpenTelemetry standard can allow you to analyze these with very good tools like Honeycomb. If all of your traffic passes through a gateway, firewall, or other central component (which is a questionable practice depending on what that central component is) then you can get some very fine-grained monitoring here.

Of course since you've got your monitoring set up it's much easier to enforce your SLA targets; most tools can set up alerts if your uptime falls below 99.99-whatever percent and the like. I'm a big fan of having these alerts. If a part of the system goes offline it's nice to have somebody get texted to address the issue. During a failure (which you could feasibly simulate) you'll also be able to see whether it cascaded to parts of the system you might have thought were isolated. I know I keep coming back to it, but if you haven't I really encourage you to read the rest of my series on the fallacies - I've already covered a lot of fault tolerance strategies so there's a lot I'm leaving out here.

Conclusion

I think the whole idea I want to convey is that every component of a distributed system is different from every other component, not just in the obvious ways but in every way. These differences might be small and subtle but they are there, and the most that these differences are acknowledged, discussed, coded for, and monitored around, the better. These differences crop up in a whole host of dimensions. There's the hardware dimension which I didn't discuss here, but then the software and architectural ones.

The focus of course is on those aspects which are often overlooked. We might forget about protocol differences because of course everyone uses HTTP, right? Or then we forget that an item is not an item when it's inside a cart. Often times these are problems which are easily resolvable once encountered, but sometimes they bring the whole show to a halt.

If we are to create truly robust distributed systems, we need to spend the time upfront to ensure our development is on the right track here, otherwise we expose ourselves to a great many more bug vectors, slowing development in the best case but impacting our users and businesses in the worst cases.

Minneapolis, January 2026

Sat, 31 Jan 2026 00:00:00 Z

If you've visited my blog before, you know that I live in Minneapolis. I state that on my about page, my now page, my GitHub profile, and in plenty of articles I've written on this blog. I've always loved this city, and I love living here.

Unless you've been living under a rock for the last month, you've seen that my city is in an interesting way right now. Several colleagues have reached out to me in the last month to check in on me. That's been very much appreciated. I am alright, considering.

There's a lot of folks in the Twin Cities that aren't alright though. In case anyone reads my blog from afar and feels a need to help out in any way, I want to encourage you to visit standwithminnesota.com. There are plenty of ways to help, financially and otherwise.

I make a point of (largely) avoiding political conversation on my professional software engineering blog. I will stress that we must not consider discussion about ICE's activity in Minneapolis to be "political;" the totality of what the federal government has fashioned here is another thing entirely.

Thanks again to all my colleagues for your care and support!

Monokai Gray

Thu, 05 Dec 2013 00:00:00 Z

About a week (?) or so ago I hashed out a quick Sublime Text color scheme called Monokai Gray. I did so because I desperately love Monokai and I love the Wombat Theme as well. Unfortunately, I couldn't use Monokai with Wombat (without throwing up) because the yellowey hue of Monokai clashed with the classy grays of Wombat. Truthfully, I had been using a similar color scheme in Visual Studio for some time (the yellowey background didn't fly there, either). So I copied many of the colors over and prettied it up a tad, and this was the result.

Here's a preview of what it looks like with some code I got here using the aforementioned Wombat Theme:

If you think it's neat, you can check it out on Package Control. I love feedback and pull requests alike!

My (Continuing) Descent Into Madness

Sat, 09 Dec 2023 00:00:00 Z

It wasn't that long ago that I tasked myself with updating a few microservice codebases to be able to work on VS Code along with Visual Studio. I really don't like the stock VS Code, so I spent a few hours customizing the colors and moving the explorer to the right. Then I added some custom keybindings so that I could navigate around the IDE efficiently.

Being clear, VS Code is not the perfect IDE. In fact, there's many things for which VS is better. VS Code is an Electron app, and you can tell even just while typing in it - VS is faster at typing. However, a combination of the customizability and several quality of life features, I became a bit envious of my colleagues who were using VS Code full-time.

So I decided to try using VS Code full-time.

Oh, how I should not have started to tweak my environment. Once I opened the first door I've ended up opening them all, tweaking every aspect of my digital life.

At this point I've completely riced my VS Code. I have custom CSS (via Custom CSS and JS Loader) to tweak each little nitpick I have. I've remapped almost all of the keybindings and added plenty of my own. When I use stock VS Code I feel dirty and disgusted, and when my colleagues use my VS Code they feel dirty and disgusted.

But this wasn't enough for me, I'd found a zeal for getting the most out of my environment, for having every command at my fingertips, for elminating visual clutter from my screen.

I found myself increasingly, then exclusively, relying on ALT+TAB to navigate windows. I'd keep my IDE maximized (side note: I should write a script to hit F11 when it starts up), then I started keeping my browser maximized, then everything else. The Windows task bar was just getting in the way now - I took all the pinned applications off it and made Windows hide it from me.

I don't keep too many windows open at any given time, but ALT+TAB takes its toll. Wouldn't it be faster if I could map ALT+<Number> to a window, or an environment?

The envy I once had for my VS Code colleagues has been replaced by an envy for my Linux colleagues who can use i3 or Qtile. I needed a tiling window manager, but on Windows? In fact, yes you can, yes I did, and yes I riced the hell out of it.

Not only could I have separate workspaces one ALT+<Number> away, windows are maximized by default! If you've been using a tiling window manager for some time you know this is routine, but I was new to this world and it only fueled the fire I'd found myself in. My coworkers now joke when I'll switch to Linux. They think it's funny, I fear it might be inevitable.

I would need icons for my workspaces of course, and Glaze recommends using Nerd Fonts for this. I'd resisted installing one of these for so many years. What's wrong with good old Consolas? Am I going to become one of those dweebs with all the glyphs in their code?

Yes, yes I did. And I have pretty icons for my workspaces. And once you give an engineer a nerd font I suppose it's only a matter of time (hours, in my case) that Oh My Posh sneaks its way in. My Powershell is now gorgeous.

Like a frog in boiling water, this has all happened to me gradually and I barely registered this transformation. I would come across an extension like Background for VS Code and install it, not thinking twice that now all of my code has pretty space images behind it. But I need to acknowledge it now, lest you too might fall into this trap.

I've gone mad and I see no end in sight. Will I start using Arch? Will I switch to Neovim? Will I take a job writing Haskell whitepapers? I do not know.

On Task Priority

Sun, 03 Mar 2024 00:00:00 Z

I don't know of a lot of successful software projects that don't have some sort of list of tasks defined. Unless you've found some magical, outlying case, we need to define our work before we can do it. This usually takes the form of a kanban board where tasks are "cards", or at the very least tasks are defined in rows in a spreadsheet.

Naturally, these tasks have priority. This isn't just useful so that I know which task to pick up next when I'm done with my current one - we usually have milestones or objectives we need to meet in our development and organizing that gives priority to the tasks. Tasks and priorities are absolutely necessary, so we slap a tshirt size on our cards, keep them ordered with higher priority at the top of the list, and consistently reevaluate priorities to keep everyone aligned, right? Right?

The trouble, as with most things in software engineering, comes from the fact that this field is filled with software engineers. Maybe one day we'll be able to do something about that root cause, but until then we'll have to deal with it. In this case it's that we software engineers can't help but organize, systematize, and insert-your-word-here-ize every problem we find. We do tend to over-engineer, and boy is task priority the sort of thing we tend to over-engineer. I want to take a look here at the best practices.

Maintain Alignment

The priority of a task is directly tied to the definition of the task and the goals of the project, and none of the variables are static. They change by the day, and sometimes by the hour. Each of these three are essential for an entire team to understand about a project, and everyone needs to be on the same page. This is the most important thing to recognize about task priority.

Reevaluate Often

Every single time we meet with stakeholders, the project changes. Sometimes dramatically. This is a feature, not a bug; we want to have the tightest loop we can between delivering value and incorporating feedback. This feedback always changes priorities. Features get reconsidered, bugs get discovered, milestones move.

The ramification is that priorities change to adapt to this, and as long as priorities are being documented they need to be revisited and revised when these parameters change. This is an opportunity to make sure the stakeholders and the team are aligned. Ask questions in terms of priority, and document and clarify why tasks have the priority they do.

Organize Tasks by Priority

This is as simple as keeping them at the top of whatever shared list the team has. It seems obvious to read, but it's often overlooked. How many times have we found an errant "high" priority task that's been at the bottom of a backlog for twenty sprints?

Reorganization should be part of reevaluation. Often times tasks will be assigned one of a small number of priority options; in this case there are several tasks with "high", "medium", or whichever priority labels you have. This doesn't tell the whole story though - the team might have different goals this week or sprint. Maybe some milestone is more pressing and its related tasks should be considered first. These fine-grained priority assignments are best encoded and documented by keeping the task list in order of priority.

Assign Meaningful Priority

Priority has meaning to everyone - the stakeholders, team members, and even users sometimes. All of these people need to be kept in alignment about priority, and so some kind of common language is needed. The priority needs to be meaningful to everyone involved, and the meaning needs to be the same for everyone across the board. This is extremely difficult to achieve since very few of us have developed adequate skills in telepathy.

The most important thing is to keep priority limited to a very small number of generic words. 3-5 is ideal. If you're able to get away with just 2, that's spectacular. Consider that each additional priority category doubles the chance for confusion for each individual that needs to be aligned on priority. If you have 15 priority categories ranging from "no priority" to "EVERYONE SWARM NOW" then nobody is going to understand anything, math being what it is (trust me, I ran the numbers).

"Highest" and "Lowest" Have no Meaning

Taking the common tshirt-size priority list of "high", "medium", and "low", it's quite common to want to tack on "highest" for a catastrophic situation and "lowest" for a backburner want-to-have. I understand the inclination, but there really is no meaning here.

For "lowest" priority items, why even have it on your set of tasks? if this is something that your team is going to do, then it should be understood in the context of the whole project and have a proper priority. If it's a wishful thought or a pipedream of one of the engineers (what this category is typically used for) then it's not a part of the project and it's wasting valuable space. Get rid of it.

Similarly, if I really get an incident that requires immediate intervention, I'm not going to create a "highest" priority card for it. I'm getting everyone on a Zoom call, starting an incident document, and fixing the issue. "Highest" priority cards lag around because some individual needed to be placated that we're definitely considering this one as important. This is a misalignment, and it should be addressed as such. Everyone should have the same idea as to what we're marching towards.

Don't use "Medium"

No, really! Just like "highest" and "lowest", this one ends up having no meaning. Using tshirt sizes, "medium" becomes the priority for 90% of all tasks - when someone creates a task they don't want to upset the boat by choosing "high", but they definitely want it done so they're not going to select "low". Instead, they use "medium" as the default "pffft, don't know don't care" option, tell the PM, then the PM doesn't reassign priority because they don't understand the full scope of why refactoring the IDoSomething interface is really needed at this stage.

A "medium" priority becomes a dumping ground for all tasks and leads to a situation where the true priority never gets documented because everything is "medium". This is a major source for misalignment acorss the entire team. Instead, my preferred set of priorities is "high", "higher", and "lower". This forces us to consider whether it's more or less important, and recognizes that most tasks on the board are there because they have some heightened priority. For ideas or far-off things, I would suggest keeping a separate, non-prioritized list for documentation purposes.

Beyond Priority

Priority is, for most of our projects, a simplified capture of many different complicated factors about our tasks - the perceived need of different groups of stakeholders, the hopes and wants of the development team, and the unkowns of new technologies or complicated use cases. In some projects however, things are more clear-cut. There might only be a single, contiguous group of users, or the focus might be so constrained as to be relatively obvious to all the parties involved.

In these cases it could be that priority is actually too ambiguous, and measuring other factors might serve us better in terms of understanding task order and importance.

Impact

If the project as a whole is addressing a specific use case or need, it might be useful to measure tasks in terms of customer impact and business impact. You can use the modified tshirt sizes of "high", "higher", and "lower", or you can just use "high" and "low" for each. By focusing on the different aspects of priority, this can greatly simplify the priority assigning process and make the documentation more meaningful.

The types of impact can be varied depending on the project type as well - if the project is a heavy refactor then team impact or codebase impact might be good inclusions.

Necessity

When the project is narrowly defined but has several different and/or disparate groups of stakeholders, defining the priority of a task instead by its necessity for different kinds of stakeholders can be the best option. Again, the modified tshirt sizes or just "high" and "low" can be used here.

The broader idea is to think about what kinds of divisions the definition of your project has, and to consider breaking priority out along those lines. By making "priority" a more concrete concept, it will be easier to assign and reevaluate priority, and it will be more meaningful for everyone who needs to be aligned on it.

Oops, I noindexed the Blog!

Sun, 26 Jan 2025 00:00:00 Z

Many months ago, I set up Umami to track views on my blog. It doesn't track users, just counts page hits and referring sources and the like. The results were encouraging: I do get a fair amount of traffic. Hello! I realized too that about 50% of it is from mobile devices. There goes my theory that I don't need to make it look good on mobile...

Last month I noticed a trend: I was getting many fewer clicks and almost no traffic was coming from Bing or Google (having a lot of .NET content I came up in Bing results a fair amount). This was odd but I attributed it to the holiday season. I was still getting plenty of traffic from Yandex and Baidu.

Well, it happens that the decrease in traffic coincided with my release of StaticBlogroll, a repo/template I made to encourage others to share a public blogroll. It's intended to be deployed with GitHub Pages, and I figured it would be good to include noindex, nofollow by default in the robots.txt for this site, lest others' articles might appear under my domain on Google. Unlikely, but I didn't want the chance.

Well, as it happens, GitHub likes to deploy Pages sites under your domain if you've already registered it. Thus, my blogroll deployed to ian.wold.guru/Blogroll. No problem by my book!

Well, as it also happens, Bing and Google will read every robots.txt file on your site. As it further happens, noindex nofollow in any robots.txt overrides everything else.

Oops.

Two learnings:

That's why I had fewer hits, and
Yandex and Baidu are (maybe predictably) not following best practices.

Luckily, Bing and Google have webmaster tools with which you can notify any updates to these files, so the problem should be corrected. I've got no clue if some of my previously-most-popular articles will regain their positions though. I'm particularly happy with my article on E2E encryption in Blazor WASM so I hope that one recovers. Finally, I've updated the default for StaticBlogroll and I've edited my (now only) robots.txt for this site to exclude /Blogroll.

Postgres: Use Views to Refactor to Soft Delete

Sat, 05 Oct 2024 00:00:00 Z

Author's note: I have updated this article with some extra bits of information as well as a repository demonstrating this solution. I hope this is helpful!

In the world of persistence, there's two main patterns (maybe "groups of patterns") to handle deletion: hard and soft deletion. Hard deletion is the default for most database systems - when you "delete" a record it is wiped from the database; as soon as the delete is committed the data is gone forever. More common in most business scenarios - particularly server-side - is to retain the deleted data, just using a flag to hide the "deleted" data from the customer. This is soft deletion: it allows us to support more comprehensive internal reporting, maintain more complicated referential integrity, and we can support an "undo" button for our users.

If you're designing a system from the ground up and you know you need to accommodate soft deletion, there's a whole host of implementations you can tailor to your needs. If you're updating an existing system from hard to soft delete though, you're more constrained. Yes, the data that has already been hard deleted is irrecoverable after your migration, but that's not your main concern. You might have a lot of tables and there's probably already a lot of code written to query against these tables. This refactor might seem like a lot of work at first glance.

Luckily because you're definitely using Postgres, this really isn't a major concern. There's a simple way using views and rules we can add soft deletion without changing any of our queries! In fact, there are two options which you might want to chose in different situations. I've implemented these strategies on a few production systems with zero downtime, and I'm sure you'll be able to make just as quick work of it as I can.

The first strategy keeps all records - those that are deleted and those that aren't - in a single table and uses a view to filter out deleted items. The second strategy adds a second table specifically for deleted items and uses a view to join the deleted and non-deleted records when needed. I have found the first option simpler to implement and maintain, while the second option is sometimes better in scenarios where I need to run a lot of different queries on the deleted vs. non-deleted data. These strategies could feasibly be combined in a single database, with some tables using one strategy and others the other, though consistency is probably better here.

I've set up a repository on GitHub which runs both of these solutions on a database. If you're seriously considering implementing a migration like this, I would encourage you to fork and play with my repo, I hope it can be an effective way to tinker with some of these ideas before moving into an actual codebase. Throughout the article I'll reference where each bit of code is in this repo.

Setting Up

I'm going to start by defining the existing system we're going to refactor. Let's consider a subset of an ecommerce system with tables items, carts, and of course cart_items:

CREATE TABLE items (
    item_id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    price DECIMAL(10, 2) NOT NULL,
    created TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE carts (
    cart_id SERIAL PRIMARY KEY,
    user_id INT NOT NULL, -- I won't define this table
    created TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE cart_items (
    cart_item_id SERIAL PRIMARY KEY,
    cart_id INT NOT NULL,
    item_id INT NOT NULL,
    quantity INT NOT NULL DEFAULT 1,
    created TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    CONSTRAINT fk_cart FOREIGN KEY (cart_id) REFERENCES carts (cart_id) ON DELETE CASCADE,
    CONSTRAINT fk_item FOREIGN KEY (item_id) REFERENCES items (item_id) ON DELETE CASCADE
);

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/Program.cs#L18 [tl! autolink]

And we're going to need to support all the various queries currently written against these tables. In addition to all the queries SELECTing from these tables, we're inserting, updating, and indeed deleting from each of these tables - those queries all need to remain the same.

There's a couple interesting queries I want to consider. Take the scenario where a cart is deleted:

DELETE FROM carts WHERE cart_id = @cartId
-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/Program.cs#L139 [tl! autolink]

In this case, it's pretty straightforward to assume that its items should delete at the same time, so we'll need to preserve the cascade functionality but with soft delete in mind. How about the following though:

DELETE FROM items WHERE item_id = @itemId;
-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/Program.cs#L138 [tl! autolink]

In our current production system, when we delete that item it will delete from the cart automatically. This is a good demonstration of why we care about soft delete - by preserving the data in our database we'll be able to tell the user that their dinner plate set is no longer available. Indeed, properly supporting that functionality is going to require some extra work in our code, but remember that that is a feature add: we can do our soft delete refactor and deploy it, preserving the current functionality with minimal effort. Later iterations can build on this work to add the new capabilities in.

This is the sort of planning work which needs to be done before beginning the refactor: The relationship between tables needs to be mapped out and understood. Typically you'll already have cascades set, so ideally this won't be difficult work to do.

Single Table

This is the first strategy I'll discuss. To implement this, we'll go through a few discrete steps. First, we'll add a deleted flag to all of our tables, then we'll create a view for each table that excludes deleted records, and finally we'll update the definition of DELETE to perform the soft delete. Those views will effectively replace our tables for all queries and commands. After the refactor, deleted items can be queried directly from the original tables, which for all other purposes will have been "hidden" by the new views.

Adding the Deleted Flag

The first step in our refactor will be to update every table we want to soft delete to add a new column. This will be the flag that determines whether the record is deleted or not. You can use just a boolean here, however this is an opportunity for us to capture more data and potentially make debugging a bit easier. I prefer using a nullable timestamp for my flag - null signifies it is not deleted, whereas the presence of a date indicates not only that it was deleted, but also when. Because soft deletion is commonly used to support internal reporting scenarios, this can sometimes be quite useful information.

Adding the column is quite straightforward of course, it can be added with no effect on the overall system:

ALTER TABLE items ADD COLUMN deleted TIMESTAMP DEFAULT NULL;

ALTER TABLE carts ADD COLUMN deleted TIMESTAMP DEFAULT NULL;

ALTER TABLE cart_items ADD COLUMN deleted TIMESTAMP DEFAULT NULL;

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SingleTableMigration.cs#L6 [tl! autolink]

This can be deployed at any time and will not affect the rest of the system unless you're doing something squirrely with your tables. Just note though that if there's a long gap between your migration steps, new tables should have this column included. Because this column isn't used, it will be null for all records, and until we proceed to the next steps records will still be hard deleted.

Hiding Deleted Data

This second step can also be deployed at any time after the first step with no change in the functionality of the system: we're going to hide any data where deleted is not null. In order to accomplish this without needing to rewrite any of our queries, constraints, or the like, we'll use views to achieve this.

For the unfamiliar, views act just like tables, but are essentially projections of other underlying tables and views. You define them with a SELECT statement just as you would query any other data. This allows you to marry commonly-referenced data together, but in our case we're going to use them a bit differently.

For each table which has a deleted column, we're going to define a new view which excludes the deleted values of the underlying column. Now, because we have how many references to the tables named items, carts, and cart_items, and it's these references from which we want to exclude the deleted items, we're going to give our views the names of the current tables and rename the current tables.

ALTER TABLE items RENAME TO items_all;
CREATE VIEW items AS SELECT * FROM items_all WHERE deleted IS NULL;

ALTER TABLE carts RENAME TO carts_all;
CREATE VIEW carts AS SELECT * FROM carts_all WHERE deleted IS NULL;

ALTER TABLE cart_items RENAME TO cart_items_all;
CREATE VIEW cart_items AS SELECT * FROM cart_items_all WHERE deleted IS NULL;

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SingleTableMigration.cs#L12 [tl! autolink]

Here I use the convention that the table name is suffixed "_all", signifying of course that the tables contain "all" of the records. Your requirements might be different, so pick a naming convention that makes sense in the context. I recommend being extremely consistent here, and I also recommend picking a convention that is obvious in distinguishing the tables.

After deploying this update, you'll find that your system is still working exactly as it has in the past. Postgres passes any commands on these views down to the underlying table, so when you INSERT into one of these views, the data will be inserted into its respective table. Neat!

Make Deletes Not Delete

This is the crucial step now: we need to redefine what a "delete" means. While the last two steps alter neither the behavior of the system nor the representation, this step will alter the representation of the data, and so long as we're dilligent it will not alter the behavior for your users. Because you're altering the data representation now though, be careful!

There's two steps we need to take here at once. First, we need to tell Postgres that when it gets a DELETE command for one of these tables that it actually needs to update the deleted column instead. Second, we need to define the new cascade behaviors to propogate the soft deletion in the intended way.

To redefine DELETE, Potgres allows us to create a rule to perform an alternate action instead. Typical for Postgres, this syntax is quite straightforward:

CREATE RULE rule_soft_delete AS ON DELETE TO items DO INSTEAD (UPDATE items_all SET deleted = CURRENT_TIMESTAMP WHERE item_id = OLD.item_id);
CREATE RULE rule_soft_delete AS ON DELETE TO carts DO INSTEAD (UPDATE carts_all SET deleted = CURRENT_TIMESTAMP WHERE cart_id = OLD.cart_id);
CREATE RULE rule_soft_delete AS ON DELETE TO cart_items DO INSTEAD (UPDATE cart_items_all SET deleted = CURRENT_TIMESTAMP WHERE cart_item_id = OLD.cart_item_id);

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SingleTableMigration.cs#L23 [tl! autolink]

And then we need to handle the cascades. Let's start with the simple case where we know that deleting a cart should cascade to each of its items. The rule we define will act on updates to the carts_all table and propogate new, non-null values for deleted to the cart items:

CREATE RULE rule_cascade_deleted_cart_items AS ON UPDATE TO carts_all
    WHERE OLD.deleted IS DISTINCT FROM NEW.deleted
    DO ALSO UPDATE cart_items_all SET deleted = NEW.deleted WHERE cart_id = OLD.cart_id;

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SingleTableMigration.cs#L27 [tl! autolink]

Recall that the other cascade case, where we delete an item, is more complicated. The current behavior (which we are just trying to preserve for now) is that items are deleted from a cart when the items themselves are deleted. The state we want to achieve (some time after this migration) is that items will still show in the cart but as "no longer available". That will take some extra coding to implement, however depending on the current state of our system we might be able to achieve some cost savings.

If you just want to keep the refactor simple or your system wouldn't currently support an alternate scheme, then you can implement the exact same rule but for items, then remove it later once you're ready. However, consider whether your current system would be able to do without the cascade right off the bat here. Suppose you have some logic on your cart page which attempts to get item data for all items in the cart and will not display items for which it cannot find data. In this case, you do not need to add the cascade, and the users will still observe the same behavior they currently have!

If you do decide not to carry over the cascade, there is the matter of the foreign key reference - if we soft delete a record from the items_all table, the foreign key on cart_items.item_id is still referencing the new items view and will break! In this case, we'll need to update the foreign key now:

ALTER TABLE cart_items_all DROP CONSTRAINT fk_item;
ALTER TABLE cart_items_all ADD CONSTRAINT fk_items_all FOREIGN KEY (item_id) REFERENCES items_all (item_id) ON DELETE CASCADE;

Again, this step does not need to be done now if you are adding the manual cascade for items.

Accessing Deleted Records

At this point, your migration is finished. Congratulations! In order to get records that have been deleted, you just need to query the _all table for that record:

SELECT * FROM items_all WHERE ...
-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SingleTableMigration.cs#L37 [tl! autolink]

In order to hard delete any records, you can DELETE from the _all table:

DELETE FROM items_all WHERE ...

Separate Deleted Table

This is the second strategy, where deleted items are kept in separate tables. We'll also be able to work through some discrete steps to implement this. First, for each table we'll create a corresponding deleted table, and then we'll also create a corresponding view to unite the two. Finally, we'll alter the definition of DELETE to perform the soft delete. After the refactor, we'll be able to consult the separate deleted tables to work with our deleted records.

Adding the Deleted Tables

The first step in our refactor will be to add a corresponding table for each existing table that will include the deleted fields. Instead of adding a deleted timestamp on the main table itself, it's sufficient for only this copy table to contain the field. We'll use LIKE to copy the schema of the base tables.

CREATE TABLE items_deleted (
    deleted TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    LIKE items INCLUDING ALL
);

CREATE TABLE carts_deleted (
    deleted TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    LIKE carts INCLUDING ALL
);

CREATE TABLE cart_items_deleted (
    deleted TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    LIKE cart_items INCLUDING ALL
);

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SeparateTableMigration.cs#L6 [tl! autolink]

My same note from earlier about naming conventions applies, though I think _deleted is probably the suffix 99% of us will ever use for these.

There is one issue with this snippet though - can you spot it? The problem is in cart_items_deleted: LIKE ... INCLUDING ALL will copy over the entire table schema, foreign key references included. In order to get around this, we'll need to not use INCLUDING ALL and manually copy over the foreign key constraints:

CREATE TABLE cart_items_deleted (
    deleted TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    LIKE cart_items,
    
    CONSTRAINT fk_cart FOREIGN KEY (cart_id) REFERENCES carts_deleted (cart_id) ON DELETE CASCADE,
    CONSTRAINT fk_item FOREIGN KEY (item_id) REFERENCES items_deleted (item_id) ON DELETE CASCADE
);

But wait, there's more! This works all well and good so long as the carts and items you're referencing from the cart_items_deleted table actually exist in items_deleted and carts_deleted. This is sure to not be the case, as it makes perfect sense that I might delete a cart item without that item having been deleted! Supposing that we did have a schema where we would expect the referant to be deleted also though, are we sure that that record is being deleted first?

This can all get quite tricky. If you don't care about the referential integrity of your soft-delted data (but trust me, you probably should), then you can just forego the foreign key constraints. If you do care about the references though, then you'll need to spend some time thinking about the best approach here. A whole article can be written on potential approaches so I won't distract us here, but beware of this trap!

This demonstrates the extra complexity of this second approach. It allows for a greater separation between deleted and non-deleted items, though it's important to consider which functionality and level of complexity your system requires.

Adding the Combined Views

There are inevitably operations we're going to want to perform on the whole dataset for each table - deleted and non-deleted items all as one. To facilitate this, we'll create a view for each table/deleted table pair which unions the two:

CREATE VIEW items_combined AS SELECT null AS deleted, * FROM items UNION ALL SELECT * FROM items_deleted;
CREATE VIEW carts_combined AS SELECT null AS deleted, * FROM carts UNION ALL SELECT * FROM carts_deleted;
CREATE VIEW cart_items_combined AS SELECT null AS deleted, * FROM cart_items UNION ALL SELECT * FROM cart_items_deleted;

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SeparateTableMigration.cs#L23 [tl! autolink]

At least that's easy!

Make Delete Not Delete

In order to make DELETEs perform a soft delete in this setup, we need to do a couple things when deleting: first, we need to copy the record into the respective _deleted table, then we need to allow the delete to actually happen.

Here we have a couple options. We could use a RULE...DO ALSO to perform the insert after the delete:

CREATE RULE soft_delete AS ON DELETE TO items DO ALSO (
    INSERT INTO items_deleted (item_id, name, price, created)
    VALUES (OLD.item_id, OLD.name, OLD.price, OLD.created)
);
CREATE RULE soft_delete AS ON DELETE TO carts DO ALSO (
    INSERT INTO carts_deleted (cart_id, user_id, created)
    VALUES (OLD.cart_id, OLD.user_id, OLD.created)
);
CREATE RULE soft_delete AS ON DELETE TO cart_items DO ALSO (
    INSERT INTO cart_items_deleted (cart_item_id, cart_id, item_id, quantity, created)
    VALUES (OLD.cart_item_id, OLD.cart_id, OLD.item_id, OLD.quantity, OLD.created)
);

However, because we're combining operations I would tend towards preferring a trigger in this specific scenario:

CREATE OR REPLACE FUNCTION items_soft_delete() RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO items_deleted (item_id, name, price, created)
    VALUES (OLD.item_id, OLD.name, OLD.price, OLD.created);
    RETURN OLD;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_items_delete BEFORE DELETE ON items FOR EACH ROW EXECUTE FUNCTION items_soft_delete();

CREATE OR REPLACE FUNCTION carts_soft_delete() RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO carts_deleted (cart_id, user_id, created)
    VALUES (OLD.cart_id, OLD.user_id, OLD.created);
    RETURN OLD;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_carts_delete BEFORE DELETE ON carts FOR EACH ROW EXECUTE FUNCTION carts_soft_delete();

CREATE OR REPLACE FUNCTION cart_items_soft_delete() RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO cart_items_deleted (cart_item_id, cart_id, item_id, quantity, created)
    VALUES (OLD.cart_item_id, OLD.cart_id, OLD.item_id, OLD.quantity, OLD.created);
    RETURN OLD;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_cart_items_delete BEFORE DELETE ON items FOR EACH ROW EXECUTE FUNCTION cart_items_soft_delete();

-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SeparateTableMigration.cs#L29 [tl! autolink]

You then have the same considerations regarding cascades and foreign key references as you had in the last refactor option. I'll leave that as an exercise for you to extend the trigger functions to include that behavior.

Accessing Deleted Records

This is either the benefit or the drawback of this setup compared to the first scheme. In order to access deleted records, you query the _deleted table:

SELECT * FROM items_deleted WHERE ...
-- https://github.com/IanWold/PostgresRefactorSoftDelete/blob/main/SeparateTableMigration.cs#L58 [tl! autolink]

And of course all records together from the separate view:

SELECT * FROM items_combined WHERE ...

If your application will need to regard these three sets differently in distinct queries, then this setup is advantageous: there's less chance of writing the queries incorrectly and causing bugs. However, if you do need to consult the blended data or perform mostly the same queries, the first option might be better.

Conclusion

Implementing soft delete on an existing system can seem like an exceptionally daunting task if you consider it from the perspective of needing to rewrite whole swaths of existing queries to suppot it. Not to mention, the incidence rate of bugs from such a hands-on refactor will be quite high!

The less we can touch our system during a refactor, the better. As with any refactor, this one requires some careful planning up front to understand the relationships between tables as data is deleted. Once that's understood, the refactor can proceed really quite fast just by changing the database schema, preserving the existing functionality and, crucially, the existing queries and the underlying semantics of those queries.

This doesn't absolve you of the careful work in ensuring your refactor works correctly (doubly so as you're tinkering around in the persistence). However, it does put more guardrails in place. Being able to move quickly and deliberately, touching as little as possible, leaves you in the best position for success in your refactor.

Principles for Successful Teams

Mon, 03 Mar 2025 00:00:00 Z

There's a heck of a lot of blabbering that's been written on the topic of what makes a team, department, or firm successful. It's (almost) entirely crap, in my experience. The bulk of it seems to be an exercise in justifying the unjustifiable jobs that middle managers and consultants take up; this is a relatively wealthy bunch so I suppose it makes some sense to sell snake oil to them.

No doubt we've all had experiences with incoherent decisions from higher-ups, illogical constraints on work, or even micromanaging. We look at each other and wonder "what if we just had the space to get our work done." Indeed it seems more often than not that altogether engineers, QAs, and the like are able to make better decisions collectively than in the typical top-down structure.

On realizing that, the principles that underlie the success are quite clear: we're all professionals who can A. deliver their work and B. collaboratively agree on decision points. In that spirit, here are (I think) the only four principles that are really needed between us and our colleagues in any team, department, or firm.

1. Each Colleague is Independent

Everyone must have ownership over their own efforts and must be able todeliver their work without micromanagement or unnecessary restrictions. Where each colleague is a professional capable of particular tasks, they must be unburdened in fulfilling them.

The diversity of thought and approaches to work strengthens the whole effort; each colleague must be able to communicate their ideas, opinions, and feedback.

2. All Colleagues Form the Whole

Our teams, departments, and firms are formed, entirely, by the sum of the contributions by colleagues. The whole cannot be formed without the contributions from everyone; each colleague is necessary irrespective of title or experience.

Effort must be distributed across all colleagues, and no individual should find themselves a "linchpin" in holding others back.

Each colleague must have access to the same resources, and all colleagues should have the same information available to them. Everyone should be "on the same page."

3. Each Colleague Supports Each Other

Collaboration is the only way to deliver efforts involving more than one colleague; competition (however friendly) and rugged individualism (however ideal) only prohibit progress on work.

Each colleague must have camraderie with each other; each must share their knowledge, time, and ability with each other.

4. All Colleagues Compromise

There can't be a proper balance between individual and team effort without compromise, and compromise is required to maintain the other principles.

Each colleague must have a voice in decision-making, feedback, and leadership over their efforts; all colleagues in turn must bring these together into a productive decision.

Each colleague must participate in the activities of the whole team, department, or firm.

Provocative Holub Opinion Tier List

Sun, 15 Feb 2026 00:00:00 Z

If you're new to my blog, one important thing to note is that my thinking regarding team and organizational practices tends to align with that of Allen Holub, at least so far as what I've gleaned from a number of talks and blog posts he's done.

Holub has come upon a number of beliefs which, while not necessarily being among the most controversial opinions in our industry, are nonetheless heterodox to the normal prescriptions we find in our industry with respect to organization. He has branded a number of these in a hashtag format such as "#NoStandups" or "#NoRetrospectives," stemming from an earlier trend around the hashtag "#NoEstimates." Below I sort these into tiers.

#NoEstimates

Link

This is the original "no" hashtag, there's a reason it began a trend. This is the one best-supported by empirical data: studies have shown that counting cards (rate of backlog accrual vs rate of completion) allow projecting the completion date of a project within plus or minus two weeks. By using completion date projection instead of the more common and opposite practice of trying to estimate completion dates, businesses and teams gain a finer level of control on what products they ship when.

This one goes in the S tier: it is very widely applicable and impossible to argue against when outside of limited cases.

#NoStandups

Link

In his opening paragraph on the matter, Holub introduces the standup meeting as "a band-aid that’s hiding ineffective communication." This may be true in a number of cases, and regular, daily standup meetings may well not be needed by a lot of teams (particularly long-running product teams). However, There are lots of cases where a daily check-in meeting is a benefit, and the standup format is a good standard for check-in meetings.

This one goes in the B tier: most teams will certainly benefit from reevaluating how they check in and monitor progress and I suspect that most honest teams will conclude that a daily standup isn't necessary, however "#NoStandups" isn't as widely applicable a banner.

#NoRetrospectives

Link

This idea is to track process impediments as cards in the same way we track work items as cards; that every sprint (or whatever iteration) at least one impediment will be worked on and resolved. This is a good idea! Is that what you thought of when you read "#NoRetrospectives" though?

This one goes in D tier: apart from being false advertising - it is not advocating "no retrospectives" - it would also probably function better in the form of "here's how to give your retro meetings some teeth."

#NoAccountability

Link

Holub argues, convincingly I think, that "accountability" in the business sphere tends to be reduced to meaning "you can be punished for this thing that I, your manager, am foisting on you." The #NoAccountability idea then is to promote shared ownership and collaboration over point-person ownership.

This one goes in A tier: while not as flashy as the more widely cited #NoEstimates, it identifies a real, universal problem and posits a workable philosophy as the solution. Unfortunately, the nature of the solution is not a step-by-step "here's what to do," instead requiring a mindset change to effect.

Publish Your Blogroll Now

Sun, 03 Nov 2024 00:00:00 Z

Do you read blogs? Of course you do! How do you keep those blogs saved? How do you keep up with their latest posts? How do you share these blogs with everyone else?

I recently became motivated (by the realization that I keep forgetting about my RSS reader) to answer these questions by creating a blogroll. But not just any blogroll, no! A super cool blogroll - a standalone, statically generated page that lists all the latest posts from the blogs I follow. It's like an RSS reader but way better: I can share it with the internet, and I set my browser's homepage to the collection of latest posts. I can keep up with my reading and share it with everyone.

The best part is you can too! I developed StaticBlogroll, a very creatively-named template repo you can use to get your own blogroll page going in no time flat. It uses GitHub Actions and Pages, so there's no work beyond just a little config to set up.

Okay, but why do I need a blogroll?

I'm a big advocate that everyone should be maintaining a blogroll. Not super actively - maybe more passively - but you should definitely have one. No doubt there's dozens of blogs a year you tune into - an article here to get help on an issue, an article there to learn about a topic, articles for fun or insight or curiousity.

A public blogroll helps others discover the blogs you enjoy. Even if only a few friends end up looking at your blogroll, you're helping the author find new readers at the same time you're helping your friends find something new. Even just recognizing an author on your blogroll is a kind gesture that shows your appreciation for their work.

If you happen to have your own blog, keeping a blogroll is an even better investment. Connecting your readers with blogs of similar interests gives a better, deeper experience for your readers. As for me, I feel like I can represent more aspects of my interests and personality by linking to other blogs that are better-focused than my blog on topics which I enjoy and influence my writing here.

So what's the utility in a statically generated blogroll? The only thing I've added to take advantage of this setup (yet) is to read the latest posts in from the RSS feeds on my blogroll. As I mentioned, this was selfish because I needed an easy way to read them. I'm certain that could be handy for you as well, but I also like the idea that this makes it easier for visitors to my blogroll to skim through for articles they might be interested in.

That sounds awesome! How do I get one?

I've published StaticBlogroll as a template repository on GitHub; you can create a new repo from that template and follow the setup steps on the readme - these should take no more than 5 minutes to get all set up!

The way it works is very simple: there's a config.json at the root level that has all the configurations. This is the name of the page, some headers and the like, but most importantly the list of feeds. These are the URLs to the RSS feeds of the blogs you want on your roll.

From there, there is a GitHub Actions workflow that generates the static site based on this config, though there's a few other files (templates and static files) you can peruse at the root level if you want to customize it further. The workflow runs whenever you push to main and it runs once a day at midnight to regenerate the latest posts.

After you've gone through the simple steps to configure Actions and Pages, you'll have a blogroll page of your own! Woohoo! The only next step would be to share it with the world. I set up a discussion topic on my repository to showcase blogrolls made with it - please add your own so I can see the cool blogs y'all read.

If you'd like a reference with some further customizations, check out my blogroll repo made off of this template.

Pull Requests are Just Fine, Thanks

Wed, 18 Dec 2024 00:00:00 Z

There's a degree of popularity around claiming that pull requests are bad and we shouldn't do them. I'm not sure that this is the majority opinion, but it's one that over the years has been popularized by some notable voices like Dave Farley, eched by a host of easy-to-find blog posts about the their disadvantages or their dysfunctionality or even how they are hated.

A review of these articles (and plenty of others which crop up from time to time) reveal a common list of gripes with PRs:

PRs give too much power to reviewers; open to exploitation
If branches live long there are merge conflicts
Large PRs are difficult to review
If the team only gives superficial reviews there's no point
PRs reduce or remove two-way communication (async over sync communication)
A review at end of dev cycle is too late
PR lag increases the amount of work in progress
PRs make the development feedback loop longer; shorter feedback loops are better
Sometimes it's easier to fix a bug than to explain the fix
PRs discourage continuous refactoring
PRs lead to negative emotions; reviewers might not be emotionally tactful
Adopting PRs for the sake of it makes purpose unclear

These complaints fall into a couple of buckets: there's problems with power dynamics, poor practices on part of reviewers, and poor practices on the part of authors. Then what to do instead of a PR? Farley would have you do pair programming before merging your code into main, while others would do away with the mandatory code review altogether. These are the practices we used before the PR came around - like many of these commentators, I did not use manual PRs for the first years I was a software engineer. In fact, I think it was almost a decade before I worked on a team which adopted the manual review.

One thing I try to espouse on this blog is a way of thinking conditionally about different options we encounter as software engineers - whether that's process, architectural, or whatever decisions. In this instance, I don't want to say "always use PRs and never use push-directly-to-main," instead I'll want to give reasonable and meaningful conditions as to when one is preferred. I don't get the same sense from most of these anti-PR articles. Sure, they do pay some service to cases where PRs are preferred, but this is superficial and unserious. The suggestions that PRs are only ever going to be useful in open source projects or when all members of a team seriously distrust each other are ridiculous and unhelpful.

When will I suggest you use PRs? Whenever you want. When to use some alternative type of code review? Whenever your team wants. When to not review at all? Whenever you want. I've developed with teams adopting all manner of schemes and at the end of the day the software all ends up getting written. Usually. The only caveat, as with all process things, is to make sure you're measuring it and able to identify if a process is holding you back. Don't be afraid to experiment with something different!

Our industry seems to have adopted (from my perspective, at least) mandatory PRs as the default way of doing code review. I'm perfectly fine with that; it's unfortunate when an organization might prohibit teams from experimenting otherwise, but that's not a hill I want to die on. Personally, I like PRs as a default, and I think the the anti-PR claims I enumerated above are well-overblown. In fact, I like PRs a lot, and I want to respond to those supposed disadvantages and argue in favor of PRs. I'll start with the last bullet I listed: we probably shouldn't be adopting processes which we're unable to justify, so if your team only adopts PRs "just because" then by all means have a rethink on that one.

Why PRs Aren't That Bad

There are a couple points made on the topic of power, that is, ways that PRs are perceived to give too much power to reviewers. When a comment is blocking a PR that shouldn't be blocking one is an example of such a power dynamic, but the more subtle one is when a very influential member of a team leaves a suggestion which - wink wink nudge nudge - isn't really just a "suggestion". These are concerns with team dynamics though. I know engineers tend to be a less social segment of society, but has the author of the PR with the blocking comment tried talking to the blocking reviewer? If PRs were not used would the very-insistent suggester still find other avenues for their so-called suggestions? Any process - PRs included - adopted by a team with unhealthy power dynamics will be brought down by those dynamics. PRs aren't more or less susceptible to this.

Two other problems deal with PR size: large PRs are difficult to meaningfully review and long-lived branches tend to have more merge conflicts. Indeed these problems can come up, and if this is a sticking problem for your team it can be time to experiment with something else. In order to prevent this problem, branches should be very small and very short-lived. Indeed, plenty of features take a lot of code to implement, so the strategy is to PR frequently with the smallest possible changes at any point. Adopting a culture in the team to integrate changes at a fine-grained level like this is important not only if you're using PRs. In fact, any process will benefit from frequent integration. In fact, if you happen to be on a team where each engineer prefers to work on large, discrete features in isolation with infrequent integration, no process will help you here.

If you're frequently engaging reviews then - as you should be doing whether or not you're using a PR process - you're solving two other issues brought up with PRs: the feedback loop is going to be much shorter and the reviews aren't only going to be coming in at the end of the dev cycle. Indeed, opening small PRs that are easily-reviewable thoughout the dev cycle ends up being a good way to engage colleagues on feedback. The other half of this is reviewer etiquitte - our teammembers need to be diligent about providing timely, meaningful reviews. If I'm only ever asking them to review a few files though, I find that this ask isn't just easy but indeed one to which my colleagues are eager to respond.

Continuing to adopt a healthy team culture will solve more of our problems with PRs and, since the issues with team culture aren't actually about the PR process, will help to solve the same problems in any non-PR processes. With frequent, small PRs that are easily and eagerly reviewed, there is not a PR lag which results in more work being in progress. Indeed, as colleagues I find that we do want to give good feedback, so having removed the burden of giving feedback more often than not I find that I get that feedback. It certainly can be a problem with PR processes that they lead to superficial reviews, but any process with reviews is susceptible to this in different ways. Whatever your process, identify how it might discourage meaningful reviews and address that. Oh look, we've crossed off another bullet!

On that point, I'm also not sure that a PR process necessarily leads to more hurt feelings than any other process. Any negative emotions I get from PRs are, in my experience, the same as what come up from other points during development. They're mitigated by communicating with colleagues; keep in mind that the PR comments section isn't the only communications channel available to us! So i'm also curious why it's felt that PRs discourage two-way communication: if you're not communicating enough with colleagues then you should ... communicate with them? If they rebuff you this is a larger contention for a broader group. Certainly if you identify a PR process as necessarily causing this in the team, by all means try something else. I just mean to be skeptical that the PR process is really the root cause of this?

I've left two negative points about PRs that I think are actually good points. First, I strongly concur that it is sometimes easier to fix a bug than to explain it. Does your team care about documenting fixes? If not, and if the team is comfortable with members pushing fixes with some other process, then I can certainly see how a PR process is an impediment. I don't think most teams are like this though; personally I very much value documenting bugs and their fixes, and I think that bugfixes are the best problems to get a second set of eyes on. In this case, the explanation issue isn't necessarily a problem with a PR process seeing as you're going to need to provide the documentation to someone anyway. It's admittedly difficult for me to imagine a scenario where I want to push a bug fix wihout having explained the problem.

I have observed it before that a PR process has discouraged continuous refactoring, though to be clear this is a problem with any process which includes a manual review, even the pair programming alternative from Dave Farley. The solution I have is the same as can be applied with any non-PR process, which is to encourage refactor and carve out a space for them. In a pair programming process, maybe allow refactors to just be pushed. In a PR process, I encourage my teams to allow "boyscout" PRs, which I named after the Boy Scout Rule. The team treats these PRs specially and celebrates members who open such PRs. The broader point is that any process adopted by a team can encourage negative behaviors, so no matter the process your team should identify those patterns and alter the process to mitigate, eliminate, or reverse those.

Why PRs Are Good

As we can see, most of the problems brought up about PR processes aren't really problems with PR processes, but problems with communication, interpersonal dynamics, and the like. The same sorts of problems exist in any process, and while they might express themselves differently given different processes, I don't know that you can say they're worse in any particular process. This is a problem I have with a lot of these hot takes: they would be more valuable if they actually addressed the root cause of the experiences described.

The same should be true of any praise; if I want to convince you that PRs are good I should cite some unique advantages of them. I could say that I like PRs because they reinforce a code review culture, or because I like getting reviews from colleagues, but those are benefits of mandatory reviews and not PRs. What are the specific things that a PR process is good for? Well, this is easier asked than answered. Try to come up with a list of benefits in your head, then go over those and see if they're really benefits of pull requests specifically?

I think the unique benefits then are all going to be related to tooling in some way. Most development tools familiar to most engineers are set up really well to work with pull requests. GitHub, GitLab, BitBucket, Azure DevOps, and plenty of others have really robust PR interfaces. Take any popular IDE and it's bound to have some level of integration with these PR features. The tooling and familiarity barrier with PRs are both very low, and this is an obvious benefit. If your team is looking to some way of doing mandatory, pre-integration reviews, pull requests are fantastic because of the ease of adoption.

The tooling benefit extends to integrations with other tooling activities, too. Pull requests and, crucially, the tooling which supports them, are great at isolating specific changes and relating them to issue tracking, build automation, and documentation repositories. Being able to integrate so many parts of the tooling with the tooling that supports the code review process is a huge advantage over alternatives.

Tooling is necessary to support PRs, I think. I'm not quite sure what a PR process without PR tooling would look like (but comment below if you've got thoughts on that). The tooling itself then has unique advantages facilitating communication and knowledge sharing. In certain contexts asynchronous communication is required, and if that applies to your team then pull requests offer a pretty good mechanism for async communication on the code review or even general convos directly related to the codebase. PR tooling also makes it incredibly easy for engineers or other observers to see what's going on in the system and the development process. Indeed there are ways other than PRs to achieve this, but the unique offer is the ease of getting all of it.

One of my favorite benefits to PRs is a documentation aspect. When I open a PR I have to write out what I've done, then any conversation related to what I've done is written out in the PR until the end of time. Or, at least, until the end of my repository's life. It's a great way to be able to retrace steps later to get both the what and the why. PRs aren't the only way but, I think, the best way to achieve that benefit, at least with the current state of our tooling.

Should I Do PRs or Not?

If you like them then use them, and if you don't like them then don't do them, by all means try something else! I want to get across that the bulk of "PR bad" articles floating around the internets aren't a meaningful contribution to the discourse. If the issue is with mandatory pre-integration reviews, then pin the blame there. If the issue is with interpersonal or team dynamics, then address the solutions in that space. That sort of analysis is what helps us make proper decisions about the conduct of our work. This popularity of terrible arguments creates a mini cacophony of almost-entirely-misdirected anti-PR sentiment - who is that good for?

PRs are fine. In fact, they have plenty of benefits which they share with the basic practices they're built on: feature branching, mandatory review, pre-integration review, and the like. PRs uniquely excel at some of those shared benefits, and the state of tooling decidedly sets PRs apart as the best way to get those benefits. Do those patterns and benefits apply to your team? Maybe so, maybe not. If you're on the fence about PRs or any other review process, I can't recommend enough Martin Fowler's writing on the subject, in which he does a great job framing PRs in the context of his branching patterns. I'd be curious in a similar analysis on review patterns, come to think of it.

Maybe in 10 years the next GitHub will have come out and revolutionized tooling in some way that something other than PRs are so easy to adopt. Maybe it will offer the same benefits in integration with other tooling, communication, visibility, and documentation, or maybe it will offer different benefits that make it a great thing to use. That's a fun thing to consider - do I actually need the benefits of PRs? Am I being impeded by feature branching or mandatory review?

Thing I Made: Question of the Week

Wed, 11 Feb 2026 00:00:00 Z

I like asking dumb questions each week as a teambuilding thing; I made a site to show a random question each week: QuestionOfTheWeek.io!

Quick & Dirty Sequential IDs in MongoDB

Wed, 01 Nov 2023 00:00:00 Z

That Mongo doesn't natively support sequential IDs is one of the many knocks against it. Sure, you should be using GUID IDs in Mongo, but suppose you're working on a microservices conversion and you have a legacy mainframe that needs to be able to know what your objects are? If you're content just using Atlas, you can create a counter collection and add a trigger for auto-incrementing IDs fairly easily.

Suppose however that you can't use a pure Atlas solution - you'll need to implement this logic yourself in your own code. If you happen to be working in a microservices environment you have concurrency concerns - there might be multiple shards of your database and/or multiple replicas of your microservice.

Is a primary key generator really the sort of thing you want "quick and dirty"? Probably not. Am I doing it in prod? Yes.

Updating a counter collection

As a prerequisite, ensure you have the Mongo driver:

go get go.mongodb.org/mongo-driver/mongo

Just as Mongo's tutorial for Atlas recommends, we'll implement a counter collection. This collection will contain one document per "kind" of ID we need to generate. If you have just one object that needs sequential IDs, then you'll only have one document in this collection. We'll represent this collection document with a struct. It only needs one field, sequence, which will represent the latest ID generated:

type MongoCounterDocument struct {
    sequence int `bson:"sequence"`
}

The ID of each document in the collection should be a string you hardcode or keep in a settings file (such as "personIdCounter"), and doesn't need to be in the document struct. Instead, we'll encapsulate that in a generator struct along with a reference to the collection:

type MongoIdGenerator struct {
    counterCollection *mongo.Collection
    counterDocumentId string
}

To implement the functionality to generate the next ID, we'll use the FindOneAndUpdate operation to increment sequence and return the new ID to us. We can specify a couple options here: we can upsert the document so that it will be created automatically if one isn't there for us (useful for integration tests), and we can specify that we want the operation to read and return us a copy of the document after the update has taken place.

func (generator *MongoIdGenerator) GetNextId() (int, error) {
    filter := bson.M{"_id": m.counterDocumentId}
    update := bson.M{"$inc": bson.M("sequence": 1)}
    options := options.FindOneAndUpdate().SetUpsert(true).SetReturnDocument(options.After)

    var updatedDocument MongoIdCounter

    err := m.counterCollection.FindOneAndUpdate(context.TODO(), filter, update, options).Decode(&updatedDocument)
    if err != nil {
        return 0, errors.New("Unable to update Mongo id counter collection.")
    }

    return updatedDocument.sequence, nil
}

FindOneAndUpdate is atomic and shouldn't have any concurrency concerns so long as you do not shard the counter collection.

But I don't want to have to hit Mongo every time I want a new id

Huh, you and I think alike, I didn't either! To get around this, we can have our app generate multiple IDs each time it hits Mongo and use these IDs until it runs out locally.

With this approach you have the concern that if your app is spinning up and tearing down too frequently, you'll start losing IDs in the mix. There are various strategies to mitigate this, such as retrieving a small number of IDs from Mongo each time or persisting the cache of IDs, but I'm not going to get into those here.

We'll add nextId and maxId properties to the generator object, as well as an increment field to specify how many IDs we should generate each time:

type MongoIdGenerator struct {
    counterCollection *mongo.Collection
    counterDocumentId string
    incrementBy       int // [tl! ++]
    nextId            int // [tl! ++]
    maxId             int // [tl! ++]
}

We'll add a func to instantiate this at startup. It'll be important that your app only has one of these objects per "kind" of ID you need to generate:

func SetupMongoIdGenerator(collection *mongo.Collection, documentId string) *MongoIdGenerator {
    return $MongoIdGenerator{
        counterCollection   : collection,
        counterDocumentId   : documentId,
        // Adjust this up or down depending on how many IDs you want to generate at once:
        incrementBy         : 25,
        nextId              : 0,
        maxId               : 0
    }
}

And we can update our GetNextId function to consult Mongo or not if nextId equals maxId:

func (generator *MongoIdGenerator) GetNextId() (int, error) {
    if generator.nextId == generator.maxId { // [tl! ++]
        filter := bson.M{"_id": m.counterDocumentId}
        update := bson.M{"$inc": bson.M("sequence": generator.incrementBy)}
        options := options.FindOneAndUpdate().SetUpsert(true).SetReturnDocument(options.After)

        var updatedDocument MongoIdCounter

        err := m.counterCollection.FindOneAndUpdate(context.TODO(), filter, update, options).Decode(&updatedDocument)
        if err != nil {
            return 0, errors.New("Unable to update Mongo id counter collection.")
        }

        generator.nextId = updatedDocument.sequence - incrementBy // [tl! ++]
        generator.maxId = updatedDocument.sequence // [tl! ++]
    } // [tl! ++]

    return updatedDocument.sequence, nil // [tl! --]
    generator.nextId += 1 // [tl! ++]
    return generator.nextId, nil // [tl! ++]
}

We do have a concurrency concern here though - we want to ensure nextId and maxId are only being accessed one at a time. We can use a mutex in the generator for this. Update the generator:

type MongoIdGenerator struct {
    counterCollection *mongo.Collection
    counterDocumentId string
    incrementBy       int
    nextId            int
    maxId             int
    mutex             sync.Mutex // [tl! ++]
}

And add the following two to the beginning of GetNextId:

func (generator *MongoIdGenerator) GetNextId() (int, error) {
    generator.mutex.Lock() // [tl! ++ **]
    defer generator.mutex.Unlock() // [tl! ++ **]

    if generator.nextId == generator.maxId {
        filter := bson.M{"_id": m.counterDocumentId}
        update := bson.M{"$inc": bson.M("sequence": generator.incrementBy)}
        options := options.FindOneAndUpdate().SetUpsert(true).SetReturnDocument(options.After)

        var updatedDocument MongoIdCounter

        err := m.counterCollection.FindOneAndUpdate(context.TODO(), filter, update, options).Decode(&updatedDocument)
        if err != nil {
            return 0, errors.New("Unable to update Mongo id counter collection.")
        }

        generator.nextId = updatedDocument.sequence - incrementBy
        generator.maxId = updatedDocument.sequence
    }

    generator.nextId += 1
    return generator.nextId, nil
}

That should be that! Here's the final code all together:

type MongoCounterDocument struct {
    sequence int `bson:"sequence"`
}

type MongoIdGenerator struct {
    counterCollection *mongo.Collection
    counterDocumentId string
    incrementBy       int
    nextId            int
    maxId             int
    mutex             sync.Mutex
}

func SetupMongoIdGenerator(collection *mongo.Collection, documentId string) *MongoIdGenerator {
    return $MongoIdGenerator{
        counterCollection   : collection,
        counterDocumentId   : documentId,
        incrementBy         : 25,
        nextId              : 0,
        maxId               : 0
    }
}

func (generator *MongoIdGenerator) GetNextId() (int, error) {
    generator.mutex.Lock()
    defer generator.mutex.Unlock()

    if generator.nextId == generator.maxId {
        filter := bson.M{"_id": m.counterDocumentId}
        update := bson.M{"$inc": bson.M("sequence": generator.incrementBy)}
        options := options.FindOneAndUpdate().SetUpsert(true).SetReturnDocument(options.After)

        var updatedDocument MongoIdCounter

        err := m.counterCollection.FindOneAndUpdate(context.TODO(), filter, update, options).Decode(&updatedDocument)
        if err != nil {
            return 0, errors.New("Unable to update Mongo id counter collection.")
        }

        generator.nextId = updatedDocument.sequence - incrementBy
        generator.maxId = updatedDocument.sequence
    }

    generator.nextId += 1
    return generator.nextId, nil
}

Reclaim Your Agile: The One Clever Trick Agile Coaches Don't Want You to Know

Wed, 13 Dec 2023 00:00:00 Z

This is the one clever trick that Agile coaches don't want you to know: you can actually become an Agile team. There is actually a light at the end of this tunnel. Okay, maybe I'm being a bit fanciful. However, there is one thing that I've done on several teams now that has led them to be able to reclaim their development process and become more Agile. With this one clever trick your team can create and dictate its own process. Again, perhaps a bit fanciful, but like I said this has worked for me and it might be beneficial to you too in some way.

I've heard a lot of my colleagues say that they are skeptical of, or even dislike, Agile. In almost every single case, I think they were actually talking about Scrum. It's a forgiveable mistake, these two words are largely used interchangeably these days. However, I encourage you to read the Agile Principles and any piece of the Scrum Guide and tell me that they're the same thing. They're not. Scrum is almost entirely antithetical to Agile ("almost" in this case being around the 99.99% range), designed to satisfy corporations who aren't quite ready to give up their waterfall yet. Agile is the emphasis on early and continuous delivery of value to the customer, whose experience is the first and foremost concern, and the prescription that each engineering team must create, own, and direct its own process.

I do think that the Agile principles are the best set of principles we've come up with so far to try to understand how to best develop software, and I want to tell you about this way I've found that's helped a team or two become more truly Agile.

It started by accident actually, many years ago. I had just been hired by Dalex Livestock Solutions, a small agricultural firm, to my first properly employed position (I was an independent consultant before that). They were a full 5 people strong, and we had a meeting in Minneapolis the week I was hired to chart the future course - they wanted to invest in developing new products (that's why I was brought on). Being a 4-person company up to that point, they had never thought of their development process. However, it was clear to them that if they wanted to scale, they'd need some process. They said they wanted Scrum, and I obliged them and went along with it. There was one hiccup though - none of the non-engineers there understood a single thing that the engineers were saying about scrum. The vocabulary was a stumbling block - terms like "scrum meeting" and "sprint" were more puzzling than helpful. And funnily enough, having a former rugby captain on the team didn’t make translating these terms any easier!

In a stroke of creativity (though I can't recall exactly whose), we decided to give a fresh spin to these scrum terms. Being in the agriculture domain, we thought of using old rancher terms to describe the concepts. This caught their attention, and we ended up using these words the entire five years I was there. We didn't have "sprints", we had "cattle drives". No "sprint planning", that was a "saddle up"; "retrospectives" were "circling the herd"; instead of a "backlog" we had a "haystack"; and so on. Not one of the words we used are in the scrum guide. Over the years, when we felt we needed to make a change, we would never be bogged down with questions like "What does the Scrum guide recommend?", but instead we'd ask "Well, what actually is a 'Cattle Drive'?". After just a couple of years we had drifted away from Scrum entirely and into something of our own - something actually Agile.

My takeaway was that by redefining and taking control over our language, we were able to expand on that to take control of our process, and I think this works - in different ways - for teams of any size. Maybe this approach will work for you, maybe it won't, but it seems to me like it's a chance for many teams to be able to fight against their Scrums and make it something actually Agile, so I want to share some ideas for how you can actually do this.

Just Start With Words

Some industries are better suited to this than others, but the central, most important, idea is to use your own words for each part of your process. I gave you some agriculture examples, but you can find examples in your own industry. Or suppose you work in insurance or banking where the domain is very dry. Do you all enjoy superheroes or Star Trek? Use Star Trek words! Find something that unites your team.

At the agricultural firm, I started with all of the words but you certainly don't have to. Start with one element or ritual and make a simple change. Given the "backlog" was the "haystack" to us, I just renamed a list on the kanban board. "Isn't that funny!" they'd exclaim. Reinforce this change with the way you speak, custom Slack emojis, or anything else to give your team a special color. When someone calls it the "backlog" errantly, call them out: "Don't you mean the 'haystack'?"

Everyone - even business - will be on board with you if you pick your first target right. They'll be impressed how your team is taking ownership over its work. If you work at a place that's too corporatish, brag about how the name change helps align the synergy of your deliveries or something of the sort.

Keep it Fun

Don't jump right in trying to redefine everything, and don't show up at retro one day ready to rock with "Hey team, I've redefined all of our words - isn't this great?". No, it's not great, not for your teammates. Make a suggestion - maybe start talking to one or two colleagues to start: "Wouldn't it be fun to rename this-or-that process to (insert word here)?"

You want to create an environment where everyone is looking forward to discussing words. When your team, probably jokingly, chooses a new word for a process - stick to it. Use it everywhere and, when appropriate, correct coworkers when they use the "wrong" word. This isn't something rigid I'm trying to prescribe, and in practice it's a natural way to have a bit of a laugh among colleagues.

You might have coworkers who wouldn't want a change in name - they might be too uptight, too set in their ways, or too anything else. More likely than anything, they might just not understand you or the idea. Go slow, keep it light, and emphasize the benefit of team cohesion. You'll win over even the most ardent colleagues.

Press the Advantage

Expand that to other parts of the process, and engage everyone on your team. At the end of a retrospective, suggest a new thing to rename. Can your team take five minutes to brainstorm a new name for this or that meeting? Sure they can!

Engender a lively and friendly debate - which word is the more "agricultural" or "Star Trek" or whatever word for this concept? Which one does the whole team like? Remember that you don't need to like the words, the team does. Keep them happy.

Before too long, you'll have your own words for everything. Your team will own its own language, and language is everything. Anybody who will want to redirect your team and change its processes will need to understand your lingo now.

Entrench Your Position

Everyone who deals with your team needs to know its lingo now. You want us to make a change to our microservice? You better say its name right! That's correct, the microservice is called "Khan" because engaging with it will entrench your team in a days-long battle deep in a dangerous nebula that will drain everyone's willpower until one of you can assert a week dominance over the other and finally destroy it (such is the world of microservice development).

Do not relent. Your language is your power. If only your team speaks this language, then you've got a special flavor and nothing else. If your team plus everyone interacting with your team speaks this language, then you've got a real power.

Inevitably, you'll have colleagues on other teams get confused or maybe frustrated at all your different words. Avoid confrontation and fall back to the old words if you need to in these contexts, but once again you'll bridge this gap if you keep it fun and take it slow. Don't create a glossary on your team's Confluence page, and even if you do don't refer folks from other teams there. Just use the new lingo in casual communication and laugh with yourself and your team when others find it strange. You'll win them all over too, with time and good nature.

Start Changing Things

Now that you have real power, use it. At retrospectives, stop asking what the right name for something is - you've already named everything. Start asking "What actually is a 'Cattle Drive'?".

At first, your teammates will be confused - there's no longer a Scrum guide to fall back on! Nobody knows what a 'Cattle Drive' is - it's not in the Scrum guide, it's not in any guide! This is when the real thinking comes out. I guarantee, buried deep in your team is a wealth of knowledge, and by removing the onerous cap of the Scrum guide you'll discover it.

The hope at this point is that by having divorced your team from the language of Scrum, you can change the process by talking about your new words instead of talking about process. This is another area that can't be rushed, and you might be surprised by what the first candidate is. Before too long, you'll surely have a sprint (or, perhaps a cattle drive) where everyone on the team has been impeded by a particular process. Motivate a conversation here about this process, but use your new word and avoid talking about process.

Keeping your team entrenched in the new language keeps the team away from considering "what does the Scrum Guide say about this process?" because the Scrum Guide says nothing about haystacks or saddle ups. Make the first process change on a process that everyone agrees needs to be changed when they are all agreeing that it needs to be. After the first change, others will be easier.

Incorporate Ideas

There will be a lot of ideas suggested. "'Circling the herd' should be X" or "A 'stampede' should be Y". Create a framework for trying ideas: agree to try ideas for a time, hold rigidly to them, then after that period of time reevaluate. Did the idea work? Keep it. Did it fail? Discard it. Something in-between? Adapt. Again: you don't need to like the ideas here. Rather, someone does, and that's the indicator of success.

When new ideas are incorporated, your process changes. Don't bill this as a process change. Instead, it's an update to a word that others don't fully understand anyway. After all, other teams are talking about your team in terms of "cattle drives" or whatever. They don't know what that is - you can tell them, and you can tell them different things months apart, just make sure you're approachable - you'll be surprised how far you can stretch your process, and somehow others will be able to work with you because humans tend to be rather adaptable. Don't worry here, just own your process. Whatever your words are, they're your team's words.

Some might yet object to new ideas because the almighty Scrum Guide doesn't contain it. You can jokingly assert that the Scrum Guide says nothing about your new word(s), but your team should be sufficiently motivated to make changes when they're considering them. If you lose a battle that's okay, you're in it for the long haul to win the war. Consensus and buying in is crucial to making fundamental changes.

Keep the Team in Control

Inevitably, outsiders will try to push on you. They'll try to compare a Cattle Drive to a sprint or other lame things like that. Don't let them. Ensure everyone on your team is on the same page so they can answer uniformly: "A 'cattle drive' is a set time in which we can develop a small set of features that directly benefit the farmers.". They might say that that sounds like a sprint - insist it's not and redirect back to how it satisfies the customers or some corporate metric.

Always redirect to your team. Start from the position that your team owns its language; anything different is entirely foreign to you. Don't speak about Scrum, just speak from the position of your team. This is how we do things, this is the way. A random person can go pretty far in an office if they're wearing a tie and carrying a clipboard, right?

Become Agile

You now own your own processes, and your team has a regular way of changing its process to respond to the environment it's in. You're now Agile, but remember it, keep it up, and don't relent.

When your process needs to change, change it but stick to your team's vocabulary. Deliver customer value early and often, and find ways to get your team as close to the customer as you can. If your engineers can regularly interact directly with the customer, that's the best. You can always fall back on the Agile manifesto and principles to give some guidance on how you might approach sharpening your process, but remember that these are only a guide and your team needs to discover, implement, and own its own process and methodologies for its own environment.

When in Doubt, Rip it Out

Most teams, especially those coming from Scrum, have too much process. A lot of them also don't have enough communication - even teams that are mostly in-office with each other. When it comes time to reevaluate a practice that you and your team have, consider ripping it out - stop the practice. It might seem unintuitive, but a lot of things in our industry are. If a particular element of your process is getting in the way somehow, maybe just remove it and get it out of the way.

Can the problem be solved with more communication? If you're in office do you need to adopt a better culture of collaboration throughout the day? If you're distributed, do you need to encourage more ad hoc video calls? If you can reduce the barrier to communication and encourage as much communication as possible, surely you'll find no more need for meetings to "align stakeholders" or the like.

Sometimes it can be difficult to rip out a process even when it's clear it needs to go. You might have colleagues who are too familiar with this process, or maybe colleagues who think they don't want to communicate that often. This is a collaborative process, and they need to be comfortable and along for the ride too, but focus on the benefit and keep the conversations focused on what your team and its product need. You can always lead by being the change you wish to see - if more video calls are needed, start hashing your problems out over video call with your colleagues.

Drive Change From Product Quality

We're in the industry of developing software products, so the working of the software that we deliver to our customers is always, invariably, the primary measure of our success. Our processes need to be honed to delivering valuable, functioning software, and so the primary impetus to change our processes should be from the metrics of our product's quality. You (probably) don't need two hundred different charts and graphs measuring every last metric you can think of, but your team does need to have agreed metrics and definitions of quality.

The greatest opportunity for change is when one of these metrics is underperforming. It's easy to orient the team around wanting to solve this issue, and often quality issues are largely - or at least in part - a fault of something in the process.

Involve the Customer

The gold standard for a software engineering process is, in my experience and humble opinion, to have the customer right beside you as you develop your software. I spent the first several years of my career as a consultant developing LOB software, and I had the opportunity to develop software in this way. Not only does it eliminate a whole swath of processes just to get me customer feedback; I have the most reliable feedback and best customer experience as a result.

This is not realistically achievable for all teams though - I work in ecommerce right now and I can't imagine what it would look like to have actual customers next to me. Would they even care? Instead, we have a team that collects customer feedback and records interviews, and I can go and read or watch those whenever I need - we've developed a process that works for us and gets me as close to the customer as possible.

Always consider the customer as you develop your processes - where do they fit in and how do you and your colleagues know who they actually are? If you can realize efficiencies here, you can probably eliminate significant parts of your process.

You Won't Ever Finish

The only constant, especially in our industry, is there are no constants. In a way this can be demoralizing; we can never create the one perfect bit of software or the one true process. But that's the environment we operate in, and we need to accommodate it. That's part of what makes Agile necessary to me, or at least that you and your team need to be empowered to develop and own their own processes.

Your process is never going to be set and it will constantly change. Keep it changing, and keep ownership over your process. To get back to my thesis here, keep the language yours. Don't let others impose their ideas or their own language on you, play uno reverse. When things need to change, this is a process to introduce new fun words or to recontextualize them to fit new problems. This process of using your own language makes it all the more fun and meaningful for teams, and will often lead to better - maybe, more agile - results.

Conclusion

There are no silver bullets, but this process has worked incredibly well for me, and it might for you as well. At the heart of it, the idea is to get your team to own its own language and exploit that as an advantage to have its own process. Confuse your superiors with funny words and slip your own process in below that.

You might also think that you don't actually want your team to be 'Agile'. If you feel that way, are you actually talking about Scrum? Agile and Scrum are very different. Read the Agile Manifesto then the Scrum Guide. Do these seem like they're saying the same thing? They're not - Scrum is a completely disingenuous bastardization of Agile for the benefit of corporate environments. Agile, on the other hand, is a loose set of guides that don't prescribe a specific system - they want to empower your team to develop its own system.

That's what I want to get at with this article. I want to give you one way, doubtlessly among many ways, to reclaim your process. Reclaim your 'Agile' and form something that works for you. I do think that the Agile principles must be followed: deliver value early and often, stay close to your customers, and always reevaluate.

On top of all of this, keep learning and keep your mind open to new ideas in process. There's many different ideas - some good, many iffy - about how to develop software. Ultimately, the best process for your team isn't written anywhere. Only by trying new things out can your team find a process that works the best.

I hope all of your future cattle drives - or whatever word you settle on - are filled with productive enhancements for the users' benefit.

Roll Your Own C# Results

Fri, 03 May 2024 00:00:00 Z

The result pattern is one of my favorite functional patterns to use in the C# world, and I don't think I'm alone on this. If you try to Google "result pattern" to try to find a Wikipedia article or something to link to as a definition of the pattern, you'll find a bunch of articles and repositories explaining the pattern in C#. Maybe that's just my search results, but this pattern is such a fundamental concept in functional programming that I imagine most folks regularly using functional languages might scoff at the suggestion that this should be elevated to the level of being a proper design pattern.

I'd like to consider it a proper pattern though. Typically and traditionally our C# programs handle errors by throwing and catching exceptions. Exceptions are abused in this case though - they're intended for truly exceptional events, occurrances which I don't expect to happen and which absolutely necessitate the interruption of my control flow. If your API's validation layer found a validation error in a request body, that's not exceptional because I expect clients to send invalid requests time-to-time. If my application wasn't able to parse a JSON response from an external resource, that's not exceptional because I know that data outside my application is never 100% trustworthy.

Nonetheless, we have all tended at one point to treat them as exceptional and throw exceptions in these cases. This can lead to bad code in a number of ways - I might forget to catch an exception at some point, I might not consider all of the error states I need to support for an operation, or my application might be made slow by excessively throwing exceptions. Instead, it's proper to only throw exceptions when the application has entered a state which we either consider impossible or from which I know it is impossible to recover. In the other 99% of cases, I want to return a result object representing the success of the operation. This is the result pattern.

Existing Implementations in C#

There's a number of libraries you can find on GitHub to support this in C#. Two well-liked ones are ardalis' Result library and altmann's FluentResult library. You'll notice that these libraries support a method returning one of a set of options - Success, Failure, Invalid, etc. The Computer Science term for this is a tagged union, also called a discriminated union or an algebraic data type. These types are foundational to functional programming and the basis of the result pattern, but not yet supported in C#. The OneOf library is an attempt to provide a generic form of these in C#, with which you can implement a result pattern.

(Note that the C# team is actively working to include discriminated unions in the langauge!)

Those and others are all fine libraries, and while each of them can solve specific scenarios quite well none of them are a one-size-fits-all approach. If you just need "success" and "failure", then you probably just need one of these libraries or you can write your own class which has an IsSuccess flag. However, in a lot of cases we have more than one kind of result we might need to return. A validation layer might have multiple levels of validation (valid, valid but for an old version, invalid), or a persistence layer might want to distinguish between multiple states (found all the objects you queried, the query succeeded but yielded no results, the query itself was invalid, the parameters were invalid, the database crashed, etc).

Among the libraries which support more than 2 result states, OneOf is the best option here, but you'll find that you need to use T0, T1, T2, and so on in your code; this is due to the generic nature of the library. To get around that you'll need to implement your own classes on top of it, which may be what you need but I'd stop here and ask: why do we not roll our own? No, it's not complicated, in fact it's very easy and probably involves no more code than if you were to extend OneOf anyway. Then of course you get the benefit that your solution is tailored to your problem and your team can extend and maintain it as your application evolves.

There's a lot of ways to get to where we want to be, and one might be more preferable for your case. I'm going to demonstrate two: first a "quick and dirty" implementation using one class with a result flag, and second a (more preferable, in my opinion) solution using multiple classes and implicit casting. For both examples I'll implement a database query result which for the purposes of this article can either be Found, Empty, ClientError, or ServerError. If it is Found, it will contain the object which we queried; if it is one of the two errors, it will contain a string description of the error.

First Approach: One Class, Result Flag

The simplest way to get off the ground with a result object is to implement one class with an enum flag signifying what type of result it is:

public enum DatabaseQueryResultType
{
    Found,
    Empty,
    ClientError,
    ServerError
}

public class DatabaseQueryResult
{
    public DatabaseQueryResultType Type { get; set; }
}

This DatabaseQueryResult will also need to hold the value of the result for the Found case, as well as the error description for either of the error cases. These need to be nullable as we can't know what kind of result this is until runtime. The class also needs to be generic as we need to support different types of values:

public class DatabaseQueryResult<T>
{
    public DatabaseQueryResultType Type { get; set; }

    public T? Value { get; set; }

    public string? ErrorDescription { get; set; }
}

This can be used as-is now:

public DatabaseQueryResult<IEnumerable<Item>> GetAllItemsFromDatabase(...)
{
    try
    {
        var result = ContactDatabaseSomehow(...);

        if (result is IEnumerable<Item> items)
        {
            return new DatabaseQueryResult<IEnumerable<Item>>
            {
                Type = DatabaseQueryResultType.Found,
                Value = items
            };
        }
    }
    catch (/*exception indicating client error*/)
    {
        return new DatabaseQueryResult<IEnumerable<Item>>
        {
            Type = DatabaseQueryResultType.ClientError,
            ErrorDescription = exception.Message
        };
    }
    catch (/*exception indicating server error*/)
    {
        return new DatabaseQueryResult<IEnumerable<Item>>
        {
            Type = DatabaseQueryResultType.ServerError,
            ErrorDescription = exception.Message
        };
    }

    return new DatabaseQueryResult<IEnumerable<Item>>
    {
        Type = DatabaseQueryResultType.Empty
    };
}

public void ShowItems()
{
    var result = GetAllItemsFromDatabase(...);
    switch (result.Type)
    {
        case DatabaseQueryResultType.Found:
            foreach (var item in result.Value!)
            {
                Console.WriteLine(item.ToString());
            }
            break;
        case DatabaseQueryResultType.Empty:
            Console.WriteLine("No items found!");
            break;
        case DatabaseQueryResultType.ClientError:
        case DatabaseQueryResultType.ServerError:
            Console.WriteLine($"Error! {result.ErrorDescription}");
            break;
        default:
            throw new Exception("This is an actual exceptional state since the code _should_ never get here.");
    }
}

This works but oh goodness the verbosity! The most glaring verbosity issue, at least as I perceive things, is all the code we spend instantiating a new DatabaseQueryResult each time. Imagine if every single method in the persistence layer was littered with this code - not only is it taking up a lot of space but we're relying on the implementor to perfectly instantiate the DatabaseQueryResult each time - they need to know to always set the Value if it's found, to always set the ErrorDescription for an error, and so on. We can implement a couple of static instantiator methods to guard this and reduce the size of the implementing code:

public class DatabaseQueryResult<T>
{
    private DatabaseQueryResult() { }

    public DatabaseQueryResultType Type { get; private set; }

    public T? Value { get; private set; }

    public string? ErrorDescription { get; private set; }

    public static DatabaseQueryResult<T> Found(T Value) => new()
    {
        Type = DatabaseQueryResultType.Found
        Value = value
    };

    public static DatabaseQueryResult<T> Empty() => new()
    {
        Type = DatabaseQueryResultType.Empty
    };

    public static DatabaseQueryResult<T> ClientError(string description) => new()
    {
        Type = DatabaseQueryResultType.ClientError,
        ErrorDescription = description
    };

    public static DatabaseQueryResult<T> ServerError(string description) => new()
    {
        Type = DatabaseQueryResultType.ServerError,
        ErrorDescription = description
    };
}

public DatabaseQueryResult<IEnumerable<Item>> GetAllItemsFromDatabase(...)
{
    try
    {
        var result = ContactDatabaseSomehow(...);

        if (result is IEnumerable<Item> items)
        {
            return DatabaseQueryResult<IEnumerable<Item>>.Found(items);
        }
    }
    catch (/*exception indicating client error*/)
    {
        return DatabaseQueryResult<IEnumerable<Item>>.ClientError(exception.Message);
    }
    catch (/*exception indicating server error*/)
    {
        return DatabaseQueryResult<IEnumerable<Item>>.ServerError(exception.Message);
    }

    return DatabaseQueryResult<IEnumerable<Item>>.Empty();
}

That simplifies and safety-ifies (that's not a word) our implementing code a lot. There's still a similar problem in the code which consumes the result object, but to a lesser degree. The switch statement is a bit cumbersome, and it's annoying to have to implement the default case always. Here's the first instance where I can rely on the nature of our specific use case to get some custom value that I might have a bit more difficulty with a library. Although I have 4 result states, for most purposes I only really care about three - did it succeed, did it fail, or is it empty? I can implement two methods - one to check for success and one to check for failure - then use an easier-to-read if/else if/else check to get the behavior I want:

public class DatabaseQueryResult<T>
{
    private DatabaseQueryResult() { }

    public DatabaseQueryResultType Type { get; private set; }

    public T? Value { get; private set; }

    public string? ErrorDescription { get; private set; }

    public bool IsFound() =>
        Type == DatabaseQueryResultType.Found;

    public bool IsError() =>
        Type == DatabaseQueryResultType.ClientError
        || Type == DatabaseQueryResultType.ServerError;

    public static DatabaseQueryResult<T> Found(T Value) => new()
    {
        Type = DatabaseQueryResultType.Found
        Value = value
    };

    public static DatabaseQueryResult<T> Empty() => new()
    {
        Type = DatabaseQueryResultType.Empty
    };

    public static DatabaseQueryResult<T> ClientError(string description) => new()
    {
        Type = DatabaseQueryResultType.ClientError,
        ErrorDescription = description
    };

    public static DatabaseQueryResult<T> ServerError(string description) => new()
    {
        Type = DatabaseQueryResultType.ServerError,
        ErrorDescription = description
    };
}

public void ShowItems()
{
    var result = GetAllItemsFromDatabase(...);

    if (result.IsFound())
    {
        foreach (var item in result.Value!)
        {
            Console.WriteLine(item.ToString());
        }
    }
    else if (result.IsError())
    {
        Console.WriteLine($"Error! {result.ErrorDescription!}");
    }
    else
    {
        Console.WriteLine("No items found!");
    }
}

This takes up much less space and, at least to me, reads much clearer with a better understanding of the intention of the code. Ive seen a lot of implementations stop here, but there's one more nagging bit - null checks! I know that when the result is Found that Value is going to be non-null, and the same for ErrorDescription when the type is one of the error types. It's annoying for the consumer to have to spam ! everywhere when it knows that these values are not null, and it's bad practice anyway to not encapsulate this behavior.

In this case I can implement the IsFound and IsError methods to return the non-null values with a sort of modified try pattern, which would also allow me to hide the internal state of the result object from the outside world. This is much better design, and it pains me when this simple step isn't taken. I'll just provide a simple implementation of these two methods, yours might involve more checks depending on your case:

public class DatabaseQueryResult<T>
{
    private DatabaseQueryResult() { }

    private DatabaseQueryResultType Type { get; set; }

    private T? Value { get; set; }

    private string? ErrorDescription { get; set; }

    public bool IsFound([NotNullWhen(true)] out T? value)
    {
        value = Value;
        return Type == DatabaseQueryResultType.Found;
    }

    public bool IsError([NotNullWhen(true)] out string? description)
    {
        description = ErrorDescription;
        return
            Type == DatabaseQueryResultType.ClientError
            || Type == DatabaseQueryResultType.ServerError;
    }

    public static DatabaseQueryResult<T> Found(T Value) => new()
    {
        Type = DatabaseQueryResultType.Found
        Value = value
    };

    public static DatabaseQueryResult<T> Empty() => new()
    {
        Type = DatabaseQueryResultType.Empty
    };

    public static DatabaseQueryResult<T> ClientError(string description) => new()
    {
        Type = DatabaseQueryResultType.ClientError,
        ErrorDescription = description
    };

    public static DatabaseQueryResult<T> ServerError(string description) => new()
    {
        Type = DatabaseQueryResultType.ServerError,
        ErrorDescription = description
    };
}

public void ShowItems()
{
    var result = GetAllItemsFromDatabase(...);

    if (result.IsFound(out var items))
    {
        foreach (var item in items)
        {
            Console.WriteLine(item.ToString());
        }
    }
    else if (result.IsError(out var description))
    {
        Console.WriteLine($"Error! {description}");
    }
    else
    {
        Console.WriteLine("No items found!");
    }
}

By implementing proper OO standards here, we've encapsulated everything about the operation of the result classes and we're exposing only exactly what's necessary. This makes it trivially simple to implement and I don't need a lot of documentation around here - most anyone could jump into the code here and implement a consumer of this result conforming to the proper standards. As a bonus, you should also move the enum definition inside the class and make it private. In addition, if your consumers care about the distinction between client and server errors, you'll want to implement some way of being able to discern that - extra IsClientError and IsServerError methods would allow consumers to choose whether they care about any error case or if they case about specific error cases.

There's another important point here that often goes overlooked in result implementations: it's impossible for me to get the value of the result without performing a success check at the same time. Often those who encourage using the result pattern in C# focus on the negative aspects of throwing exceptions more than the positive effects of forcing success checks; indeed, this is because a lot of result implementations in C# don't force these checks. However, this is an essential - maybe the essential - part of this pattern. It helps to put you and your team in the pit of success by forcing proper (or at least better) explicit error case handling.

If you take nothing else away from this article, take away the above paragraph: Always use the result pattern to enforce proper error handling.

There's one bit yet that annoys me about this code, and I think we can do better. However, it's going to force us to rewrite our solution here. My problem is the code which produces one of these results - in the empty or error cases, I still need to write out the full return type even though I don't have one! This seems like a minor inconvenience but look at the code above - if your objects are named longer that's a lot of eyesore. Wouldn't it be nice if there was some type inference to write something like:

public DatabaseQueryResult<IEnumerable<Item>> GetAllItemsFromDatabase(...)
{
    try
    {
        var result = ContactDatabaseSomehow(...);

        if (result is IEnumerable<Item> items)
        {
            return DatabaseQueryResult.Found(items);
        }
    }
    catch (/*exception indicating client error*/)
    {
        return DatabaseQueryResult.ClientError(exception.Message);
    }
    catch (/*exception indicating server error*/)
    {
        return DatabaseQueryResult.ServerError(exception.Message);
    }

    return DatabaseQueryResult.Empty();
}

Being clear, that won't compile now. Instead, we'll look at a second approach.

Second Approach: Multiple Classes, Implicit Casting

This approach is going to get us closer to how this pattern tends to be used in actual functional languages, and it's much closer to how I assume we'll be coding once the C# team adds discriminated unions to the language. With this approach, we're going to create one record per each result type, then use implicit casting on a generic base type to allow instances of these records to instantiate the base type. This will allow methods to use the base type as a return type. This base class will contain the flags and values and isFound checks and everything we had before, these extra classes will just be giving us a nicer way to instantiate and think about our results.

Why not use inheritance? Because that would not get me away from having to specify the generic type for all result values. You can certainly implement a result object using inheritance, but I don't think that the majority of use cases benefit from it.

Though we could extend the first approach, I'm going to start from the ground up here for the sake of simplicity. The first thing I'm going to do is to define all of my result types:

public record Found<T>(T Value);

public record Empty();

public record ClientError(string Description);

public record ServerError(string Description);

The other half of this is the base result class, which we can implement with the implicit casts off the bat:

public sealed class DatabaseQueryResult<T>
{
    private DatabaseQueryResult<T>() { }

    public static implicit operator DatabaseQueryResult<T>(Found<T> found) => new();

    public static implicit operator DatabaseQueryResult<T>(Empty empty) => new();

    public static implicit operator DatabaseQueryResult<T>(ClientError clientError) => new();

    public static implicit operator DatabaseQueryResult<T>(ServerError serverError) => new();
}

Now the records which represent our result states already contain all of the information we need, so we don't need to duplicate those properties on the DatabaseQueryResult object. In order to keep our state we can define an object field for our result class to store whichever result state we've gotten. Since our result object is carefully guarding its instantiation and internal state, I don't think we need to worry about type safety here. If you've got a different situation you might need to implement something more robust, though.

public sealed class DatabaseQueryResult<T>
{
    object _value;

    private DatabaseQueryResult<T>(object value) =>
        _value = value;

    // Implicit casts
}

Then we can use type checking to implement our IsSuccess and IsError methods:

public sealed class DatabaseQueryResult<T>
{
    public bool IsSuccess([NotNullWhen(true)] out T? value)
    {
        if (_value is Found<T> found)
        {
            value = found.Value;
            return true;
        }

        _value = default;
        return false;
    }

    public bool IsError([NotNullWhen(true)] out string? description)
    {
        description = null;

        if (_value is ClientError clientError)
        {
            description = clientError.Description;
        }

        if (_value is ServerError serverError)
        {
            description = serverError.Description;
        }

        return description is null;
    }

    // Rest of implementation
}

We could imagine that a consumer might care only about an empty state, and it's easy to implement an IsEmpty, and this implementation is quite human-readable:

public bool IsEmpty() =>
    _value is Empty;

That should give us everything we need for this nicer result type. Compare our previous implementation of GetAllItemsFromDatabase to this new one:

public DatabaseQueryResult<IEnumerable<Item>> GetAllItemsFromDatabase(...)
{
    try
    {
        var result = ContactDatabaseSomehow(...);

        if (result is IEnumerable<Item> items)
        {
            return new Found<IEnumerable<Item>>(items);
        }
    }
    catch (/*exception indicating client error*/)
    {
        return new ClientError(exception.Message);
    }
    catch (/*exception indicating server error*/)
    {
        return new ServerError(exception.Message);
    }

    return new Empty();
}

I find myself very much in favor of this since return new ServerError(exception.Message) makes a lot more senes to me than return DatabaseQueryResult<IEnumerable<Item>>.ServerError(exception.Message). It's not just shorter, it reads more clearly and in six months when I revisit my code I won't be scratching my head why this line needs to provide IEnumerable<Item> as a type argument when it's got nothing to do with a list of items.

This works great, but there's one more quality of life improvement we can give ourselves. When we consume one of these result objects, we'll need to consult either IsSuccess, IsError, or IsEmpty - maybe two or all three of these - in order to direct our control flow, and this involves several conditional statements. Suppose we want to map this result onto a different value depending on its state - that will look even worse! The afrementioned OneOf library contains a method for this purpose on the result object which accepts functions for each possible state to facilitate an easier way to achieve this mapping, though our custom implementation gives us the option of being able to name the arguments to this method in a way that match to the specific state of each, rather than the t1, t2, etc naming in that library.

public sealed class DatabaseQueryResult<T>
{
    public U Map<U>(
        Func<T, U> found,
        Func<U> empty,
        Func<string, U> error
    )
    {
        if (_value is Found<T> foundValue)
        {
            return found(foundValue.Value);
        }
        else if (_value is ClientError clientErrorValue)
        {
            return error(clientErrorValue.Description);
        }
        else if (_value is ServerError serverErrorValue)
        {
            return error(serverErrorValue.Description);
        }

        return empty();
    }
    
    public U Map<U>(
        Func<T, U> found,
        Func<U> empty,
        Func<string, U> clientError,
        Func<string, U> serverError
    )
    {
        if (_value is Found<T> foundValue)
        {
            return found(foundValue.Value);
        }
        else if (_value is ClientError clientErrorValue)
        {
            return clientError(clientErrorValue.Description);
        }
        else if (_value is ServerError serverErrorValue)
        {
            return serverError(serverErrorValue.Description);
        }

        return empty();
    }

    // Rest of implementation
}

These two overloads for Map allow our consumer to choose whether it cares about the client/server error distinction but still enforces that they must provide a complete mapping from any possible result state to an object. We could use this to rewrite our ShowItems method from earlier in a clearer way:

public void ShowItems() => Console.WriteLine(
    GetAllItemsFromDatabase(...)
    .Map(
        found: items => string.Join(items.Select(i => i.ToString()), "\n"),
        empty: () => "No items found!",
        error: description => $"Error! {description}"
    )
);

Now that's looking properly functional!

Future Approach: Discriminated Tag Unions

Update: I added this section to give a little more context on my reference to discriminated unions in C# and the context in which this article exists.

I'm a big fan of these, and a big proponent of their inclusion in C#. The language does not currently support them, but the language design team is actively discussing their inclusion. As I discussed earlier, our result is one of their use cases.

Once DUs are included in the language, the above code will become obsolete. As nice as the API we're able to create is, it might still be a bit interesting for a new member of your team discovering .Map everywhere. It's not that it isn't usable or easy to code in, but it is different. Wouldn't it be nice to be able to use switch? And on that point, wouldn't it be nicer to not need to include all the boilerplate?

Indeed, at the end of the day while the solution I presented above is I think the best option currently, it is still hacky. I don't think this solution - or any others - will offer anything beyond what the language's own upcoming DU features will. The syntax for this feature is not set in stone, but a recent meeting and presentation by the language design team shows their current thinking, so I want to outline a snapshot of what the code that replaces my solution will look like.

To start with, the boilerplate code that sets up the DatabaseQueryResult and each of its options in 140 lines long, and while that isn't really a lot of code, it's certainly a lot of code given how simple the concept is. The syntax to create a union type may end up looking something like the following:

union DatabaseQueryResult<T>
{
    Found(T Value);
    Empty();
    ClientError(string Description);
    ServerError(string Description);
}

No Map function necessary to boot, since the language will allow the use of switch:

public void ShowItems() => Console.WriteLine(
    GetAllItemsFromDatabase(...) switch
    {
        Found(items) => string.Join(items.Select(i => i.ToString()), "\n"),
        Empty() => "No items found!",
        ClientError(description) or ServerError(description) => $"Error! {description}"
    }
);

And ... that's it! From 164 lines total in my sample code down to 16.

The complete code from this article is available on GitHub.

Scrum is not Agile

Wed, 05 Jun 2024 00:00:00 Z

One of my favorite lectures concerned with software engineering methodology is Allen Holub's War is Peace, Freedom is Slavery, Ignorance is Strength, Scrum is Agile, in which I believe he convincingly articulates that Scrum is harmful and that it is not equivalent to Agile. This lecture is particularly useful in drawing a clear distinction between Scrum and Agile - even if you do disagree with Holub's assertion that Scrum is not a proper implementation of Agile, you must agree that Scrum and Agile are different things.

The terms are thrown around interchangeably in our industry, and I think that does a disservice to both. I've heard a lot of criticisms levied against "Agile", but it happens that the criticism is actually against specific practices in the Scrum guide, and while not as ubiquitous I've heard the opposite as well. If you disagree with the rest of my article here, I'd kindly suggest that you at least take this point away and encourage others to be a bit more precise when speaking in this area - neither criticism nor praise are useful if misattributed.

Holub's thesis is stronger than just "Scrum and Agile are different things", though, and it's the stronger point which I find convincing and which I want to defend here. That point is that Scrum is not an implementation of Agile. Holub means this as an indictment of Scrum, where he sees the Agile princiles as being, I think it's not unfair to say, the best set of principles we have right now to guide development methodology. I've written about opinions on this before, so I'm not going to rehash opinion on the matter here. Rather, I think I can defend a dispassionate form of this statement, that based on definitions Scrum can be said to not be an implementation of Agile, even though it claims to be.

Agile is very broadly defined, as well it should be. Its purpose nowadays (I do not know the original intent) is to be a category of processes - there are processes which are Agile and those which are not, and helpfully there's going to end up being a murky middle ground with a fair amount of debate. Agile is bounded by a manifesto and twelve principles, making it obvious that any implementation of Agile needs to satisfy these properties. Is it necessary though for an implementation to allow for these properties to be included with the specific methodology practiced, or is it necessary for an implementation to support or even require these properties? Take for example the following principle:

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

Certainly, if I invent a process which claims to implement Agile but in its definition precludes the ability for engineers to trust each other (say, one of its principles is "Trust nobody but yourself"), I can say that the process does not actually implement Agile. But is it necessary for an implementing system to specify that individuals within the system must trust each other, or is the necessary component simply that the implementing system allows for trust? Rather than splitting hairs, this is an important distinction which will affect how we interpret things, maybe drammatically. What do we do with the following Agile principle then, given this open question:

Continuous attention to technical excellence and good design enhances agility.

This doesn't directly recommend a practice or category of practice unless we consider "enhancing agility" to be a good quality, but that doesn't necessarily emerge from the rest of the writing on the matter. As an aside, if you or a coworker are engaged in a definitional project as the authors of the Agile Manifesto were two decades ago, I can't recommend enough that you consult a philosopher, they have 2500 years of experience in this sort of work.

Vaguely defined things like Agile can be broken down into strong and weak forms. A strong Agile will be an implementation which makes clear and explicit accommodations for each of the Agile principles - trust in work would be encouraged or required, "enhancing agility" would be said to be good, and so on. A weak Agile is one which might or might not explicitly require anything from the principles, but does not preclude or discourage any of them. This is a good distinction, but not terribly useful to us as it doesn't really give us a lot of teeth to be able to determine that something is or is not Agile, but rather a scale of Agility.

We do actually see Agile and non-Agile things though, so we need some way of being able to draw a line in the sand. I'll propose that there are a smaller set of well-defined qualities which can be used to properly delineate Agile, and I think this can be derived from the manifesto and those principles which prescribe practice. Thus, I'll suggest that a system is not Agile unless it explicitly requires the following:

Software must be delivered in continuously iterations, its proper function for the user must be continuously monitored, and the improvement of the software for the requirements of the user must be the highest priority at all times;
The stakeholders in the software all must be continuously engaged with by the engineers of a software, the engineers must be the sole owners of the software, and this relationship and working pace must be able to be carried on ad infinitum;
The engineers must be the sole owners of their own specific processes, they must continuously evaluate and adapt these, and they must accommodate change in any scope at any point in time.

These are the hard requirements which form the "core" of Agile; without specifically and explicitly adhering to these, a system is said to not be an implementation of Agile. While I've used three bullets for brevity, note that these are 9 points grouped into three.

Defining Scrum is, helpfully, much easier, It has a very specific guide to reference. There are specific roles, practices, and structure defined. I'll use that document as a reference for the specific definition of Scrum.

The question then is whether this document satisfies each of the above points - I'll go one-by-one.

Scrum certainly requires the delivery of software in continuous iterations, a sprint is defined:

They are fixed length events of one month or less to create consistency.

Check. Is its function being continuously monitored? Scrum artifacts certianly are:

The Scrum artifacts and the progress toward agreed goals must be inspected frequently and diligently to detect potentially undesirable variances or problems.

Though the document leaves it open whether these artifacts includes the software itself, I think that it's proper to grant a charitable reading to say so. As to whether properly-functioning software is elevated to the highest priority, it seems that a team's goals are rather the highest priority:

The Scrum Team commits to achieving its goals and to supporting each other. Their primary focus is on the work of the Sprint to make the best possible progress toward these goals.

We would certainly hope that teams would, themselves, prioritize the functioning software, but this is not explicitly stated. As for the engineers being continuously engaged with the stakeholders, the Scrum guide falls short as well:

The Product Owner is one person, not a committee. The Product Owner may represent the needs of many stakeholders in the Product Backlog. Those wanting to change the Product Backlog can do so by trying to convince the Product Owner.

Being fair, this isolation can be beneficial in discrete applications, but to specify it as a part of the process goes against our second requirement for Agile. This Product Owner is also charged with developing the Product Goal, which is not explicitly defined. This is a problem as it ought at least be defined to exclude the possibility that the Product Owner might supplant the engineers as the sole owners of the software. Certainly nothing in the Scrum guide, I don't think, might be construed to suggest a working relationship or pace which is not infinitely maintainable.

The Scrum guide fails again in keeping the engineers the owner of their own process, the most glaring way is the inclusion of the Scrum Master role - this is not a simple inclusion and wrestles process control, at least in part, away from the engineers. The Scrum guide is quite successful, by contrast, in mandating Review and Retrospective meetings, both of which are opportunities to evaluate and adapt the process. Accommodating change in any scope at any point in time is a bit more tricky though, take the following:

During the Sprint: [...] No changes are made that would endanger the Sprint Goal;

Surely there's some way to change course though? Well, there is this:

A Sprint could be cancelled if the Sprint Goal becomes obsolete. Only the Product Owner has the authority to cancel the Sprint.

I don't take that inclusion to be accommodating of change though, these prescriptions might allow for some change but they place impediments in its way and refocus the attention away from change being in support of the software to change being antagonistic to the sprint goals.

Is Scrum Agile? No, in the sense that they are just different things, but also no in the sense that Scrum is not an implementation of Agile. It's not just that there are required Agile prescriptions missing from Scrum, were that the case then I might be able to say that Scrum is an implementatino of a subset of Agile. I can't say that though because Scrum contradicts some Agile requirements. This all isn't to say anything negative or disparaging about Scrum, but rather to dispassionately say that it is not an implementation of Agile. Those committed to the whole of Scrum are, I think, committed to the position that not all of Agile is desirable. So as to not throw the baby out with the bathwater, that might be alternatively phrased that only some portion of Agile is desirable. In the same way, those who are committed to the whole of Agile are committed to the position that not all of Scrum is desirable. Perhaps portions of it or ideas from it are.

Shenanigans as an Accelerator

Wed, 03 Sep 2025 00:00:00 Z

If you're a fan of Wordle-ing with your work colleagues, you've probably run into a similar problem as I have. The NYTimes app has a feature to compete with friends on Wordle, but you'd (presumably) be signing into the NYTimes app with a personal account. Or, not everyone has the app, or a NYTimes account. Both of these are barriers to friendly office Wordling.

My company uses Slack, which makes setting up workflows very easy. I presume Teams does as well. Crucially, Slack support two actions out of the box: it can paste form submissions into a Google Sheet, and it can set up an endpoint to ingest and parse any JSON string. On the Google side, their Apps Scripts make interacting with Sheets quite easy, and allows sending HTTP requests. Creating a Wordle challenge Slack channel was all to obvious then: I have a form workflow in which colleagues paste Wordle results that get saved to the spreadsheet, then each morning at 10 an Apps Script posts the results and winners to the Slack channel and clears out that day's submissions.

If you're already familiar with Slack and/or Apps Sctipts then that's nothing but obvious. Maybe a bit annoying to have to read spelled out. However, this ended up being different for me and my colleagues, as we sorted into a cohort defined by two facts: We did not know all that much about Slack workflows and Apps Scripts; and we have a number of scheduley-like things that could greatly benefit from such an automation.

Setting up automations for common scheduley/reporting things can be a huge accelerator for any team. Teams develop processes to make their work flow smoothly, and while those processes evolve over time they generally settle into a static state, with someone on the team tasked with upholding them. Where these processes involve maintaining lists, updating work items, or other manual interventions, would I rather that the process maintainer spend some hours per week performing this work themselves or some (much fewer) hours per year maintaining automations to resolve this work? As with any technological solution, poorly created or architected automations can end up requiring burdensome maintenance, but I want to focus here on the positive result of well-set-up, simple automations with common tools.

What could be more common (for my firm) than Slack and Sheets, I do not know. I do know a couple other things though. For one, having ended up using Slack and Sheets to set up common automations, my team is much better off. One more important thing I know is that, many times, that work done for work is rigid (maybe rote), can tend towards over-engineering, but above all is typically non-exploratory. Maybe I'm getting a bit ahead of myself in my own blog post - how do those two relate?

Well, it wasn't the case that we figured out how to set up the Wordle channel after having tasked someone with learning how to set up automations with Slack and Sheets. Rahter, it was the other way around: the automations we developed were a result of having set up the Wordle channel. In fact, we didn't even think of some of the automations we might do until possibilities were revealed to us by the Wordle channel. And, I assert this is the only direction we could have done this work - we could not have assigned out a task to develop the automations.

Only by making something engaging we were able to stimulate the motivation to explore. Only by stepping outside - clearly outside - the boundaries of day-to-day work we were able to get a different perspective on problems, or what problems existed that needed solving. Easy solving, too.

A Wordle competition Slack channel is surely some shenanigans, and thsoe shenanigans have a tangible, positive result on our work. Some might say that's work that was distracted from by play, but there's no denying the benefits realized. I'm resolving to shenanigan again, and I would encourage the same for you!

"Should I Learn (Insert Some Tech Here)?"

Tue, 14 Nov 2023 00:00:00 Z

Yes.

An Introduction to Sprache

Fri, 22 Jan 2016 00:00:00 Z

As my activity on this blog and my GitHub account may attest, I'm quite fond of a C# library called Sprache. Sprache is a parser-combinator that uses LINQ (Language INtegrated Query) to allow for the elegant construction of parsers in C#. I've been using Sprache for three years now, before I started college, and I've used it to implement a number of domain-specific languages (DSLs) both in side projects on my GitHub and on applications I've worked on. It's only natural I would want to share my favorite C# library with my fellow undergraduate classmates, but there are several factors which make it rather unapproachable for the average undergraduate computer science student. Thus, I have written this piece to provide a completely introductory tutorial to using Sprache.

I'll explain LINQ and BNF, and then I'll walk you through the implementation of a few simple grammars in Sprache such that I may touch upon all the most important concepts in the Sprache library to allow the reader to immediately begin to implement the grammars which they desire. At the end of this post, I link to several articles which cover the framework and other related readings. In the future, I may also write a short handbook/reference to certain Sprache concepts.

One does not necessarily need to have an understanding of C# to begin using Sprache, but a familiarity of a similar language (i.e. Java) would go a long way. I'm going to assume the reader has an understanding of object-oriented programming. I won't be going into an explanation of what a parser-combinator is, nor what a "combinator" is, in general. If you would like to become more involved in the development of Sprache, though, you should definitely familiarize yourself with the concept. I provide some links at the end of this tutorial to that end.

Prerequisites

To begin with, of course, you'll need to download Sprache. You can find it on GitHub.

LINQ

LINQ, short for Language INtegrated Query, is a wonderful feature of Visual C# which adds data-querying operators to C#. LINQ expressions are sometimes (grammatically incorrectly) referred to as "LINQ queries" as they read rather fluently as a query on a data set. Here is an example of a LINQ expression:

var myList = new List<string>()
{
    "hello",
    "world",
    "how",
    "are",
    "you"
};

var startsWithH =
    from s in myList
    where s.ToCharArray()[0] == 'h'
    select s;

foreach (var a in startsWithH)
	Console.WriteLine(a);

Here, we start with a list of words, and we desire to print to the console each word which begins with the letter 'h'. The variable startsWithH is defined with the following LINQ expression, which is how we sort out those words which start with 'h':

from s in myList
where s.ToLower().ToCharArray()[0] == 'h'
select s;

Let's look at what's going on here. First, we have a from statement. This will iterate over each object in myList, using s as the iterator variable. Next, we have a where statement, which filters out the objects in myList based on the condition provided. Note that several where statements could be specified here. At the end of this LINQ expression, as with every LINQ expression, we have a select statement, which returns each "queried" object. In this case, we only desire to return the strings which begin with the letter 'h'.

LINQ supports several operators apart from from, where, and select, though these are the main ones. Microsoft, naturally, provides a very in-depth list of LINQ operators, though Wikipedia has a much more succinct list.

Sprache uses LINQ to construct its parsers. This allows for quick implementation and easy and intuitive readability.

Backus-Naur Form

Backus-Naur Form, or BNF for short, is a metalanguage used to describe the grammars and syntax of context-free grammars (essentially, for our purposes, this means the grammars of computing languages). BNF defines expressions in terms of other expressions and strings using a number of rules which will become more familiar as we begin implementing these grammars in Sprache.

As an example, suppose I want to define a grammar which specifies an arithmetic expression which might add, subtract, multiply, or divide two digits. I'll provide a BNF definition of this grammar, and then explain it.

<expr>      ::= <add> | <subtract> | <multiply> | <divide>

<add>       ::= <digit> "+" <digit>
<subtract>  ::= <digit> "-" <digit>
<multiply>  ::= <digit> "*" <digit>
<divide>    ::= <digit> "/" <digit>

<digit>     ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"

Let's look at each of the elements and what they do. First, the most notable and important element is the reference for an expression, which looks like the following:

<expression_name>

The expressions are referenced by this convention, and they are defined with the ::= operator. In defining such expressions, a number of rules can be used. Look at the definition for expr:

<expr>      ::= <add> | <subtract> | <multiply> | <divide>

The bar ('|') denotes an or relationship. That is, an expression expr can be either an add, subtract, multiply, or divide expression.

Now let's look at the definition of the add expression:

<add>       ::= <digit> "+" <digit>

This specifies that an add can be a digit, followed by a plus sign, followed by another digit.

You might notice a bit of an inefficiency in the grammar I defined above. Namely, we define add, subtract, multiply, and divide separately, but due to the similarity in their structures, it feels like we should be able to define them all together. While there are certainly good reasons one might want to define them separately as I did above, for succinctness one might desire to redefine the grammar as such:

<expr>      ::= <digit> ("+" | "-" | "*" | "/") <digit>

<digit>     ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"

Here I introduce a grouping of terms, defined by the parentheses. Now, expr is defined to be two digits separated by either an addition, subtraction, multiplication, or division symbol. This is, however, a rather dumb grammar, in that only two digits can be used in the arithmetic expression, while we might want to allow any number to be used. We can extend the grammar further to allow for this:

<expr>      ::= <number> ("+" | "-" | "*" | "/") <number>

<number>    ::= <integer> ["." <integer>]
<integer>   ::= +("0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9")

Here we have broken a number into two parts, a number and an integer. Where I define integer, I introduce a plus sign, which allows the expression which it suffixes to be repeated one or more times. Where I define number, I introduce the square brackets, which surround optional expressions. Thus, the following terms are captured by the expression number: 0, 125, 3.14, and 123.456. However, the following terms are not captured by number, and I will allow the reader to postulate why they are not, and how the grammar might need to be altered to capture them: .31, -12, -12.56, and -.987.

When we want to parse a language with Sprache (or any other parser, for that matter), we will first define the language in BNF, so that we can easily reference the pieces of the parser we must create, and to keep track of our progress.

Sprache

Ultimately, once you get used to using LINQ to construct parsers, Sprache is just another library, and becoming proficient in Sprache is the same process one should be used to of learning the methods given by the library and learning how to ask questions on stack overflow.

Let's begin familiarizing ourselves by constructing a parser which can parse the string "hello" into a string "hello." This is an entirely non-useful task for Sprache, but it gets our feet wet:

Parser<string> myParser =
    from str in Parse.String("hello").Text()
    select str;

string val = myParser.Parse("hello");

val will, unremarkably, be "hello."However, the parser should be very easy to understand, especially given our understanding of the working of a LINQ expression. The method String(string) is a parser which parses any string you desire (in this case, we desired to parse the string "hello"). The String parser returns an enumerable of chars, so we need to use Text to turn the enumerable into a string. From there, it should be rather obvious what is going on.

Now, let's suppose we want to parse the string "hello" multiple times, separated by whitespace, and we want to know how many times "hello" appears. We can extend our parser above like so:

Parser<int> myParser =
    from str in
        Parse.String("hello").Text()
        .DelimitedBy(Parse.WhiteSpace.Many())
    select str.Count();

Testing this parser with the string "hello hellohello hello" should return a result of 4. Because of the way our parser is constructed, it is relatively straightforward to read it as "parse the string "hello" delimited by whitespace." But let's look at what's going on here. DelimitedBy will attempt to match the "hello" parser, and then it will look for whitespace (WhiteSpace().Many() is a parser itself which matches 0 or more different whitespace characters in a row), and will then look to match "hello" again and more whitespace, until the parser is no longer able to match either "hello" or whitespace, at which point it returns an IEnumberable containing several "hello"s. Our select statement can then select the Count of that IEnumerable, and thus we can obtain the number of times "hello" is parsed.

This idea of chaining parsers onto each other, as DelimitedBy is chained onto String, is the entire concept behind parser-combinators.

A slightly more complicated task might be to try to parse a variable name surrounded by whitespace (often called an "identifier"). This example is given on the Sprache GitHub page:

Parser<string> identifier =
    from leading in Parse.WhiteSpace.Many()
    from first in Parse.Letter.Once()
    from rest in Parse.LetterOrDigit.Many()
    from trailing in Parse.WhiteSpace.Many()
    select new string(first.Concat(rest).ToArray());

Thus, " abc123 " should come out as "abc123". Notice how staggering several from statements in a row reads as though we are saying "then". For example, this parser could be read by a human as "Parse many whitespace characters, then parse one letter, then parse 0 or more letters or digits, then parse more whitespace, and return the first letter and the rest of the letters/digits concatenated to it".

Our First Language

So, let's now define the grammar for a small DSL, and we'll try to parse it. Let's make a language that defines variables: we can have an identifier, followed by a colon, and then a string, and we can define as many variables as we want on different lines. Ultimately, we want to parse this into a dictionary. So, our resulting language could look like this:

identifier1 : "hello"
identifier2 : "world"
identifier3 : "yay parsing"

The BNF for the language looks like this:

<block>        ::= <expr> *(<newline> <expr>)
<expr>         ::= <identifier> [<whitespace>] ":" [<whitespace>] <string>
<identifier>   ::= <letter> *(<letter> | <digit>)
<string>       ::= "\"" *(<any_character>) "\""

I'll imagine you can determine what newline, letter, digit, and any_character are. Note, though, that we technically want any_character to parse any character except a quotation mark.

Sprache already contains parsers for a letter, digit, and any character, so we should be all good to go from here. We already have our identifier parser, so let's add to that by constructing our string parser:

Parser<string> identifier =
    from first in Parse.Letter.Once()
    from rest in Parse.LetterOrDigit.Many()
    select new string(first.Concat(rest).ToArray());

Parser<string> stringParser =
    from first in Parse.Char('"')
    from text in Parse.AnyChar.Except(Parse.Char('"')).Many().Text()
    from last in Parse.Char('"')
    select text;

Here we use Except to add an exception to the AnyChar parser. In addition, we use Text at the end to tell sprache to convert the IEnumerable returned by Many into a string. Now, we can add the parser for the expressions:

//Adding to the code above:

Parser<Dictionary<string, string>> expr =
    from id in identifier
    from colon in Parse.Char(':').Token()
    from str in stringParser
    select new Dictionary<string, string>() { { id, str } };

Optional is used here - that does exactly what it says - it makes the parser optional. In addition, I introduced Token, which will parse whitespace before and after the CHar parser. Notice how we are able to reference the parsers we created earlier, and we can use the values they return to create a new object. Let's finish it off by creating the "block" parser, which is supposed to parse several expressions separated by newlines:

//Adding to the code above:

Parser<IEnumerable<Dictionary<string,string>>> block =
    expr.DelimitedBy(Parse.Char('\n'));

Notice how we are not using the fancy LINQ expressions here. Because our parser fits on one line, and DelimitedBy returns the type that we want, then we can condense our parser a bit. Now that we're done with our parser, we should be able to parse our example file just fine into an IEnumberable of dictionaries containing our identifier-string pairs.

Comma-Separated Values

CSV files are extremely popular for storing tables in plaintext, and they're very easy to parse, as you might imagine. Frequently, programs which read from CSV files desire to read the files into their own data structures. So, we'll imagine a couple different scenarios involving CSVs, and we'll look at how we can go about parsing each one.

First, I'll provide a rough CSV parser that sorts the CSV into a list of a list of strings, and from there we can talk about a custom data structure for it.

Parser<IEnumerable<IEnumerable<string>>> csv = 
    Parse.AnyChar.Except(Parse.Char(',')
        .Or(Parse.Char('\n'))).Many().Text()
        .DelimitedBy(Parse.Char(',').Token())
        .DelimitedBy(Parse.Char('\n'));

This parser is quite fun, as it can be written in one line, yet it parses a CSV file pretty much alright - you might notice that none of the values can contain a comma (Bonus problem: see if you can get the parser to recognize escape characters so that the user can insert commas. Later, in the JSON parser, we'll implement escape characters).

Let's digress now, and suppose we have the following simplistic data structure:

class Row
{
    public string Title { get; set; }
    public IEnumerable<string> Items { get; set; }
}

And let's further suppose that in a CSV, the first item of every row is the title of that row, and the remaining elements in that row are the items, corresponding to the structure above. So, we want to get a list of these rows, presumably. We can very easily modify our parser:

Parser<Row> line =
    from first in
        Parse.AnyChar
        .Except(Parse.Char(',')
        .Or(Parse.Char('\n')))
        .Many().Text()
    from comma in Parse.Char(',').Token()
    from rest in
        Parse.AnyChar
        .Except(Parse.Char(',')
        .Or(Parse.Char('\n'))).Many().Text()
        .DelimitedBy(Parse.Char(',').Token())
    select new Row() { Title = first, Items = rest };

Parser<IEnumerable<Row>> csv = line.DelimitedBy(Parse.Char('\n'));

Naturally, if you're just interested in obtaining the rows, then this parser works perfectly. But let's suppose we didn't want the nested lists to contain the rows, but the columns. In this case, we can do some nifty snafu:

Parser<IEnumerable<string>> line =
    Parse.AnyChar.Except(Parse.Char(',')
        .Or(Parse.Char('\n'))).Many().Text()
        .DelimitedBy(Parse.Char(',').Token());

Parser<IEnumerable<IEnumerable<string>>> csv =
    from l in line.DelimitedBy(Parse.Char('\n'))
    select Transform(l);

//Here's the Transform method:
//Assume the table is n-by-n
static IEnumerable<IEnumerable<string>> Transform(IEnumerable<IEnumerable<string>> t)
{
    var toReturn = new List<List<string>>();
    
    for (int i = 0; i < t.ElementAt(0).Count(); i++)
    {
        for (int j = 0; j < t.Count(); j++)
        {
            if (toReturn.Count == i) toReturn.Add(new List<string>);
            if (toReturn[i].Count == j) toReturn[i].Add("");
            
            toReturn[i][j] = t.ElementAt(j).ElementAt(i);
        }
    }

    return toReturn;
}

Transform just rotates the list of lists as though it's a matrix, so we're not adding anything too special here. What if we wanted to do what we did above with the rows, but with the columns? Try modifying this code to do just that. Bonus points if you can eliminate Transform and perform the transformation within the parser!

XML

Obviously, XML is a rather complex language, and a complete BNF specification is thus very large. Therefore, we'll be using a much simpler variation of XML, which we can see below:

<tag> ::= <single_line_tag> | <multi_line_tag>
<short_tag> ::= "<" <identifier> <whitespace> <attribute>* "/>"
<full_tag> ::= "<" <identifier> <whitespace> <attribute>* ">" <tag>*
    "</" <identifier> ">"
<attribute> ::= <identifier> "=" "\"" <any_characters> "\""

You might notice I'm leaving out a few unnecessary components: Expressions which are intuitively obvious aren't defined, whitespace is only used where necessary (we'll use Token prolifically to allow for flexibility on the user's part), and when defining a "full" tag, the identifier of the opening and closing tags must be the same. The latter component cannot be defined in vanilla BNF, so it's something we'll need to account for in our code.

Instead of writing this code out, I'll reference the XML example which is included with Sprache, and I'll explain the new elements and solutions found there.

Right off the bat, looking at the Document parser, we're introduced to a new use of LINQ. As I alluded to previously, any LINQ query can be written in one line without the LINQ statements. Here's the critical portion of line 98:

Node.Select(n => new Document { Root = n })

To make this easier to comprehend, we can write it out using the regular LINQ notation we're familiar with:

from n in Node
select new Document() { Root = n };

In fact, the LINQ statements we've been are just a shorthand (or perhaps more of a "paraphrasing," as they tend to be longer) for the inline notation. Document very well could have been written using solely the LINQ statements, and that would have looked like this:

public static readonly Parser<Document> Document =
    from leading in Parse.WhiteSpace.Many()
    from doc in from n in Node.End()
                select new Document() { Root = n }
    select doc;

The next parser up from Document is Item. We'll ignore the code regarding the comments (if you're interested in this, please see my post Parsing Comments with Sprache). This makes it easy to see that an Item is either a Node cast as an Item, or a Content, and a Node (looking above in the document) is either a short or full node.

Looking at the ShortNode parser, we can see it seems completely familiar, except that the LINQ expression is used as an argument to the Tag method. The Tag method returns a parser that parses a greater than and less than sign before and after the parser you specify. This abstraction allows us to write cleaner code.

FullNode is fun for a couple reasons. First, look at how they solved the issue of requiring opening and closing tags to be named the same with the EndTag method. In addition, notice the use of the Ref parser. In FullNode, we need to use Item, but it has obviously not yet been created. Ref allows us to reference a parser later in the document, thus allowing us to create some recursive or ambiguous grammars with Sprache.

JSON

JSON, or JavaScript Object Notation, is kind of like XML. It's a way of storing data in plaintext (in a key-value pair manner) which is also easily readable by a human. In addition, it's very easy to construct a parser for it. The BNF form is very clear and concise - here, I have transcribed the informal definition on json.org into the more formal BNF notation which we have been using in this tutorial:

<object>    ::= "{}" | "{" <members> "}"
<members>   ::= <pair> | <pair> "," <members>
<pair>      ::= <string> ":" <value>
<array>     ::= "[]" | "[" <elements> "]"
<elements>  ::= <value> | <value> "," <elements>
<value>     ::= <literal> | <array> | <object>
<literal>   ::= <string> | <number> | <bool> | "null"

An example of a valid JSON file might be the following:

{
  "firstName": "John",
  "lastName": "Smith",
  "age": 25,
  "address": {
    "streetAddress": "21 2nd Street",
    "city": "New York",
    "state": "NY",
    "postalCode": "10021-3100"
  },
  "phoneNumbers": [
    {
      "type": "home",
      "number": "212 555-1234"
    },
    {
      "type": "office",
      "number": "646 555-4567"
    }
  ]
}

This should allow us to construct the data structure we're going to store our data in (called an Abstract Syntax Tree):


public class JSONValue {}

public class JSONObject : JSONValue
{
    public Dictionary<string, JSONValue> Pairs { get; set; }

    public JSONObject(IEnumerable<KeyValuePair<string, JSONValue>> pairs)
    {
        Pairs = new Dictionary<string, JSONValue>();
        if (pairs != null)
            foreach (var p in pairs)
                Pairs.Add(p.Key, p.Value);
    }
}

public class JSONArray : JSONValue
{
    public List<JSONValue> Elements { get; set; }

    public JSONArray(IEnumerable<JSONValue> elements)
    {
        Elements = new List<JSONValue>();
        if (elements != null)
            foreach (var e in elements)
                Elements.Add(e);
    }
}

public class JSONLiteral : JSONValue
{
    public string Value { get; set; }

    public LiteralType ValueType { get; set; }

    public JSONLiteral(string value, LiteralType type)
    {
        Value = value;
        ValueType = type;
    }

    pubilc static enum LiteralType
    {
        String,
        Number,
        Boolean,
        Null
    }
}

To implement the parser, we'll start from the bottom and work our way up, as we usually do. Literal values are expressed nicely by our JSONLiteral class, which stores every value as a string, and also keeps track of the type of literal it is. Parsing them all out is a bit of a pain, so I'll post each parser here and explain it briefly.

Parser<JSONLiteral> JNull =
    from str in Parse.IgnoreCase("null")
    select new JSONLiteral(null, JSONLiteral.LiteralType.Null);

Parser<JSONLiteral> JBoolean =
    from str in Parse.IgnoreCase("true").Text()
    			.Or(Parse.IgnoreCase("false").Text())
    select new JSONLiteral(str, LiteralType.Boolean);

Parsing a literal null or boolean value isn't all too complicated. We just need to parse the strings which represent them, ignoring the case, and return new JSONLiteral objects.

Parser<string> JExp =
    from e in Parse.IgnoreCase("e").Text()
    from sign in Parse.String("+").Text()
                 .Or(Parse.String("-").Text())
                 .Optional()
    from digits in Parse.Digit.Many().Text()
    select e + ((sign.IsDefined) ? sign.Get() : "") + digits;

Parser<string> JFrac =
    from dot in Parse.String(".").Text()
    from digits in Parse.Digit.Many().Text()
    select dot + digits;

Parser<string> JInt =
    from minus in Parse.String("-").Text().Optional()
    from digits in Parse.Digit.Many().Text()
    select (minus.IsDefined ? minus.Get() : "") + digits;

Parser<JSONLiteral> JNumber =
    from integer in JInt
    from frac in JFrac.Optional()
    from exp in JExp.Optional()
    select new JSONLiteral(integer +
                           (frac.IsDefined ? frac.Get() : "") +
                           (exp.IsDefined ? exp.Get() : ""),
                           LiteralType.Number);

Parsing a number is much more exciting. We need to account for integers, decimals, negation, and 'e'. The code above for JNumber knows we need at least an integer, and can be optionally followed by the fraction or the exponential term. Notice that Optional returns a special object which may or may not be defined. Thus, we need to check whether it is defined with IsDefined before we can Get its value.

List<char> EscapeChars = new List<char>
    { '\"', '\\', 'b', 'f', 'n', 'r', 't' };

Parser<char> ControlChar =
    from first in Parse.Char('\\')
    from next in Parse.EnumerateInput(EscapeChars, c => Parse.Char(c))
    select ((next == 't') ? '\t' :
            (next == 'r') ? '\r' :
            (next == 'n') ? '\n' :
            (next == 'f') ? '\f' :
            (next == 'b') ? '\b' :
            next );

Parser<char> JChar =
    Parse.AnyChar
    .Except(Parse.Char('"')
    .Or(Parse.Char('\\')))
    .Or(ControlChar);

Parser<JSONLiteral> JString =
    from first in Parse.Char('"')
    from chars in JChar.Many().Text()
    from last in Parse.Char('"')
    select new JSONLiteral(chars, LiteralType.String);

To parse a string, we want to make sure that we allow for control characters (the control characters are all given on json.org). As you can see, the string will be zero or more characters, which are in turn any character except a quotation mark or the escape character. Where ControlChar is defined, EnumerateInput is used on our list EscapeChars. This instance of EnumerateInput will return the following parser:

Parse.Char( '\"').Or(Parse.Char('\\')).Or(Parse.Char('b')) ...

That is, it chains each element in EscapeChars along as the parser Parse.Char() using the Or parser.

Parser<JSONLiteral> JLiteral =
    JString
    .XOr(JNumber)
    .XOr(JBoolean)
    .XOr(JNull);

Finally, we're able to piece them all together to form our JLiteral parser. Luckily, this is half of our entire parser!

As you can see from our JSON BNF, the rest of the grammar is recursive. That is, self-referential. This is where Ref will come in handy. We need to implement objects and arrays, and those two plus literals will be defined as a value. So, let's define our JValue parser, and proceed from there.

Parser<JSONValue> JValue =
    Parse.Ref(() => JObject)
    .Or(Parse.Ref(() => JArray))
    .Or(JLiteral);

Here, we are using Ref to reference our yet-undefined JObject and JArray parsers. Of course, we've already created our JLiteral parser, so we do not need to use Ref to access it.

Now we just need to parse JSON arrays and objects. For convenience, let's recall the portion of the JSON BNF which defined them:

<object>    ::= "{}" | "{" <members> "}"
<members>   ::= <pair> | <pair> "," <members>
<pair>      ::= <string> ":" <value>
<array>     ::= "[]" | "[" <elements> "]"
<elements>  ::= <value> | <value> "," <elements>

We can see that arrray and object look very much alike, and the definition of array appears to be a tad more simple. Therefore, we should write our array parser first, and we can copy it down to create our slightly more complicated object parser.

Parser<IEnumerable<JSONValue>> JElements =
    JValue.DelimitedBy(Parse.Char(',').Token());

Parser<JSONValue> JArray =
    from first in Parse.Char('[').Token()
    from elements in JElements.Optional()
    from last in Parse.Char(']').Token()
    select new JSONArray(elements.IsDefined ? elements.Get() : null);

Notice how our JElements parser almost perfectly matches the definition of elements in the BNF. DelimitedBy will parse any number of JValue here, so long as they are separated by commas - this removes our need to call JElements recursively. Our JArray parser, then, just encases the JElements parser in square brackets. If we desired we could combine the parsers into one. The reason I separated them here, however, was to demonstrate the close relationship between BNF and parsers like Sprache. Here is how the combined parsers would look:

Parser<JSONValue> JArray =
    from first in Parse.Char('[').Token()
    from elements in
        JValue.DelimitedBy(Parse.Char(',').Token()).Optional()
    from last in Parse.Char(']').Token()
    select new JSONArray(elements.IsDefined ? elements.Get() : null);

Now, we can move on to write our JObject parser.

Parser<KeyValuePair<string, JSONValue>> JPair =
    from name in JString
    from colon in Parse.Char(':').Token()
    from val in JValue
    select new KeyValuePair<string, JSONValue>(name.Value, val);

Parser<IEnumerable<KeyValuePair<string, JSONValue>>> JMembers =
    JPair.DelimitedBy(Parse.Char(',').Token());

Parser<JSONValue> JObject =
    from first in Parse.Char('{').Token()
    from members in JMembers.Optional()
    from last in Parse.Char('}').Token()
    select new JSONObject(members.IsDefined ? members.Get() : null);

By now, this should all be trivial to you - especially considering JObject and JMembers are copies of JArray and JElements, respectively. With that, we should now be able to parse any document which conforms to the JSON standard. Notice that every JSON document is itself a single JSON object. Thus, given a JSON document, we would parse it with our JObject parser.

If you would like to see the parser in full, there is a version on my GitHub.

My Work With Sprache

As I mentioned above, I've been working with Sprache for three years now, after seeing a presentation about it at the Twin Cities Code Camp, which is totally awesome and you should all go (it's even free).

I've contributed to Sprache by adding a comment parser, I've published a JSON serializer/mapper, and I'm working on a Markdown parser.

I've used Sprache in small amounts in a couple other projects, and I enjoy using it wherever I'm able. An idea suggested to me at an Iowa Code Camp, which is also awesome and free and you should all go, was to write a tool to convert BNF into Sprache. I haven't done anything with this concept yet, but that is further work that could be done - if you're feeling the Sprache bug and you want to tackle that, go right ahead!

SpracheDown

Fri, 26 Sep 2014 00:00:00 Z

About a year and a half ago I attended one of the Twin Cities Code Camps, and there I was shown a nifty library called Sprache. Sprache is, by no means, a novel invention. It's a monadic parser combinator based on years of programming done with similar libraries popular in functional languages. I say it's nifty because it seems to be the cleanest monadic parser combinator made in C#. It's also got a large enough following that keeps it up-to-date well enough, and it's what I was taught at the code camp.

Half a year after that, almost a full year ago now, I attended my first Iowa Code Camp. One guy I taught about Sprache and monadic parser combinators asked if it was possible to create a markdown parser with it, and my answer was a brief "Well, yeah, you can totally do that." I've been mulling that over for about a year now, and I decided to look for a Markdown parser implemented with a monadic parser combinator to see what it would look like.

I found an excellent article by Greg Hendershott who implemented a MarkDown parser with a variation of Parsec for Racket. I browsed through his parser on GitHub, and to my alarm it seemed that the parser itself was 1,000 lines long (I could be misreading that, I've no prior experience with Racket, but I'm assuming "parse.rkt" contains the parser). This compelled me to attempt such a parser with Sprache. If nothing else, it would be an interesting comparison.

So I began writing this parser in steps, gradually adding features to it. I started out with headers, then lists, then paragraphs, and so forth. As I was writing this parser, I deliberately omitted features from MarkDown, notably reference-style links and inline HTML. I think if you can parse the majority of MarkDown the little bits could be implemented with equal ease, so I don't believe this invalidates the parser. I may go on and add inline HTML (Sprache ships with an XML parser example), but so long as this is a neat little pet project, I don't think I'll go too far beyond that.

At first, I parsed the MarkDown text directly into strings representing the HTML output. This was efficient, but of course it wouldn't do for any parser - the user needs to be able to manipulate the output. To save time, I borrowed the syntax tree that Sprache's XML example comes with. I adapted and modified the objects with a couple methods to bend them to my will, so to speak, and from there it only took a minute to plug them into my parser.

One problem that Mr. Hendershott faced was parsing MarkDown's nested list feature (this is achieved by inserting spaces before the asterisk in MarkDown). I don't know if this was due to the language he was using or a limitation of the parser,

Now, I can't say I actually ran into any major problems when I implemented the parser. Granted, I ignored the MarkDown I didn't like so well, but as I said, I don't believe that invalidates the parser. In fact, my success with this parser speaks to the beauty inherent in parser combinators, specifically Sprache in this case. Over the past two days, I've spent a total of six hours working on this, and it's already relatively well-polished. The code is clear and readable, thoroughly commented, and the syntax tree is easily scalable. In addition, my parser is significantly smaller than Mr. Hendershott's parser (perhaps an advantage of Sprache in C# over Parsack in Racket?), and I don't believe I could top 1000 lines in the parser alone if I were to bring the parser up to speed with all of MarkDown's features.

As I've been saying, I wrote this as a proof of concept, and I don't really intend for it to go anywhere, but if you think this is the coolest thing ever and you want it to be something, you're more than welcome to submit pull requests, or you can fork it and do your own thing with it. I believe I've documented everything thoroughly, so it should be easy to find your way around, but if I've missed something don't wait to contact me. Have fun!

You can find SpracheDown on GitHub right here.

SpracheJSON

Fri, 10 Apr 2015 00:00:00 Z

I've been meaning to write about this for a while now, but college got in the way of that. I wrote a deal I called SpracheJSON to parse JSON text into C# objects with Sprache, and it's kinda neat. I also played with the GitHub pages feature on this project, but that's not really interesting.

I got this thing to the point where is parses the JSON just fine, no problem. But I also figured it made sense to serialize between JSON and C# objects. Like the other parsers I've written have ended up, it's got its own AST to throw the JSON into, but I can't imagine I'd ever want to use generic objects to store my data (isn't that the whole point of JSON?) That said, I've never been a fan of ISerializable, so I've got my own custom serializer thing going on. Obviously that makes it kinda imperfect, but that's how the world goes, I suppose.

In the future I'm going to use ISerializable to make that part of the project go easier, but until then, I've got a nice half-baked parser here. I'm not going to detail the functionality of the library too much, but it's pretty slick, so you should definitely go check it out here.

Parsing Comments with Sprache

Fri, 06 Feb 2015 00:00:00 Z

I recently made a comment parser for the Sprache framework, and I wanted to give a basic run-down on how it works.

The CommentParser class gives you the option to define the header styles for comments, and it can parse both single- and multi-line comments. It's rather basic as of right now, but that's (hopefully) subject to change in the future.

Using CommentParser is pretty simple, but it's a tad different from the rest of the flow of Sprache as a combinator library. You'll need to make an instance of the CommentParser class, using the comment headers and (optional) newline character you require as arguments:

static CommentParser comments = new CommentParser("//", "/*", "*/");

From there, CommentParser gives you a couple parsers you can use to parse single- and multi-line comments:

static Parser<string> myParser = Parse.String("foobar").Text().Or(comments.AnyComment);

CommentParser.AnyComment will parse either single- or multi-line comments for you, while CommentParser.SingleLineComment and CommentParser.MultiLineComment will parse those individually.

A real, working example using the CommentParser class can be found in Sprache's XMLParser example.

In the future, it would be awesome if multiple comment headers could be included, and if whitespace could be defined to include comments. Some work towards this effort has been done on my GitHub here.

Successful Personal Projects

Wed, 02 Jul 2025 00:00:00 Z

I have this one specific psychic power, when I speak to a software engineer I can tell them how many half-finished, abandoned, sometimes barely-started personal projects they have scattered between source control providers and dusty hard drives. I can even prove to you I have this psychic power! Just focus your mind on your favorite color, and I'll tell you now how many such projects you have. Ready? The answer that's coming to me is "a lot."

Uncanny, isn't it?

We all spend a lot of time with software, and we spend a lot of time thinking about cool software things we can do. We know we have the ability to create these things, we see others releasing their fun projects, but this or that happens and our project stalls. Then the next one and the next one, to the point that for every successful project we've released we have several unfinished ones.

This isn't a problem in itself; they're on our own time and impacting nobody. However, we did start each project for some reason (presumably), and failing to have finished a project is to have failed to get what we wanted from it. Maybe we wanted to learn a new technology, fully develop an idea, or have the satisfaction of having built something. Consistently engaging with personal projects - and that means finishing them - comes with a number of benefits in increasing our abilities and confidence in our work.

I'm not going to claim to be great at finishing personal projects, but I can claim to frequently see folks stepping in the same potholes I've learned to avoid, so I can share my approach to personal projects.

1. What do you want to gain?

The most important consideration for a personal project is having a realistic personal goal. Not a product goal necessarily, but a personal project is done, ultimately, to fulfil a personal need or want. What is that goal? Are you trying to learn a framework, a pattern, a protocol, or a language? Are you trying to gain a deeper understanding of a technology you commonly use? Are you trying to deliver some kind of product - for yourself, a friend, or a club? Are you trying to showcase your work to colleagues, potential employers, or a blog audience?

I'm under no illusions that the majority of personal projects are begun under the motivation of creating some end result; most are not contrived after the decision upon the personal goal. That's entirely fine, and I think it's short-sighted to consider such a workflow as putting the cart before the horse. I'm definitely more inspired thinking about delivering a cool thing than I am by a satisfaction with being able to bring up cool facts about RFC 1605.

Nonetheless, before you start building you must be upfront about what the personal goal is and make sure you define it well. To reiterate, because these projects are done on personal time with personal resources, they won't be successful without being rooted firmly in personal aims.

2. Define done

Indeed, there's no escaping the same "definition of done" we must engage in at work. The only real difference being that your definition of done for your personal project should be significantly lower in scope than you have at work. Descope everything and MVP-max. It's not glamorous but it is realistic. Sure I know that I can deliver a set of 12 features for a project, but I also know that I definitely will deliver the project if it's only 3 of those 12 features. It's not insulting to my abilities to descope the project, it's a guarantee that I can achieve what I want to achieve. Remeber, delivering the resulting software is subordinate to filling my personal requirements.

Done means two things: the software is working at the state that I want and I have achieved my personal goal. These go hand-in-hand: the software and technology need to support the personal goal, and the personal goal constrains the scope of the software. Just as I'm focusing on a single personal goal, I'm focusing on achieving a single piece of software, that does one thing. It's okay if it's a little ugly.

The faster I can get to "done," the better. Sensible personal goals and super-MVP-ified software gets me there. This always leaves extra bandwidth at the end of the project for me to continue it. It's a separate project in itself, often, to add more features to this piece of software - here we can rinse and repeat the personal project cycle. Or, I can publish the project open source and have a very helpful backlog of interesting addons. Sometimes others on the internet do engage with your projects this way!

3. Architect and develop pragmatically

Or, put another way, focus on the core competency of the project. If your project requires auth but your goal is to learn about VOIP, don't write your own auth layer. If your goal is not to learn a new language then use a familiar one. There's no need to dress your codebase up with all of the bells and whistles endemic to professional engineering. C# and Java code tends to be riddled with too many interfaces, passthrough layers, and unit tests. Your personal projects sute don't need unit tests, not out of the gate at least.

Architecture is a very common pitfall. Really consider what you actually need. If you can get away with hosting your site on GitHub pages then do that. If you only absolutely require one small piece of server functionality then spin up a serverless function (I love Railway's functions for this). Seriously interrogate every piece of infrastructure. Do you need Postgres or can you get away with a JSON file or Google Sheets? Do you need a server or can you serve using existing technologies? Do you need all your favorite NPM packages or do you just need simple reactivity?

Remember you've only got one goal, not one main goal and a bunch of secondary ones. I'm sure you'd really enjoy learning a new JS framework, but if that isn't the one goal of your project then that's not what we're going to be doing on this project, we'll be doing it in another project. Conversely, if your one goal is to learn a JS framework then our software isn't going to be using an unfamiliar or burdensome architecture, and it definitely should have similar functionality to other projects you've developed before!

To keep development from stalling, I specifically keep two things in mind. First, I want to get to an executable piece of software as step #0 in development; I won't start writing any code until I can execute code, and then I'll keep the project ni a state of being able to run. Second, I'll always focus on the difficult and "core competency" component first, at the exclusion of anything else; if a CSS style is taking too long I'll let it be ugly until I can get the core component where I want it.

4. Finish it

Every step you take on the project should be a tangible, obvious step towards the end. Because you've kept your project scoped as low as possible you'll have the time and energy to do the coding work, and because you're clear about your personal goal you'll identify when you've achieved that.

If you're spinning your wheels you'll need to reevaluate what you haven't scoped down. Sometimes we make mistakes, sometimes projects take off on the wrong trajectory and need to be seriously rethought. Sometimes projects do need to be killed - not abandoned, but killed. I advise restarting with the same personal goal and software concept, but after a rearchitect. That itself might be an architecture project!

When we finish something we get the satisfaction of having finished, the confidence that we can finish it, and we've gained whatever knowledge, experience, or product that we set for ourselves.

Testing Logging in ASP.NET Core

Sun, 12 Jan 2025 00:00:00 Z

While I'm not much a fan of unit tests, I'm a big fan of integration tests, particularly for distributed services. I'm in agreement with Grug that this is the right level for long-lived automated tests, and I try to structure my projects in such a way that everything which I need to test can be and is tested this way. Now, this strategy doesn't apply to every project type, but any project that can be seen as a request-response black box - like an ASP.NET Core API - is a proper candidate.

A lot of API integration testing will focus on HTTP requests and the responses to those and call it a day. That's perfectly fine, and surely captures most all the necesssary business logic to test. We would be mistaken though to think these are the only inputs and outputs to an API. If you're running in production, I'm sure your API is able to be configured by a DevOps administrator, and maybe it publishes events to an event queue or schedules emails to be sent. These are all different inputs and outputs that can also be tested. One output of our services that often goes untested are their logs.

Why would we need to test log output? You might have many reasons, but I suspect the primary reason would be if you have a monitoring system reading the logs from the API with alert conditions set up in specific cases. Suppose you have a critical path in your API and you want to get pinged about any errors in this flow. You might configure your monitoring service to send you a text if it ever sees an ERROR-level log with text "critical failure." This log output has just become a business rule, and you may well want to keep a test to ensure that if the API ever enters that condition, the log with the correct level and string will be output, as expected by the monitoring suite.

At least, that's the condition I found myself in recently, needing to ensure certain logging conditions are met for specific failure situations. So, I'll document how I went about setting these up for an integration test suite for an ASP.NET Core API. My whole sample solution and all the code on this article can be found on my GitHub.

To set up the tests, we just need a simple ASP application with an endpoint that logs an error:

var app = WebApplication.CreateBuilder(args).Build();

app.MapPut("/test", (ILogger<Program> logger) =>
    logger.LogError("Test error!")
);

app.Run();

If you start a new project today, you'll find that the ASP templates will set you up with a "classless" Program.cs file, but this is a bit of an issue when it comes to testing as we need to make the Program class public or internally visible to the test project. To set up the demonstration I just updated my Program.cs file to use an actual Program.Main method.

The Program class is used in our test code to set up a client to which we can send HTTP requests. I'll use XUnit to set this test up, but this solution works with any test runner. (Note that the test code requires you to add the Nuget packages Microsoft.Extensions.Logging and Microsoft.AspNetCore.Mvc.Testing).

[Fact]
public async Task TestErrorLogged()
{
    var client = new WebApplicationFactory<Program>().WithWebHostBuilder(builder => {}).CreateClient();
    var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Put, "/test"));
    Assert.True(response.IsSuccessStatusCode);
}

If we run this test now we'll see it passing. The /test endpoint is logging an error, but note that it is always responding 200 by default. Now, we need to capture that error log and update our test to check for it.

The web host builder, which we configure with the helpfully-named WithWebHostBuilder, allows us to alter the configuration and services of the ASP app. You can think of it as ammending the normal setup logic in Program so that the tests can inject their own configuration, replace services with fakes, or any other alterations that might be needed to set up the test.

In this case, we're going to want to configure how the ASP app deals with logs. ASP is set up with a default logger provider which it uses to resolve the ILogger<Program> dependency on our endpoint. We're going to want to write our own ILogger to be injected there, and in order for ASP to use our ILogger we'll also need to write our own ILoggerProvider and configure our app-under-test to use this logger provider.

These interfaces are quite easy to implement, especially since we only need to store logs in-memory so we can search through them in the test. I'll use a ConcurrentBag as the collection type to avoid any possible concurrency issues.

ILogger only contains three methods, and we'll really only be interested in the Log message, in which we'll simply add the log into the in-memory ConcurrentBag:

public record LogMessage(LogLevel LogLevel, string Message);

public class TestLogger(ConcurrentBag<LogMessage> logs) : ILogger
{
    public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter) =>
        logs.Add(new(logLevel, formatter(state, exception)));

    public bool IsEnabled(LogLevel logLevel) => true;

    public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
}

ILoggerProvider only has one method which returns an ILogger implementation given some category name. We don't care about the category name, so we can just return our custom ILogger:

public class TestLoggerProvider : ILoggerProvider
{
    private readonly ConcurrentBag<LogMessage> _logs = [];

    public IReadOnlyCollection<LogMessage> Logs => _logs;

    public ILogger CreateLogger(string categoryName) => new TestLogger(_logs);

    public void Dispose() => GC.SuppressFinalize(this);
}

I personally prefer to not expose a ConcurrentBag and instead expose it as an IReadOnlyCollection, but that's a stylistic preference.

The only thing left is to revisit our test method - we need to configure our ASP app to use this logger provider, and we should add a test for the error log. We can use the ConfigureLogging method on the web host builder to ensure our custom logger provider is the only provider, and to make sure logs at all levels are captured. After we send the request, we can then inspect the logs in our custom logger provider to confirm the expected log was logged.

[Fact]
public async Task TestErrorLogged()
{
    var loggerProvider = new TestLoggerProvider(); // [tl! focus]
    var client = new WebApplicationFactory<Program>()
        .WithWebHostBuilder(builder => // [tl! focus:7]
            builder.ConfigureLogging(logging =>
            {
                logging.ClearProviders();
                logging.AddProvider(loggerProvider);
                logging.SetMinimumLevel(LogLevel.Trace);
            })
        )
        .CreateClient();

    var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Put, "/test"));

    Assert.True(response.IsSuccessStatusCode);
    Assert.Contains(loggerProvider.Logs, l => l.LogLevel == LogLevel.Error && l.Message.Contains("Test error")); // [tl! focus]
}

And that's all! Again, all the code above is in a working solution on my GitHub.

There's Always Money in the Banana Stand

Fri, 30 Aug 2024 00:00:00 Z

The penultimate fallacy of distributed computing is "transport cost is zero." Anyone who has stood up their own test project on a cloud provider can intuit the fallacy here: dispatching any message across a transport layer requires more processing and uses network resources, each of which cost runtime and ultimately money for our application. In the context of a distributed system these costs can end up being huge, making it important to not just avoid the fallacy but to always be cognizant of the costs associated with any transport.

Naturally there are plenty of options to mitigate this issue, as I've explored on previous posts in this series. If (de) serialization is causing poor performance at the application level, MessagePack or ProtoBuf can save you a lot of resources - LinkedIn reduced latency by 60% with ProtoBuf. Keeping inter-service communication on a local network and geolocating friendly services will further reduce latency and isolate bandwidth concerns.

I think the 80/20 rule applies here - 80% of the achievable cost reduction will take 20% of the effort. That's not to say that it's easy work, but that it's relatively easy. How easy do you think it would be to switch your production distributed system to ProtoBuf? Now imagine how much work there is in the other 80%! I take this to indicate that there's a practical plateau as far as cost saving is concerned - at a certain point the juice is no longer worth the squeeze. That said, even if you were to realize all the potential cost savings in a distributed system, you are still distributing, which is inherently more costly than not distributing.

With everything optimized, you still incur a baseline cost in latency and network resource consumption with the message. On either end of that message, you incur a serialization cost. You incur a cost from the systems required to guarantee your required level of resiliency and fault tolerance (retry logic, load balancers, etc), and you incur a cost from your security systems (firewall, auth, etc).

In a way, each of the previous fallacies has preempted this fallacy, so there's not a lot to say here that isn't review. The key lesson here is that if you are distributing, then you are spending more money, period. Depending on your domain you might be spending a lot more money than if you weren't distributing. There are plenty of problems with monolithic systems, and while some of those problems are only solvable with distribution, most of the common problems that arise in these systems are just regular problems. Distribution isn't the magic wand to unspaghetti your bad codebase.

Looking more broadly than the costs associated in just transporting one message, you will incur costs in the telemetry, logging, monitoring, and alerting required to properly maintain a distributed system, even if your observability requirements aren't terribly deep. You'll incur costs with the engineering resoruces required to develop these systems, the operations resources required to deploy and maintain these systems, and the organizational costs required to develop and enforce the procedures that arise from owning these systems.

These fallacies demonstrate that all of these costs are necessary for distributed systems, and they reinforce the big leap in effort to create and maintain these systems. Taken all together, I see the fallacies all pointing to one glaringly obvious conclusion: unless the problem you're trying to solve cannot be solved but by distributing, then don't distribute. Maybe you've got a problem-ridden legacy monolith and you're looking at a large expenditure involved in refactoring. Microservices look attractive in this situation, but beware of underestimating the proper costs associated with the microservices transformation. Remember Grug on Microservices:

grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too

seem very confusing to grug

There Are Infinite Administrators

Fri, 23 Aug 2024 00:00:00 Z

In my recent exploration of the eight Fallacies of Distributed Computing, the first five have all regarded, almost exclusively, the technical aspects of distributed systems. However, as is the case for every software project, the human aspect is equally important. Each project is a combination of technical knowledge and humans - intelligent and flawed - capable of wielding it. This fallacy however - that there is one administrator - is almost entirely about the people aspect, and I'm very excited about that.

One of the things I find fun about producing small, personal projects (like FreePlanningPoker.io) is that I can know everything about the system. I know exactly how to build, deploy, and monitor the system, through the code, configuration, and cloud environment. I am capable of being the one administrator for this system.

The systems we're generally paid to develop, however, are rarely as simple as our small, personal projects. Indeed, even our personal projects can grow to the point that we don't know everything about them! That's generally the point where my velocity on them drops off a cliff, but that's another matter. What I mean to get at though is that our professional projects tend to be the size or complexity (often both) that there can't possibly be one administrator - one person (not one DevOps team, not one manager, but one individual human) who knows everything about running the system.

On the whole, it's not even advisable to only have one administrator even if it's possible for a given system. People come and go from their positions or firms with some frequency, or sometimes they just forget things; knowledge and experience needs to be distributed throughout the firm. As projects grow in size and complexity it's generally more efficient anyway to distribute the administration across the firm. And not only does our company have multiple administrators, the administrators aren't the only administrators! Engineers are administrators, managers are, BAs are. External companies can be, too, and that idea becomes very interesting considering cloud providers might introduce a lot of potential administrators.

The problem of course is that each administrator has a different understanding of the system, and sometimes these understandings conflict with each other. Each administrator's understanding of the system changes over time, and to successfully keep our system going they all need to interact with each other, with their overlapping knowledge. These administrators aren't just tasked with observing the system to diagnose problems, but they also need the ability to change the system in various ways. This introduces the issue that they can - and will - change our system architecture on the fly! Ahem topology doesn't change ahem

So our systems need to be open to change from various sources, and resilient enough to keep running if these changes aren't, shall we say, properly thought out. They can't become reliant on huge configuration files though, as we lose the ability to teach others about the workings of the system. Finally, they do need to have clear and high quality observability so that folks who might not be terribly knowledgeable about their workings can diagnose problems. This is particularly key in our distributed environment, where problems are frequently the result of many complex interactions between various components.

Service Design

Aside from some specific considerations regarding configurability, this fallacy doesn't inform our service design more than the other fallacies do, but it does serve to reinforce many of the same learnings we've taken from the others. It's another pin in the hat to reinforce our understanding that distributed systems are difficult and do genuinely require that much extra care.

Decoupling and Isolation

One of the primary themes (for lack of a better word) around developing distributed systems, and consequently my writing on these fallacies, is isolation. This is a double-edged sword which too easily looks like a cudgel to wield against any of the destabilizing factors which affect distributed services. On one hand, after doing the difficult task of identifying the unstable areas of a system, it's very easy and obvious to say that this is the part we need to code defensively against.

Indeed, there should be some such defense. The trouble comes in that each layer of defense is its own layer of complexity. Never forget evil demon spirit complexity. Isolating and decoupling services are thus better suited as guiding principles when approaching architectural decisions, with the goal being to find the simplest architecture, most parsimonious with the domain requirements, maximizing te degree of isolation without growing the architecture. This is a major reason that message queues come up so frequently in discussions regarding distributed systems; I think it's fair to say they can provide the most intelligent sort of isolation.

This is all with respect to the sometimes interesting effects which administrators can have on the system. In short, administrators can be the direct causes of the problems we explored with the other fallacies, so in a sense we are providing some degree of protection against the crazy and whacky admins by following best practices there.

Admins can (and will) change around the network topology, alter security management and configuration, and tinker with any number of properties (particularly with respect to cloud providers) that can cause any number of network unreliabilities. Those articles which I linked are focused on guarding against these negative properties of distributed systems, however the outlook is slightly different when we're considering this from the administration perspective. Admins need to have the ability to change these things, even though they can't know all of the deleterious ripple effects a change might have throughout the system. This means that the systems we create need to go the extra step to enable these changes to happen to the system.

Configuration and Observability

I had tried writing about these two under separate headers so as to explain them in more detail, but as far as the administrators are concerned they go hand-in-hand. The configurability of the system allows the administrators to be able to adapt the system in response to new errors, a change in the environment, or whatever they might need to do. Typically, it's the observability of the system that is able to give them the insight that something changed, there's a problem, and points to where the fix needs to go. Sure, often we do need to reengineer parts of our system's components in response to errors we observe. However, as we've discussed at length on this series on the fallacies, a lot of system errors come in through the environment and configurations, and configuration can be the solution - if we let it.

Both the observability of the whole system and the configuration of each individual component needs to be obvious. Remember - there is no one individual with everything in their head, nor can we count on everyone's knowledge being up-to-date. As engineers, we tend to talk a lot about self-documenting code, but remember that my observability graphs and the system configurations should also strive to achieve the same level of clarity!

Clarity being the most important factor in designing this aspect of our system, it should also be designed to fit its purpose - observability should be strictly focused on identifying what issues are happening, why they are happening, and where they are happening. A report on the throughput of the entire system is useless, as while it might identify that throughput drops too low it cannot identify where the issue is. Nonspecific reports shouldn't be included, and each report needs to be able to be correlated - easily - to activity and system components.

Then too there's a potential problem in moving from an identified issue to a configuration fix in a particular area of the system. Configurations need to be designed with a particular mind for who might be updating these configurations and why. Just the other week I was attempting to diagnose an issue in a microservices architecture and identified that I wanted to attempt increasing the timeout policy for outbound messages from a microservice, only to open up its config file and find a separate timeout configuration for each outbound request, each named for the method in its code which made the outbound request. Needless to say, I increased every one seeing as I wasn't going to spend a day reading its source.

DevOps and Deployment

Sometimes I take this for granted (and you might too) but it's worth writing out: Continuous integration and continuous delivery (CI/CD) is one of the most important practices for any software system, distributed or not. These practices emphasize that engineers should continuously integrate their working code with the main codebase, and that the software should be continuously delivered as updates are made to it.

To me, the gold standard for components of a distributed system is to deploy automatically when any code is integrated into the main branch. This requires a fair amount to be in place in order to be effectively supported. First and most obviously, you do need a build gate guarding the main branch, and it's incredibly unwise to not include a comprehensive suite of automated tests along with that. You do need some kind of review process so that a single engineer can't yeet whatever they please into prod, and both the review and build processes need to be fast - like, measured in minutes fast. Finally, you must have appropriate monitoring and a people process around reacting to it.

With all of these in place, I can get features out quickly, but importantly I can get a bugfix out quickly. At my current firm, I've been able to have prod issues discovered, triaged, coded, reviewed, and deployed in 20 minutes.

These practices serve to reinforce robustness and redundancy in the system not by giving us a way to roll back, but a way to roll forward. I notice that having more components - particularly distributed components - in a system makes it more and more difficult for any one component to be able to roll back in the case of a failure. These system components are all connected by (sometimes not entirely explicitly defined) contracts that cause a high level of complexity in any system, well-maintained or not. Rolling forward is sometimes - a lot of the time even - the only realistic option.

Process Design

If the technical considerations I listed above seem a bit surface-level and handwaving at my previous writing, I think you're getting the same feeling that I do. As I mentioned before, this fallacy really concerns the human aspect of software development more than the technical aspect - that there is more than one human administrator is a recognition that impacts how we design the human administration of the system.

To reiterate, these human administrators need to be able to jump into parts of a system at any time and understand what it is doing, if something is going wrong, and the options available to them if something is going wrong. Then they need to be able to manipulate the system (as much as they can reasonably be afforded) without needing to engage engineers to change the code. Indeed, these administrators might well be engineers too, but when facing a high-impact incident it's really nice to be able to click a few buttons to get customers back up and going.

Each human administrator is going to have a different understanding of the system. Some will have a greater familiarity with larger parts of the system, but it's incredibly rare that anyone knows everything about the system. To complicate it, our memories aren't as solid as we typically like to think. Each person's knowledge of the system will change over time, and that doesn't just mean that they will eventually become more and more familiar with the system. On the contrary, they will forget certain aspects or old knowledge might become stale as components or the environment updates over time. Yet, each one needs to be able to administer the system, particularly if you're not sure which of them might be able to respond to the 2 AM P1 on Christmas.

Knowledge Sharing

This is maybe the most crucial thing to integrate into our processes at all levels - it's not just for administrators! This is the only tool you have to improve your bus factor, and the only tool you have to increase the creativity and maximize the contribution from your whole team. Continuously knowledge sharing is the boyscout juice of productivity.

But today we're talking about administrators, so today we'll share knowledge about them. Adopting a continuous, diligent practice of sharing knowledge will help administrators understand how to understand the system, the options they have for manipulating it, and staying abreast of changes and updates.

I have no evidence for this claim, but it seems to me that the majority of teams and firms have a limited approach to knowledge sharing, and I wonder if this would be one of those cases where if we polled the teams on how they felt that they would feel that they do a better job of it than they actually do. I mean to say that there tends to be a bit of complacency here - if Bob really knows a particular API, it's just easier to only go to him to address problems. Over time everyone might just start calling it "Bob's API", and over even more time everyone absolves themselves of needing to know anything about it - Bob always takes care of it! Bob's API has an unenviable bus factor, and the kicker is that none of this is even Bob's fault, even though his name is on it now!

Knowledge sharing takes a lot of forms, but it's inextricably linked with work sharing. Lectures and lunch-and-learns are great ways for an individual to offload knowledge, but they're not a great way for the others to ingest knowledge. The best way for individuals to offload knowledge are through active-engagement exercises like hackathons or pair work. The very best way to transfer knowledge is the "thrown in the deep end of the pool" approach, but this can also have detrimental effects if the target doesn't first know how to swim.

The best underutilized tool to get knowledge from one person to another is repetition. If our gradeschool teachers had only once told us that the mitochondrea is the powerhouse of the cell, it would not have stuck with us. They didn't tell us once though, they told us this once per year at least, and that repetition has reinforced that knowledge in all of us, to the benefit of meme culture a decade ago. If you have particular knowledge about key points regarding specific aspects of the system, you should repeat these points regularly across the various forms of communication you have with your organization. You don't need to start each morning in Slack with the same sentence, or set that sentence as your email signature (that's an example I just came up with but I might actually try that now). Rather, for these bits of knowledge which you can deliver as a soundbite, take the opportunity to do so just whenever it's relevant just with larger groups of people. Particularly do this for those more ephemeral, touchy-feeling aspects of the system - there's a lot more knowledge which we can convey in feelings than we typically consider.

Documentation

I will be the first to accuse myself of taking the typical software engineer's approach to writing documentation - that is, I tend not to, at least for technical documentation. The more technically focused they are, these wiki pages tend to become stale rather quickly. Whether outlining the broader system architecture, call patterns, the configurability of a component, take your pick! The facts change and it's a coin flip whether the document gets updated. If the change occurs while fixing a P1 it's several coin flips as to whether the documentation gets updated.

Documentation is necessary though, and I don't want to convey that I write no documentation. On the contrary, I just don't write documentation regarding those things. Rather, I find the best documents are those about static facts which don't need updating as time goes on. My prime example is decisions - what was decided and why was it? Historical snapshots which hold long-lived knowledge about a system. Long after I'm gone and Vendor A is replaced by Vendor B, it's still going to be true that we picked Vendor A at one point, and for some reasons. Some of those reasons might persist through to the Vendor B days, and this knowledge is as valuable as gold.

For an administrator dealing with a malfunctioning system, stale technical documentation is not just unhelpful but entirely detrimental. However, being able to scour through "we did X because Y" gives the kind of knowledge helpful in a debugging context. Such documentation, properly done, should list the facts of the system at the time the decision was considered, the thinking that went into the decision incorporating those facts, and the specific decision or action that was actually taken. In future, I can always compare the facts then to the facts now, and I can always consider whether the reasoning would still be sound given the context.

I've touched on this idea before in my writing, and perhaps it deserves its own treatment, but it will suffice here to just offer this advice. Document with great frequency, and treat any point where your writing is recorded (PR descriptions and comments, meeting minutes, kanban card comments) as opportunities for documentation for posterity. Take every documentation opportunity to state the relevant facts at the time, what the thinking behind a decision was, and never forget to write out what the specific decision was.

It's All Culture

So we've done all of the above. Our services are appropriately decoupled, each with great observability and a short turnaround time to deploy to prod. We've spread the knowledge around the team and our decisions are all documented in an easy-to-search manner. Yes, I can hear you laughing at that last one, but just suppose we've done that - what's it going to look like when it really hits the fan?

Maybe you'll be able to effectively triage and address the issue, maybe not. This hammers home the people element - how deeply have you integrated this all into your culture? There's always a relationship between our proess and culture, as well as our technology and culture; it's always that both are dictated more by culture than anything. I've seen it happen too many times that we sit down to define a process only for it to fall apart - if you want to change the process you need to change the culture. process = f(culture).

Open documentation and knowledge sharing, avoiding silos and planning for risk, redundancy, and succession need to be the fundamental ways that your firm operates. It's not good enough for a single team either! Your software (probably) doesn't exist because your team formed itself from the aether - on the contrary, a bunch of stuffy business folks got some idea or another and here you are. Your architecture is dictated by their requirements, and their cultural influence is just as important - maybe more important - than your own.

Remember that these stuffy business folks (okay, maybe they're not all stuffy) are administrators too sometimes, and they've probably got a very different understanding of things than your senior performance engineer. Well, they both probably agree that they want the whole thing to be fast. Point being: these considerations all become circular in a way, each pointing to the next.

In a way, this is the most important lesson for developing distributed systems. Maybe it's the most important lesson in developing software. Focus on people first, most things will fall into place if you get that right. In my work I'm reminded almost daily of a great Joe Strummer quote: "Without people you're nothing."

The Art of Hype-Driven Development

Wed, 22 Jan 2025 00:00:00 Z

From microservices to blockchain, from the dotcom bubble to AI, our industry has a surprisingly high hype cycle density. Many of us find this distasteful. However, as the old saying goes: if you can't beat them, join them; in this spirit I've developed the following principles to guide any aspiring grifters through the hypeverse. If you grind hard enough at conferences and social media - reputation be damned - you too can earn a massive profit for someone in Silicon Valley!

1. Follow the Money

If someone else is profiting massively over some new technology, do what you can to increase their profit. If money is the smart bet, then the biggest single pile of cash is the smartest bet. Since the best returns are always just five years away, forget present-day realities; focus all your attention on imagining the glorious future of this new technology.

2. Confidence over Comprehension

Historically speaking, the more revolutionary an idea is, the less it is generally understood. Thus, the most revolutionary things don't need to be understood, and trying to articulate them is a waste of time. Just be confident, and be sure to have a promise to counter every uncertainty. Remember that specific statements are often false statements, so stick to more truthful statements like generalisms and benign platitudes.

3. Ruthless Simplicity

Always simplify, then simplify some more, and don't stop. Don't write a paper when a sentence will suffice, and then turn that sentence into a single word - one with a good buzz to it. Don't release a complete product when an MVP will suffice, and then don't code that because you remember that the best MVP is a Minimum Viable PowerPoint. Use your buzzy word on that PowerPoint a lot.

4. Minimize Risks

The best way to minimize risks is to not have risks, and the best way to not have risks is to have opportunities instead. Your goal is to have zero risks - yes, zero! Pivot every risk into a future opportunity. Simplify your explanation into a word with lots of buzz, and put this word on your PowerPoint.

5. Organize your Community

If all else fails, you can always try starting a cult! The AI hypers have had to resort to this tactic but it's been very successful for them.

The Case for Single-Reviewer PRs

Sat, 13 Jul 2024 00:00:00 Z

As with all things, there are a lot of different practices our teams might adopt around pull requests (which I prefer to call "peer reviews", but we can agree on "PRs"). Different teams in different environments or circumstances will certainly benefit from different processes. If you've suddenly inherited a huge legacy codebase and you need features now then you're going to have a different process to a team that's cooking up large batches of copypasta microservices because your startup just got too much funding and decided to justify it by blowing money on AWS.

The majority of teams doing actual development in the real world (the percentage of teams this describes is inversely correlated with the amount of VC money available, I assume) don't need a whole lot of process. Less is more, really. There's one process that I prefer as a baseline for most teams, light enough to be able to flex when it needs but still providing comprehensive benefits.

This is the single-reviewer PR, where each engineer on a team assigns each of their PRs to a single other engineer on the team on a rotating basis per PR, such that each member of a team has an even number of PRs from each other member throughout the course of development.

Maybe you've tried some form of this before and it didn't work out, or maybe you've never tried it before and it doesn't sound quite right to you. Maybe you like this, but you're trying to articulate the benefits of this scheme to a skeptic on your own team. I'm going to try to show the benefits of this setup and extra considerations you might need to make, and I'm going to do my darndest to not say "oh, you just didn't do it right!"

No Review Jam

Having multiple reviewers on a PR can lead to a situation where each reviewer implicitly defers to each other reviewer. "There's 6 reviewers on this PR, surely someone will get to it!" This makes PRs take longer to be reviewed, or I have to constantly pester my peers for a review. I might need to constantly field questions like "Am I really the right one to review the PR?"

If the process requires multiple approvals, or approvals from specific people, has the problem of creating bottlenecks. These are particularly insidious; I've seen such processees implemented before to the effect that PRs would take multiple days on average to merge.

Having a single reviewer eliminates all these issues. It's a level set with the team - you're going to get PRs assigned to you and you need to review them then. There's no deferring to other people, and if you're a teammember who consistently fails to find time to review PRs then we have an opportunity for a conversation.

In typical fashion, I would keep a running tally in my head of the next engineer who'd be picking up my next PR, but there are advantages in massaging that order now and then. I worked on one team with a member in Frankfurt, and if I had an early evening PR in I knew I could assign it to him and have a review by the next morning.

Ultimate Knowledge Distribution

Most teams do, I think, value distributing knowledge across each of its members. This is such a significant consideration that Uncle Bob suggests it be part of our code of ethics. Not only is the single-reviewer PR the best PR process to encourage knowledge distribution, it might be the best single process to do so.

Each member of the team - from the junior out of college to the principal who's been there just a bit too long - is going to be reviewing the full breadth of work that the team produces. Of particular importance to those lower on the ladder, the juniors will be reviewing PRs from the principals.

It becomes everyone's professional responsiblity to take the time to understand what's being engineered. While you'd expect a more senior engineer to be able to provide a good review for a more junior engineer, they're advantaged by being required to consider the full breadth of work. The junior though is particularly assisted by needing to consider sometimes quite advanced code. There's no hiding, you've got to learn it.

Now, we certainly wouldn't consider it good to leave any engineer alone with a bit of code they don't understand and ask them to sign their name to it. These cases give us the extra advantage of promoting communication. When I get assigned a PR that doesn't make a lot of sense to me, I want to have two video calls at least (assuming we're remote): one with the author to walk through the what and why of the PR, and another call with a second engineer who understands the code better to answer any more technical questions. You might adopt something else, but this gives me great contact with the team. I especially recommend this for juniors to maximize exposure to others' knowledge.

Here's the most fun I have with this process: when someone is about to open a PR in an area that one member of the team knows a lot about, I will suggest that they ask for a review from anyone but that engineer. Indeed, the reviewer will (probably) need to reach out to that knowledgeable engineer in order to complete the review. This is, to me, the easiest way to achieve this sort of knowledge distribution.

Quality Reviews

If you haven't worked this way before, it might not seem intuitively obvious that it produces, I think, the highest quality reviews on average. I think the inclination would be that by distributing PRs around, you'd get quality reviews from those more experienced but not necessarily from others.

Instead, I find that the review-ability of everyone on the team increases with this scheme. The key is it's not just that you are receiving reviews equally from each member of the team, but that they are receiving reviews equally as well. After a few cycles of reviews, it becomes clear who is good at giving reviews and who isn't. Behavior tends to normalize across the team with more contact between its members, and team members are biased towards adopting the practices of the quality reviewers.

This, as with all things in software, requires communication, but the advantage is that the communication between members of the team is at the minimum required through this PR system. Indeed, it naturally creates more communication between members on average. This is the key that causes quality reviewership to spread across the team. And again, if there's one member who is consistently failing to pick up what's being put down, there's an excellent opportunity for a conversation.

Transparency and Trust

The natural outcome of increasing both knowledge distribution and communication across the team is increasing transparency. Team members are more aware of what's happening in the code, what features and bugs are being tackled, and even just how their peers are feeling. Sometimes it's the small things that we don't pay enough attention to.

Our tooling nowadays usually boast excellent automation features. One that I particularly like is when a distributed team uses one software for communication (Teams, Slack, etc.) you can set up an automation from GitHub to copy a conversation between a PR in GitHub and a thread on your communication platform, ensuring that your PR conversations are all recorded in the same place all of your other conversations are. This makes it incredibly easy for others to follow along, or to go back over previous conversations and decisions at a later date.

Now here's one disadvantage you might be imagining. Wouldn't members of the team who feel a particular attachment to some part of the system interject themselves here? Indeed, I have seen this happen, but it subsides over time. Because this process encourages knowledge distribution and accountability across the entire team, I find that members trust each other much more with this process, and tend not to interfere. Indeed, interference should be discouraged as in the PR the author and reviewer are engaged in a sort of mini pair programming session; their decisions are professional and trustable.

More often than not - in fact almost always - when I see the situation that one member feel a knee jerk need to intervene, it's a learning opportunity for them. Frequently the author and reviewer are engaged with a new problem which alters some fundamental part of the original code. Sometimes the documentation is entirely insufficient leading to maybe more of a reach in th code than what's strictly required. These are not cases which require interference, they're cases where the original engineer can stand to learn - maybe to learn from a mistake they made or to learn from the wisdom of their peers. And, yes, sometimes it's a principal engineer learning from the wisom of two junior engineers struggling to comprehend the demon-speak of some terrible regex. Shame on you for checking in that regex in the first place!

In reality though, these situations are rare even though they're incredibly beneficial (if for unexpected reasons). The knowledge distribution naturally causes a work distribution, and paired with the increased trust across the team means that a codebase will, over time, have fewer and fewer landmines of particular engineers' passions. It encourages us to keep attachments at a professional distance, and allows the sort of iterations necessary across the entire codebase.

The Modular Monolith Won't Save You

Wed, 19 Feb 2025 00:00:00 Z

It seems like it was just the other day that microservices were the cool thing to be doing, but as fads come and go the microservice fad has waned. What are the evangelists to evangelize now? Some have dipped their toes into the "modular monolith," which at times has both pleased and confused me.

I've been pleased that conversation has shifted to the benefits of monolithic codebases, although it would be nicer yet to replace this with conversation about the realistic drawbacks of distributed systems. I would think that monolithic code would be the default, that we wouldn't dream of distributing a system unless it developed to a point where some clear, specific principles would force the question of distributing. Maybe I'm an idealist.

That leads partially to my confusion over the new focus on the modular monolith. "Modular?" Isn't that just how monoliths - nay, any professional code - should be written? Can't it just be the default that we do our best to architect and engineer a really good bit of code, and that code all executed together until it absolutely can't? Oh yes, I'm definitely an idealist.

I fear that the modular monolith might take the same place that microservices had in the 2010s, with some degree of dogma, some degree of specific aesthetic concerns, and some degree of very vague rationale in their favor. Realistically, the microservices vs. modular monolith question is a false dichotomy, and this allows me to discuss the broader role of patterns and principles. Or, maybe I'm swapping out my idealism for skepticism.

Microservices vs. Modular Monoliths

In one sense these are natural opposites - many of those generally opposed to the proliferation of the microservices pattern holding the new modular monolith pattern up as a reaction. In any other sense these are two of an infinite number of ways to think about the organization of software.

The Rise and Fall of Microservices

Much has been written about this subject, some of which by myself on this blog, so I'll stay brief here. I can't recommend enough Death by a Thousand Microservices by Renegade Otter as a primer. "Microservices" is a term that loosely refers to a distributed system composed of dozens to hundreds of very-small independently-deployed services, each focusing on a narrow and specific domain problem. Ideally each one wholly owns its domain area and contains its own persistence.

The pattern arose at firms like Amazon and, maybe most famously, Netflix, as they attempted to tackle the most difficult distributed problems at a global scale. These firms didn't adopt this architecture for fun, nor did they do so out of an idea that it is the right way to architect software (or even specifically distributed software). Rather, they adoped the architecture because they had to - there was no other way to solve planet-scale problems as theirs any other way.

Being good colleagues, the architects and engineers involved with these firms published their results for the rest of us to admire and maybe learn from. Being bad colleagues, the tech evangelists observing this figured they could gain a lot of money or prominence by "helping" much smaller firms and teams with much smaller problems adopt this architecture. Microservices were presented as the antidote to all the crappy legacy code we had to work with up to that point. Code which, coincidentally, was monolithic. "Monolith" was made to mean "legacy" and "spaghetti", and "microservices" were made to mean the opposite.

Of course, the nicest thing to say about that line of thinking is it's misguided; bad code is bad code, but bad code that is distributed is more expensive. In the disarray of the hype, cloud providers have made a fortune, and each year now there's an increasing amount of crappy legacy distributed code that is so much more expensive to maintain than its previous generation of crappy legacy monolithic code. The architectures are neither the problem nor the solution; it's the code.

That was all as unsustainable as it sounds, and I feel confident now in saying that the microservices bubble has most certainly burst. The problem of course is that distributed systems are already hard - really hard - and they'll bite a team not knowing what it's doing. If pain isn't felt setting up a new, distributed system, it's certain to be felt down the line when some new business requirement reveals the system has been coded into a corner. The pendulum of excitement has been swinging in a distinctly new direction over the last couple of years.

The Rise of the Modular Monolith

I'm not very good at history, so I'm not sure when this term came about. Maybe it's been around for 20 years, maybe it came up a few years ago when I first heard it. Whatever the case, it's maybe the most commonly-cited anti-microservices pattern. Why that is, I'm unclear. I wonder if it's not partially a bit of clever marketing - microservices having been somewhat of a reaction to the perceived weakness of monolithic software, it makes sense that advocating for the monolith requires some answer to that perceived weakness.

However the case, "modular monolith" comes about and is now excitedly and widely discussed. Loosely it means a monolithic architecture that has good separation between domain boundaries. But wait, I ask: isn't that just how software is always supposed to be made - with proper separation of concerns? If this is all the "modular monolith" is, then why aren't we just saying "do a monolith but don't suck at coding?" I might prefer a step further even: if monolithic architectures should be the default then maybe we should say "don't suck at coding and don't distribute unless you really, really need to."

But that hasn't got a good ring to it so now we've got the "modular monolith," almost as though before microservices there were no well-written monoliths. And here's exactly the problem I perceive: because it needs some overarching explanation of the right way to structure software, it becomes a term that is open to being coopted into meaning something dogmatic. "Microservice" started as something incredibly different to how it ended up largely due to the widespread adoption of ill-formed ideas as to why software is the way it is.

Which One is Better?

I'm not going to answer that. Not just because I don't want to, but because I can't. Well, also because I don't want to. Microservices are better for Amazon and Netflix and the like. If you're starting up a brand-new system then a monolithic architecture is probably better. This is a false dichotomy though: these are not the only two options! In fact, you might adopt different definitions of "monolith"; consider whether you think a C# server with a standalone React frontend and a Postgres database is a monolith or a distributed system. Most systems today require some kind of distribution, but it's also the case that almost every single microservices system doesn't need to have a microservices architecture.

I do feel comfortable committing to the position that monolithic architectures are the proper default for new systems development, but even then I need to attach the boring caveat that "default" practices are overridable by well-considered reasoning and understanding of the system's domain. This lands me back at a more fundamental truth though: naïveté is the highest virtue when approaching a new system. Well, naïve in a smart sense, I suppose. Keep it simple; less is more; start with what you need then grow from there; so on and so forth.

If you're with me on that, then you'll probably agree it's a point in favor of the so-called modular monolith. "monolith more simple than distribute" + "more bias towards simple" = "more bias towards monolith than distribute". And look how cleverly I can continue to avoid taking a definite position! That's not due to a commitment towards noncommitment on my part; you simply can't make objective statements about the quality of any pattern, their goodnesses can only be judged in relation to some system, problem, or domain.

There are plenty more foundational principles on which I feel quite comfortable taking objective positions, though. In addition to the simplicity bias, I think it's easy to say that code which is written and formatted neatly and tidily is better. At least, more professional. Code with the right balance of abstraction and modularity is better, and even that is quite a bit more definite than our architectural patterns: less is more but it grows with the scale of the domain, system, and complexity.

These principles all seem to be pointing in favor of our modular monolith pattern, but taking that step from principles to an architectural pattern that may well pan out to be some new buzzword (just like what happened to "microservices") gives me pause. Let me re-ask a question I hinted at earlier in this article: Does the "modular monolith" just refer to "proper, professional coding practices also it's a monolith", or is it instead an actual architectural pattern? If it's the former it's not a pattern; "try to code good" is a universal ideal.

Patterns Won't Save You

Increasingly I get the sense that the "modular monolith" term is being defined as something of a fad pattern. Some patterns are quite well-defined, even broad ones: if I ask you to implement dependency injection you can get to work and your result is probably not going to spark a lot of debate as to whether your code "is" or "is not" a proper example. Other patterns are poorly defined: I've never seen "single responsibility pattern" mentioned during a code review that didn't spiral into debate.

Fad patterns, particularly the architectural ones, are in a weird state as regards quality of definition. We can make some definite statements about them: microservices should have lots of distributed services, blockchain systems are slow, and so on. At the same time there is a general confusion, or at least lack of consensus: implementation details get fuzzy and lots of ideologues have divergent prescriptions for different problems that arise. They have a general, agreed thrust that might be more aesthetic than technical, along with a large cloud of nebulous hype and half-thought-out implementation details. The microservices debacle shows that even with a concerted educational effort by our colleagues who actually know what they're doing, the nebula still wreaks havoc over plenty of systems.

The Role of Patterns

We certainly shouldn't understand architectural patterns as being too wishy-washy to be useful. Yes, they don't have definite prescriptions regarding implementation, but architecture sits around the implementation details a lot of the time anyway. These sorts of patterns give us a common language at a 10,000-foot view. For all the folly, if in a conversation I say "microservices" my interlocutor will get a good sense that the following conversation is going to involve topics of distribution, infrastructure, domain separation, and a lot of complaining about patterns!

Patterns are good pedagogical tools too. In spite of the fact that university educations are typically poor preparations for professional software engineers, any serious course needs some way to give a name to different concepts discussed. Students need easy ways to be able to distinguish between broad concepts, and patterns give that sort of naming foundation.

Similarly, patterns are useful for those of us who have been in the industry for a bit of time. Seeing as things change so frequently as they do, I don't only need some way to be able to latch on to new concepts, but also a way to apply those new concepts in my considerations at work. Patterns define a space where we can explore exercises in abstract thinking about problems, which is crucial to being able to integrate new thinking into our work.

These (and maybe a couple others I've missed) constitute the proper utility of architectural patterns. None of them have anything to do with the implementation of systems, as none of them are capable of describing an implemented system. They're abstract and exist for the benefit of our thinking and communication, not for the benefit of our fingers-on-keyboards work. These patterns are informed by real-world problems, and in turn real-world solutions are informed by them, but "informed" doesn't do any heavy lifting for us.

The Real World

The real world is a world that has a lot of variables in it. When we're considering how to implement a real system we've got (off the top of my head) business requirements, quality attributes, specific infrastructures or cloud providers, performance targets, security of both the system and the environment it'll be working in, the hopes and dreams of managers, particular egos within a team, competitions with other deliverables for the firm, technical complexity, and the degree of understanding of the domain or solution. Surely, there's dozens more.

These variables all present requirements your solution may need to fill, and don't roll your eyes at "egos within a team!" I'm quite serious about that one - keeping up development at a sustainable pace means keeping everyone happy. We're engaged from 8 to 5 in doing implementation, so details about those implementations affect our contentedness. Egos are an often-overlooked variable. At the same time though don't necessarily take it as being on par with "business requirements" - that is a huge hole itself filled with many variables.

The broader point is that the patterns account for a small set of requirement variables and propose a solution to satisfy those. Variables which aren't accounted for will alter the implementation of the pattern or maybe make it irrelevant. Maybe your business requirements do contain the set of factors that are addressed by some pattern, but also include an extra one which makes the pattern unsuitable.

There's no formula to apply here, and there's (as best I can tell) no way to teach how to navigate discovering a proper implementation for a set of requirements. There's only experience and diligence, and constant learning. Our patterns only help us with that third point. If you take from patterns the principles of problem solving, then you can solve any problem with the same principles. However, if you only satisfy requirements by applying predefined patterns, you've missed the forest for the trees.

Avoid the Trap

With this understanding, let me ask: how many of the ill effects of microservices implementations are ultimately attributable to a misunderstanding of the roles of patterns? How frequently has it been that some demonstration of microservices at a pattern level has found its way into our codebases? The problem might not be the pattern, it might be us. Both the "us" that becomes lazy when considering implementing a real system, and also the "us" that becomes lazy when teaching or mentoring. These are two sides of the same coin: the misunderstanding of the role of patterns gives way to a confusion of the pattern for an implementation prescription. This is the "pattern trap."

If the recent buzz around the modular monolith resembles the same follies as that around the microservice, are we falling into the same trap again? If I wanted to be cynical I might throw my arms up in despair that our industry is eternally captive to hype. There's plenty of indication that a similar thing is happening - popular teachers are lazily making prescriptions from a loose understanding of a pattern and we're all not immune from a bout of laziness while inflicting those prescriptions upon our own code.

I'll go back to the most hilarious aspect of this all (at least, hilarious to me): I still don't think that "modular monolith" means anything! Take a gander over to Wikipedia on 'modular programming' - "modular" isn't some ultra-sophisticated way of handling domain separation; it's just using classes. Or packages, or "namespaces", or whatever your language supports. You're already doing it! Monoliths are "modular." What gives?

This has potentially alarming implications. A survey of articles about the "modular monolith" offer a wide array of interpretations about what work "modular" is doing in that name, many of which incompatible with each other. One writer might take it to mean using DDD and another might take it to mean writing each module in a way that it might easily be separated out into a distributed service. These are wildly different.

One of the main problems with "microservices" was that the term came to mean less and less over time. It started as a very specific pattern describing architecture and practice, and over the years evangelists watered down and confused the meaning to the point that the popular understanding of the term became something like "vaguely distributed system and also the word 'event' should probably be used." The end result was to create an environment for a hefty set of poorly-considered and many times contradictory prescriptions to take hold. These became implemented on enough codebases that the popular zeitgeist now has a bad taste for the word "monolith."

And to think - "modular monolith" has started off not meaning anything! The pattern trap is a real trap, and trends tend to set these traps. There's plenty of new trends constantly about, and the modular monolith is one of them.

Principles over Patterns

None of this is to say that any pattern should be avoided, and that includes writing on both microservices and modular monoliths. The point is how we understand them, and how those come to inform our implementations. The role these patterns have is not one of prescribing solutions; they're intellectual exercises. Studying patterns will, hopefully, help one develop ways to consider problem solving in the face of different sorts of requirements. They're a component of developing principles to guide our approach to writing software. Principles which, if you're doing things right, will change over time.

The real requirements of our systems are sometimes contradictory even, and this frequently makes us have to choose least-worst options. Often, this even results in a contradiction of principles we hold. As a result, we engineers are frequently engaged in a process of reconciling these contradictions, and it's that process of working through contradictions that causes us to emerge with solutions to our problems, implementations of our systems, and hopefully new principles we can keep going forward.

The systems we actually implement are synthesized from the requirements and contradictions of the multitude of real variables surrounding them. The actual design of our systems will never follow 1-to-1 how microservice or modular monolith or any other pattern might describe. To implement a system we need to have the skills to be able to construct it based on the real requirements it has, not based on the ideal or ephemeral assumptions of the patterns.

The Network is Unreliable and Reliability is Scary

Wed, 21 Feb 2024 00:00:00 Z

When the Fallacies of Distributed Computing were first written in the 90s, networks were unreliable. The internet was unreliable, intranets were unreliable, even radio was sometimes spotty back then. In the last thirty years, we as an industry have taken this unreliable infrastructure and ... left it there. Packet failures, client timeouts, and the occasional solar flare continue to be a problem not because of any inadequacy on our part but because it's a flaw which is inherent in the system; no network can ever guarantee reliability. Radio is still sometimes spotty because, just like the internet, sending any information over large physical distances is always going to have interruptions and loss. The first Fallacy of Distributed Computing is to assume the opposite of this - the network is reliable - and it's first for a good reason: it really matters.

Now, I wonder if in the last thirty years we haven't actually made this problem worse. In the 90s there weren't a lot of microservice systems making exorbitant use of load balancers, firewalls, gateway APIs, and the like. These are useful tools, but each new component in the distributed stack adds a point of failure. These systems can and do fail in their own right, but the incidence rate of network failures specifically will increase as more of these components are included.

It seems then that we have a good candidate for a first fix: simplify the architecture! Does every microservice need its own firewall? Do we have multiple gateway APIs? A serious and focused audit of the system architecture can reduce a lot of layers in the distributed stack, and overall improve the reliability of the system - the most simple systems tend to be more reliable. However, this only gets us to a certain point - a lot of systems still require load balancing, and you're going to need a firewall somewhere.

Store and Forward (and Retry)

The simplest implementation to get some fault tolerance is to implement a retry when we see a network error after sending a request. This is store and forward, but I prefer to call it "store/forward/retry" as these all tend to be related. Intermediate systems like gateway APIs and load balancers and the like might have simple implementations of this pattern themselves. To implement this, you'll need to "store" the message, send it ("forward"), and then you can retry on failures as needed by resending the stored message. This pattern works well because it's very simple and provides a good level of recovery from some network errors. It's best practice to implement some sort of retry system - if the packet is dropped en route from client to server, you'll want to be able to resend that.

Idempotency

There is a problem here though - it is possible for our client to get a network error even if the operation did succeed on the server: suppose the response packet was dropped or the client timed out before receiving the success response. That means that we will potentially send the same request to the server more than once, leading to potential error on the server from reprocessing the same valid, successful request more than once. This is solved by ensuring the operations on the server are idempotent - that replaying the same request twice won't cause these sorts of errors. Indeed, idempotency should be a default for all operations in a distributed system.

Here's a problem on top of that though: some operations cannot be idempotent. I'm currently working in ecommerce, which is rife with examples of single-fire operations: ship an item from an order, send an email when an item was bought from your gift registry, charge a credit card, and so on. In some cases we're able to architect the systems around these to make it so that the web requests are idempotent while the operation isn't. In other cases, we need to be a bit more clever.

Here operation IDs can solve the majority of your problems - generating a UUID for each operation and storing which ones we've processed is a simple solution that works in many cases. That said, non-idempotent operations are sometimes dependent on message ordering in order to work properly. If you're not able to architect away from this, you probably need some more robust patterns. I'll touch on some of these throughout this article.

Boilerplate and Complexity

There's yet another problem with implementing store-and-forward everywhere - one that affects my day-to-day life much more annoyingly than misordering a customer's packets: now I'll have a bunch of boilerplate around my code! This is a bigger problem for some codebases and less a problem for others. If your application doesn't have a lot of requests and you only need a base level of resilience, then moving the boilerplate into a shared library, or consuming an existing third party library for this, can be sufficient.

On the other hand, if you're setting up a distributed system of even a moderate size, you've probably got a fair amount of traffic going around, and it might make sense to set up a more comprehensive scheme. Some third party libraries do help out here, and allow a sharing of settings across components or provide more intricate solutions for orchestrating some of the request policies across a whole system. Still there are problems here - suppose the client application goes offline before it's able to resend its request, now I might need to add some persistence somewhere if the system requirements need that level of resiliency.

A more powerful alternative to solve this problem is a message queue (MQ). An MQ acts as a standalone message bus for a distributed system, allowing your clients to send their requests into the queue and letting the queue handle all of the considerations to make sure it gets to the client. Most of them offer robust UIs to give you a good level of insight and control over the system, and there are several options that are widely used for this purpose. Now your client needs very little logic in the way of sending a request - it just needs to make sure the request gets to the MQ.

More Complicated Patterns

The very minimal amount of work that we must do when implementing communication across a distributed system - whether "distribution" in your context means a client and a server or a microservices cluster spanning the globe - is to implement redundancy against the fragile network. In my mind, this means that store/forward/retry and idempotency are the default. As our requirements scale and our systems inevitably become more complicated, these become insufficient either in that they can't satisfy the system requirements or that they don't appropriately guard us against the chaos of the network.

There's another complication on top of this; there's always another complication. Peter Bailis and Kyle Kingsbury, writing for the ACM in 2014 cited some of the only research I am aware of regarding the effects of network fragility on our systems and users. Their work produced this quote (a summary of one such citation), which, to paraphrase what the kids say these days, lives within me without payment of room or board:

Perhaps more concerning is [Microsoft's and the University of Toronto's] finding that network redundancy improves median traffic by only 43%; that is, network redundancy does not eliminate common causes of network failure.

Can we go beyond redundancy and introduce some patterns that will 100% eliminate the effects of network failures? Well, we can introduce some patterns beyond redundancy but we'll never get to 100%.

Asynchronous Communication

TCP, UDP, and HTTP primarily support request-response type models: I send a request, I wait, and then I get a response. So far we've been considering how this doesn't work, so I think it's natural here to feel that this way of thinking has adopted, at least a little bit, of the fallacy the network is reliable. If I can't rely on getting a response, or even that my request will reach its destination, or if I can't even guarantee that redundancy will be as helpful as I need, then it intuitively follows that relying exclusively on request-response isn't sufficient.

I touched very briefly on the utility of MQs for distributed communication. Indeed, they're a great tool to be able to add a wealth of error handling (and other messaging-related) logic without polluting my codebase(s). I don't think that's the most attractive aspect of these systems though. While most MQs are quite good at supporting a request-response type messaging model, they're exceptional useful if we can move to a fire-and-forget model where I send the request and know I won't get a response.

But wait, my OrderMicroservice needs to be able to get item data from the ItemMicroservice! How do we do this without request-response? The answer in a microservice context is data duplication. The service which is responsible for manipulating the data publishes notifications on changes to that data, and systems which rely on this data will listen to those notifications and maintain their own copies of that data as they need. This is how asynchronous communication works in distributed systems: notify, don't request. This way of thinking changes our system a lot; indeed, it upends our entire architecture. Beyond the change to the communication pattern, it makes data-owning services only singularly responsible for manipulating data, rather than being responsible for both manipulation and querying.

As different as it is from the "traditional" modus operandi, this has almost become the default communication scheme in distributed systems. Particularly systems with varied, geographically distributed components such as IoT and banking (yes, not everything is microservices yet!) It eliminates whole categories of errors, and in our case it helps to eliminate not just one area as a source of network errors (the response) but also supports other best-practice patterns that feed back to helping us maintain our rigidity against the network.

There's tradeoffs though, as there are with all things. Asynchronously communicating systems have trouble reaching consensus, and a lot of times this can be entirely impossible. That's not a dealbreaker for a lot of systems, though it's a major one if your requirements necessitate it.

Outbox

One common point raised is that as more logic is added around outbound requests, the slower it is to handle those requests. In cases where my hot path is very hot and still needs to produce a fair number of outbound requests (as you might need to if you're notifying on all data change operations), I'll want to optimize my logic as much as I can. Perhaps it will seem attractive to not provide adequate robustness around my requests to make them faster.

The obvious pattern to use here would be to shuffle your message off to a queue running in a background process that will eventually publish the message, just outside the thread the hotpath is on. This works in a lot of scenarios, but there are robustness concerns yet with this. A helpful pattern here is the Outbox Pattern. The core concept is the same - we maintain a background process in a separate thread which handles sending messgaes with proper resiliency against the faulty network, however the enqueueing mechanism is the clever bit.

This pattern suggests that your database should have a table, or tables, containing the messages which you want to enqueue - this is the "outbox" table. When your application makes the update to the business objects in the database, in the same transaction it would add the message to be sent to the outbox table. The background message sending process then listens to this table (either by polling or by having the database raise events) to perform the sends. This is clever because, while you do need to write the logic to insert the message into the table, you don't specifically need to call the message publishing service to enqueue the message. On top of this, that you're using your database as the queue gets you a persisted queue for free.

This pattern is worthwhile if you've got a hot hot path, need the extra resiliency in your queue, and the extra cost of running the background process is worth it to you.

Event Sourcing

Some applications have a high need to preserve message ordering, usually because the state of the system is dependent on the temporal changes over the course of several events. These systems are good candidates for event sourcing, and this pattern can help us alleviate some of the pain of hardening our system against a faulty network.

This pattern imposes (very broadly) that you should save all of the events which alter your state, and that the state should subsequently be derived from these events. This is opposed to our traditional way of persisting data, where we process an event, update the state to reflect the changes specified by that event, and then forget the event. Event sourcing allows a number of benefits like being able to replay state, but what's interesting to us is that it allows inserting an event in the middle of a set of events which have already been processed.

This is beneficial to us if message ordering is high on our considerations list. If we're implementing proper resiliency when messages are dropped on the wire, we're going to be retrying messages, and there's a fair chance we're going to be sending some messages out of order in this scenario. As long as our events are properly dated, they can be ordered appropriately (and change the state appropriately) in our eventually-consistent system. This pattern also has the power to transform some non-idempotent operations into idempotent ones - instead of changing the state directly in a non-idempotent way, we'd be inserting/upserting/updating the single event in an idempotent way.

One word of warning though - event sourcing is a huge pain to implement and maintain. This pattern can very quickly get very complicated, and if you're careless then you can mangle data over time. Systems like EventStoreDB or the postgresql-event-sourcing plugin can help to make this easier, but that of course requires an investment in those systems. This is a pattern to study carefully and only use if it's appropriate for your use case.

Chosing the Right Solution

The trend which I think is plainly obvious here is that each next step we take in the battle against the unreliability of the network introduces greater and greater rippling effects across our code, infrastructure, and architecture. It's common advice for most any problem in distributed computing - not just this one - that the best solution is for us to upend just about everything we know about software engineering! Okay, I'm being a bit facetious, but as solid and well-proven as patterns like asynchronous communication are, they're quite different to other ways of writing software.

Difference between domains isn't bad in itself. A CRUD API is never going to look anything like a desktop application, which is going to look entirely different itself from any video game code you might find. It's probably a sign of being on the right path that we find that radically different domains produce radically different styles of engineering and architecture. Distributed systems aren't a single, contiguous domain though: the architecture we consider for a client-server system is itself going to be radically different from a microservices cluster, and yet they're both "distributed". Some patterns are going to work better or worse depending on the requirements of each different system.

Cost-Benefit

Extending the observation that the increasing steps exacerbate the complexity of our systems, we can say that the more reliable of a system you engineer, the more costly it is. I have no evidence for this, but it is maybe helpful to assume a sort of exponential or logarithmic growth in cost vs. reliability (I know "exponential" and "logarithmic" are very different but I slept through math class, passed by answering "C" on all the questions, and for my purposes they occupy the same category in my brain, right next to Snoopy and Woodstock). 100% reliability can never be achieved, so there is some point of reliability which becomes cost ineffective for your firm.

I don't have an answer for this. I can bring up patterns with a pros and cons list all day - I've presented a couple here - but this comes down to a study by your team and firm for its use case. Start by engineering a redundant system and extend (or replace) the components and architecture where you need. Since we can't ever achieve 100% reliability, we'll need to throw in the rag at some point and accept that failure has to be an option, but remember that failure is only an option if we've explicitly coded for it.

How Do We Know Our Solutions Work?

How do we know we've engineered a system which is appropriately reliable given our domain? Here's one more quote from that ACM paper:

Moreover, in this article, we have presented failure scenarios; we acknowledge it is much more difficult to demonstrate that network failures have not occurred!

Indeed, you can't prove that something didn't happen. With appropriate monitoring you can get an insight into how frequently network failures tend to occur, and you should track these and line them up with any cascading system failures. Generally, the absence of these indicators suggests strongly enough that there are no failures occurring.

Using this data to feed back to know if you've over- or under-engineered your system is a different beast, though. This is part of the art of our field. Especially with the patterns we're talking about here, it's almost impossible to do A/B testing. I recommend a gradual approach: start with the lowest amount that you need, and if your monitoring indicates failures in a particular way, then address those in the minimally-invasive way and repeat.

Conclusion

The network is unreliable and it won't ever be reliable. I've presented some of the general patterns to consider here, but there's a wealth of ideas on this topic. The job of creating properly resilient systems is about balancing tradeoffs as your requirements demand. There are dozens of patterns out there for dealing with this problem in different domains, trading off one bit of reliability for performance or one style of communication for redundancy, or any other tradeoff you might make.

Like the approach to the other fallacies, the right approach to this one (insofar as there is any "right" approach) is one of mindset more than code. The fallacies are so-named not because they're actual logical fallacies but because the point is to remind us it's a mindset problem. If we engineer our systems from the mindset that the network is reliable, then we'll end up with a deficient system because it's not true. If we adopt the proper mindset that not only is the network unreliable but also that it's a difficult and domain-specific problem to solve, we'll end up engineering thoughtful and considered systems.

There's More to Network Security than the Network

Thu, 30 May 2024 00:00:00 Z

As I'm reviewing and writing overviews of the fallacies of distributed computing, I'm trying to keep observations and suggestions at a 10,000-foot view: I want to describe each problem, related patterns, and convey the mindset behind the fallacy. Each fallacy is quite discrete, has obvious examples, and clear patterns that directly relate to it. The fourth fallacy, however, is a gigantic topic - "The Network is Secure" covers all of network security. I wonder if it's misplaced in the fallacies then, as we could very easily imagine a distinct set of the Fallacies of Network Security.

I'm not a security expert, I avoided the security class in university, and I typically find myself relying on colleagues for the more involved security considerations. That's unfortunate because security is so important and overlooked. One study cited by a University of Wisconsin paper suggests that 52% of networks are only defended at the perimeter. The link to that study no longer exists and I have a lot of questions about methodology and specific definitions, but I think we can all intuit that there's some truth here.

I'm not a great authority, then, to recommend specific practices for mitigating security issues; indeed, I'll allow someone else to write a series on the "Fallacies of Network Security". However, I can speak about the areas where security concerns come up, and why those areas are a problem. Just as well then, otherwise this article could be far too long! I hope I can convey the idea that even for those of us developing alongside a strong, security-minded colleague, we can adopt a holistic view of security issues and keep them in mind in the course of our work. I don't think I'm going to point out any new, groundbreaking information here, but it can be helpful to have these top-level ideas collated in one place.

Keep the Services Secure

The very baseline of any security is at the level of the services, and not the network itself as one might assume. This is the fallacy - the assumption that the whole network is secure leads us to develop more vulnerable services. We might develop services that are relaxed in their authorization or authentication, don't implement proper (or any) access control, or that don't properly validate user input.

There's a lot to be said about authorization and authentication. That you have auth in place is the first step, but there's a lot of considerations when it comes to the specifics. How you handle keys and passwords is important, and if you're relying on third party auth then the ways you go about securing your communication with them, and how you handle the keys you receive from them, matter. This is an important area of focus, for you and your team to thoroughly consider how these are all handled.

Access control plays into this - which users can access what data? Ensure you're appropriately applying the principle of least privilege - users should only have as much access as they've been authed to have. This is important at the level of each individual service, being skeptical of the security of the whole network is the idea here. Even guarding against internal denial of service attacks by implementing rate limiting might not be a bad idea, depending on your situation.

Even when I can be sure I'm correctly dealing with requests from a valid user, I still need to handle their inputs appropriately. This doesn't just end at sanitizing SQL inputs to guard against injection attacks - is your user able to input anything - say, regex - that can get around other data protections? Maybe your service's auth code is so complicated that it's easy for incorrect auth attributes to be associated with a request as it's processing and you accidentally give users higher privileges based on their own inputs. You might roll your eyes at that suggestion but I guarantee you it's less funny to encounter in the wild!

To boot, you can get issues with poor configurations - the use of default passwords is one that I encounter with some regularity. Most systems should have guidelines on security practices as regards their configuration, and this shouldn't be overlooked. This also bleeds from the service itself to how the servie itself is configured when running; yes, those running your service should be following your practices, but you should set them up for success by creating a service which is difficult or impossible to misconfigure.

Keep the Data Secure

The next obvious security concern is data transmission. I hope you're using HTTPS for everything! HTTPS makes it easy to start securing this area but it's not the end. Particularly if you're handling sensitive data like credit card numbers, you're going to need extra protections in place.

If you do have sensitive data, you probably want that data encrypted as it goes over the wire. Using proper, not-outdated encryption algorithms is important - there's still a fair amount of SHA-1 out there. Keeping up-to-date here is important too - algorithms and practices fall out of date like libraries, frameworks, or other practices.

Data integrity can also be a concern depending on your circumstance. It's good practice generally to use a hash as a checksum for your data, not just for the security aspect but to ensure your packets are transmitting correctly. That's quite circumstantially dependent and up to you and your team to choose whether it's needed in your case, but if you're transmitting sensitive data encrypted anyway you probably want this guard as well.

Access controls certainly help keep the data secure as well - if we appropriately guard who can receive which information then we've won half the battle. The regular suggestions for securing a network - firewall, maybe a VPN - help here too, but remember that if you're developing services within a network you can't rely on these being present or even helping you if they are. Security can't be assumed.

Data security has an internal aspect, too - sensitive data should be masked if it's stored internally in logs or audits. Properly enforcing how sensitive information is handled on your team is quite important not just because you don't want a rogue employee misusing it with malicious intent, but you want everybody protected if any of your internal systems are compromised. This is a vigilance mindset which is important to develop and maintain whenever we're working on our systems - you don't want to be the one whose laptop was breached and exposed PII!

Guard Against Third Parties

In my article on latency I pointed out that third parties can introduce latency onto us, and there's a similar relationship in a security sense - third parties can introduce vulnerabilities on us. This can happen whenever we have a connection with a third party, the obvious one being whichever third party services ours call into. A less obvious occasion for third-party-induced vulnerabilities that still effects everyone is when the building or delivery of our software is compromised - this process almost always involves external systems.

I think the first, most obvious step to guarding against vulnerabilities from a third party system is to make sure the system and the company owning it was appropriately vetted before relying on them. Something like Azure seems to be a pretty safe bet that they're going to do, and continue to do, what they can on their end to keep vulnerabilities away from you. Other vendors might be more or less trustworthy, but the point then is that you probably shouldn't replace MailChimp with BobsSuperGreatEmailService after a thorough review.

If you do need to consume BobsSuperGreatEmailService though, hiding it behind an isolation layer would be a smart move. That can be said for any system security-wise. Does every external system need an isolation layer? Maybe, that's up to you and your team to determine. There's pros and cons here, and the point is that security should be one of the considerations when working through this.

The supply chain vulnerabilities are the more interesting ones here. We all rely on third parties - sometimes a lot of them - to build and deploy our systems. Every codebase I know brings in several libraries through a package manager, just like those libraries rely on others, and so on down the line. You can be dependent on dozens or hundreds of libraries, and each of them is a security concern. Including a library should be the result of an audit, and these need to be continuously reviewed to catch vulnerabilities if they occur.

This might seem like a far-off case, but my friends in the .NET world will recall the recent incidence where the maintainer of the most popular C# mocking library put malware in a release, harvesting emails from the machines of any developers who built their code with the latest version of his library. The most charitable reason to give why he did this is profound ignorance, and it shows that security vulnerabilities have nearly infinite vectors to exploit. I commend Nick Chapsas for having shared that video as soon as he did; I was able to raise that alarm at my firm that morning and over the next two weeks one of my colleagues painstakingly removed Moq from our main codebase. Indeed, incidents like this are largely reactive.

Some of the most impactful and famous attacks have been supply chain attacks. The SolarWinds attack was a compromise of their delivery system, and the CCleaner attack saw legitimate versions of the product compromised with malicious code. While these third party vectors are some of the most obscure and difficult to guard against, these compromises can result in serious damage.

Guard the Whole Network Against Attacks

This seems like the easiest and first thing to do really - keep malicious actors outside my network of distributed components. As I keep mentioning though, this is the source of the fallacy! Indeed, we do have to do what we can to secure the network, however it's vital to ensure the individual components within the network, their connections outside the network, and the data transmitting across the network are completely secure. If those components are engineered to be appropriately secure, then the security of the network as a whole is an extra layer ensuring a lower number of vulnerabilities, rather than being the last line of defense.

On the flip side, you might observe that since all of the constituent components of the network are secure, it seems like the network itself doesn't need to be a focus of our vigilance. Au contraire! Not only do you lose the multiple layers of security which I described above, but there are some advantages you get from the network security layer.

Firewalls, VPNs, and intrusion detection systems can be configured for the entire network, giving a blanket level of security for all of its systems - not that they're relying on this - important! These intrusion detection systems are particularly interesting and can't really operate as well within the context of an individual service.

Denial-of-service attacks, particularly of the distributed sort, are famous and capable of taking down entire systems. Individual services should be guarded against denial of service attacks, but the network as a whole should be as well. Rate limiting, traffic filtering, and a good load balancer can be employed at the network level to alleviate this concern. I think cloud providers have products specifically for these sorts of attacks as well, but I'm not sure how effective they are.

The topology of the network can help its security too. Segmenting the network into isolated areas can help contain breaches, which combined with firewalls and intrusion detection systems can give you a lot of control and insight into the state of the system. Ensuring that no actor in the network are trusted - even internal ones - helps to maintain the focus on security throughout the system, forcing proper auth checks at each stage.

React to Attacks and Vulnerabilities

As I mentioned before, the number of vectors for attack are too many to be able to guard against all of them. Unless you have a particularly honed security mind (I do not) then you're unlikely to be able to imagine a lot of them. I don't want to discount the importance of building secure systems, but in a lot of ways the most important action to ensuring a secure system is to actively monitor and respond to attacks and detected vulnerabilities. Adopting a security mindset isn't just about being cognizant of the technical aspects of security, but realizing our limitations and building practices and processes to shore up our shortcomings.

Routinely scanning and auditing for vulnerabilities is the first step, and when we know that we don't know what we don't know (how many times can I say "know" in a sentence), this practice can be the best teacher for us. If your code is hosted in GitHub, it has excellent security scanning products which can help to stay on top of known vulnerabilities, particularly with respect to your code's dependencies. If you're fortunate enough to have a security-minded person on your team or a security team at your firm, they can potentially help you set up a process of auditing attack vectors in your services and system.

You do need to prioritize fixing these, however! It's no good finding vulnerabilities only to shrug and say "probably not gonna get found". Just because a system hasn't been exploited yet doesn't make it secure, and part of our professional duty is to deliver software as secure as we're able to. Known vulnerabilities must be prioritized.

Once we're in prod, we still need to observe our systems as they run. We can't prevent all security issues before we deploy, so being able to catch them when they happen is crucial. Security Information and Event Monitoring (SIEM) systems are purpose-built for this kind of monitoring - if you're securing a distributed system I'd make the assumption that you're able to devote resources for a product like this.

Again here, when alerts are raised they need to be prioritized and responded to - it's no good knowing the vulnerability exists (an if detected here, probably exploited) only to not fix it. It would be good to have a conversation about what the plan should be when a security breach is detected - this likely involves a lot of parts of the organization outside just engineering. I've seen a lot of firms - much bigger than you might expect - that didn't have these clearly laid out. Being explicit about what should be done for security incidents makes it easy for anyone to be able to react to the incidents when they occur, which strengthens the overall response from the team or firm.

Thing I Made: CFWeaver

Wed, 05 Feb 2025 00:00:00 Z

Last week, I wrote about some thoughts I had related to test generation and control flow in business logic. This followed an effort at work that naturally turned out to be a good candidate for a model-based testing strategy. From this, I developed the tool CFWeaver to generate test scenarios from control flow models, which I've worked on over the last couple of weeks to get into a state that might be useful for others in a similar situation.

CFWeaver is a very simple CLI: you input a control flow model and it outputs test scenarios to exhaustively cover all the possible paths through the control flow. The input file is written in valid Markdown with a special syntax, and it's able to output either HTML or Markdown. You can read more about its function on its GitHub repository, but I'll give a small example here from the readme. I might have the following control flow model defined:

# Get an Item

* Authenticate: Success | Failure = 403 ? I am not authenticated to access items
* Validate: Success | Failure = 400 ? the request is invalid
* GetFromDatabase: Found ? the item exists | Not found = 404 ? the item does not exist | Error = 500 ? The items table errors on select
* Authorize: Success = 200 | Failure = 401 ? I am not authorized to access the specific item

CFWeaver can understand this model, compute the various combinations of control flow states, and generate the following (each row of the table being a test scenario):

I've published CFWeaver at v0.9.0, with a small laundry list of tasks to get to "1.0" - I'll push some of those up to the repo as good first issues, if this project is interesting to you please do feel free to contribute!

This is far from a jack-of-all-trades though; I think there are some factors that constrain testing this way generally, and the utility of this tool specifically, which I think make the ideal use case for it a bit unique. There's not much to say about a simple CLI tool that can't be understood from its README, so below I'll elaborate a fair amount on my thoughts on using a model-based testing strategy in various scenarios.

Model-Based Testing

The overarching concept here is model-based testing, which covers any number of approaches to test generation where some model of the system under test (SUT) is made, from which tests are generated. The model could be any number of things we normally use to model our software: a model of the input parameters, state diagrams, Markov chains, or in my case a control flow diagram. Some tools generate full test suites you can repeatedly run, some will randomly compute tests at the point of test execution, and others will stop short of generating the full test in favor of some description of the tests.

That latter behavior - only generating a description of the tests - is what CFWeaver does. Generating a full test suite which could be run directly against the SUT would have a few negative consequences. First, CFWeaver would almost certainly require much more fine-grained input about the nature of the SUT to be able to generate full, meaningful tests. This, in turn, raises the complexity and therefore development effort on my end. Finally, it would probably also restrict the types of systems for which CFWeaver could generate tests.

So, in order to maximize utility and ease development, CFWeaver only generates scenarios to test, with the conditions to cause the scenario and the expected result. This makes CFWeaver suitable to generate scenarios for any kind of system at any level of testing (unit, integration, e2e, and so forth), though there are natural constraints on when you'd want to employ a strategy like this.

On constraints, the most important consideration before employing any model-based strategy is the utility of the model from which the tests are generated. Since the tests aren't generated from the SUT itself, there needs to exist some model independent of that system. Does that separate model have a utility other than allowing tests to be generated from it? How difficult is it to develop that model? Does that model need to be maintained and evolved in tandem with the system in order to continue to be able to adequately test the system?

These are all important questions to ask. If the model has no use outside of being used to generate tests and is difficult to create and/or maintain, I'd suggest you have a poor candidate for model-based testing. Any practice, testing practices included, will not work well if they don't work harmoniously with your whole software process. If you naturally have models as a product of your process and the sorts of tests that can be generated from that model are useful for the sort of system you have, then you might want to consider a model-based testing strategy.

Take a web service, for example. I work on web services a lot these days. Often, services will generate an OpenAPI spec file as a result of their build processes. This model is easily generated, and it has a lot of utility in providing documentation to consumers, maybe generating clients, and so on. A tool like Schemathesis that randomly tests your service based on its OpenAPI spec might naturally fit into the workflow for a service with a good OpenAPI definition.

I developed CFWeaver from a similar position. I wrote a new service for my company, and the business logic of this service was based on what essentially amounted to control flow diagrams which had been collaborated on by various stakeholders. These models define the system and are maintained by the stakeholders independently of the system; when the system changes it will be as a result of some collaboration over these models. Being able to generate all the test scenarios from this model was greatly valuable in that I was able to verify the expected business requirements for the various technical faults that were possible.

Black or White Box Testing?

Model-based testing supports both of these strategies, depending on how you come about your models. For review, black-box testing treats the SUT as a "black box" into which it cannot see; tests are based on external understandings of the system or the output from the system. White-box is the opposite; tests are based on information gleaned from the code of the system itself.

If I go into a codebase and read the code (or perform some other sort of static analysis) in order to come up with a model of a system, then I'm using model-based testing in a white-box manner. The project I described above that caused me to make CFWeaver is a black-box project; the model exists independently of the system, and the system is tested in a code-agnostic manner to ensure it conforms to that specification.

White-box approaches are good for testing technical aspects of the system and uncovering unexpected business requirements or lack thereof. For a web service, the return codes for various types of technical failures often fall under the purview of the business requirements for a system, and sometimes these scenarios can't be understood independently of the implementation of the system. Creating an independent model when analyzing a codebase for these sorts of technical states is usually beneficial, and a tool like CFWeaver can help after setting up the model if manually generating all the different scenarios is too complex or tedious.

Such an approach is (probably; usually) a one-time spelunking, and the model is probably discarded after the analysis and test scenario generation. However, it is a natural product of the analysis and useful for engineers collaborating while understanding a system. That's particularly useful when analyzing legacy (or otherwise poorly-understood) code.

I'm generally of the opinion though that the great majority of business logic can - and should - be tested from a black box approach. The software doesn't exist on its own, it exists because of an external requirement for its specific function. I largely don't care about implementation when it comes to ensuring the system meets those independent requirements, and on top of that it's beneficial to keep tests decoupled from implementation so that I can refactor the SUT without having to touch the tests; this makes refactoring easy and best ensures the success of the refactor (or any sort of change to the code, for that matter: feature add, bug fix, and so on).

I do not know, but if you asked me to guess I would say that model-based testing probably is less naturally suited to black-box testing because it's more rare that formal models exist independently of the SUT. Indeed, the OpenAPI spec example is probably the most ubiquitous one, and I luckily landed in a project for which I was able to formalize the independent model. Rarity aside, if the stars do happen to align in a way that a project does maintain models independently of the system, then a model-based approach can be a powerful black-box testing tool.

Black-box strategies naturally exist in some level of ambiguity; there is a large bit of information about the SUT that is missing: the implementation. As mentioned, this is desirable to focus the testing on the actual business requirements, but difficulty arises in needing to comprehend the full scope of those requirements. Different permutations of input or function or environmental states all need to be considered, so any tool that can help enumerate these states is desirable.

Different systems are naturally suited to different kinds of modeling, and while every system does have a control flow, it isn't necessarily the case that every system is best understood by a model of its control flow. Plenty of system are though, and in those cases CFWeaver will be a good fit.

Three Laws

Wed, 20 Nov 2024 00:00:00 Z

There is plenty of conventional "folk"-ish wisdom to be had, and a lot of it is widely known if sometimes unintuitive. This knowledge coalesces in the form of popular "laws," particularly in areas like management and economics. Plenty has been written about the application of these laws to the software industry, to varying degrees of acceptability. The implications they have for us can be surprising, and studying them can give an intuition for the unintuitive: how to organize the best team, deliver the most successful project, or architect the best codebase.

Here's three which I find particularly helpful.

Parkinson's First Law

Work expands so as to fill the time available for its completion

That is, if I go through a pointing session and come out with a two week estimate for a task, it will take two weeks. If I estimate one week, it just might come out to one week. There are indeed efforts which take more or less time, but efforts afforded more time will use that time.

This is one of the inefficiencies caused by estimating, and it becomes especially difficult if a culture of "it's okay to point a little higher just in case" develops. This has the potential to cause a feedback loop, where tasks are estimated with a bit of wiggle room, that room is used (per Parkinson's), a new baseline is thus set and estimates expand again to maintain that "little room." If this sets in it will grind a team to a halt eventually - there won't be time for small tasks and important quality work like refactors and vulnerability patches will be punted in favor of feature development.

Do not estimate! There are other reasons beyond Parkinson's to not estimate, but this is a big one. Forecast instead. In fact, you don't need to measure time taken on cards, you just need to count cards to get forecasting that's more specific than estimation.

Law of Triviality

People in an organization devote a disproportionate amount of time to trivial issues

I think of this like an 80/20 proposition: 80% of our resources will be spent on the trivial issues and 20% on the actual thing we need to focus on. One explanation for why this happens is that the trivial issues are easier for many people to understand, biasing people to discuss these. Another reason is that these issues are easier to disagree about - if I'm wrong about a complicated technical issue that might call my competence into question, whereas being wrong about inconsequential issues has, obviously, no consequence.

Following the 80/20 idea, only commit 20% (the bare necessity) to tackling the issue needed. Should you have 15 folks on the call to go over the issue, or 3? Should that call be 60 minutes or 15? Should your meeting have an open - or no - agenda (allowing any topics to be discussed as they arise), or should you have an agenda with the 1-3 items that need discussing?

Do not expand! Liberally cull resources from the problem-solving space.

Jevon's Paradox

As the efficiency of consuming a resource is increased, the net consumption of that resource increases

This is a paradox as you'd expect the resource consumption to decrease, but instead the new efficiency induces an extra demand on that resource. This is a well-known phenomenon in urban planning: when lanes are added to a highway, it tends to cause the highway to have more jams than it did before.

As software engineers, we tend towards making things more efficient. New processors have more cores, algorithms run faster than old algorithms, cloud platforms can offer more "computes" per dollar, etc. We can fall in a trap thinking this can afford us to write code at higher-than-necessary levels, less efficient business logic, or that we can "just" use more cloud products. In the extreme this has manifested in an outsized adoption of distributed architectures; particularly microservices.

Do not complexity! Complexity very, very bad! When we write software or make architectural diagrams or the like, we talk about taking several "passes". There's the "get it to work" pass, the "make it pretty" pass, and so on. Always make sure you have a "make it simple" pass. Heck, do several! How many arrows and boxes can you take out of the architectural diagram? How much code can you get rid of? How many iterations over that list of widgets can you consolidate?

The Topologies They Are a-Changin'

Wed, 19 Jun 2024 00:00:00 Z

Some of my earliest memories are from stories that my grandfather used to tell. He had an interesting career and knew a lot of story-able folks, so like a lot of grandfathers he had a compendium of fantastic stories. One of my favorites was a time he taught a course to his colleagues at his work. He was an engineer at Honneywell and was required to give this lecture in spite of his less-than-adequate abilities as a teacher, and it happened to be on an unfamiliar subject. After this lecture his colleagues were asked to fill out a short review form for him; the punchline of the story was that my grandfather's favorite review was "Leigh took a boring subject and ... left it there!" This was greatly amusing to him.

The topic on which he was required to lecture was, of course being germane to this article, topology. His was a lecture on the subject in mathematics, so I'm hoping that network topology is a more interesting subject for us here, but be warned that I will take some delight if you leave a comment about this being a boring subject! I do not think so - network topology consists of the actual layout of all of the components and connections between them within a network, which is an area with which I think we all have hands-on experience.

The fifth fallacy of distributed computing is "topology never changes", the implication being that it changes constantly. Twenty years ago when these fallacies were written that was a concern, and I will suggest that it's a significantly higher concern today. Not only do we tend to build systems with more distributed components and more connections than we did twenty years ago, but nowadays we have managed systems, we throw load balancers everywhere, and I'm not sure any of us knows everything that cloud providers do behind the scenes. Heck, each time a client connects or disconnects that's a change in the topology, and sometimes it's not a trivial one.

Our topologies change constantly and mostly due to automated processes. Thankfully, this is an area where practice has mostly kept up with the times, and the industry actually seems to be relatively good at writing systems which are resistant, to some degree, to changes in topology. Cloud providers assign names to services by default, modern application frameworks make some best practices the default, and so on. This is an excellent start but we oughtn't allow ourselves to be lulled into a false sense of security here. Just as with the other fallacies, errors induced by topology can cause some wild behaviors in our systems.

Problems from Topology

Topology changes can be the cause of any of the problems covered in the other fallacies. Changing the location or address of a service can prevent its dependents from being able to contact it, adding ride-along services can induce latency, closely locating services can surface bandwidth issues, and so on for each of the issues we have (and will) discuss. To some extent then, applying the lessons learned from our exploration of the other fallacies will guard us against the problems we can encounter from topology changes.

Were the list of fallacies just a list of sources for errors from distributed computing, then including topology changes might well be redundant. Instead of being that list though, the fallacies are a matter of mindset - a list of human assumptions we ought keep ourselves from making. By adopting the assumption that the topology will change, and frequently, we can eliminate this category of network instabilities from affecting our systems.

Services

Obviously, a service failing and removing itself from the topology can have knockon effects if there are other services which depend on it; this creates a network instability and can be protected against with a number of patterns. Equally concerning though is when services add themselves into the topology.

These situations can cause balancing errors, sometimes localized to the area of the service itself, sometimes with ripple effects across the whole network. The configurations of the load balancers take on a greater importance here, even small inconsistencies can cause a service to underperform. At scale this can introduce issues related to latency and bandwidth.

Even just when the services change, say when an update is rolled out, should not be discounted as a change to the topology - while the new version occupies the same node as the previous version and fulfills the same contract, but it's not the same as the topology before the update. This makes these update times sensitive; the team should monitor the system at the time of the update, and it would also be best to retain with your historical logs when updates are pushed out. The more subtle a problem, the further down the line it's likely to be caught.

Connections

Services can be connected with each other in various ways. If a service needs to dispatch a call to another, that's an obvious connection. If one service consumes events which another is responsible for producing, that's another more abstracted connection. These connections can chain into very complicated dependency chains, and that complexity is a source of problems itself. If the complexity grows to the point that it can't be understood, or if appropriate telemetry isn't set up to give you insight into the system, those introduce human problems. Human problems are the most difficult to solve.

There's a physical aspect to the connections as well - services which are physically, geographically more separated will have more connection issues. Even in a perfect world without packet drops, those communications will be slower. Software-defined networking approaches can really help, depending on the scope of the system, and this introduces the same sort of configuration issue I mentioned with the load balancers. These systems can also become very much affected by service updates or other triggers that might cause the dependency graph to change - either by calling different systems or by routing more or less traffic over a particular connection.

This is where it's important to have a proper understanding of the conceptual topology; you might not be able to predict all the ramifications of a change, but at least you can get some good insights into what you might need to test. This is not just useful for large changes; if your topology is well-understood, this sort of analysis can become part of the working process.

Decouple Services from Topology

The way to prevent changes in topology from inducing these errors in the system is to decouple the system from the topology. This is almost a question-begging argument: "Disconnecting the system from the topology disconnects the topology from the system!" However, the system can't be disconnected from the topology, not entirely, as the topology arises from the system itself. The statement rather is to say that the system should be written with abstractions over the pieces which interact with the topology such that a specific topology is not required for any component to work correctly.

Maybe the most obvious example of decoupling services from the specifics of the topology is IP and DNS. I'm sure you would give me quite a quizical look if I were to suggest that send requests from one service to another using the target service's IP address instead of its domain name; the IP address of the service will almost certainly change, so we'd definitely break our connection if we did that! We use DNS to escape this problem. Well, others too, but we're focused on topology. Instead of having to rely on a specific about the topology (IP), we use an abstraction (DNS) that hides underlying changes from us.

Just as we have an expectation that services' IP addresses will change, we should expect that all aspects of the topology can change. I'm almost always a fan of not introducing abstractions until a codebase or system reveals a pattern that is clearly asking for an abstraction, but this is not a case for that. Distributed computing increases the complexity of our systems tenfold (at least), and a significant contributing factor to that is the necessity of a host of abstractions.

Abstract the Message from its Delivery

This seems simple conceptually but ends up being more interesting in practice. The idea is that messages dispatched across a network have a message that's being sent and some address it's being sent to; these should never be coupled with each other. That seems simple - we'll just throw the URL of the target service (ServiceA) in the configuration of the service making the request (ServiceB)!

That's a start - suppose we include a ServiceABaseUrl config in ServiceB, then when ServiceB needs to send a request it can send it thus:

result = httpHandler.SendRequest(
    url: $"{serviceAConfig.BaseUrl}/some/endpoint",
    request: packet
);

Oh wait, ServiceB is coupled to the endpoint of ServiceA now! In some very simple scenarios this could well be enough, but we would probably be wise to be more resilient. Sure, we could throw all of the individual endpoints into the configs as well, and that gives us more flexibility. But there's still a couple problems. Less importantly, we're going to have tons of config values. More importantly, even with the values of ServiceA's address(es) in configs in ServiceB, the latter is still coupled to specific addresses about the former. We'll need to involve another party in order to properly decouple these.

A configuration management system might be a good next step given our hypothetical ServiceB -> ServiceA topology. This would be a third party in our architecture, distinct from ServiceA or ServiceB, which would manage the configuration values across the system, or across parts of the system you care about. Instead of having ServiceB's configurations specified in a configuration file or some other scheme deployed with the service, they'd be in this distributed component. Not only would ServiceB dynamically access these values at runtime, but the configurations in the management system can be dynamically updated at runtime.

Thus, when the location of ServiceA changes, the configuration in the management system would be updated to point to the new location at ServiceA. This can get quite fancy if you also introduce a "downtime" config, specifying whether ServiceA is available or not, as it might be down momentarily while changing locations. While this gives us the proper level of abstraction, it has some negative effects.

By introducing this distributed component, ServiceB will have to dispatch a network request each time it needs a config value. Sure, we could do a cache about that, but now ServiceB has to go through the whole rigamaroll of handling that. Caches are difficult and we want to avoid them. This ties in with the other problem with using a configuration management system in this way - it adds a fair bit of complexity, needing multiple config values potentially in order to satisfy the abstraction requirements we have.

Fortunately, this idea describes the widely-known pattern service discovery, and there are dedicated systems which can support this. Employing one of these systems is essentially the same pattern as above but with the complex logic already built in. If your use case(s) are appropriately small, using your own configuration management with your own logic on top might be the right solution, otherwise I'd encourage you to look into dedicated service discovery systems.

You might notice a flaw or two with this pattern though. Each dependent service will need to maintain its own logic to connect to and understand the service discovery system; more microservices in the network will result in more updates to that state being required; the dependent services still need to dispatch requests in order to understand addresses; and so on. In short, if I have a complex topology, then this pattern may be insufficient.

The next step up would be a service mesh to provide an independent component dedicated to managing the traffic across the network, rather than just managing address configurations. Service mesh systems like Envou or Istio will have you deploy a sidecar alongside each service to handle inbound and outboud communication. These are centrally coordinated through a control plane to route requests correctly (as well as security, telemetry, and whatnot). This allows each service to minimize the amount of configuration required to dispatch its requests.

Async Communication

It might seem quite dramatic to deploy a sidecar application along with every service in a system, and indeed it's more resource-intensive and complicated on the infrastructure side. In practice it isn't terribly burdensome from an engineering standpoint, and it's not terribly complicated for the infrastructure team once they get it up and going. However, I sympathize with the inclination to want to avoid this if possible. Less is more.

I've touched on the concept of asyncronous communication in my other posts on the fallacies, but to quickly recap, this pattern can essentially be thought of as reacting to events instead of sending requests to get a response. In our previous example, ServiceB needed some resource from ServiceA so it dispatched a request to get that resource. In an asynchronous context, ServiceA would be publishing events notifying the system about updates to that resource, which ServiceB could then consume in order to maintain its own state for this resource.

From a data normalization mindset this can seem like an antipattern, but in a distributed context it's often quite beneficial, as it allows an ultimate abstraction between services (particularly within the context we're discussing related to topology) while minimizing processing and communication overhead. Not all communication can be made asynchronous, but adopting asyncronous communication largely obviates the need for the more complicated patterns discussed earlier to handle request-response communication.

Self-Healing Architecture

This is a key concept running a line through the considerations I've made so far. Self-healing architectures are, as their name implies, architectures which are designed to automate fault recovery as much as possible. For example, service discovery and service meshes provide tolerance by adjusting the routing through a system in the event of a failure at one point. Just as our services can be coupled to the topology when they dispatch requests onto the network, they can also be coupled to the topology when they encounter errors from it. Thus, implementing a self-healing architecture is a necessity in ensuring a proper decoupling between the services and topology.

Implementing proper deployment strategies with automated rollbacks are quite beneficial at the service level, and these are facilitated by the patterns discussed earlier. Blue-green deployments, whereby a new version of a service is installed alongside the old version, allows requests to be rolled onto the new version and rolled off if necessary. This can be done intelligently in a canary release strategy for more control. Using appropriate health check or other monitoring tools, these can be entirely automated.

Looking at a bit of a higher level, some of our self-healing patterns will themselves alter the topology, and so need to be carefully considered in the whole context so as to not introduce the errors we've discussed. Auto-scaling systems and load balancers will provide isolation across services and communication. The circuit breaker pattern, also discussed previously, is widely employed to isolate failures by breaking the connection to failed services.

Each of these, combined with rollback strategies at the service level, provide an excellent resilience for the system. Their necessity also demonstrates the central point that the topology is always changing, giving us a strange feedback loop where the topology has to change in order to react, in part, to the effects of a changing topology. This cascade is an interesting subject for study!

Communication and Monitoring

I've discussed communication, monitoring, and people process at the end of each of my posts on the fallacies, and I'm curious whether I've said enough or whether this discussion should take up half of each fallacy. Indeed, our processes are as important as the software we develop. We're never going to be able to engineer distributed systems which are perfectly resilient to the constantly-changing environments they run in, so errors are going to happen. People who implement a process involving both proactive and reactive communication with each other (very difficult, I know) and monitoring the production systems are the glue that holds it all together.

This is very important then. Proactively, when manual changes are made to the topology these should be communicated in a way that all stakeholders can efficiently be made to know about them and have an opportunity to react if necessary. Most changes in topology are automatically done nowadays, I think, so it occurs to me that manual changes are probably quite serious or impactful. All the more reason to communicate.

But that the vast majority of topology changes are minor and done without human intervention mean that we can only react to unhandled errors that crop up as a result. Where possible, systems which do these sorts of alterations should log what they do and when, allowing us to use our normal log monitoring tools to be able to correlate errors we observe with these updates.

Chaos engineering is, for a lot of engineers I know, the most exciting related process - probably more so for the destructive aspect. I don't know that employing this strategy is right for most organizations though, and this testing strategy really benefits larger distributed systems. In order to effectively deploy it, not only do you need excellent communication and collaboration, but the folks running the tests need to be quite diligent and competent.

Conclusion

The saving grace is that topology errors will impact more complicated systems over time. Sure, that can probably be said about all the fallacies, but I think this is the one where that primarily shows through. Given my simple example topology of ServiceB -> ServiceA, the topology isn't going to change that much and simple strategies will resolve problems there. However, that network still requires that we set up proper security, and network failues and latency still impact the user experience.

I tried to outline, particularly with request-response messages, a way to be able to ramp up onto more elaborate strategies to deal with topology abstraction. If you're developing a new system or distributing a legacy system, you might not need to worry about these issues as much up front, but you certainly need to plan for worrying about them. Architect your components in a way that they can be updated with these considerations in future, and ensure you have proper monitoring and processes in place to start with.

I think that network topology - understanding it and coding around it - is one of the more interesting aspects in distributed systems. I think I share that opinion with several colleagues, but maybe not most. If you find this subject boring generally, leave a comment as to whether I was able to convey a sense of interest or whether I left the subject there, so to speak!

Using Interfaces

Sat, 08 Jun 2024 00:00:00 Z

I've previously written about this problem in the C# world that interfaces are overused. Some codebases seem to have the idea that a DI container is incapable of handling a service class if it doesn't have an interface. Other codebases define inexplicable, empty IModel interfaces which just serve as a handle for reflection or something. Other codebases yet have the even stranger idea that every single class should have its own interface, and then if polymorphism is required, there are interfaces on top of those!

Last I checked, which was admittedly a while ago, this is a problem in the Java world as well. To limited extents the problem exists in other interface-supporting languages like Go or TypeScript, though there are many more opportunities than just the interfaces to footgun yourself with TypeScript types. I hope that we might be able to disabuse our codebases of these faulty notions about interfaces. I'm not going to propose some sort of SOLID but for interface use, I just finished writing about the problems of that methodology.

Rather, I'm going to motivate some instances when interfaces should be used, the suggestion being that if you are not encountered with one of the following then you should not use an interface. Interfaces are abstractions, and abstractions should be avoided until the codebase reveals their necessity. I'm very skeptical of so-called principles which suggest abstractions should be implemented in all cases, or indeed to be implemented before they are revealed through a first iteration of the code.

Polymorphism

Obviously, if you need multiple implementations of the same service, you should use an interface. PayPalPaymentService and GooglePayPaymentService are probably required to share an IPaymentService interface.

Polymorphism is particularly useful when I need to migrate from one implementation of a service to another implementation, but I can't do it all at once. Suppose my software had always processed payments through PayPal, but our firm decided to not renew a contract and now we need to switch entirely to Google Pay. Bit of a contrived scenario, but you get the gist.

If I had only ever had one implementation of PaymentService, it would not need an interface, and indeed I'm suggesting that this would have been correct. Much of the time we put interfaces on services with single implementations thinking about this potential future case that we're going to need to alter implementation. Being clear, I've encountered this situtation of needing to swap implementation a lot. However, I advocate for not implementing the interface before I start doing this swap because it's easy to add the interface at that point in time. I don't know what the future brings, I don't know which services are going to need implementations swapped out, so I don't want to pollute my codebase with interfaces until it becomes necessary.

Any IDE, even Vim, is going to be able to support you on this. If I have the single concrete implementation PaymentService, I can do a global rename of PaymentService to IPaymentService, create IPaymentService as a separate interface which gets a compiler error, then rename the payment service class to PayPalPaymentService implementing the new interface. All of my code now references the interface, and I'll only need to update the DI registration and pull public methods up into the interface. This takes 5 minutes, or no more than 30 if you have a particularly tortured codebase.

Layer Boundaries

Almost every legitimate use of an interface is going to be because there are multiple concrete implementations of that service, but there are legitimate cases when I need to hide the implementation from components across a system. I contend that this could not possibly be anything other than a very large system which requires lots of wrangling.

The best example would be the modular monolith architecture. In this architecture, the various modules (domains, areas, or the like) of the application are not allowed to reference each other. Instead, there is a shared library which provides the abstractions (interfaces) of each of the modules which they in turn reference. A central host or shell module will provide the entrypoint for the application and use DI or a similar pattern to marshal the correct implementations from each module to the others depending on their abstractions.

This pattern facilitates the individual modules to be distributed from the core monolithic application as becomes necessary, and it's a first step in implementing the strangler pattern. It's an important pattern in making sense of very large codebases, and although it introduces a number of problems itself which do require care to resolve, I don't think that its reliance on interfaces is one of them. It's certainly overkill to use this pattern on a small or a new codebase, which leads me to suggest that implementing interfaces for services in anticipation of a future implementation of the modular monolith is improper.

The Hexagonal architecture (AKA "onion" or "ports and adapters" architecture) is another example of using interfaces to support layer boundaries, where there is a domain boundary which all other layers reference, which contains the abstractions necessary to perform domain operations. Again here we want to be careful that this architecture only be used for codebases of an appropriate size and type, but it's inoffensive to use interfaces to support this sort of architectural style.

All Other Scenarios

Don't use interfaces. YAGNI. Ideally, when we see an interface in our code - when we consume one - some flashing light should be going off somewhere in our brain: this is more dangerous than consuming an implementation! I'm not suggesting anything dramatic, but consuming interfaces should be handled more diligently and with an absolute expectation of a change in underlying behavior. The casual use of interfaces causes us to forget this in a sort of boy-cried-wolf sort of way: because interfaces are ubiquitous and their implementations don't actually change that much in practice, we stop caring. An interface is an abstraction, and abstractions are to be treated with respect and fear.

One objection I can anticipate is that I did not account for using interfaces to facilitate mocking in unit tests. You might be inclined towards permitting interfaces for this case, but I am not. Indeed, I am decidedly anti-unit-test but I recognize a difference in opinion exists here. I prefer a combination of integration testing and property testing, along with relegating business logic to an area of the code that doesn't have dependencies - ports and adapters is good for this!

Okay but Seriously, Why Don't we Work in Single-Week Increments?

Sun, 11 Jan 2026 00:00:00 Z

Even if you're not following the Scrum Guide, a "sprint" has become the default unit of time for software engineering teams to encapsulate a single chunk of deployable work. This seems natural: we increment the product, test it, release it, then repeat. If you're not using Scrum you're probably still using sprints, and even if you rename it something dumb like "cattle drive" it's still a sprint. Sprints are defined by setting a goal for the team: in the next X days, we want to accomplish Y. Y is usually a list.

Two weeks is the default length of sprint time in our industry I think (I have no source to cite), though because one month was (arbitrarily?) selected as a maximum by the Scrum Guide you can find plenty of teams blowing this up to 4 or 5 weeks. There's some products and/or teams for whom this long measure is a necessity, maybe I guess. If that's you then this article doesn't apply.

For the last many years, across two companies and many products delivered, my update every Monday morning has been "I do not remember Friday, here's what I'm going to do today." I've recently stopped saying the bit about not remembering Friday. I genuinely don't remember Friday, and that's not entirely due to my upper-Midwest-level of alcohol consumption; plenty of teetotalers can't either. Or refuse to. It's probably more that I refuse to remember anything past the weekend. I've been told Severance is a good commentary on this phenomenon.

Weekends are an excellent natural break in the flow of things. 4 to 5 days (or 30 to 40 hours) is a nice chunk of time to accomplish a task. Why do I have to set a goal longer than a week? The longer I work in more-than-one-week sprints the less I understand the idea. There's plenty of sorts of work that require more than one week to figure; the week length is an individual human-level division of time. If there's work that needs to be scheduled on longer timeframes, I propose also having broader monthly goals which can be chipped away at in weekly increments.

A separate, but related, topic is that meeting every single morning about daily progress is probably functionally useless in the majority of cases, unless grinding swathes of sofware engineers down into nihilists is "functional". Meet on Monday and Friday to set and reflect on goals, respectively. Meet extra twice a month for the same purpose for the whole month. 10-12 meetings/month.

I've read of teams that already effectively do this in practice but they call the month unit the "sprint." I have no beef with you all, carry on your excellent work. To the rest of us, I propose that we should unite as we have nothing to lose but our "I don't remember what I did on Friday"s.

Why I Have This Blog

Mon, 09 Sep 2024 00:00:00 Z

In 2013 I, a young software engineer, purchased ianwold.com and set up a simple site to serve as a landing page for myself. I figured it would be good to have some internet presence, considering I wasn't terribly attracted to Twitter or other social media as a way of advertising myself (as was the fashion at the time). I published a quick post about a Sublime Text color scheme, and this initiated a nearly 10-year period with just six measly articles on my blog.

The last article I published then in 2016 as an introduction to the Sprache framework isn't so bad re-reading it. It could certainly be better, no doubt.

Over that decade I continued to work on my personal site - I have always made use of it as my landing page - though I neglected the blog. I spent a fair amount of time getting the site to look just as I want it to, I created my own static site generator for it (that project might be next in my sights to actually get across the finish line), and most importantly I upgraded my domain to the ultra-cool ian.wold.guru.

I continued to not touch my blog though. Being fair, it wasn't really something which I needed for any particular reason, and I've never really aspired to being a writer. No need + no want = no work; fair enough.

This changed a year ago when I decided to break my streak to write about deploying ASP projects to Railway, the minimalist cloud provider. I was deploying a separate project on Railway and couldn't find the information I needed in one place, so I wrote that article to document the process for my future self and others. This isn't terribly different from the technical documentation I wrote for work, but crucially it is for a popular audience. Such writing should be more approachable, and it seems that at least some effort should be made to prevent it from being too dry.

This coincided with a period where I was making some updates to my site (this is a good annual practice, BTW), and one of those updates I made was to include Giscus to allow for comments on blog posts, so I wrote another post on that. Days later, having returned from an excellent conference, I was inspired to share a set of links to some papers I like, which I thought would be pretty funny to call a "book club". Okay, maybe just a bit amusing at best.

That was my first three posts since 2016, all back-to-back, because I had a reason to publish them. Equally importantly, it did convey a sense of being alive on my homepage - there's nothing like opening someone's site only to see no obvious signs of activity for six years. That was exciting to me, and at that moment I remember jokingly thinking that I was then set for the next eighteen years!

As it happens, that was not meant to be. In the same way that all bad ideas come into being, I started thinking. It felt good to have those articles on my homepage, but did it feel good to write them? In fact, it felt a bit clunky and forced. I feel the information on them is good, but there's no doubt the prose could be better. That was an interesting thought to me, as I do need to write for my work, and increasingly so - I need to be able to convey sometimes complex decisions and requirements though writing somewhat frequently. As I mentioned above, at work I do have the luxury of assuming a baseline knowledge with the domain and the ability to be somewhat dry (seeing as it's a captive audience).

Wouldn't it be better if I was able to write more inclusively and with more interesting prose professionally though? This is a tautology ("wouldn't it be better if it was better?") so the obvious answer is yes. It occurred to me though, doubtless as it has for others, that writing publicly would force me to develop this skill. Thus, last year I began a writing streak that continues to this day - hoping to get four posts per month on average. And not just any posts, I figure the exercise should be purposeful for any poor souls who accidentally fall upon this blog.

So I'm here to report, one year later, on the success of this project. Though I have had a bit of a lower output in recent months with a busier schedule in the summer, I've been able to keep to publishing posts, some of which are even useful! I like to think I've gotten a bit better at writing in this way, though I suppose it will probably take a good deal longer to properly develop. Importantly though, I find that I do enjoy it a fair bit.

There's practical benefits to it beyond developing a personal skill. Chiefly, I think it's a smart professional move: it gives a wider SEO footprint, demonstrates your professional engagement, and it seems to carry a distinctive aesthetic quality which is overall beneficial for one's image. Now I might have gone and sullied that image with some clunky posts, but the other benefits are there!

Looking back, I think it's safe to say that reengaging with my blog was among the best decisions I made last year. I'm very happy with the result, and I'm going to keep this blog going forward. It's the best way to continue to hone a writing skill, I think it does (or at least can) present me professionally quite well, and it allows me to give a small bit of konwledge back to the community. If you read this and want to start blogging, I would certainly encourage you to start. I started with an HTML page hosted by GitHub pages, and that's really all you need, though there are a plethora of fun platforms or tools or packages you can tinker with.

If you like anything that I've written, I'd encourage you to reach out - you can comment on any of my posts, or cooler yet you can webmention me! I continue to have a general aversion to social media, and I'm excited by the prospect that a homepage can stand in as a platform for connecting with folks across the web.

Write Your Own RDBMS Versioned Migration Boilerplate

Sat, 25 Nov 2023 00:00:00 Z

If you're using a relational database, even perhaps for a small personal project, you've almost certainly had to have a serious think about migrations. Some databases require very heavily-engineered migration systems to be able to handle large, complex, and/or frequently-changing data. Other databases are very small and might do just fine with a single migrations file, or maybe even manual migrations computed off a shared schema script, if updates are few and far between.

There's many different migration strategies, and there's no one-size-fits-all approach - different databases and applications can require vastly different strategies. One of the best and most ubiquitous strategies is the versioned migration, where individual updates are stored in SQL scripts that have an incremental version. I like a setup where, on startup, my application consults a migration history table in my database to see what the latest migration is, and to run any new migration scripts in sequential order.

In my experience this setup works very well, and it can scale to a large size of project, database, or team. I start almost all of my projects - big or small, personal or professional - with this strategy, and I think you should consider making this strategy (or some variation of it) your default as well.

One holdup though - isn't that a lot of overhead for a small, personal project? Should I really be investing the time to set up Flyway for every little API I want to set up? I contend no and no. I keep a snippet of boilerplate code to handle these migrations that I copy for every new project. It's a small amount of code, and I can modify it as needed per-project if it requires anything special. Best of all, my entire database from the start is migration-versioned, making it easy in future to switch to another system or onto Flyway if needed.

Versioned Migrations

As I described, this strategy involves breaking your migrations down into individual scripts for each discrete migration, and assigning them a version. What belongs in an individual file is up to you. You can restrict each file to only containing a single update on a table or column, or you could say that each feature card should have a single migration script. I prefer an in-between where each script contains a logically coupled, discrete set of changes, such that the database is valid at any migration version.

However you split these up, you might have several migration scripts:

/ Migrations
  |- 01_CreateUsersTable.sql
  |- 02_CreateItemsTable.sql
  |- 03_CreateListsTable.sql
  |- 04_AddListIdToItemsTable.sql
  |- 05_AddUserIdToListsTable.sql

You will need some way to assign a version to each script. I prefer adding the version to each file name (as Flyway does, if I recall correctly) so that it's easy to see, and is aligned by version in my file system. If you prefer otherwise, you could maintain a separate map in your code or a config file from a file name to a version number, or no doubt any number of other strategies.

The other aspect of this migrations strategy is that we will need to maintain a table containing the migration history of the database. I prefer a simple table myself:

CREATE TABLE migration_history(
    "version" bigint primary key,
    "migrated" timestamp default NOW()
)

In this way, I can keep track of where each database is at and apply migrations accordingly. When my app starts up, I'll look at this table to see what the latest version is. If the app sees that there are migration files exceeding this version, then I can run those migration scripts in the order they're intended.

Run the Migrations

Our migration-running code needs to do the following:

Check the latest migration version, or create the migration_history table if it's a new database,
Find all the migration scripts after the latest version
Execute these scripts in order, updating the migration_history as it goes
Commit the changes (important)

I'll be demonstrating this with C# and PostgreSQL (via Npgsql), but this approach will work in any language with any RDBMS. The code should be straightforward enough for you to translate to whatever your case is.

public static class DatabaseMigrator
{
    static int GetLatestVersion(NpgsqlConnection connection, NpgsqlTransaction transaction) // TODO
    static IEnumerable<(int, string)> GetNewMigrationFiles(int latestVersion) // TODO
    static void RunMigrationFile((int version, string name) file, NpgsqlConnection connection, NpgsqlTransaction transaction) // TODO

    public static void Migrate(string connectionString)
    {
        using var connection = new NpgsqlConnection(connectionString);
        using var transaction = connection.BeginTransaction();

        var latestVersion = GetLatestVersion(connection, transaction);
        var newMigrationFiles = GetNewMigrationFiles(latestVersion);

        foreach (var file in newMigrationFiles)
        {
            RunMigrationFile(file, connection, transaction);
        }

        transaction.Commit();
        connection.Close();
    }
}

Then call DatabaseMigrator.Migrate(connectionString); from your startup logic, and it's all wired up! We can focus then on implementing each of the TODOs here.

GetLatestVersion is probably the most complicated of these, because we'll want to check whether migration_history exists before we try to consult it, and create it if not. Before we get started implementing that method though, we'll want to write a little boilerplate to excute some queries on the database.

static NpgsqlCommand GetCommand(string query, NpgsqlConnection cnonnection, NpgsqlTransaction? transaction)
{
    var command = connection.CreateCommand();

    command.Connection = connection;
    command.CommandText = query;

    if (transaction is not null)
    {
        command.Transaction = transaction;
    }

    return command;
}

static void Command(NpgsqlConnection connection, NpgsqlTransaction transaction, string query)
{
    var command = GetCommand(query, connection, transaction);
    command.ExecuteNonQuery();
}

static T Query<T>(NpgsqlConnection connection, string query)
{
    var command = GetCommand(query, connection);
    return (T)command.ExecuteScalar();
}

If you wanted to implement all the logic here to check that the result actually is a T and handle that in a special flow, you can do. However, this is good enough for me - this code is running in a way where it's unlikely for me to encounter an exceptional scenario but in an exceptional scenario, and if this code fails I want to let it throw anyway so that my app crashes and its health endpoint responds with a failure. Your scenario may well differ though.

With that, I can outline GetLatestVersion:

static int GetLatestVersion(NpgsqlConnection connection, NpgsqlTransaction transaction)
{
    var migrationHistoryExists = Query<bool>(
        connection,
        "SELECT EXISTS(SELECT 1 FROM pg_tables WHERE tablename = 'migration_history')"
    );

    if (migrationHistoryExists)
    {
        return Query<int>(
            connection,
            "SELECT MAX(version) FROM migration_history"
        );
    }
    else
    {
        Command(connection, transaction,
            """
            DROP SCHEMA public CASCADE;
            CREATE SCHEMA public;
            GRANT ALL ON SCHEMA public TO postgres;
            GRANT ALL ON SCHEMA public TO public;
            COMMENT ON SCHEMA public IS 'standard public schema';

            CREATE TABLE migration_history(
                "version" bigint primary key,
                "migrated" timestamp default NOW()
            )
            """
        );

        return -1;
    }
}

Given that, it's quite easy to implement GetMigrationFiles. The only peculiarity of that method is that it will return a tuple containing the version and file name for each file, so that it's easy for the other code to reference. Here I'm assuming all the migrations are in the "/Migrations" directory.

static IEnumerable<(int, string)> GetNewMigrationFiles(int latestVersion) =>
    new DirectoryInfo("/Migrations").GetFiles()
    .Where(f => f.Extension == ".sql")
    .Select(f => (version: Convert.ToInt32(f.Name.Split('_')[0]), file: f.FullName))
    .Where(f => f.version > latestVersion)
    .OrderBy(f => f.version);

This can probably be more concise using query syntax and defining the version with let but I'll leave that as an exercise for the reader.

The only thing left then is to run these scripts and update the migration history:

static void RunMigrationFile((int version, string name) file, NpgsqlConnection connection, NpgsqlTransaction transaction)
{
    var query = "";
    using (var reader = new StreamReader(file.name))
    {
        query = reader.ReadToEnd();
    }

    conn.Command(connection, transaction, query);
    conn.Command(connection, transaction, $"INSERT INTO migration_history (version) VALUES ({file.version})");
}

That's it! With just a 100-line C# file we've got fully-versioned migrations!

Conclusion

You can find all the code together on this GitHub Gist. I copy this for every project I start, and I start each database out with versioned migrations.

You don't need to start your project off with a dependency on a third party migration library, you don't need to jump through any hoops - technical or conceptual - in order to get versioned migrations, and starting out with this puts you on the most solid path from the start. In future as your project evolves, if you end up in the rare situation of needing more features in your migrations, the code is right here for you to add it! If you end up needing so many migration features that a library like Flyway makes more sense, your story for switching to Flyway will be very easy. indeed.

Happy migrating!