Sam HogarthSam Hogarth's articles and presentationshttps://samhogy.co.uk/favicon.ico2024-03-25T00:00:00Zhttps://samhogy.co.uk/feed.xmlSam HogarthSingle Table Design for the Pick'Em App2024-03-25T00:00:00Zhttps://samhogy.co.uk/2024/03/single-table-design-for-the-pickem-app/This one was a bit of a mind-bender!<style>
table { border: 1px solid #aaa; overflow-x: scroll; }
td, th { padding: 0.5rem; border: 1px solid #aaa; }
th, .a{ text-align: center; font-weight: bold; }
.l { text-align: left; }
th, td { border-right: 1px solid #aaa; }
.tc { overflow-x: scroll; }
</style>
<p>My next job for the Pick'Em app is to define the application's data model. In the previous post, I <a href="/2024/03/creating-a-pickem-app-with-aws-amplify/">set up all the cloud infrastructure with AWS Amplify</a>.</p>
<p>I chose to use <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html">DynamoDB</a> for the backend storage, mainly to familiarise myself more with this style of database. DynamoDB is a scalable 'NOSQL' database, which in some respects can be considered a Key-Value hash map, with arbitrary runtime data attributes.</p>
<p>Traditional SQL databases specify a data schema up front, encoding entities and their relationships into tables, with primary and foreign keys as identifiers. Data storage is highly normalized, removing the need for data dpplication. At runtime, a user can execute arbitrary queries through constructs such as <code>JOIN</code>s. Although there have been decades of optimization, SQL performance degrades with data size, where the joining becomes a bottleneck.</p>
<p>DynamoDB does away with all of this in the name of scalability. Using DynamoDB effectively requires putting quite a lot of ingrained SQL knowledge aside and paying close attention to the application's data access patterns. Indeed, the prefered practice in DynamoDB land is to represent the application's data with a <strong>single table</strong>! DynamoDB's scalability comes from using the key-value structure as a means of <em>partitioning</em> data - with the right partitioning strategy, reads are distributed and fast, even under high loads. This also comes with the benefit of having a much simpler query structure, as SQL-style data joins are not supported.</p>
<p>The resulting schema looks nothing like what you may be conventionally used to from a traditional SQL database. Before delving in to the thinking and schema I devised for the Pick'em app, a note of caution. I'm in no way an expert and I'm sure there are improvements that could be made here. Out of a personal curiosity, please do reach out if there's anything I should be aware of! As it stands though, I'm quite happy with this design.</p>
<h2>Entities and Relationships and Keys (Oh My!)</h2>
<p>NoSQL doesn't mean there aren't entities and relationships! They're still crucial to developing a workable schema.</p>
<p>In a nutshell: <code>User</code>s will make <code>Prediction</code>s for the <code>Match</code>es in an <code>Event</code>, and their <code>Result</code>s will be aggregated into league <code>Standing</code>s.</p>
<p>As I am aiming for a single table, I've opted for some generic names for some key attributes.</p>
<ul>
<li>The primary key, known as the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.PrimaryKey">partition key</a>, is named <code>PK</code></li>
<li>The sort key is named <code>SK</code></li>
<li>The attribute that acts as the partition for the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.SecondaryIndexes">global secondary index</a> is <code>GSI_PK</code></li>
<li>The attribute that acts as the sort key for the global secondary index is <code>GSI_SK</code>.</li>
<li>Each data item will have an attribute called <code>type</code> defining which entity it represents.</li>
</ul>
<p>Together, a partition and sort key comprise a <strong>composite key</strong> to uniquely identify a data item. It is possible to store many items with the same primary key (partition key) in DynamoDB, providing the sort key uniquely identifies them. Further, as expected by the name, the sort key identifies the order in which queries items are returned.</p>
<p>With generic names for the key attributes, it's natural that I would also use a generic structure for the values for these partition keys too. A DynamoDB convention is to use the <code>ENTITY#ID</code> format for the key values. In some cases it may be required to go further, to <code>ENTITY#ID#EXTRAS</code>. The purpose of this structure is to enforce consistency, which becomes particularly important when it comes to sorting. As strings, these will be <strong>lexicographically</strong> sorted by default.</p>
<h2>The Database Schema</h2>
<p>Working this back-to-front, here's the single-table design I came up with. It requires a main table and one global secondary index. After which, I'll cover how this satisfies each of the data access patterns. Trust me, it will make sense.</p>
<h3>Main Table</h3>
<div class="tc">
<table>
<thead>
<tr>
<th colspan="2">Primary Key</th>
<th colspan="10" rowspan="7">Attributes</th>
</tr>
<tr>
<th>Partition Key: PK</th>
<th>Sort Key: SK</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="6s">EVENT#2023-03-03-aew-revolution</td>
<td rowspan="2">EVENT</td>
<td class="a">type</td>
<td class="a">GSI_PK</td>
<td class="a">GSI_SK</td>
<td class="a">date</td>
<td class="a">name</td>
<td class="a">state</td>
</tr>
<tr>
<td>event</td>
<td>EVENT</td>
<td>EVENT#2023-03-03-aew-revolution</td>
<td>2024-03-03</td>
<td>AEW Revolution</td>
<td>upcoming</td>
</tr>
<tr>
<td rowspan="2">MATCH#01</td>
<td class="a">type</td>
<td class="a">championships</td>
<td class="a">id</td>
<td class="a">name</td>
<td class="a">points</td>
<td class="a">preshow</td>
<td class="a">teams</td>
</tr>
<tr>
<td>match</td>
<td>[...]</td>
<td>03973653-e1f9-4262-86d5-02cbfd2cf962</td>
<td>Tag Team</td>
<td>5</td>
<td>true</td>
<td>[ ... ]</td>
</tr>
<tr>
<td rowspan="2">SCORE#140#sam</td>
<td class="a">type</td>
<td class="a">belongsto</td>
<td class="a">mcorrect</td>
<td class="a">points</td>
<td class="a">predictions</td>
</tr>
<tr>
<td>prediction</td>
<td>USER#sam</td>
<td>10</td>
<td>140</td>
<td>{ ... }</td>
</tr>
</tbody>
<tbody>
<tr>
<td rowspan="2">STANDINGS#2024</td>
<td rowspan="2">SCORE#140#sam</td>
<td class="a">type</td>
<td class="a">belongsto</td>
<td class="a">eplayed</td>
<td class="a">mcorrect</td>
<td class="a">points</td>
</tr>
<tr>
<td>standings</td>
<td>USER#sam</td>
<td>1</td>
<td>10</td>
<td>140</td>
</tr>
</tbody>
</table>
</div>
<h3>Global Secondary Index</h3>
<div class="tc">
<table>
<thead>
<tr>
<th colspan="2">Primary Key</th>
<th colspan="10" rowspan="7">Attributes</th>
</tr>
<tr>
<th>Partition Key: GSI_PK</th>
<th>Sort Key: GSI_SK</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2">EVENT</td>
<td rowspan="2">EVENT#2023-03-03-aew-revolution</td>
<td class="a">type</td>
<td class="a">PK</td>
<td class="a">SK</td>
<td class="a">date</td>
<td class="a">name</td>
<td class="a">state</td>
</tr>
<tr>
<td>event</td>
<td>EVENT#2023-03-03-aew-revolution</td>
<td>EVENT</td>
<td>2024-03-03</td>
<td>AEW Revolution</td>
<td>upcoming</td>
</tr>
</tbody>
</table>
</div>
<h2>Data Access Patterns</h2>
<p>In SQL schema design it's possible to initially define a normalized schema pretty much independent of the application's data access patterns, but in DynamoDB single table design this isn't really an option. I forsee the following data requests from the applications:</p>
<ol>
<li><strong>Get the league standings</strong> - a table of user's scores, highest first</li>
<li><strong>Get the details of an event</strong> (broadcast time, matches) so that a user can make a prediction (and if they've already made a prediction, return that too so they can amend it)</li>
<li>The same view as 2, but when the event has concluded and been scored, also return the results and a league table just based on scores for this event.</li>
<li><strong>Get a list of events</strong>, most recent first, which may be filtered/limited/paginated in future as the number of events grow</li>
</ol>
<p>These map pretty closely to the 'screens' of the application. What I'm aiming to do is get all of the data needed for one screen of the application in a single query. I don't want to have to make multiple calls to the databse, whether independent or sequential, to service one screen - as this is where latency catches up to you.</p>
<p>With this schema design, I've aimed to utilise sort keys very heavily to sort:</p>
<ul>
<li>the reverse-chronological order of events (most recent first)</li>
<li>the order of matches on an event card (in general, pre-show first, main event last)</li>
<li>the league standings (highest points first)</li>
</ul>
<p>I've also aimed to group the data as closely as possible.</p>
<h3>Get the League Standings</h3>
<p>Retrieving the league standings is a relatively simple query. For a given year (i.e 2024) the partition key is <code>STANDINGS#year</code>. The sort key is where the heavy-lifting is done. It's a combination of <code>SCORE</code>, the number of points, and the username: if user <code>sam</code> has <code>140</code> points, this is stored as <code>SCORE#140#sam</code>.</p>
<p>Note earlier I mentioned that strings in DynamoDB are lexicographically sorted. This is important as it leads to a potential hazard.</p>
<p>Consider the list of unsorted numbers: <code>11, 2, 10, 1</code>:</p>
<ul>
<li>Sorted numerically: <code>1, 2, 10, 11</code></li>
<li>Sorted lexicographically: <code>1, 10, 11, 2</code></li>
</ul>
<p>Avoiding this requires applying padding to the numbers. Based on the point scoring system, the number of matches per typical event and the number of expected events in a year, I've estimated an extremely gifted user would be able to score a total amount of points in the order of <strong>tens-of-thousands</strong> within a calendar year. Therefore, scores would be stored in the sort key as: <code>10000</code>, <code>01000</code>, <code>00100</code> <code>00010</code>.</p>
<p>Prefixing the username to the sort key adds further disambiguation in the event of a tie in scoring.</p>
<p>To achieve a result set that is reverse-chronological, rather than the chronological default, it's possible to specify the direction within the database query.</p>
<p><code>PK = STANDINGS#2024, ScanIndexForward=false</code></p>
<h2>Get the Details of an Event</h2>
<p>To predict, the user must first see which matches are on the card for an event. They may also want to revise their existing predictions, so if a user has any existing predictions, they should be returned too.</p>
<p>Event data is partitioned with a key comprising the entity type <code>EVENT</code>, the broadcast date (<code>YY-MM-DD</code>) and the name.</p>
<p>The event's metadata, such as the <code>name</code>, broadcast <code>date</code> and <code>state</code> are stored within an item with the sort key <code>EVENT</code>.</p>
<p>Each match is represented as a single item, with its sort key being the entity type <code>MATCH</code> and its numeric sort order. The same lexicographical sorting technique for points is applicable here too. Sort orders are padded to support <strong>tens</strong> of matches per event. Separate to the match position on the card (which can and does change in the build-up!) is a consistent <code>id</code>. This is just another attribute within the item and is a uuid. Alongside this attribute, there's a bunch of attributes to list the championship(s) being defended, how many points are available, whether it's a pre-show match (boolean), and an array of <code>teams</code>. And, for when it is eventually scored, a <code>result</code> attribute storing the index of the winning team.</p>
<p>Actually, it's probably worth diving into the <code>teams</code> object. Wrestling matches can have multiple participants, optionally grouped into teams. Those teams may have names. One (or more!) of the participants, or teams, may be defending a championship. To satisfy the available combinations, a match consists of:</p>
<ul>
<li>An array of teams, minimum 2</li>
<li>Each team has at least one member
<ul>
<li>A team may (optionally) have a name</li>
<li>A team may (optionally) be defending a championship</li>
</ul>
</li>
</ul>
<p>Let's illustrate with an example:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"championships"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"AEW World Tag Team Championship"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"id"</span><span class="token operator">:</span> <span class="token string">"05389aab-1f1b-44d0-9cfc-2e6d9cfd487f"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Tornado Tag Team"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"points"</span><span class="token operator">:</span> <span class="token number">25</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"result"</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"teams"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"champion"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"members"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"Sting"</span><span class="token punctuation">,</span> <span class="token string">"Darby Allin"</span> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"members"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"Matthew Jackson"</span><span class="token punctuation">,</span> <span class="token string">"Nicholas Jackson"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"The Young Bucks"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Also stored within this partition are the user's predictions and results. I actually store these in one item per user. The sort key uses the same <code>SCORE#value#username</code> format as the league standings, with the points value supporting <strong>hundreds</strong> of points. In order to associate a prediction/results item to a user, I add a <code>belongsto</code> attribute, which is crucial when trying to retrieve just a single user's prediction.</p>
<p>In the spirit of DynamoDB, with this schema I can get all of the data about a single query just by looking at the partition key:</p>
<p><code>PK = EVENT#id</code></p>
<p>In fact, at the point where an event is in the <code>scored</code> state (i.e. the results are in!), this is what I query for, with <code>ScanIndexForward = false</code> so that the highest-points scores come first.</p>
<p>When an event is still <code>upcoming</code> and accepting predictions, I just need to return the event details, matches and that particular user's predictions. DynamoDB supports filter expressions, which are executed after the Scan operation. Using the <code>belongsto</code> attribute on the user's prediction/result item, it's possible to create a filter expression that says "on a result where the <code>belongsto</code> attribute is present, it must equal the user's id". This will still pass through the event detail and match entries, but only then pass through a particular user's predictions, rather than everyone's.</p>
<p><code>PK = EVENT#id, Filter = attribute_not_exists(belongsto) Or belongsto = user</code></p>
<h3>Get a List of Events</h3>
<p>I've kept this for last because this query is satisfied via the secondary index. In the schema above, the details for an event are separate items to each of the matches. But, those detail items (that have the sort key <code>EVENT</code>) are stored within different partitions. Not great. Thankfully, it's possible to partition the same data differently on a secondary index. These are kind of like materialized view in SQL, projections of data into a different set of partitions. It's handled under the hood through data duplication, but essentially it enables another dimension of querying upon the data.</p>
<p>By swapping the order of the <code>PK</code> and <code>SK</code> attribute values, I'd end up grouping the data all under the same partition key (<code>EVENT</code>) and the sort key would be the lexicographically-sortable <code>EVENT#date-name</code>. So, this is what I do! The value of <code>GSI_PK</code> is the same as <code>SK</code> and <code>GSI_SK</code> is the same as <code>PK</code>. Querying for a list of events (just the details, not the matches!) is a simple query on the global secondary index, even retrieving the event listings in reverse-chronological order.</p>
<p><code>GSI_PK = EVENT, ScanIndexForward = false</code></p>
<div class="tc">
<table>
<thead>
<tr>
<th>Requirement</th>
<th>Params</th>
<th>Table</th>
<th>Operation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Get Events</td>
<td>-</td>
<td>GSI</td>
<td>GSI_PK = EVENT, ScanIndexForward=false</td>
</tr>
<tr>
<td>Get Standings</td>
<td>year</td>
<td>main</td>
<td>PK = STANDINGS#year, ScanIndexForward=false</td>
</tr>
<tr>
<td>Get Event Listings and Predictions</td>
<td>event, user</td>
<td>main</td>
<td>PK = EVENT#id, Filter = attribute_not_exists(belongsto) Or belongsto = user</td>
</tr>
<tr>
<td>Get Event Listings, Scores and User's Predictions</td>
<td>event, user</td>
<td>main</td>
<td>PK = EVENT#id, ScanIndexForward=false</td>
</tr>
</tbody>
</table>
</div>
<h2>Where We're Going, We Don't Need Servers</h2>
<p>Now that the data layer is in place, it's time to move over to the world of serverless! The next post will cover implementing the HTTP API for the application user interface, using AWS API Gateway and a Lambda function.</p>
Creating a Pick'Em App with AWS Amplify2024-03-15T00:00:00Zhttps://samhogy.co.uk/2024/03/creating-a-pickem-app-with-aws-amplify/Building the infrastructure for a cloud-native application!<p>This post journals setting up AWS Amplify for my Pick'Em side project application, covering the desired application architecture and how to provision that infrastructure through the tooling.</p>
<h2>About the App</h2>
<p>My friends and I watch All Elite Wrestling (I've recently <a href="/2024/03/aew-revolution-24-review/">reviewed Revolution</a>). Whenever they run a Pay Per View event, we usually exchange messages with our predicted outcomes for the matches on the event card. Many sports and live events have fantasy games where you can sign up, make your choices and score points based on the results - you can see similar Pick'Em apps for the <a href="https://picks.nba.com/pickem">NBA</a>, <a href="https://nflpickem.co.uk/">NFL</a> and the <a href="https://fantasy.premierleague.com/">Premier League</a>.</p>
<p>I thought that the scope of this would be self-contained enough that it would make a good candidate to build an application. Unlike the Premier League, there's not a Pay Per View event every week, so there is space to breathe with a near-term target date. For a while I've wanted to get properly stuck in to a cloud-native application that I have complete autonomy over, so all the mistakes can be made and learned.</p>
<p>The idea for the application then is that a user, upon invitation, can:</p>
<ul>
<li>See the current league table</li>
<li>View the events, both upcoming and previously broadcast</li>
<li>For an upcoming event, view the match card and submit predictions</li>
<li>For a broadcast event, view the results and how everyone scored</li>
</ul>
<p>I chose <a href="https://aws.amazon.com/amplify/">AWS Amplify</a> (gen 1) as a kickstarter framework for this project. It's a quick way of scaffolding out the core infrastructure that you need for a web or mobile application, such as authorization/authentication, storage, hosting and so on. As you add resources, Amplify generates all the necessary CloudFormation. It also tracks the deployment statuses of resources and provides a mechanism for pushing updates to the cloud. There's a visual development environment (Amplify Studio, which I didn't use), alongside out-the-box UI components for various frontend libraries. A JavaScript SDK wraps the AWS SDK functions with user-friendly helpers. In essence, it's a very quick way of getting to MVP.</p>
<h2>Proposed Architecture</h2>
<p>My bank balance isn't ready for the entire world to start using my application, so I'll deliberately limit it in scope to a handful of users. That said, I'll try and do stuff "properly" and learn as much as possible.</p>
<p>User accounts will be managed through AWS Cognito, as will their authorization permissions, being either a standard user or an administrator. Given the limited scope, there's no need to keep machines running constantly serving up the web application and backend APIs. I could build my own EC2 instances and let them autoscale down to zero, or embrace AWS Lambda functions. I'll choose the latter. The lambda functions will need to sit behind an API Gateway, integrating with Cognito so that only authenticated users can access the services. For data storage, I'll use DynamoDB. The database will track the game state - the matches, predictions, results and league table. The frontend application itself will be written in Vue, mainly because I want to try out a new framework.</p>
<img src="https://samhogy.co.uk/assets/pickem/architecture.png" width="800" />
<h2>Setting Things Up</h2>
<p>The Amplify CLI isn't part of the AWS CLI, it's installed separately (and globally) via <code>npm</code>:</p>
<pre class="language-sh"><code class="language-sh"><span class="highlight-line"><span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">-g</span> @aws-amplify/cli</span></code></pre>
<p>Once I'd installed that, alongside the AWS V2 CLI (new machine, fresh start), I then configured Amplify <a href="https://docs.amplify.aws/react/start/getting-started/installation/">as per the documentation</a> to refer to a specially-created IAM User. This is good practice as it limits the blast radius and capabilities of what can be done. The Amplify Administrator role is still fairly permissive, but it operates on the principle of least privilege.</p>
<p>Setting up a Vue app with Vite was as simple as running its creation script, which sets up a boilerplate application with a hot-module-reloading dev server that can be run with <code>npm run dev</code>:</p>
<pre class="language-sh"><code class="language-sh"><span class="highlight-line"><span class="token function">npm</span> create vite@latest</span></code></pre>
<p>Next is to create the Amplify backend by running <code>amplify init</code> and following the instructions. Most of the defaults were fine. This generates an <code>amplify</code> directory, which should be checked into source control. This is where any resources that you create, their configuration, and their code will live. Some contents are <code>.gitignore</code>'d by default, such as the auto-generated Cloud Formation, but inspecting it is a good learning opportunity to see what is happening under the hood.</p>
<p>Once initialised, you can then add your resources through the <code>amplify add <resource-name></code> command.</p>
<h3>Authentication</h3>
<p>First, authentication.</p>
<pre class="language-sh"><code class="language-sh"><span class="highlight-line">amplify <span class="token function">add</span> auth</span></code></pre>
<p>sets up a Cognito user pool with the authentication means of your choosing. It's possible to handle federated login from multiple identity providers, but for simplicity's sake I chose email. The setup instructions don't let you add user pool groups, to do that I had to run <code>amplify update auth</code> and follow the options to set up the groups.</p>
<img src="https://samhogy.co.uk/assets/pickem/auth-add-groups.png" width="800" />
<h3>Storage</h3>
<p>Next, the database. Now, Amplify's documentation pushes hard for a combination of DynamoDB with GraphQL and AppSync. I'll go into more detail about why I'm not taking this route in the next post, but for now, I'm going to go off-piste. The main reason for this is that I want to design my DynamoDB table according to the <a href="https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/">Single Table Design pattern</a>.</p>
<p>To add a DynamoDB database, run the command and follow the instructions!</p>
<pre class="language-sh"><code class="language-sh"><span class="highlight-line">amplify <span class="token function">add</span> storage</span></code></pre>
<p>It will prompt for a partition key, sort key, and if you want to add Global Secondary Indexes, how they should be partitioned too. I set up a basic table with generic partition and sort keys (<code>PK</code> and <code>SK</code> respectively), plus a Global Secondary Index with two more generic keys (<code>GSI_PK</code> and <code>GSI_SK</code>). I was surprised to learn that this sets up the table in provisioned mode - this is where you specify the expectations of read/writes in terms of capacity units. To be quite honest, given the low traffic levels this is most likely a very sensible default. But I'm a stickler for only paying for what I use. It took me a while to figure out how to switch to On Demand mode - it's handled by overriding the resource: <code>amplify override storage</code>.</p>
<p>Overriding a resource provides an <code>override.ts</code> file in the resource's subdirectory, from which you can modify the CloudFormation. I needed to delete the pre-configured <code>provisionedThroughput</code> property on the main table and the secondary index, and then set the <code>billingMode</code>.</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> AmplifyDDBResourceTemplate<span class="token punctuation">,</span> AmplifyProjectInfo <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@aws-amplify/cli-extensibility-helper'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">function</span> <span class="token function">override</span><span class="token punctuation">(</span>resources<span class="token operator">:</span> AmplifyDDBResourceTemplate<span class="token punctuation">,</span> amplifyProjectInfo<span class="token operator">:</span> AmplifyProjectInfo<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">delete</span><span class="token punctuation">(</span>resources<span class="token punctuation">.</span>dynamoDBTable<span class="token punctuation">.</span>provisionedThroughput<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">delete</span><span class="token punctuation">(</span>resources<span class="token punctuation">.</span>dynamoDBTable<span class="token punctuation">.</span>globalSecondaryIndexes<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>provisionedThroughput<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> resources<span class="token punctuation">.</span>dynamoDBTable<span class="token punctuation">.</span>billingMode <span class="token operator">=</span> <span class="token string">"PAY_PER_REQUEST"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h3>API</h3>
<p>I'm going to define two Lambda functions, for now. One is to handle API traffic for users, the second for administrators. Lambdas are added, you guessed it, with the add command:</p>
<pre class="language-sh"><code class="language-sh"><span class="highlight-line">amplify <span class="token function">add</span> <span class="token keyword">function</span></span></code></pre>
<p>As the functions are going to connect to DynamoDB, I opted for the DynamoDB template, which boilerplates you with an express application with some use cases for retrieving and storing data. Once defined, I needed to then run <code>amplify update function</code> for each, so that I could select "Resource access permissions" and permit access to the <code>auth</code> and <code>storage</code> resources.</p>
<img src="https://samhogy.co.uk/assets/pickem/function-update-resource-access.png" width="800" />
<p>The Lambda functions are not invoked directly by a user, rather, they're invoked by events trigged from an API Gateway. Adding this is a case of running <code>amplify add api</code>, choosing REST from the options (opting for GraphQL takes you down the whole AppSync route). You're then prompted to define API routes and what to do with them. I defined user API traffic as coming in on the <code>/app</code> path, and directed this traffic to the existing Lambda function created earlier. I did the same for admin traffic, on the <code>/admin</code> path, to the other Lambda. Each route is protected - for the <code>/app</code> path the user can be in either the users or admin user pool groups, <code>/admin</code> is just for admins. When creating the API Gateway, you are prompted with a choice to restrict access to a path. Choose "Individual Groups" to select from the Cognito user pool groups defined in your project.</p>
<h3>Hosting</h3>
<pre class="language-sh"><code class="language-sh"><span class="highlight-line">amplify <span class="token function">add</span> hosting</span></code></pre>
<p>sets you up with a CloudFront distribution for static content (that's the HTML, CSS, JavaScript and images) for the web application. I opted for manual deployments, although it is possible to connect to a git repository and have a fully-integrated CI/CD pipeline, such that when code is pushed it is automatically built and deployed. I'll probably come back to this at a later stage.</p>
<p>Adding hosting gives access to the <code>amplify publish</code> command for when you're ready to update the frontend.</p>
<h3>Domain Name</h3>
<p>Configuring a domain name is not done through the Amplify CLI. Running <code>amplify update hosting</code> will redirect you to the Amplify section of the AWS Console. I purchased a new domain name from Route 53, and then using the Amplify "Manage Domains" page, I could configure subdomain routing.</p>
<p>Right now, I've only the one amplify environment created (defaults to 'dev') and so all I need to do is point the subdomain to the Amplify environment. This will then begin to provision an SSL certificate.</p>
<img src="https://samhogy.co.uk/assets/pickem/manage-domain.png" width="800" />
<p>Amplify instructed me to add two <code>CNAMEs</code> into the DNS record for my domain, this is part of the SSL verification process and to redirect to the CloudFront distribution.</p>
<img src="https://samhogy.co.uk/assets/pickem/make-it-so.gif" />
<p>Making all of this infrastructure appear in the cloud is a case of running <code>amplify push</code>. Magic!</p>
<h2>Next Up</h2>
<p>Before getting started on the frontend application, I'm going to resist the magpie "ooh shiny" urge and focus on the database design. As mentioned above, I'm following the Single Table Design pattern for DynamoDB and this requires quite a bit of thinking and explanation to get right. But once in place, most other stuff should be a relatively straightforward implementation.</p>
<p>The next post will focus on defining the data model, based on the application's access patterns.
After that, I'll talk about building out the lambda functions for the app API, and then move on to the frontend.</p>
AEW Revolution '24 Review2024-03-14T00:00:00Zhttps://samhogy.co.uk/2024/03/aew-revolution-24-review/Wow!!<div style="text-align: center;">
<img src="https://samhogy.co.uk/assets/aew-revolution.png" />
</div>
<p>All Elite Wrestling has just held its first major Pay Per View event of the year, and it's one of the best wrestling events I have seen put together in a long time. 2022 and 2023 had been somewhat turbulent and the "World's End" event on 30th December 2023 acted as a form of 'reset', after what I thought had been a period of the company spinning the wheels following the hugely successful "All In" event at Wembley Stadium.</p>
<p>Since the 'reset', there's been a sharp return to form and the build for this event has been brilliant. Going in, we knew in advance this would be Sting's final match after a 40-year career. The event sold out quickly. It was going to be something special. Sting debuted at Winter is Coming 2020, with Revolution 2021 being his first match in AEW. Revolution 2024 bookends his three-year run in the company on an undefeated streak. Throughout this run, Sting has been booked as an icon. The '80s kids may remember the bleach blonde hair of Surfer Sting, us '90s kids remember him brooding in the rafters as Crow Sting. The vast majority of the card lived up to the expectations that come with saying goodbye to the in-ring career of one of the greats.</p>
<p>Let's dive into the matches. I'm not going to use a reductive star-rating system; I'm going to describe what I thought and felt.</p>
<h2>The Bang Bang Scissor Gang vs Jeff Jarrett, Jay Lethal, Satnam Singh, Willie Mack, Isiah Kassidy and Marq Quen (Pre-show)</h2>
<p>Whenever you get a match with the "Total Nonstop Assholes": Jarrett, Lethal and Singh, you're going to get something that's fun. This match isn't a bad way to start a pre-show, especially with main event talent Jay White in the mixup and hitting his finisher for the win. There's some storyline stuff going on, with Max Caster fluffing his rap during the entrance. I fully expect Bullet Club Gold to turn on The Acclaimed and unify the AEW Trios and ROH 6-Man tag belts at some point. It's a serviceable match to start things off.</p>
<h2>The Return of Pac (Pre-show)</h2>
<p>The Bastard Pac is from my neck of the woods, and I do get slightly discombobulated seeing his vignettes containing Newcastle cityscapes and hearing a promo in a rather broad Geordie accent. But I'm glad he's back from injury and ready to go. He's used very sparingly in AEW but is at that championship level in terms of performance. I'm hoping for a solid and consistent run in the future.</p>
<h2>Kris Statlander and Willow Nightingale vs Julia Hart and Skye Blue (Pre-show)</h2>
<p>I'm fully on board with the idea that AEW needs a Women's Tag division sooner rather than later. Former TBS champion Kris Statlander and Willow Nightingale face off against current TBS champion Julia Hart, and Skye Blue. Another serviceable pre-show match that moves the storyline forward, down the road Willow Nightingale challenging Julia Hart for her TBS title should happen. It was the correct result that Skye Blue took the pin from Nightingale.</p>
<h2>Christian Cage vs Daniel Garcia</h2>
<p>Christian Cage, Father of The Patriarchy faction, defends his TNT Championship against Daniel Garcia. Garcia had a poor run of form in the Continental Classic tournament at the end of 2023, since which he's been on a hot streak. This was a well-deserved championship shot, but I was certain it wasn't the right time for a change. Christian's gimmick is still massively over, and ultimately, his reign ought to culminate with a rematch against Adam Copeland. Christian's gimmick has kind of bantered its way into existence, but he's leant into it and does a very fine job. Throughout the match, there's the usual heel stuff of faking injuries, distractions on the outside, and so on. This match is about elevating Garcia and does just that.</p>
<h2>All Star Scramble</h2>
<p>Originally booked as a "meat madness" match, but due to injury, this was suddenly re-booked into an "All-Star Scramble" match. It was ... fine. It did the job. A bit of fun where everyone gets their spots in. Given the killer promos that Wardlow had been delivering of recent, and with a World Championship title shot on the line for the winner, I expected a Wardlow win.</p>
<h2>Orange Cassidy vs Roderick Strong</h2>
<p>It was the right time for Orange Cassidy to drop the title, and I think dropping it to Roderick Strong was also the right thing. It was a very well put together match, in terms of Roddy capitalizing on the accumulation of injuries from Cassidy's second reign. I wasn't too keen on the idea of Orange winning the title back so soon after his first reign, but that said the title never felt unimportant. Strong is a veteran and a very fluid performer. After a rather goofy run with the 'Neck Strong' gimmick last year, winning the workhorse title is going to give him a fantastic opportunity to demonstrate on a regular basis just how good he is.</p>
<p>Following the match, Kyle O'Reilly returned after two years out due to an injury. It's nice to see him back, as I felt his run didn't really get started.</p>
<h2>Eddie Kingston vs Bryan Danielson</h2>
<p>This was a very retro-feel match with some spectacular nearfalls. It gave New Japan vibes, which is entirely to be expected - a mix of strong style and technical wrestling. After a poor run at the beginning of the Continental Classic tournament, Eddie Kingston turned things round and defeated all members of the Blackpool Combat Club to win the whole thing. But he didn't win the respect of Danielson. This match is about that, stipulating that if Kingston wins, they must shake hands. Danielson does all his heel-technical stuff - the joint manipulation, the snapping back of limbs, submission holds. The crowd gets behind Kingston as he builds to win with a powerbomb finish. Great stuff.</p>
<h2>Blackpool Combat Club vs FTR</h2>
<p>This is the only match that I didn't predict correctly going in, mainly because I don't bet against FTR in a tag team match! The BCC pulling off a shock victory against one of the greatest tag teams ever is a bold move. The match started slow, but built up to an intense finish, which is fantastic storytelling. With these two brawler teams, you're looking for a good fight, and this was delivered in droves. Of course, there was blood, quite a lot of it from Dax Harwood. Claudio Castagnoli's swing leading into a dropkick from Moxley is a brilliant double-team move. Castagnoli showed good ring sense to counter FTR's finisher, the Shatter Machine. A dual submission hold finish, with Moxley putting Harwood to sleep, cinched the victory for the BCC. Very enjoyable.</p>
<h2>'Timeless' Toni Storm vs Deonna Purazzo</h2>
<p>I love the 'timeless' gimmick, it is simply <em>the best</em> thing in wrestling right now. However, this match wasn't particularly interesting.</p>
<p>Purazzo is new to AEW, but carries with her a reputation for strong technical wrestling, and a personal history with Toni Storm. One of the weird things about wrestling is how they weave their stories through real life and fantasy - this is the case with the former friends turned enemies. It's too soon for a wrestler new to company to come in and win a major championship, so this bout is about maintaining the challenger as a credible threat and setting up a rematch down the line, which this match just about manages to do. Purazzo makes Storm submit, albeit with the referee's back turned due to outside interference, before ultimately losing the match. Classic heel behaviour, an OK title defense.</p>
<h2>Will Ospreay vs Konosuke Takeshita</h2>
<p>GOOD GRIEF this was good. I remember watching Will Ospreay wrestle in the Northumbria University Sports Hall several years ago and it's very pleasing that him and Takeshita put on a match of the year contender. Ospreay and Takeshita are both in the same faction - The Don Callis Family. The story being told is that these guys are so good, they've only got themselves for a challenge. That said, there was a little tension before the matchup; you knew they wouldn't hold back. Both the performance AND the pace of the performance are worth commending here - this was 100% bell-to-bell. There were no sells, kicking out at one, a brutal turnbuckle brainbuster from Takeshita (catching Ospreay on the back), an <em>incredibly</em> fluid reversal into a Blue Thunder Bomb by Takeshita, and a Tiger Driver '91 from Ospreay. You need extreme precision to pull that off. Like Swerve/Hangman, this will be one of these long-running rivalries that define this era of AEW, if they have any sense!</p>
<h2>Samoa Joe vs Hangman Adam Page vs Swerve Strickland</h2>
<p>The Hangman / Swerve rivalry spills over into the Main championship. With the 'reset', the rankings system returned into AEW, so wins and losses matter in terms of who's in the running for the title picture. Naturally, this is a storytelling device, and going in to this Hangman and Swerve had an equal win/loss record, so the decision was made to turn this into a Three-Way match. Hangman, having lost the Texas Deathmatch against Swerve last year, did not take this news well. His fall to a heel is complete in this match, assaulting referees and doing anything possible to stop Swerve winning this match. Everyone comes off well in this match, Samoa Joe deservedly retained (I do think he'll drop to Swerve <em>eventually</em>), but both Hangman and Swerve had opportunities to clearly score victories. They didn't take them, instead being consumed by their own rivalry. Some fantastic storytelling in this match.</p>
<h2>Sting and Darby Allin vs The Young Bucks</h2>
<p>There was a beautiful pre-roll showcasing Sting's career (well, none of the WCW footage because The Fed owns it now) which was genuinely very emotional. On stage, Sting's sons adorned the costumes and makeup of his earlier gimmicks (surfer Sting and Wolfpac Sting), before Sting himself arrived. They walked to the ring together, beginning a fast and intense Tornado Tag match. Oh, they didn't hold back!</p>
<p>Sting and Darby started on the upper hand, rapidly beating down The Young Bucks, who play these heel roles so well. Darby Allin's senton bomb from the 10ft+ ladder onto a plane of glass supported by steel chairs is brutal. Uncomfortable to watch. Surprising to see on mainstream professional wrestling. I'm not a huge mark for the deathmatch stuff, especially the levels of goriness on the indies. In this match, though, it worked. Darby Allin put his body on the line in a big way to make this match special, later requiring twelve stitches for a pretty gnarly wound.</p>
<p>That's when things turned. With Darby essentially out of the match, the Bucks began to punish Sting. He was put through tables, from ladders. He was put through panes of glass. He's a few weeks shy of 65 years old. Just imagine for a moment jumping off tall stuff and being put through tables in your mid '60s. And don't give me any of that "wrestling's fake" stuff. But of course, it is pre-scripted, and although the convention is to go out on your back, Sting would not stay down. Despite everything thrown at him, he got up, kept fighting, and eventually put away the Bucks. What a match. What a career! Presumably, the Tag Team titles will now be vacated?</p>
<h2>Closing Thoughts</h2>
<p>This PPV wasn't just a goodbye and a celebration of an iconic career. It was also a statement of what's to come. Tony Khan has put together one of the best wrestling rosters of all time, and their capabilities and talents are not being wasted. The secret is basically out about who's coming to the company next, one of the biggest New Japan Pro Wrestling names of all time, and one of the most successful female stars from the modern WWE era. Five years in, the Revolution is still on. Jump onboard for the ride!</p>
February '24 Monthly Review2024-03-01T00:00:00Zhttps://samhogy.co.uk/2024/03/february-24-monthly-review/All change!<p>This post is a continuation of my series of monthly review posts - what I've been doing and all that kind of good stuff. Other entries in the series:</p>
<ul>
<li><a href="/2024/02/january-24-monthly-review/">January 2024</a></li>
</ul>
<h2>Work</h2>
<p>We're wrapping up our sixth Programme Increment since the transition to working in SAFe. The Home Insurance web team that I lead completed the migration work for the remaining two of our price comparison partners. Now all Home Insurance customers are served with our new web journey, completing a fairly significant investment in modernising that application journey. Similar work on our Motor Insurance journey continues - although we're using the rewritten application as a platform to launch 'tiered' products. These are designed to closely align with customer needs that we have identified. Given the ongoing cost of living crisis and various factors such as supply-chain issues affecting the cost of insurance, tiered products let us better align to market demands. At the end of last year we launched the tiered products for the direct route into the Car Insurance journey and we've spent this month getting ready to integrate this with our first price comparison commercial partner.</p>
<p>One of the things that I really enjoy about where I work is that we believe in each other. As a line manager I advocate strongly for those "in my chain", I take pride in assisting with career development. This goes all the way to the top too. Recently, a Principal Engineering role came up within the Platforms side of the business - they're responsible for looking after our cloud platforms, data, VMWare and so on. My line manager strongly recommended that I apply. I'd admit to being a little hesitant, with it being what I described as a 'diagonal' move. Not just upwards, but also into a different discipline. Thinking things through a bit more and reflecting on my time within Insurance, I realised the vast majority of the value I brought was in cloud expertise. So much of my time has been spent immersed in the infrastructure, the pipelines, the operability aspects of what we build. It seemed less daunting. So I bet on myself and applied. Suffice to say, it went well and I was offered the position! We're still working through the specifics of start dates and handover.</p>
<p>UK News readers may be aware that this month <a href="https://www.bbc.co.uk/news/business-68249167">Tesco Bank announced a sale of its Banking arm to Barclays, subject to regulatory approval</a>. Internally, we did not know about this until it was announced to the marketplace - and so this added an extra dimension of complexity to weighing up the Principal role. However, it also means that I'm going to have 'a seat at the table' for how we technically separate the Insurance and Banking sections of the business. Possibly the most disruptive time to announce my new role to the team, but also probably the best timing in terms of the opportunities that lie ahead!</p>
<h2>Side Projects</h2>
<p>I've been getting hands-on with Serverless programming, building a predictions application for my friends and I. Whenever All Elite Wrestling runs a Pay Per View event, we usually exchange messages with our predictions for the match results. You can see similar Pick'Em apps for the <a href="https://picks.nba.com/pickem">NBA</a> and <a href="https://nflpickem.co.uk/">NFL</a> respectively.</p>
<p>I'll write about this in more detail in the near future, but this short project has really been eye-opening. Learning how to design using the "Single Table Design" approach in AWS DynamoDB, how user management works in AWS Cognito, how to write and deploy AWS Lambda functions (including the CloudFormation stuff), and then composing all this together using either the Serverless framework and/or AWS Amplify. There's a lot I need to unpack!</p>
<h2>Books</h2>
<p>Last month was pretty fiction heavy, so February switches it up to (mostly) non-fiction:</p>
<h3><strong>How to Decide</strong> by <em>Annie Duke</em></h3>
<img src="https://samhogy.co.uk/assets/books/duke-howtodecide.jpg" />
<p>This book came in extremely helpful, for the reasons outlined above! It's a great book from a former professional poker player about decision-making. How can we make good decisions? What gets in the way and clouds our judgement? Some choice highlights include:</p>
<blockquote>
<p>Any decision is, in essence, a prediction about the future</p>
</blockquote>
<blockquote>
<p>Your decisions will only be as good as your ability to anticipate how they might turn out.</p>
</blockquote>
<blockquote>
<p>For any options you're considering, ask yourself, "if this were the only option I had, would I be happy with it?"</p>
</blockquote>
<p>As humans, we're prone to <em>resulting</em> - the idea that we evaluate a decision based on its outcome. This is a form of hindsight bias. We can make bad decisions that have good outcomes (unexpectedly), or vice versa. But this actually impairs our future decision-making by cutting out the ability to retrospect on our past decision-making. Good decisions go beyond simplistic pros and cons lists - it's about exploring the possible outcomes, evaluating the likelihood, potential upsides and downsides of those outcomes, and <em>then</em> deciding. There's a lot to learn from books like this.</p>
<h3><strong>Quit</strong> by <em>Annie Duke</em></h3>
<img src="https://samhogy.co.uk/assets/books/duke-quit.jpg" />
<p>The other side of the decisioning aspect is, once you've made a decision, when is it prudent to back out of it? Culturally we often seeing quitting as a 'bad' thing - persistence and knuckling down and grinning and bearing it are all considered virtues. Often, they can be actively harmful. The key is to understand when to quit and particularly, how you should approach this decision. I read <em>Quit</em> back-to-back with <em>How to Decide</em> and highly recommend that you do the same.</p>
<h3><strong>Tidy First?: A Personal Exercise in Empircal Software Design</strong> by <em>Kent Beck</em></h3>
<img src="https://samhogy.co.uk/assets/books/beck-tidyfirst.jpg" />
<p>Tidy First? is such a curious book. It's lightweight in terms of page count but not content. Going in, I expected this book to be about tidyings, and it <em>kind of</em> is. There's the familiar "keep the kitchen clean as we cook" adage to keeping code clean, but what I hadn't initially expected is how much the question mark in the title comes into play. This book is more about the <strong>economics</strong> behind tidying - there's the cost of change aspect: code that is difficult to understand is difficult to change, and therefore costlier. Going back to the decisioning around options from Annie Duke's books, Kent Beck applies this to software:</p>
<blockquote>
<p>Design we do today is the premium we pay for the option of buying the behavior change tomorrow.</p>
</blockquote>
<p>I think I'll need another read-through to fully digest the contents.</p>
<h3><strong>Actionable Agile Metrics for Predictability (Vols 1 & 2)</strong> by <em>Dan Vacanti</em></h3>
<img src="https://samhogy.co.uk/assets/books/vacanti-aamfp.jpg" />
<p>These books are a fantastic resource for those who want to get deeper into flow metrics, the relationship between those metrics, how to visualize data and how to interpret the visualizations for process improvement. The first of the volume culminates in the Cumulative Flow Diagram - which shows you the number of items arriving and leaving a process over time. Many tools will further break this up into different item statuses. As a cumulative graph, from left to right the data trends upwards so in order to interpret its meaning, you're looking at the <em>rate of change</em> to tell you whether items are arriving faster than they're departing, or whether items are becoming stuck 'in progress'. Ultimately, the books focus on making uncertain processes <strong>predictable</strong>, which can be done through probabalistic analysis of data. The Monte Carlo Simulation is a good example of this - taking a sample of throughput data (i.e "this item took 3 days") it is possible to run many simulations of your process and arrive at a probablistic forecast. It won't guarantee that an item will be done in X days, or that in Y days a team will achieve Z items - but it will help with expressing <strong>confidence</strong> in those forecasts. Volume two extends further into the nature of <strong>variability</strong> in processes. There's always variability, the challenge is identifying what is and isn't within your control. Process Behaviour Charts such as an xMR chart helps to visualise the variability, separating the signal (exceptional variability) from the noise (routine variability) so that you can make sure improvements are targeted where they'll have impact. There aren't entry-level books, for that I'd recommend the companion piece <em>When Will It Be Done?</em>.</p>
<h3><strong>The Sad Ghost Club (Vols 1-3)</strong> by <em>Lize Meddings</em></h3>
<img src="https://samhogy.co.uk/assets/books/meddings-sadghostclub.jpg" />
<p>Currently a three-volume graphic novel in which characters wearing ghost sheets (a visual metaphor for the isolation of anxiety and depression) meet their 'kindred spirits', forming the eponymous Sad Ghost Club, to support each other in challenging times. Very quick reads, but also quite profound.</p>
<h3><strong>Godkiller</strong> by <em>Hannah Kaner</em></h3>
<img src="https://samhogy.co.uk/assets/books/kaner-godkiller.jpg" />
<p>This is a brilliant fantasy novel that hooked me from the start with its fantastic worldbuilding, strong character development and an enticing final sequence setting up its forthcoming sequel <em>Sunkiller</em>. A girl survives a sacrifice to a fire God and grows up to become a Godkiller by trade, in a land where worship is outlawed. She meets an orphaned noble who has, somehow, became intertwined with a god of white lies. As they meet up with a former-knight-turned-baker, they embark on a quest to a ruined city, pursued by demons. The middle third dragged for me, but overall this is a strong debut novel.</p>
<h2>Games</h2>
<img src="https://samhogy.co.uk/assets/games/balatro.jpg" />
<p><a href="https://www.playbalatro.com/">Balatro</a> on my SteamDeck has been like catnip - it's a poker-based deckbuilding roguelite. From what I can gather, it was built by one developer and has made a huge amount of money since its launch. The core idea of the game is that you play your best poker hand from a set of available cards, trying to reach a target score. You win money, you buy upgrades to your deck, such as Joker cards that modify scoring, Tarot cards that change suits and so on. Boss battles impose further conditions - such as you can only play one <em>type</em> of hand. It's such a fun, lightweight game with a ridiculous amount of depth to it.</p>
January '24 Monthly Review2024-02-01T00:00:00Zhttps://samhogy.co.uk/2024/02/january-24-monthly-review/Wot I've gone and dun!<p>This post is the first in a new quest to 'work with the garage door up' - a monthly roundup of what I have been up to.</p>
<h2>Work</h2>
<p>For the last year, the teams that I lead have been re-writing Tesco Bank's Home and Car insurance online application journeys. We launched the new journeys last year for direct applications - i.e from our website. This month, we finalised the integration between the Home insurance application and three of our price comparison commercial partners. There's a large amount of cloud optimization we can take off the back of this work, which means we'll be running a leaner, greener and more user-friendly product.</p>
<p>A significant piece of development that I got to do in January was around our continuous delivery strategy. My applications run on auto-scaling EC2 instances which run in a blue/green setup utilising weightings on a load balancer. An S3 bucket serves the static content for the applications, which didn't operate under a blue/green setup. I developed a process to estalish which colour stack is the current online leg, deploys the static content to the opposite colour and ensures the EC2 instances load from the correct bucket.</p>
<p>The engineering lead role gives me opportunity to deep dive into unfamiliar territory. We use DataDog for monitoring and alerting, which integrates via a webhook to ServiceNow. Our build tooling provisions a bunch of system-level monitors in Datadog for all the infrastructure it Terraforms. We'd noticed that our load balancer latency and error monitors were going haywire, seemingly sporadically. They always self-recovered and other monitors indicated there wasn't a problem - the applications were up, serving traffic and not under heavy load. What was the problem? I did some digging and found out the issue pertained to sampling and averaging data in a small rolling window - which is notoriously prone to spikes in activity. At certain points in the day a single delayed request was enough to trigger these monitors. Once I identified the fix it became a matter of finding the sweet spot - where anomalous data alone wouldn't trigger, but sustained failures and network delays would.</p>
<h2>Books</h2>
<p>I had a pretty productive month for reading in January. Here are some of the standout books:</p>
<h3><strong>Before the Big Bang: The Origin of the Universe and What Lies Beyond</strong> by <em>Laura Mersini-Houghton</em></h3>
<img src="https://samhogy.co.uk/assets/books/mersini-houghton-bigbang.jpg" />
<p>Part autobiography and part summary of a life's work into multiversal theory, including string theory and the cosmic microwave background. The author grew up in Communist Albania and covers her journey into academia and how this perspective led to her contributions in cosmology. There's little about 'before' the Big Bang - perhaps a quirk of the publishers?</p>
<h3><strong>Tomorrow, and Tomorrow, and Tomorrow</strong> by <em>KGabrielle Zevin</em></h3>
<img src="https://samhogy.co.uk/assets/books/zevin-tomorrow.jpg" />
<p>This a fantastic work of fiction. Two brilliant but highly-flawed individuals lives revolve around each other. They develop a video game together, build a successful video game company off the back of it, and become intertwined with their mutual friend Marx (who is by far the most interesting character). The story is emotional. It doesn't pull its punches, and gets dark in places. The documentary-like style gives the morally-gray characters the space they need. There's timeline jumping and in and out-of-game segments - I found the real world stuff being much more engaging.</p>
<h3><strong>The Starless Sea</strong> by <em>Erin Morgenstern</em></h3>
<img src="https://samhogy.co.uk/assets/books/morgenstern-starlesssea.jpg" />
<p>This book did not resonate with me. I loved her previous work <em>The Night Circus</em> (despite not sticking the ending) and had high hopes for the sophomore novel. The mysteries are preserved through obfuscation and amount to nothing. The prose has an almost lyrical style, highly fantastical. Lots of description of pretty much nothing, with an ending that once again, just fizzled out.</p>
<h3><strong>There is Nothing for You Here: Opportunity in an Age of Decline</strong> by <em>Fiona Hill</em></h3>
<img src="https://samhogy.co.uk/assets/books/hill-nothing.jpg" />
<p>This hits home, almost literally. It documents the author's childhood in North East England. Despite the challenges in place by her background, her sex and her location, the book covers getting a place at the University of St Andrew's, working in the USSR on an exchange program, moving to the US and then to the White House in the National Security Council during the Trump Administration. Fiona Hill would make an excellent Labour politician, she is incredibly passionate on the politics of education, aspiration and opportunity.</p>
It Depends Is About Tradeoffs2023-12-12T00:00:00Zhttps://samhogy.co.uk/2023/12/it-depends-is-about-tradeoffs/Dig a little deeper.!<p>The joke is that the consultant's answer to any question is "it depends". An even funnier joke - it actually does!</p>
<p>"It depends" is not a get-out clause; it's an opportunity to surface and understand the trade-offs underpinning a decision. As much as we may reap the benefits of happy accidents, nobody likes to suffer unintended consequences.</p>
<p>Having an understanding of the critical trade-offs at play helps us to make better decisions.</p>
<p>At the coding level, common trade-offs focus on concerns such as resource usage (speed, memory and complexity) and integrity (duplication, consistency, naming). Higher up, we perform "build versus buy" decisions - should we accept a dependency (in the form of a framework or a library) or go bespoke and roll our own implementation? Further, trade-offs inform commercial decisions. For example, launching a new feature before the end of the financial year will have a predicted return. Deferring that feature may mean missing the window of opportunity - a competitor could beat us, or market desires may move elsewhere. Trade this off against what's needed to make this happen and the impact elsewhere. There may be an old product ready for decommissioning, but the longer it stays live, the more maintenance costs it will incur and the higher the chance our team may spend rectifying incidents.</p>
<p>There isn't a correct decision on what to do in these situations. Our role is to make the best decision we can with the information we have. It depends!</p>
Fun With Javascript Map / Reduce2023-12-04T00:00:00Zhttps://samhogy.co.uk/2023/12/fun-with-javascript-map-reduce/FUNctional!<p>I'm a huge fan of functional programming. The JavaScript language treats functions as first-class-citizens (meaning, they're <a href="https://developer.mozilla.org/en-US/docs/Glossary/First-class_Function">just another variable</a>) and so there are ample opportunities for coding in a functional style.</p>
<p>Some of my favourite functions are on the <code>Array</code> prototype: <code>filter</code>, <code>map</code>, <code>reduce</code>. I like that these are chainable, meaning that it's possible to create a <strong>pipeline</strong> of functions to execute for a dataset. The more formal name for this pattern is <a href="https://en.wikipedia.org/wiki/Fluent_interface">Fluent Interface</a>. In this post, I cover these functions and provide some examples of how to use them.</p>
<h2>Digging Into the Functions</h2>
<h3><code>filter</code> Copies All Array Entries That Pass a Test</h3>
<p>Use <code>filter</code> when you need some elements in an array based on whether a condition evaluates to true.</p>
<p>Filtering a dataset before transforming the relevant data minimises the amount of operations performed. This is of particular importance for large datasets or for large chains.</p>
<p>The C# version of this function is more readable to me (it's <code>.Where</code>) as the filter operation <strong>retains</strong> the items that satisy the test. It filters <strong>out</strong> the items that don't!</p>
<p>A filter statement requires a callback function that returns a boolean - <code>true</code> to keep the element in the result, <code>false</code> if not.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">6</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">8</span><span class="token punctuation">,</span> <span class="token number">9</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> x <span class="token operator"><</span> <span class="token number">5</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token comment">// returns [1, 2, 3, 4]</span></span></code></pre>
<h3><code>map</code> Transforms Each Value in the Array</h3>
<p>Use <code>map</code> to create a new array of transformed elements.</p>
<p>I use mapper functions often when handling data from an API call, typically to transform a collection of data into <a href="https://en.wikipedia.org/wiki/Domain_model">domain models</a> that I control. I frequently find myself then mapping from domain model objects into another format specific to a UI component.</p>
<p>As with <code>filter</code>, <code>map</code> accepts a callback function, executed for each element in the array, and whose return value becomes an element in the resultant array.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">6</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">8</span><span class="token punctuation">,</span> <span class="token number">9</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> x<span class="token operator">*</span>x<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token comment">// returns [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]</span></span></code></pre>
<h3><code>reduce</code> Consolidates All the Array Values to a Single Value</h3>
<p>Use <code>reduce</code> for when the array elements need combined into a single value. The reduce function has a slightly different signature to <code>map</code> and <code>filter</code>, because it works slightly differently.</p>
<p>It's best to start back-to-front with the call signature.
Argument two is the <strong>initial value</strong> for the result. It could be a <code>0</code> if the reducer is summing all the array elements, or an empty object <code>{}</code> if the result is an object.</p>
<p>The callback function has two arguments - the <strong>current</strong> value of the accumulator and the <strong>current</strong> array element. Its return value becomes the value passed as the accumulator argumnet to the next callback.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">6</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">8</span><span class="token punctuation">,</span> <span class="token number">9</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">acc<span class="token punctuation">,</span> x</span><span class="token punctuation">)</span> <span class="token operator">=></span> acc <span class="token operator">+</span> x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token comment">// returns 55, whereas</span></span>
<span class="highlight-line"><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">6</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">8</span><span class="token punctuation">,</span> <span class="token number">9</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">acc<span class="token punctuation">,</span> x</span><span class="token punctuation">)</span> <span class="token operator">=></span> acc <span class="token operator">+</span> x<span class="token punctuation">,</span> <span class="token number">100</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token comment">// returns 155, and</span></span>
<span class="highlight-line"><span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token string">"eggs"</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token string">"ham"</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token string">"cheese"</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">acc<span class="token punctuation">,</span> x</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> acc<span class="token punctuation">[</span>x<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token operator">=</span> x<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> acc</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token comment">// returns { "eggs": 1, "ham": 2, "cheese": 3 }</span></span></code></pre>
<p>The <code>reduceRight</code> variant applies from right-to-left.</p>
<h2>Putting It All Together</h2>
<p>Let's put this learning into practice with some real-world code.</p>
<p>For a recent side project I needed to query the JIRA API to list all the (completed and active) sprints for a project. The JIRA API returns a lot of information, not all are actually needed in my application. I used the <code>filter</code> and <code>map</code> functions to exclude any future sprints and then transform the returned objects to include the properties I need, in the format I want:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> data <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">getFromJiraAPI</span><span class="token punctuation">(</span>boardId<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">return</span> data</span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">item</span><span class="token punctuation">)</span> <span class="token operator">=></span> item<span class="token punctuation">.</span>state <span class="token operator">!==</span> future<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">item</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>item<span class="token punctuation">.</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> item<span class="token punctuation">.</span>name<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">state</span><span class="token operator">:</span> item<span class="token punctuation">.</span>state<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">goal</span><span class="token operator">:</span> item<span class="token punctuation">.</span>goal<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">started</span><span class="token operator">:</span> <span class="token function">toMs</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>activatedDate<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">finished</span><span class="token operator">:</span> <span class="token function">toMs</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>completeDate <span class="token operator">||</span> item<span class="token punctuation">.</span>endDate<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>In another part of the code, I have an array of objects containing timing data and a need to produce a sum total of all timings. This calls for a <code>reduce</code>.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">return</span> flowTimes<span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token punctuation">(</span><span class="token parameter">acc<span class="token punctuation">,</span> entry</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> acc<span class="token punctuation">.</span>leadTime <span class="token operator">+=</span> entry<span class="token punctuation">.</span>leadTime<span class="token punctuation">;</span></span>
<span class="highlight-line"> acc<span class="token punctuation">.</span>cycleTime <span class="token operator">+=</span> entry<span class="token punctuation">.</span>cycleTime<span class="token punctuation">;</span></span>
<span class="highlight-line"> acc<span class="token punctuation">.</span>activeTime <span class="token operator">+=</span> entry<span class="token punctuation">.</span>activeTime<span class="token punctuation">;</span></span>
<span class="highlight-line"> acc<span class="token punctuation">.</span>idleTime <span class="token operator">+=</span> entry<span class="token punctuation">.</span>idleTime<span class="token punctuation">;</span></span>
<span class="highlight-line"> acc<span class="token punctuation">.</span>blockedTime <span class="token operator">+=</span> entry<span class="token punctuation">.</span>blockedTime<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> acc<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">leadTime</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">cycleTime</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">activeTime</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">idleTime</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token literal-property property">blockedTime</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>I wanted to visualise the proportion of stories, versus defects and technical debt tickets in a sprint, as a stacked bar chart. Given an object listing each ticket type and the number of that type:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> ticketTypesCount <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token string-property property">"Story"</span><span class="token operator">:</span> <span class="token number">10</span><span class="token punctuation">,</span> <span class="token string-property property">"Defect"</span><span class="token operator">:</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token string-property property">"TechDebt"</span><span class="token operator">:</span> <span class="token number">2</span> <span class="token punctuation">}</span></span></code></pre>
<p>I can (generically) calculate the sum of all the tickets by:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> total <span class="token operator">=</span> Object<span class="token punctuation">.</span><span class="token function">keys</span><span class="token punctuation">(</span>ticketTypesCount<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">acc<span class="token punctuation">,</span> k</span><span class="token punctuation">)</span> <span class="token operator">=></span> acc <span class="token operator">+</span> ticketTypesCount<span class="token punctuation">[</span>k<span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p><code>Object.entries</code> is a useful function for producing an array containing each property in an object. On the above, this will output <code>["Story", "Defect", "TechDebt"]</code>; Each callback of <code>reduce</code> provides an individual index to a property of the <code>ticketTypesCount</code> object.</p>
<p>Dynamic access to an object's property uses the square-bracket syntax, just like accessing an array element at an index. When <code>k="Story"</code>, <code>ticketTypesCount[k]</code> is functionally-equivalent to <code>ticketTypesCount.Story</code>, with the exception that the property name did not need to be hard-coded!</p>
<p>To 'stack' the data, transform each data point to the <em>proportion of the total</em> it accounts for. This requires a <code>map</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> band <span class="token operator">=</span> Object<span class="token punctuation">.</span><span class="token function">keys</span><span class="token punctuation">(</span>ticketTypesCount<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">type</span><span class="token operator">:</span> k<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">value</span><span class="token operator">:</span> ticketTypesCount<span class="token punctuation">[</span>k<span class="token punctuation">]</span> <span class="token operator">/</span> total</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Outputting</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">"Story"</span><span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> <span class="token number">0.625</span> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">"Defect"</span><span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> <span class="token number">0.25</span> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">"TechDebt"</span><span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> <span class="token number">0.125</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">]</span></span></code></pre>
<p>These three functions, <code>map</code>, <code>filter</code>, <code>reduce</code>, alongside some supporting functions such as <code>Object.keys</code> are often sufficient for the vast majority of data manipulation needs. I also find the code cleaner than the alternative, which is to do all the operations over the array in one large <code>for</code> statement, as there is less state to maintain overall. Each operation remains nicely contained to its own callback function in the pipeline.</p>
<p>Why not give this style a go? See if it makes a difference!</p>
Why So Negative?2023-09-20T00:00:00Zhttps://samhogy.co.uk/2023/09/why-so-negative/Does Agile have an image problem?!<p>Why is so much advice in the Agile space framed from a negative perspective?</p>
<p>It's all "don't do this" and "stop doing that".</p>
<p>I'm sure I've done it myself - mea culpa.</p>
<p>Maybe it's today's Internet climate, the need to put over strongly-opinionated #content to get #engagement. Should you ever get the chance to speak to an Internet firebrand for realsies (pot meet kettle), you'll often find their actual opinions are more reserved and nuanced - leaning towards the consultant's answer of "it depends". <em>Of course it does</em>!</p>
<p>Take the Agile Manifesto - it's a reflective and forward-looking document. Note the <em>"x over y"</em> language and how it recognises both sides have value, but one side is valued more.</p>
<p>I value the Agile Manifesto for how it communicates its message. I read it and become inspired.</p>
<p>As I reflect more on how leaders influence and shape the environment around them, I'm becoming more aware that "stop doing x, don't do y, do z instead" advice <strong>harms</strong> more than it helps. Why?</p>
<h2>Journey Before Destination</h2>
<p>Let's take a concrete example. A team have recently moved from estimating work in hours to using story points. In discussion, you provide advice to stop using story points and instead right-size stories to roughly a day, because "story points are trash". You may have lots of data to back up your point. But on the other hand, that team improved their process, which still should be celebrated, rather than attacked.</p>
<p>Agility advocates can be criticised for using the <a href="https://en.wikipedia.org/wiki/No_true_Scotsman">No True Scotsman fallacy</a> as an argument - what you're doing isn't right, it's not "Agile". Holding an impossible-to-meet standard against those trying their best, regardless of <em>where</em> they are in their journey.</p>
<p>What would it look like if, instead, you meet people where they are? Recognise their achievements and help them towards their next steps. This isn't to say your opinions are invalid. How you communicate those opinions matters.</p>
<p>Enquire about what led them to adopt story points. How do they feel it's going? What patterns are they noticing? What differences has it produced? Hmm, you looked uncomfortable talking about "measuring velocity" there - what worries you about that?</p>
<p>Sparking curiosity invites participation and provides growth opportunities. A 'guru' giving the answer and downplaying progress creates disengagement, which can lead to resentment.</p>
<p>Instead of telling people not to do something, have you tried listening to them?</p>
Plague2023-08-17T00:00:00Zhttps://samhogy.co.uk/2023/08/plague/Bleurgh!<p>Lord Nurgle has finally struck the samhogy household - we came back from holiday and brought some COVID home as a souvenir.</p>
<p>It's been a rough few days. No loss of smell or taste (thankfully), but nasty coughs, sore throats and snotty noses abound. Shivering from fever and muscle pain.</p>
<p>Hopefully we've broken the back of it.</p>
<p>This is the first time that we're sure we've had it, over the years there's been a few near misses and false starts. I suppose in the grand scheme of things, getting to 2023 is a decent streak. But it's also a reminder that this thing still isn't over.</p>
Re-reading Royce2023-07-22T00:00:00Zhttps://samhogy.co.uk/2023/07/re-reading-royce/Don't go chasing waterfalls!<p>Winston W. Royce's article, "<a href="https://www.praxisframework.org/files/royce1970.pdf">Managing the Development of Large Software Systems</a>", is conventionally and mistakenly perceived as the origin of the Waterfall method of Software Development. However, the actual contents of the paper couldn't be more different!</p>
<p>The term 'waterfall' is never mentioned. What is commonly understood to be the 'waterfall' approach - a linear sequencing of activities described in Figure 2 - is described as being "risky and invites failure".</p>
<img src="https://samhogy.co.uk/assets/royce-waterfall.jpg" alt="Royce's basic linear sequencing of software phases" />
<h2>Risk Accumulates</h2>
<p>Royce's core observation is that a linear approach accumulates risk throughout the process. Testing, the "phase of greatest risk", occurs at the end. Without mitigation measures, any issues can have a significant impact later in the process.</p>
<blockquote>
<p>"The required design changes are likely to be so disruptive that the software requirements upon which the design is based ... are violated."</p>
</blockquote>
<p>Royce's approach to mitigating this risk is reducing scope - by incorporating feedback loops throughout the process to "maximize the event of early work that is salvageable and preserved".</p>
<p>The first of these mitigation measures is a "preliminary design phase" in which program designers contribute to the system design before detailed analysis to "impose constraints". Royce intends for the non-functional requirements of a system to provide scope to the solution space, such that the process "will culminate in the proper allocation of execution time and storage resources".</p>
<p>Nowadays, I see similarities to the role of solution architects working early in a project's lifecycle to define a high-level solution design, incorporating factors such as operability, system communication and coherence with organisational standards and governance.</p>
<h2>Some of this Sounds A Bit Like XP</h2>
<p>Two mitigation measures, if re-written in more modern lingo, are familiar to those who practice Kent Beck's Extreme Programming (XP).</p>
<p>As part of the preliminary program design phase, Royce recommends to "Do It Twice" - essentially, run the project on a smaller scale and throw away the resulting solution, but incorporate the learnings in later phases.</p>
<p>Complexity emerges in software as our understanding of the problem grows. Deliberately exploring solutions through rapid prototyping makes it possible to get an early sense of the trouble spots. Through the prototype (Royce calls it a 'simulation'), one can "perform experimental tests of some key hypotheses".</p>
<p>Throwaway prototypes uncover risk in a controlled way and enable teams to experiment with potential solutions. In XP, these are called Spike Solutions and serve as a means to gather information and increase the reliability of estimates for the actual work.</p>
<p>Moving forward to the last point on customer involvement, the "On-Site" always-available customer is vital to XP. Involving the customer throughout the development of a system ensures that their feedback is incorporated early and often, reducing the risk of building the wrong solution. Often, programmers have created what clients asked for, only for them to be dissatisfied upon delivery. Royce acknowledges this:</p>
<blockquote>
<p>"For some reason what a software design is going to do is subject to wide interpretation even after previous agreement. It is important to involve the customer in a formal way so that he has committed himself at earlier points before final delivery."</p>
</blockquote>
<h2>Documentation Ain't All Bad</h2>
<p>Royce advocates for "quite a lot" of documentation, which may seem at odds with the Agile Manifesto valuing "working software over comprehensive documentation", but I'm willing to give Royce some leeway.</p>
<p>Documentation is essential in software development. Code cannot unambiguously describe everything. For instance, Royce suggests documenting decisions, which is eminently sensible. Architectural Decision Records are a modern-day approach which comes to mind.</p>
<p>Some of this section shows its age, particularly around the separation of development and operations. Royce highlights the unsuitability of developers managing their software in live environments:</p>
<blockquote>
<p>"Generally these people are relatively disinterested in operations and do not do as effective a job as operations-oriented personnel."</p>
</blockquote>
<p>Royce's operating context is that of large-scale software systems in which a 'Thing' is being built. Nowadays, evergreen software is the mainstream operational context, where there isn't necessarily a pre-defined 'Thing' but rather an ongoing, iterative and evolutionary approach to design. In this context, it does make more sense for developers and operations to be closely aligned.</p>
<p>Another point on documentation relates to system specifications:</p>
<blockquote>
<p>"During the early phase of software development the documentation is the specification and is the design. Until coding begins these three nouns (documentation, specification, design) denote a single thing".</p>
</blockquote>
<p>The relationship between those three nouns brings me to Royce's suggestions about testing.</p>
<h2>Back to Testing</h2>
<p>Royce understands testing is a continual activity but can't <em>quite</em> say this. He notes earlier stages are</p>
<blockquote>
<p>"aimed at uncovering and solving problems before entering the test phase".</p>
</blockquote>
<p>Testing, as a fundamental concept, is a means of <strong>clarifying assumptions</strong>, of gaining shared understanding through experimentation.</p>
<p>Documentation, in the form of rigid specifications, acts as a proxy for shared understanding and provides no guarantees that all consumers will arrive at that same understanding. Methodologies such as Behaviour-Driven Development provide space to arrive at a shared understanding and capture that through executable documentation.</p>
<blockquote>
<p>"During the early phase of software development the documentation is the specification and is the design. Until coding begins these three nouns (documentation, specification, design) denote a single thing".</p>
</blockquote>
<p>Documentation can serve as a focus. Consider Amazon's six-pagers which provide the top-level overview and raison d'être. I advocate for documentation, but not to the detriment of, or a replacement for, shared understanding.</p>
<p>Royce sees testing as a means of gaining feedback and therefore reducing risk. He advocates for human and technical approaches. Consider the parallels between:</p>
<blockquote>
<p>"Every bit of an analysis and every bit of code should be subjected to a simple visual scan by a second party"</p>
</blockquote>
<p>and modern-day peer review or pair programming. Likewise,</p>
<blockquote>
<p>"Test every logic path in the computer program at least once with some kind of numerical check"</p>
</blockquote>
<p>should be familiar to those automated writing programmer tests (although, note 100% code coverage alone is not an indicator of quality).</p>
<h2>More Agility Than Previously Advertised</h2>
<p>In many respects, this paper shows its age. Computing power has improved significantly in the succeeding half-century, making some previously-prohibitively-expensive activities trivial to perform on a typical developer machine.</p>
<p>The most radical change is the notion that a pre-defined 'Thing' is being built. Instead, we have initial expectations and assumptions about a solution that teams will successively refine based on real-life learnings.</p>
<p>To some extent, I feel guilty about judging an old paper by today's standards - the discipline has evolved. It strikes me that a lot here still applies to so-called "Agile" teams today, despite the paper's age and a conventional misunderstanding of its primary point.</p>
<p>But let's bury the notion that Royce designed and advocated for a 'waterfall' approach.</p>
Does Scrum Enable Sustainable Development?2023-07-11T00:00:00Zhttps://samhogy.co.uk/2023/07/does-scrum-enable-sustainable-development/You can run a sprint or you can run a marathon, but you can't sprint a marathon - Ryan Holmes!<p>Whenever I think about developing software at a sustainable pace, my right foot flares with pain.</p>
<p>In 2014 I picked up an injury by running too many miles and not sufficiently resting following exercise. Rest periods are when the body heals and adapts. Too much activity without rest leads to injury.</p>
<p>Working at a sustainable pace is a key principle of the <a href="https://agilemanifesto.org/">Agile Manifesto</a>:</p>
<blockquote>
<p>"Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely."</p>
</blockquote>
<p><strong>Indefinitely</strong> is the operative word. "Crunch" periods of late-night feature development and bug-fixing close to a fixed release date aren't sustainable. People will burn themselves out from the stress, and then the company loses their expertise when they leave. Code that is difficult to reason about, modify and verify becomes increasingly expensive to change. Friction slows down the flow of value to the customer and revenue to the business. Sustainable development is a crucial economic factor in the ongoing success of a product.</p>
<p>I've been thinking about how Scrum promotes working at a sustainable pace. The first thing that struck me was the terminology around time-boxed iterations, which Scrum calls "Sprints".</p>
<p>Physiologically speaking, sprinting isn't sustainable. High-intensity (anaerobic) activities cause a buildup of lactic acid and lead to muscle failure.</p>
<p>Given this, why was the term Sprint chosen? As per the <a href="https://scrumguides.org/">Scrum Guide</a>, a team sprints, and then it sprints again and again:</p>
<blockquote>
<p>"A new Sprint starts immediately after the conclusion of the previous Sprint."</p>
</blockquote>
<p>Jeff Sutherland, co-creator of Scrum, in his book <a href="https://www.scruminc.com/new-scrum-the-book/">"Scrum: The Art of Doing Twice the Work in Half the Time"</a>, explains that:</p>
<blockquote>
<p>"We called them that because the name evoked a quality of intensity. We were going to work all out for a short period of time and stop to see where we were."</p>
</blockquote>
<p>Sprints in Scrum contain high-intensity bursts of focus and periods of reflection. Although "intensity" and working "all out" do not appear in the Scrum Guide, the author's intent is clear enough.</p>
<h2>Reflection</h2>
<p>Scrum's periods of reflection are the Sprint Planning, Sprint Refinement and Sprint Retrospective events.</p>
<p>During Sprint Planning, the team collaboratively define the Sprint Goal - the thing to intensely focus on achieving during the next time box. The Developers then plan how they will work to accomplish this. Sprint Planning promotes sustainable development in two ways:</p>
<ol>
<li>By decomposing items into smaller chunks of work of a day or less in size, progress towards the goal can be meaningfully inspected and adapted during the Daily Scrum.</li>
<li>Developers have the discretion of defining 'how' the work is done, removing the ability for others to force work (and working conditions) upon the team.</li>
</ol>
<p>Whilst the Sprint Review focuses on inspecting and adapting the product for future Sprints, the Sprint Retrospective focuses on the process aspect - the "quality and effectiveness". Retrospectives offer a forum to raise concerns around sustainable development, for instance, whether quality has decreased (accidentally or deliberately),</p>
<p>The Scrum Framework has a reflect-do rhythm, but is that enough to enable working at a sustainable pace? I don't think so.</p>
<h2>Purposefully Incomplete</h2>
<p>Scrum describes itself as a "purposefully incomplete" framework of empirical process control, differentiating itself from prescriptive methodology by "only defining the parts required to implement Scrum theory".</p>
<p>Purposeful incompleteness is a blessing and a curse - the Scrum Framework alone cannot enable sustainable development. Constructive ambiguity is at play here. What's missing?</p>
<h2>Scrum Requires Strong Technical Foundations</h2>
<p>Technical excellence is missing from the Scrum Guide, but reading between the lines, it's assumed and expected.</p>
<p>Developers should be "Holding each other accountable as professionals" - '90s corporate speak for maintaining standards. Regarding quality, Scrum sets an expectation it doesn't decrease during an iteration, instilled through the Definition of Done and refined through the Retrospective.</p>
<p>On-the-ground technical practices, such as Test-Driven Development, refactoring, Continuous Integration, and Trunk-Based Development, aren't explicitly mentioned. Scrum has applicability beyond software and, as a framework, doesn't prescribe methods.</p>
<h2>Sustainable Development is on You</h2>
<p>Scrum cannot guarantee or ensure that a sustainable pace is maintained. It provides opportunities to reflect on the process and sets rough expectations on quality and professionalism.</p>
<p>I come back to the Sprint Planning section of the Scrum Guide:</p>
<blockquote>
<p>"How this (the work) is done is at the sole discretion of the Developers. No one else tells them how to turn Product Backlog items into Increments of value."</p>
</blockquote>
<p>What quality means and how the team does the work is entirely on them - that means it's for the team to work out how to work sustainably. If refactoring or maintenance is needed to support a feature, the Developers can establish it as part of their Sprint Planning. They don't need permission from a Product Owner, and it doesn't need prioritising by commercial.</p>
<p>If you, as I do, feel this is an unsatisfying conclusion, then reflect on why that is. There's the Scrum Guide, what it says and expects, and the contrasting experience of Scrum in practice - what Ron Jeffries calls <a href="https://ronjeffries.com/categories/dark-scrum/">"Dark Scrum"</a> and the issues described in Ryan Ripley and Todd Miller's book <a href="https://pragprog.com/titles/rrscrum/fixing-your-scrum/">"Fixing Your Scrum"</a>. As members of the software development industry, our challenge is to be active participants in product development.</p>
<p>We shouldn't be looking to Scrum to answer how we work at a sustainable pace, but ask how we can use the opportunities available within the Scrum Framework to enable sustainable ways of working.</p>
Attention Canaries2023-07-04T00:00:00Zhttps://samhogy.co.uk/2023/07/attention-canaries/Hey! Listen!!<p>Coal mining is a dangerous activity. Deep underground, it is difficult to detect the presence of toxic gases. Coal produces carbon monoxide when it burns. Methane gas, formed during coalification and emitted during mining, carries both suffocation and implosion risk.</p>
<p>At the turn of the 20th Century, John Scott Haldane suggested a novel (if horrific) safety measure - canaries. Physiologically they served as effective early warnings on air quality, as they'd get sick before the miners did. This allowed workers to fit their <a href="https://miningheritage.co.uk/self-rescuer/">self rescuers</a> and evacuate the area. The canaries were eventually replaced with <a href="https://www.bbc.co.uk/archive/nationwide--pit-animals/z62f2sg">electronic devices</a> in the late '80s, around the time when Maggie Thatcher's government was shutting the mines, decimating communities across the country.</p>
<p>Since then, "canary in the coal mine" has been a synonym for early-warning systems.</p>
<p>A warrant canary is a similar technique that warns users against circumstances where the service has been investigated by law enforcement authorities, and there is a legal restriction from reporting on this. A notice is publicised well before any investigations, for example, "The FBI has not been here". If this notice is ever removed, it's not categorically stating that the event has now happened. Still, it can be interpreted as being indicative. Users may then change their behaviour accordingly. Apple and Reddit have previously published and removed warrant canaries.</p>
<p>Recently in longer-form communications and in certain documents I really want someone to pay attention to, I've devised a similar technique that I've called an attention canary.</p>
<p>The idea is to carefully place a certain piece of information so that you can work out who's paying attention. The human attention span wanes between 10-20 minutes, so it's somewhere in this ballpark you're targeting. People will pick up on the odd thing, and their attention is jolted.</p>
<p>At a recent internal presentation, I added my birthday onto a slide detailing the timeline of a new product rollout. It's whimsical and entirely non-harmful. After the presentation, I received a few messages remarking on it! I can use this as a sense check on levels of audience engagement.</p>
<p>Alternatively, this could influence document design, particularly the placement of figures, charts, headings, bullets or images. Break up long blocks of text, and change the pace of the document. Grab back the user's attention by summarising the crucial details or preparing them for the next burst of focus. This pattern can also be used between logical sections of a presentation, chapters in a book, and so forth. The canary is a call to action to reorient, but it's also a jumping-back-in point.</p>
<p>As a parting exercise, consider the communication activities you participate in as a presenter or audience. What interests you? What bores you? What keeps you engaged? How could you restructure that complicated explanation to be more accessible?</p>
Do It As A Last Resort2023-07-03T00:00:00Zhttps://samhogy.co.uk/2023/07/do-it-as-a-last-resort/Drop, Defer, Delegate, Do - in that order!<p>There's a deluge of things coming your way through every channel available. Email. Phone. Instant Messenger. The blokey who pops to your desk and asks if you have a minute. Working from home hasn't made this any better. In fact, it's probably made it worse.</p>
<p>I'm talking about the everyday, reactive, side-of-desk stuff that comes in and disrupts our flow. There's a cost to this. Humans are consistently not good multi-taskers. Performance drops when there is a need to switch contexts. Interruptions add cognitive overhead that, when sustained, leads to stress and burnout.</p>
<p>Cal Newport calls this the <a href="https://www.nytimes.com/2021/03/05/podcasts/ezra-klein-podcast-cal-newport-transcript.html">"Hyperactive Hive Mind"</a> in his book <em>A World Without Email</em>. It's (unfortunately) rapidly becoming just the way we do things. Business as usual. So many group chats, so many emails, so many <a href="https://nohello.net/en/">"Hi Sam"</a> direct messages.</p>
<p><strong>How are we supposed to get <em>anything</em> done?!</strong></p>
<p>I have a simple mantra that I run through to work out how to deal with an incoming request:</p>
<ol>
<li>Does it have to be done?</li>
<li>Does it have to be done <em>now</em>?</li>
<li>Does it have to be done <em>by me</em>?</li>
<li>Do it.</li>
</ol>
<p>This system, the <em>Four D's of time management</em> - Delete / Defer / Delegate / Do - originates from <em>The Power of Focus</em>.</p>
<h2>Saying No</h2>
<p>There are only so many hours of the working day. Therefore, a decision to do something is explicitly also a decision <strong>not</strong> to do some other stuff. Being conscious of this helps you focus on what's truly the most valuable use of your time. The flip side is by saying no to the wrong things you can say yes to the right things.</p>
<p>Learning to say no, or at least <em>not now</em>, is an important time management skill to develop. As you progress to a leadership role this becomes even more important, as more folks will make demands of your time. Others in the organisation who aspire to your role will look to you to set an example. The way you do that is by showing, not telling.</p>
<p>After all, one of the core <a href="https://agilemanifesto.org/principles.html">principles of the Agile Manifesto</a> tells us</p>
<blockquote>
<p>Simplicity - the art of maximizing the amount of work not done - is essential.</p>
</blockquote>
<h2>The Urgency of the Recent</h2>
<p>Bob Carter famously wrote:</p>
<blockquote>
<p>A lack of planning on your part does not necessitate an emergency on mine</p>
</blockquote>
<p>Not all requests coming your way are last-minute oh-shit-I've-dropped-the-ball requests. Nor are they always indicative of poor planning (especially in an environment where work is <em>emergent</em>). Often they're line of business requests. It is relatively infrequently that stop-the-bus requests come through.</p>
<p>Think of the quote and the "does it have to be done now" step as a defence mechanism against <a href="https://en.wikipedia.org/wiki/Recency_bias">recency bias</a>. Just because it's new doesn't mean it needs to be done now.</p>
<h2>Growing Teams</h2>
<p>Delegation isn't about palming the work off on some unwilling schmuck. Rather, it's an opportunity to develop the team's cross-functional capabilities.</p>
<p>One perspective on the role of a manager is to "shield the team", acting as an umbrella and giving the team space to focus. While important to some degree, it places you in the unenviable and inescapable position of becoming a go-between. In the long run this hurts and hinders the team, reducing the <a href="https://en.wikipedia.org/wiki/Bus_factor">bus factor</a> to one.</p>
<p>So, make that work visible. Delegate it down to the team. They'll learn something along the way.</p>
<h2>Working Hard, Hardly Working?</h2>
<p>Give tasks you need to do your full, undivided attention. The 4 D's technique is one that carves out the space for focus, so turn off the chat and email clients. It's remarkable what can be achieved in a block of pure focus time. Don't worry, if something genuinely more important comes up you'll get a phone call. It is within this space that you close loops and get stuff done.</p>
<p>In today's always-on, always-busy world, focusing on one thing at a time may seem basic. You may worry others will think this is slacking. Put it this way - how can you reasonably expect to get <strong>more than one</strong> thing at a time done, if you can't reliably and repeatedly get one thing at a time done?</p>
State of the Sam 20222022-12-24T00:00:00Zhttps://samhogy.co.uk/2022/12/state-of-the-sam-2022/Once more round the sun!<p>Turns out last year that I forgot to write one of these. 2021 was a difficult year, put it that way. A job move didn't work out for me over the summer, but on reflection I absolutely made the right to decision to walk away when I did. Some would consider that a failure - I don't. It's only a failure if you don't learn from it. In the summer my second son was born and that really brought to the fore that I need to make the most of my time working, so that I could make the most of the time at home. If the two start bleeding together then bad things happen.</p>
<p>So anyway, I rejoined Tesco Bank. First I was part of a team releasing a product launch, which was great fun. That work was rewarded with a promotion to a Lead Engineer role in Insurance, which I started August 2022. Every day is different, and challenging in its own ways. My promotion coincided with the adoption of SAFe (first through a pilot, soon to be rolled out across the org) and with supporting the release of a new project. It's been... busy, but fun. I'm making a difference, there's a great cohort of developers that I'm supporting and in many ways we have made a huge amount of progress compared to where we were in 2020.</p>
<p>As a parent of (now) 4 and 1 year-olds, this hasn't been a year of huge creative output, and I won't lie that in many respects it's been a struggle. The first year of a child is <strong>brutal</strong> in terms of its relentless schedule and impact on sleep. But it's been a joy to see both boys grow, the eldest into a role of big brother. The youngest is just starting to talk now, soon we'll be in that lovely 18month period which is incredibly rewarding!</p>
<h2>Return to Speaking</h2>
<p>I fell into a right funk with public speaking. I lost confidence, then motivation. Only two public-facing events this year - both a virtual and then physical workshop for my Example Mapping course. The latter went really well and brought back my mojo. Luckily I've already got my first speaking engagement for next year sorted out - I'll be speaking about Smart Notes for Developers at NEBytes in January.</p>
<h2>Podcasting!</h2>
<p>Podcasts are a good friend of mine, they're the perfect partner for a run. Some of my favourites are:</p>
<ul>
<li><a href="https://www.podmasters.co.uk/oh-god-what-now">Oh God What Now</a></li>
<li><a href="https://www.podmasters.co.uk/the-bunker">The Bunker</a></li>
<li><a href="https://www.podmasters.co.uk/origin-story">Origin Story</a></li>
<li><a href="https://www.podmasters.co.uk/doomsday-watch">Doomsday Watch</a></li>
<li><a href="https://therestispolitics.supportingcast.fm/">The Rest is Politics</a></li>
<li><a href="https://www.calnewport.com/podcast/">Deep Questions with Cal Newport</a></li>
<li><a href="https://athleticomince.com/">Athletico Mince</a></li>
<li><a href="https://www.techwontsave.us/">Tech Won't Save Us</a></li>
<li><a href="https://www.agileuprising.com/">Agile Uprising</a></li>
</ul>
<p>This year I was fortunate enough to appear on three podcast episodes - answering <a href="https://samhogy.co.uk/2022/04/do-you-actually-need-a-microfrontend/">"Do You Need a Microfrontend"</a> on Scott Logic's Beyond the Hype podcast, conducting a <a href="https://samhogy.co.uk/2022/05/a-two-year-retrospective-on-remote-work/">Two Year Retrospective on Remote Work</a> and talking about <a href="https://samhogy.co.uk/2022/07/the-waterfall-in-the-room/">The Waterfall in the Room</a> with the Agile Uprising folks. Joining in with podcasts has been a fun and slightly out-my-comfort-zone experience and something I'd very much like to continue honing next year.</p>
<h2>Running</h2>
<p>A mix of ups and downs this year. Poor start. After my birthday I made a serious effort to correct that, using the longer summer hours to make up distance once the kids were asleep. For the first time since 2014 I ran over 30 miles in a month, finally getting back and comfortable running 10ks! As the nights got longer and the need to build my Halloween decorations kicked in, running tailed off, leaving the end of the year very much back to where it started. Consistency is the key here. I've found that running gives me mental resiliency, it clears away the cobwebs, gives me the time and space to decompress and I really need to pay attention to this as I get older. Perhaps even complementing running with some other exercise types.</p>
<table>
<thead>
<tr>
<th>Month</th>
<th>Distance</th>
</tr>
</thead>
<tbody>
<tr>
<td>Jan</td>
<td>7</td>
</tr>
<tr>
<td>Feb</td>
<td>10</td>
</tr>
<tr>
<td>Mar</td>
<td>0</td>
</tr>
<tr>
<td>Apr</td>
<td>0</td>
</tr>
<tr>
<td>May</td>
<td>11</td>
</tr>
<tr>
<td>Jun</td>
<td>28</td>
</tr>
<tr>
<td>Jul</td>
<td>30</td>
</tr>
<tr>
<td>Aug</td>
<td>24</td>
</tr>
<tr>
<td>Sep</td>
<td>21</td>
</tr>
<tr>
<td>Oct</td>
<td>17</td>
</tr>
<tr>
<td>Nov</td>
<td>6</td>
</tr>
<tr>
<td>Dec</td>
<td>12</td>
</tr>
</tbody>
</table>
<h2>Entertainment</h2>
<p>Games-wise at the back end of last year I was thoroughly playing the shit out of Metroid Dread on the Nintendo Switch, which is just an amazingly well-designed game, with just the right amount of difficulty to keep me engaged but challenged. This then sent me on a bit of a retro kick, my wife gifted me a Retroid Pocket 2+ for Christmas and I used it to play some old Game Boy games I've got shoved somewhere in the loft - Metroid Fusion and Metroid Zero Mission, then Wario Land 2, including all the secret endings! Following that was the release of Splatoon 3 on the Nintendo Switch, which has got me back into the fantastic world of colouring in maps with brightly-coloured paint. I still play a lot of Splatoon 3. Late in the year I purchased a Steam Deck so I've also been enjoying playing some metroidvania games that have been kicking around in my Steam library, unplayed, for quite a while: Guacamelee, SteamWorld Dig 1 and 2.</p>
<p>This year I've been watching a lot of wrestling, again, in particular AEW. I don't think this has been the strongest year the company's had, but they always put on a good show filled with a variety of high-quality wrestling. It's a shame that some off-screen drama detracted from the product this year. Yes, I know it's fake, but so is most of the stuff you watch on telly too. Standout performances this year are from Jon Moxley (who carried the company during turbulent times - again), Jamie Hayter (so glad she's finally women's champ!) and Chris Jericho, who even at this stage in his career is consistently driving the industry forward. Both Daniel Garcia and Wheeler Yuta are going to be big stars (their ROH Pure Championship feud has been great), as is Jungle Boy Jack Perry, whose program with Luchasaurus and Christian Cage ended in one of the best Steel Cage matches I've seen.</p>
<p>Music-wise, probably as expected, my highlight is IMPERA by Ghost. This is a band at full confidence, full strength, breaking new ground through a modern-day lens of nostalgia. Don't be put off my the costumes and make-up.</p>
<p>As for books, my favourite two tech books of the year have been Sooner Safer Happier (which I describe as "everything we've learnt about this Agile thing in the last twenty years") and The Staff Engineer's Path. The former has really aided me when speaking to commercial folk at work, bridging the gap between the technical foundations of agility and business agility. The latter has helped me with wayfinding in the new role, helping me shape it into what I want it to be, working on the things I want to be working on, and how best to elevate the teams that my efforts support. A special mention goes to my favourite tech book of last year: Fixing Your Scrum, which is genuinely like a book of cheat codes. It takes an approach I like - starting from the fundamental values - and engaging in practical problems through the lens of those values. How can we demonstrate more openness? That kind of thing. It's really good. All these books are ones you'll be encouraging others to buy and read!</p>
<p>For non-techie stuff I've recently been re-reading era 2 of Brandon Sanderson's Mistborn series, ready for the finale of that phase of books. Prior to that I caught up on his Stormlight Archive series, which are epic-length books that remain surprisingly readable. if you're into your fantasy, you can't go wrong here.</p>
<h2>What's Next?</h2>
<p>Genuinely? I don't know. I suspect the transition to SAFe will really hold a mirror up to some processes that we have at work, and present us with a real opportunity to adapt how we do things. We've already made great strides on the engineering front around continuous delivery - I'm going to be championing that (and the related "build quality in" concerns required to enable continuous delivery) over the course of the year. Almost like a consultant but on the full-time payroll.</p>
<p>Anyway, I'm off for my traditional Christmas Eve curry. Be merry, let's do it all over again in the next solar orbit.</p>
The Waterfall in the Room2022-07-17T00:00:00Zhttps://samhogy.co.uk/2022/07/the-waterfall-in-the-room/Project Management - what is it good for?!<p>Once again I stayed up late and joined my friends over at the <a href="https://agileuprising.libsyn.com/">Agile Uprising Podcast</a> for a panel discussion on traditional project management. What can we learn from those in that role, and what maybe have we lost, or forgotten?</p>
<p>You can listen to the discussion here, or wherever you get your podcasts:</p>
<iframe width="640" height="360" src="https://www.youtube-nocookie.com/embed/2eSBzBLPll0" frameborder="0" allowfullscreen></iframe>
A Two-Year Retrospective on Remote Work2022-05-08T00:00:00Zhttps://samhogy.co.uk/2022/05/a-two-year-retrospective-on-remote-work/Today is March 799th, 2020!<p>I was delighted to join my friends over on the <a href="https://agileuprising.libsyn.com/">Agile Uprising Podcast</a> (one of my favourite running podcasts!) to have a two-year retrospective on remote working, since that sodding virus forced us all to work from home.</p>
<p>You can listen to the discussion here, or wherever you get your podcasts:</p>
<iframe width="640" height="360" src="https://www.youtube-nocookie.com/embed/XT4pEJrPAME" frameborder="0" allowfullscreen></iframe>
<p>We covered a <em>lot</em>, and also not enough!</p>
<p>Note, I was the only "not a coach" on the panel, so my perspective was somewhat different to those whose role involves a lot of travel, a lot of on-site work. But, that's the advantage of panels - you get diverse perspectives.</p>
<p>Last year I outlined my thesis on <a href="https://samhogy.co.uk/talks/the-next-normal">the Next Normal</a> and my contributions in this discussion were massively influenced by that desired direction of travel.</p>
<p>Thanks a bunch to Murm, Claudia, SKellogg and MikeC for forming such a lively panel!</p>
Do You Actually Need a Micro-Frontend?2022-04-05T00:00:00Zhttps://samhogy.co.uk/2022/04/do-you-actually-need-a-microfrontend/Hype or not hype?!<p>I recently joined my former colleagues Colin Eberhardt and Dean Kerr from Scott Logic, to discuss whether microfrontends are hype or not hype.</p>
<p>You can listen to the discussion here, or wherever you get your podcasts:</p>
<iframe title="Embed Player" src="//play.libsyn.com/embed/episode/id/22367951/height/192/theme/modern/size/large/thumbnail/yes/custom-color/ffffff/time-start/00:00:00/playlist-height/200/direction/backward" scrolling="no" allowfullscreen="" webkitallowfullscreen="true" mozallowfullscreen="true" oallowfullscreen="true" msallowfullscreen="true" style="border: none;" width="100%" height="192"></iframe>
<p>Colin focuses a lot on the definition of micro-frontend, and whether it fits the concept of a pattern, given how many different interpretations and implementations of the style exist. I prefer seeing micro-frontends as both a decision framework and then a collection of techniques derived from decisions taken from that framework. This approach is formalised in <a href="https://learning.oreilly.com/library/view/building-micro-frontends/9781492082989/">Building Microfrontends</a> by <a href="https://lucamezzalira.com/">Luca Mezzalira</a>.</p>
<p>Although we ended up sitting on different scales of hype or not hype, we were actually all in agreement that micro-frontends solve a specific problem that many teams probably won't immediately encounter. I really enjoyed having this discussion and reminiscing on past projects mirroring the evolution of the web.</p>
Words and Phrases I'm Trying to Stop Using2022-03-31T00:00:00Zhttps://samhogy.co.uk/2022/03/words-and-phrases-im-trying-to-stop-using/If I'm gonna catch up, first I gotta 'fess up!<p>We are all on journeys, learning to improve. I'm mindful of how language affects how we work effectively together. I want to speak more precisely, but productively. "Being blunt" is not an excuse for being curt. These are some phrases in my own vocabulary that I have identified as requiring improvement. Listing these publicly is important to me, not only to clarify to myself <em>why</em> I want to improve on these words and phrases, but also to make a commitment (to the ether, I guess) to making the improvement.</p>
<h2>I disagree</h2>
<p>Now, "I see this differently". Conflict is an essential team dynamic as it surfaces differences in perspective. What is important is facilitating conflict to be funneled into something productive and progressive: conflict should be resolved with positive benefit to the team. Disagreements are entirely legitimate. Voicing them both requires, and is key to, psychological safety. Crucially, conflict shouldn't be made personal as that is where things get ugly. So, it's not that I'm disagreeing with you (although I am), it's that I see things <em>differently</em>. I'm focusing more on how we can resolve the tension, rather than putting the other person on the defensive.</p>
<h2>Yes, and No</h2>
<p>From my experience at the (role-playing) tabletop, I'm more inclined to use "Yes, and", or "No, but" now. As with above, this is about making sure we're swimming in the same direction and moving things along. Rather than stonewalling someone with a hard no, I will instead try follow up with a "but" and offer an alternative or next course of action. This applies to "yes, and" too. Build on good ideas, establish positive momentum!</p>
<h2>The business</h2>
<p>There isn't an exact replacement, but something along the lines of: "product", "operations", "analysts", "our business" (depending on context).</p>
<p>The idea of a separate unit being "the" business comes from the IT as a Cost Centre thinking, with service delivery contracts, functional silos and internal accounting wrangling. There are better words to use when referring to the perspectives of a person acting in a particular role (the Product Owner, or even use their name!). This change in language aims to prevent and discourage the development of Us Vs Them mentalities. Everyone is the business.</p>
<h2>As Soon as Possible</h2>
<p>As soon as possible can be coded for many things, but a generous interpretation is that something is needed immediately but not urgently. More: at your convenience, but very soon. I now prefer to be more specific regarding my expectations on the time boundary. If it's needed immediately, say so. Within the hour? Say so. It can be done any time today? You get what I mean. Furthermore, by communicating this expectation the recipient can reconcile this with their own workload and suggest an alternative path.</p>
<h2>Command and Control</h2>
<p>Command and control organisations are considered traditional top-down, hierarchical, managerialist working environments where employees have little agency and autonomy. It is in these places where waterfall management practices are rife.</p>
<p>I now prefer to remove the military metaphor and instead use "plan and execute". I do this because "command and control" <em>is</em> a pejorative and people on the command side (relative to those who they "control") get defensive about it, and themselves are controlled from a higher command. So, I feel "plan and execute" places more focus on the work and how we don it, rather than the people, whilst still accurately conveying the message of separating the thinking and doing.</p>
<h2>Resources</h2>
<p>They're people. With thoughts, feelings, concerns, dreams and families. I am not a number, I am a free man! Distilling people down to resources / work units / whatever other Taylorist term you can imagine is an abstract, inhumane way of organisation. Rather than bringing the work to the resources, bring the people to the work! Get them involved, get them motivated, listen to them. Treat people as people.</p>
<h2>Agile</h2>
<p>I'm not necessarily sure <em>what</em> to replace this with yet. And I think replacing it will be the most difficult just because of its usage as a noun. I'm going to try and speak more about <strong>agility</strong>, but less under the "Agile" umbrella. Maybe it's product economics, or technical practices, or whatever. It's not that <a href="https://www.youtube.com/watch?v=a-BOSpxYJ9M">Agile is Dead</a>, it's that it's got too much emotional baggage. And that's clouding the message.</p>
<h2>On/Off/Near/shoring</h2>
<p>I'm editing this post as of 1st April 2022 to include this. Thanks, <a href="https://twitter.com/chrismurman">Murm</a>, for reminding me of this one.</p>
<p>Previously I worked in a consultancy role who offered nearshoring services - which means the consultants are in (roughly) the same time zone, albeit in a secondary city to offer the benefits of lower salaries than a capital city. I'm not opposed to this business model, but I have concerns with the term, as with offshoring. It's Anglocentric (or maybe Western-centric), because "off"shore normally means India, or perhaps Eastern Europe at a push. It highlights that these people are somewhere else. That they're "off" and we're "on". I've seen superiority arguments about near versus offshoring which I've found uncomfortable and you can compute for yourself easily enough.</p>
<p>I don't want to use these terms any more. I might instead use the office name, but if there's a team name I'd use that. Anything to try and reduce the Us Vs Them mentality.</p>
Making the Most of Snapshot Testing2022-03-04T00:00:00Zhttps://samhogy.co.uk/2022/03/making-the-most-of-snapshot-testing/Tips and tricks to avoid the common snapshot pitfalls!<p>I am a huge fan of <a href="https://jestjs.io/docs/snapshot-testing">Jest's Snapshot Testing</a> - it's a powerful but often misused feature of the testing library. In this post, I'd like to give some advice about how to use snapshot testing effectively within your code, and not just for React components!</p>
<h2>How Does It Work?</h2>
<p>Essentially, any output that you would like to capture in a snapshot uses a special assertion: <code>expect(something).toMatchSnapshot()</code>.</p>
<p>The first time that a test with this assertion is run, Jest will generate a snapshot file, that's stored in a <code>__snapshots__</code> directory. <strong>Commit this to source control</strong>, as it will be used on subsequent runs of that test, both locally and on CI, to determine whether there has been a change to the contents of that file. if there has, the test fails.</p>
<p>A snapshot is just that - a snapshot in time. The test will fail when the snapshot content changes, but that's not necessarily a bad thing. It may be that you've made a legitimate change and want this recorded in the snapshot. So, you can re-run the tests with an instruction to update the snapshots. This test run will edit the snapshot file, producing a diff that you can supply to the team at code review and capture the change. It's common to hook up the run of <code>jest</code> to the <code>test</code> npm script in <code>package.json</code>, so that testing can be kicked off running <code>npm test</code>, as such:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"test"</span><span class="token operator">:</span> <span class="token string">"jest"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>So to then update the snapshots, run <code>npm test -- -u</code>. If you use Jest in watch mode, then there's an option (<code>i</code>) to update snapshots interactively.</p>
<h2>My Tips and Tricks for Snapshotting Effectively</h2>
<h3>Don't Snapshot Everything</h3>
<p>Because <code>toMatchSnapshot</code> is such an easy assertion to use, in the past I've ran into problems where I have captured an entire React component tree in a single snapshot. What this ends up doing is producing a tens-of-lines-long snapshot, which reviewers assume is correct at the point of code review. But later on, it becomes undiffable when there's a change, and because it's capturing such a large portion of code, it may change for unrealted reasons (for instance if a css class name changes).</p>
<p>Any testing technique should provide an <strong>unambiguous, actionable signal</strong> as to the failure case. Eagerly snapshotting deep component trees, or large JSON structures, and so on, goes against this rule. Be really careful about only capturing what matters for <strong>this particular test</strong>. Kevlin Henney describes this as <a href="https://97-things-every-x-should-know.gitbooks.io/97-things-every-programmer-should-know/content/en/thing_81/">testing "precisely and concretely"</a>.</p>
<p>The <a href="https://github.com/jest-community/eslint-plugin-jest">Jest ESLint Plugin</a> has a preconfigured rule to discourage large snapshots. By default this is 50 lines, which is still too big in my opinion, bu this can be changed through config options.</p>
<h3>Consider Inline Snapshots for Simple Use Cases</h3>
<p>If your snapshot is relatively small it can be more readable to use <code>.toMatchInlineSnapshot</code>. There's two ways to set this up:</p>
<ol>
<li>If you know what the snapshot is going to be up front, pass it as a string to <code>.toMatchInlineSnapshot</code></li>
<li>If you want the initial inline snapshot autogenerated for you, call <code>.toMatchInlineSnapshot</code> without any arguments and it will be written into the test file for you.</li>
</ol>
<h3>Use Readable Snapshot Names</h3>
<p>By default, the snapshot name is based on the test name, plus an incrementing integer to support multiple snapshots per test. Following standard conventions on good test naming goes a long way here. Consider that when the test is nested within multiple <code>describe</code> blocks, the snapshot name will contain the full context.</p>
<p>Should any further context be needed, it is possible to pass a string as an argument to <code>.toMatchSnapshot(hintString);</code>. This will then be appended to the snapshot test name. Tests don't just exist for behaviour verification - they can be examples on how to use the code. We spend more time reading code than we do writing code, so every little helps when it comes to making tests more readable.</p>
<h3>Ensure Snapshots Are Deterministic</h3>
<p>In a nutshell, the same inputs should produce the same output, as the purpose of the snapshot is to capture a save point that can be used for later comparison purposes! This may seem easy, but there's always subtleties that can make this tricky.</p>
<p>For instance, anything that uses <code>Date.now()</code> is going to produce a different result on every execution. To make these tests stable, mock out this global function to return a static date:</p>
<p><code>Date.now = jest.fn(() => 1646402310215); // That value is 22-03-04 at 13:58</code></p>
<p>This can also be used to mock out other indeterminate functions, such as <code>Math.random()</code>. But, setting a mock with a hardcoded value might be undesireable for some other test cases in your suite. When snapshotting JavaScript objects, Jest can use a <strong>Property Matcher</strong> syntax to apply some fuzzy matching logic to indeterminate properties. This lets the test run without any static mocks, but also remain deterministic where it matters:</p>
<pre class="language-js"><code class="language-js"><span class="highlight-line"><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'uses property matchers for indeterminate properties'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> item <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">created</span><span class="token operator">:</span> <span class="token keyword">new</span> <span class="token class-name">Date</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> Math<span class="token punctuation">.</span><span class="token function">random</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">tag</span><span class="token operator">:</span> <span class="token string">'test'</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>item<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toMatchSnapshot</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">created</span><span class="token operator">:</span> expect<span class="token punctuation">.</span><span class="token function">any</span><span class="token punctuation">(</span>Date<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> expect<span class="token punctuation">.</span><span class="token function">any</span><span class="token punctuation">(</span>Number<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Which produces:</p>
<pre class="language-js"><code class="language-js">exports<span class="token punctuation">[</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">uses property matchers for indeterminate properties 1</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<span class="highlight-line">Object {</span>
<span class="highlight-line"> "created": Any<Date>,</span>
<span class="highlight-line"> "id": Any<Number>,</span>
<span class="highlight-line"> "tag": "test",</span>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></code></pre>
<p>Note that the non-property matched property <code>tag</code> in the <code>item</code> object is written verbatim to the snapshot.</p>
<h3>You Can Add Custom Serializers!</h3>
<p>The Jest <code>expect</code> global is configurable. In addition to adding your own custom assertions, it's also possible to add your own custom snapshot serializers! This controls how Jest will translate an incoming piece of data into a textural format to be stored in the snapshot file. Supply a <code>test</code> function to specify the conditions this serializer is used for, and a <code>print</code> function to produce the output.</p>
<p>Here's a really simple example that serializes all strings to lowercase:</p>
<pre class="language-js"><code class="language-js"><span class="highlight-line">expect<span class="token punctuation">.</span><span class="token function">addSnapshotSerializer</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function-variable function">test</span><span class="token operator">:</span> <span class="token parameter">value</span> <span class="token operator">=></span> <span class="token keyword">typeof</span> value <span class="token operator">===</span> <span class="token string">'string'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">print</span><span class="token operator">:</span> <span class="token parameter">value</span> <span class="token operator">=></span> value<span class="token punctuation">.</span><span class="token function">toLowerCase</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>You cna be really inventive here, formatting arrays as asciidoc tables, for example. Just remember, make sure that whatever output you make produces <strong>clean diffs</strong> when something changes and helps the developer understand if a change to a snapshot was intended or unintended - so that they can update the snapshot, or fix their code, respectively.</p>
<h3>Multiple Snapshots per Test? Try Only Snapshotting the Changes</h3>
<p>A common pattern when testing frontend code is to capture the state of a component in a snapshot before and after some interaction. In the spirit of making the changes easy to read and easy to diff, consider introducing the <a href="https://github.com/jest-community/snapshot-diff">snapshot-diff</a> utility. This provides a <code>snapshotDiff</code> function that accepts before and after arguments, the only outputs what's different between the two!</p>
<pre class="language-js"><code class="language-js"><span class="highlight-line"><span class="token function">expect</span><span class="token punctuation">(</span><span class="token function">snapshotDiff</span><span class="token punctuation">(</span><span class="token operator"><</span>SayHello name<span class="token operator">=</span><span class="token string">"sam"</span> <span class="token operator">/</span><span class="token operator">></span><span class="token punctuation">,</span> <span class="token operator"><</span>SayHello name<span class="token operator">=</span><span class="token string">"ewan"</span><span class="token operator">></span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toMatchSnapshot</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>The alternate syntax <code>expect(a).toMatchDiffSnapshot(b)</code> exists, hooking into Jest as a cusom assertion. There's some funky syntax to be aware of if you use custom snapshot serializers, so check out the docs if that applies to you.</p>
<h2>Go Forth and Snapshot</h2>
<p>Snapshot testing is a great addition to your utility belt! It doesn't need to just be limited to testing React components. As JSON is a nice serializable format, snapshot testing can be a great tool for <a href="/2018/07/jest-snapshots-for-api-contract-testing/">API contract testing</a>Developing the expertise about when to and when not to snapshot test is something that comes with time, but with the tips above, you should be well to understand how to make the most out of snapshot testing without causing too much of a headache.</p>
Building a Brainforest - The Tech2022-02-01T00:00:00Zhttps://samhogy.co.uk/2022/02/building-a-brainforest-the-tech/Choose Your Own Adventure, through my brain!<a href="https://brainforest.samhogy.co.uk">
<div>
<img src="https://samhogy.co.uk/assets/brainforest.png" height="300" alt="Brainforest logo" />
<p>Visit the Brainforest</p>
</div>
</a>
<p>For the last year or so, I've been experimenting with a new form of note-taking. At school age I discovered quite quickly that by <strong>writing things down in my own words</strong> I could keep good amounts of working information in my head - and got quite good at passing exams. I also started to find interesting connections between subjects. But I never really tugged at those strings properly, so now as an adult I'm giving it another go.</p>
<p>The tool I use for this is <a href="https://obsidian.md/">Obsidian</a> - a wiki-style markdown editor application designed for linked note-taking. Inspired by the <a href="https://takesmartnotes.com/">How to Take Smart Notes</a> book, itself drawing upon the <a href="https://zettelkasten.de/posts/overview/">Zettelkasten methodology</a>, I have developed a process in which I orient notes based upon ideas, rather than the resources which those ideas sprouted from. By linking these notes together, one builds a web of synthesized concepts, out of which new ideas emerge. This is my current Obsidian graph:</p>
<img src="https://samhogy.co.uk/assets/obsidian-note-graph.png" alt="Obsidian notes graph" />
<p>The core concept in this note-taking system is the production of evergreen notes - concept-oriented, deeply-linked, atomic notes. I'll touch more on how I go about writing these in a later blog post, but in this post, I'd like to talk about how I go publishing these evergreen notes to my website. There is no formal hierarchical structure to these notes - it's a web. Therefore, I'd like the app to almost take on the "choose your own adventure" feel, giving little in the way of direction and allowing the user to explore based on what interests them.</p>
<p>Although an off-the-shelf paid-for solution exists in the form of <a href="https://obsidian.md/publish">Obsidian Publish</a>, I decided to try and build something myself as a weekend challenge. In particular, I didn't want to use any JavaScript frameworks such as React. On the first Saturday a working version with some dummy notes was up and running. Sunday afternoon was used to add some animation effects. During breaks at work through the week I tinkered further - adjusting the styling, tidying up the build process, adding stats collection and some helper pages.</p>
<p>This article covers the implementation as of commit <a href="https://gitlab.com/sh1989/brainforest/-/tree/850f393a68367a00f81492418625849db715c2ac">850f39</a> - which will no doubt evolve, but this is the latest at the time of writing.</p>
<h2>The Build Process</h2>
<p>First, I need to transform my Obsidian evergreen notes from their Markdown form into HTML pages. This is run as a simple NodeJS script utilising the builtin <code>fs</code> file reading library. Because my Obsidian workspace contains more than the evergreen notes - I simply copy over the <code>evergreen/</code> folder into the brainforest repo. Hey, it's manual, but it does act as an extra backup!</p>
<p>The main function orchestrates the following steps:</p>
<ul>
<li>Read the directory containing the markdown notes, to get a list of files - using <code>fs.readdir</code></li>
<li>Read each file and transform Obsidian wiki-links to standard Markdown links</li>
<li>Append additional templates to the content, such as the stats section on the introduction page, or a backlinks section on a note page</li>
<li>Convert to HTML and write to the file system</li>
</ul>
<p>Throughout the process, I maintain an object (called <code>nodes</code>) to track each page and its outgoing and incoming links. This object is useful for generating stats about the notes in the brainforest, but it is also used to calculate backlinks to a given page - essentially, what other notes link <em>to</em> this page?</p>
<p>The file names in Markdown are plain English, spaces and all. This is deliberate, as it makes the links in Obsidian much more readable. As part of the transformation to Markdown, I 'slugify' the file names to turn them into a string which is more suitable to be used as an <code>id</code> in HTML:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">slugify</span><span class="token punctuation">(</span><span class="token parameter">s</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> s</span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">toLowerCase</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">[^a-z0-9 -]</span><span class="token regex-delimiter">/</span><span class="token regex-flags">g</span></span><span class="token punctuation">,</span> <span class="token string">""</span><span class="token punctuation">)</span> <span class="token comment">// remove invalid chars</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\s+</span><span class="token regex-delimiter">/</span><span class="token regex-flags">g</span></span><span class="token punctuation">,</span> <span class="token string">"-"</span><span class="token punctuation">)</span> <span class="token comment">// collapse whitespace and replace by -</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">-+</span><span class="token regex-delimiter">/</span><span class="token regex-flags">g</span></span><span class="token punctuation">,</span> <span class="token string">"-"</span><span class="token punctuation">)</span> <span class="token comment">// collapse dashes</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">^-+</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span> <span class="token string">""</span><span class="token punctuation">)</span> <span class="token comment">// trim - from start of text</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">-+$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span> <span class="token string">""</span><span class="token punctuation">)</span> <span class="token comment">// trim - from end of text</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">-</span><span class="token regex-delimiter">/</span><span class="token regex-flags">g</span></span><span class="token punctuation">,</span> <span class="token string">"-"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The <code>nodes</code> object is constructed by iterating over the array of file names, and converting it (through a reducer function) into an object with the following shape:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> nodes <span class="token operator">=</span> files<span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">agg<span class="token punctuation">,</span> fileNameWithPath</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token punctuation">{</span> fileName<span class="token punctuation">,</span> slug <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">fileDetails</span><span class="token punctuation">(</span>fileNameWithPath<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> agg<span class="token punctuation">[</span>slug<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> fileName<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">outboundLinks</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">inboundLinks</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> agg<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>File links in Obsidian are written using the wiki-style convention. Unlike a conventional markdown link, <code>[which looks like this](./a-path-to-go-to.html)</code>, an internal link is represented <code>[[Like this]]</code>. When reading a file, I also run the Markdown content through a replacement function that searches and replaces these links. It also is supplied with the <code>nodes</code> object so that it can check whether an internal link actually exists. It is this function that updates the <code>nodes</code> object to track outgoing and incoming links:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">deObsidianify</span><span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> fileName<span class="token punctuation">,</span> slug <span class="token punctuation">}</span><span class="token punctuation">,</span> data<span class="token punctuation">,</span> nodes</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// Add file name as a <h1> element</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> title <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string"># </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>fileName<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token string">'.'</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>title<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">\n\n</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>data<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\[\[[A-Za-z,'\- ]+\]\]</span><span class="token regex-delimiter">/</span><span class="token regex-flags">g</span></span><span class="token punctuation">,</span> <span class="token parameter">match</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> trimmed <span class="token operator">=</span> match<span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token string">'[['</span><span class="token punctuation">,</span> <span class="token string">''</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">replace</span><span class="token punctuation">(</span><span class="token string">']]'</span><span class="token punctuation">,</span> <span class="token string">''</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> linkedSlug <span class="token operator">=</span> <span class="token function">slugify</span><span class="token punctuation">(</span>trimmed<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>nodes<span class="token punctuation">[</span>linkedSlug<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> nodes<span class="token punctuation">[</span>linkedSlug<span class="token punctuation">]</span><span class="token punctuation">.</span>inboundLinks<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>slug<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> nodes<span class="token punctuation">[</span>slug<span class="token punctuation">]</span><span class="token punctuation">.</span>outboundLinks<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>linkedSlug<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">[</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>trimmed<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">](</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>linkedSlug<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">.html)</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token string">''</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Before running through the conversion function, there's some dynamic content that I'd like to append to the files. For the introduction page, I display some simple stats about the number of notes and links between them, alongside a last published time. For an individual note, if there are backlinks to it, I append a section linking to those notes. Without a JavaScript framework to rely on, I instead used JavaScript Template Strings - a modern language feature for dynamically constructing strings:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> <span class="token function-variable function">backlinksHTMLTemplate</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> inboundLinks <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> nodes <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> inboundLinks<span class="token punctuation">.</span>length <span class="token operator">?</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">\n
<span class="highlight-line"><section id="backlinks"></span>
<span class="highlight-line"><h3>Links here</h3></span>
<span class="highlight-line"><ul></span>
<span class="highlight-line"></span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>inboundLinks<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">l</span> <span class="token operator">=></span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string"><li><a href="./</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>l<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">.html"></span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>nodes<span class="token punctuation">[</span>l<span class="token punctuation">]</span><span class="token punctuation">.</span>fileName<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"></a></li></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span><span class="token string">'\n'</span><span class="token punctuation">)</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"></span>
<span class="highlight-line"></ul></span>
<span class="highlight-line"></section></span>
</span><span class="token template-punctuation string">`</span></span> <span class="token operator">:</span> <span class="token string">''</span>
<span class="highlight-line"></span>
<span class="token keyword">const</span> <span class="token function-variable function">statsHTMLTemplate</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> lastPublished<span class="token punctuation">,</span> links<span class="token punctuation">,</span> nodes <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">\n
<span class="highlight-line"><h2>The Details</h2></span>
<span class="highlight-line"><ul></span>
<span class="highlight-line"><li>Last published: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>lastPublished<span class="token punctuation">.</span><span class="token function">toLocaleDateString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"></li></span>
<span class="highlight-line"><li>There are </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>Object<span class="token punctuation">.</span><span class="token function">keys</span><span class="token punctuation">(</span>nodes<span class="token punctuation">)</span><span class="token punctuation">.</span>length<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> notes in the brainforest with </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>links<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> links</li></span>
<span class="highlight-line"></ul></span>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></code></pre>
<p>Some interesting things about those templates:</p>
<ul>
<li>Note the <code>.join('\n')</code> after the <code>.map</code> over the array. By default, a comma is used as a separator when joining an array! If you omit a manual join, <a href="https://github.com/sh1989/today-i-learned/blob/master/javascript/why-does-a-map-in-a-template-string-add-commas.md">this is called behind the scenes</a> and the outputting content will have an errant comma.</li>
<li>I wrote the content in HTML rather than Markdown, because I wanted the backlinks piece to live in a <code><section></code>, which Markdown doesn't have a unique construct for. I could, on reflection, have written an extension to the markdown->html converter to support this. But any HTML content is passed straight through unchanged.</li>
</ul>
<p>Conversion from Markdown to HTML is incredibly easy, using the <a href="https://github.com/showdownjs/showdown">showdown</a> library. Just init the converter with any options you need:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> converter <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">showdown<span class="token punctuation">.</span>Converter</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">noHeaderId</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">parseImgDimensions</span><span class="token operator">:</span> <span class="token boolean">true</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>And then call it, writing the content to a file using <code>fs</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">convertToHtml</span><span class="token punctuation">(</span><span class="token parameter">file</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> html <span class="token operator">=</span> converter<span class="token punctuation">.</span><span class="token function">makeHtml</span><span class="token punctuation">(</span>file<span class="token punctuation">.</span>data<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> fileName <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>file<span class="token punctuation">.</span>slug<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">.html</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> fs<span class="token punctuation">.</span><span class="token function">writeFileSync</span><span class="token punctuation">(</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>outputDir<span class="token punctuation">,</span> fileName<span class="token punctuation">)</span><span class="token punctuation">,</span> html<span class="token punctuation">,</span> <span class="token punctuation">{</span> encoding <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>What I end up with is a directory containing a HTML page for each individual note. At runtime, the code in the browser can then retrieve these and stitch them onto the page! For local development, these HTML pages are served from an <code>express</code> server to mimic how the content is hosted on Gitlab Pages.</p>
<h2>The Browser Script</h2>
<p>Moving on to the job of the browser code, then. In a nutshell, it should load the introduction note and then listen out for any clicks on <code><a></code> links. Rather than allowing the browser to handle these and navigate to a new page, it should intercept the event and load the link's content using an XHR request, appending the resulting content to the current page. Simple!</p>
<p>There are three core steps to this:</p>
<ol>
<li>Retrieve the article content</li>
<li>Adding the new article content to the page DOM</li>
<li>Setting up event listeners to intercept link clicks.</li>
</ol>
<p>The first piece is rather trivial, using the <code>fetch</code> API:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">getArticleContent</span><span class="token punctuation">(</span><span class="token parameter">url</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> res <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">fetch</span><span class="token punctuation">(</span>url<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>res<span class="token punctuation">.</span>ok<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span>res<span class="token punctuation">.</span>status<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">await</span> res<span class="token punctuation">.</span><span class="token function">text</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span></code></pre>
<p>Creating the article is actually itself rather trivial too. I want to keep the semantic structure of the overall page, as well as the loaded article content. Luckily, the HTML5 <code><article></code> element serves this purpose perfectly - allowing for its own nested structure of headings. I convert the article url path into an id by removing the relative path prefix ('/') and the trailing <code>.html</code>. This is then used to assign the article a unique id in the DOM.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">createArticle</span><span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> content</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> article <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">createElement</span><span class="token punctuation">(</span><span class="token string">'article'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> article<span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> article<span class="token punctuation">.</span>innerHTML <span class="token operator">=</span> content<span class="token punctuation">;</span></span>
<span class="highlight-line"> notes<span class="token punctuation">.</span><span class="token function">appendChild</span><span class="token punctuation">(</span>article<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">markLocation</span><span class="token punctuation">(</span>id<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">bindEvents</span><span class="token punctuation">(</span>id<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Finally, binding the events. Given that an article has its own unique ID. I use this to grab the newly-created element and query it for any links (that don't have the <code>external</code> class). These represent links to other notes in the system. For each of those links, I add an event handler to the <code>click</code> event. It prevents the event from bubbling up to the browser - which would trigger a full page navigation. Instead, it retrieves the HTML and kickstarts step one of the process.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">bindEvents</span><span class="token punctuation">(</span><span class="token parameter">id</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> article <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">getElementById</span><span class="token punctuation">(</span>id<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> links <span class="token operator">=</span> article<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'a:not(.external)'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> links<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">link</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> link<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'click'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> e<span class="token punctuation">.</span><span class="token function">preventDefault</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">navigateToArticle</span><span class="token punctuation">(</span>link<span class="token punctuation">.</span><span class="token function">getAttribute</span><span class="token punctuation">(</span><span class="token string">'href'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">catch</span><span class="token punctuation">(</span><span class="token parameter">_</span> <span class="token operator">=></span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>If you look at the source code, you'll see there's a little bit more functionality there than this. I added some CSS animations to slide articles in, some error-handling functionality around loading duplicate articles.</p>
<p>One final feature I'd like to cover is the ability to link to a specific note. Whenever a note is inserted into the DOM, I call a function called <code>markLocation</code> with the element id. This uses the history API to update the URL. When a url is suffixed with <code>#an-id</code>, a browser will load the page and scroll to the DOM element with that id. I want to replicate this as close as I can.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">markLocation</span><span class="token punctuation">(</span><span class="token parameter">id</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> history<span class="token punctuation">.</span><span class="token function">replaceState</span><span class="token punctuation">(</span><span class="token keyword">undefined</span><span class="token punctuation">,</span> <span class="token keyword">undefined</span><span class="token punctuation">,</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">#</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>So when the page loads, in addition to loading the introduction article, if there's an id in the URL pointing to a different note, <em>also</em> load that:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> directLink <span class="token operator">=</span> window<span class="token punctuation">.</span>location<span class="token punctuation">.</span>hash<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> intro <span class="token operator">=</span> <span class="token string">'welcome-to-the-brainforest'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">await</span> <span class="token function">navigateToArticle</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>intro<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">.html</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>directLink <span class="token operator">&&</span> directLink <span class="token operator">!==</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">#</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>intro<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">await</span> <span class="token function">navigateToArticle</span><span class="token punctuation">(</span>directLink<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token string">'#'</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">+</span> <span class="token string">'.html'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> console<span class="token punctuation">.</span><span class="token function">error</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Next Steps</h2>
<p>At this moment in time, the app is pretty functional, and using the CSS (and altering the <code>flex-direction</code> from <code>column</code> to <code>row</code>), I can have decent views of the content in both desktop and mobile browsers. The JavaScript and CSS code itself isn't minified, but it's small enough to not gain a huge amount of benefit from doing so. There are a few little bugs that I'll fix when I can, but aside from that this is pretty much where I need it to be. I'm tempted to build a graph view dynamically as a user explores, in a Roguelike manner...</p>
<p>In the next post, I'll talk more about the note-taking aspect itself and what I've discovered from building my Brainforest!
Don't forget that you can view the <a href="https://gitlab.com/sh1989/brainforest">Brainforest source code repository</a> and <a href="https://brainforest.samhogy.co.uk">explore the Brainforest</a> for yourself!</p>
Gherkin Golf2021-07-02T00:00:00Zhttps://samhogy.co.uk/2021/07/gherkin-golf/I try my hand at writing the best Gherkin I can!<p>Over on LinkedIn <a href="https://www.linkedin.com/posts/john-ferguson-smart_four-simple-rules-for-better-gherkin-scenarios-activity-6816629042993762304-918v">John Ferguson Smart posted an interesting comparison</a> between two Gherkin scenarios illustrating a business rule. This is from his excellent book, <a href="https://www.manning.com/books/bdd-in-action">BDD in Action</a>.</p>
<p>The original scenario is:</p>
<pre class="language-gherkin"><code class="language-gherkin"><span class="highlight-line"><span class="token scenario"><span class="token keyword">Scenario:</span><span class="token important"> Earning point test case</span></span></span>
<span class="highlight-line"> <span class="token atrule">Given</span> <span class="token atrule">I</span> open the Purchase Flights page</span>
<span class="highlight-line"> <span class="token atrule">And</span> <span class="token atrule">I</span> purchase a Return ticket from London to New York</span>
<span class="highlight-line"> <span class="token atrule">And</span> <span class="token atrule">I</span> upgrade the return flight to Business</span>
<span class="highlight-line"> <span class="token atrule">And</span> <span class="token atrule">I</span> complete the flight</span>
<span class="highlight-line"> <span class="token atrule">When</span> <span class="token atrule">I</span> log on to the Frequent Flyer site</span>
<span class="highlight-line"> <span class="token atrule">Then</span> <span class="token atrule">I</span> go to the account summary page</span>
<span class="highlight-line"> <span class="token atrule">When</span> <span class="token atrule">I</span> click on <span class="token string">"Earned Points"</span></span>
<span class="highlight-line"> <span class="token atrule">Then</span> <span class="token atrule">I</span> see the total number of points of 1350</span></code></pre>
<p>This is a common occurrence in the wild. It's usually a sign that a manual test script (remember those?!) was transcribed over to an automated test in a like-for-like fashion. Or if it's greenfield, it's written by someone with extensive experience in manual script-based testing who is still applying that frame of reference to scenario formulation.</p>
<p>It's okay. Not great though. This kind of test is doing quite a lot, masking the actual business rule under test. The scenario is tightly coupled to the user interface, which is error-prone and difficult to reason about. Usually, it's because the test is trying to test both the business rule <strong>and</strong> the particular user interface. It loses focus in doing so. And woe betide anyone who has to debug one of these when they fail for seemingly arbitrary reasons!</p>
<p>John suggests a <strong>much nicer</strong> improvement:</p>
<pre class="language-gherkin"><code class="language-gherkin"><span class="highlight-line"><span class="token scenario"><span class="token keyword">Background:</span><span class="token important"></span></span></span>
<span class="token atrule">Given</span> the following flight points schedule:<span class="token table-head">
<span class="highlight-line"> <span class="token punctuation">|</span><span class="token th variable"> From </span><span class="token punctuation">|</span><span class="token th variable"> To </span><span class="token punctuation">|</span><span class="token th variable"> Class </span><span class="token punctuation">|</span><span class="token th variable"> Points </span><span class="token punctuation">|</span></span><span class="token table-body"></span>
<span class="highlight-line"> <span class="token punctuation">|</span><span class="token td string"> London </span><span class="token punctuation">|</span><span class="token td string"> New York </span><span class="token punctuation">|</span><span class="token td string"> Economy </span><span class="token punctuation">|</span><span class="token td string"> 550 </span><span class="token punctuation">|</span></span>
<span class="token punctuation">|</span><span class="token td string"> London </span><span class="token punctuation">|</span><span class="token td string"> New York </span><span class="token punctuation">|</span><span class="token td string"> Business </span><span class="token punctuation">|</span><span class="token td string"> 800 </span><span class="token punctuation">|</span></span>
<span class="highlight-line"> <span class="token atrule">And</span> Stacey is a Frequent Flyer member</span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token scenario"><span class="token keyword">Scenario:</span><span class="token important"> Flights earn points based on distance travelled and cabin class</span></span></span>
<span class="token atrule">Given</span> Stacey has purchased the following tickets:<span class="token table-head">
<span class="highlight-line"> <span class="token punctuation">|</span><span class="token th variable"> From </span><span class="token punctuation">|</span><span class="token th variable"> To </span><span class="token punctuation">|</span><span class="token th variable"> Class </span><span class="token punctuation">|</span></span><span class="token table-body"></span>
<span class="highlight-line"> <span class="token punctuation">|</span><span class="token td string"> London </span><span class="token punctuation">|</span><span class="token td string"> New York </span><span class="token punctuation">|</span><span class="token td string"> Economy </span><span class="token punctuation">|</span></span>
<span class="token punctuation">|</span><span class="token td string"> New York </span><span class="token punctuation">|</span><span class="token td string"> London </span><span class="token punctuation">|</span><span class="token td string"> Business </span><span class="token punctuation">|</span></span>
<span class="highlight-line"> <span class="token atrule">When</span> she completes the flights</span>
<span class="highlight-line"> <span class="token atrule">Then</span> she should earn 1350 points</span></code></pre>
<p>This does so much right. It's decoupled from the UI - how this is presented as a system interface does not change the business rule, therefore it doesn't impact the expression of the rule under test. Sure, under the hood these steps are doing more work than the steps in the original example. The use of tense makes the context, action and expectations clear. I really like this solution.</p>
<p>Here's my take on how I'd formulate this business rule:</p>
<pre class="language-gherkin"><code class="language-gherkin"><span class="token feature"><span class="token keyword">Feature:</span><span class="token important"> Frequent flyer points allocation</span>
</span>
<span class="highlight-line"> Rule: Points allocated upon return flight completion</span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token scenario"><span class="token keyword">Scenario:</span><span class="token important"> Transatlantic flight with upgraded return leg</span></span></span>
<span class="token atrule">Given</span> frequent flyer Freda has purchased a return journey<span class="token table-head">
<span class="highlight-line"> <span class="token punctuation">|</span><span class="token th variable"> From </span><span class="token punctuation">|</span><span class="token th variable"> To </span><span class="token punctuation">|</span><span class="token th variable"> Class </span><span class="token punctuation">|</span><span class="token th variable"> Points </span><span class="token punctuation">|</span></span><span class="token table-body"></span>
<span class="highlight-line"> <span class="token punctuation">|</span><span class="token td string"> London </span><span class="token punctuation">|</span><span class="token td string"> New York </span><span class="token punctuation">|</span><span class="token td string"> Economy </span><span class="token punctuation">|</span><span class="token td string"> 550 </span><span class="token punctuation">|</span></span>
<span class="token punctuation">|</span><span class="token td string"> New York </span><span class="token punctuation">|</span><span class="token td string"> London </span><span class="token punctuation">|</span><span class="token td string"> Business </span><span class="token punctuation">|</span><span class="token td string"> 800 </span><span class="token punctuation">|</span></span>
<span class="highlight-line"> <span class="token atrule">When</span> she completes the return flight</span>
<span class="highlight-line"> <span class="token atrule">Then</span> she has earned 1350 points</span></code></pre>
<p>It's in a very similar spirit to John's example, but with some key differences. I'll outline what I think are the important parts.</p>
<h2>Drop the I, It's Not About Me</h2>
<p>Seb Rose and Gáspár Nagy in <a href="https://bddbooks.com">The BDD Books</a> (vol 2: Formulation) discourage the use of "I":</p>
<blockquote>
<p>Use of the first person "I" in scenarios is common throughout industry and is advocated in many tutorials. However, we encourage people to avoid this ... because it encourages imprecision. In particular, consider that the reader of a scenario will identify themselves as "I". This leads to the meaning of a scenario being dependent on the person who is reading it</p>
</blockquote>
<blockquote>
<p>Instead, we recommend that all scenarios are written in the third person, identifying all actors by role or persona</p>
</blockquote>
<p>Therefore, similar to John, I've utilised a persona. I originally used Stacey as the persona name because he did, but I've changed it to Freda because the alliteration ("Freda the Frequent Flyer") helps me easily recall the context. Gherkin doesn't have formal support for personas - I normally create a <code>personas.md</code> file alongside the <code>.feature</code> files to outline each one.</p>
<h2>Rule, Feature, Scenario</h2>
<p>The <code>Rule</code> keyword is a more recent addition to the Gherkin language. A feature may be an aggregate of several distinct rules, each of which can be illustrated by a series of example scenarios. Looking at John's solution, I was curious that frequent flyer points were only allocated on completion of a return flight. But from experience (don't ask), not all flights have returns. So what happens there? It seems like calling out the behaviour specifically for a return flight has value. That's what a <code>Rule</code> is for!</p>
<p>Furthermore, that rule can be broken down into explicit scenarios. The example given is one where the return leg of the journey is at Business class. Sounds like someone got an upgrade! Using real-world data is useful for retaining the context of the test. <a href="https://cucumber.io/blog/bdd/example-mapping-introduction/">Matt Wynne's Example Mapping</a>, an activity that's fantastic to perform at the Discovery stage of BDD, recommends examples have names that fit into the naming convention of an episode of Friends - "the one where...". I decided to tell a bit of a story - Freda's booked a transatlantic trip (how exotic) and received a business class upgrade on the return leg.</p>
<h2>Is the Background Necessary? No!</h2>
<p>A big difference between John's example and mine is that I've dropped the <code>Background</code> in favour of putting the data in the scenario. It's the reason mine is more concise.</p>
<p>John argues that the <code>Background</code> section is useful for establishing shared facts - his example clearly maps to "set up"-style methods common in automation tools. Feature files with lots of different scenarios benefit from the top-level shared context. Advocates of "don't repeat yourself" will be appealed.</p>
<p>However, Rose and Nagy argue against a <code>Background</code> section in most cases because:</p>
<blockquote>
<p>it has a negative effect on the readability of the scenarios. When reading a scenario, it is necessary to bear in mind what steps (if any) are included in the Background ... which is tedious and error-prone.</p>
</blockquote>
<p>It's hard to disagree with that. Higher cognitive load, higher maintenance burden, difficult to reason about the impact of a change. Don't like!</p>
<p>My perspective is that the <code>Given</code> step establishes the context. It mentions the persona, so I'd encode the automation tooling behind-the-scenes to set up the persona appropriately in the implementation of the <code>Given</code> step. The flights and the frequent flyer points live in the <code>Given</code> block's data table, too. Do the frequent flyer points make sense to be expressed at this level, or should they have remained in the <code>Background</code>? I can see arguments both ways here. What swung it for me is that in the real world, there'd be more than one scenario in this file. For that reason I didn't want to have a <code>Background</code> section containing a large mapping of journeys/classes -> points combinations. Each scenario should be distinct enough to encapsulate its own data. On balance, given the <code>Given</code> clause already has a data-table outlining the journey(s) under test, adding an extra row didn't feel too naughty.</p>
<h2>Have I Lost Something In Doing This?</h2>
<p>Ooh, good question. Note the focus of my test versus John's is actually slightly different. John's Scenario is that "Flights earn points based on distance travelled and cabin class", whereas my business rule is "Points allocated upon return flight completion". Perhaps John's example was written against a version of Gherkin that didn't support <code>Rule</code> and is therefore serving dual purposes of outlining a rule AND an example. But it does mean that the point John was making explicit has become implicit in mine. Maybe that's because the business logic for point calculation needs separated out <strong>entirely</strong> from the rules regarding <strong>when</strong> those points are allocated to a user.</p>
<p>One to chew on a bit more, I think. And a good example that we all come at these problems from different angles. Finding consensus and consistency is key!</p>
<h2>What About the Should?</h2>
<p><a href="/2016/03/should-you-use-should-in-tests/">A few years back I wrote about my dislike of the word should</a> in a test case. In short, "should" is different from "must" with regards to precision. So, in my example, I removed it in favour of a more active and direct expression.</p>
<p>If the test passes, it's because Freda has 1350 points. If it fails, it's because she doesn't.</p>
<p>Daniel Terhorst-North and Liz Keogh both disagree, advocating <em>for</em> the world "should". <a href="https://lizkeogh.com/2005/03/14/the-should-vs-will-debate-or-why-doubt-is-good/">The reasoning is compelling</a>, so read it for yourself. As John points out over on LinkedIn, it's usage in BDD is deliberate, as it's intended to make you question what's there:</p>
<blockquote>
<p>The word "should" has a long history in BDD, and the uncertainty it introduces is actually very deliberate: it is designed to make you think "should it?" rather than accepting a rule on face value.</p>
</blockquote>
<p>This is an important point and I don't disagree. My mind works the same. Where we differ is that I've found the "should it" discussion is more fruitful during discovery and early formulation. By the time the scenario is formulated and essentially ready to be committed to the repository - I'd prefer those "shoulds" to be "musts". Otherwise, there's more discovery to do! My experience is one of existing tests expressed as "shoulds" making me more uncertain about the <em>current state</em>, and therefore hinders my reasoning of the impact of my current change.</p>
<p>But, I'm used to being frequently and reliably told I'm wrong on this one!</p>
<h2>Golfin' Makes Me Feel Good</h2>
<p>It's a good to flex the muscles on stuff like this when not under the pressure of a deadline. It helps clarify thinking and gain some experience when the principles and the real world intersect. Whether you agree or disagree with my solution, or my reasoning, I don't mind. I'd love to see your take on this (please do get in touch!). If anything, I hope it helps you deepen your knowledge on how to write clear Gherkin.</p>
The Next Normal: A Micro-Talk2021-04-09T00:00:00Zhttps://samhogy.co.uk/2021/04/the-next-normal-a-microtalk/I'm on Youtube now!<p>Following on from my post last year about <a href="/2020/05/the-office-post-covid-19/">the office, post COVID-19</a>, I've recorded a micro-talk about what happens next.</p>
<div class="callout">
<p><a href="https://samhogy.co.uk/talks/the-next-normal.html">📹 Watch it here
<img src="https://samhogy.co.uk/assets/talks/the-next-normal.jpg" width="760" /></a></p>
</div>
<p>I decided to push myself out my comfort zone and develop a <a href="https://www.pechakucha.com/about">PechaKucha</a>-style talk: 20 slides, 20 seconds. I've found constraints inspire creativity and help focus the mind. All in, this creates about a six-minute talk. Long enough to be of substance, short enough to keep an attention span.</p>
<h2>Why I Made It</h2>
<p>In many respects, we're stuck in our ways. Working from home, even during the turmoil of the pandemic, has opened my eyes to different and more humane ways of working. That respects people's time, commitments, and outside-working life.</p>
<p>The UK government are considering the relaxation of restrictions, opening up the possibilities to return to the office. Senior government figures have also demonstrated their lack of understanding and empathy on this issue. <a href="https://www.theguardian.com/world/2021/mar/27/boris-johnson-branded-irresponsible-over-back-to-the-office-call">Boris Johnson suggested we've had enough "days off"</a> and <a href="https://news.sky.com/story/reopen-offices-after-lockdown-ends-or-your-staff-may-quit-rishi-sunak-warns-british-businesses-12256912">Rishi Sunak suggested staff would leave if they weren't given an office desk</a>. Both miss the point profoundly - that the open-plan office was a cost-saving exercise mired in the remnants of 20th Century workplace thinking.</p>
<p>Sure, we're two decades into the 21st Century now, but COVID has forced the issue of what modern working will look like. Now videoconferencing is normalised. Now the jobs that "could never" be done from home, for "security" or whatever other excuse was contrived at the time, are being done from home. Now that things begin to open up again and the kids are back at school. What <em>should</em> work look like?</p>
<p>I think this presents us an opportunity. This talk summarises where I am with my thinking on this.</p>
<h2>How I Made It</h2>
<p>For now, I'm still using PowerPoint for slides. So I created a template slide that allocated room for a webcam to be overlaid, and added a ticking timer bar. This time bar starts its animation immediately and lasts for 20 seconds. Then, the slide automatically advances. Duplicate it 19 times, you've got yourself a template for a micro-talk!</p>
<img src="https://samhogy.co.uk/assets/pechakucha-ppt.png" width="800" />
<p>Historically, I've preferred to riff from the slides over being particular in what I say. It makes each iteration of the talk unique and provides space for reading the room. But this isn't 'live' in that way. It's a one-and-done scenario, and you're up against the timer. So, I wrote a script. I did this on a slide by slide basis, using the constraints to clarify the thought I wanted to present on a slide. I rehearsed and refined until each slide had 17-20 seconds of content.</p>
<p>The recording was handled in <a href="https://obsproject.com/">OBS</a>, which was used successfully during last year's <a href="https://ne-rpc.co.uk">NE-RPC events</a>. It's simple to set up - all I needed to do was add the PowerPoint window, my mic and webcam, rearrange and crop to fit the template, and hit record.</p>
<img src="https://samhogy.co.uk/assets/pechakucha-obs.png" width="800" />
<p>I've a second monitor that's capable of producing 1920x1080 output, which I use to host the slideshow. My main, 37" curved monitor holds OBS and the Presenter View. To get the windows re-arranged quickly, I added a "Get Ready" slide. That meant I could hit record in OBS, start the show, and have a few seconds before the show begins in earnest. All that fluff gets removed in post, before uploading to YouTube.</p>
<p>Now I've essentially got a reusable template for running these talks, I'll think about doing more. Alongside blogs, of course.</p>
Making the Most of Contract Testing2021-04-02T00:00:00Zhttps://samhogy.co.uk/2021/04/making-the-most-of-contract-testing/What makes a good contract test?!<p>All the hard work in <a href="/2021/03/consumer-driven-contracts-with-pact/">setting up consumer and provider apps</a> with a <a href="/2021/03/a-consumer-driven-contract-workflow-on-gitlab/">fully-integrated CI/CD workflow</a> is now done. Hooray! Now though, it's time to turn our eyes towards good practice when it comes to writing contract tests for realsies.</p>
<p>📖 <a href="https://gitlab.com/my-pact-experiments">Click here for a fully-worked example of the code in this blog post</a></p>
<p>The contract that I generated for the provider's <code>/ping</code> endpoint - that it returns a hard-coded piece of text - is a bit lacking to put it politely. For demonstration purposes of setting up a workflow, it's sufficient. As a reminder, this is what it looks like:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"consumer"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"pact-consumer"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"provider"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"pact-provider"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"interactions"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"description"</span><span class="token operator">:</span> <span class="token string">"a request"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"providerState"</span><span class="token operator">:</span> <span class="token string">"ping"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"request"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"method"</span><span class="token operator">:</span> <span class="token string">"GET"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token string">"/ping"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"response"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"status"</span><span class="token operator">:</span> <span class="token number">200</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"headers"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"body"</span><span class="token operator">:</span> <span class="token string">"pong"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"metadata"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"pactSpecification"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"2.0.0"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Pretend you didn't see that, as we discuss how to make the most from contract testing.</p>
<h2>What Should Contracts Test?</h2>
<p>Contract testing and functional testing are distinct activities. Be careful about mixing them. Whenever a consumer writes a contract, it places <strong>constraints</strong> upon the provider's API. This isn't itself a bad thing - the purpose of a contract test is to encode the shared understanding about usage, so that an automated signal exists for breaking changes. The constraint is "don't break my usage". But this constraint, enforced too particularly, limits the ability for the API to evolve.</p>
<p>Contract testing veers into functional testing when the contracts end up expressing the business logic of a provider. Instead, contracts should be written from the perspective of the consumer: how is the API used? What must be constrained to prevent a breakage in the consumer application?</p>
<p>In the example above, the consumer app is calling the simple <code>/ping</code> endpoint and displaying the response text. Does it matter if that changes from 'pong' to something elese? As the consumer app currently uses it - not at all! The constraint can be loosened into a type - the contract expresses the response must be a string.</p>
<p>There will be times when an exact value will be necessary to assert, I'm sure of it. There may be times where a certain format is expected, such as a URL containing <code>http://</code> or <code>https://</code> (so from a consumer's perspective, their usage in an <code><a></code> tag works correctly). There will also be times where a particular structure to the response object needs to be specified in the contract.</p>
<p>Pact contract tests let contracts be expressed through:</p>
<ul>
<li>Exact values</li>
<li>Types (string, boolean, integer, uuid, and some built-in date-time matching!)</li>
<li>Regex</li>
<li>Object/array 'shapes'</li>
</ul>
<p>The exercise of <em>consumer-driven</em> contract tests is not for a consumer to <strong>dictate</strong> how an API behaves. It's to <em>have a conversation</em> and form the shared understanding. Constraints placed in a contract should be expressed as loosely as possible, whilst still giving convidence that the <strong>consumer</strong> use case is satisfied. Keep this principle in mind while developing contract tests to ensure the contracts don't become proxy-functional tests.</p>
<h2>Provider States</h2>
<p>Taking a step back, it's common to see a <code>Given / When / Then</code> pattern form in testing. Also expressed as <code>Arrange / Act / Assert</code>, the idea is that sometimes there's some pre-test setup that's needed to establish the context of the item under test. Contract tests are no different - think of applications that communicate with a database, or have user-driven workflows. When contract testing a particular API, it's useful to be able to request that the provider performs some setup code first.</p>
<p>This is where provider states are useful. The simple <code>/ping</code> example from earlier in the series did specify a provider state - but it was being ignored. So let's do something more interesting!</p>
<p>I'm creating a new API so that for a given user, it returns some favourite quotations from TV shows. The rules of the contract are that:</p>
<ul>
<li>When there's no user, return 404</li>
<li>When there's a specified user, return a list of quotations</li>
<li>Quotations need to have the quote text, an author and a source in the form of a URL.</li>
</ul>
<p>Why is there no need to contract test the empty list case, you may ask? Well, this is where it's worth recalling the line between contract and functional testing. For me, that leans closer to the functional testing side of things, because the contract around the 'shape' of the array satisfies the contract. So, I opted to omit this case.</p>
<h3>Consumer</h3>
<p>On the consumer side, I can express this as a contract test:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> AxiosError<span class="token punctuation">,</span> AxiosResponse <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"axios"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> pactWith <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'jest-pact'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> Matchers <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@pact-foundation/pact'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> QuoteService <span class="token keyword">from</span> <span class="token string">'./quoteService'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">pactWith</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">consumer</span><span class="token operator">:</span> <span class="token string">'pact-consumer'</span><span class="token punctuation">,</span> <span class="token literal-property property">provider</span><span class="token operator">:</span> <span class="token string">'pact-provider'</span><span class="token punctuation">,</span> <span class="token literal-property property">cors</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token parameter">provider</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">let</span> <span class="token literal-property property">quoteService</span> <span class="token operator">:</span> QuoteService<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">beforeEach</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> quoteService <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">QuoteService</span><span class="token punctuation">(</span>provider<span class="token punctuation">.</span>mockService<span class="token punctuation">.</span>baseUrl<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'Quotes API'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'not found when no quotes'</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">await</span> provider<span class="token punctuation">.</span><span class="token function">addInteraction</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">state</span><span class="token operator">:</span> <span class="token string">"no user"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">uponReceiving</span><span class="token operator">:</span> <span class="token string">"a request for quotes"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">withRequest</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">method</span><span class="token operator">:</span> <span class="token string">'GET'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">path</span><span class="token operator">:</span> <span class="token string">'/quotes'</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">willRespondWith</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">status</span><span class="token operator">:</span> <span class="token number">404</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">await</span> quoteService<span class="token punctuation">.</span><span class="token function">getAQuote</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>err<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>err<span class="token punctuation">.</span>response<span class="token punctuation">.</span>status<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token number">404</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">"receives quotes"</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">await</span> provider<span class="token punctuation">.</span><span class="token function">addInteraction</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">state</span><span class="token operator">:</span> <span class="token string">"a comedy-loving user"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">uponReceiving</span><span class="token operator">:</span> <span class="token string">'a request for quotes'</span><span class="token punctuation">,</span> </span>
<span class="highlight-line"> <span class="token literal-property property">withRequest</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">method</span><span class="token operator">:</span> <span class="token string">'GET'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">path</span><span class="token operator">:</span> <span class="token string">'/quotes'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">willRespondWith</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">status</span><span class="token operator">:</span> <span class="token number">200</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">body</span><span class="token operator">:</span> Matchers<span class="token punctuation">.</span><span class="token function">eachLike</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">quote</span><span class="token operator">:</span> <span class="token string">"I'm Pickle Rick!"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">author</span><span class="token operator">:</span> <span class="token string">"Rick Sanchez"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">source</span><span class="token operator">:</span> Matchers<span class="token punctuation">.</span><span class="token function">regex</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">matcher</span><span class="token operator">:</span> <span class="token string">"http(s)?:\/\/([^?#]*)"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">generate</span><span class="token operator">:</span> <span class="token string">"https://www.imdb.com/title/tt5218268/"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token literal-property property">response</span><span class="token operator">:</span> AxiosResponse <span class="token operator">=</span> <span class="token keyword">await</span> quoteService<span class="token punctuation">.</span><span class="token function">getAQuote</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>status<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token number">200</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>data<span class="token punctuation">.</span>length<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Here, I'm using a mix of Pact features. The source URL is specified using a (rather naive) regex - as it's important to the consumer app this link is fully-resolvable. Using <code>eachLike</code> is a handy convenience method for specifying an array of JSON objects, based on a sample object. This sample object uses strings for the <code>quote</code> and <code>author</code>, but unlike the earlier contract for <code>/ping</code>, these aren't absolute values. Rather, they're matching the <strong>type</strong> of string, but using the sample values as an example.</p>
<p>Recall I'm using the <code>jest-pact</code> library, which is a wrapper on <code>pact-js</code>. This means after each test I don't need to call <code>verify</code> myself. But, I do need to <code>await</code> the call to <code>provider.addInteraction</code>. As there's multiple of these now, I opted to move individual interactions into each test, rather than in the catch-all <code>beforeEach</code>.</p>
<p>Note that each test has a different <code>state</code> value. These are provider states which <strong>must</strong> be implemented on the provider side, during provider verification.</p>
<h3>Provider</h3>
<p>There's now a Plain Ol' Java Object to represent a quote:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">Quote</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token class-name">String</span> quote<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token class-name">String</span> author<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token class-name">String</span> source<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token class-name">Quote</span><span class="token punctuation">(</span><span class="token class-name">String</span> quote<span class="token punctuation">,</span> <span class="token class-name">String</span> author<span class="token punctuation">,</span> <span class="token class-name">String</span> source<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>quote <span class="token operator">=</span> quote<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>author <span class="token operator">=</span> author<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>source <span class="token operator">=</span> source<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token class-name">String</span> <span class="token function">getQuote</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> quote<span class="token punctuation">;</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token class-name">String</span> <span class="token function">getAuthor</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> author<span class="token punctuation">;</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token class-name">String</span> <span class="token function">getSource</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> source<span class="token punctuation">;</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Alongside a service class which has a pre-canned Quote response. As a <code>@Component</code>, it's injectable in the Spring Boot context and effectively made a singleton. This means that I can define a public method to modify the pre-canned response - which is what I'll be doing later during provider verification to satisfy the provider states.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@Component</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">QuoteService</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token class-name">List</span><span class="token generics"><span class="token punctuation"><</span><span class="token class-name">Quote</span><span class="token punctuation">></span></span> quotes <span class="token operator">=</span> <span class="token class-name">Arrays</span><span class="token punctuation">.</span><span class="token function">asList</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Quote</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token string">"You Come At The King, You Best Not Miss."</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">"Omar Little"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">"https://www.imdb.com/title/tt0749433/"</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token class-name">List</span><span class="token generics"><span class="token punctuation"><</span><span class="token class-name">Quote</span><span class="token punctuation">></span></span> <span class="token function">getQuotes</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> quotes<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setQuotes</span><span class="token punctuation">(</span><span class="token class-name">List</span><span class="token generics"><span class="token punctuation"><</span><span class="token class-name">Quote</span><span class="token punctuation">></span></span> quotes<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>quotes <span class="token operator">=</span> quotes<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Finally, the new <code>GET</code> endpoint for quotes. This uses Spring Boot dependency injection to <code>@Autowire</code> the singleton instance of <code>QuoteService</code>. Upon receiving a call to get quotes, it gets the list from the service. If it's empty a 404 error is returned. Otherwise, the list is returned as a JSON array. Sure, this isn't production-quality either, but it demonstrates the contract well enough.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@SpringBootApplication</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@RestController</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@CrossOrigin</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">DemoApplication</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@Autowired</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token class-name">QuoteService</span> quoteService<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@GetMapping</span><span class="token punctuation">(</span><span class="token string">"/quotes"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token class-name">List</span><span class="token generics"><span class="token punctuation"><</span><span class="token class-name">Quote</span><span class="token punctuation">></span></span> <span class="token function">quote</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">List</span><span class="token generics"><span class="token punctuation"><</span><span class="token class-name">Quote</span><span class="token punctuation">></span></span> quotes <span class="token operator">=</span> quoteService<span class="token punctuation">.</span><span class="token function">getQuotes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>quotes<span class="token punctuation">.</span><span class="token function">isEmpty</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">ResponseStatusException</span><span class="token punctuation">(</span><span class="token class-name">HttpStatus</span><span class="token punctuation">.</span><span class="token constant">NOT_FOUND</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> quotes<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Calling <code>localhost:8081/quotes</code> produces:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"quote"</span><span class="token operator">:</span> <span class="token string">"You Come At The King, You Best Not Miss."</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"author"</span><span class="token operator">:</span> <span class="token string">"Omar Little"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"source"</span><span class="token operator">:</span> <span class="token string">"https://www.imdb.com/title/tt0749433/"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">]</span></span></code></pre>
<p>Now, onto provider verification. Note that I'd previously mentioned the provider application performs two variants of provider verification: one using <code>pact-cli</code> (where the app is treat as a black box), another using <code>junit5-spring</code> (which uses JUnit and can interact with the Spring setup at test-time). This is complete overkill - choose one of them only. I only did this for completeness in an earlier post. But, I'll continue, and implement provider states in both ways.</p>
<h4>Provider Verification via pact-cli</h4>
<p>To the DemoApplication class, I added a POST endpoint to handle provider states. Provider verification, when performed using the pact-cli tool, posts a JSON object for each provider state it encounters:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span> consumer<span class="token operator">:</span> “consumer-name”<span class="token punctuation">,</span> state<span class="token operator">:</span> “state-name” <span class="token punctuation">}</span></span></code></pre>
<p>For convenience sake, I mapped a POJO object for this. For the provider states known about, the QuoteService’s quote list is modified:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@PostMapping</span><span class="token punctuation">(</span><span class="token string">"/provider-states"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">providerStates</span><span class="token punctuation">(</span><span class="token annotation punctuation">@RequestBody</span> <span class="token class-name">ProviderState</span> providerState<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token string">"pact-consumer"</span><span class="token punctuation">.</span><span class="token function">equals</span><span class="token punctuation">(</span>providerState<span class="token punctuation">.</span><span class="token function">getConsumer</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token string">"no user"</span><span class="token punctuation">.</span><span class="token function">equals</span><span class="token punctuation">(</span>providerState<span class="token punctuation">.</span><span class="token function">getState</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> quoteService<span class="token punctuation">.</span><span class="token function">setQuotes</span><span class="token punctuation">(</span><span class="token class-name">Collections</span><span class="token punctuation">.</span><span class="token function">emptyList</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token string">"a comedy-loving user"</span><span class="token punctuation">.</span><span class="token function">equals</span><span class="token punctuation">(</span>providerState<span class="token punctuation">.</span><span class="token function">getState</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> quoteService<span class="token punctuation">.</span><span class="token function">setQuotes</span><span class="token punctuation">(</span><span class="token class-name">Arrays</span><span class="token punctuation">.</span><span class="token function">asList</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Quote</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token string">"I don’t know how many years I got left on this planet, I’m going to get real weird with it."</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">"Frank Reynolds"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">"https://www.imdb.com/title/tt1504565/"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>When running <code>pact-cli verify</code>, there's an option to pass in</p>
<h4>Provider Verification via junit5-spring</h4>
<p>Curiously, I couldn’t find a way to get the <code>junit5-spring</code> library to work with a provider state url, so I ended up duplicating the provider state setup in the tests. As this is a Spring Boot test, the test class itself can utilise the DI context, so it’s possible to inject the same instance of <code>QuoteService</code> that the application receives, and configure it appropriately. Each provider state is given its own method to perform its setup, and annotated with <code>@State("<the state name>")</code> so that it is called at the appropriate time.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@Provider</span><span class="token punctuation">(</span><span class="token string">"pact-provider"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@PactBroker</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@SpringBootTest</span><span class="token punctuation">(</span>webEnvironment <span class="token operator">=</span> <span class="token class-name">WebEnvironment</span><span class="token punctuation">.</span><span class="token constant">RANDOM_PORT</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">ProviderVerificationTests</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token annotation punctuation">@LocalServerPort</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token keyword">int</span> port<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@Autowired</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token class-name">QuoteService</span> quoteService<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@BeforeEach</span></span>
<span class="highlight-line"> <span class="token keyword">void</span> <span class="token function">before</span><span class="token punctuation">(</span><span class="token class-name">PactVerificationContext</span> context<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> context<span class="token punctuation">.</span><span class="token function">setTarget</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">HttpTestTarget</span><span class="token punctuation">(</span><span class="token string">"localhost"</span><span class="token punctuation">,</span> port<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@TestTemplate</span></span>
<span class="highlight-line"> <span class="token annotation punctuation">@ExtendWith</span><span class="token punctuation">(</span><span class="token class-name">PactVerificationInvocationContextProvider</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token keyword">void</span> <span class="token function">pactVerificationTestTemplate</span><span class="token punctuation">(</span><span class="token class-name">PactVerificationContext</span> context<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> context<span class="token punctuation">.</span><span class="token function">verifyInteraction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@State</span><span class="token punctuation">(</span><span class="token string">"no user"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setupEmptyState</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> quoteService<span class="token punctuation">.</span><span class="token function">setQuotes</span><span class="token punctuation">(</span><span class="token class-name">Collections</span><span class="token punctuation">.</span><span class="token function">emptyList</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@State</span><span class="token punctuation">(</span><span class="token string">"a comedy-loving user"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setupUserState</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> quoteService<span class="token punctuation">.</span><span class="token function">setQuotes</span><span class="token punctuation">(</span><span class="token class-name">Arrays</span><span class="token punctuation">.</span><span class="token function">asList</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Quote</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token string">"I don’t know how many years I got left on this planet, I’m going to get real weird with it."</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">"Frank Reynolds"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">"https://www.imdb.com/title/tt1504565/"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h4>Test Output</h4>
<p>When running Provider verification, the output looks quite like the <code>Given / When / Then</code> format. This shows that it's useful to think of the provider state in that way. Don't overspecify the provider state - exactly how the provider wants to implement that is their business - but it forms part of the <strong>conversation</strong> and shared understanding between the consumer and provider. My provider state represents a use-case: "a comedy-loving user", rather than an overspecified "a user who has a Frank Reynolds quotation saved".</p>
<pre><code>Verifying a pact between pact-consumer and pact-provider
Given no user
a request for quotes
with GET /quotes
returns a response which
has status code 404
Given a comedy-loving user
a request for quotes
with GET /quotes
returns a response which
has status code 200
has a matching body
</code></pre>
<h2>Wrapping Up</h2>
<p>Well, this has been fun! Over four posts, I've built a full contract testing workflow using Pact Broker and Gitlab. It's been fun! I've learnt a lot. I hope you have too. Please get in touch (details in footer) if you'd like to discuss more in private, or have any suggestions for where to go next.</p>
<p>In my opinion, contract testing is an essential step in creating, maintaining and operating a distributed service-oriented architecture. Done well, it avoids the need for "big bang" deployments and costly phases of manual end-to-end testing. With integrations such as Pact Broker, it can also act as a vehicle for Continuous Delivery by letting individual services be deployed to an environment with surety they're compatible with the existing setup in that environment. Obviously, other contract testing techniques than Pact exist, but what I like especially about the Pactflow is how it's specifically designed to be an enabler of good practices.</p>
Addendum: Pact Provider Verification2021-03-13T00:00:00Zhttps://samhogy.co.uk/2021/03/addendum-pact-provider-verification/Performing provider verification locally against a remote Pact Broker!<p><a href="/2021/03/a-consumer-driven-contract-workflow-on-gitlab/">Previously</a>, I took provider and consumer applications and built a full CI/CD workflow, interacting with the Pact Broker.</p>
<p>Next on the list is to talk through contract testing best practices, introducing Provider States. But first, an addendum to the previous post. The changes to perform provider verification using the Pact Broker need to be ported back for when a developer wants to perform provider verification locally. Although there's only a need to do this in one way, in my <a href="/2021/03/consumer-driven-contracts-with-pact/">first post</a>, I demonstrated two solutions supported on the JVM.</p>
<p>In this brief post, I'll cover the changes I needed to interact with the remote Pact Broker.</p>
<p>First off, I deleted the Pact file I copied from the consumer side. Pacts will be retrieved instead from the Broker.</p>
<p>📖 <a href="https://gitlab.com/my-pact-experiments">Click here for a fully-worked example of the code in this blog post</a></p>
<h2>Docker-Compose</h2>
<p>Rather than hard-coding the Pact Broker Url and authentication token into a file that's committed to source-control, I defined a <code>.env</code> file within the repository, but added it to the project's <code>.gitignore</code>. This file is a simple key-value pairing of properties:</p>
<pre><code>PACT_BROKER_BASE_URL=https://url-to-pact-broker
PACT_BROKER_TOKEN=this-should-be-a-token
</code></pre>
<p><code>.env</code> files are read automatically during <code>docker-compose up</code> and are then available as environment variables.</p>
<p>Now, I need to run the <code>pact-cli</code> image, passing in the broker url, token and the consumer tag to use. Similar to in CI, I chose <code>master</code> to pull in the latest mainline consumer pact:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">version</span><span class="token punctuation">:</span> <span class="token string">"3"</span></span>
<span class="highlight-line"><span class="token key atrule">services</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">app</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span> pact<span class="token punctuation">-</span>provider<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">ports</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> <span class="token string">"8081:8081"</span></span>
<span class="highlight-line"> <span class="token key atrule">pact</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span> pactfoundation/pact<span class="token punctuation">-</span>cli<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">depends_on</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> app</span>
<span class="highlight-line"> <span class="token key atrule">links</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> app</span>
<span class="highlight-line"> <span class="token key atrule">volumes</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> ./src/test/resources/pacts<span class="token punctuation">:</span>/tmp/pacts</span>
<span class="token key atrule">command</span><span class="token punctuation">:</span> <span class="token punctuation">></span><span class="token scalar string">
<span class="highlight-line"> verify</span>
<span class="highlight-line"> --consumer-version-tag master</span>
<span class="highlight-line"> --provider pact-provider</span>
<span class="highlight-line"> --provider_base_url http://app:8081</span>
<span class="highlight-line"> --provider-states-setup-url http://app:8081/provider-states</span>
<span class="highlight-line"> --pact-broker-base-url ${PACT_BROKER_BASE_URL}</span>
<span class="highlight-line"> --broker-token ${PACT_BROKER_TOKEN}</span>
--wait 10</span></code></pre>
<h2>JUnit</h2>
<p>The alternative approach, using JUnit5 test templates, needs a change too. Rather than specifying a <code>@PactFolder</code>, I switch over to <code>@PactBroker</code>. Configuration options can be passed directly to this annotation, I don't want to hard-code any values containing secrets.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">package</span> <span class="token namespace">co<span class="token punctuation">.</span>uk<span class="token punctuation">.</span>samhogy<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junitsupport<span class="token punctuation">.</span>loader<span class="token punctuation">.</span></span><span class="token class-name">PactBroker</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token comment">// Other imports</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token annotation punctuation">@Provider</span><span class="token punctuation">(</span><span class="token string">"pact-provider"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@PactBroker</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@SpringBootTest</span><span class="token punctuation">(</span>webEnvironment <span class="token operator">=</span> <span class="token class-name">WebEnvironment</span><span class="token punctuation">.</span><span class="token constant">RANDOM_PORT</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">ProviderVerificationTests</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// Nothing changed in here </span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Instead, the Pact library will look for the following system properties and drive its configuration from there:</p>
<ul>
<li><code>pactbroker.scheme</code></li>
<li><code>pactbroker.host</code></li>
<li><code>pactbroker.port</code></li>
<li><code>pactbroker.auth.token</code></li>
<li><code>pactbroker.consumerversionselectors.tags</code></li>
</ul>
<p>So, I can pass these in as options when running a Maven build:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">./mvnw clean install \
<span class="highlight-line"> -Dpactbroker.scheme="https" \</span>
<span class="highlight-line"> -Dpactbroker.host="<broker-host>" \</span>
<span class="highlight-line"> -Dport="443" \</span>
<span class="highlight-line"> -Dpactbroker.auth.token="<token>" \</span>
<span class="highlight-line"> -Dpactbroker.consumerversionselectors.tags="master"</span>
</span></code></pre>
<p>The options here are similar to the <code>docker-compose</code> version. It uses the auth token and pulls contracts from consumers with the tag <code>master</code>.</p>
A Consumer-Driven Contract Workflow on GitLab2021-03-12T00:00:00Zhttps://samhogy.co.uk/2021/03/a-consumer-driven-contract-workflow-on-gitlab/GitLab CI/CD, using Pact Broker to identify whether it's safe to deploy!<p><a href="/2021/03/consumer-driven-contracts-with-pact/">Previously</a>, I built a React front-end application backed by a Spring Boot Java service to demonstrate consumer-driven contract testing with Pact. This was all running locally, of course, so now it's time to take this one step further and get it working on CI/CD.</p>
<p>My CI/CD provider is choice is Gitlab, although I expect the instructions will roughly translate to others. Regardless, the workflow is the same.</p>
<p>I initially wanted to focus on just setting up the basics in Gitlab, but quickly realised that demonstrating the full Pact-flow is far more interesting. So, this post details:</p>
<ul>
<li>Building the consumer and provider applications into deployables (Docker images)</li>
<li>Running all automated tests</li>
<li>Sharing consumer contracts to the Pact Broker</li>
<li>Provider verification, using these shared contracts</li>
<li>Tracking which version of an application (consumer or provider) is being deployed to the production environment</li>
<li>Performing a safe-to-deploy check prior to deployment</li>
</ul>
<p>📖 <a href="https://gitlab.com/my-pact-experiments">Click here for a fully-worked example of the code in this blog post</a></p>
<h2>The Pact Broker and Workflow</h2>
<p>Contracts on the consumer side need to be shared with the service provider. Copy-and-pasting isn't sufficient nor scalable. One could utilise git-enabled service workers to automatically commit updated contracts to a provider's repository. However, I personally find that using the Pact Broker service is far more scalable and brings additional benefits.</p>
<p>The Pact Broker is a service (hosted, or run your own) for storing and sharing contracts. It provides both a REST API and a command-line utility to publish contracts from a consumer build, retrieve contracts in a provider test run and then share these verification results. As a central store of contracts, it builds a <strong>matrix</strong> of compatible consumer-provider versions. Combined with tracking information about what versions are deployed to an environment, it can then provide deploy-time safety checks. It's the spider, at the middle of a web of services, keeping track of compatibility.</p>
<p>As mentioned, it's possible to host the Pact Broker yourself. For simplicity's sake, I signed up to the free tier on <a href="https://pactflow.io">Pactflow.io</a>, which takes care of hosting the Broker and supports up to 5 contracts.</p>
<p>The Pact Broker provides you with a read-only developer authentication token, and another read-write token for CI purposes. I extracted these to environment variables for both the consumer and provider CI/CD builds in Gitlab. I opted to make <code>PACT_BROKER_TOKEN</code> a masked variable, preventing the plain-text value being printed to logs. Neither variable is marked as a protected variable, as I want these to be available to all branches. In Gitlab terminology, a protected branch is one with additional constraints to prevent forced pushes and deletions.</p>
<img src="https://samhogy.co.uk/assets/pact/ci-variables.png" alt="Setting Pact Broker URL and authentication Token in Gitlab CI/CD" />
<h2>Consumer</h2>
<p>The pipeline has four stages:</p>
<ol>
<li><strong>Test</strong> - Run the automated test suite, including Pact consumer-driven contract tests</li>
<li><strong>Imaging</strong> - Generate a Docker image of the frontend app from a production build</li>
<li><strong>Publish Contracts</strong> - Push contracts to the Pact Broker</li>
<li><strong>Deployment</strong> - "Deploy" to a production environment</li>
</ol>
<img src="https://samhogy.co.uk/assets/pact/consumer-ci.png" alt="Visualisation of Consumer CI/CD Pipeline" />
<p>The test stage is simple, performing an install then kicking off Jest tests:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">test</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">stage</span><span class="token punctuation">:</span> test</span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span> node<span class="token punctuation">:</span><span class="token number">10</span></span>
<span class="highlight-line"> <span class="token key atrule">variables</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">CI</span><span class="token punctuation">:</span> <span class="token string">"true"</span></span>
<span class="highlight-line"> <span class="token key atrule">script</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> npm ci</span>
<span class="highlight-line"> <span class="token punctuation">-</span> npm test</span></code></pre>
<p>The imaging step (and this is common between both consumer and provider pipelines) utilises <a href="https://github.com/GoogleContainerTools/kaniko">Google's Kaniko builder</a>, to build Docker images from the repository's <code>Dockerfile</code> then push to Gitlab's project container registry. Kaniko is an excellent resource for daemonless Docker builds. And as Gitlab Runners are effectively Docker containers themselves running in Kubernetes, this is an ideal solution for Gitlab. The alternative is to use docker-in-docker.</p>
<p>Getting this to work requires a little jiggery-pokery in Gitlab. As each stage of a pipeline runs in a Docker container (with you specifying the image), any image that contains an <code>ENTRYPOINT</code> runs into trouble with Gitlab's <code>script</code> stage. To fix this I cleared the <code>entrypoint</code> by setting it to <code>[""]</code>.</p>
<p>Although the mechanics of the Kaniko build aren't relevant here, in a nutshell it copies some Gitlab environment variables to a config file, so that Kaniko can authenticate with the project's container registry. It builds from a Dockerfile and pushes the image with two tags - one being the commit hash, another a sanitised branch name:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">imaging</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">stage</span><span class="token punctuation">:</span> imaging</span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">name</span><span class="token punctuation">:</span> gcr.io/kaniko<span class="token punctuation">-</span>project/executor<span class="token punctuation">:</span>debug</span>
<span class="highlight-line"> <span class="token key atrule">entrypoint</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token key atrule">script</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> echo "<span class="token punctuation">{</span>\"auths\"<span class="token punctuation">:</span><span class="token punctuation">{</span>\"$CI_REGISTRY\"<span class="token punctuation">:</span><span class="token punctuation">{</span>\"username\"<span class="token punctuation">:</span>\"$CI_REGISTRY_USER\"<span class="token punctuation">,</span>\"password\"<span class="token punctuation">:</span>\"$CI_REGISTRY_PASSWORD\"<span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token punctuation">}</span>" <span class="token punctuation">></span> /kaniko/.docker/config.json</span>
<span class="highlight-line"> <span class="token punctuation">-</span> /kaniko/executor <span class="token punctuation">-</span><span class="token punctuation">-</span>context $CI_PROJECT_DIR <span class="token punctuation">-</span><span class="token punctuation">-</span>dockerfile $CI_PROJECT_DIR/Dockerfile <span class="token punctuation">-</span><span class="token punctuation">-</span>destination $CI_REGISTRY_IMAGE<span class="token punctuation">:</span>$CI_COMMIT_SHORT_SHA <span class="token punctuation">-</span><span class="token punctuation">-</span>destination $CI_REGISTRY_IMAGE<span class="token punctuation">:</span>$CI_COMMIT_REF_SLUG</span></code></pre>
<p>Once the image is built, I know I've got a <em>potentially</em> deployable unit of code. Now is the time to share any consumer contracts with the Broker, so that compatibility tests can be ran at deploy-time.</p>
<p>Pact's CLI tools are built into a Docker image, which is ideal for Gitlab. However, it also needs the <code>entrypoint</code> fix I used for Kaniko. A particular pecularity of this service! Using some pre-defined environment variables, I specified the consumer Pact version to be the Git commit hash and tag it with the branch it was built from. The Broker URL and Token are passed in too, from the variables I set up earlier.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">pact-publish</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">stage</span><span class="token punctuation">:</span> pact<span class="token punctuation">-</span>publish</span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">name</span><span class="token punctuation">:</span> pactfoundation/pact<span class="token punctuation">-</span>cli<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">entrypoint</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token key atrule">script</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> "pact publish pact/pacts </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>consumer<span class="token punctuation">-</span>app<span class="token punctuation">-</span>version=$CI_COMMIT_SHORT_SHA </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>tag=$CI_COMMIT_REF_NAME </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>base<span class="token punctuation">-</span>url=$PACT_BROKER_BASE_URL </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>token=$PACT_BROKER_TOKEN"</span></code></pre>
<p>So far, running this pipeline builds a deployable and publishes the contract to the Pact Broker. The contract is visible in the Broker UI, tagged and versioned appropriately. But notice it's listed as "unverified". Before I can think about deployment, I need to move over to the provider side pipeline.</p>
<img src="https://samhogy.co.uk/assets/pact/pact-broker-unverified.png" alt="The published contract, marked as unverified in Pact Broker" />
<h2>Provider</h2>
<p>The pipeline has three stages:</p>
<ol>
<li><strong>Imaging</strong> - Same as the consumer side</li>
<li><strong>Pact Provider Verification</strong> - Verify the mainline consumer contract against this build</li>
<li><strong>Deployment</strong> - Same as the consumer side</li>
</ol>
<img src="https://samhogy.co.uk/assets/pact/provider-ci.png" alt="Visualisation of Provider CI/CD Pipeline" />
<p>Imaging is performed with Kaniko, exactly the same as on the consumer pipeline.</p>
<p>The Pact Provider Verification step looks remarkably similar to the docker-compose solution from the previous post, except now that it's querying the Pact Broker, there's some more options passed in. For a general build of the provider, I want to verify the latest consumer contract available. To do that, I look for the <code>master</code> tag. Additionally, passing in <code>--publish-verification-results</code> will post back the result of this verification to the Pact Broker.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">pact-provider-verification</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">stage</span><span class="token punctuation">:</span> pact<span class="token punctuation">-</span>provider<span class="token punctuation">-</span>verification</span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">name</span><span class="token punctuation">:</span> pactfoundation/pact<span class="token punctuation">-</span>cli<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">entrypoint</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token key atrule">services</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>CI_REGISTRY_IMAGE<span class="token punctuation">}</span><span class="token punctuation">:</span>$<span class="token punctuation">{</span>CI_COMMIT_SHORT_SHA<span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token key atrule">alias</span><span class="token punctuation">:</span> app</span>
<span class="highlight-line"> <span class="token key atrule">script</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> "pact verify </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>consumer<span class="token punctuation">-</span>version<span class="token punctuation">-</span>tag=master </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider=pact<span class="token punctuation">-</span>provider </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider<span class="token punctuation">-</span>app<span class="token punctuation">-</span>version=$CI_COMMIT_SHORT_SHA </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider<span class="token punctuation">-</span>version<span class="token punctuation">-</span>tag=$CI_COMMIT_REF_NAME </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider_base_url=http<span class="token punctuation">:</span>//app<span class="token punctuation">:</span><span class="token number">8081</span> </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider<span class="token punctuation">-</span>states<span class="token punctuation">-</span>setup<span class="token punctuation">-</span>url=http<span class="token punctuation">:</span>//app<span class="token punctuation">:</span>8081/provider<span class="token punctuation">-</span>states </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>pact<span class="token punctuation">-</span>broker<span class="token punctuation">-</span>base<span class="token punctuation">-</span>url=$PACT_BROKER_BASE_URL </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>token=$PACT_BROKER_TOKEN </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>wait=10 </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>publish<span class="token punctuation">-</span>verification<span class="token punctuation">-</span>results"</span></code></pre>
<p>Now for my simple contract file, running this pipeline produces a verified Pact in the Broker:</p>
<img src="https://samhogy.co.uk/assets/pact/pact-broker-verified.png" alt="The published contract, now marked as verified in Pact Broker" />
<p>From here, it's possible to view the Matrix of pacts in the Broker. The Broker uses a combination of version numbers, tags and file hashes to understand when a contract has <em>actually</em> changed, and conversely when it's unchanged. This is useful because the contracts aren't going to change every single commit. This prevents a n^2 explosion of provider verification tests!</p>
<p>The Matrix view is essentially the visualisation of verified, unverified and incompatible contracts, which can be queried to perform the "can I deploy" check when it's time to deploy a service.</p>
<img src="https://samhogy.co.uk/assets/pact/pact-broker-matrix.png" alt="A matrix visualisation of verified, compatible consumer and provider versions" />
<p>Before moving onto deployment, there's one remaining puzzle to solve...</p>
<h2>Webhooks for Provider Verification</h2>
<p>What happens when a consumer contract is generated (or updated) and published to the Pact Broker? Does verification of this have to wait until the next provider CI build? Nope! What should happen is that a special provider verification build is triggered on the provider-side. It doesn't need to rebuild the entire application - but it does need to verify the new contract and publish the results.</p>
<p>The Pact Broker can use webhooks to trigger notifications to other systems when a particular event occurs. <a href="https://docs.gitlab.com/ee/ci/triggers/#adding-a-new-trigger">Gitlab pipelines can also be triggered over HTTP</a>. Putting the two together - a webhook in the Pact Broker triggers a pipeline build in the provider repository. First, I generated an API token from the provider's CI/CD Settings page:</p>
<img src="https://samhogy.co.uk/assets/pact/gitlab-ci-trigger.png" alt="Gitlab CI/CD Trigger creation" />
<p>The API token is a secret (no, you can't have it) and I don't want this to be stored in plain-text anywhere. In the Pact Broker, I add this as a secret value with the name <code>ProviderVerificationToken</code>:</p>
<img src="https://samhogy.co.uk/assets/pact/pact-broker-secret.png" alt="Defining the trigger token as a Pact Broker secret" />
<p>When creating the webhook to trigger provider verification, I can reference this secret with <code>${user.ProviderVerificationToken}</code>. This won't be printed to the logs!</p>
<p>I configured the broker to fire the URL supplied by Gitlab when creating the trigger, but with the ref <code>master</code>. This is because I always want provider verification to be performed against the latest mainline branch of the provider application. Additionally, it's possible to pass additional variables to a Gitlab pipeline via the URL. I passed the consumer tag from the updated consumer contract, so that the triggered pipeline knows what contract to verify:</p>
<img src="https://samhogy.co.uk/assets/pact/pact-broker-webhook.png" alt="Creating a Pact Broker webhook for contract changes" />
<p>The full URL takes the form:</p>
<p><code>https://gitlab.com/api/v4/projects/<project-id>/ref/master/trigger/pipeline?token=${user.ProviderVerificationToken}&variables[PACT_CONSUMER_TAG]=${pactbroker.consumerVersionTags}</code></p>
<p>A full set of Webhooks and available variables are listed <a href="https://docs.pact.io/pact_broker/advanced_topics/api_docs/webhooks">here</a>, opening up a wide range of possible integrations with your own toolchain.</p>
<p>The job to trigger is similar, but different, to the standard provider verification job. The only differences are that the provider app uses the latest image built on <code>master</code>, and the consumer tag is supplied by the incoming variable. Gitlab exposes this as the <code>PACT_CONSUMER_TAG</code> variable. Provider verification is configured to run against the latest mainline image, which has the tag <code>master</code>. To mark it as a job that should execute only from the trigger and not part of a regular CI build, I used the <code>only</code> expression:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">pact-new-contract-verification</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">stage</span><span class="token punctuation">:</span> pact<span class="token punctuation">-</span>provider<span class="token punctuation">-</span>verification</span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">name</span><span class="token punctuation">:</span> pactfoundation/pact<span class="token punctuation">-</span>cli<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">entrypoint</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token key atrule">services</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>CI_REGISTRY_IMAGE<span class="token punctuation">}</span><span class="token punctuation">:</span>master</span>
<span class="highlight-line"> <span class="token key atrule">alias</span><span class="token punctuation">:</span> app</span>
<span class="highlight-line"> <span class="token key atrule">script</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> "pact verify </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>consumer<span class="token punctuation">-</span>version<span class="token punctuation">-</span>tag=$PACT_CONSUMER_TAG </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider=pact<span class="token punctuation">-</span>provider </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider<span class="token punctuation">-</span>app<span class="token punctuation">-</span>version=$CI_COMMIT_SHORT_SHA </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider<span class="token punctuation">-</span>version<span class="token punctuation">-</span>tag=master </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider_base_url=http<span class="token punctuation">:</span>//app<span class="token punctuation">:</span><span class="token number">8081</span> </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>provider<span class="token punctuation">-</span>states<span class="token punctuation">-</span>setup<span class="token punctuation">-</span>url=http<span class="token punctuation">:</span>//app<span class="token punctuation">:</span>8081/provider<span class="token punctuation">-</span>states </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>pact<span class="token punctuation">-</span>broker<span class="token punctuation">-</span>base<span class="token punctuation">-</span>url=$PACT_BROKER_BASE_URL </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>token=$PACT_BROKER_TOKEN </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>wait=10 </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>publish<span class="token punctuation">-</span>verification<span class="token punctuation">-</span>results"</span>
<span class="highlight-line"> <span class="token key atrule">only</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> triggers</span></code></pre>
<p>Conversely, to prevent the trigger performing every other job in the pipeline, I added, where necessary:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">except</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> triggers</span></code></pre>
<p>To test this, I made a breaking change to the contract on the consumer-side in a new branch, by expecting a different response to <code>/ping</code>:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line">provider.addInteraction(<span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">state</span><span class="token punctuation">:</span> <span class="token string">"ping"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">uponReceiving</span><span class="token punctuation">:</span> <span class="token string">'a request'</span><span class="token punctuation">,</span> </span>
<span class="highlight-line"> <span class="token key atrule">withRequest</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">method</span><span class="token punctuation">:</span> <span class="token string">'GET'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">path</span><span class="token punctuation">:</span> <span class="token string">'/ping'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">willRespondWith</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">status</span><span class="token punctuation">:</span> <span class="token number">200</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">body</span><span class="token punctuation">:</span> <span class="token string">"pong!!!!"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"><span class="token punctuation">}</span>)</span></code></pre>
<p>Pushing this shares a new contract with the broker, and then initiates the provider-verification trigger. Naturally, the build fails, now that the contract expects <code>pong!!!</code> instead of <code>pong</code>.</p>
<img src="https://samhogy.co.uk/assets/pact/failed-provider-verification-gitlab.png" alt="A failing provider verification job on Gitlab, triggered by a Webhook" />
<p>This provider verification failure is also visible in the Pact Broker. If it ever came to pass that this consumer version would attempted to be deployed, the deploy-time safety check will (correctly) fail.</p>
<img src="https://samhogy.co.uk/assets/pact/failed-provider-verification-broker.png" alt="A failing provider verification result in the Pact Broker" />
<p>Ok, now onto those deployment checks!</p>
<h2>Deployment</h2>
<p>The deployment steps in both the consumer and provider pipelines are exactly the same. Conceptually, they have three sub-steps:</p>
<ol>
<li>Perform a deploy-time safety check? This is provided by Pact-Broker's <code>can-i-deploy</code> functionality.</li>
<li>Do the deployment. For simplicity's sake, this is a simple <code>echo</code> of the commit hash.</li>
<li>Tag the deployment as successful in the Pact Broker, with <code>create-version-tag</code>.</li>
</ol>
<p>These all execute within the <code>pact-cli</code> image, because my deployment is simply an <code>echo</code>. This may need to be split into three separate jobs in a more complex setup.</p>
<p>Post-deployment, the contract version is marked as being in production by adding a new tag: <code>production</code>. This is used as part of the <code>can-i-deploy</code> check. Different tags can be used for different environments, such as development and testing.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">deploy</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">stage</span><span class="token punctuation">:</span> deploy</span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">name</span><span class="token punctuation">:</span> pactfoundation/pact<span class="token punctuation">-</span>cli<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">entrypoint</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">""</span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token key atrule">when</span><span class="token punctuation">:</span> manual</span>
<span class="highlight-line"> <span class="token key atrule">environment</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">name</span><span class="token punctuation">:</span> production</span>
<span class="highlight-line"> <span class="token key atrule">only</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">refs</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> master</span>
<span class="highlight-line"> <span class="token key atrule">except</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> triggers</span>
<span class="highlight-line"> <span class="token key atrule">script</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> "pact broker can<span class="token punctuation">-</span>i<span class="token punctuation">-</span>deploy </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>pacticipant=pact<span class="token punctuation">-</span>consumer </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>version=$CI_COMMIT_SHORT_SHA </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>to=production </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>base<span class="token punctuation">-</span>url=$PACT_BROKER_BASE_URL </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>token=$PACT_BROKER_TOKEN"</span>
<span class="highlight-line"> <span class="token punctuation">-</span> echo "Deploying $<span class="token punctuation">{</span>CI_COMMIT_SHORT_SHA<span class="token punctuation">}</span> to production<span class="token tag">!</span>"</span>
<span class="highlight-line"> <span class="token punctuation">-</span> "pact broker create<span class="token punctuation">-</span>version<span class="token punctuation">-</span>tag </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>pacticipant=pact<span class="token punctuation">-</span>consumer</span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>version=$CI_COMMIT_SHORT_SHA </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>tag=production </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>base<span class="token punctuation">-</span>url=$PACT_BROKER_BASE_URL </span>
<span class="highlight-line"> <span class="token punctuation">-</span><span class="token punctuation">-</span>broker<span class="token punctuation">-</span>token=$PACT_BROKER_TOKEN"</span></code></pre>
<h2>Next Steps</h2>
<p>Returning to the task list from the last post, I've covered off quite a bit here:</p>
<ul>
<li>✔️ Get this working on Gitlab CI</li>
<li>Create a more complex interaction and utilise provider states</li>
<li>✔️ Share Pactfiles between the consumer and provider in a better way than copy/paste</li>
<li>✔️ Demonstrate how contracts can be used to perform deploy-time safety checks</li>
</ul>
<p>So now both applications have a working CI/CD integration with the Pact Broker, it's time tofinish off this series by looking at how to write good Pactfiles and make the most of contract testing. I'll also look into how provider states can be used to isolate behaviour between interactions. Stay tuned!</p>
Consumer-Driven Contracts with Pact2021-03-05T00:00:00Zhttps://samhogy.co.uk/2021/03/consumer-driven-contracts-with-pact/Setting up an API consumer and provider, and contract testing them!<p>Welcome to a new post series! Here, I'm going to investigate building a CD pipeline for a system using consumer-driven contract testing, with Pact (and Pact Broker).</p>
<p>This first post introduces the theory of contract testing, then proceeds to set up a Java Spring Boot API provider app, a React consumer app, generates a consumer-driven contract and then performs provider verification.</p>
<p>📖 <a href="https://gitlab.com/my-pact-experiments">Click here for a fully-worked example of the code in this blog post</a></p>
<h2>Why Do Consumer-Driven Contract Testing?</h2>
<p>A single component rarely does it all itself - normally relying on other components for service functionality and data. Traditionally, testing this will involve deploying all the components and performing "end-to-end" verification of user workflows, checking that the system works in its integration. As appealing as this sounds, it's difficult to get right. End-to-end tests have a habit of being brittle, difficult to maintain and time-consuming, due to the complexities of setting up and then observing behaviour in unpredictable environments.</p>
<p>Larger systems will have components developed and maintained by many different teams, adding to the challenge of getting everything into working order. Without supporting processes to give teams the confidence they can deploy their piece of the whole without breaking anything, practices develop which make the overall system more rigid: heavy focus on end-to-end testing, co-ordination of deployments, big-bang releases. Bottlenecks, bureaucracy and delays to feedback.</p>
<p>Even with highly automated environment provisioning and observability tooling, teams require faster feedback than the above situation can provide. To shift testing "left" is to query whether the assurances sought from a particular stage of testing can be done earlier in the development process. Do those end-to-end tests <em>need</em> to be done as end-to-end tests? Or could the same failure signal be accurately derived in a simpler environment, that provides <strong>faster feedback</strong> and reduces the need for rework?</p>
<img src="https://samhogy.co.uk/assets/pact/web-of-services.png" title="Can I deploy a new version of a service?" />
<p>Enter <a href="https://pact.io/">Pact</a>: a programming language-agnostic, consumer-driven contract testing tool. Its purpose is to answer the question "can I safely deploy my service"? It can answer this through <strong>contracts</strong> - a relationship between a service provider and a consumer, from the consumer's perspective. The consumer expresses its expectations: for a given requet, what's form should a response take, what constraints must be upheld, and what's used?</p>
<p>These contracts are shared with the provider. They shift testing left by:</p>
<ul>
<li>Giving a provider a means to test whether it still fulfills all its contractual obligations, at build-time, in isolation from other services</li>
<li>Supplying deploy-time assurances that a new component can be safely released.</li>
</ul>
<p>Rather than replacing the entire web of deployments and performing end-to-end tests, it's possible to switch out individual components independently and safely. If the contracts are satisfied, the components are compatible.</p>
<img src="https://samhogy.co.uk/assets/pact/new-deployment.png" title="Does the new service satisfy the existing contracts?" />
<p>Pact doesn't remove the need for <em>all</em> end-to-end testing. But combined with a good set of acceptance tests that cover the functional behaviour of a component in isolation, it gets you most of the way there.
Of course, there are still operational concerns that Pact doesn't concern itself with. For example, blue/green deployments or staggered rollouts. It can certainly provide the initial compatibility sanity check prior to a deployment. You may still wish to run some test transactions through the integrated system and observe the trace from source to sink(s). The important thing is that end-to-end tests are used where they're most appropriate!</p>
<h2>A Worked Example</h2>
<p>Time to get started! Firstly, I will set up a Spring Boot application to act as a provider for an API. This API will be consumed by a React application. Then, I'll generate the contract on the consumer side as part of an automated test run, supply it to the provider, and perform provider verification testing of that contract.</p>
<h3>Provider-Side API</h3>
<p>I used the <a href="https://start.spring.io/">Spring Boot Initializer</a> to create a Spring Boot app. Adding the Spring Web dependency pulls in <code>spring-boot-starter-web</code> alongside <code>spring-boot-starter-test</code>, which gives me enough to get started with a simple REST API.</p>
<p>When developing API services I normally add a simple 'ping' API call, that in the early days can serve as a proto-healthcheck endpoint. Call <code>/ping</code>, get "pong" back.</p>
<p>The Spring Boot starter kit unzips to contain a single class file, which kickstarts the application. For simplicity's sake, I annotated this with <code>@RestController</code> so I can define the endpoint as part of this class:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@SpringBootApplication</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@RestController</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@CrossOrigin</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">DemoApplication</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@GetMapping</span><span class="token punctuation">(</span><span class="token string">"/ping"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token function">home</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token string">"pong"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">static</span> <span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token class-name">String</span><span class="token punctuation">[</span><span class="token punctuation">]</span> args<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">SpringApplication</span><span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token class-name">DemoApplication</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">,</span> args<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Now, this application will build: <code>mvn clean install</code>. run: <code>java -jar target/provider-0.0.1-SNAPSHOT.jar</code> and serve on <code>http://localhost:8081/ping</code>. Fantastic!</p>
<h3>Consumer-Side Setup</h3>
<p>I generated a React-based web application using Create React App. I used the Typescript template for additional developer-time type information:</p>
<p><code>npx create-react-app pact-consumer --template typescript</code></p>
<p>I defined a class that calls the <code>/ping</code> endpoint on the provider, using the XHR library of the day: <code>npm install --save axios @types/axios</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> axios<span class="token punctuation">,</span> <span class="token punctuation">{</span> AxiosResponse <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"axios"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">PingService</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">readonly baseUrl<span class="token operator">:</span> string</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">getPing</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">:</span> Promise<span class="token operator"><</span>AxiosResponse<span class="token operator">></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> axios<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span><span class="token keyword">this</span><span class="token punctuation">.</span>baseUrl<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">/ping</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">responseType</span><span class="token operator">:</span> <span class="token string">'text'</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> PingService<span class="token punctuation">;</span></span></code></pre>
<p>And I then hooked this into the main React component, so that "pong" is displayed on screen following a load:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">import</span> <span class="token class-name">React</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> useState<span class="token punctuation">,</span> useEffect <span class="token punctuation">}</span> from <span class="token char">'react'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token namespace">logo</span> from '<span class="token punctuation">.</span>/logo<span class="token punctuation">.</span>svg'<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> '<span class="token punctuation">.</span>/<span class="token class-name">App</span><span class="token punctuation">.</span>css'<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token class-name">PingService</span> from '<span class="token punctuation">.</span>/pingService'<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">function <span class="token class-name">App</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token punctuation">[</span>message<span class="token punctuation">,</span> setMessage<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">useState</span><span class="token punctuation">(</span><span class="token string">""</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">useEffect</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">PingService</span><span class="token punctuation">(</span>'http<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span>localhost<span class="token operator">:</span><span class="token number">8081</span>'<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">getPing</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span>r <span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">setMessage</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>data<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token operator"><</span>div className<span class="token operator">=</span><span class="token string">"App"</span><span class="token operator">></span></span>
<span class="highlight-line"> <span class="token operator"><</span>header className<span class="token operator">=</span><span class="token string">"App-header"</span><span class="token operator">></span></span>
<span class="highlight-line"> <span class="token operator"><</span>img src<span class="token operator">=</span><span class="token punctuation">{</span>logo<span class="token punctuation">}</span> className<span class="token operator">=</span><span class="token string">"App-logo"</span> alt<span class="token operator">=</span><span class="token string">"logo"</span> <span class="token operator">/</span><span class="token operator">></span></span>
<span class="highlight-line"> <span class="token generics"><span class="token punctuation"><</span>p<span class="token punctuation">></span></span><span class="token punctuation">{</span>message<span class="token punctuation">}</span><span class="token operator"><</span><span class="token operator">/</span>p<span class="token operator">></span></span>
<span class="highlight-line"> <span class="token operator"><</span><span class="token operator">/</span>header<span class="token operator">></span></span>
<span class="highlight-line"> <span class="token operator"><</span><span class="token operator">/</span>div<span class="token operator">></span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">export <span class="token keyword">default</span> <span class="token class-name">App</span><span class="token punctuation">;</span></span></code></pre>
<img src="https://samhogy.co.uk/assets/pact/cra-pong.png" alt="Create React App, displaying text from API response" />
<h3>Defining a Consumer-Driven Pact</h3>
<p>As discussed, it's the responsibility of the consumer to define its interactions with and expectations of a service provider.</p>
<p>With Pact, an API request and response expectation is encoded as a contract, known as a Pactfile. The process for generating these is simple, and should run as part of the consumer's automated test suite.</p>
<img src="https://samhogy.co.uk/assets/pact/mock-provider.png" alt="Consumer makes API requests to mock provider" />
<p>A Pact test sets up a mock server, then specifies that for a given request, a particular response should be sent back. The test will then execute consumer code to make this API call and handle the mocked response. The overall interaction is written to a Pactfile, which is then committed to source control and shared with the provider. The test may then perform additional assertions, such as checking the consumer code handles the request correctly.</p>
<p>To set up a Pact mock provider service, I pulled in the following dependencies:</p>
<p><code>npm install --save @pact-foundation/pact @pact-foundation/pact-node jest-pact</code></p>
<p>Let's look at what this interaction looks like, as captured in a Jest test:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> AxiosResponse <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">"axios"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> pactWith <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'jest-pact'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> PingService <span class="token keyword">from</span> <span class="token string">'./pingService'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">pactWith</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">consumer</span><span class="token operator">:</span> <span class="token string">'pact-consumer'</span><span class="token punctuation">,</span> <span class="token literal-property property">provider</span><span class="token operator">:</span> <span class="token string">'pact-provider'</span><span class="token punctuation">,</span> <span class="token literal-property property">cors</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token parameter">provider</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">let</span> <span class="token literal-property property">pingService</span> <span class="token operator">:</span> PingService<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">beforeEach</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> pingService <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">PingService</span><span class="token punctuation">(</span>provider<span class="token punctuation">.</span>mockService<span class="token punctuation">.</span>baseUrl<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'ping API'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">beforeEach</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> provider<span class="token punctuation">.</span><span class="token function">addInteraction</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">state</span><span class="token operator">:</span> <span class="token string">"ping"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">uponReceiving</span><span class="token operator">:</span> <span class="token string">'a request'</span><span class="token punctuation">,</span> </span>
<span class="highlight-line"> <span class="token literal-property property">withRequest</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">method</span><span class="token operator">:</span> <span class="token string">'GET'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">path</span><span class="token operator">:</span> <span class="token string">'/ping'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">willRespondWith</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">status</span><span class="token operator">:</span> <span class="token number">200</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">body</span><span class="token operator">:</span> <span class="token string">"pong"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">"sends a request"</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token literal-property property">response</span><span class="token operator">:</span> Response <span class="token operator">=</span> <span class="token keyword">await</span> pingService<span class="token punctuation">.</span><span class="token function">getPing</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">await</span> <span class="token function">expect</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>status<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token number">200</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Note that I had to add the <code>cors: true</code> option to get Axios to play nicely with access to the Pact mock provider server, running on a random <code>localhost</code> port.</p>
<p>Using a neat mini-DSL expression, an interaction is defined. The request (verb and path) alongside the response (status and body text) are specified. Notice there's also a <code>state</code> field - this will be useful in future when we touch on provider states.</p>
<p>On running the tests with <code>npm test</code>, the contract is outputted to a <code>pact</code> directory within the consumer project.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"consumer"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"name"</span><span class="token punctuation">:</span> <span class="token string">"pact-consumer"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"provider"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"name"</span><span class="token punctuation">:</span> <span class="token string">"pact-provider"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"interactions"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"description"</span><span class="token punctuation">:</span> <span class="token string">"a request"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"providerState"</span><span class="token punctuation">:</span> <span class="token string">"ping"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"request"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"method"</span><span class="token punctuation">:</span> <span class="token string">"GET"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"path"</span><span class="token punctuation">:</span> <span class="token string">"/ping"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"response"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"status"</span><span class="token punctuation">:</span> <span class="token number">200</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"headers"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"body"</span><span class="token punctuation">:</span> <span class="token string">"pong"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token key atrule">"metadata"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"pactSpecification"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token key atrule">"version"</span><span class="token punctuation">:</span> <span class="token string">"2.0.0"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Changes to a Pactfile then form part of code review on the consumer side to ensure that interactions have been correctly captured. I'll cover in a future post how to make the most of a contract test - in general, aim to express only what's needed and what's used on the consumer side. Don't fall into the trap of testing business rules in the contract.</p>
<h3>Provider Verification</h3>
<p>Finally, over to the provider side. In an ideal world, I'd want a mechanism that facilitates the sharing of Pactfiles between consumer and provider automatically. This is crucial if there are separate teams involved in the consumer/provider divide, but also reduces operational overhead. I shall address this properly later in the series. For now, I'm simply going to copy/paste the Pactfile over into the Provider's source code. As it's a testing-time resource, Java convention dictates that <code>src/test/resources</code> is an appropriate location for this file to live.</p>
<p>Pact provider verification sets up a mock consumer to run against the provider. This is essentially the inverse of the consumer-side testing above. Consumer-side, a mock provider records interactions against it and outputs a Pactfile. On the provider-side, a mock consumer takes a Pactfile as an input and uses this to make API calls to the Provider, ensuring those responses conform to the contract.</p>
<p>If, for example, the above service replied with "ping" instead of "pong" - this would violate the contract!</p>
<img src="https://samhogy.co.uk/assets/pact/mock-consumer.png" alt="Mock consumer uses contracts to make API calls to provider" />
<p>There are two ways of performing provider verification. Both are at automated testing time, but the difference is in how they are triggered. For demonstration purposes, I'm going to show both. They are:</p>
<ol>
<li>Use the pact-cli to set up a mock consumer</li>
<li>(If supported by your programming language) Incorporate a library to set up a mock consumer as part of the test suite.</li>
</ol>
<p>Firstly, to keep everything ticking over, I need a small change to cope with provider states. As this is a simple API service with a hard-coded response, I don't yet need to think provider states other than to define a no-op POST endpoint. To the singular <code>DemoApplication</code> class, I've added:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@PostMapping</span><span class="token punctuation">(</span><span class="token string">"/provider-states"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">providerStates</span><span class="token punctuation">(</span><span class="token annotation punctuation">@RequestBody</span> <span class="token class-name">Map</span> body<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">System</span><span class="token punctuation">.</span>out<span class="token punctuation">.</span><span class="token function">println</span><span class="token punctuation">(</span>body<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">"state"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>As more complexity is added to the application, it will become necessary to setup the provider to behave differently based on incoming pacts. Provider State verification is Pact's mechanism for controlling this whilst maintaining test isolation. I'll touch on this more in a later article.</p>
<p>Now, let's cover those two options.</p>
<h4>Dockerised, with Pact-CLI</h4>
<p>The generic way to run Pact provider verification is using the CLI tool. I opted to Dockerize the Spring Boot application, so that it can be packaged and run alongside the pact-cli tool in a docker-compose setup.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="highlight-line"><span class="token key atrule">version</span><span class="token punctuation">:</span> <span class="token string">"3"</span></span>
<span class="highlight-line"><span class="token key atrule">services</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">app</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span> pact<span class="token punctuation">-</span>provider<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">ports</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> <span class="token string">"8081:8081"</span></span>
<span class="highlight-line"> <span class="token key atrule">pact</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token key atrule">image</span><span class="token punctuation">:</span> pactfoundation/pact<span class="token punctuation">-</span>cli<span class="token punctuation">:</span>latest</span>
<span class="highlight-line"> <span class="token key atrule">depends_on</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> app</span>
<span class="highlight-line"> <span class="token key atrule">links</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> app</span>
<span class="highlight-line"> <span class="token key atrule">volumes</span><span class="token punctuation">:</span></span>
<span class="highlight-line"> <span class="token punctuation">-</span> ./src/test/resources/pacts<span class="token punctuation">:</span>/tmp/pacts</span>
<span class="token key atrule">command</span><span class="token punctuation">:</span> <span class="token punctuation">></span><span class="token scalar string">
<span class="highlight-line"> verify</span>
<span class="highlight-line"> --provider_base_url http://app:8081</span>
<span class="highlight-line"> --provider-states-setup-url http://app:8081/provider-states</span>
<span class="highlight-line"> --provider pact-provider</span>
<span class="highlight-line"> --wait 15</span>
/tmp/pacts/pact-consumer-pact-provider.json</span></code></pre>
<p>The tests can then be ran with <code>docker-compose up --abort-on-container-exit --exit-code-from pact</code>. The additional argument ensures that after test completion, everything is torn down, and the exit code from the pact-cli image is used as the overall exit code. Without this argument, the Spring Boot service will remain up and accessible on <code>localhost:8001</code>.</p>
<p>In the docker-compose file, note that I'm mounting the <code>/src/test/pacts</code> volume into <code>/tmp/pacts</code> of the <code>pact-cli</code> image and then instructing pact-cli, through its <code>verify</code> command, which Pactfile to verify, and against which service. As docker-compose uses DNS to references running containers, the app is accessible on <code>http://app:8081</code>, as that's the name I gave that container within the compose file.</p>
<p>I personally prefer this solution as it's neat, generic, and ties closely with how it will run in CI (spoilers!!).</p>
<h4>Pact Provider Verification as a Spring/JUnit Test</h4>
<p>In JVM-land, it is alternatively possible to incorproate Pact provider verification as part of a JUnit suite.</p>
<p>I'll demonstrate what is needed for a bare-bones provider verification test - but to repeat for clarity - you only need one or the other!</p>
<p>First, I declared the test-scoped dependency for the <code>junit5spring</code> Pact provider library:</p>
<pre class="language-xml"><code class="language-xml"><span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>dependency</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>groupId</span><span class="token punctuation">></span></span>au.com.dius.pact.provider<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>groupId</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>artifactId</span><span class="token punctuation">></span></span>junit5spring<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>artifactId</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>version</span><span class="token punctuation">></span></span>4.1.17<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>version</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>scope</span><span class="token punctuation">></span></span>test<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>scope</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>dependency</span><span class="token punctuation">></span></span></span></code></pre>
<p>Next, I set up a new <code>ProviderVerificationTests</code> class. This is configured to run as a Spring Boot test on a random port, so a <code>@BeforeEach</code> block is necessary to instruct the Pact provider context what port to look for on <code>localhost</code>.</p>
<p>With JUnit5, it's possible to autogenerate tests from a <code>@TestTemplate</code>, which means that given the contents of a <code>@PactFolder</code>, individual tests can be auto-generated for each interaction in the Pactfiles.</p>
<p>Finally, provider state set up is handled . As this isn't being used yet in my Pactfiles, I can go ahead and create an empty method.</p>
<p>The resulting test class is:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">package</span> <span class="token namespace">co<span class="token punctuation">.</span>uk<span class="token punctuation">.</span>samhogy<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junit5<span class="token punctuation">.</span></span><span class="token class-name">HttpTestTarget</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junit5<span class="token punctuation">.</span></span><span class="token class-name">PactVerificationContext</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junit5<span class="token punctuation">.</span></span><span class="token class-name">PactVerificationInvocationContextProvider</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junitsupport<span class="token punctuation">.</span></span><span class="token class-name">Provider</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junitsupport<span class="token punctuation">.</span></span><span class="token class-name">State</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">au<span class="token punctuation">.</span>com<span class="token punctuation">.</span>dius<span class="token punctuation">.</span>pact<span class="token punctuation">.</span>provider<span class="token punctuation">.</span>junitsupport<span class="token punctuation">.</span>loader<span class="token punctuation">.</span></span><span class="token class-name">PactFolder</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">org<span class="token punctuation">.</span>junit<span class="token punctuation">.</span>jupiter<span class="token punctuation">.</span>api<span class="token punctuation">.</span></span><span class="token class-name">BeforeEach</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">org<span class="token punctuation">.</span>junit<span class="token punctuation">.</span>jupiter<span class="token punctuation">.</span>api<span class="token punctuation">.</span></span><span class="token class-name">TestTemplate</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">org<span class="token punctuation">.</span>junit<span class="token punctuation">.</span>jupiter<span class="token punctuation">.</span>api<span class="token punctuation">.</span>extension<span class="token punctuation">.</span></span><span class="token class-name">ExtendWith</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">org<span class="token punctuation">.</span>springframework<span class="token punctuation">.</span>boot<span class="token punctuation">.</span>test<span class="token punctuation">.</span>context<span class="token punctuation">.</span></span><span class="token class-name">SpringBootTest</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">org<span class="token punctuation">.</span>springframework<span class="token punctuation">.</span>boot<span class="token punctuation">.</span>test<span class="token punctuation">.</span>context<span class="token punctuation">.</span></span><span class="token class-name">SpringBootTest</span><span class="token punctuation">.</span><span class="token class-name">WebEnvironment</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">org<span class="token punctuation">.</span>springframework<span class="token punctuation">.</span>boot<span class="token punctuation">.</span>web<span class="token punctuation">.</span>server<span class="token punctuation">.</span></span><span class="token class-name">LocalServerPort</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token annotation punctuation">@Provider</span><span class="token punctuation">(</span><span class="token string">"pact-provider"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@PactFolder</span><span class="token punctuation">(</span><span class="token string">"pacts"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@SpringBootTest</span><span class="token punctuation">(</span>webEnvironment <span class="token operator">=</span> <span class="token class-name">WebEnvironment</span><span class="token punctuation">.</span><span class="token constant">RANDOM_PORT</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">ProviderVerificationTests</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token annotation punctuation">@LocalServerPort</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token keyword">int</span> port<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@BeforeEach</span></span>
<span class="highlight-line"> <span class="token keyword">void</span> <span class="token function">before</span><span class="token punctuation">(</span><span class="token class-name">PactVerificationContext</span> context<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> context<span class="token punctuation">.</span><span class="token function">setTarget</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">HttpTestTarget</span><span class="token punctuation">(</span><span class="token string">"localhost"</span><span class="token punctuation">,</span> port<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@TestTemplate</span></span>
<span class="highlight-line"> <span class="token annotation punctuation">@ExtendWith</span><span class="token punctuation">(</span><span class="token class-name">PactVerificationInvocationContextProvider</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token keyword">void</span> <span class="token function">pactVerificationTestTemplate</span><span class="token punctuation">(</span><span class="token class-name">PactVerificationContext</span> context<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> context<span class="token punctuation">.</span><span class="token function">verifyInteraction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@State</span><span class="token punctuation">(</span><span class="token string">"ping"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setupPingState</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>If you've complex Spring set up then this option may be more suitable than using pact-cli directly.</p>
<h2>Next Steps</h2>
<p>I now have barebones front- and back-end applications with a contract for a simple API call, with provider verification of that contract. For demonstration purposes I covered two ways of performing provider verification in Java - one in <code>docker-compose</code> with <code>pact-cli</code>, another directly in JUnit.</p>
<p>The next stages are to:</p>
<ul>
<li>Get this working on Gitlab CI</li>
<li>Create a more complex interaction and utilise provider states</li>
<li>Share Pactfiles between the consumer and provider in a better way than copy/paste</li>
<li>Demonstrate how contracts can be used to perform deploy-time safety checks</li>
</ul>
<p>In the next post, I'll move onto Gitlab!</p>
Traffic Lights and Watermelons2021-01-27T00:00:00Zhttps://samhogy.co.uk/2021/01/traffic-lights-and-watermelons/You probably don't need traffic light reporting. But if you do, use it properly!<img src="https://samhogy.co.uk/assets/traffic-lights-and-watermelons.jpg" alt="A traffic light, and a watermelon" />
<p>Project-based delivery has a natural waterfall-y inclination. Being budget-based, a project usually ends up fixing time, scope, or both. Work progresses through phases of activity, sequenced by milestones. The developers disappear for a few months, emerging from the cave with a shiny treasure. The testers then try to smash it with a hammer. More complex projects can involve dependencies, internal or external. Those additional handoffs need management too.</p>
<p>Progress towards completion is measured by proxy. This is the <strong>status</strong> of the project. The traffic light (RAG) model represents status visually:</p>
<ul>
<li>🟢 Green - On track to complete by expected date</li>
<li>🟠 Amber - At risk, under active management</li>
<li>🔴 Red - Not on track, escalate for resolution</li>
</ul>
<p>Organisations with many in-flight projects aggregate RAG reports for high-level situation awareness. However, RAG reports omit vital context and dampen useful signals. They are too abstract a visualisation technique.</p>
<h2>Watermelons</h2>
<p>It is common to see RAG reports used as an indicator of project health. Red -reporting projects are then considered unhealthy, necessitating senior management intervention. Environments of low trust and low autonomy are ones of fear and blame. In that context, reporting an unhealthy project status demonstrates a failure. To avoid this perception, bad news becomes suppressed.</p>
<p>This suppressive effect isn't always malicious. It can arise from putting a positive spin on the situation. In my experience it requires less effort to shift from green to amber than it does amber to red. To upper management, amber is 'pay closer attention', not 'get involved'. Switching from amber to red is often a negotiation: is help necessary?</p>
<p>A 'greenshifting' effect occurs as statuses work their way up the chain of command. Reds become ambers, which become greens.</p>
<p>This is unsustainable. Reality catches up at the point of criticality. Green projects turn red without room to course-correct, with nothing to show for it. To much chagrin the project suffers delays. Heads may roll.</p>
<h2>On Track?</h2>
<p>With a big bang release, green represents being on track to hitting the planned release date. What usually happens is that a project starts and immediately reports green. The initial plan is 'correct' in that reality has not yet invalidated or impeded it. But, as time goes on, continued reporting of green says nothing about reducing risk. Or validating assumptions. Or realising value. There is a distinct lack of transparency, risking an accumulation of issues.</p>
<p>The concept of being 'on track' becomes less useful as organisations embrace agility. Project-based delivery gives little breathing room for discovery or validation. Agility embraces the uncertainty.</p>
<p>Three principles of the Agile Manifesto come to mind:</p>
<ul>
<li>"Working software is the primary measure of progress"</li>
<li>"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."</li>
<li>"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."</li>
</ul>
<p>These exist to provide focus on what matters: <strong>working product</strong>. Status is a proxy measure for progress. Working software is the progress, in material form. Demonstrating progress this way makes status meetings disappear!</p>
<p>Scrum's three pillars are: transparency, inspection, and adaptation. These pillars build on one another. Only through the shared understanding of transparency can a team inspect-and-adapt. Concepts such as the 'Definition of Done' aid this shared understanding. Focus is on producing product increments to a regular cadence. The intention is to assess progress not through status, but through working product.</p>
<p>What if RAG reporting is still required in these environments? Start it on red. Only move to green when there is working product that is producing value. This better portrays that the risk is greatest at the start; the point of least knowledge. This will be a difficult sell. Especially in blame cultures. But this is coming from a position of openness. It requires honesty and courage. Focus and commitment will demonstrate its effectiveness.</p>
<h2>Boots on the Ground</h2>
<p>Rather than keeping activity at arm's length, get involved! Work with teams, the boots on the ground, to set up a sustainable delivery path. Report on progress, not status.</p>
<p>Up front, the incentive is to get something working. To produce an increment that kick-starts development, but also begins to produce value. Through the feedback, plans can be re-adjusted. Features on the 'critical path' may not be as important as suspected. New opportunities present themselves. Large budget overheads become less relevant too. Value streams from the working product provide self-sustaining investment for future development.</p>
<p>Talking about the 'status' moves from the abstract to the concrete. From traffic light colours, to work done. Another feedback loop shortened.</p>
Six Sprints Ahead2021-01-16T00:00:00Zhttps://samhogy.co.uk/2021/01/six-sprints-ahead/Delivery with water... slides!<p>How far should a team plan ahead in an agile setting? We can take some guidance from the <a href="https://agilemanifesto.org/">Agile Manifesto</a> itself, in that it values:</p>
<blockquote>
<p>Responding to change over following a plan
(That is, while there is value in the items on the right, we value the items on the left more.)</p>
</blockquote>
<p>Additionally, there's also the principle where we:</p>
<blockquote>
<p>Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.</p>
</blockquote>
<p>No-brainer then, right? Don't do it. Yet, I've worked on (and with) many teams which have a strong desire to fully spec out many sprints ahead. Before we jump into why this is misconceived, here's a point-of-view experience of a water slide from a (recently closed; 🕯️) water park near where I live.</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/RMSmMGw2LUo" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>This slide is called the Calamity Canyon. It's a series of cascading flumes and pools, targeted as a gentle family experience. As a ten-year-old it was great fun playing WWF Royal Rumble in these pools, avoiding the <s>genuine safety concerns</s> scorn of the lifeguards. I'll let you figure out the rules.</p>
<p>Do your sprints feel like that? A series of cascading waterfalls? Mine sure have in the past.</p>
<h2>Roadmaps and Product Goals</h2>
<p>It's common wisdom that teams should have "a sprint or two" ahead planned out according to a definition of ready. Any further than that, the team has done too much planning.</p>
<p>The team's refinement efforts should be focused on the items at the top of the backlog, so the work is understood to a point where it can be taken into Sprint Planning. This can involve some form of effort/complexity estimation, breaking the items into smaller, deliverable chunks, and having conversations with interested parties. Actually fitting this work into sprints in advance is wasteful. Firstly, teams judging sprint capacity through story point velocity won't know this upfront. It will be subject to change. Secondly, doing this ahead of Sprint Planning deprives the Developers from utilising this timebox effectively, to establish <em>how</em> they will deliver the work. There's also the notion that if, priorities do change (and they do), all this effort was for naught.</p>
<p>Recently, the <a href="https://www.scrumguides.org/scrum-guide.html">2020 Scrum Guide</a> introduced the concept of the <strong>Product Goal</strong>, which sits above a <a href="/2020/02/sprint-goals/">Sprint Goal</a> as a guiding, top-level objective the team works towards. This focuses sprint goals as concrete steps to achieving the product goal, giving a nod to strategic product decisions made by the Product Owner, in conversation with stakeholders and customers.</p>
<p>That's a long-winded way to say that it is perfectly possible for teams working with agility to have roadmaps. The key is for those roadmaps to express a direction of travel through high-level objectives. If they become too specific - of the form "X will be delivered on Y" for example - then the roadmap becomes a Gantt Chart in disguise. A desire to plan several sprints ahead comes from attempting to present or express a concrete roadmap in a more "agile-friendly" format. But I'm calling a spade a spade here. Crucially, an agile roadmap is one that recognises the changing, complex environment and embraces the uncertainty. A Product Owner will have a concrete understanding of the goal for the next sprint, ambitions for future sprints, but be willing to adapt if necessary.</p>
<h2>Planning - Incrementally, Iteratively - Why Not Both?</h2>
<p>Agile development is not against planning as a concept. Seriously, it's not. Agile teams plan <strong>all the time</strong>. The key distinction is that our planning is "just in time" rather than ahead of time. Planning this way maximises the effectiveness of plans, minimising the chances of obsoletion through the passage of time. When a plan is formulated and later must change, that's wasted effort.</p>
<p>Planning on a just in time basis conforms to decision-making at the point of most information. As outlined by Mary and Tom Poppendieck in <em>Lean Software Development: An Agile Toolkit</em>, this is the <strong>last responsible moment</strong>:</p>
<blockquote>
<p>[...] delay commitment until the last responsible moment, that is, the moment at which failing to make a decision eliminates an important alternative. If commitments are delayed beyond the last responsible moment, then decisions are made by default, which is generally not a good approach to making decisions.</p>
</blockquote>
<p>So, Scrum teams have several planning ceremonies. Sprint Planning is planning how to deliver a working Increment of product that achieves a Sprint Goal. The Sprint Review is planning the next move, based on the produced Increment. The Retrospective plans changes to the process, based on experience rather than an presumptive application of theory. The Daily Standup is a daily <strong>planning</strong> event, where the Developers plan their day around meeting the Goal.</p>
<p>But there's the elephant in the room. It's those mini-waterfalls that I enjoyed at visits to the water park. It's that it's really easy to take a roadmap and deliver it incrementally through the Scrum framework. To do so "isn't Scrum", but common enough to warrant discussion.</p>
<p>Incremental-only product development is when there's a fixed idea for a product, which is built in stages. Sprints correspond to milestones. The expectation is clear, the scope is pretty much fixed. Value is realised and risk is reduced early through incremental delivery. Priorities will probably not massively shift based on user feedback. Not a huge amount of learning is generated. In this instance, the idea of planning "six sprints ahead" is appealing, because the product is known and the stakeholders are effectively projecting a delivery date. More correcly, they're attempting to.</p>
<p>It's also possible to do iterative-only development. This is an extreme form of discovery called rapid prototyping. It's where an idea is formulated and rapidly tested. The prototype is thrown away when unsuitable. It may not produce value itself, but discover where value can be realised. In startups, this process establishes the basis of an MVP and is incredibly useful when a team has identified a need but doesn't yet have an answer for meeting that need.</p>
<h2>Options, Not Requirements</h2>
<p>More desirable is to combine the gradual, incremental-based approach to delivery with the discovery-based approach of iteration. Have a roadmap, but understand that these are, at best, <a href="/2020/12/whose-requirement-is-it-anyway/">guesses for what should be built</a>. The prioritised Product Backlog is a series of <strong>options</strong> that the Product Owner suspects will deliver value. These are like the financial instrument that gives the holder the right, but not the obligation, to be exercised. Product Owners prioritize to "maximise the value" of the Scrum Team. As such, the Product Backlog gives the team an order of options to exercise next, based on their theorized value.</p>
<p>An Increment of product, produced at the end of a Sprint, is a stepping stone to achieving a Product Goal. It makes a material impact on your roadmap. But it's also an opportunity to iterate on the product's direction by re-examining the options available and establishing whether a pivot is required. This requires delivering working increments to users, gaining value and generating learning.</p>
<p>Planning too far ahead of time exercises these options in advance. It incurs cost (in time and effort) to explore the problem space - doing refinement, estimation, etc - without returning value. Further cost is incurred in the event of having to re-work the plan. Keeping options available until the point of most information is a waste-reducing, cost-reducing, risk-reducing exercise. Working with agility is to be able to pivot on a dime, for the cost of a dime.</p>
The 'Just' Alarm2020-12-26T00:00:00Zhttps://samhogy.co.uk/2020/12/the-just-alarm/Oh, the complexities concealed behind that word!<p>This post is inspired by a Tweet of mine on the subject of Behaviour-Driven Development. As my tweets are set to auto-delete after 30 days (as a matter of digital hygeine), I shall reproduce it below:</p>
<blockquote>
<p>In the same way that a new requirement drives a new test, perhaps the use of the word "just" is a good indicator for an exercise in BDD</p>
</blockquote>
<p>As a big fan and proponent of Behaviour-Driven Development, or Specification by Example, I'm always on the lookout for ways to better understand and model the "business domain", using business language. In the developer community we like to utilise the three-act play structure (<code>Given/When/Then</code>) as a way of semantically expressing these business rules. The exact format matters less to me than the conversations that happen in order to produce these outputs. What also matters are the examples that we use to decorate business rules, adding context and specificity.</p>
<p>One of my 'trigger' words is the word 'just'. It glosses over complexity. As someone who likes to get into the weeds of problems, that frustrates me. Very often the 80/20 rule often applies, where 'just' is mapping the simple business-as-usual processes, rather than the exceptional and more-complicated behaviours. But as developers building systems to model business processes, we need to think about what lurks in the weeds of the 20%. We'll probably spend 80% of our time there.</p>
<p>Some colleagues in the past have tried to ban the word 'just' for this reason, but I've found that synonyms will crop up instead, such as 'simply'. So instead, these days I use the word as an opportunity to begin an exercise of BDD. Let's put some examples down on the board to demonstrate how this works. The business representative is a subject matter expert after all, let them tell the story. Get them involved! Software development is a complex, multi-disciplinary endeavour. We require shared understanding to ensure we're building the right thing, in the right way.</p>
<p>So, don't ban the word 'just', use it as a jumping off point to have more conversations and gain a deeper understanding. The vast majority of the issues we run into during software development aren't technical, they're about communication.</p>
State of the Sam2020-12-24T00:00:00Zhttps://samhogy.co.uk/2020/12/state-of-the-sam/Well, that was indeed a year!<p>One of my favourite authors, Brandon Sanderson, posts an annual "state of" on his birthday, summarising works in progress and the year hence. I'm going to do the same, although not on my birthday, because despite the Global Bastard stopping the world, quite a lot has happened. And it's good to talk.</p>
<h3>A Little Bit of DevOps</h3>
<p>My last project at Scott Logic started with a discovery phase in the last six weeks of 2019, followed by a ramp-up in the New Year. I was leading a team building an MVP on a limited budget for a UK financial institution. My focus was primarily on spec'ing out the technical architecture for the solution, fitting it within the bank's existing Google Cloud estate. This dropped me head-first into the world of Cloud, DevOps and infrastructure as code with Terraform! I learned a lot in a very short space of time, including familiarity with containerisation, Kubernetes, CI/CD and platform engineering. The security aspect of this project later spun out into a talk that I presented at BSides NCL, and later at DDD: <a href="https://samhogy.co.uk/talks/ci-cd-pipeline-security">CI/CD Pipeline Security</a>.</p>
<p>All said, this was a challenging and emotionally draining project, which ended up with me leaving in exhaustion in the middle of a pandemic. Oh yeah, about that...</p>
<h3>Pandemic? Great, Let's Change Job!</h3>
<p>In May, <a href="/2020/05/new-horizons/">I changed job</a>. With hindsight it is strangely a good mix of being a backwards, forwards and sideways move. The team are great, the company have been incredibly patient and supportive, especially with someone as anxious as I about the COVIDs. I've been in the office four times and one of those was just to pick up the laptop! The job has been effectively entirely remote. The day to day coding isn't the most interesting thing I've done professionally, but there's a chance to modernise a legacy system and practice some of the stuff I preach in talks and blogs. That, I'm thankful for. The role has given me the opportunity to get in on the ground floor with turning a very big ship around. Naturally establishing momentum is tough, but with continued effort next year we'll start to see some very fruitful results. Which brings me onto...</p>
<h3>Agile Blogging</h3>
<p>This year's blogging has mostly, aside from a few curiosities, taken the form of opinion (from experience) pieces on the topic of Agile development. I'm clearly passionate about this subject, even more so than most technical topics, because experience is teaching me that the big problems we encounter on teams are usually down to communication.</p>
<p>At the start of the year, I recalled a conversation that occurred in a pub nearly ten years ago. I was very junior in my career and talking with a well-known and highly-regarded coach, after they gave an agile training session to my grad scheme cohort. I was adamant that Sprints need goals to provide the necessary focus, product vision and room for experimentation. This idea was dismissed out of hand. Well, I wrote a blog about the power of <a href="/2020/02/sprint-goals/">Sprint Goals</a> because on reflection, there's a huge difference between teams that set themselves sprint goals and teams who don't. Remember folks, just because a popular person says something, doesn't mean they're right!</p>
<p>From that, I've further blogged on how <a href="/2020/07/overcoming-fud-when-improving-processes/">to fight back against FUD during process improvement</a>. How to establish a move towards <a href="/2020/08/going-with-the-flow/">optimising for flow</a> and how, rather than quality being something only testers are concerned with, <a href="/2020/08/quality-is-a-team-responsibility/">quality is a team responsibility</a>. I've written a <a href="/2020/10/stuffing-the-backlog/">silly parody song</a> about how the backlog is where work goes to die, and how rather than focusing on delivering projects of work, we gain more purpose from our jobs and build better products by <a href="/2020/10/why-focus-on-outcomes/">focusing on outcomes</a>. Finally, I pondered the very <a href="/2020/12/whose-requirement-is-it-anyway/">concept of a requirement itself</a>, arguing that in a lot of cases we should <em>embrace uncertainty</em>.</p>
<p>There's a lot there, most of which I'm proud of. It's good to capture your thoughts and to let go the fear that in a few year's time, you may look back differently. Writing down and communicating your thoughts clarifies your thinking, even if nobody reads them.</p>
<h3>Conferences, and Conference Talks</h3>
<p>In order to help keep the community lively and engaged during our lockdown hibernation, I came up with the idea to host a <a href="https://ne-rpc.co.uk">virtual conference - NE-RPC</a>. With the help of some former colleagues, new and old friends, we put this together... twice! On zero budget, we hosted two separate multi-track full-day events and learned <strong>so much</strong> about running a virtual conference.</p>
<img src="https://samhogy.co.uk/assets/ne-rpc.png" />
<p>The advantage of a virtual event is that if you have the option to pre-record a talk, there's very little for you to do on the day. Our talks were all pre-recorded (apart from a last minute live fill-in), but we hosted live Q+As with speakers afterwards. This gave a very nice balance of communication and rapport with speakers. Next year I might do something more experimental with the virtual format, such as utilising OpenSpace, to try and recreate some of the inter-personal dynamics that virtual events can easily omit.</p>
<p>At the start of the year I performed my <a href="https://samhogy.co.uk/talks/in-retrospect">retrospectives talk</a> in-person twice. Once at DDD North in Hull, the second at DevHub North. The latter was luckily recorded and it's always nice to have physical evidence of your performances. It helpes get a foot in the door at larger events. Both of these were before the plague hit, and naturally everything has gone a bit virtual since then.</p>
<p>Here's my last 'live' performance of In Retrospect, from DevHub North. Note the talk has evolved a bit since then, I've ditched a lot of the theory at the beginning to focus more on the end piece, creating meaningful actions. My favourite version of the talk is the most recent iteration from Agile Scotland, which will hopefully be online soon.</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/q2JpAZQxYKg" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Here's the latest version of my CI/CD Pipeline Security talk, as presented at DDD:</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/kkWIvm9WMJg" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Finally, here's the talk I gave on unit testing at NE-RPC, Testing that the Code Does What the Code Does!</p>
<iframe width="640" height="360" src="https://www.youtube-nocookie.com/embed/hdKGlgk-34g" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>A good year of talking. Although I do miss the in-person events, I also like how easy it is to present at a virtual event. With a small child (and therefore limited travel capabilities in pre-apocalyptic times) this has opened up so many events from an accessibility point of view.</p>
<h3>Halloween</h3>
<p>I am very proud with how this year's Halloween display project turned out. I had the idea in the back of my mind since last Halloween. Although I was somewhat anxious about whether even supporting trick or treating was the right thing to do in the year of the apocalypse, in the end I decided to go ahead with a display but to stick a sign up saying we're not handing out sweets. Oh, and barricade the doorway with pumpkins! As we also found with Christmas, people went BIG on decorations this year, due to being stuck at home. I'm not complaining, just hoping this trend continues to next year! There's already a few ideas poking around the old noggin for 2021. Fingers-crossed this can involve actual tricking and treating.</p>
<img src="https://samhogy.co.uk/halloween/2020/scene.jpg">
<p>Take a look at the <a href="https://samhogy.co.uk/halloween/2020">Halloween sub-site</a> for more 2020 pics!</p>
<h2>What's Next?</h2>
<p>Well, there's a few things that I have planned for next year. I'll announce formally in the new year! But, I do plan to do more Agile Coaching within my organisation, circumstances depending, and I'm going to look at a few scrum.org certifications. Most likely the PSM track.</p>
<p>I think that I'll develop my retrospectives talk into a workshop, as it's kinda crying out to take that format. I don't forsee that a grand re-opening of face-to-face society is going to happen any time soon, so some thought will be needed about how to deliver this virtually, alongside my existing full-time work commitments. There'll be a lot more blogging too - the drafts folder is already bulging at the seams.</p>
<p>2020 has been a rollercoaster year, with Coronavirus only being part of the reason for that. We're very clearly not out of the woods yet, but I'm lucky to be in an industry where working from home is trivial, where I have the space to work mostly uninterrupted, lots of open space for exercise and a supportive, loving, small family.</p>
<p>Please, enjoy safe festivities and see you on the other side!</p>
<p>Sam</p>
Whose Requirement is it Anyway?2020-12-03T00:00:00Zhttps://samhogy.co.uk/2020/12/whose-requirement-is-it-anyway/When we are told something is a requirement, it's very often... not!<blockquote>
<p>"It is a requirement that the system..."</p>
</blockquote>
<p>How many times have you heard this in your career? And of those, how many times has it been used to actually mean "shut up and do as you're told"?</p>
<p>Software has a very interesting property - it's <em>soft</em>. Flexible. Malleable. It can be dynamically adjusted to meet changing user needs, as they become apparent. Unlike a manufacturing production line for a physical product (i.e a car) that undergoes 'production' before 'delivery', software runs these concurrently, continuously.</p>
<p>Software will transform radically throughout its operational lifetime. Features once considered foundational may be jettisoned entirely, should things move in a different direction. So given this, to what extent can something actually be a requirement?</p>
<h2>Definition of Done</h2>
<p>Requirements come in at the start of the process, so let's start by looking at the end, where we consider work to be done. Using this let's tease out the tensions that exist between short-term projects and a long-term product.</p>
<p>Within the agile community, we talk lots about the 'definition of done'. What do you describe as 'done'? In a previous role, the team commonly referred to 'done' (devs had finished), done done (scheduled for release), and done done done (running in production). I laugh at this now, but getting teams to define done is hard. That's because done means different things to different people.</p>
<p>With a project-based approach of delivering requirements according to a schedule, 'done' ultimately means the project is delivered. Should the project have separate waterfall-like phases of dev > test > production, then within a given phase, done signals progression to the next phase. Realistically, this means "done... <em>probably</em>" because there's always a chance the work can come back for rework.</p>
<p>Teams <a href="/2020/08/going-with-the-flow/">going with the flow</a> instead look to reduce these handoffs and rework exercises, which gets us to 'done' in the 'live in production' sense sooner and safer. But even with a super-agile Kanban workflow, there's still a chance work comes back - even if it's via an incident ticket from the production system!</p>
<p>Truthfully, we want 'done' to mean <strong>"we can work on something else"</strong>. Even though a project might be done when the work is delivered, even though the team move onto the next project, the business still carries the maintenance costs of that delivered work, right up until the point it's decommissioned. Is a business <em>truly done</em> with work while it's providing value in a live system? Only when we die are we done with life.</p>
<p>Okay, this is a very broad definition of done. You're splitting hairs, Sam. This isn't useful for planning our Sprint Velocities, or for moving work to the 'done' column of the Kanban board. All true and fair enough - you need a definition of done that you can use as a nearly-always-reliable indicator that something else can be worked on now. On a separate note, when that indicator becomes unreliable, that's your signal you have a quality problem.</p>
<p>There's a distinction that exists between projects, with an end date, and the continued usage, maintenance and costs incurred beyond the initial scope of that project. I'm making this point because it's within this space that the usage of the word 'requirement' sits.</p>
<h2>Requirements</h2>
<p>We've talked about the dynamics of software, its always-changing, adapting nature. We've talked about fixed periods of 'projects' that exist within this continuum of a piece of software's lifetime. How does this all impact on software requirements?</p>
<p>Projects have requirements. A business expectation is set. Analysts analyse, transmorgifying these expectations into a series of requirements in a change request document. The work is estimated. The estimate is treated as a quote. Funding is sought. A code is allocated in the timesheet for an individual, named developer, to work on ticket <code>WTF-123: blue button background MUST be changed to green button background</code>. Very often the business context - the why - isn't included in the work ticket. Sometimes it is expressed in the <code>Given / When / Then</code> or <code>As A / I Want / So That</code> format - but even then, it's considered a given that this ticket WILL produce the desired outcome. And you as a developer will never find that out.</p>
<p>Is it <em>actually</em> a requirement of the system that this button <em>needs</em> to be changed from blue to green? Surely requirement means required. Are we saying that without this change the system isn't functional? Of course, this is a rather rigid and literal interpretation of the word requirement.</p>
<p>The honest answer is that for most of our purported requirements, not many of them are required in that sense of the word. Although it can be the case in organisations working in projects, that the requirement is actually on <strong>you</strong>, dear Developer, to <strong>do what is being asked of you</strong>.</p>
<p>Of course, there are regulatory requirements (and security requirements from the back of this, that incur fines and public outrage from non-compliance). But what is seen as requirements for work that delivers business value? Well, similar to a practical definition of done above, actually, these aren't requirements but <strong>guesses</strong>. The guess is "does this work meet the identified needs and produce business value"? Is the button background colour change a requirement, or a guess that it will result in more users clicking this button, thus bringing in more revenue? More radically, is the need to have a button a requirement <em>at all</em>, or could this be achieved more effectively through some other means? Fancy taking a punt?</p>
<p>Projects fix this uncertainty upfront by expressing a solution in the form of requirements. They remove all the context of why we're doing something. They remove all the creativity of how something will be done. And then the work is driven through a delivery framework. Only then can the output be inspected. If it's discovered it didn't work, start a new project I guess?</p>
<h2>Requirements -> Hypotheses</h2>
<p>Prof J Hackman's book "Leading Teams - Setting the Stage for Great Performances" outlines what happens when both the "ends" (the outcomes we want to achieve) and the "means" (the requirements - the way we're going to achieve those outcomes) are fixed:</p>
<img src="https://samhogy.co.uk/assets/ends-and-means.png" />
<p>We talk often about how software (both the development, and the testing) is a knowledge-based, creative activity. But in environments where both the ends and the means are specified we don't have a huge opportunity for creativity. Traditional project thinking and project management plans to turn uncertainties into certainties, then works the plan. By doing so, the work is expressed through tickets that can be completed almost by rote, through precise itemisation. The desired outcomes of the change has been almost forgotten, over the outputs considered "required" to be delivered.</p>
<p>What if, instead, we embraced the idea of uncertainty? What if we harnessed the creative powers of our team? What if we used time-boxed iterations to express a goal (a desired outcome) and experiment towards what delivers us value? From a costing/budget perspective this is simple - your cost is the headcount of the team. This is the power of <a href="/2020/10/why-focus-on-outcomes/">focusing on outcomes</a> over the outputs.</p>
<p>The 2020 Scrum Guide talks about a <a href="https://www.scrum.org/resources/blog/scrum-guide-2020-update-introducing-product-goal">Product Goal</a> - the intended direction, strategy or vision of your product. The aim is to give clarity to the team the purpose of the work undertaken. To achieve this goal, there's a Product Backlog - a series of steps to reaching the goal, or perhaps more powerfully, a series of <em>hypotheses</em> about meeting parts of the product goal, and future value that can be attained by doing so. Scrum also distinguishes between product backlog items and sprint backlog items. During <a href="https://www.scrumguides.org/scrum-guide.html#sprint-planning">Sprint Planning</a> the developers plan the work. By expressing product backlog items in the form of hypotheses, the developers are given the freedom to innovate. Likewise, through product backlog refinement, large-scale hypotheses are broken down into sprint-sized hypotheses that can be worked on. This naturally forms the objective of a sprint - the <a href="/2020/02/sprint-goals/">sprint goal</a>.</p>
<p>Interestingly, agile-based frameworks such as Scrum do not eliminate the planning that is present in projects. However, it moves it from an "up front" activity to a "just-in-time" activity. By deferring planning until the most-responsible moment, when the team possesses as much information as possible, they are able to better experiment with options available to work out what they should be doing. Note that at this point, requirements virtually fade into existence. Rather than dealing with requirements, the team is considering <strong>user stories</strong>, the outcomes of those user stories, <strong>why</strong> the business wants to meet them, <strong>how</strong> they will be met and, for the very mature teams on this journey, begin to understand <strong>when</strong> it will be known that these user needs are being met.</p>
<p>By creating a cross-functional <strong>product</strong>-aligned team, rather than ephemeral <strong>project</strong>-aligned teams, the inconsistency of formally expressing uncertainties through requirements is removed. Instead, teams focus on the true purpose of meeting user needs. Teams are empowered and trusted by respecting their autonomy, unleashing their creativity and having them own what they build.</p>
Why Focus on Outcomes?2020-10-22T00:00:00Zhttps://samhogy.co.uk/2020/10/why-focus-on-outcomes/Software is in the business of improving lives, not shipping sausages from a production line.!<p>Yesterday <a href="/2020/10/stuffing-the-backlog/">I wrote a silly song parody about agile backlogs</a>. Today I'm going to get serious, a bit vulnerable and a bit passionate about why I build software.</p>
<p>The products and services that we build are designed to achieve outcomes. These outcomes - behavioural changes in our users, the acquisition or development of skills, empowering a new capability - carry business value. When we say we are in the business of "delivering business value", we mean that we do this by achieving these outcomes.</p>
<p>Agile development, as I understand it, places a large focus on outcomes. But we often fall into the trap of thinking about our output. Of course, we need output to achieve outcomes. However, too big of a focus on the management of output, at the expense of losing sight of the outcomes, leaves teams thinking that maybe this Agile thing isn't giving them what was promised.</p>
<p>Agile frameworks make it easy to be output focused. Let's pick on Scrum as a <em>lingua franca</em>. You start with a backlog, engage in intense and continual processes to <em>groom</em> this backlog, alongside the <em>estimation</em> of work and the <em>capacity planning</em> of how much can be done in an iteration. It's so easy for traditional project management to be ported onto this kind of framework. The glorified accountancy can even surface and worsen micro-management capabilities, with obsession on burn-down and perfect estimation.</p>
<p>Scrum of course provides the way out with <a href="/2020/02/sprint-goals/">sprint goals</a>, but these themselves are difficult to use effectively if the way your work is arranged in an output-focused way. Work such as "change the design of this button", "reword this piece of copy", "make the header banner blue". Your default sprint goal is "do the work in the sprint", which is not helpful and entirely output-driven.</p>
<p>For clarity, I'm not saying I don't value this output-driven stuff, I just value the outcome-driven stuff <em>more</em>. And I don't mean to bash on Scrum, it's simply a convenient jumping-on point.</p>
<p>So perhaps it's time to detail how we can focus on outcomes rather than outputs by way of a use case.</p>
<h2>An Example: Changing the question text of a web form</h2>
<p>Let's say that we work on a product and in order to buy that product, the user needs to answer a series of a questions. We present these on a web form as part of the purchasing workflow. Our support operatives are receiving a lot of queries from confused customers, because the wording of the questions is confusing. We're losing customers and the ones we're retaining aren't that happy either. We agree that something needs to be done.</p>
<p>The output-driven approach is to "change the text of question". We can ask a designer to re-word the question - they may even run a small focus group - and then a ticket with the exact wording to change the copy on the web page is raised on the team's backlog. Fantastic. Job done. The output of a text change is delivered. Next project, please.</p>
<p>But how do you <em>know</em> that you've changed anything for the better? You can only answer this question by focusing on the outcome.</p>
<p>I'd propose the outcome that we're trying to achieve is that "customers should not be confused when purchasing our product".</p>
<p>That is your user story. That is the outcome that you wish to achieve - not "change the text of question to ...".</p>
<p>The exact text from the focus group? That's our <strong>experiment</strong> - it's how we <em>think</em> we'll achieve the outcome. Now we can take a stab at answering how we can measure the outcome. Putting the text change into production isn't enough. So we look elsewhere. Are support queries on this issue dropping? Are signups increasing? Does our analytics software identify users spending less time answering this question? Could we survey customers? It could even be possible to set the outcome as the sprint goal, experiment with alternatives through A/B testing, measure some criteria and present these at sprint review alongside data that demonstrates business value.</p>
<h2>Why does this matter?</h2>
<p>About a decade ago, a close family member suffered a brain haemorrhage. Don't worry, they're fine. When leaving hospital, they were left with no job and a huge cognitive task of having to re-learn basic things. Memory, reading comprehension, concentration - they all took a hit. Filling out a benefits form to claim the support they were entitled to was such a complicated task it drove them to tears. Without delving too much into the politics, in this case the outcome the UK Government were aiming to achieve was not aligned with the outcome of the users of their service. The questions were deliberately set out to confound, deter and disrupt the ability for people to claim their entitlements. This was ideologically-driven and financially incentivised.</p>
<p>When we design our products and services, we hold an immense power over people's livelihoods and wellbeing. The question text example seems superficially noddy, but when you realise that even simple things like this can have a profound impact on people's lives in times of need, the true power of aligning your work to outcomes becomes apparent. Indeed, I'd argue it's a <strong>responsibility</strong>. Work thrown over the wall as "delivered", without answering the question "how has this made things better" is not <em>done</em>. This obsessive focus on user outcomes is what gives our work meaning, what provides teams with a purpose and ultimately why we do this.</p>
Stuffing the Backlog2020-10-21T00:00:00Zhttps://samhogy.co.uk/2020/10/stuffing-the-backlog/The team don't like it / Stuffing the Backlog, stuffing the Backlog!<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/bJ9r8LMU9bQ" frameborder="0"></iframe>
<p>Now the team told the manager<br>
"This nonsense needs to stop"<br>
The work kept on piling in<br>
But pushed straight to the top<br>
New stuff was top priority<br>
The backlog wasn't groomed<br>
So it became a pointless<br>
Endless list of to's to do-ooo</p>
<p>The team don't like it<br>
Stuffing the Backlog, stuffing the Backlog<br>
The team don't like it<br>
Stuffing the Backlog, stuffing the Backlog</p>
<p>By order of the manager<br>
They ban that waterfall<br>
Filled up a brand new backlog<br>
They're in it for the long haul<br>
But the sprints had no goals just an endless stream of work<br>
The deadlines kept a'comin meetings added busywork<br>
And when the manager asked when they'd be done<br>
They began to wail</p>
<p>-- chorus</p>
<p>Now over in a pull request<br>
Oh, there's something going on<br>
Some required work is missing<br>
It was gonna take to long<br>
They add a task to do the rest<br>
At a later point in time<br>
But the devs knew the backlog<br>
Is where work goes to die</p>
<p>-- chorus</p>
<p>The business called a team meeting<br>
They said how long will it take<br>
We need to-the-hour estimates<br>
A waterfall headache<br>
As soon as the business were laughed outta there<br>
The team brought in Kanban to build the software<br>
Limited work in progress to stop the nightmare<br>
Then they all wailed</p>
<p>--chorus</p>
<p>The team don't like it, they think it's not agile<br>
Stuffing the Backlog, stuffing the Backlog<br>
The team don't like it, fundamentally can't take it<br>
Stuffing the Backlog, stuffing the Backlog<br>
The team don't like it, you know that it's not agile<br>
Stuffing the Backlog, stuffing the Backlog<br>
The team don't like it, really, really hates it</p>
<hr>
<p>I was going to write something serious about how if you've got a backlog that extends out 18, 24 months, even further beyond than that, then you're kidding yourself as to when this work will be done. Your backlog is a dumping ground for stuff that's never going to get done, because new work will always conveniently jump straight to the top of the priority list and be demanded immediately. This is why when someone's working on a task and they start splitting out the hard bits into separate backlog items, I get upset, because it's done in the knowledge these will never get selected for prioritisation.</p>
<p>Or that when you start an Agile transformation from the perspective of forming a backlog of work, rather than understanding <em>how</em> we can go about delivering value and framing our work in terms of the outcomes we want to achieve, not the outputs - all you've done is created an ordered list of work (which has value) and shoe-horned it into a delivery framework, with no guarantee of achieving any outcomes.</p>
<p>But I decided instead to write a song. I'm sorry.</p>
How Often Do You Commit?2020-10-20T00:00:00Zhttps://samhogy.co.uk/2020/10/how-often-do-you-commit/More often than you think, with good reasons!<p>Pretty much every minute or so.</p>
<p>This answer seems to surprise many, so I thought I'd flex the old writing muscles and justify this seemingly-bonkers process.</p>
<h2>Commits are Save Points</h2>
<p>The easiest way of visualising a change (herein: a commit) is like a save point in a video game. You're creating a snapshot of how things are at a point in time, writing this down, and giving yourself the ability to go back to that point in time.</p>
<p>In older version control systems, creating a commit was quite formal as the change was synced immediately to the central server. This put a big onus, socially-speaking, on making them meaningful and 'complete'; they were expected to work.</p>
<p>I found this restrictive when working locally. The tooling <em>constrained</em> how I worked - I ended up writing commits that were quite large, corresponding to a feature or a significant chunk of a large feature. The push towards 'all or nothing' meant I wasn't giving myself room to experiment and I often lost track of smaller changes within a wider change.</p>
<p>Recently I've been converting some JavaScript code to replace JQuery with native-DOM functionality. Within a particular file are lots of individual chunks of JQuery that I'm tackling one by one. Sometimes these are easy, sometimes it takes a bit of experimentation to get right. When doing the latter fiddly changes it's common to undo the change and try a different technique. Effectively, I want to go back to the save point!</p>
<p>But I don't want to have to undo the changes to the <em>entire file</em> to get to that save point - because I'd be throwing mostly working code away. So here's what I do.</p>
<h2>'Public' Commits vs 'Work In Progress' Commits</h2>
<p>Git's distributed nature means that changes you apply locally aren't shared with the rest of the team until you <code>push</code>. I can commit what I want, when I want, how I want. While commits are local, they're mine to play with and do as I please. I can use the tooling to <em>empower</em> how I work, rather than constrain it.</p>
<p>In the example above, I'd commit every time I got an individual JQuery replacement working. I'd also commit when I need to go to a meeting, or when I leave my desk for a tea/toilet break. Or at the end of the day. Very often these commit messages will be terse and very contextual. "brb going for a wee" is a common occurrence in my commit message! I can then have a look through my git log to get an accurate understanding of where I'm at with a task. I don't need to worry about going into a huge amount of detail unless I think it would be helpful - the last commit of the day commonly has a bullet-list of thoughts and TODOs.</p>
<p><code>git log --oneline -n 5</code> is a compact way of showing the last five commits in a terminal. I <a href="https://github.com/sh1989/dotfiles/blob/master/.gitconfig">have this aliased</a> to <code>git slg</code> because I use it so much.</p>
<p>Now, I can guarantee that when I need to leave my desk, the commit I make won't compile. That's fine. The reason I do it anyway is to help defend against the "Doorway Effect". Ever walk into a room only to forget why you're there? Or even worse, what you were doing before? Well, by creating a small save point just before stepping through the doorway, I can easily jump back into that old frame of reference.</p>
<p>To undo a commit in git <strong>AND put those changes back into your local staging area</strong>, you can run <code>git reset --soft HEAD^</code>. I run this when I get back to my desk to reinstate that frame of reference.</p>
<p>(To backup these piecemeal changes somewhere remote, you can push them to a branch on <code>origin</code>, or if you want to keep that prim and proper, onto a private fork. Most source control services like GitHub, BitBucket and GitLab allow for private forks of repositories. They're useful dumping grounds and a godsend if your laptop decides to die.)</p>
<p>When I'm ready to share the work I've done, I now need to go through and turn these 'work in progress' commits into more meaningful, well-structured and properly documented commits. I call these 'public' commits because I'm going to push them to the remote repository. They'll be reviewed and become part of the project history.</p>
<p>With the type of work in progress commits I write, I know I need to go back and tidy these up. This gives me the opportunity to perform <strong>deliberate practice</strong> in writing good commit messages. Indeed, it forces me to!</p>
<p>To take these commits and arrange them into commits for publication, I'll use the <code>rebase</code> command. This also presents me with an opportunity for <strong>deliberate practice</strong> with rebasing, so that when I need to use it for realsies it's scary and undaunting.</p>
<ul>
<li>First, I'll make sure my Git's understanding of the <code>origin</code> repository is up to date with reality: <code>git fetch origin</code>.</li>
<li>(Let's say I branched from the <code>develop</code> branch). I'll then make sure my changes are applied on top of what's in the base branch: <code>git rebase origin/master</code>. This will replay every commit one-by-one, asking for fixups if there are any conflicts.</li>
<li>I'll check everything works, then perform an <strong>interactive rebase</strong> on all of my unmerged (to <code>develop</code>) commits, so that I can tidy them up: <code>git rebase -i origin/develop</code></li>
</ul>
<p>Interactive rebase is a mechanism for you to instruct git what to do with each individual commit you have. A text editor will open with a list of commits and explain the options available. You can, amongst other things:</p>
<ul>
<li>keep a commit as is (<code>pick</code>, or <code>p</code> for short)</li>
<li>keep a commit as is, but <code>reword</code>/<code>r</code> its commit message</li>
<li><code>squash</code>/<code>s</code> two commits into one (combining both their commit messages too)</li>
<li><code>fixup</code>/<code>f</code> a commit - this squashes it onto the previous commit, but discards its message</li>
</ul>
<p>I'll do some thinking as to what the commits are, and then us a combination of reword and fixup to build out the commits that I want. Once you save and exit the text editor, the instructions are executed one-by-one on the commits, in the order specified.</p>
<p>In the text editor that's opened up when performing the interactive rebase, you can also re-arrange the commits. This is really useful if you added a "quick fix" commit to something and want to combine it into the original commit where that should have been made, but that commit is several commits back in the history. Sneaky!</p>
<h3>Making the Rebase Experience Nicer</h3>
<p>Note, to make this a bit easier to work with, I have the following set in my <code>~/.giconfig</code> file:</p>
<pre><code>[rebase]
abbreviateCommands = true
</code></pre>
<p>This turns:</p>
<pre><code>pick f7f3f6d Change my name a bit
pick 310154e Update README formatting and add blame
pick a5f4a0d Add cat-file
</code></pre>
<p>Into:</p>
<pre><code>p f7f3f6d Change my name a bit
p 310154e Update README formatting and add blame
p a5f4a0d Add cat-file
</code></pre>
<h3>Being Paranoid With Changes</h3>
<p>One of the interactive rebase options, on top of the ones above, is to "exec"/"x" a command. You can manually add one of these after each commit in the interactive rebase to, amongst other things, build and test your code. But doing that is time-consuming and boring.</p>
<p>So let's say that running the command <code>npm test</code> builds and executes our unit test. We can automatically run this command after every commit in our interactive rebase without having to manually add it, by adding it to the <code>git rebase</code> command:</p>
<p><code>git rebase -i -x "npm test" origin/develop</code></p>
<p>Now, if a <code>npm test</code> fails, the rebase stops at this point. I can investigate, make any changes and <code>git rebase --continue</code> when I'm happy.</p>
<h2>How Do You Do It?</h2>
<p>This workflow has suited me well for several years. I'm not saying the way I use git is right or wrong. It's just what I do. I'd love to hear about how you use git, or any improvements to my workflow. Please do get in touch!</p>
NE-RPC - It's Happening Again!2020-08-25T00:00:00Zhttps://samhogy.co.uk/2020/08/ne-rpc-its-happening-again/Going back to school, virtually!<p>Following the success of the <a href="https://ne-rpc.co.uk/2020-june/">The Inaugural NE-RPC</a> back in June, I'm excited to announce the agenda for our follow-up event in September: <a href="https://ne-rpc.co.uk/">NE-RPC: Back 2 Sk00l</a>. Tickets are free and available now. We're streaming Friday 25th September!</p>
<h2>Back 2 Sk00l</h2>
<p>The thinking this time around was to theme the event. Given the time of the year and the ongoing debate/debacle regarding the return to schools following the COVID-19 lockdown, we though the concept would fit nicely.</p>
<h2>Iterative Improvements</h2>
<p>We've learned a lot from the previous event and had some great feedback from participants and speakers about the overall organisation of the event. It's great to hear, but it kind-of managed itself.</p>
<p>We set up a Discord group, with all ticket-holders getting an invite. This acted as the 'hub' for the day. We had chat rooms for each of the two streams and would post links to the talks and links to join the Zoom Q+A session in there. Having a central hub allowed us to replicate some of that conference feel, by hosting (or kicking off) discussions and providing a means to give continuity announcement to the day.</p>
<p>All videos were pre-recorded and uploaded to Youtube in advance, allowing us to set a 'go live' time. That meant we had little to tinker with on the actual day, as Youtube itself would facilitate the publication of content. There were a few teething issues with this - we didn't realise that prior to a video going live, Youtube displays a countdown video. The volume of that was significantly higher than that of the talks themselves. To correct this we're going to provide further sound normalisation instructions to speakers in September. In one case, we had a session whose audio got out of sync with the video. Turns out this was beyond our control, but it's down to how YouTube actually "live streams" the video when it premieres. Replays of the video, or watching on a delayed feed, didn't experience this issue. Weird but bugs happen.</p>
<p>One of the successes the day was the ad-hoc Lean Coffee session we arranged, days before the event. We'd deliberately allocated a long lunchtime so that those balancing work, childcare and house chores could still get other stuff done, but it gave us an opportunity to bodge in a lean coffee session without timetable changes elsewhere. This time around, we're allocating proper time for it, so that people don't have to make a choice between lunch and lean coffee.</p>
<h2>I'm Speaking Too</h2>
<p>Being on the panel and having a session selected (by the others) is a bit weird, I must admit. For the selection of this talk I was clear that I'd only take the place if the talk stood on its own merits, everything else considered. Thankfully it did, so I'll be giving an outing to a new talk - "Testing that the code does what the code does" - covering TDD principles, where it's gone wrong and what we can do to avoid the common pitfalls and traps.</p>
<h2>See You There?</h2>
<p>All the videos will remain on Youtube after the event, so you can still catch up even if you can't join us on the day, or for the full day. However, I do hope to see you at the event!</p>
Quality is a Team Activity2020-08-22T00:00:00Zhttps://samhogy.co.uk/2020/08/quality-is-a-team-responsibility/Why telling testers to stay in their lane harms your team's ability to deliver work!<p>In my last article, <a href="/2020/08/going-with-the-flow/">"Going with the Flow"</a>, I argued that building functional silos around different disciplines hinders our ability to deliver value. In particular, testing shouldn't be seen as a separate activity involving a handoff. Instead we should seek to eliminate handoffs by working together. Since then I've been doing more thinking about the role of testers on a team. Having worked on 'agile' (and supposedly 'agile') teams for a decade, I've noticed the same frequent patterns of behaviour. Why is it testers are treat lesser to the devs? Why do the testers feel stressed and pressured? Why do arguments about the nature of testing break out between the testers and the business, in a way that rarely happens between business and devs?</p>
<p>My suspicions are that this stems from how we see testing as separate - in terms of <em>who</em> performs testing, <em>why</em> we test, and <em>where</em> we see it as an activity within the overall software development lifecycle.</p>
<p>Let's delve into it.</p>
<h2>Juking the Stats</h2>
<p>Commonly, 'agile' project tracking tools such as JIRA are set up in a way that enforce handoffs. Projects set up with separate swimlanes for "development", "code review" and "testing" are a prime example of this. They're activities performed at different times by different people. This makes me uncomfortable, because it's similar to the concept of <a href="https://www.youtube.com/watch?v=_ogxZxu6cjM">"juking the stats", as explained in the Wire</a>.</p>
<p>From a team perspective, it allows us to continue to enforce Work In Progress (WIP) limits, whilst also making it seem like we're doing more. However, this comes back to bite us because we end up committing to too much. Projects set up this way end up encouraging poor practice among the teams. In particular, I've observed all of the following through my career:</p>
<ul>
<li>A developer can fling an item into code review and immediately start working on something else. But that item in code review is not <em>done</em> - so why is something new being started?!</li>
<li>The need to salami-slice work, so that everyone is working "to capacity". At any point, we all need to be working on our own ticket. This prioritises individual productivity, disregarding and compromising the ability for the team to focus as a whole on getting something fully developed to completion.</li>
<li>The requirement to "project manage", because when developers prioritise new work over code reviews, code reviews never get done.</li>
<li>The introduction of the term "dev complete", which creates the logical fallacy that the work is done pending bureaucracy. Therefore, testers become a barrier to getting things done, being just another hoop to jump through.</li>
<li>Testing is seen as a "bottleneck" which itself needs to be "project managed". Typically developers outnumber testers, so when we prioritise everyone working to capacity, testers are overwhelmed by having to test lots of tickets at the end of a sprint.</li>
<li>Testers getting the blame when things don't get done, because their activity occurs later on the path to "done".</li>
</ul>
<p>Nowadays, I prefer to simply have an "in progress" column that encompasses development, testing AND business sign-off/UAT activity. This simpler approach reduces the swimlanes down to "To Do", "In Progress" and "Done". It helps us maintain our focus and WIP limits, ensuring that as a team we work together to finish stuff that we start, before starting on new stuff. Project managers who want "visibility" can still get this by <strong>speaking to the team</strong>.</p>
<p>This is a dire, but all-too-common state of affairs. We do our testers harm by rigging the game like this.</p>
<h2>Testers as QAs - Quality Advocates</h2>
<p>Thinking about this more, if my argument is for work to be treat as "in progress" until it's done, what is the role of a tester on a team? Fair question.</p>
<p>I've spoken to many senior IT bods who fundamentally view testing as a cost-sink activity that rarely delivers the "assurance" of quality they're (not) paying for. This thinking comes from experiences with offshore testing, especially when performed with cheap labour in a rote-based manual fashion. We all understand these folks are committed to quality and are doing their best, but when testing is seen as something that gets in the way and rarely delivers results, it's difficult to come to the conclusion that quality is being under-invested. Instead, they'll try to make the entire department redundant by "automating the testing".</p>
<p>Automation is a good thing, for what it's worth. Automation provides fast, reliable, repeatable, auditable feedback. Compared to manual, rote-based activity conducted via a hand-off to another department, automation is a leaner model for delivering value. We should recognise though that automation works well for some activities, but not all. Laborious tasks that are repeated frequently and require precision checking are prime candidates for automation. Computers don't get tired or change-blind. Computers aren't creative and so we still need that human touch of ingenuity on top of our automation packages. This 'exploratory' testing is a high-value activity that requires a deep contextual understanding of the product being developed and the environment in which it operates.</p>
<p>That's the crux of it, really. Testing, when under-invested, becomes something that rots to the point where we pay for it because we have to, in order to get sign-off. If you see it as a cost-sink rather than a value-add then you're incentivised to reduce costs. You need to pay for quality.</p>
<p>Quality cannot come simply from one discipline. Although used extensively in this post, I don't like the term tester. I prefer QA. However, I don't mean QA in the traditional sense of <strong>quality assurance</strong>. Quality cannot be assured. It can only be advocated for. Therefore, I see QA as meaning <strong>quality advocate</strong>, not quality assurance.</p>
<p>(Incidentally, I'm not the first person to suggest this. <a href="https://alisterbscott.com/kb/quality-advocate/">Alister B Scott proposed the same idea 7 years ago</a>, demonstrating that someone always beats me to an idea!)</p>
<p>Quality Advocate more accurately describes the role being performed on the team. The QA may perform certain testing activities, but so will a developer, and so will the product owner. A QA champions quality, empowering the team to develop skills and put them into practice. The QA will care very deeply about building a high-quality product and seek to coach the team. They can advise on where automation is best-placed, they question if we can get faster feedback, they challenge our ways of working.</p>
<p>The tester is not there as the safety net, catching bugs before they go to production. They're questioning why you need the proverbial 'safety net', when a belay system and good rope discipline will stop the person falling in the first place.</p>
<p>Just as professional sports players and musicians have coaches, so do professional software development teams. It's not a sign of failure, it's a sign that the team is supported to self-improve their ways of working.</p>
<h2>Mindset -> Frame of Reference</h2>
<p>I take umbrage to the widely-circulated argument that testing is a "mindset" which you either have or don't, and that this is why some people are testers and some people are developers. This is complete and utter bollocks.</p>
<p>A mindset, like a culture, is not a fixed thing. As mentioned in my previous post, culture is a lagging, not a leading indicator. The sample applies to mindsets.</p>
<p>"Testing is a mindset" risks the subtext of "you don't have it, stay in you lane". That's not empowering at all. At large, it causes all the problems described above, because testing becomes a separate thing done by separate people at a separate time.</p>
<p>It can't be true though. As a developer I write tests every day, so how can I <em>not</em> have <strong>the</strong> testing mindset? It just doesn't make sense when viewed this way. A mindset is not some innate quality that you possess by gift of birth. It's a lagging indicator because it is indicative of your <em>frame of reference</em> - what your training and experience has led you to. Developers have deep understanding of programming languages, development frameworks and whatnot because of the work they've done and the training they've received. Similarly, testers have a deep understanding of the common causes of failure in applications because of their backgrounds. Both are entirely learnable.</p>
<p>Some express concerns with the notion of individuals in teams having "specialisms". But viewed in this way, it makes sense that someone with a test-focused frame of reference can be well-equipped for exploratory testing, just in the sense the database person is well-equipped for fine-tuning the application's database performance. They're all important roles.</p>
<h2>Testing is Everyone's Responsibility</h2>
<p>Testing is <em>already</em> an activity everyone on the team commits to. Developers should be writing a lot of tests: unit, integration and acceptance. QAs champion quality within the team, but may also allocate some of their time for exploratory testing sessions. They're well-placed to do this not because of their "mindset", but because of their <em>frame of reference</em> as advocates of quality. The notion of testing a ticket in isolation blurs away to a ticket being "in progress", with quality being built in. Testing is a <strong>value-add</strong> activity, rather than a cost-sinking bureaucratic hurdle. As a continuous activity throughout the software development lifecycle, it doesn't fall upon one person's (or discipline's) shoulders. Rather, it's the responsibility of everyone in the team.</p>
<p>To the QAs I've worked with in the past - I'm genuinely sorry for this state of affairs, and as a developer I commit to empowering you to become advocates of change within your teams.</p>
Going With the Flow2020-08-17T00:00:00Zhttps://samhogy.co.uk/2020/08/going-with-the-flow/Why we shouldn't be cramming sprints to capacity. We should be focusing on getting stuff done.!<p>There are a variety of hoops that need to be jumped through in order to get software from someone's head to someone's screen. Hoops such as "build the thing" and "test the thing" and "sign off that the thing can be released". Historically we've fallen into the trap of making these separate activities happen at separate times - one after another. After all, how can you test what hasn't been built? From these seemingly discrete activities, organisations have built silos around each 'discipline' leading to a waterfall of work cascading between these silos, but not much getting delivered.</p>
<p>At the turn of the Millennium, the Agile Manifesto was published. Sure, it wasn't the first document to espouse the ideas within. It drew upon decades of practice, from the Toyota Lean Manufacturing ideals, Kent Beck's Extreme Programming and even further back, to the Scientific Method of empiricism. To me, this was akin to a modern-day Martin Luther <a href="https://en.wikipedia.org/wiki/Ninety-five_Theses">pinning his 95 Theses to a church door</a>. Just like those demands begun the Protestant Reformation, so did the manifesto fuel the Agility Revolution.</p>
<img src="https://samhogy.co.uk/assets/old-man.png" alt="Old Man Yells at Cloud (from The Simpsons)">
<p>Out with waterfall, in with Agile. Capital A 'Agile', that is. One Agile please. <a href="/2019/09/agile-is-not-a-solved-problem/">We treat Agile like a solved problem, but it isn't</a>). There's frameworks like Scrum and SAFe and even this Kanban thing which tell us how to build software, so in they came. Unfortunately, this doesn't solve anything <em>by itself</em>. When it doesn't turn out as expected it's written off as not having enough buy-in, requiring too large of a culture / mindset change. Does this ring true? Do you feel seen?</p>
<p>Truth be told, culture changes are a lagging, not a leading indicator. If we wait for the culture to change, we're allowing perfect to be the enemy of good. We need to be the change we want to see in the world - as only by doing this can we <a href="https://www.franklincovey.com/the-7-habits/habit-1.html">extend our influence outwards</a> and concern ourselves with larger cultural shifts.</p>
<p>How do we go about this then? How do we do the agiles so that we can get the desired results? The sage advice is "you need to find out for yourself", because it's in the journey where the learning takes place. This of course, is why we can't simply off-the-shelf an Agile solution. It doesn't mean you can't be pointed in a starting direction though, so I shall do that.</p>
<p>Focus on that <em>flow</em> of value from idea to delivery. Make it lean.</p>
<h2>Mapping Value Streams</h2>
<p>Agile processes advocate the delivery of working software to users frequently in small increments. Compared to the 'big bang' of waterfall, frequent delivery reduces risk. It provides the ability for rapid experimentation. For many organisations, getting to a position of frequent delivery will require taking a holistic look at their software delivery processes and successively refining them to be lean by removing waste. <a href="https://en.wikipedia.org/wiki/Value-stream_mapping">Value Stream Mapping</a> describes this process. By plotting out the organisational and developmental activities that take place to build software, we can begin to understand where waste accumulates in the process. Removing this waste makes the value stream more efficient.</p>
<p>More efficient value streams enable agility. Fast feedback throughout the lifecycle provides a competitive advantage by removing the time to market and creating a culture focused on its customers, rather than its own internal functional silos.</p>
<h2>Reduce Handoffs, to Reduce Waiting</h2>
<p>When one silo completes its piece of the puzzle - whether that's front-end development, back-end development, testing, security etc - the work needs to be handed off between these silos for it to move along the path to delivery. No individual silo is responsible for delivering value, they're each responsible for their bit. Then we hire managers to oversee this process, championing work under their stewardship over other priorities within each silo.</p>
<p>Let's put this in practice with an example. Consider an organisation whose work is arranged into sprints of two weeks. However, their developers and testers work in different, staggered sprints. The devs complete a sprint, build their integrated work and hand it over to the testers for a sprint of testing. Although the work is superficially taking part within an 'agile' framework - it's clearly still a waterfall process oriented around functional silos. While the testers are looking at the previous (dev) sprint's worth of features, the devs are off doing something new entirely.</p>
<p>This methodology seems nice if you're interested in having the people in your silo working to capacity. However, what value have the developers produced at the end of their sprint? Nothing. None of their features will be delivered to production. It doesn't matter if they're planning their sprints using story point velocity. Only functionality delivered to production can be considered 'done', and nothing's getting done if you're not aligned to that goal.</p>
<p>You may think it seems unfair, and it is. It's unfair to teams further to the right of the process who inherit all of this work, who operate closer to deadline day, who've lost contextual information because of the handoff between silos. They suffer the cascading snowball effect and receive all the blame when things go wrong. The handoff has introduced waste into the process, which slows us down and affects our ability to get stuff done. To see why, we need to additionally look at waiting times.</p>
<p>When a tester finds an issue with a given feature, it needs to be raised back on the developer queue of work to be looked at. Even if it jumps right to the front of the queue and is worked on immediately, that overall feedback took a minimum of one sprint - 2 weeks - of waiting to be received. That's because when a developer completes a feature during a sprint, it isn't looked at by a tester until the start of the <em>next</em> sprint. Between these two dates the work is sitting idle, making no progress towards providing value to end users.</p>
<p>It's clear when viewed holistically that handoffs between teams introduces waiting, which indicates process inefficiency. For instance, each silo has their own queue of work. They're not aligned on the <em>same</em> queue and seeing it through to production!</p>
<p>So an improvement to this would be to align the teams to the one value stream, working within the <strong>same sprint</strong> on the <strong>same stuff</strong> - all aligned towards a common <a href="/2020/02/sprint-goals/">sprint goal</a>. By doing this, when a developer completes a feature they can get a tester to look at it immediately. Even better, they could work together! (I know, imagine that! People speaking to each other!) The handoff is reduced, the waiting times eradicated.</p>
<p>But what about my capacity? Won't this mean devs sitting idle towards the end of a sprint and testers sitting idle at the beginning? It might. Or it might not. It probably won't. Remember that we're not in the game of stacking each individual to capacity. We're in the game of delivering value to our users. Telling team members to stay in their lane doesn't happen when the team is aligned to the same goal. They all chip in, doing what's necessary to see the work get done. I'm sorry this doesn't align to your expectations of each person must be working on their own unique task.</p>
<h2>A Safe Framework for Experimentation</h2>
<p>When we achieve this continuous flow of value to users we open up a competitive advantage. We can get features to market quicker, gaining feedback sooner. We don't suffer the sunk cost of having to build the whole thing before we ship it and find out it's not what's wanted. Rather, we can perform small experiments and trim what doesn't work early, before it's broken our budgets. The flow from left-to-right then becomes a mechanism for learning. We've built ourselves a laboratory, where we can develop a hypothesis, put it to the test and evaluate the results. We're utilising the most powerful algorithm known to humanity - the Scientific Method. When we have full control of our ability to deploy to production, we can even target these experiments on a smaller user base. A/B testing lets us experiment with varying implementations. Feature toggling and gradual rollouts let us beta and battle-test functionality on real users before releasing en-masse. If this sounds exciting, it's because it is!</p>
<h2>That's the Power of Flow</h2>
<p>The ideals behind agility, devops, bizdevsecops (whatever we're calling it now) all point towards the same ideal. That is, to speak in Geordie, not "hoying stuff over the wall", but instead being aligned to delivering value to customers. We do this by breaking barriers, by talking to each other and by learning from each other. Maximising flow should be the top aspiration for all organisations. No longer should the 'IT Department' be following a contractor-based model, providing a fixed-scope service to the business in exchange for a fixed sum of money. Instead, the development, operations and business teams should all working together, in full alignment.</p>
UK Gameshow Theory2020-08-14T00:00:00Zhttps://samhogy.co.uk/2020/08/uk-gameshow-theory/In which I ramble about the surprising depth to some old UK gameshows!<p>Just for fun, in this post, I'm going to look at some classic UK gameshows and see what we can learn from how they were played. As I'm not a mathematician, I won't be going into the underlying decision theory and game theory mathematics, instead preferring plain-English.</p>
<h2>Deal or No Deal</h2>
<p>Deal or No Deal had 22 contestants, each of whom was assigned a sealed box. Within each box was a unique prize amount, increasing from 1p to £250,000. One of these contestants is selected to play the game. Over the course of several rounds, they ask the other contestants to open their box, revealing its value. At the end of each round, the contestant would be given the opportunity to 'sell' their box to 'the banker'. Ultimately, they will win the amount contained within <em>their</em> box unless they sell at an earlier point in the game.</p>
<p>The offer made by the banker was usually just above or below the average of the remaining cash values (although clearly the values were fudged for entertainment and storytelling purposes). The game, then, was one of trying to get the best value for the box based on the information available. Do you sell a worthless box for a decent amount, or the high prizes for pennies?</p>
<p>(The contestants and presenter, out of entertainment, genuine belief, or both, frequently ascribed spiritualism and mysticism to the gradual opening of the boxes. Ultimately they were simply randomly opening boxes.)</p>
<img src="https://samhogy.co.uk/assets/noel.jpg" />
<p>This shows that the value of an asset, which remains unknown, is judged based relatively to the available information in the marketplace. Now of course, this is a special case because there's a range of specific and unique values, but we can still learn from this. Whether to sell or continue at the end of the round depends upon the contestant's adversity to risk. And that this adversity to risk is context-specific - not only are you on TV, but your decision-making risks being 'anchored' by the sequencing of events. A good run of removing entirely low-value boxes, or a bad run of the opposite, can influence whether a contestant decides to push their luck or cash out. By doing so, they're over or under-valuing their asset.</p>
<p>You see the same type of thinking down at the bookies on the roulette machines. The gambler will say "a red's due because I've had five black spins". This isn't the case. Each box opening is random, and independent inasmuch as the value will be one of the discrete prize amounts remaining.</p>
<p>Crucially, when there was only one other box remaining, the banker would offer a swap. You may be familiar with the Monty Hall problem. Based on the 1970s US gameshow "Let's Make a Deal" - the problem is traditionally stated as such:</p>
<blockquote>
<p>Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?</p>
</blockquote>
<p>Your probability of choosing the correct door is 1/3. As the host has revealed an incorrect door, it is common to think that the "stick or swap" choice offers a 50/50 chance. However, it doesn't. Counterintuitively, by swapping you have a 2/3 probability of winning.</p>
<p>The original choice you made still carries the 1/3 probability of being correct. Out of the two remaining doors, you were shown a door that was a losing door. This leaves one door, which like your original choice, could still be a loser. However this remaining door contains <strong>all of the remaining probability</strong> for the door being correct. That's 2/3.</p>
<p>Deal or No Deal's final swap offer <strong>isn't</strong> a Monty Hall problem, because neither contestant nor the banker know what is in each box. It's still 50/50.</p>
<p>There's a lot to learn from the mechanics of Deal or No Deal and thankfully none of it is Noel's electrosmog nonsense. For me, the key takeaway is that the information that's available can be used to inform better decision-making, but beware of local minima and maxima!</p>
<h2>Golden Balls</h2>
<p>Jasper Carrott presented Golden Balls in the late 2000s. The game focused on collaboratively building a cash prize, while eliminating contestants - normally for their untrustworthiness, sometimes because they've been scapegoated.</p>
<p>The final round of the show was called "split or steal", taking place between the remaining two contestants. Each contestant had two choices - to split the prize money or to steal all of it for themselves. They'd have a time-limited opportunity to discuss strategy, before simultaneously revealing their choices. The rules were:</p>
<ul>
<li>If they both choose to split - the prize money is shared equally</li>
<li>If one steals and the other splits - the stealer wins all the money</li>
<li>If they both steal - nobody wins the prize money</li>
</ul>
<p>Classically this problem is presented as the Prisoner's Dilemma. Two gangmembers have been arrested on lesser charges. The prisoners, whom do not have the ability to communicate with each other or discover the other's choice, are separately interviewed and offer to 'betray' their co-conspirator. If they do so, they're let free while the other can be convicted on more severe charges. The rational, self-interested thing to do is to opt to be set free by betraying. However, if both betray each other then they'll both serve the higher sentences. What plays out in reality is that humans bias towards the <strong>co-operative</strong> behaviour, in which the parties independently choose to work together (by not betraying each other).</p>
<p>Golden Balls plays this out for entertainment value - with the difference being that the contestants <em>can</em> confer during the discussion. So clearly, we're not dealing with a traditional Prisoner's Dilemma problem space. It was fun watching the contestants agree to split (or try to convince that of course they wouldn't <em>dare</em> steal). One contestant though played an incredibly interesting and now infamous game:</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/S0qjK3TWZE8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>This remarkable play turns the prisoner's dilemma on its head in a captivating way. By revealing information, Nick has disrupted the ability for Ibrahim to influence the game. Nick immediately communicates his desire to steal (with a promise to split privately after the show ends). There's no point to Ibrahim stealing because the two steals will cancel each other out. His only choice to win any money is to comply by opting to split. Ibrahim still has to take it on trust Nick would privately split after the show - however he'd have to take it on trust they'd both choose to split. The host mentions there's no legal requirement, which is true, but doesn't change the situation. Ibrahim can only win some money by splitting - he cannot win it all by stealing.</p>
<p>Ultimately, Nick was lying. He chose to split. The genius in the play was to present a situation in which the only rational choice that his opponent could make was to split. That puts the output of the game directly in his hands. Note he could have chosen to steal at this point, and the rational, self-interested line would be to do exactly that. But Nick is a nice guy and split - once again showing that human angle to co-operate.</p>
<hr>
<p>In both cases above I've focused on how the availability of information affects (positively or negatively) our ability to make decisions. Most gameshows in the UK are simple quizzes, but now and then we do get a show which gives us a surprising amount of depth to decision-making and game plans!</p>
Overcoming FUD When Improving Processes2020-07-01T00:00:00Zhttps://samhogy.co.uk/2020/07/overcoming-fud-when-improving-processes/Focus today on the problems you have today. Focus tomorrow on the problems you may have tomorrow.!<p>Changing ways of working is hard - it's why nearly two decades after the publication of the Agile Manifesto we're still talking about it. Yes, you read that correctly; we're getting old. After all, <a href="/2019/09/agile-is-not-a-solved-problem/">Agile is not a solved problem</a>, it's a framework for stepwise process improvement.</p>
<p>Resistance to process improvements come in many forms. A more insidious form is that of Fear, Uncertainty and Doubt. FUD stops change from getting out the blocks. It can manifest in many ways, but for argument's sake let's use the following example:</p>
<blockquote>
<p>In response to a proposed change to a way of working, a team member expresses concern that this will introduce another problem. They refuse to commit to the change until that concern too is addressed. Rinse and repeat.</p>
</blockquote>
<p>It is not helpful to dictate changes to process. All team members should buy in to a change. For the skeptical in the group, they may need some convincing. Their FUD is most likely coming from somewhere, such as a prior bad experience. Bringing this team member along means we cannot simply dismiss their contribution to the discussion. Acknowledging their concerns have been recognised is important for psychological safety - raising concerns, however well-founded, requires a level of trust and goodwill. You wouldn't want to recklessly erode that as it will make the future more difficult. Healthy skepticism should be encouraged, as through questioning we can challenge assumptions, refine our understanding and gain clarity on our goals.</p>
<p>So, what can we do?</p>
<p>The proposal to change the ways of working is coming from somewhere - most likely an actual problem the team is currently experiencing. Reiterating that this is the thing that is currently trying to be fixed will refocus the discussion. Problems that may come about as a result of making this change are, at this stage, entirely hypothetical. They may, or may not happen. In reality, we won't know until we've tried it. We can dismiss this as counterfactual, but it's more useful to acknowledge these concerns and the uncertainty around our solutions. This is why we need the ability to experiment! It will take trust on both parts - knowing that the team is invested in improving itself, that we don't have all the answers up front, and we may not get it right on the first try. Never have I seen a magic one-size-fits-all solution that can be applied without negative consequences.</p>
<p>The 'disagree and commit' management philosophy aims to ensure that disagreements are aired. This discourages groupthink, allows teams to consider alternative viewpoints, but ensures that a lack of consensus doesn't lead to inaction. It does ask that when the team makes a decision, all the members align to making a success of it. Critics say this introduces cognitive dissonance, but thankfully that can be alleviated by the agile processes frequent <strong>inspect and adapt</strong> checkpoints. These let us course correct based on learning. Just as easily as we can start doing something, we can stop. This helps with the buy-in as it's not necessarily for the long term (unless it works).</p>
<p>Software developers will be familiar with Donald Knuth's infamous quote:</p>
<blockquote>
<p>Premature optimization is the root of all evil</p>
</blockquote>
<p>He was talking about spending too much time in the weeds of small optimizations, instead focusing on the more critical areas. We can generalise this statement further to encompass making decisions ahead of time when we're not fully informed. It's far better, and more efficient, to defer decisions until as late as possible. An additional problem introduced by a change may not have as much impact as the initial problem. Even if it eventually does have more impact, it's still theoretical until it's manifest, so we can't even measure this up front. By trying to solve it in advance and in an uninformed manner, we're essentially gambling on something that might not happen. Wouldn't you rather observe, listen, learn and <em>then</em> decide?</p>
<p>That's the power of agility. The more we do it, the better we get at it. The better we get at it, the more we stop letting FUD cause our teams to rot through inaction. Focus today on the problems you have today. Focus tomorrow on the problems you may have tomorrow.</p>
The Office Post COVID-192020-05-22T00:00:00Zhttps://samhogy.co.uk/2020/05/the-office-post-covid-19/Calling time on cramming people into open-plan offices, reorganising our work-life balance and making offices spaces for meaningful inter-personal interactions.!<div class="callout">
<p>The contents of this article were factored into a micro-talk, <em>The Next Normal</em>.</p>
<p><a href="https://samhogy.co.uk/talks/the-next-normal.html">📹 Watch it here!</a></p>
</div>
<p>COVID-19 has normalised working from home in a way unthinkable prior to the pandemic. We now have the opportunity to radically rethink how we conduct our work, ensuring that our spaces work <em>for</em> us, not <em>against</em> us.</p>
<p>Before diving into how we can go about this, we should acknowledge that this is not "normal" working from home. This is something mandated, rather than a voluntary choice. Aside our work we're having to juggle childcare, our partner's work schedules and lining up at the shops for loaves of bread. Pre-lockdown levels of productivity cannot be expected. Deadlines set prior to the pandemic but not revisited since are worthless. Call them out!</p>
<p>The lockdown has given us the opportunity to familiarise ourselves with the tech, explore how to work remotely and forced the hands of employers to strengthen their working from home setups. Us in tech community might not have seen much of a difference, but in other industries this has involved a significant effort just so that businesses can keep the lights on.</p>
<p>Uncertainty reigns supreme over the timetable for returning to 'normality'. Questions exist as to whether the old 'normality' is even possible or desired. Politically a lot of topics well outside the Overton Window are back up for grabs. I think that the opportunity is now present for a large rebalancing of <em>how</em> we work. I will explore:</p>
<ul>
<li>why I think the "old way" of sitting in rows in an open office is harmful</li>
<li>why we shouldn't return to this any time soon</li>
<li>how we can better utilise these spaces to work <em>for</em> us, not <em>against</em> us.</li>
</ul>
<h2>The Old Open Plan Office</h2>
<p>The idea was to tear down barriers, to allow for collaboration - but this didn't work out as planned. The space changed us. <a href="https://royalsocietypublishing.org/doi/10.1098/rstb.2017.0239" title="The impact of the ‘open’ workspace on human collaboration">Face-to-face communication decreased significantly, with electronic interactions taking over</a>. If someone's looking like they're busy, they won't be deliberately interrupted. That said, distractions in an open plan office are commonplace - another team's loud phonecalls or discussions, for example - leading many to purchase noise-cancelling headphones and further withdraw. Focus-based activities have a cognitive cost - if you get distracted, it takes a while to find your groove again. <a href="https://onlinelibrary.wiley.com/doi/10.1002/9781119992592.ch6" title="The Physical Environment of the Office: Contemporary and Emerging Issues">One meta-study showed consistent negative impacts</a> on creative thinking, productivity, stress, motivation and concentration.</p>
<img src="https://farm3.staticflickr.com/2899/14199370746_560cd25b82_b.jpg" width="800" alt="An Open Plan Office">
(Image by [K2 Space](https://www.flickr.com/photos/123056804@N04), licensed [CC BY 2.0](https://creativecommons.org/licenses/by/2.0/?ref=ccsearch&atype=html))
<p>So why did we continue to persist with this way of working? There's two reasons I can think of:</p>
<h3>Seen to be Doing Work</h3>
<p>Bentham's Panopticon prison was designed to be a space-efficient means for a security guard to observe all prisoners, without a single prisoner knowing whether they were actively being observed. Architecturally, the structure enforced compliance. Less dramatically, the same concept appears within open plan offices. When you're physically present, you can be "seen" to be working by your colleagues and managers.</p>
<p>Whether you're working on the right thing, or whether you're missing out on other opportunities is difficult to quantify. This can manifest in subtle ways, but the idea that an employee is not being productive because they're reading a book or watching a training course, can discourage those kinds of activities from being performed. Ultimately this is a short-termist practice, discouraging innovation.</p>
<h3>Sardines</h3>
<p>Office space is expensive, especially in city centres. Positioning your office at a transportation nexus has a strategic advantage, enabling employees to easily travel from home to the office. These two competing forces then balance out with wanting to get a return on investment from the office space - a high yield of employees who will earn money. Open plan offices are the optimal usage of the space, with their rows of desks and lack of boundaries. You get the most bang for your buck; the most sardines in the tin.</p>
<p>I've worked with organisations who've "downsized" and saved on cost by cramming into open plan offices and enforcing desk rotas. Staff would work from home so that the desk space could service multiple members of staff over a week. The result? There was little to no personalisation of the space. You didn't "own" this space, you existed in it.</p>
<h2>Remote By Default</h2>
<p>So now we're working from home, more or less, let's look at what we'll need to do as lockdowns are restricted. It's looking likely that social distancing in some or other will be with us for quite a while. Returning to the big open plan offices is essentially a non-starter.</p>
<p>Large companies like Facebook and Twitter are acknowledging this new state of play by shifting to "remote by default" working practices. The days of travelling in a box to sit in another box with other people, to do work we now know you can do in your own box, are over. We have the means, motive and opportunity to create a happier, healthier and more productive working environment for all staff - giving them the flexibility to work in the ways that they need.</p>
<p>Embracing home working gives you the ability to have focus time. If you need to get your head down and do something - you're not going to have the unnecessary distractions. If you need to read a book, go through some research papers, catch up with some virtual conferences (plug: <a href="https://ne-rpc.co.uk" title="NE-RPC">NE-RPC</a>) or Pluralsight/Safari training courses, you can plan your day to accommodate for these long-form activities. You can do so knowing the panopticon won't be criticising your work choices.</p>
<p>It gives you the chance to <strong>own</strong> your space and customise it. My office has a whiteboard, photos, a minifridge and a sofabed (yes, I'm privileged to have the space; your mileage may vary, etc).</p>
<p>Finally, the commute is much easier. You're giving staff the gift of time. More time with the family. More time for leisure. More time to get the housework done. This is a considerable improvement to work-life balance that people won't want to give up.</p>
<h3>The Virtual Water Cooler</h3>
<p>You're not a one-person army. Despite working from home, you're still part of a team. Developing the appropriate techniques to still work effectively as a team takes time. Embedding these practices into the company culture won't be doable in a day. I've found the following things useful:</p>
<ul>
<li>Switching many email, or text-based conversations into video calls</li>
<li>Turning the webcam on by default (and encouraging others to do so)</li>
<li>Doing code reviews as video calls</li>
<li>Pairing on features over video calls!</li>
<li>Prefixing or suffixing calls with a bit of personal chat. Simple things such as asking how the day is going builds morale</li>
<li>Putting on a group chat when I'm wanting a break. This often spins up "ad-hoc" video calls where the team can socialise together</li>
<li>Encouraging Mario Kart tournaments on the Nintendo Switch (other game systems and racing games are available)</li>
</ul>
<p>A balance still needs to be struck between being private and being present.</p>
<h3>When We Still Need In-Person Meetings</h3>
<img src="https://live.staticflickr.com/3161/2336990347_6b3604ee1c.jpg" alt="Kanban">
<p>(Image by <a href="https://www.flickr.com/photos/16031249@N00">Christopher Huffman</a>, licensed <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/?ref=ccsearch&atype=html">CC BY-NC-ND 2.0</a>)</p>
<p>That said, I'm skeptical that we can do <em>all</em> work remotely. Or more correctly, that we <em>should</em>. There are some interactions that I've found to be more productive when conducted in person.</p>
<p>Agile Retrospectives are collaborative meetings which need to be performed in safe spaces. The physical boundary of the room acts as that barrier of safety. It also allows the team to feel the 'energy' of the room - reading body language, emotions, etc. Retrospectives often utilise co-operative activities to build understanding. Having physical space to conduct this and display any artefacts is vital for spotting trends and patterns from what's produced. Sprint planning and user story mapping sessions utilise space in a similar way. Physical artifacts are an output of those meetings. Moving things around the space - in scope, out of scope, high priority, low priority, grouping things together - is integral to those meetings not being boring.</p>
<p>Think on that, as we draw some conclusions.</p>
<h2>Design for Collaboration</h2>
<p>The pandemic leaves companies paying a LOT of monthly rent, and not much of a need to cram in rows of desks. How could we instead utilise this space, given our new constraints, our new desires, and a bit of blue-sky thinking?</p>
<p>The answer lies in creating environments that foster creativity and collaboration. That's going to mean private office space - for teams or individuals will depend upon circumstance. Allow for in-person collaboration through hot-desking, for those cases where you just need to prototype something or work together side-by-side. Create breakout space. Have quiet rooms, spaces for social interaction, spaces for people to create. Allow the space to be owned physically by those using them. The work will then follow.</p>
<p>Frankly, if I'm going to travel to an office in the future, I want to go there for the right reasons. That is, to spend time having interactions best suited for those environments. It's obviously clear that I can do the programming from home.</p>
<p>One-size-fits-all approaches don't exist here. Most importantly, design your spaces to promote positivie, constructive, collaborative interactions. Give your employees what they need to work in a happy and healthy way, then watch their productivity and morale increase.</p>
Announcing NE-RPC!2020-05-11T00:00:00Zhttps://samhogy.co.uk/2020/05/announcing-ne-rpc/A virtual conference in times of turmoil.!<p>I'm pleased to announce the first online event of a new North-East tech meet-up, the <a href="https://ne-rpc.co.uk">North East Remote Programming Conference</a>!</p>
<img src="https://samhogy.co.uk/assets/ne-rpc.png" />
<p>The call for speakers is now open and I'm hoping that alongside a few extra volunteers, we see some fantastic submissions! Although the Prime Minister hinted that a phased return from lockdown over the coming months is possible, I still think that there's space for virtual conferences in the fray of activity within the developer community. Regardless of what happens next month, working from home is the new normal. Ensuring that members of the community remain connected is of vital importance for our personal growth and mental health!</p>
<p>The name, NE-RPC, comes from <a href="https://en.wikipedia.org/wiki/Remote_procedure_call">remote procedure calls</a> as I was trying to think of something to indicate the distributed nature of things, and that the event will primarily be a showcase of local talent.</p>
<p>For now this is a one-off. I'll be looking at the tech side of things to determine how exactly we'll run this. Current thinking is a combination of Zoom and Twitch. Also, I'm looking for an appropriate charity that attendees can donate to in lieu of a ticket fee. This honesty system has worked well with events I've ran in the past and I don't see why it won't work here too.</p>
<p>Fingers crossed this is a success! 🤞</p>
New Horizons2020-05-06T00:00:00Zhttps://samhogy.co.uk/2020/05/new-horizons/In which I change jobs during a pandemic.!<p>I haven't blogged for a while - sorry about that! This year has been incredibly busy and being in lockdown hasn't slowed things down.</p>
<p>Since the beginning of the year I've been working on an MVP within the finance sector. It's been a great opportunity to take on a DevOps role, scaffolding the infrastructure in GCP via Terraform, crafting and honing build pipelines in Gitlab - and learning lots about observability. The wider environment in which the product was developed was interesting to say the least. As such, that engagement has ended and I'm now moving on to become an employee of Tesco Bank.</p>
<p>What a time to change jobs, you might say. But actually, the wider context of lockdown has refined my thinking around this. I've been seeking a better work-life balance since becoming a father. Finishing at 5 when you're working from home is brilliant. But finishing at 4:15 is much better.</p>
<p>It's difficult to talk about anything else in the current COVID-19 crisis. Beyond saying that I'm not happy with how the UK Government has handled the situation (too little, too late, too confused), there'll be time for reflection in the months to come. Within the software community we're lucky that in the vast majority of cases home working is trivial, normalised and relatively straightforward. Ensuring that you move around the house, get some air and exercise and not consume vast amounts of chocolate is where I'm struggling! I also miss the sporadic social interactions - those off-chance kitchen conversations, or catching someone on their way to/from the train station. We've attempted to make up for this with social rooms on video chat software. It's helpful, but not entirely like-for-like.</p>
<p>Finding things to distract you from the wider picture is also useful. A toddler is a prime example of a suitable distraction (luckily of an age where we don't have to explain what's going on), but the new Animal Crossing game on the Nintendo Switch and the backlog of wrestling videos have also provided useful crutches for temporarily escaping reality.</p>
<p>I set out to write something here, not with a particular focus in mind or an intended audience. The main message is that I'm still here, safe, healthy, working away and seeking to focus on the things that matter. Best wishes to you, dear reader.</p>
Sprint Goals!2020-02-10T00:00:00Zhttps://samhogy.co.uk/2020/02/sprint-goals/Sprints need a purpose to, well, sprint towards. Otherwise, a sprint risks becoming a mini-waterfall.!<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2020/02/10/sprint-goals.html">the Scott Logic blog</a></span></div>
<p>The youth of today are always talking about squad goals - aspirational attainments for a group. It's a nice concept and actually built into the Scrum framework, as Sprint Goals.</p>
<p>I'm personally a big advocate of having a Sprint Goal, but they seem to be the least-adopted of all the named things within the Scrum Guide. Without one, you have an implicit goal: "do all the stuff in the sprint". That's boring! It doesn't motivate, inspire or help us understand the value in what we're doing. Let's do better than this!</p>
<h2>What's the Value?</h2>
<p>A Sprint Goal provides the team with a shared <em>focus</em> - a method to concentrate our efforts. A good goal clarifies the understanding of <em>why</em> we're doing the work. It gives the team the tools to prioritise the <em>what</em> and inspires them to devise the <em>how</em>. It's a motivational tool that unites us as a team.</p>
<p>The<a href="https://www.scrumguides.org/scrum-guide.html" title="Scrum Guide"> Scrum Guide</a> itself has the following to say on the goal:</p>
<blockquote>
<p>The Sprint Goal is an objective that will be met within the Sprint through the implementation of the Product Backlog, and it provides guidance to the Development Team on why it is building the Increment.</p>
</blockquote>
<p>Note that the Sprint Goal is a singular, named term. There is one per Sprint, agreed upon by the team during Sprint Planning. This provides the shared vision and forms the commitment. As you're putting backlog items into the Sprint, you're doing so with a common purpose - does this help us meet the goal? Once everyone's happy that the work is doable (to their understanding) and are aligned that this will help achieve the goal, the Sprint can start.</p>
<p>Your Sprint Goal therefore describes <em>the</em> priority of your team. Note that priority is also a singular term. Don't believe me? Here's some useful links to persuade you!</p>
<ul>
<li><a href="https://www.quora.com/Did-the-word-priority-indeed-only-exist-in-singular-form-until-recently?share=1">Quora - Did the word 'priority' indeed only exist in singular form until recently?</a> (Spoiler: pretty much)</li>
<li><a href="https://brightonleadership.com/2016/04/05/can-priority-plural/">Brighton Leadership Group - How Can Priority Be Plural?</a></li>
<li><a href="https://dazne.net/priority/">Rishab R. Dassani - Priority is a Singular Word</a></li>
</ul>
<p>Still not convinced? Tell me what your priorities are (note: plural) when the fire alarm sounds. You're not going to instruct one person to stay behind to work on the "secondmost priority". But we try and doublespeak this away with the notion of a "top priority". A Sprint Goal can overcome this misuse of the term, aiding the prioritisation effort in a binary way - something's in the Sprint, or it isn't. During your daily standups you inspect your progress towards the Sprint Goal, adapting as necessary.</p>
<p>(Pragmatism point of order - there'll always be stuff going into a Sprint, like bugs and minor style points, that aren't perfectly aligned with the goal. That's life.)</p>
<p>Reporting is much easier too - rather than taking "the sprint board" to the stakeholders, or even worse, transcribing it into Excel and changing stuff (please don't) - you can communicate the goal itself. This is why I find it useful to keep the Sprint Goal at the elevator pitch level. Not only does it keep you out of the detail, but it clearly sets out what it is the team wants to achieve in a simple way.</p>
<h2>Setting Your Goal</h2>
<p>Now to the difficult part, actually setting a goal.</p>
<p>Don't get too distracted by attempts to find the exact magic incantation - I've witnessed teams say "keep it in a tweet" (back in the 140 character limit days), or stick to a sentence. I personally like the linguistical challenge of the latter, but that's not the purpose of the exercise.</p>
<p>Be cautious about the word "and" in a goal - that implies a split focus, or that you're possibly re-stating the sprint items. Note the difference between a task and a goal. My wife informs me that there's a useful concept at play in the education sector to help explain the difference. Within a lesson (read: Sprint) you have a series of success criteria, which seek to achieve the overall learning objective. The success criteria, like our sprint's tasks, are the stepping stones to achieving the goal. If the items in the Sprint amount to building a lunar lander, then the Sprint Goal would be to land on the moon.</p>
<p>I was curious about the choice of the word "Sprint" within the Scrum Guide. Superficially, "iteration" seems like a more descriptive fit. It's partially to communicate that the timebox is short in length, typically less than a month. Moreover, it's suggesting a motivation to rapidly do <em>something</em>. What is that something? The purpose of an iteration isn't <em>just</em> to deliver some features. Whatever's on your roadmap is of unknown utility until it's out there in the wild. An iteration, therefore, is a dash towards challenging assumptions, reducing the risk, gathering data from your experimentation and <em>increasing our understanding</em>. As per the <a href="https://agilemanifesto.org/principles.html">Agile Manifesto</a>, we want to:</p>
<blockquote>
<p>satisfy the customer through early and continuous delivery of valuable software</p>
</blockquote>
<p>and be flexible to:</p>
<blockquote>
<p>welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.</p>
</blockquote>
<p>so that we can agree that:</p>
<blockquote>
<p>working software is the primary measure of progress</p>
</blockquote>
<p>In a rapidly-changing competitive marketplace, you need to get your ideas out there and start learning from them. Ideally, make some money too! Orient your goals in that way - what problems are you trying to solve for your users? What uncertainty are you wanting to address? That's why we sprint towards a goal!</p>
<p>Goals framed in this manner are so potent because of the avenues for creativity they open. The desire to answer the question, to learn the unknown is tantalising. What a way to motivate your team!</p>
<p>You will also need to specify how you'll know when the goal is achieved. It's not necessarily when all the tickets are in the DONE column. All good hypotheses have a quantifiable aspect. Very often this aligns with key business metrics, which then helps you further understand how your product is used by the end users, so you can perform future iterations!</p>
<h2>Shoot and Score</h2>
<p>Overall, Sprint Goals help you avoid the trap of doing waterfall within an agile-like framework. Without a clear goal your Sprint is simply a bucket of work. The items in Sprint will have confused priorisation. It will seem like there's progress on lots of different things, but no progress overall on the higher-level outcomes. Instead, work intensely to achieve the one goal. See how that changes things and how happier it will make your team members.</p>
Unit Test Your Architecture with ArchUnit2019-12-05T00:00:00Zhttps://samhogy.co.uk/2019/12/unit-test-your-architecture-with-archunit/ArchUnit asserts architectural constraints at the unit-test level, giving developers guiderails on good practice with fast feedback.!<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2019/12/05/unit-test-your-architecture-with-archunit.html">the Scott Logic blog</a></span></div>
<p>📖 <a href="https://github.com/sh1989/archunit-example">Click here for a fully-worked example of the code in this blog post</a></p>
<p>Automated tests provide fast, frequent and early feedback when code changes are made. Historically we've talked about unit tests, integration tests and acceptance tests, looking at increasing levels of code to business functionality. What if we could apply the same concepts to our architecture?</p>
<p>Enter stage left ArchUnit - a Java testing library that inspects your code's architecture. Assertions can be made on your code's structure, properties and relationships to provide further automated quality guarantees about the health of your codebase.</p>
<p>For the purposes of this article, I'm going to be using ArchUnit with Junit 5. Other test frameworks are compatible and available. I'll explain concepts as we go along.</p>
<h2>Tooling Up</h2>
<p>Firstly, add the following <em>test-scoped</em> dependencies into your project's POM:</p>
<pre class="language-xml"><code class="language-xml"><span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>dependency</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>groupId</span><span class="token punctuation">></span></span>com.tngtech.archunit<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>groupId</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>artifactId</span><span class="token punctuation">></span></span>archunit-junit5-api<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>artifactId</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>version</span><span class="token punctuation">></span></span>0.12.0<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>version</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>scope</span><span class="token punctuation">></span></span>test<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>scope</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>dependency</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>dependency</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>groupId</span><span class="token punctuation">></span></span>com.tngtech.archunit<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>groupId</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>artifactId</span><span class="token punctuation">></span></span>archunit-junit5-engine<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>artifactId</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>version</span><span class="token punctuation">></span></span>0.12.0<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>version</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>scope</span><span class="token punctuation">></span></span>test<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>scope</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>dependency</span><span class="token punctuation">></span></span></span></code></pre>
<p>Now, it's time to write our test class.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@AnalyzeClasses</span><span class="token punctuation">(</span>packages <span class="token operator">=</span> <span class="token string">"uk.co.samhogy.example.archunit"</span><span class="token punctuation">,</span> importOptions <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token class-name">ImportOption<span class="token punctuation">.</span>DoNotIncludeTests</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">,</span> <span class="token class-name">ImportOption<span class="token punctuation">.</span>DoNotIncludeJars</span><span class="token punctuation">.</span><span class="token keyword">class</span> <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">ArchitectureTests</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The <code>AnalyzeClasses</code> annotation sets up ArchUnit to load classes in the specified package. ArchUnit inspects the compiled bytecode and builds it into an analyzable data structure for usage within your tests. I've specified some customisation rules for loading classes to prevent the inclusion of third-party JARs and test classes.
From here, individual architectural tests are tagged with the <code>@ArchTest</code> annotation, similarly to how traditional JUnit tests are tagged with <code>@Test</code>.</p>
<p>I'd also recommend adding the following <code>logback-test.xml</code> configuration, to prevent a huge amount of DEBUG statements from cluttering your log files:</p>
<pre class="language-xml"><code class="language-xml"><span class="highlight-line"><span class="token prolog"><?xml version="1.0" encoding="UTF-8"?></span></span>
<span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>include</span> <span class="token attr-name">resource</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>org/springframework/boot/logging/logback/base.xml<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span></span>
<span class="highlight-line"> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>logger</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>com.tngtech.archunit<span class="token punctuation">"</span></span> <span class="token attr-name">level</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>INFO<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span></span>
<span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span></span></code></pre>
<p>The rule can always be switched off if you fancy getting into the weeds of how the library initialises and operates.</p>
<h2>Including Pre-Configured Tests</h2>
<p>ArchUnit ships with a set of includable tests that cover common use cases. The following snippet demonstrates two of these - enforcing that we throw meaningful (rather than generic) exceptions, and that we don't use JodaTime. I've worked in repositories that have combined JodaTime and Java Time. It gets messy quick. Let's enforce it rather than argue about it!</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">import</span> <span class="token keyword">static</span> <span class="token import static"><span class="token namespace">com<span class="token punctuation">.</span>tngtech<span class="token punctuation">.</span>archunit<span class="token punctuation">.</span>library<span class="token punctuation">.</span></span><span class="token class-name">GeneralCodingRules</span><span class="token punctuation">.</span><span class="token static">NO_CLASSES_SHOULD_THROW_GENERIC_EXCEPTIONS</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token keyword">static</span> <span class="token import static"><span class="token namespace">com<span class="token punctuation">.</span>tngtech<span class="token punctuation">.</span>archunit<span class="token punctuation">.</span>library<span class="token punctuation">.</span></span><span class="token class-name">GeneralCodingRules</span><span class="token punctuation">.</span><span class="token static">NO_CLASSES_SHOULD_USE_JODATIME</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> no_generic_exceptions <span class="token operator">=</span> <span class="token constant">NO_CLASSES_SHOULD_THROW_GENERIC_EXCEPTIONS</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> do_not_use_jodatime <span class="token operator">=</span> <span class="token constant">NO_CLASSES_SHOULD_USE_JODATIME</span><span class="token punctuation">;</span></span></code></pre>
<p>There are further <a href="https://static.javadoc.io/com.tngtech.archunit/archunit/0.12.0/com/tngtech/archunit/library/GeneralCodingRules.html">common coding rules</a> that you can include, which prevents <code>System.out</code> calls, ensures the correct usage of a logger and so on.</p>
<h2>Defining Architectural Boundaries</h2>
<p>Let's envisage an application with a traditional 3-layered architecture:</p>
<h3>Web Layer</h3>
<ul>
<li><code>web</code> package - defines Spring MVC <code>RestControllers</code></li>
<li><code>dto</code> package - defines API Request/Response objects</li>
</ul>
<h3>Domain Layer</h3>
<ul>
<li><code>service</code> package - defines services that perform business transactions</li>
<li><code>domain</code> package - framework-independent object modelling of our business domain</li>
</ul>
<h3>Persistence Layer</h3>
<ul>
<li><code>entity</code> package - defines Object-Relational Mapping entities</li>
<li><code>repository</code> package - for querying and data access</li>
</ul>
<p>Good architectural practice ensures that we have a clear separation of responsibilities between the layers. Returning Hibernate entities from our API endpoints should be discouraged. Instead, prefer the usage of separate DTOs to prevent tight-coupling to your database schema. Likewise, the core business domain is expressed by a set of domain objects that remain independent of both. Sure, in a small sample project there's a quite a bit of duplication, but as an applications grows these will have separate concerns.</p>
<p>ArchUnit has out-of-the-box functionality to assert that your layered architecture is respected. Define layers by specifying the corresponding packages and then express a network of permitted dependencies between these layers. This test then provides automated assurances that access and usage is maintained according to your defined boundaries:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> layeredArchitecture <span class="token operator">=</span> </span>
<span class="highlight-line"> <span class="token function">layeredArchitecture</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">layer</span><span class="token punctuation">(</span><span class="token string">"Entity"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">definedBy</span><span class="token punctuation">(</span><span class="token string">"..entity.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">layer</span><span class="token punctuation">(</span><span class="token string">"Repository"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">definedBy</span><span class="token punctuation">(</span><span class="token string">"..repository.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">layer</span><span class="token punctuation">(</span><span class="token string">"Domain"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">definedBy</span><span class="token punctuation">(</span><span class="token string">"..domain.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">layer</span><span class="token punctuation">(</span><span class="token string">"Service"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">definedBy</span><span class="token punctuation">(</span><span class="token string">"..service.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">layer</span><span class="token punctuation">(</span><span class="token string">"DTO"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">definedBy</span><span class="token punctuation">(</span><span class="token string">"..dto.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">layer</span><span class="token punctuation">(</span><span class="token string">"Web"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">definedBy</span><span class="token punctuation">(</span><span class="token string">"..web.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">whereLayer</span><span class="token punctuation">(</span><span class="token string">"DTO"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">mayOnlyBeAccessedByLayers</span><span class="token punctuation">(</span><span class="token string">"Web"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">whereLayer</span><span class="token punctuation">(</span><span class="token string">"Service"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">mayOnlyBeAccessedByLayers</span><span class="token punctuation">(</span><span class="token string">"Web"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">whereLayer</span><span class="token punctuation">(</span><span class="token string">"Domain"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">mayOnlyBeAccessedByLayers</span><span class="token punctuation">(</span><span class="token string">"Web"</span><span class="token punctuation">,</span> <span class="token string">"Service"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">whereLayer</span><span class="token punctuation">(</span><span class="token string">"Repository"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">mayOnlyBeAccessedByLayers</span><span class="token punctuation">(</span><span class="token string">"Service"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">whereLayer</span><span class="token punctuation">(</span><span class="token string">"Entity"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">mayOnlyBeAccessedByLayers</span><span class="token punctuation">(</span><span class="token string">"Repository"</span><span class="token punctuation">,</span> <span class="token string">"Service"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>If you follow a Ports and Adapters (Hexagonal Architecture, Onion Architecture) approach, don't worry. ArchUnit <a href="https://www.archunit.org/userguide/html/000_Index.html#_onion_architecture">provides assertions</a> according to the terminology and dependency structures outlined <a href="https://jeffreypalermo.com/2008/07/the-onion-architecture-part-1/">here</a>.</p>
<h2>Asserting Classes are in the Correct Package</h2>
<p>The test above is fantastic for making quite sweeping assertions. It clearly communicates the design decision and acts as the source of truth. However it's easy to avoid - simply put a different type of class in a different package and use it the wrong way. Naughty, devious, nefarious! So we'll want to defend against that too.</p>
<p>ArchUnit provides a fluent DSL which lets you attach conditions to a given situation. This lets us assert that classes that have particular annotations must belong to particular packages, or that they have particular names. By doing so, we're establishing our coding conventions in a form that's easy to test!</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> entities_must_be_suffixed_in_correct_package <span class="token operator">=</span> </span>
<span class="highlight-line"> <span class="token function">classes</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">that</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">areAnnotatedWith</span><span class="token punctuation">(</span><span class="token string">"Entity"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">should</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">resideInAPackage</span><span class="token punctuation">(</span><span class="token string">"..entity.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">andShould</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">haveSimpleNameEndingWith</span><span class="token punctuation">(</span><span class="token string">"Entity"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> services_must_be_in_correct_package <span class="token operator">=</span> </span>
<span class="highlight-line"> <span class="token function">classes</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">that</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">haveSimpleNameEndingWith</span><span class="token punctuation">(</span><span class="token string">"Service"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">should</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">resideInAPackage</span><span class="token punctuation">(</span><span class="token string">"..service.."</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> rest_resources_must_be_in_correct_package <span class="token operator">=</span> </span>
<span class="highlight-line"> <span class="token function">classes</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">that</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">areAnnotatedWith</span><span class="token punctuation">(</span><span class="token string">"RestController"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">should</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">resideInAnyPackage</span><span class="token punctuation">(</span><span class="token string">"..web.."</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> dtos_must_be_suffixed_in_correctPackage <span class="token operator">=</span> </span>
<span class="highlight-line"> <span class="token function">classes</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">that</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">resideInAPackage</span><span class="token punctuation">(</span><span class="token string">"..dto.."</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">should</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">haveSimpleNameEndingWith</span><span class="token punctuation">(</span><span class="token string">"DTO"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>It's also possible to encode your decisions on particular discussions that largely come down to personal preference. Don't like using the <code>Impl</code> suffix for interface implementations? Enforce it with a test! As an added extra, the <code>because</code> block lets you go into detail about your choice. Or to be passive-aggressive.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@ArchTest</span></span>
<span class="highlight-line"><span class="token keyword">static</span> <span class="token class-name">ArchRule</span> classes_must_not_be_suffixed_with_impl <span class="token operator">=</span> </span>
<span class="highlight-line"> <span class="token function">noClasses</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">should</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">haveSimpleNameEndingWith</span><span class="token punctuation">(</span><span class="token string">"Impl"</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">because</span><span class="token punctuation">(</span><span class="token string">"seriously, you can do better than that"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>The Sell</h2>
<p>Other static analysis tools exist on the marketplace. Some are more declarative compared to the programmatic nature of ArchUnit. The power of this library comes, in my opinion, from its fluent expressions and its extensibility. By being a Java library, the language you're already familiar with, you benefit from the additional tooling capabilities and leads to an overall lower cost of maintenance.</p>
<p>As a tool which helps makes your knowledge of the system's operation <em>explicit</em> instead of <em>implicit</em>, ArchUnit also aids new developers who join the project at a later date. Architectural decisions are defined in code and enforced in an automated manner. Rather than relying on hearsay or reading a wiki page to establish conventions, look at the tests. They're a form of executable documentation.</p>
<p>It also helps you stay on the straight and narrow. There are plenty of times where a corner's cut or a rule is violated and subsequently not spotted at code review. This risks introducing a bug or incurring tech debt. We're humans. I forgive you. But by encoding these choices in an executable form we can provide fast feedback on whether changes satisfy our architectural boundaries, lowering the cost of correcting such mistakes. Also, it's one less thing to review manually.</p>
<p>ArchUnit has <a href="https://www.archunit.org/userguide/html/000_Index.html">a comprehensive user guide</a> detailing the core concepts, alongside <a href="https://github.com/TNG/ArchUnit-Examples">a treasure trove of examples</a> covering a myriad of use cases. Suffice to say I've merely scratched the surface in this article.</p>
<p>In summary, this is a fascinating tool that can aid keeping a codebase clean, maintainable and pleasant to work in. It's definitely worth investing in a solution that strengthens your game in this area.</p>
Agile Is Not a Solved Problem2019-09-23T00:00:00Zhttps://samhogy.co.uk/2019/09/agile-is-not-a-solved-problem/Agile is not an off-the-shelf product that can be dropped into an organisation. It's a learning journey with a willingness to embrace change.!<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2019/09/23/agile-is-not-a-solved-problem.html">the Scott Logic blog</a></span></div>
<p>It occurred to me that the Agile Manifesto turned 18 this year, making it old enough to drink in pubs in the UK. That's quite a long time. In all honesty, how do you think it's going?</p>
<p>From experience, projects which are decreed as "Agile" up front translates into adopting a particular framework (Scrum, Kanban or SAFe). The implication is that this is how we're "doing Agile". Sure, you can tinker with the edges if something's not quite working out. But ditching the whole framework and doing something else? Unthinkable. Here's your box. Get back to work.</p>
<p>I'm not saying that these frameworks are fundamentally bad; far from it actually. They're fantastic <em>starting points</em> in a journey. Scrum gives you a minimal setup in terms of ceremonies and inspection/adaptation cycles to facilitate you <em>being agile</em>. SAFe provides portfolio-scale delivery projects, typically accustomed to more traditional software development practices, with a transition mechanism to more flexible and adaptable ways of working. Sometimes the benefits these frameworks bring are overlooked because the bleeding-edge have rejected them, or because the <a href="https://www.scaledagileframework.com/">SAFe organisational diagrams</a> are daunting to look at. There's a perception of the "enterprising" of Agile Development at play here, especially as the term "business agility" contains an aspect of appropriation. There is good stuff in each of these frameworks that is worth recognising.</p>
<p>All of this overlooks the matter at hand. Software delivery is difficult. Agile methodologies are undoubtedly and provably better than what came before, but in no way have they "solved" the difficult problem.</p>
<p>Returning to the <a href="http://agilemanifesto.org/">Agile Manifesto</a>, I was surprised by its subtle use of language. The document says "we <em>are uncovering</em> better ways of developing software", not "we <em>have uncovered</em>". Following this are a set of comparative and qualitative values, not entirely rejecting what came before it, but placing more value on some other aspects at play. There's no commandments to follow. Just principles and values.</p>
<p>Dave Thomas, one of the signatories, <a href="https://www.youtube.com/watch?v=a-BOSpxYJ9M">remarks on the difference between the noun Agile and the verb agile</a>. Confusingly the manifesto is for Agile Software Development - i.e a named thing. Yet the principles and values describe agility, which is what you seek to practice.</p>
<p>Scrum, Kanban, SAFe - these process frameworks are noun-Agile. You can be verb-agile within those, but there's no guarantee. When an Agile project fails, or doesn't seem to be bringing you the proclaimed benefits, we feel disheartened. Very often that's because the culture isn't promoting <em>agility</em>, or is resisting its adoption. This manifests in several forms, such as: Scrum-But ("we're doing scrum, but…"), the "wagile" pejorative, or SAFe's release trains being misused to drive Gantt-chart deliveries, to name a few. At best you've established a division of labour and a management framework but have made no inroads into solving the problem of making software delivery easier.</p>
<p>These are not reasons to feel dejected. Rather, this is empowering. The sages of times gone by have not uncovered a set of universal truths, just painted a rough outline for a way forward. Absolutely, go ahead and adopt Agile frameworks if you need a helping hand, but don't feel bad about moving past them if they no longer suit your needs. You're finely attuned to the workings of your team, your domain, your ways of working. What you think about this matters immensely. The actions your team takes to improve its way of doing things are important. That needs to come from you. Listen to each other. Take what's good, drop what's bad. Repeat.</p>
Reflections from Build IT Right2019-04-05T00:00:00Zhttps://samhogy.co.uk/2019/04/reflections-from-build-it-right/DevOps and Continuous Delivery learnings from the first Build IT Right event.!<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2019/04/05/reflections-from-build-it-right.html">the Scott Logic blog</a></span></div>
<p>Yesterday I attended Build IT Right, just around the corner from the Newcastle Scott Logic offices. Advertised as a conference to adapt to emerging patterns, it certainly delivered on that promised, turning out to be one of the more thought-provoking events that I've attended in a long time.</p>
<p>Normally I report conferences by covering each of the sessions that I attended. The sessions I attended yesterday were great, so I extend my thanks to <em>Dave Farley</em>, <em>Jesse Cary</em>, <em>Bartosz Jedrzejewski</em>, <em>Timandra Harkness</em>, <em>Stephen Mounsey</em> and <em>Gary Fleming</em> for some insightful talks. But, I'm going to do it a bit differently this time, take a step back, and reflect on some of the larger trends that emerged.</p>
<img src="https://samhogy.co.uk/assets/builditright/stage.jpg" width="600" />
<h2>Continuous Delivery Is Your Future</h2>
<p>Last year, I read Jez Humble and Dave Farley's "Continuous Delivery" book and then quickly flew through "The Phoenix Project", "The DevOps Handbook" and the more recent "Accelerate". These books have been a real light bulb moment for me, clarifying some of my own thinking on how to build software. The fundamental idea is the continuous delivery of value to our users (and therefore to our business). Achieving this is a root-and-branch transformation of an organisation's approach to software development and delivery, with the added side-effect that we create infrastructure which is reliable, environments that are repeatable and production releases that are uneventful. All this while giving us the means to rapidly iterate and experiment with our product.</p>
<p>Many who have worked in larger organisations will be familiar with spending hours in "go / no-go" meetings. During these meetings senior team members assign risk values to a set of features in a release candidate build, then make a judgement call on whether the business can "afford" the risk. For a long time such meetings and processes have seemed to be designed to slow down delivery. The only truly "safe" form of software delivery is to not deliver at all. Under this philosophy, attempting to release anything is actually an exercise in box-ticking, form-filling and pleading with stakeholders to agree sign-off. Make no mistake, software is a risky business. There's always the risk that a production release causes an outage, a financial loss or reputational damage. But traditionally teams have fallen into the trap of releasing less often as a way of mitigating the risk. Actually, they're compounding it by doing so. Releases become a big deal, a huge event, with lots of ceremony and lots of features being dumped into production in one large chunk.</p>
<p>A core principle of Continuous Delivery is that if it hurts, do it more often. Get good at it. Refine the process, make it as simple as it needs to be. Make it boring. Remove the humans in as much of the boring stuff as you feasibly can - their time and attention is better spent on things that humans are good at, like creativity and exploration, rather than repetitive and laborious run-books.</p>
<p>A Continuous Delivery Pipeline is the process by which an individual commit to your repository can work its way into production. You can consider this an extension of Continuous Integration. A commit turns into a built version of your application, which is itself a release candidate. We run the release candidate through a series of quality gates - each acting as a falsification mechanism. Release candidates are rejected if they fail to meet certain criteria. It's common to see automated unit, integration and system/acceptance level testing quality gates, in addition to security audits and manual exploratory testing. Running a quality gate may require the automated set-up and deployment to an environment. To satisfy this as a team, we need to refine our DevOps abilities. We can't have <a href="https://martinfowler.com/bliki/SnowflakeServer.html">snowflake servers</a> which are long-lived and whose configuration becomes increasingly opaque and immutable.</p>
<p>All of this is designed to reduce risk whilst delivering to production safely and sustainably. Ideally, release candidates which pass all of your quality gates can be released to production. That step itself can be automated! Incredibly large organisations have cycle times (a measurement of how long it takes an idea to get from conception to production) in the order of <em>minutes</em>. We don't know which version of Amazon we're using, the concept is so alien to that platform. End users don't really need to know what version of Chrome they have - it's just "the latest". Adopting this mindset is incredibly powerful, because if you can get good quality code into production this fast, you're not accumulating risk over time. To the risk-based gatekeepers out there, there's still no guarantee the software that will be used by end users will be bug free (it's mathematically impossible anyway). But by passing the code through the quality gates, we've verified it against our known risks. Furthermore, the releases that will be produced from this pipeline will be inherently smaller and therefore inherently less risky than the old large weekly/monthly "drops" into production.</p>
<p>To achieve this as a development team, we must also think about how we develop. We must integrate more often. Long-lived branches are almost unthinkable under this paradigm. Therefore, this necessitates practices such as <a href="https://trunkbaseddevelopment.com/">trunk-based development</a>, being able to place features behind <a href="https://martinfowler.com/articles/feature-toggles.html">toggles</a>, being able to perform rolling upgrades through <a href="https://octopus.com/docs/deployment-patterns/blue-green-deployments">blue/green deployments</a>. I will go in-depth into these topics through a later series of blog posts, but for now, let's appreciate how <em>agile</em> this all feels.</p>
<h2>Embracing The Conflict Inherent To Agile</h2>
<p>These processes are, effectively, an application of the Scientific Method: making observations, formulating a hypothesis, devising an experiment and measuring the outcomes. When put on repeat this algorithm has driven the rapid development of our species. When it comes to software, it inherently leads to conflict.</p>
<p>With the engine of Continuous Delivery providing a framework for experimentation and our attention freed from laborious production deployments, we have time for creative innovation. The members of an agile team all have their own priorities, their own incentives, their own personality traits. Conflict arises when these pull in different directions. Effective agile teams can work through conflict to form consensus. Visibility, accountability and transparency are key aspects here, allowing teams to work through their problems.</p>
<p>Agile teams are the foundry of software development. Teams that embrace conflict in order to resolve it become the lubricant of your business value stream, helping you deliver to customers more effectively.</p>
<h2>Entropy Is Also Your Future - If You Let It Be</h2>
<p>Entropy, the gradual decline of order and predictability into disorder. Closed systems, according to the laws of thermodynamics, are doomed to this fate. Is our software too?</p>
<img src="https://samhogy.co.uk/assets/builditright/entropy.jpg" width="600" />
<p>The practices of continuous delivery fight against the forms of entropy our software can run up against. Operating systems, programming languages, frameworks, libraries all get updated. Older versions fall into maintenance windows and then obscurity. If we don't keep up to date, we inherently make our system harder to work with and more difficult to change. But there are other forms of entropy we must think about: people leaving teams, the setup of your developer machines. Knowledge risks being transient - stuck in certain team member's heads, hidden in configuration files and small patches. If those team members leave, or if you need to set something up from scratch you can be often left wondering "why are things this way" and never knowing. Bodge jobs become permanent. Entire sections of domain knowledge just vanish.</p>
<p>We must think of ways and means to prevent the entropy. The only way we can do this is turning the closed system into an open one. That is, to put energy in. To do work to make things last longer, make our processes and systems more resilient, make our architecture more adaptive to change.</p>
<h2>Conclusion</h2>
<p>A realisation that I made is the Humble/Farley book was published 9 years ago. The Agile Manifesto is 18 this year, that's old enough to drink in pubs. These ideas aren't new. They're tried and tested with an ever-increasing data sample. Adopting these techniques are still considered the bleeding-edge of software delivery, but I suspect that is because the trailing-edge often trails by several decades. Your future in software engineering will be working with teams practising Continuous Delivery and we'll be better as an industry for it. Should you not be adopting these practices, your software risks being overcome by entropy, increasingly intolerable to change and requiring more and more (highly-paid) specialists to maintain it. Our world is a fast-moving one. We are uncovering better ways of developing software by doing it and helping others to do it.</p>
<p>This is the first conference in a long time that has really shaken up the way that I think about software and for that I extend my thanks to the Build IT Right organisers for putting on such an excellent event!</p>
Converting a React App to TypeScript2018-09-29T00:00:00Zhttps://samhogy.co.uk/2018/09/converting-a-react-app-to-typescript/<p>📖 <a href="https://github.com/SundayNightDice/snd-webapps">Click here for a fully-worked example of the code in this blog post</a></p>
<p>I've been looking at <a href="https://www.typescriptlang.org/index.html">TypeScript</a> recently (yes, I'm a bit late to this party) as a means to improve the readability and maintainability of an existing large web application. I thought I'd initially start small with something that I own, so in this post I'll describe the steps that I took to convert a suite of small React apps to using TypeScript.</p>
<h2>Setting Things Up</h2>
<p>I have two small web apps - they're small utilities that I host on my RPG website: <a href="https://github.com/SundayNightDice/warlock-invocation-tracker">here</a> and <a href="https://github.com/SundayNightDice/barbarian-combat-tracker">here</a> - and as such they use bare-bones React and nothing else. In the future I envisage these becoming either part of a larger web app, or for there to be a suite of similar-sized web apps. I also wanted to re-use some shared components between them. For that reason I imported the two existing git repos into a Lerna monorepo. This isn't a necessity however and all of the steps below can be run on each project individually.</p>
<p>To use TypeScript, I need some new dependencies. I require the typing definitions for React and ReactDOM, which are shipped separately from the main packages. I also require a Webpack loader that can trigger TypeScript compilation. Using yarn/npm/lerna, install: <code>@types/react @types/react-dom @typescript awesome-typescript-loader --save-dev</code></p>
<p>TypeScript looks for a <code>tsconfig.json</code> file, which instructs the TypeScript compiler where to look for code, where to produce code, and any specific compilation rules that should be followed during compilation. My configuration is explicitly disabling the <code>noImplicitAny</code> rule as I'm looking for strict typing. It's also instructing TypeScript to produce ES2015-compilant JavaScript and use the commonjs module format.</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"compilerOptions"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"outDir"</span><span class="token operator">:</span> <span class="token string">"./dist/"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"sourceMap"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"noImplicitAny"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"module"</span><span class="token operator">:</span> <span class="token string">"commonjs"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"target"</span><span class="token operator">:</span> <span class="token string">"es2015"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"jsx"</span><span class="token operator">:</span> <span class="token string">"react"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"include"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token string">"./src/**/*"</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>TypeScript's compilation step replaces Babel's transpilation step, meaning that I could delete Babel, it's dependencies, and the <code>.babelrc</code> files.</p>
<p>I then renamed all my <code>.jsx</code> files to <code>.tsx</code>. In Visual Studio Code this started to produce quite a lot of errors almost immediately!</p>
<p>I was already using Webpack to build my distributables and would like to continue to do so. After upgrading from Webpack 2 to 4, I then had to make some small changes to the config file - to tell it to look for TypeScript files, and run them through the TypeScript loader.</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line">var webpack = require('webpack');</span>
<span class="highlight-line"></span>
<span class="highlight-line">module.exports = <span class="token punctuation">{</span></span>
<span class="highlight-line"> entry<span class="token operator">:</span> <span class="token string">"./src/index.tsx"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> output<span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> filename<span class="token operator">:</span> <span class="token string">"barbarian-combat-tracker.js"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> path<span class="token operator">:</span> __dirname + <span class="token string">"/dist"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> devtool<span class="token operator">:</span> <span class="token string">"source-map"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> resolve<span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> extensions<span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">".ts"</span><span class="token punctuation">,</span> <span class="token string">".tsx"</span><span class="token punctuation">,</span> <span class="token string">".js"</span><span class="token punctuation">,</span> <span class="token string">".json"</span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> module<span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> rules<span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> test<span class="token operator">:</span> /\.tsx?$/<span class="token punctuation">,</span> loader<span class="token operator">:</span> <span class="token string">"awesome-typescript-loader"</span> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> enforce<span class="token operator">:</span> <span class="token string">"pre"</span><span class="token punctuation">,</span> test<span class="token operator">:</span> /\.js$/<span class="token punctuation">,</span> loader<span class="token operator">:</span> <span class="token string">"source-map-loader"</span> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> test<span class="token operator">:</span> /\.scss$/<span class="token punctuation">,</span> use<span class="token operator">:</span> <span class="token punctuation">[</span>'style-loader'<span class="token punctuation">,</span> 'css-loader'<span class="token punctuation">,</span> 'sass-loader'<span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> test<span class="token operator">:</span> /\.png$/<span class="token punctuation">,</span> loader<span class="token operator">:</span> 'url-loader?limit=<span class="token number">25000</span>' <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> externals<span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"react"</span><span class="token operator">:</span> <span class="token string">"React"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"react-dom"</span><span class="token operator">:</span> <span class="token string">"ReactDOM"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span>;</span></code></pre>
<p>The way that imports works in TypeScript is slightly different to ES imports. Therefore I had to change all occurrences of: <code>import React from 'react'</code> to: <code>import * as React from 'react'</code></p>
<h2>Converting Functions</h2>
<p>Most functions are quite simple to convert to TypeScript - specify the types of any parameters and the type of the returned object:</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">const</span> proficiencyBonus <span class="token operator">=</span> <span class="token punctuation">(</span>level<span class="token operator">:</span> <span class="token builtin">number</span><span class="token punctuation">)</span> <span class="token operator">:</span> <span class="token builtin">number</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>level <span class="token operator">>=</span> <span class="token number">17</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token number">6</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>level <span class="token operator">>=</span> <span class="token number">13</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token number">5</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>level <span class="token operator">>=</span> <span class="token number">9</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token number">4</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>level <span class="token operator">>=</span> <span class="token number">5</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token number">3</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">else</span> <span class="token keyword">return</span> <span class="token number">2</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<h2>Converting the React Components</h2>
<p>The vast majority of the components that I use are stateless components. These are simple to convert manually and act as a good starting point. If you envisage your React components as a tree, the simplest ones will most likely live at the leaf nodes and will most likely be stateless, receiving properties from their parent components and producing output wholly dependent upon those inputted properties. Take the following React component for example:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">Effect</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> effect<span class="token punctuation">,</span> invocation <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span> <span class="token attr-name">className</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>description<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token punctuation">{</span>effect<span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span> <span class="token attr-name">className</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>invocation<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token punctuation">{</span>invocation<span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span></code></pre>
<p>The first step was to take that list of properties from the component and define an interface:</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">interface</span> <span class="token class-name">EffectProps</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> effect<span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> invocation<span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Then, I declared the type of your functional component to be one of <code>React.SFC</code>. This is a generic type, with a type parameter corresponding to the interface declared for the component's properties:</p>
<pre class="language-tsx"><code class="language-tsx"><span class="highlight-line"><span class="token keyword">const</span> Effect<span class="token operator">:</span> React<span class="token punctuation">.</span><span class="token constant">SFC</span><span class="token operator"><</span>EffectProps<span class="token operator">></span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">{</span> effect<span class="token punctuation">,</span> invocation <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span> <span class="token attr-name">className</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>description<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token punctuation">{</span>effect<span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span> <span class="token attr-name">className</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>invocation<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token punctuation">{</span>invocation<span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span></code></pre>
<p>From there I progress upwards in the component, doing the same thing. tree. Every time I saw a property that was complex object, I gave it the type <code>any</code>. These will be revisited later on, but help in the immediate sense that you can progressively work through your files and see them go from red to green.</p>
<p>Functions that you pass as properties into React components, for example for use as a callback function, must also have typing information supplied.</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">interface</span> <span class="token class-name">CheckboxProps</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> value<span class="token operator">:</span> <span class="token builtin">boolean</span></span>
<span class="highlight-line"> <span class="token function-variable function">onChange</span><span class="token operator">:</span> <span class="token punctuation">(</span>checked<span class="token operator">:</span> <span class="token builtin">boolean</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token keyword">void</span></span>
<span class="highlight-line"> disabled<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">boolean</span></span>
<span class="highlight-line"> children<span class="token operator">:</span> React<span class="token punctuation">.</span>ReactNode</span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>My top-level root React component is a stateful component. It's using its internal state to track anything that can change within the application - I thought the state was small enough not to warrant using a state management library such as Redux. These too must be typed! Rather than using <code>React.SFC</code>, I use <code>React.Component</code>. It's also a generic type with two type params - the first represents the type of the props, the second the type of the state. As I don't care about the props I'm comfortable using <code>any</code>. For my state I use a custom type called <code>Model</code>, which I need to define myself.</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Container</span> <span class="token keyword">extends</span> <span class="token class-name">React</span><span class="token punctuation">.</span>Component<span class="token operator"><</span><span class="token builtin">any</span><span class="token punctuation">,</span> Model<span class="token operator">></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span>props<span class="token operator">:</span> <span class="token builtin">any</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">super</span><span class="token punctuation">(</span>props<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>state <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// Define the state inital values</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">render</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// Render the component tree</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Defining Domain Object Models</h2>
<p>Let's talk a bit about how to define those custom domain-based types. I need a custom type to define the shape of the top-level state, and also replace the instances of <code>any</code> that I've sprinkled temporarily throughout the code into something more meaningful. This is as straightforward as defining some more interfaces:</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">interface</span> <span class="token class-name">Pact</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> name<span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> value<span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>For some of my domain types, I had some properties which were optional. TypeScript doesn't expect optionals by default - you need to instruct the compiler to allow for them by denoting the property with a <code>?</code>:</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">interface</span> <span class="token class-name">Invocation</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> id<span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> name<span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> eldritchBlast<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">boolean</span></span>
<span class="highlight-line"> effect<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> minLevel<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">number</span></span>
<span class="highlight-line"> pact<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> patron<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> source<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"> tome<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">string</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>One of the more esoteric types I had to define was for an object which was a map of <code>string -> boolean</code>. Of course all objects in JavaScript are keyed by string, which effectively makes them maps by default. But in order to represent this as a type, specifically a type that makes sense in the domain, I firstly had to define a helper <code>Map</code> type, which I can then extend:</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">interface</span> <span class="token class-name">Map<span class="token operator"><</span><span class="token constant">T</span><span class="token operator">></span></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span>key<span class="token operator">:</span> <span class="token builtin">string</span><span class="token punctuation">]</span><span class="token operator">:</span> <span class="token constant">T</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">interface</span> <span class="token class-name">ActiveInvocations</span> <span class="token keyword">extends</span> <span class="token class-name">Map<span class="token operator"><</span><span class="token builtin">boolean</span><span class="token operator">></span></span> <span class="token punctuation">{</span><span class="token punctuation">}</span></span></code></pre>
<p>As this is such as small project I decided to extract the domain-based types into a separate <code>model.ts</code> file. However as an application grows I could quite conceivably see that you would end up defining these in separate files. Once again by working upwards from the leaf React components up to the top-level root React component, I was able to replace all instances of <code>any</code> which represented a domain object with the newly-defined types.</p>
<p>From here all I need to do is run webpack, load the <code>index.html</code> file and everything is working!</p>
<h2>Conclusion</h2>
<p>This was a relatively straightforward exercise. I didn't have any extra dependencies such as state management libraries to handle. Overall it took me a few hours to grasp (enough of) the TypeScript syntax and perform the conversion. Visual Studio Code is my primary IDE of choice and it's great to see its TypeScript integration works so well that you can use it as a guide while converting. It also called out a few errors in my existing code (oops!) which I was able to correct as part of the conversion. I can totally see the benefits of introducing TypeScript, although hopefully</p>
<p>Next steps for me are to investigate how well TypeScript plays with a React-Redux project, and then something such as the Apollo GraphQL client - the prospects of using a GraphQL schema to strongly-type the frontend code is very exciting to me!</p>
Lerna Version 3 Upgrade Guide2018-08-18T00:00:00Zhttps://samhogy.co.uk/2018/08/lerna-version-3-upgrade-guide/<p>Lerna version 3 is now out of beta! Under the hood there have been a fair few changes, but it is backwards-compatible so that you can upgrade at your convenience. Here are some of the important deprecations that I have found through upgrading my build scripts:</p>
<h2>Lerna Publish</h2>
<p><code>lerna publish --skip-npm</code> is now deprecated. Use <code>lerna version</code> instead.</p>
<p><code>lerna publish</code> only handles publishing to an npm repository itself. It calls through to <code>lerna version</code> in order to handle the package.json version bumps and the git tagging, so if you don't need to publish to an npm repository, call the <code>version</code> command directly yourself. For backwards-compatibility and practical reasons, flags passed to <code>lerna publish</code> that are recognised by <code>lerna version</code> are passed through as expected.</p>
<h2>--skip-git</h2>
<p><code>lerna publish --skip-git</code> is now deprecated. Use <code>lerna version --no-git-tag-version --no-push</code> (or <code>lerna publish --no-git-tag-version --no-oush</code>) instead.</p>
<p>I've used this flag to the <code>lerna publish</code> command many times while debugging CI scripts, just so that I don't have to manually unwind the changes in git. As described above, now that the version bumping is handled by <code>lerna version</code>, <code>--skip-git</code> is still supported on that command although marked as deprecated.</p>
<p>You can run with the <code>--no-git-tag-version</code> flag to prevent a git commit and a git tag from occurring during the version bump. Adding <code>--no-push</code> prevents a push of these changes to a git remote, if one is configured.</p>
<h2>--cd-version and --repo-version</h2>
<p><code>lerna publish --cd-version=patch</code> and <code>lerna publish --repo-version 1.0.1</code> are both deprecated. Run <code>lerna version <version-number></code> (or <code>lerna publish <version-number></code>) with either the exact version number, or a semver keyword such as major/minor/patch.</p>
<p>A great simplification of the CLI - I never fully understood why these were separate flags!</p>
<h2>Lerna List</h2>
<p><code>lerna ls</code> is now <code>lerna list</code>, although <code>lerna ls</code> is still supported. Additionally you can run <code>lerna ll</code> as shorthand for <code>lerna list -l</code> and <code>lerna la</code> as shorthand for <code>lerna list -la</code>. Longform versions <code>--long</code> and <code>--all</code> are also supported. Additionally, you can retrieve the output as parseable JSON with the <code>--json</code> flag - ideal for running build scripts!</p>
<h2>Wrapping Up</h2>
<p>Remember that you don't have to do anything immediately because all existing commands are still supported. Lerna will print deprecation warnings to the console with the suggested changes that you can make to your build scripts.</p>
Lerna Independent Mode with Semver2018-08-16T00:00:00Zhttps://samhogy.co.uk/2018/08/lerna-independent-mode-with-semver/<p>In my previous post on <a href="/2018/04/moving-to-a-monorepo/">Moving to a Monorepo</a> I discussed a journey of taking several existing npm-based git repositories and migrating to a Lerna monorepository, alongside some common techniques for testing, building and deploying code within that context. One of the topics I briefly covered, but not in a huge amount of detail, was Lerna's independent versioning mode. This is largely because in my day job I'm currently running in a fixed mode monorepo. I'm now considering making the switch to using independent versioning and I thought I'd document some of the benefits on why this would be a good idea.</p>
<p>It would be best to follow along as much as possible with this and as such I'll provide lots of detail at each step.</p>
<h2>Setup</h2>
<p>We're going to set up an incredibly simple Lerna monorepo, with two packages, one depending on the other. There's nothing fancy here. First, initialise a lerna repo in independent mode into a new directory:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">mkdir lerna-demo
<span class="highlight-line">cd lerna-demo</span>
<span class="highlight-line">lerna init --independent</span>
</span></code></pre>
<p>This will create the necessary top-level files (<code>lerna.json</code>, <code>package.json</code>) as well as initialise an empty git repository. Examine your <code>lerna.json</code> file to notice that the <code>version</code> field is not hard-coded, instead it's set to 'independent':</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"lerna"</span><span class="token operator">:</span> <span class="token string">"2.11.0"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"packages"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token string">"packages/*"</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"independent"</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Next we'll create the two packages:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">mkdir packages/app packages/lib
<span class="highlight-line">lerna create app</span>
<span class="highlight-line">lerna create lib</span>
</span></code></pre>
<p>And link lib package as a dependency of the app package:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">lerna add lib --scope=app
</span></code></pre>
<p>The lib package will export a simple greeting function, in <code>packages/lib/index.js</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">greeting</span><span class="token punctuation">(</span><span class="token parameter">name</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token string">"Hello, "</span> <span class="token operator">+</span> name <span class="token operator">+</span> <span class="token string">"!"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line">module<span class="token punctuation">.</span>exports <span class="token operator">=</span> greeting<span class="token punctuation">;</span></span></code></pre>
<p>The app package will consume this function, call it, then log the output (<code>packages/app/index.js</code>):</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> greeting <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'lib'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token function">greeting</span><span class="token punctuation">(</span><span class="token string">"Sam"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Add a start script to the <code>packages/app/package.json</code>:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"start"</span><span class="token operator">:</span> <span class="token string">"node ."</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>For completeness, don't forget to add a <code>.gitignore</code>:</p>
<pre><code>**/node_modules
lerna-debug.log
</code></pre>
<p>So that when we run <code>lerna run start</code>, this produces in your terminal:</p>
<blockquote>
<p>Hello, Sam!</p>
</blockquote>
<h2>How Does Independent Versioning Work?</h2>
<p>If we take our sample code above and commit it under the following format:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">git commit -m "feat: Log a name"
</span></code></pre>
<p>It is possible to run the <code>lerna publish</code> command, with an additional <code>--conventional-commits</code> flag. This will look at the files changed in each package, combined with the associated commit messages, to work out what packages to publish and their versions.</p>
<p>So as we wrapped up all the above into one commit of type <code>feat</code>, running <code>lerna publish --conventional-commits</code> produces:</p>
<blockquote>
<p>Changes:</p>
<ul>
<li>app 1.0.0 -> 1.1.0</li>
<li>lib 1.0.0 -> 1.1.0</li>
</ul>
</blockquote>
<p>If you accept the changes, then Lerna will publish to npm (unless you add the <code>--skip-npm</code> flag). It will also create a new commit in your repo, whose mesage contains the names of the versions that you're updating (unless you add the <code>--skip-git</code> flag):</p>
<blockquote>
<p>Publish</p>
<ul>
<li>app@1.1.0</li>
<li>lib@1.1.0</li>
</ul>
</blockquote>
<p>A tag is created for each package that was built, pointing at this commit:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">git tag -l
</span></code></pre>
<blockquote>
<p>app@1.1.0
lib@1.1.0</p>
</blockquote>
<p>The git commit message itself can be configured in one of two ways: either with a flag:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">lerna publish -m "chore: release"
</span></code></pre>
<p>Or alternatively via your <code>lerna.json</code> file:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"commands"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"publish"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"chore: release"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Finally, in each package a <code>CHANGELOG.md</code> file is created/appended to containing the details of exactly what's changed in this version. This is a handy utility to pass to your QA, release team and customers if necessary, making the change process relatively self-documenting. An example from above:</p>
<blockquote>
<p><a name="1.1.0"></a></p>
<h1>1.1.0 (2018-08-16)</h1>
<h3>Features</h3>
<ul>
<li>Log a name beeab41</li>
</ul>
</blockquote>
<h2>Semantic Versioning</h2>
<p>The key advantage with independent mode over fixed mode is (naturally) that the versions of your packages can be updated independently, and therefore communicate semantic information with their version numbers. Larger monorepo solutions, which may host several "top-level" application packages as well as a collection of utility/library packages can therefore benefit from evolving independently, whilst using version numbers as a guide as to what has functionally changed within a</p>
<p>Say, for instance, that we make a change to the <code>lib</code> package above, so that the format of the argument expected in the <code>greeting</code> function is now different:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">greeting</span><span class="token punctuation">(</span><span class="token parameter">value</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token string">"Hello, "</span> <span class="token operator">+</span> value<span class="token punctuation">.</span>name <span class="token string">"!"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>This is of course a breaking change, so we'll commit this change to the repo. Using the <a href="https://conventionalcommits.org/">Conventional Commits specification</a> specification, we can clearly communicate breaking changes:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">git commit -m "feat: Re-work the greeting function
<span class="highlight-line"></span>
<span class="highlight-line">BREAKING CHANGE: The argument to greeting is no longer an object, just a string"</span>
</span></code></pre>
<p>Secondly, we'll need to adjust the <code>app</code> package to work to this new format:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token function">greeting</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">"Sam"</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Now, depending on your personal opinion, you may class this as a breaking change to the consumer of <code>lib</code>. I personally don't, because the output of the consumer hasn't changed - it's just changed what it needed to do in order to get that output. There's no right answer to whether a breaking change in a library is a breaking change in a consumer, it will depend on a case-by-case basis. But what's great is you have the choice on a case-by-case basis, you're not constrained to dedciding this up-front. So for this instance I'll class it as a <code>chore</code>, which is a minor version bump:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">git commit -m "chore: update to new lib version
</span></code></pre>
<p>Now when I run the next <code>publish</code> command, I'll see the following change:</p>
<blockquote>
<p>Changes:</p>
<ul>
<li>app 1.1.0 -> 1.1.1</li>
<li>lib 1.1.0 -> 2.0.0</li>
</ul>
</blockquote>
<p>Notice that only the minor version of <code>app</code> has changed, but the major version of <code>lib</code> has changed.</p>
<p>You'll notice to get this to work I had to implement the change as two separate commits. Fans of single, atomic changes may find this slightly frustrating, but if everything is rolled into one commit, you're constrained to declaring all packages affected by that change under the same semantic version type - both <code>app</code> and <code>lib</code> would be bumped a major version to <code>2.1.0</code>. This may make incremental commits break if ran in isolation, unfortunately this is something you'll have to live with if you go down this route. In my opinion the benefit outweighs the marginal cost.</p>
<h2>Conclusion</h2>
<p>Working with independent mode in Lerna is pretty easy and pretty powerful. With the combined usage of Conventional Commits you can effectively utilise semantic versioning for your package versions, to communicate the reaason for changes to a package and automatically generate a changelog file. You could further combine the use of Conventional Commits with a tool such as <a href="http://commitizen.github.io/cz-cli/">Commitizen</a>, which will ensuring that commits to the repository all utilise the correct syntax.</p>
Jest Snapshots for API Contract Testing2018-07-10T00:00:00Zhttps://samhogy.co.uk/2018/07/jest-snapshots-for-api-contract-testing/<p><a href="https://jestjs.io">Jest</a> is a great JavaScript testing tool - it's simple, fast and easy to use. One of the key features that I enjoy using for UI testing is its [snapshot testin](https://jestjs.io/docs/en/snapshot-testing], a quick way of capturing the output of a test and comparing against an expected result. Opinion is mixed on snapshot testing and where it fits into the testing pyramid, but it's a useful tool in the utility belt. One trick to bear in mind is that snapshot testing isn't just useful for testing UI output, you can use it for a variety of purposes. In this post I will demonstrate how it can be used for testing your API contracts.</p>
<h2>How Snapshots Work</h2>
<p>In a Jest test, use the <code>expect(...).toMatchSnapshot()</code> method - that's it! The first time you run a test with a snapshot assertion, you'll see a new file is outputted into a <code>__snapshots__</code> folder. This represents the output of your test and should be committed to source control. On subsequent runs, the output of the test is diffed against the version stored in the file system. The test will pass if the outputs are equal. Should they not be, the test fails and produces a diff of expected vs actual recorded output. In the situation where you would like the change to be accepted, say you have redesigned a UI component, then you can run the tests with the <code>-u</code> or <code>--updateSnapshot</code> flag. This will overwrite the snapshot file with the new changes, which are again committed to source control.</p>
<h2>API Contract Testing</h2>
<p>When building an API it's best to avoid introducing breaking changes where necessary, such as by altering the response that is produced from an endpoint. As a developer I would like fast feedback if a change I make alters the result of any existing endpoint - so that I can quickly and safely iterate on changes. Jest snapshot testing is an excellent fit for this use case, as the data under test is inherently serializable!</p>
<p>Let's demonstrate this in action! First off, I will define an incrediblky simple Node server. It uses an express server to serve up some static data from two API endpoints: <code>/users</code> to get all users, or <code>/user/:id</code> to get a user by its id. Responses will be served up as JSON.</p>
<p>To set up the project:</p>
<pre><code>npm init --yes
npm install --save express jest supertest
</code></pre>
<p>Then define server.js</p>
<pre><code>const express = require('express');
const path = require('path');
const users = {
1: {
id: 1,
name: "Sam",
age: 29
},
2: {
id: 2,
name: "Taylor",
age: 10
},
3: {
id: 3,
name: "Cleo",
age: 11
}
};
const app = express();
app.get('/users', (req, res) => {
res.json(users);
});
app.get('/user/:id', (req, res) => {
const id = req.params.id;
const user = users[id];
if (user) {
res.json(user);
}
else {
res.sendStatus(404);
}
});
module.exports = app;
</code></pre>
<p>And index.js:</p>
<pre><code>const app = require ('./server');
app.listen(8080, () => {
console.log('server running);
});
</code></pre>
<p>Add the following scripts to the <code>package.json</code> to allow starting the express server and running the tests:</p>
<pre><code>"scripts": {
"start": "node .",
"test": "jest"
}
</code></pre>
<p>(Are you liking the simplicity of this as much as I am?!)</p>
<p>We'll start off by firstly writing a test against the <code>/users</code> endpoint, by creating a new file: <code>__tests__/server.spec.js</code>. I'll be using the <a href="https://www.npmjs.com/package/supertest">supertest</a> library to assist testing - it allows me to fire off requests against the express server, wait for the response and then make assertions. This ties in nicely with Jest, which natively supports <a href="https://jestjs.io/docs/en/tutorial-async">asynchronous tests</a> and if you're running on a recent version of node, you can write async-await syntax out of the box too.</p>
<p>The first test looks as follows:</p>
<pre><code>const request = require('supertest');
const app = require('../server');
describe('the server API', () => {
it('returns all users', async () => {
const response = await request(app).get('/users');
expect(response.body).toMatchSnapshot();
});
});
</code></pre>
<p>Run this and you'll notice a new file has been saved at <code>__tests__/__snapshots__/server.spec.js.snap</code>. This is the snapshot output:</p>
<pre><code>// Jest Snapshot v1, https://goo.gl/fbAQLP
exports[`the server API returns all users 1`] = `
Object {
"1": Object {
"age": 29,
"id": 1,
"name": "Sam",
},
"2": Object {
"age": 10,
"id": 2,
"name": "Taylor",
},
"3": Object {
"age": 11,
"id": 3,
"name": "Cleo",
},
}
`;
</code></pre>
<p>Remarkably, it's incredibly easy to read as it's effectively stringified JSON. Now if I were to change the output, such as changing the structure of the user object, and re-run the tests, I'll get a test failure which looks as follows:</p>
<img src="https://samhogy.co.uk/assets/jest-snapshot-failure.jpg" />
<p>As you can see, the difference has been clearly communicated. Through these tests we have defined and are enforcing the 'contract' that the API should adhere to. Of course, additive changes are typically considered 'fine', in that they won't 'break' existing consumers use of an endpoint. This is true, and for additive changes the above method would note that the output has changed. For these cases a simple re-run with the <code>--updateSnapshot</code> flag can record that this change was intentional.</p>
<p>Your own API boundary might not be the only API contract that you wish to test. Perhaps an API call made to your server causes a request to an upstream system? This use case is common, for example if your API acts as a gateway to a microservice-based backend, or places requests on a message queue, or speaks to mainframe systems, and so on. It may be important in your use case to additionally capture those requests too! In these situations you can then utilise snapshot testing at the integration test layer, utilising your server as a black box, inputting some data and observing the outputs.</p>
<p>Snapshot testing is almost certainly not a pancea - but for this use particular use case, where you're looking to record and diff serializable output for the purpose of regression testing - it is an ideal match. I would highly recommend considering it for use cases other than UI testing!</p>
Moving to a Monorepo2018-04-24T00:00:00Zhttps://samhogy.co.uk/2018/04/moving-to-a-monorepo/<p>In this post I will discuss a recent structural change that I made on a project - switching to hosting all of our front-end code in a single repository - a monorepo.</p>
<p>📖 <a href="https://github.com/sh1989/lerna-demo">Click here for a fully-worked example repository</a></p>
<p>I've been working on a suite of web applications, each with a consistent visual language, shared components and utility functions. This was originally developed as a single application with a run-time check determining which content the application would present. Out of necessity we quickly split the application into three standalone artifacts, which resulted in cloning the repository and deleting the unnecessary code. Over time we found ourselves struggling with the ability to share code between the repositories, without creating more repositories and increasing the amount of administration and co-ordination required to publish a single change. It led to many more copy-and-past changes, with the long-term risk of our "shared" libraries differing over time and become tough to re-integrate. Something had to give, so I researched ways to improve our process.</p>
<img src="https://samhogy.co.uk/assets/monorepo.png">
<h2>Monorepos as a Productivity Tool</h2>
<p>The NPM ecosystem favours small dependencies as a means of promoting code sharing and reusability. Investigate the <code>node_modules</code> folder of any node project and you'll see the abundance of small libraries, composed together to make larger pieces of work. But when it comes to writing applications at the user-level, we sometimes forget this compositional nature, instead hosting all of that code in one large package.</p>
<p>There's a strong benefit to breaking applications up into libraries - even if they end up being bundled together and deployed as one minified JavaScript file. Libraries introduce bounded contexts - logical abstractions that assist us from an organizational and maintenance perspective. In my particular case, the library structure also affords the ability to re-use code between applications.</p>
<p>When your libraries live in separate repositories, there's a further administrative overhead to managing those dependencies. There are multiple pull requests necessary and it becomes more involved for developers to test the integration works correctly. Especially where libraries are shared between multiple teams, implementing changes costs a notable amount of time. In larger corporate environments I've seen cases where code that really should live in a library ends up living in a team's application repository instead, as it's just quicker that way.</p>
<p>The reason this situation comes about is the one-to-one relationship between a repository producing a publishable NPM artifact. This doesn't actually have to be the case, it's just convention. Babel, Jest, Angular and React (amongst others) have all taken the approach of storing all their code in a single repository, breaking the one-to-one relationship between a single library being housed in a single repository. It's immediately obvious that in this situation, a lot of the organizational headaches discussed above simply disappear, improving developer productivity significantly.</p>
<h2>Using Lerna</h2>
<p><a href="https://lernajs.io/">Lerna</a> is a tool used to create monorepos for npm packages. In additional to describing how multiple packages should be stored, it provides a mechanism to publish artifacts to NPM and a means of 'linking' packages together where local dependencies exist. Furthermore it provides an efficient means of sharing third-party dependencies.</p>
<p>Setting up a new Lerna repository is as simple as installing <code>lerna</code> globally and then running the initialisation script: <code>lerna init</code> This will create a <code>lerna.json</code> file at the root of your repository.</p>
<h3>What Does It Look Like?</h3>
<p>In addition to a root-level <code>package.json</code>, each package will also have its own <code>package.json</code> descriptor. Packages are therefore free to define dependencies and scripts just like they would in a standalone repository.</p>
<p>The root-level <code>lerna.json</code> file describes where packages live within the repository. By default they're expected to live in a directory called <code>packages</code>.</p>
<p>For web applications, consider the difference between applications and libraries. At a high-level, applications are the entities that are built using a tool such as Webpack, Browserify or Parcel and deployed to a production environment. Libraries are smaller utilities that are composed into applications. As such a bundler that runns from an application-level package will include the code in a library-level package. Therefore libraries themselves don't need to be built themselves using a bundler. I typically export my library-level packages as ES6 modules and let the bundler tool take care of it.</p>
<p>If you don't wish to have your packages published to npm as standalone artifacts, set <code>private: true</code> in that package's <code>package.json</code> descriptor.</p>
<h3>Running Scripts</h3>
<p>One of the features that I really enjoy about npm is the ability to define scripts in the <code>package.json</code> file and run them using <code>npm run <script-name></code>. This manages the complexity of common scripts that are regularly run and simplifies CI on build servers. Lerna understands this common use case and supports it. By running <code>lerna run <script-name></code>, Lerna will run the script <code><script-name></code> in each package that contains a script of that name.</p>
<p>At any point you can always <code>cd</code> into a package directory and run the npm scripts defined there. But sometimes it's more convenient to have these scripts aliased in the root directory of your monorepo. That way there's no need to jump in and out of packages. When running a script via the <code>npm run</code> command, it's possible to supply a <code>--prefix</code> argument, which instructs to run the command from a given directory.</p>
<p>In my situation, I have a monorepo which hosts a series of applications. Sometimes I need to run all three in parallel. Each of these are bundled with Webpack, so for some packages, I could define the following scripts in my root-level <code>package.json</code> file:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"dev:package-a"</span><span class="token operator">:</span> <span class="token string">"npm run dev --prefix packages/package-a"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"dev:package-b"</span><span class="token operator">:</span> <span class="token string">"npm run dev --prefix packages/package-b"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"dev:package-c"</span><span class="token operator">:</span> <span class="token string">"npm run dev --prefix packages/package-c"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"prod:package-a"</span><span class="token operator">:</span> <span class="token string">"npm run prod --prefix packages/package-a"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"prod:package-b"</span><span class="token operator">:</span> <span class="token string">"npm run prod --prefix packages/package-b"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"prod:package-c"</span><span class="token operator">:</span> <span class="token string">"npm run prod --prefix packages/package-c"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> ...</span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>This small change has a significant impact when it comes to developer familiarisation with your monorepo structure!</p>
<h3>Importing From Existing Git Repositories</h3>
<p>Lerna provides a script to import existing git repositories into its repository structure while preserving git history. This is vital for developer productivity, otherwise useful contextual information disappears into a black hole.</p>
<p>Unfortunately, I struggled to import my existing git repositories using <code>lerna import</code>. I suspect that this was down to merge conflicts that required manual resolution. There is a <a href="https://github.com/lerna/lerna/blob/master/doc/troubleshooting.md">documented method</a> for fixing this, but this involves re-writing history in a way I was not entirely comfortable with. Instead, I utilised the magic that is the git subtree command to manually import into the new destination:</p>
<ol>
<li>Add the old repository as a remote: <code>git remote add <name> <path></code>.</li>
<li>Create a subtree to the remote. Use the prefix option to specify where you would like the subtree to be written to. By default that will be a directory in <code>packages</code>: <code>git subtree add -P packages/<package-name> <remote> <branch></code>.</li>
</ol>
<p>If you ever have to merge from the old repository, you can handle that via a subtree merge: <code>git subtree merge -P packages/<package-name> <remote> <branch></code>. Note that whilst this technique preserves git history, I lost tag information. I didn't mind so much because I can always <code>grep</code> for exact commits where a tag was created.</p>
<h3>Managing Dependencies</h3>
<p>This is where Lerna really comes into its own. Each package is free to declare its own dependencies, but for cases where several packages wish to depend on the same dependency, Lerna's bootstrap process will "hoist" the common dependencies so that they're stored in one location. For painless hoisting, add <code>lerna bootstrap --hoist</code> as a <code>postinstall</code> command in your top-level <code>package.json</code>. This means every time that <code>npm install</code> is run, the hoisting stage will happen immediately after, ensuring that all packages that depend on one another are correctly referenced.</p>
<h4>Example 1</h4>
<p>There are two packages that both depend on the same dependency, i.e React v16.3. When you bootstrap, React will be installed at the root-level <code>node_modules</code> folder only. Node's module resolution will find these as expected.</p>
<pre><code>node_modules
\ react
packages
\ package-a
\ package-b
</code></pre>
<h4>Example 2</h4>
<p>Similar to above, two packages depend on the same dependency, but require it to be available from the command-line. An example is Webpack, as you may wish to run Webpack from the package's folder. When you bootstrap, Webpack will be installed at the root-level <code>node_modules</code>, although a "symlink" exists in the <code>.bin</code> folder of each package's <code>node_modules</code></p>
<pre><code>node_modules
\ webpack
packages
\ package-a
\ node_modules
\ .bin
\ webpack.cmd
\ package-b
\ node_modules
\ .bin
\ webpack.cmd
</code></pre>
<h4>Example 3</h4>
<p>The Lerna repository contains two packages, where one depends on the other (package-b depends on package-a). When you bootstrap, a "symlink" to package-a exists in the <code>node_modules</code> folder of package-b.</p>
<pre><code>packages
\ package-a
\ package-b
\ node_modules
\ package-a
</code></pre>
<p>Note that in this case Lerna handles the versioning of the dependency to a local package for you, so as you publish new versions, it will update <code>package.json</code> files of your packages automatically.</p>
<p>The modern web development toolchain is quite advanced - including transpilation, bundling, linting and minification. Very often your packages will wish to perform these steps in a consistent manner. To achieve that, place your configuration files, such as your <code>.babelrc</code> and <code>eslintrc</code> files, at the root directory of the repository. This way they will be found through Node's dependency resolution algorithm and applied consistently.</p>
<h3>Testing Packages</h3>
<p>I use <a href="https://facebook.github.io/jest/">Jest</a> as a test runner and assertions library so I'll be referring exclusively to its operation here. In the interests of balance, other runners and frameworks are available!</p>
<p><a href="https://facebook.github.io/jest/blog/2017/05/06/jest-20-delightful-testing-multi-project-runner.html">Jest supports monorepos out of the box</a> which is super-handy, but opinions differ regarding whether you should run your tests as one large run, or parallelize running Jest tests from each package. I've used both.</p>
<h4>As One Runner</h4>
<p>For the first options, as with other shared config files, declare your jest config file in the root directory of your rpeository.</p>
<p>Here's an example of a bare-bones top-level Jest config file:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">module<span class="token punctuation">.</span>exports <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">collectCoverage</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">collectCoverageFrom</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token string">'packages/*/src/**/*.{js}'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string">'!**/node_modules/**'</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">moduleNameMapper</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token string-property property">".+\\.(css)$"</span><span class="token operator">:</span> <span class="token string">"identity-obj-proxy"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string-property property">"common"</span><span class="token operator">:</span> <span class="token string">"<rootDir>/packages/common/src"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">roots</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token string">'packages/'</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>This file is providing a mock (<a href="https://github.com/keyanzhang/identity-obj-proxy">specifically identity-obj-proxy</a>) so that CSS files aren't included. It's instructing Jest to look for tests and source files in the packages directory. Code coverage is configured to include this directory too, and exclude <code>node_modules</code> for obvious reasons. The only interesting part is the declaration of <code>common</code> in <code>moduleNameMapper</code> - in my sample repository this is a library package that lives within the monorepo and is used by other packages. In order for Jest to correctly find this, I found that I had to specifically add this module name transform.</p>
<p>From there, it's just a case of adding <code>"test": "jest"</code> in the root-level <code>package.json</code> scripts section.</p>
<h4>As Multiple Runners</h4>
<p>Should you wish to have Jest run standalone in each package, that's perfectly fine too! As you would if the repository were standalone for your package, define a <code>jest.config.js</code> in that package's directory. Similar to above, that package should have <code>"test": "jest"</code> in its <code>package.json</code> scripts section.</p>
<p>On your CI server you'll most likely wish to run <code>lerna run test</code>, which will execute the <code>test</code> script in each package that defines one. You can do that while developing locally too if you wish, but you may also find yourself just wanting to run the tests from a single package. It may be beneficial to therefore define a top-level <code>package.json</code> script to only run the tests in a given package, using the <code>--prefix</code> trick mentioned above. For a package called package-a, that could look like this:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token property">"test:package-a"</span><span class="token operator">:</span> <span class="token string">"npm test --prefix packages/package-a"</span></span></code></pre>
<p>Note however that if you wish to pass arguments, such as to instruct Jest to update snapshot files, you have to use two instances of <code>--</code> in the arguments:</p>
<pre class="language-shellsession"><code class="language-shellsession"><span class="token output">npm run test:package-a -- -- -u
</span></code></pre>
<h3>Publishing Packages</h3>
<p>Lerna has two versioning modes: fixed and independent. These versioning modes determine what occurs when you wish to publish npm packages from the Lerna repository.</p>
<p>Fixed mode is the default and it's my preferred option - in this mode the <code>lerna.json</code> file at the root of the repository tracks the version number for each package in the application. When running a publish build, for all packages that have been marked as updated (via the <code>lerna updated</code> command) then a new version is published with the <em>same version number</em>.</p>
<p>Given a Lerna repository with three packages - if the version is currently <code>0.1.0</code> and two of the three packages have been updated, Lerna will publish the two updated packages using the version number that you specify, let's say <code>0.2.0</code>.</p>
<p>Independent versioning is where the exact version number of each package can differ. Whilst this may not be immediately obvious, consider the case where a change to a library package constitutes an incremental change to one package that depends on it, but a breaking change in another. If you're an adopter of semantic versioning then you would wish for your new version numbers to reflect this change. Managing independent versioning requires more co-ordination, so I would only recommend it if absolutely necessary. By the way, I've written more on <a href="/2018/08/lerna-independent-mode-with-semver/">Lerna Independent Versioning</a>.</p>
<p>Lerna's publish command will create a new release, utilising several lerna sub-commands. It will identify any packages that have changed, bump their version and then push those packages to NPM. Finally a commit is applied to the repository and a new tag created. The version number can be supplied to the command - either exactly or using a sem-ver keyword like <code>patch</code>, <code>minor</code> etc. Alternatively do not supply a version for an interactive prompt. Remember to add <code>--yes</code> in CI environments to skip interactive prompts! A typical CI setup would typically look like:</p>
<ol>
<li><code>npm install</code> (with the above <code>postinstall</code> step to run <code>lerna bootstrap --hoist</code>)</li>
<li><code>lerna run build</code></li>
<li><code>npm test</code> (or <code>lerna run test</code>)</li>
<li><code>lerna publish patch --yes</code></li>
</ol>
<p>For further configurability you may wish to extend step four into a custom script that reads some configuration from a file, to determine the versioning scheme (cd-version or repo-version) to use, and the subsequent values of those options. This could be useful if you wish to publish a prerelease version or an exact version number from a branch, but your day-to-day incremental builds run on a patch basis. Furthermore this can be extended to a CI/CD pipeline scenario, where following your publish step you auto-deploy to environments.</p>
<h2>Conclusion</h2>
<p>I've enjoyed the flexibility of storing multiple packages in one repository. Especially in corporate environments where there's a level of administration required to set up repositories, Lerna's structure affords me the ability to create packages arbitrarily. This is great for code re-use, modularisation and provides an efficient way of building code via dependency re-use and parallel execution of npm scripts.</p>
<p>Taking this one step further, Yarn Workspaces takes the concept of Lerna but tightly integrates it into the package manager CLI. This makes installing dependencies even more efficient. Lerna is <a href="https://github.com/lerna/lerna#--use-workspaces">fully compatible with Yarn Workspaces</a>, so it's not necessarily a choice of one or the other.</p>
Evaluating Redux Saga Test Libraries2018-01-16T00:00:00Zhttps://samhogy.co.uk/2018/01/evaluating-redux-saga-test-libraries/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2018/01/16/evaluating-redux-saga-test-libraries.html">the Scott Logic blog</a></span></div>
<p>📖 <a href="https://github.com/sh1989/redux-saga-test-analysis">Click here for a fully-worked example of the code in this blog post</a></p>
<p>If you're a fan of <a href="https://github.com/redux-saga/redux-saga">Redux Saga</a> then you will have noticed the abundance of libraries to assist testing your sagas. This post takes an in-depth look into the different approaches to testing sagas, describing where five popular testing libraries fit into these approaches:</p>
<ul>
<li>"Native" testing (i.e without a helper library)</li>
<li><a href="https://github.com/wix/redux-saga-tester">redux-saga-tester</a></li>
<li><a href="https://github.com/stoeffel/redux-saga-test">redux-saga-test</a></li>
<li><a href="https://github.com/antoinejaussoin/redux-saga-testing/">redux-saga-testing</a></li>
<li><a href="https://github.com/jfairbank/redux-saga-test-plan">redux-saga-test-plan</a></li>
<li><a href="https://github.com/DNAinfo/redux-saga-test-engine">redux-saga-test-engine</a></li>
</ul>
<p>First, a brief introduction...</p>
<h2>What's a Saga?</h2>
<p>A Redux store represents the immutable global application state. Modifications to the state are handled through a process of dispatching an action. The action is handled by a (tree of) reducer functions, which <strong>reduce</strong> the action and the previous state into a new state object.</p>
<p>This functional approach makes Redux <a href="/2018/01/testing-redux-state/">easy to test</a>, but it means that your reducers become limited to storing the state. How do you make an API call? In functional programming, this is considered an 'impure' side-effect due to its unpredictability, reliance on external influences and time-dependence. Side-effects have no place in a pure functional reducer.</p>
<p>Enter Redux Saga. A saga uses the dispatching of an action as a signal to asynchronously perform a side effect. Your Redux application becomes nicely separated: the pure state updates to your reducers, and the impure work in your sagas. Alternatives do exist, such as <a href="https://github.com/gaearon/redux-thunk">redux-thunk</a>, but over time I've come to appreciate that sagas are more expressive and are my go-to choice.</p>
<p>Everything connects together using Redux's middleware chain. When an action is dispatched, Redux passes this through a chain of middleware functions. Reducers run after the middleware chain. Redux Saga is a middleware, which generates effects that will run <a href="https://redux-saga.js.org/docs/api/index.html#selectselector-args">after the reducers have updated the state</a>. This may seem initially strange, but makes sense when you consider that you want your <code>select</code> calls to be returning up-to-date values.</p>
<p>Redux Saga's middleware is responsible for starting, pausing and resuming sagas, as well as executing the effects that are yielded from a saga.</p>
<h2>Effects</h2>
<p>The following saga calls an API and then dispatches an action (either success or fail) with the response:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> call<span class="token punctuation">,</span> put <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'redux-saga/effects'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token comment">// Action creators</span></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">loadUser</span> <span class="token operator">=</span> <span class="token parameter">username</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'LOAD_USER'</span><span class="token punctuation">,</span> <span class="token literal-property property">payload</span><span class="token operator">:</span> username <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">loadUserSuccess</span> <span class="token operator">=</span> <span class="token parameter">user</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'LOAD_USER_SUCCESS'</span><span class="token punctuation">,</span> <span class="token literal-property property">payload</span><span class="token operator">:</span> user <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">loadUserFailure</span> <span class="token operator">=</span> <span class="token parameter">error</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'LOAD_USER_FAILURE'</span><span class="token punctuation">,</span> <span class="token literal-property property">payload</span><span class="token operator">:</span> error <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token comment">// Selectors</span></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">getContext</span> <span class="token operator">=</span> <span class="token parameter">state</span> <span class="token operator">=></span> state<span class="token punctuation">.</span>context<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token comment">// Reducer</span></span>
<span class="highlight-line"><span class="token keyword">const</span> defaultState <span class="token operator">=</span> Object<span class="token punctuation">.</span><span class="token function">freeze</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">loading</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">result</span><span class="token operator">:</span> <span class="token keyword">null</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">error</span><span class="token operator">:</span> <span class="token keyword">null</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">context</span><span class="token operator">:</span> <span class="token string">'test_app'</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">function</span> <span class="token function">reducer</span><span class="token punctuation">(</span><span class="token parameter">state <span class="token operator">=</span> defaultState<span class="token punctuation">,</span> action</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">switch</span><span class="token punctuation">(</span>action<span class="token punctuation">.</span>type<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token string">'LOAD_USER'</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token literal-property property">loading</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token string">'LOAD_USER_SUCCESS'</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token literal-property property">loading</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token literal-property property">result</span><span class="token operator">:</span> action<span class="token punctuation">.</span>payload <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token string">'LOAD_USER_FAILURE'</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token literal-property property">loading</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token literal-property property">error</span><span class="token operator">:</span> action<span class="token punctuation">.</span>payload <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">default</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> state<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token comment">// Saga</span></span>
<span class="highlight-line"><span class="token keyword">function</span><span class="token operator">*</span> <span class="token function">requestUser</span><span class="token punctuation">(</span><span class="token parameter">action</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> context <span class="token operator">=</span> <span class="token keyword">yield</span> <span class="token function">select</span><span class="token punctuation">(</span>getContext<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> user <span class="token operator">=</span> <span class="token keyword">yield</span> <span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> action<span class="token punctuation">.</span>payload<span class="token punctuation">,</span> context<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">yield</span> <span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserSuccess</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>error<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">yield</span> <span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserFailure</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>(Note - I will be using this reducer and saga for any test cases below!)</p>
<p><code>function*</code> indicates the saga is a generator function. When a generator <code>yield</code>s a value to its caller, execution of the generator is 'paused' until the calling function advances it, by calling either <code>next</code> or <code>throw</code>. It is possible to pass a value back to the generator function by supplying arguments to either of these calls. The generator then runs until the next <code>yield</code> or <code>return</code> statement.</p>
<p>In the example above, the saga is yielding a JavaScript object returned from a call to <code>select</code>, <code>call</code>, or <code>put</code>. These objects are known as <strong>effects</strong> - they are a <em>description</em> of some asynchronous operation that is to be performed by the Redux Saga middleware. <code>select</code> selects from the state, <code>call</code> calls a function and <code>put</code> dispatches an action. Others exist for more complex workflows, but these three are the most common.</p>
<h2>Testing Sagas</h2>
<p>The separation between the description of an effect and the execution of that effect is incredibly valuable for testing, for two reasons. Firstly, as the tests do not call an external function directly, mocking is easy. All that is needed is to pass the mock return value back into the saga. Secondly, the effects are just objects, so can be asserted using deep equality.</p>
<p>In my research, I have discovered there are different styles of testing sagas:</p>
<h3>1. Test the exact order</h3>
<p>It is possible to test a saga by simply stepping through its yielded effects. From there you can make assertions by performing a deep equality check on the effect object. This is the most straightforward approach to testing, with test code calling <code>.next</code> or <code>.throw</code> on the saga function to advance it forward.</p>
<p>Commonly, this approach to testing is best suited at the unit level, where you are most interested in verifying the saga works in isolation. By stepping through sagas in this manner, testing will typically focus on testing the exact order of yielded effects. Skipping a step is possible, albeit manually. There are times where drilling down to the exact order of effects is necessary to capture in tests. For example, I have a saga which is responsible for orchestrating polling an API endpoint, which means making sure that <code>delay</code> and <code>select</code> effects occur in a prescribed order, ensuring the saga is using up-to-date data. This testing approach suits this use case very well. However, a slight restructuring of a saga can lead to many test failures and cause annoyance. As with all testing, there's a fine line and you can risk <strong>testing that the code does what the code does, not what it ought to be doing</strong>.</p>
<h3>2. Recording only the effects that you're interested in</h3>
<p>Rather than assert the exact order of a saga's yielded effects, there is another approach which offers more flexibility. Sometimes it is not desirable to ensure a <code>select</code> happens at a particular point, just that it happens.</p>
<p>Testing in this style involves setting up a saga in advance with some mocks - normally for <code>select</code> and <code>call</code> effects. The saga is then started, run until completion, and then you can make your assertions. Test libraries that follow this style still provide you with a history of yielded effects, but also offer more capability than simply asserting that something happened.</p>
<p>This style aims for tests to assert that under a specific run condition a specific effect is observed. This is closer to integration testing than the style above, but still displays many characteristics of unit testing.</p>
<p>Placing this into a distinct bucket is difficult. As an approach it is definitely closer to an integration test than the style above, but still carries many characteristics of unit testing. Effectively the entity that runs the saga collects (or records) all of the effects, for you to assert on as you wish.</p>
<p>This approach offers a more stable test, it will be less brittle to change. It is common to see testing libraries that support this style of testing also be used to cover the exact ordering style, for times when it is desirable to be more prescriptive.</p>
<h3>3. Integration testing</h3>
<p>At the top end of the scale, there is the integration testing approach. Rather than test your saga in isolation, as a unit, integration testing libraries act as a mock middleware environment. They are supplied the saga, the reducer tree and an initial state. When the saga is started, some effects (i.e. <code>select</code> and <code>call</code>) will be applied to the state. Mock values can be provided for other effects (i.e. <code>call</code>).</p>
<p>Naturally these tests are slightly slower than a unit test, but fundamentally is just an extension of the methods described above. This is most suitable when testing a saga which manages a complex workflow and requires tight co-ordination with some state held in a reducer. Assertions on integration tests can still involve testing an exact order, or testing that certain effects happened during execution. They can also make assertions on the final state, or even the state at some point within the execution.</p>
<h2>Comparing the Libraries</h2>
<p>Each testing library implements one of the above styles in a slightly different way. This section provides a brief overview of each library.</p>
<h3>Native Testing</h3>
<p>Testing without any helper library involves manually stepping through the saga function and asserting effects as needed. It is most useful for testing the exact order of effects that are yielded.</p>
<p>Redux Saga also provides a <code>cloneableGenerator</code> utility function to reduce duplication in test code when a saga has branching logic. It wraps the saga's generator function, returning a new generator that can be advanced as normal. When a branch is approached, create a clone per branch and diverge at that point.</p>
<p>A complete example, therefore, looks like this:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'with redux-saga native testing'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> generator <span class="token operator">=</span> <span class="token function">cloneableGenerator</span><span class="token punctuation">(</span>loadUserSaga<span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> user <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token literal-property property">username</span><span class="token operator">:</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token literal-property property">isAdmin</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'gets the execution context'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> generator<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>value<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">select</span><span class="token punctuation">(</span>getContext<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'calls the API'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> generator<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token string">'tests'</span><span class="token punctuation">)</span><span class="token punctuation">.</span>value<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'tests'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'and the request is successful'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">let</span> clone<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">beforeAll</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> clone <span class="token operator">=</span> generator<span class="token punctuation">.</span><span class="token function">clone</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'raises success action'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> clone<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">.</span>value<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserSuccess</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'performs no further work'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> clone<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>done<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'and the request fails'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">let</span> clone<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">beforeAll</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> clone <span class="token operator">=</span> generator<span class="token punctuation">.</span><span class="token function">clone</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'raises failed action'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> error <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token string">"404 Not Found"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> clone<span class="token punctuation">.</span><span class="token function">throw</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">.</span>value<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserFailure</span><span class="token punctuation">(</span>error<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'performs no further work'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> clone<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>done<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toBe</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h3>redux-saga-test</h3>
<p>redux-saga-test provides a convenient shorthand syntax for asserting effects yielded from a saga. Rather than:
<code>expect(generator.next().value).toEqual(select(getContext));</code></p>
<p>You can instead use:
<code>expect.next().select(getContext);</code></p>
<p>I categorise this library as a "quality of life" improvement over native testing. Your tests are still following the native, exact-ordering style, but with fewer keystrokes to make your assertions. It is still required to manually advance the saga in your test code.</p>
<p>If you use Jest as your testing framework and choose to use <em>redux-saga-test</em>, you will need to provide <code>fromGenerator</code> with a function called <code>deepEqual</code>, which maps to Jest's <code>equals</code> function. Ideally you'd provide a global wrapper onto <code>fromGenerator</code> that your tests can import, preventing the need to do this in every single test file.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'with redux-saga-test'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> generator <span class="token operator">=</span> <span class="token function">loadUserSaga</span><span class="token punctuation">(</span><span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> expect <span class="token operator">=</span> <span class="token function">fromGenerator</span><span class="token punctuation">(</span>assertions<span class="token punctuation">,</span> generator<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'gets the execution context'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> expect<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">select</span><span class="token punctuation">(</span>getContext<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'gets the user'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> expect<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token string">'test_app'</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token operator">...</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h3>redux-saga-testing</h3>
<p>The redux-saga-testing approach is to override the test function (ie <code>it</code>), so that each test case advances the generator. The yielded value is then passed through to the test function:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> sagaHelper <span class="token keyword">from</span> <span class="token string">'redux-saga-testing'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> requestUser <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./saga'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'with redux-saga-testing'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> it <span class="token operator">=</span> <span class="token function">sagaHelper</span><span class="token punctuation">(</span>requestUser<span class="token punctuation">,</span> <span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> user <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'gets the username'</span><span class="token punctuation">,</span> <span class="token parameter">result</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">call</span><span class="token punctuation">(</span>getUsername<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> user<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'raises the success action'</span><span class="token punctuation">,</span> <span class="token parameter">result</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserSuccess</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'performs no further work'</span><span class="token punctuation">,</span> <span class="token parameter">result</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span>not<span class="token punctuation">.</span><span class="token function">toBeDefined</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>By adopting this library, you tightly couple the execution of your tests to the execution of a saga. You will be testing using the exact style of testing. individual steps can be skipped using an empty test function:</p>
<p><code>it('', () => {})</code></p>
<p>However, it can magnify the issues encountered by exact testing because of its lack of support for <code>cloneableGenerator</code>. Small structural changes to a saga will cause many test failures - especially so if there are multiple <code>describe</code> block to cover the saga's branching logic.</p>
<h3>redux-saga-test-plan</h3>
<p>redux-saga-test-plan supports exact order testing with its <code>testSaga</code> function. It provides a means of chaining assertions into a single test:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'with redux-saga-test plan'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'works as a unit test'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">testSaga</span><span class="token punctuation">(</span>loadUserSaga<span class="token punctuation">,</span> <span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">select</span><span class="token punctuation">(</span>getContext<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token string">'tests'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'tests'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserSuccess</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">isDone</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>When using the <code>testSaga</code> function it is possible to avoid having to assert on every effect, although as you have to call <code>next</code> yourself, you are still coupling your tests to the exact order in which the saga is yielding effects.</p>
<p>By instead using its <code>expectSaga</code> function, you can instead have the saga run until completion without having to manually advance it. You can provide any mock values for effects during setup. The chainable assertion syntax used in <code>testSaga</code> is supported on <code>expectSaga</code> too:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'works as an integration test'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">expectSaga</span><span class="token punctuation">(</span>loadUserSaga<span class="token punctuation">,</span> <span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">provide</span><span class="token punctuation">(</span><span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token function">select</span><span class="token punctuation">(</span>getContext<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> user<span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserSuccess</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>The <code>expectSaga</code> function can be enhance with a reducer, or some static state, so that it can run as an integration test. With this approach you can additionally assert on the final state of the reducer, after the saga runs until completion:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'works as an integration test with reducer'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">expectSaga</span><span class="token punctuation">(</span>loadUserSaga<span class="token punctuation">,</span> <span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">withReducer</span><span class="token punctuation">(</span>reducer<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">provide</span><span class="token punctuation">(</span><span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> user<span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">hasFinalState</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">loading</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">result</span><span class="token operator">:</span> user<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">error</span><span class="token operator">:</span> <span class="token keyword">null</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">context</span><span class="token operator">:</span> <span class="token string">'test_app'</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h3>redux-saga-test-engine</h3>
<p>redux-saga-test-engine adopts a similar approach to redux-saga-test-plan. It provides the <code>createSagaTestEngine</code> function, which accepts a list of effect types to record during a test run. You then start the saga and provide any mock return values, for effects such as <code>select</code> and <code>call</code>.</p>
<p>The result from the test function is a list of effects of the types you asked to record. It is possible to assert the exact ordering, albeit on a slightly reduced set.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'with redux-saga-test-engine'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> user <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token literal-property property">username</span><span class="token operator">:</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token literal-property property">isAdmin</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> collectEffects <span class="token operator">=</span> <span class="token function">createSagaTestEngine</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'PUT'</span><span class="token punctuation">,</span> <span class="token string">'CALL'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> actualEffects <span class="token operator">=</span> <span class="token function">collectEffects</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> loadUserSaga<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token function">select</span><span class="token punctuation">(</span>getContext<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> user<span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'gets the user'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>actualEffects<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">call</span><span class="token punctuation">(</span>getUser<span class="token punctuation">,</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token string">'test_app'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'raises the success action'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>actualEffects<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token function">put</span><span class="token punctuation">(</span><span class="token function">loadUserSuccess</span><span class="token punctuation">(</span>user<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'performs no further work'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>actualEffects<span class="token punctuation">.</span>length<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h3>redux-saga-tester</h3>
<p>As an integration test framework, redux-saga-tester provides an class to run a saga alongside a reducer, an initial state and potentially some extra middleware. Create an instance of <code>SagaTest</code> and then <code>start</code> the saga. From there, it is possible to assert that the final state is as expected. It also maintains a history of effects, making it possible to assert the exact order of effects, or a smaller set.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'with redux-saga-tester'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'works as an integration test with reducer'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> user <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token literal-property property">username</span><span class="token operator">:</span> <span class="token string">'sam'</span><span class="token punctuation">,</span> <span class="token literal-property property">isAdmin</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> <span class="token literal-property property">context</span><span class="token operator">:</span> <span class="token string">'test_app'</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">const</span> sagaTester <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">SagaTester</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">initialState</span><span class="token operator">:</span> defaultState<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">reducers</span><span class="token operator">:</span> reducer</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> sagaTester<span class="token punctuation">.</span><span class="token function">start</span><span class="token punctuation">(</span>loadUserSaga<span class="token punctuation">,</span> <span class="token function">loadUser</span><span class="token punctuation">(</span><span class="token string">'sam'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>sagaTester<span class="token punctuation">.</span><span class="token function">wasCalled</span><span class="token punctuation">(</span><span class="token constant">LOAD_USER_SUCCESS</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>sagaTester<span class="token punctuation">.</span><span class="token function">getState</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">loading</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">result</span><span class="token operator">:</span> user<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">error</span><span class="token operator">:</span> <span class="token keyword">null</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">context</span><span class="token operator">:</span> <span class="token string">'test_app'</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Conclusion</h2>
<p>In summary, the following table can be used for a quick outline as to where each testing library is best utilised:</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Exact</th>
<th>Recording</th>
<th>Integration</th>
<th>cloneableGenerator</th>
</tr>
</thead>
<tbody>
<tr>
<td>Native testing</td>
<td>Y</td>
<td>N</td>
<td>N</td>
<td>Y</td>
</tr>
<tr>
<td>redux-saga-test</td>
<td>Y</td>
<td>N</td>
<td>N</td>
<td>Y</td>
</tr>
<tr>
<td>redux-saga-testing</td>
<td>Y</td>
<td>N</td>
<td>N</td>
<td>N</td>
</tr>
<tr>
<td>redux-saga-test-plan</td>
<td>Y</td>
<td>Y</td>
<td>Y</td>
<td>N</td>
</tr>
<tr>
<td>redux-saga-test-engine</td>
<td>N</td>
<td>Y</td>
<td>N</td>
<td>N</td>
</tr>
<tr>
<td>redux-saga-tester</td>
<td>N</td>
<td>N</td>
<td>Y</td>
<td>N</td>
</tr>
</tbody>
</table>
<p>Depending on the nature of the saga under test, I'd advise picking what you consider to be the most appropriate method. <code>redux-saga-test-plan</code> provides comprehensive support for all styles of testing, but mixing and matching is equally a valid option. What's more important in my view is that the developer crafts tests with an understanding of the strengths and weaknesses of the approach that they are taking.</p>
<p>A complete working solution is provided on <a href="https://github.com/sh1989/redux-saga-test-analysis">Github</a>.</p>
Testing Redux State2018-01-02T00:00:00Zhttps://samhogy.co.uk/2018/01/testing-redux-state/<p>Redux at heart is a fundamentally simple concept. It's somewhere to put your global application state and it's a mechanism to mutate that state. One aspect of Redux I'm particularly interested in is its testability - how to approach your testing and what to actually test. In this post I'll dive into my preferences and the reasoning behind these choices.</p>
<p>On the surface testing Redux state is very simple. Let's quickly define a reducer, some actions and a selector function that I will use throughout:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token constant">INCREMENT</span> <span class="token operator">=</span> <span class="token string">'INCREMENT'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">increment</span> <span class="token operator">=</span> <span class="token parameter">c</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token constant">INCREMENT</span><span class="token punctuation">,</span> <span class="token literal-property property">payload</span><span class="token operator">:</span> c <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token constant">DECREMENT</span> <span class="token operator">=</span> <span class="token string">'DECREMENT'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">decrement</span> <span class="token operator">=</span> <span class="token parameter">c</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token constant">DECREMENT</span><span class="token punctuation">,</span> <span class="token literal-property property">payload</span><span class="token operator">:</span> c <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getCount</span> <span class="token operator">=</span> <span class="token parameter">s</span> <span class="token operator">=></span> s<span class="token punctuation">.</span>count<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> defaultState <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token literal-property property">count</span><span class="token operator">:</span> <span class="token number">0</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">reducer</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">state <span class="token operator">=</span> defaultState<span class="token punctuation">,</span> action</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">switch</span><span class="token punctuation">(</span>action<span class="token punctuation">.</span>type<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token constant">INCREMENT</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token literal-property property">count</span><span class="token operator">:</span> state<span class="token punctuation">.</span>count <span class="token operator">+</span> action<span class="token punctuation">.</span>payload <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token constant">DECREMENT</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token literal-property property">count</span><span class="token operator">:</span> state<span class="token punctuation">.</span>count <span class="token operator">-</span> action<span class="token punctuation">.</span>payload <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">default</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> state<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> reducer<span class="token punctuation">;</span></span></code></pre>
<p>Nothing too strange there, just a reducer to track a count with actions to increment and decrement. There's also a selector function, <code>getCount</code>. These functions are performing data retrieval, reading from the store and encapsulating exactly where on the store the data are. Typically your UI components will access state via the selectors, rather than reading from the state themselves. By doing so, in In larger Redux applications it's then possible to use libraries such as <a href="https://github.com/reactjs/reselect/">reselect</a> to provide efficient combination and memoization of selectors, to reduce redraws.</p>
<p>Let's write a test for the increment behaviour!</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> reducer<span class="token punctuation">,</span> <span class="token punctuation">{</span> getCount <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./state'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'counter'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> state <span class="token operator">=</span> <span class="token function">reducer</span><span class="token punctuation">(</span><span class="token keyword">undefined</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'none'</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'initially has a count of 0'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>state<span class="token punctuation">.</span>count<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'increments the counter'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> state<span class="token punctuation">.</span>count <span class="token operator">=</span> <span class="token number">3</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span>state<span class="token punctuation">.</span>count<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'retrieves count'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> state<span class="token punctuation">.</span>count <span class="token operator">=</span> <span class="token number">4</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span><span class="token function">getCount</span><span class="token punctuation">(</span>state<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>These tests aren't utilising the reducer in the way it does in production. The tests are directly manipulating and reading the state, whereas in production this will be handled through dispatching actions. Notice that there's also a duplication: the selector test is doing exactly the same as the test that's reading directly from the state. I'm not being too simplistic here, I've seen these kind of tests in real code. This typically happens when a developer wishes to demonstrate that they're practicing the craft of unit testing, albeit in the literal sense of unit testing everything. Whilst these tests will gain you some good code coverage metrics, in the long term they will hinder your project.</p>
<p>Refactoring the state will mean that your tests will fail. You'll then have to update the tests. But wait, isn't this a good thing? Red-green-refactor? Sure, it can help, but do you want to spend all of your time fixing failing tests? I want my tests to fail if I've broken a requirement or some sort of contract, rather than just rejigging the code around. I'd like for my tests to capture that the system is doing what <em>ought</em> to be doing, rather than testing that the it's doing what it's doing.</p>
<p>How can the tests be improved? I'd make my tests match how the code will be executed in reality. Dispatch actions to the reducer, don't set the state directly. Use the selectors to read the state. Unit testing purists may be thinking that this is no longer a 'unit' because it's not the smallest possible scope. Instead, I'd argue that it's the smallest <em>logical</em> scope for my tests.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> reducer<span class="token punctuation">,</span> <span class="token punctuation">{</span> getCount<span class="token punctuation">,</span> increment <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./state'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">'counter'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">let</span> state <span class="token operator">=</span> <span class="token function">reducer</span><span class="token punctuation">(</span><span class="token keyword">undefined</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'none'</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'initially has a count of 0'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span><span class="token function">getCount</span><span class="token punctuation">(</span>state<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">it</span><span class="token punctuation">(</span><span class="token string">'increments the counter'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> state<span class="token punctuation">.</span> <span class="token operator">=</span> <span class="token function">reducer</span><span class="token punctuation">(</span>state<span class="token punctuation">,</span> <span class="token function">increment</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">expect</span><span class="token punctuation">(</span><span class="token function">getCount</span><span class="token punctuation">(</span>state<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">toEqual</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>When I refactor the state, if I don't also update the selector, I've broken a contract and my test will fail. In the first example, test 3 would fail but test 2 would pass. Test 2 isn't representative of reality, it's just noise that slows the developer down. I can refactor with more safety. Furthermore, my tests are now focusing on what we're actually trying to capture and less that each moving part works in isolation.</p>
<p>Understandably, this is a fine line. To some degree it's playing with semantics and in such a small example it can feel marginal. However, consider a project with a larger state, complex interactions and a more involved workflow. A solid and representative approach to testing will help maintain test cases, making sure they will not hinder changing the code and act as sensible red flags for when functionality deviates from expected behaviour.</p>
Solving the GraphQL N+1 Problem2017-11-15T00:00:00Zhttps://samhogy.co.uk/2017/11/solving-the-graphql-n-plus-one-problem/<p>📖 <a href="https://github.com/sh1989/graphql-consultancy/tree/3e2ba987cbb5a23d8776cc99ed3ca266d23c2415">Click here for a fully-worked example of the code in this blog post</a></p>
<p>If I log all SQL queries in my GraphQL consultancy application and then execute a nested query against the GraphQL schema:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> developers <span class="token punctuation">{</span></span>
<span class="highlight-line"> name<span class="token punctuation">,</span></span>
<span class="highlight-line"> project <span class="token punctuation">{</span></span>
<span class="highlight-line"> name</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The following output is printed to the console:</p>
<pre><code>SELECT d.id, d.name, d.role FROM developer d - undefined
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":1}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":2}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":3}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":4}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":5}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":6}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":7}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":8}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":9}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":10}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":11}
SELECT a.projectId, p.name, p.description FROM assignments a LEFT JOIN project p ON (p.id = a.projectId) WHERE a.developerId = $id - {"$id":12}
</code></pre>
<p>That's a <strong>lot</strong> of SQL for a relatively small query!</p>
<p>Breaking it down, the first line isn't too bad. It's retrieving all of the developers in one go - you can probably guess that this is analagous to the <code>developers</code> field on the GraphQL root query. But as the query above is requesting the <code>project</code> field on each <code>Developer</code> type that is returned, for each developer there is a subsequent database lookup.</p>
<p>This is a common problem in the area of object-relational mapping, known as the n+1 problem. ORM frameworks such as Hibernate and Entity Framework can normally help you avoid this problem by batching requests, which is incredibly important when your database is slow, or scalability and performance are a concern. Efficient batching of SQL also helps avoid strange data inconsistencies that can arise from inefficient queries.</p>
<p>Over this series I have deliberately built the GraphQL application in a naïve manner, mainly to demonstrate that it is quite easy to run into performance issues such as this, if you're not careful. I see this as both a strength and weakness of GraphQL. When Facebook conceived GraphQL, they didn't want to replace an entire carefully-optimized backend stack. GraphQL was therefore envisaged as a thin wrapper around an existing backend stack. As such it makes no assumptions about your underlying data store. It could be SQL, it could be text files, it could even come from another web service. There's little magic under the hood, so effective GraphQL usage requires profiling, inspection and optimisation based on your own context. You are best-judged to make those decisions.</p>
<p>So, how can I solve this? First, let's take a step back.</p>
<h2>What's in a Query?</h2>
<p>Ultimately a GraphQL query is just a bunch of function calls. At any point in time there's an object in scope that defines a set of functions. These are your GraphQL types, and the fields on that type, respectively.</p>
<p>The query above therefore breaks down to:</p>
<ul>
<li>On the root query object, call the function to get all developers.</li>
<li>On each object that is returned, call the function to get the project.</li>
</ul>
<p>GraphQL is asynchronous by default. It will not respond to a caller until all fields have been resolved. Some fields can be resolved immediately (e.g. returning a property on an already-initialised object), whereas others may require some time to resolve (e.g if they require a database lookup). The entity that handles incoming queries will await all fields resolving.</p>
<p>To avoid the n+1 problem, I'd like to somehow batch the SQL queries into a single query. Something like <code>SELECT * from project WHERE id IN (1, 2, 3)</code> would be ideal. This would require us to <em>reduce</em> the incoming requests into one query and then <em>map</em> the results back out to the individual fields.</p>
<p>So here's an idea. Rather than making an individual SQL request per project, I could instead:</p>
<ol>
<li>In the root query field, create a data structure to represent queries that should be batched.</li>
<li>Add the id to a collection of queries to be batched</li>
<li>Compose an optimized SQL query</li>
<li>Resolve each field with the entities from the database.</li>
</ol>
<h2>Introducing Data Loader</h2>
<p><a href="https://github.com/facebook/dataloader">DataLoader</a> is Facebook's open-source utility for asynchronous batching of data loading. This is commonly used within GraphQL applications to do just the type of batching discussed above.</p>
<p>A DataLoader instance is constructed by passing a function, which maps an array of ids into an array of values. As this too is asynchronous by default you can return a Promise that resolves to the array of values.</p>
<p>In my code, I define a function that accepts the database accessor and returns an object of loaders. One DataLoader is defined to retrieve the project assignments for a list of developers, identified by id.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">buildLoaders</span> <span class="token operator">=</span> <span class="token parameter">db</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">projectAssignments</span><span class="token operator">:</span> <span class="token keyword">new</span> <span class="token class-name">DataLoader</span><span class="token punctuation">(</span><span class="token parameter">developerIds</span> <span class="token operator">=></span> <span class="token function">getProjectAssignments</span><span class="token punctuation">(</span>developerIds<span class="token punctuation">,</span> db<span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>When constructing your GraphQL endpoint, it's possible to add to a context object, which is passed down to all fields. This technique is useful if you have a singleton that you'd like to share. I already use this to give fields access to the database object, so I can also add the object containing my loaders to it:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">sqlite<span class="token punctuation">.</span><span class="token function">open</span><span class="token punctuation">(</span><span class="token string">'consultancy.db'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">cached</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> sqlite<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'PRAGMA foreign_keys=on'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> graphQLApp <span class="token operator">=</span> <span class="token function">express</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> db <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function-variable function">get</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token operator">...</span>args</span><span class="token punctuation">)</span> <span class="token operator">=></span> sqlite<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token operator">...</span>args<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">all</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token operator">...</span>args</span><span class="token punctuation">)</span> <span class="token operator">=></span> sqlite<span class="token punctuation">.</span><span class="token function">alll</span><span class="token punctuation">(</span><span class="token operator">...</span>args<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">run</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token operator">...</span>args</span><span class="token punctuation">)</span> <span class="token operator">=></span> sqlite<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token operator">...</span>args<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> loaders <span class="token operator">=</span> <span class="token function">buildLoaders</span><span class="token punctuation">(</span>db<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> graphQLApp<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token string">'/'</span><span class="token punctuation">,</span> <span class="token function">graphQLHTTP</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">graphiql</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">pretty</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> schema<span class="token punctuation">,</span></span>
<span class="highlight-line"> rootValue<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">context</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> db<span class="token punctuation">,</span></span>
<span class="highlight-line"> loaders</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> graphQLApp<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span><span class="token constant">GRAPHQL_PORT</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Now on the field <code>project</code> on type <code>Developer</code>, rather than calling a function that accesses the database directly, I add the id onto the loader:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Developer</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> name<span class="token punctuation">,</span> role</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>role <span class="token operator">=</span> <span class="token function">toRole</span><span class="token punctuation">(</span>role<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span><span class="token parameter">obj<span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> ctx<span class="token punctuation">.</span>loaders<span class="token punctuation">.</span>projectAssignments<span class="token punctuation">.</span><span class="token function">load</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>id<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>When the data loader function resolves with its array of values, it fans these values back out to their callers, so the result of the <code>load</code> function above will be a single object.</p>
<p>Finally, I need to change the database query that I construct, because now it's looking up several entities rather than a just one:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getProjectAssignments</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ids<span class="token punctuation">,</span> db</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span>'</span>
<span class="highlight-line"> <span class="token constant">SELECT</span> d<span class="token punctuation">.</span>id<span class="token punctuation">,</span> p<span class="token punctuation">.</span>id <span class="token keyword">as</span> projectId<span class="token punctuation">,</span> p<span class="token punctuation">.</span>name<span class="token punctuation">,</span> p<span class="token punctuation">.</span>description</span>
<span class="highlight-line"> <span class="token constant">FROM</span> developer d</span>
<span class="highlight-line"> <span class="token constant">LEFT</span> <span class="token constant">JOIN</span> assignments a</span>
<span class="highlight-line"> <span class="token constant">ON</span> <span class="token punctuation">(</span>d<span class="token punctuation">.</span>id <span class="token operator">=</span> a<span class="token punctuation">.</span>developerId<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token constant">LEFT</span> <span class="token constant">JOIN</span> project p</span>
<span class="highlight-line"> <span class="token constant">ON</span> <span class="token punctuation">(</span>p<span class="token punctuation">.</span>id <span class="token operator">=</span> a<span class="token punctuation">.</span>projectId<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token constant">WHERE</span> d<span class="token punctuation">.</span>id</span>
<span class="highlight-line"> <span class="token constant">IN</span> <span class="token punctuation">(</span>$<span class="token punctuation">{</span>ids<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">i</span> <span class="token operator">=></span> <span class="token string">'?'</span><span class="token punctuation">)</span><span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> '<span class="token punctuation">,</span> ids<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">rows</span> <span class="token operator">=></span> rows<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token parameter">id</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> entry <span class="token operator">=</span> rows<span class="token punctuation">.</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token parameter">row</span> <span class="token operator">=></span> row<span class="token punctuation">.</span>id<span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">===</span> id<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>entry<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>entry<span class="token punctuation">.</span>projectId<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>projectId<span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> r<span class="token punctuation">.</span>name<span class="token punctuation">,</span> r<span class="token punctuation">.</span>description<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">null</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">No Developer exists with id </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Obviously the latter is more complex than the former. It took me a while to get right.</p>
<p>DataLoader throws an error if the array of values is different in length to the array of ids. Therefore it's important to make sure that the loader function returns something for each requested key. The order matters too. The query above uses <a href="http://www.sqlite.org/c3ref/bind_blob.html">bound parameters</a>, so that if an id isn't found in the database it doesn't cause the entire query to fail. This also gives you some protection against SQL injection, compared to embedding the ids into the query directly. But, when mapping the resultant rows to instances of the <code>Project</code> type, I must ensure that for ids that didn't produce a result they're padded out with an <code>Error</code>, and for ids that did produce a result but not an inner result (i.e a developer with no project), I return a <code>null</code>.</p>
<p>(Note: I ended up extracting the above function into a generic "whereIn" function to avoid code duplication. You can see that form of the solution in the example linked at the beginning of the post.)</p>
<p>Re-running the original GraphQL query, the database logging output becomes:</p>
<pre><code>SELECT d.id, d.name, d.role FROM developer d - undefined
SELECT d.id, p.id as projectId, p.name, p.description
FROM developer d
LEFT JOIN assignments a
ON (d.id = a.developerId)
LEFT JOIN project p
ON (p.id = a.projectId)
WHERE d.id IN (?,?,?,?,?,?,?,?,?,?,?,?) - ["1","2","3","4","5","6","7","8","9","10","11","12"]
</code></pre>
<p>Hooray!</p>
<h2>Where Else?</h2>
<p>One other area that you can consider using DataLoader for is when your schema has a field to resolve a single instance of some type. Seems strange, but remember that with GraphQL, you can assign these to values, making it possible to actually request several single instances in one query.</p>
<p>This query does just that, except one of the ids is obviously erroneous.</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line">query <span class="token punctuation">{</span></span>
<span class="highlight-line"> sam<span class="token operator">:</span> developer(id<span class="token operator">:</span> <span class="token string">"2"</span>) <span class="token punctuation">{</span></span>
<span class="highlight-line"> name</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> gary<span class="token operator">:</span> developer(id<span class="token operator">:</span> <span class="token string">"3"</span>) <span class="token punctuation">{</span></span>
<span class="highlight-line"> name</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> dustin<span class="token operator">:</span> developer(id<span class="token operator">:</span> <span class="token string">"eleven"</span>) <span class="token punctuation">{</span></span>
<span class="highlight-line"> name</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>By defining another DataLoader to load developers, these can be batched. As the method I used above maps any unfound entries to instances of an <code>Error</code>, GraphQL can accommodate this in its response, by highlighting the error:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"errors"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"No developer exists with id eleven"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"locations"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"line"</span><span class="token operator">:</span> <span class="token number">8</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"column"</span><span class="token operator">:</span> <span class="token number">3</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token string">"dustin"</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"data"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"sam"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Gary"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"gary"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Chris"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"dustin"</span><span class="token operator">:</span> <span class="token null keyword">null</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Conclusion</h2>
<p>We've seen how easy it can be to write sub-optimal SQL using GraphQL. DataLoader is a simple mechanism by which you can avoid the n+1 problem by batching multiple database requests into a more efficient database call. I'd highly recommend considering this solution early in development to prevent scalability issues. But always remember to profile these things!</p>
GraphQL Mutations2017-08-26T00:00:00Zhttps://samhogy.co.uk/2017/08/graphql-mutations/<p>📖 <a href="https://github.com/sh1989/graphql-consultancy/tree/61bc0af67ac76c6efad54f2cbc905670433e0f86">Click here for a fully-worked example of the code in this blog post</a></p>
<p>In this post I will continue the development of a GraphQL endpoint. So far it is able to return the developers, projects and skills for a fictitious software consultancy firm. In past posts I've developed the initial schema, added a SQLite database and then added the ability to make queries more selective with arguments. These changes have all been one-way as they all relate to retrieving data. Many APIs support the ability to write data too - in REST this is a simple HTTP POST action. Let's now take a look at how GraphQL addresses this - using <em>mutations</em>.</p>
<p>A GraphQL schema has a root <code>Query</code> type, where you define the top-level queries. Mutations are very similar to queries: they have a name, they accept arguments (known as inputs) and then they return some data of a given type. The main difference is that mutations are performing a write operation, immediately followed by a read. By convention a mutation returns the data that was affected during the write.</p>
<p>Just like queries, for each mutation you define GraphQL will call a function. This means that there's no actual restriction on a query from performing a write - the distinction only exists in your schema. A mutation is actually a query, but it's doing something else first. By separating out pure reads into a <code>Query</code> type and writes into a <code>Mutation</code> type, it makes it clear to the user what they can expect to happen when invoking your GraphQL endpoint.</p>
<p>I'm going to be defining mutations that broadly fit within three categories. I won't cover every single mutation in this post, instead I'll address some of the more interesting ones within each category. You'll be able to inspect the source code if you'd like to see the complete set.</p>
<ol>
<li><strong>Mutations that add data</strong> - a developer, a project, a skill</li>
<li><strong>Mutations that delete data</strong> - a developer, a project</li>
<li><strong>Mutations that update data</strong> - assigning a project, changing a developer's role, updating a developer's competencies</li>
</ol>
<h2>Inputs</h2>
<p>Let's cover input types in a broad sense. Unlike REST, GraphQL expresses a type system via its schema. In the previous posts I introduced the concept of arguments that can passed into a query. Mutations combine the type system and argument capabilities as a way of defining what data a mutation is expecting to be supplied.</p>
<p>However, the types that you can define on a mutation are more restrictive. This is to prevent the possibility for circular dependencies. Only scalars (Int, String, Boolean, Enums, etc) are accepted. If you wish to define a complex argument, it must be defined as a special <code>input</code> type, whose fields are restricted to scalar values. You can also use the <code>!</code> operator to define a field as mandatory.</p>
<p>By convention all of my mutations accept one mandatory argument, named input. It is an <code>input</code> type and it defines all of the data that's expected for that mutation. It may seem like overkill when there's just one piece of data required, but I've chosen to do this for consistency.</p>
<p>I define the root <code>Mutation</code> type on my schema, alongside an input type for each mutation:</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">input DeveloperInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">role</span><span class="token operator">:</span> Role<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input DeleteDeveloperInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> String<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input ProjectInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">description</span><span class="token operator">:</span> String<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input DeleteProjectInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> String<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input SkillInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input AssignCompetencyInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">developer</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">skill</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">rating</span><span class="token operator">:</span> Int<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input AssignProjectInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">developer</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">project</span><span class="token operator">:</span> String<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">input AssignRoleInput <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">developer</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">role</span><span class="token operator">:</span> Role<span class="token operator">!</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">type Mutation <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">addDeveloper</span><span class="token punctuation">(</span>input<span class="token operator">:</span> DeveloperInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> Developer<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">deleteDeveloper</span><span class="token punctuation">(</span>input<span class="token operator">:</span> DeleteDeveloperInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> String<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">addProject</span><span class="token punctuation">(</span>input<span class="token operator">:</span> ProjectInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> Project<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">deleteProject</span><span class="token punctuation">(</span>input<span class="token operator">:</span> DeleteProjectInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> String<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">addSkill</span><span class="token punctuation">(</span>input<span class="token operator">:</span> SkillInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> Skill<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">assignCompetency</span><span class="token punctuation">(</span>input<span class="token operator">:</span> AssignCompetencyInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> Developer<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">assignProject</span><span class="token punctuation">(</span>input<span class="token operator">:</span> AssignProjectInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> Developer<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">assignRole</span><span class="token punctuation">(</span>input<span class="token operator">:</span> AssignRoleInput<span class="token operator">!</span><span class="token punctuation">)</span> <span class="token operator">:</span> Developer</span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>As you'd expect by this point, each mutation is just another function call that is defined on my resolver object. Notice how these are just calling through to functions that I will implement in my <code>api.js</code> file.</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> rootValue <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function-variable function">developer</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">developers</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> assigned <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDevelopers</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> assigned<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">project</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProject</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">projects</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter">obj<span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProjects</span><span class="token punctuation">(</span>ctx<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">skill</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getSkill</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">skills</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> order <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getSkills</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> order<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">addDeveloper</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">createDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">deleteDeveloper</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">removeDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">addProject</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">createProject</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">deleteProject</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">removeProject</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">addSkill</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">createSkill</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">assignCompetency</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">setCompetency</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">assignProject</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">setProject</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">assignRole</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> input <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">editRole</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> input<span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>Invoking a mutation requires you to first state that you're performing a mutation, followed by the name of the mutation. This is a slightly more verbose syntax than just making a simple query:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">mutation <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">addDeveloper</span><span class="token punctuation">(</span>input<span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">"Colin"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">role</span><span class="token operator">:</span> <span class="token constant">LEAD</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> id<span class="token punctuation">,</span></span>
<span class="highlight-line"> name<span class="token punctuation">,</span></span>
<span class="highlight-line"> role</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>As they're part of the schema, interactive environments like GraphiQL are able to provide auto-completion!</p>
<img src="https://samhogy.co.uk/assets/graphql/mutation_required.png" />
<h2>Addition Mutations</h2>
<p>Time to drill down into how some of these mutations are implemented.</p>
<h3>Adding a Developer</h3>
<p>When it comes to adding a developer, I'm only asking for some of the fields that are defined on the <code>Developer</code> type: namely a role and a name. I'm assuming that it is unlikely that project assignment will happen immediately and that it will be up to the developer at a later point to update their list of competencies (where I work, this is arranged via a periodic staff survey). So, I define a function to perform an <code>INSERT</code> operation into the database. This will auto-generate an id by incrementing from the last used ID.</p>
<p>As a mutation is a write and then a read, I also need to make this function return something. Database functions are Promise-based in <code>node-sqlite</code>, so I am able to chain a function to do a read onto the function that performed the write. When Promises are chained in this way, the result of one function is passed to the next in the chain. The result object returned from the write function contains a <code>lastID</code> field that's set to the id of the new item. I already have functions to lookup by an ID, so this can be invoked. Should something go wrong with the insertion, then I'd need to report an error back to the caller. The result object from the database also contains a <code>changes</code> field. If this is zero, the operation was unsuccessful. I rerturn an <code>Error</code> in this scenario, which rejects the Promise:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">createDeveloper</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> developer</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'INSERT INTO developer (name, role) VALUES (?, ?)'</span><span class="token punctuation">,</span> developer<span class="token punctuation">.</span>name<span class="token punctuation">,</span> <span class="token function">fromRole</span><span class="token punctuation">(</span>developer<span class="token punctuation">.</span>role<span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>changes <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> result<span class="token punctuation">.</span>lastID<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token string">'Unable to create developer'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>The role of a developer is specified as an enum. GraphQL enums are string-based (so the <code>Role</code> type has the allowed values of GRAD, DEV, SENIOR, LEAD). In SQL these are stored as integer values, so I also had to define a helper function <code>fromRole</code> which converted the enum from a String to an Int.</p>
<p>A quick note about security. While the GraphQL validation layer is checking that the <code>name</code> of the developer is a String, it is <em>not</em> performing any further validation on the string itself, such as whether it's a sensible string to store in the database as-is. In a production system I'd remain hesitant and perform my own validation before storing it in the database. SQL injection attacks are remarkably still one of the top 10 attack vectors for applications. GraphQL is adding little, by default, towards solving SQL injection. This is deliberate - GraphQL makes no assumptions about <em>where</em> your data are coming from. Although, it is possible to define further restrictions with <a href="https://www.npmjs.com/package/graphql-custom-types">custom types</a>.</p>
<h3>Adding a Skill</h3>
<p>A skill is something that a developer can possess. In the database the <code>competencies</code> table represents an association between a <code>developer</code> and a <code>skill</code>. As a developer I could say that I possess the C# skill, and I'd rate myself as 8/10 on that front. It doesn't make sense for us to have two skills named C#, so I'd like for it not to be possible to have a duplicate skill.</p>
<p>Unfortunately I made a mistake by not adding this as a constraint to the database up-front. This is fixed by defining a new migration file, which adds the unique constraint. At startup the migration files are executed sequentially. Up until this point all the data are static, so I'm comfortable making this change:</p>
<p><em>002-skills-unique-names.sql</em></p>
<pre class="language-sql"><code class="language-sql"><span class="highlight-line"><span class="token comment">-- Up</span></span>
<span class="highlight-line"><span class="token keyword">CREATE</span> <span class="token keyword">UNIQUE</span> <span class="token keyword">INDEX</span> skill_name <span class="token keyword">ON</span> skill<span class="token punctuation">(</span>name<span class="token punctuation">)</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token comment">-- Down</span></span>
<span class="highlight-line"><span class="token keyword">DROP</span> <span class="token keyword">INDEX</span> skill_name</span></code></pre>
<p>A SQL constraint violation is naturally an error. <code>node-sqlite</code> will reject the promise outright when this happens:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">createSkill</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> skill</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'INSERT INTO skill (name) VALUES (?)'</span><span class="token punctuation">,</span> skill<span class="token punctuation">.</span>name<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>changes <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token function">getSkill</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> result<span class="token punctuation">.</span>lastID<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token string">'Unable to create skill'</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token string">'Unable to create skill'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Deletion Mutations</h2>
<p>Deletion operations are straightforward - they're just <code>DELETE</code> calls into the database. The foregin key constraints on the table will handle cascade the delete onto tables that contain derived data. Deleting a developer therefore will also delete the entry in the <code>assignments</code> table (if they're assigned to a project) and remove all their <code>competencies</code>.</p>
<p>The interesting question is what is a sensible thing to return. Some prefer to return the deleted item - although be careful doing that, as your database might perform further lookups using an id that's no longer present. I prefer to return the id of the deleted item.</p>
<p>For deleting a project, this looks like:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">removeProject</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> <span class="token punctuation">{</span> id <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'DELETE FROM project WHERE id = $id'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">$id</span><span class="token operator">:</span> id <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>changes <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Unable to delete project with id </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Update Mutations</h2>
<p>This is where things get interesting! I've defined three update mutations and as they're a bit more complicated in nature than the basic additions or deletions, allow me to cover each one individually.</p>
<h3>Updating a Developer's Role</h3>
<p>Promoting or demoting a developer is a case of asking the caller to provide the id of the developer alongside the new <code>Role</code> which they'd like to have. As <code>Role</code> is an enum, it's scalar and therefore permitted to be used in a GraphQL input type. The entire developer object is then retrieved from the database and returned to the caller.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">editRole</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> roleChange</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'UPDATE developer SET role = $role WHERE id = $id'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">$role</span><span class="token operator">:</span> <span class="token function">fromRole</span><span class="token punctuation">(</span>roleChange<span class="token punctuation">.</span>role<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token literal-property property">$id</span><span class="token operator">:</span> roleChange<span class="token punctuation">.</span>developer <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>changes <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> result<span class="token punctuation">.</span>lastID<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Unable to edit role </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>roleChange<span class="token punctuation">.</span>role<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> for develoepr </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>roleChange<span class="token punctuation">.</span>developer<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h3>Assigning a Project</h3>
<p>The input for this mutation is simple, requiring a developer's id and the project id.</p>
<p>The complexity with this mutation is that I have to be careful of the fact that a developer may already be assigned to a project. Therefore it is not possible to run a straightforward <code>UPDATE</code> statement, as above. SQLite supports the <code>INSERT OR REPLACE</code> operation, which will update an existing entry in a table if there is already a matching entry in the table. Otherwise it will add a new entry.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">setProject</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> assignment</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'INSERT OR REPLACE INTO assignments (developerId, projectId) VALUES (?, ?)'</span><span class="token punctuation">,</span> assignment<span class="token punctuation">.</span>developer<span class="token punctuation">,</span> assignment<span class="token punctuation">.</span>project<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>changes <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> result<span class="token punctuation">.</span>lastID<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Unable to assign developer </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>assignment<span class="token punctuation">.</span>developer<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> to project </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>assignment<span class="token punctuation">.</span>project<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Assigning a Competency</h2>
<p>To assign a competency, the user will specify a developer's id, a skill id and a rating. There is a similar problem to above, where a competency may already exist. Perhaps a developer has attended a training course and wishes to claim they're now more knowledgeable at C#. They may also have recently developed a new skill (let's say F#) and wishes to tell the company about that. I use the <code>INSERT OR REPLACE</code> operation for this scenario:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">setCompetency</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> assignment</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">run</span><span class="token punctuation">(</span><span class="token string">'INSERT OR REPLACE INTO competencies (developerId, skillId, value) VALUES (?, ?, ?)'</span><span class="token punctuation">,</span> assignment<span class="token punctuation">.</span>developer<span class="token punctuation">,</span> assignment<span class="token punctuation">.</span>skill<span class="token punctuation">,</span> assignment<span class="token punctuation">.</span>rating<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>changes <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> assignment<span class="token punctuation">.</span>developer<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Unable to assign developer </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>assignment<span class="token punctuation">.</span>developer<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> to skill </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>assignment<span class="token punctuation">.</span>skill<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Next Steps</h2>
<p>This GraphQL endpoint is developing quite the feature set now. It's possible to imagine a range of line-of-business applications that all speak to this one GraphQL endpoint to update the business' data. A HR web application could be responsible for adding developers to the staff roster. A client management application could then be responsible for managing the list of projects and assigning staff to these projects.</p>
<p>However, for it to be truly representative, let's say that we only want to enable client managers to add projects, and we only want HR to be able to add staff. This is a pretty interesting challenge and I'll cover how to achieve it in the next part of the blog series.</p>
GraphQL Arguments2017-08-23T00:00:00Zhttps://samhogy.co.uk/2017/08/graphql-arguments/<p>📖 <a href="https://github.com/sh1989/graphql-consultancy/tree/c29b3e42da11092a4b96d58de39d26cf65a9107f">Click here for a fully-worked example of the code in this blog post</a></p>
<p>I'm building a GraphQL server to provide and API for a fictitious software consultancy firm. So far a user can query for a list of developers, their competencies and for their project assignments.</p>
<p>In this post I'm going to take this a bit further and look at how <em>arguments</em> are another useful tool in the GraphQL arsenal. They make your server more interactive, allowing for callers to be more specific when querying for data. With the use of GraphQL's introspection feature, it's also possible for a user to query which arguments are supported. Interactive environments like GraphiQL implement this feature to provide as-you-type auto-completion. It's a fantastic way of conveying the ability to "explore" your API surface to a user.</p>
<img src="https://samhogy.co.uk/assets/graphql/graphiql_autocomplete.png" />
<p>Arguments are added to fields, either at the root query level, or an individual field on a type. They're defined in the schema and they have too have types. I'm going to support some arguments in my GraphQL API, in order to demonstrate the different ways in which arguments can be used:</p>
<ul>
<li>Getting the highest-rated N competencies for a developer - Optional Arguments</li>
<li>Getting an individual developer or project - Mandatory Arguments</li>
<li>Getting all assigned (or unassigned) developers</li>
<li>Getting skills, ordered A-Z or Z-A - Default Arguments</li>
</ul>
<p>Let's dive in!</p>
<h2>Getting the Highest-Rated Competencies</h2>
<p>When defining new GraphQL arguments, I always like to think of a <em>question</em> that is being asked. Allowing to display the highest-rated competencies for the developers answers "What are the staff's specialisms?". This could be useful when the consultancy firm is looking for new projects to take on.</p>
<p>I already have a field that retrieves a list of competencies, on the <code>Developer</code> type. I add the new argument to this field - <code>top</code>, of type <code>Int</code>:</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">type Developer <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">competencies</span><span class="token punctuation">(</span>top<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> <span class="token punctuation">[</span>Competency<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">role</span><span class="token operator">:</span> Role<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">project</span><span class="token operator">:</span> Project</span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Although the schema has changed, this is not a breaking change for existing callers. As the argument is optional it has no effect on any existing consumers of this field. The change is purely additive.</p>
<p>Before jumping into the underlying implementation, a quick recap. Previously the <code>Developer</code> class looked like this:</p>
<p><em>developer.js</em></p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Develoepr</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span>id<span class="token punctuation">,</span> name<span class="token punctuation">,</span> role<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>role <span class="token operator">=</span> <span class="token function">toRole</span><span class="token punctuation">(</span>role<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">competencies</span><span class="token punctuation">(</span>obj<span class="token punctuation">,</span> ctx<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">getCompetenciesForDeveloper</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>id<span class="token punctuation">,</span> ctx<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span>obj<span class="token punctuation">,</span> ctx<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">getProjectAssignment</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>id<span class="token punctuation">,</span> ctx<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>There are two functions that resolve data dynamically, to show how fields can be resolved asynchronously. The first argument to these functions was previously unused. When arguments are queried for on a particular field, they'll be defined on this first object. Therefore I retrieve the <code>top</code> argument from this object as such:</p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token function">competencies</span><span class="token punctuation">(</span><span class="token punctuation">{</span> top <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">getCompetenciesForDeveloper</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>id<span class="token punctuation">,</span> ctx<span class="token punctuation">,</span> top<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The <code>getCompetenciesForDeveloper</code> function queries the database and maps the results to objects of type <code>Competency</code>. I have updated this to use the <code>top</code> argument, but as it's optional, it may not be defined. To achieve the effect of providing the highest n competencies for a given developer, the SQL query sorts the data and then (optionally) applies a restriction to the number of items returned:</p>
<p><em>api.js</em></p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getCompetenciesForDeveloper</span> <span class="token operator">=</span> <span class="token punctuation">(</span>id<span class="token punctuation">,</span> ctx<span class="token punctuation">,</span> top<span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>top <span class="token operator"><</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">throw</span> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token string">'top cannot be negative'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> limit <span class="token operator">=</span> top <span class="token operator">?</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">LIMIT </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>top<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span> <span class="token operator">:</span> <span class="token string">''</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span></span>
<span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">SELECT s.name, c.value FROM competencies c
<span class="highlight-line"> LEFT JOIN skill s ON (c.skillId = s.id)</span>
<span class="highlight-line"> WHERE c.developerId = $id</span>
<span class="highlight-line"> ORDER BY c.value DESC</span>
</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>limit<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="highlight-line"> <span class="token punctuation">{</span> $id<span class="token operator">:</span> id <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span>result <span class="token operator">=></span> result<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span>r <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>name<span class="token punctuation">,</span> r<span class="token punctuation">.</span>value<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Getting an Individual Developer or Project</h2>
<p>Arguments can be specified as <em>mandatory</em>, meaning that they must be defined. GraphQL will return with an error if you execute a query with a mandatory argument missing. I can use this technique to define queries to return a single item: either a <code>Developer</code> or a <code>Project</code>. As with other mandatory fields in GraphQL, mandatory arguments are identified with a <code>!</code>. I add these new queries to the root query object:</p>
<p><em>schema.js</em></p>
<pre class="language-typescript"><code class="language-typescript"><span class="highlight-line"><span class="token keyword">type</span> <span class="token class-name">Query</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">developer</span><span class="token punctuation">(</span>id<span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">)</span><span class="token operator">:</span> Developer<span class="token punctuation">,</span></span>
<span class="highlight-line"> developers<span class="token operator">:</span> <span class="token punctuation">[</span>Developer<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span>id<span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">)</span><span class="token operator">:</span> Project<span class="token punctuation">,</span></span>
<span class="highlight-line"> projects<span class="token operator">:</span> <span class="token punctuation">[</span>Project<span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Retrieving the single item from the database is a simple <code>SELECT WHERE</code> query. If the result is not defined then an error is returned to the client instead:</p>
<p><em>api.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getDeveloper</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> id</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">'SELECT id, name, role FROM developer WHERE id = $id'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">$id</span><span class="token operator">:</span> id <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Developer</span><span class="token punctuation">(</span>result<span class="token punctuation">.</span>id<span class="token punctuation">,</span> result<span class="token punctuation">.</span>name<span class="token punctuation">,</span> result<span class="token punctuation">.</span>role<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">No developer exists with id </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getProject</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> id</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">'SELECT id, name, description FROM project WHERE id = $id'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">$id</span><span class="token operator">:</span> id <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span>result<span class="token punctuation">.</span>id<span class="token punctuation">,</span> result<span class="token punctuation">.</span>name<span class="token punctuation">,</span> result<span class="token punctuation">.</span>description<span class="token punctuation">)</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Error</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">No project exists with id </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span></span></code></pre>
<p>A quick note about GraphQL error handling. With a REST API one would expect a 400, 404 or 500 depending on the context of the error. With GraphQL, you always get a 200 OK response, even for errors! This is because with GraphQL you're not always querying for a single item. Indeed, it's encouraged to query deeply nested data. The decision is therefore to "fail forward" - return what can be returned, and identify what aspects of the query failed.</p>
<p>To demonstrate this, for this example query:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> developer(id<span class="token operator">:</span> <span class="token string">"2"</span>) <span class="token punctuation">{</span></span>
<span class="highlight-line"> id<span class="token punctuation">,</span> name</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> project(id<span class="token operator">:</span> <span class="token string">"eggs"</span>) <span class="token punctuation">{</span></span>
<span class="highlight-line"> id<span class="token punctuation">,</span> name</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>A response object is received with the following structure:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"errors"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"message"</span><span class="token operator">:</span> <span class="token string">"No project exists with id eggs"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"locations"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"line"</span><span class="token operator">:</span> <span class="token number">5</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"column"</span><span class="token operator">:</span> <span class="token number">3</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"path"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token string">"project"</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"data"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"developer"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"id"</span><span class="token operator">:</span> <span class="token string">"2"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Gary"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"project"</span><span class="token operator">:</span> <span class="token null keyword">null</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>A GraphQL response is always wrapped in the outer object with the <code>errors</code> and <code>data</code> keys, although <code>errors</code> is omitted in the case of a fully-successful query. As the project field on the query produced an error, it's set to <code>null</code> in the response, but a helpful error is also specified.</p>
<p>Should the default error format not be appropriate for you (perhaps displaying a location or path risks a leakage of implementation detail), then there are projects out there that can <a href="https://github.com/kadirahq/graphql-errors">improve or customise GraphQL errors</a>.</p>
<h2>Get all assigned (or unassigned) Developers</h2>
<p>I can imagine a project manager in a resourcing meeting wishing to see at-a-glance which developers are currently assigned to a project, or are still waiting assignment.</p>
<p>As I have an existing query to return all developers, I can make a simple adjustment to define an <code>assigned</code> argument of type <code>Boolean</code>. By leaving it optional I am indicating that if it's not present, it will fetch all developers. Again, this makes the change non-breaking for existing callers.</p>
<p>The root query now becomes:</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">type Query <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">developer</span><span class="token punctuation">(</span>id<span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">)</span><span class="token operator">:</span> Developer<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">developers</span><span class="token punctuation">(</span>assigned<span class="token operator">:</span> Boolean<span class="token punctuation">)</span><span class="token operator">:</span> <span class="token punctuation">[</span>Developer<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span>id<span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">)</span><span class="token operator">:</span> Project<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">projects</span><span class="token operator">:</span> <span class="token punctuation">[</span>Project<span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>This new argument is passed through to the API-layer function, just like the previous examples:</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> rootValue <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function-variable function">developer</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">developers</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> assigned <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDevelopers</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> assigned<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">project</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProject</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">projects</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter">obj<span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProjects</span><span class="token punctuation">(</span>ctx<span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>The <code>getDevelopers</code> function itself is inherently now more complicated. The <code>SELECT</code> aspect is still the same, but I now optionally add a <code>WHERE</code> clause if the <code>assigned</code> argument is defined. Remember that in JavaScript <code>false</code> and <code>undefined</code> are actually different but as <code>undefined</code> is falsey, <code>!assigned</code> is true for both values. You need to be more specific with your undefined checks!</p>
<p>When the argument is set, I build up a subquery and concatenate it to the <code>SELECT</code> statement. ES6 string templating makes this approach simple. In SQL, a <code>WHERE EXISTS</code> statement lets you filter a result set based on the success of a sub-query. In this case, the sub-query asks if there is an entry in the <code>assignments</code> table for a given developer's id. <code>WHERE NOT EXISTS</code>, as you'll expect, does the inverse:</p>
<p><em>api.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getDevelopers</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> assigned</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> subquery <span class="token operator">=</span> assigned <span class="token operator">===</span> <span class="token keyword">undefined</span> <span class="token operator">?</span></span>
<span class="highlight-line"> <span class="token string-property property">''</span> <span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>assigned <span class="token operator">?</span> <span class="token string">'WHERE'</span> <span class="token operator">:</span> <span class="token string">'WHERE NOT'</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> EXISTS (SELECT a.developerId FROM assignments a WHERE a.developerId = d.id)</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> query <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">SELECT d.id, d.name, d.role FROM developer d </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>subquery<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span>query<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">r</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Developer</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>id<span class="token punctuation">,</span> r<span class="token punctuation">.</span>name<span class="token punctuation">,</span> r<span class="token punctuation">.</span>role<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<h2>Getting Skills, Sorted Alphabetically or Reverse-Alphabetically</h2>
<p>So far I've only exposed competencies in terms of a developer, where it comes alongside with an ability rating. But on a more general level, a caller could wish to just see a raw list of all the skills in the organisation. I could also imagine these wishing to be sorted either A-Z, or Z-A.</p>
<p>It's possible to use an enum as the type of a particular argument. Furthermore, with any argument, you can also supply a default value. If the caller does not specify the argument, then the default value is taken. If you're wishing to add a new mandatory argument to an existing field, you can retain backwards-compatibility this way.</p>
<p>A <code>Skill</code> is an entity with a unique id and a name. I'll also need an enum for a sort order, and a new field on the root query:</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">type Skill <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String</span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">enum</span> Order <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token constant">ASCENDING</span><span class="token punctuation">,</span> <span class="token constant">DESCENDING</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line">type Query <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">developer</span><span class="token punctuation">(</span>id<span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">)</span><span class="token operator">:</span> Developer<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">developers</span><span class="token punctuation">(</span>assigned<span class="token operator">:</span> Boolean<span class="token punctuation">)</span><span class="token operator">:</span> <span class="token punctuation">[</span>Developer<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span>id<span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">)</span><span class="token operator">:</span> Project<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">projects</span><span class="token operator">:</span> <span class="token punctuation">[</span>Project<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">skills</span><span class="token punctuation">(</span>order<span class="token operator">:</span> Order <span class="token operator">=</span> <span class="token constant">ASCENDING</span><span class="token punctuation">)</span> <span class="token operator">:</span> <span class="token punctuation">[</span>Skill<span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>This results in a new resolver function:</p>
<p><em>schema.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> rootValue <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function-variable function">developer</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDeveloper</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">developers</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> assigned <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDevelopers</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> assigned<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">project</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> id <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProject</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> id<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">projects</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter">obj<span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProjects</span><span class="token punctuation">(</span>ctx<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">skills</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> order <span class="token punctuation">}</span><span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getSkills</span><span class="token punctuation">(</span>ctx<span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>I've also defined a JavaScript clas for the new Skill type:</p>
<p><em>skill.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Skill</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> name</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> Skill<span class="token punctuation">;</span></span></code></pre>
<p>When querying the database I add an <code>ORDER BY</code> to the selection from the <code>skills</code> table. The direction of the ordering is handled depending on the argument, respecting the default direction. Again, ES6 templating makes this incredibly easy:</p>
<p><em>api.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getSkills</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">ctx<span class="token punctuation">,</span> order</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">SELECT id, name FROM skill ORDER BY name </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>order <span class="token operator">===</span> <span class="token string">'DESCENDING'</span> <span class="token operator">?</span> <span class="token string">'DESC'</span> <span class="token operator">:</span> <span class="token string">'ASC'</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">r</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Skill</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>id<span class="token punctuation">,</span> r<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Next Steps</h2>
<p>GraphQL makes it incredibly easy to add arguments to any of your queries and they're fully integrated into the schema's type system. In this post I actually thought the most difficult aspect was ensuring that an optional argument was set (false rather than undefined, as JavaScript coercion can catch you out here) and translating that into the appropriate SQL query.</p>
<p>In the examples above, I'm sorting and filtering the data that's returned to the caller. I've chosen to implement these at the database layer, in the SQL queries. Remember though that as GraphQL makes no assumptions about where it retrieves its data from, it may be necessary for you to apply the effects of your arguments somewhere else, even possibly in the resolver functions directly. That's absolutely fine and entirely your decision to make. The context of your application and profiling tools will assist you in determining where to make the choice. You may be retrieving your data from a static text file and have to do the <code>.filter</code> yourself!</p>
<p>The API is currently one way - the work in this post only allows for a caller to be a bit more specific when asking for data. It would be nice if the user could post back data, just like a HTTP POST request would in a REST API. In the next post, I'll take a look into how GraphQL's mutations can be used for this purpose.</p>
Connecting SQLite to GraphQL2017-07-23T00:00:00Zhttps://samhogy.co.uk/2017/07/connecting-sqlite-to-graphql/<p>📖 <a href="https://github.com/sh1989/graphql-consultancy/tree/f534c2981b7041f4f00b3d8c05ef07c25a474bcc">Click here for a fully-worked example of the code in this blog post</a></p>
<p><a href="/2017/07/getting-started-with-graphql/">Previously</a>, I created a GraphQL endpoint that returned data for a fictitious software consultancy firm. I discussed the reasoning behind this approach, but the implementation was slightly smoke-and-mirrors. Instead of a using real data source, I cheated and created a static, in-memory file. Let's swap that implementation out with some real database technology!</p>
<h2>Setting up the Database</h2>
<p>Searching for Node SQL clients yielded several possible options. Initially, the <a href="https://www.npmjs.com/package/sqlite3">sqlite3</a> package seemed like the best choice. Upon installation it installs the necessary binaries and provides a simple API to set up and query a SQLite database. However, I found the API too simple - in particular, it had quite a heavy focus on using callbacks for running database code, which is a tad old-fashioned and cumbersome to work with.</p>
<p>Instead I opted for the similarly-named <a href="https://www.npmjs.com/package/sqlite">sqlite</a> package. It wraps the <code>sqlite3</code> library but provides a promise-based API. Furthermore I found its support for <a href="https://github.com/kriasoft/node-sqlite/tree/master/migrations">SQL-based migration files</a> to be a more maintainable way of handling the setup of the database.</p>
<h3>Defining the Data</h3>
<p>Time to put the data into the database! When designing the structure I opted to normalise the data as much as possible. In the spirit of the single responsibility principle, things that can change for different reasons exist in different tables of the database. Therefore a developer's project assignments live in a separate table, as do the list of competencies recorded by a particular developer. These are linked by foreign keys. Sure, it makes the SQL queries to query the data more complex, but it keeps the overall database more maintainable for the future.</p>
<img src="https://samhogy.co.uk/assets/graphql/db_schema.png" width="600" height="330" />
<p>The <code>sqlite</code> package includes a database migrations feature, which lets you define a numbered series of <code>.sql</code> files containing both upgrade and downgrade scripts. This allows your database to evolve over time, accommodating for rollbacks where necessary. These live within a root-level directory named 'migrations'. To begin, I'll only need one file to create the database tables and populate them with data:</p>
<p><em>migrations/001-initial-schema.sql</em></p>
<pre><code>-- Up
CREATE TABLE project (
id INTEGER PRIMARY KEY,
name TEXT,
description TEXT
);
CREATE TABLE developer (
id INTEGER PRIMARY KEY,
name TEXT,
role INT
);
CREATE TABLE skill (
id INTEGER PRIMARY KEY,
name TEXT
);
CREATE TABLE assignments (
developerId INTEGER,
projectId INTEGER,
FOREIGN KEY(developerId) REFERENCES developer(id) ON DELETE CASCADE,
FOREIGN KEY(projectId) REFERENCES project(id) ON DELETE CASCADE,
PRIMARY KEY(developerId)
);
CREATE TABLE competencies (
developerId INTEGER,
skillId INTEGER,
value INTEGER,
FOREIGN KEY(developerId) REFERENCES developer(id) ON DELETE CASCADE,
FOREIGN KEY(skillId) REFERENCES skill(id) ON DELETE CASCADE,
PRIMARY KEY(developerId, skillId)
);
# There's a lot of INSERT INTO statements here!
-- Down
DROP TABLE project
DROP TABLE developer
DROP TABLE skill
DROP TABLE assignments
DROP TABLE competencies
</code></pre>
<h2>Hooking up to the Database</h2>
<p>Now I will need to set up the database connection and find a way to ensure that the GraphQL resolvers have a means to query the data.</p>
<p>To recap: For each field in a graphql schema, you're expected to define a function that returns some value. For convenience, if you return an object for a field, then sub-fields that exist as properties will be auto-resolved. I used this to great success by defining a class for each type in my schema, with properties on the class corresponding to GraphQL fields for that type, with class-level methods for any asynchronous data resolution.</p>
<p>The main <code>server.js</code> file is the most appropriate place to do any initial setup, for here it's possible to create the database and then create the express server with the GraphQL middleware. When creating the database you specify whether you'd like it to be in-memory, or persisted to a file. I've chosen in-memory still as this keeps things simple. As the <code>sqlite</code> package is promise-based I can chain together <code>then</code> statements. First, calling <code>sqlite.migrate()</code> will cause the migration scripts to be run in sequential order. Once completed, the express server can be established.</p>
<p>There's one change in how the GraphQL middleware is constructed. The database connection is effectively a global entity. But it's bad form to simply make the <code>sqlite</code> instance a global field, so to instead make sure that resolver functions are able to access it by reference, upon setting up the GraphQL middleware it is possible to pass in a <em>context object</em>. Mine is a simple wrapper around the database, proxying two of its accessor functions.</p>
<p><em>server.js</em>
%}
import express from 'express';
import graphQLHTTP from 'express-graphql';
import sqlite from 'sqlite';
const { schema, rootValue } = require('./data/schema');</p>
<p>const GRAPHQL_PORT = 8080;</p>
<p>sqlite.open(':memory:', { cached: true })
.then(() => sqlite.run('PRAGMA foreign_keys=on'))
.then(() => sqlite.migrate())
.then(() => {
const graphQLApp = express();
graphQLApp.use('/', graphQLHTTP({
graphiql: true,
pretty: true,
schema,
rootValue,
context: {
db: {
get: (...args) => sqlite.get(...args),
all: (...args) => sqlite.all(...args),
run: (...args) => sqlite.run(...args)
}
}
}));</p>
<p>graphQLApp.listen(GRAPHQL_PORT, () => {
console.log(
<code>GraphQL server is now running on http://localhost:${GRAPHQL_PORT}</code>
);
});
})
.catch(e => console.error(e));</p>
<pre><code>
An interesting point to make is that I need to turn on the enforcement of foreign key constraints by immediately running a `PRAGMA` statement. For backwards-compatibility the enforcement of foreign keys is not enabled by default in SQLite v3, something which will be addressed in v4.
The context object is passed into every resolver function as an argument. Here's what my root query field resolver functions now look like:
*schema.js:*
```javascript
import { buildSchema } from 'graphql';
import { getDevelopers, getProjects } from './api';
export const schema = buildSchema(`...`);
export const rootValue = {
developers: (obj, ctx) => getDevelopers(ctx),
projects: (obj, ctx) => getProjects(ctx)
};
</code></pre>
<p>Recall that all data was retrieved by calling methods defined in a file named <code>api.js</code>. For the smoke-and-mirrors implementation it would call through to a file containing some static, in-memory JavaScript objects held within <code>database.js</code>. Now there's no need for the fake database file as <code>api.js</code> can simply make calls against the real database. The purpose of <code>api.js</code> remains the same - it is effectively an incredibly naïve ORM mechanism, making database calls and turning them into objects.</p>
<p><em>api.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> Competency <span class="token keyword">from</span> <span class="token string">'./competency'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> Developer <span class="token keyword">from</span> <span class="token string">'./developer'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> Project <span class="token keyword">from</span> <span class="token string">'./project'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getDevelopers</span> <span class="token operator">=</span> <span class="token parameter">ctx</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span><span class="token string">'SELECT id, name, role FROM developer'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">r</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Developer</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>id<span class="token punctuation">,</span> r<span class="token punctuation">.</span>name<span class="token punctuation">,</span> r<span class="token punctuation">.</span>role<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getProjects</span> <span class="token operator">=</span> <span class="token parameter">ctx</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span><span class="token string">'SELECT id, name, description FROM project'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">r</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>id<span class="token punctuation">,</span> r<span class="token punctuation">.</span>name<span class="token punctuation">,</span> r<span class="token punctuation">.</span>description<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getProject</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token string">'SELECT id, name, description FROM project WHERE id = $id'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">$id</span><span class="token operator">:</span> id <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span>result<span class="token punctuation">.</span>id<span class="token punctuation">,</span> result<span class="token punctuation">.</span>name<span class="token punctuation">,</span> result<span class="token punctuation">.</span>description<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getCompetenciesForDeveloper</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> ctx</span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="highlight-line"> ctx<span class="token punctuation">.</span>db<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">SELECT s.name, c.value FROM competencies c LEFT JOIN skill s ON (c.skillId = s.id) WHERE c.developerId = $id</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span> <span class="token literal-property property">$id</span><span class="token operator">:</span> id <span class="token punctuation">}</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">r</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span>r<span class="token punctuation">.</span>name<span class="token punctuation">,</span> r<span class="token punctuation">.</span>value<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span></code></pre>
<h3>A Change to the Model</h3>
<p>To demonstrate how GraphQL supports fields being resolved asynchronously, in the <code>Developer</code> type I made getting the assigned <code>Project</code> asynchronous. It was simply a promise that resolved immediately, creating a <code>Project</code> from an id. Now that there's an actual database in place, this can be replaced with a real database lookup, searching for the project referenced in the <code>assignments</code> table. Similarly, the list of <code>Competencies</code> are no longer available up-front, as in the database they're stored within a separate <code>competencies</code> table. These will require a further lookup too.</p>
<p><em>developer.js:</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> getCompetenciesForDeveloper<span class="token punctuation">,</span> getProject <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./api'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> toRole <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./role'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Developer</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> name<span class="token punctuation">,</span> role<span class="token punctuation">,</span> projectId</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>role <span class="token operator">=</span> <span class="token function">toRole</span><span class="token punctuation">(</span>role<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>projectId <span class="token operator">=</span> projectId<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">competencies</span><span class="token punctuation">(</span><span class="token parameter">ctx</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Promise</span><span class="token punctuation">(</span><span class="token parameter">resolve</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">resolve</span><span class="token punctuation">(</span><span class="token function">getCompetenciesForDeveloper</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>id<span class="token punctuation">,</span> ctx<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span><span class="token parameter">ctx</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Promise</span><span class="token punctuation">(</span><span class="token parameter">resolve</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">resolve</span><span class="token punctuation">(</span><span class="token function">getProject</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>projectId<span class="token punctuation">,</span> ctx<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> Developer<span class="token punctuation">;</span></span></code></pre>
<p>When building this I noticed that there was some further work required to get enums to behave correctly. My schema defines the <code>Role</code> enum in terms of some string values: GRAD, DEVELOPER, SENIOR, LEAD. I stored roles in the <code>developer</code> table using the convention of having a unique integer represent each possible value. So GRAD = 0, DEVELOPER = 1 and so on. It was necessary to define a lookup function to convert an integer to a string so that GraphQL would correctly identify the returned value as valid for the <code>Role</code> enum type:</p>
<p><em>role.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> Role <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token constant">GRAD</span><span class="token operator">:</span> <span class="token string">'GRAD'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token constant">DEV</span><span class="token operator">:</span> <span class="token string">'DEV'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token constant">SENIOR</span><span class="token operator">:</span> <span class="token string">'SENIOR'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token constant">LEAD</span><span class="token operator">:</span> <span class="token string">'LEAD'</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> roleMappings <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token number">0</span><span class="token operator">:</span> Role<span class="token punctuation">.</span><span class="token constant">GRAD</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token number">1</span><span class="token operator">:</span> Role<span class="token punctuation">.</span><span class="token constant">DEV</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token number">2</span><span class="token operator">:</span> Role<span class="token punctuation">.</span><span class="token constant">SENIOR</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token number">3</span><span class="token operator">:</span> Role<span class="token punctuation">.</span><span class="token constant">LEAD</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">toRole</span> <span class="token operator">=</span> <span class="token parameter">val</span> <span class="token operator">=></span> roleMappings<span class="token punctuation">[</span>val<span class="token punctuation">]</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> Role<span class="token punctuation">;</span></span></code></pre>
<h2>Next Steps</h2>
<p>I was pretty pleased to discover that it was quite easy to drop-in a database to my application without too many hiccups along the way. The two main gotchas were discovering the ability to pass through a context object (and then working out which argument this corresponded to in the resolver functions), and also the necessity of having to manually convert an integer-based enum to a string-based one.</p>
<p>In the next post I'll start looking at adding some interactivity to the GraphQL API. Particularly, I'll be covering how to pass arguments to GraphQL fields, and how to allow a client to change the data in the API, which is known as a mutation.</p>
Getting Started with GraphQL2017-07-09T00:00:00Zhttps://samhogy.co.uk/2017/07/getting-started-with-graphql/<p>📖 <a href="https://github.com/sh1989/graphql-consultancy/tree/3f66fc73bd6537023a411793cb7f58f9267e0634">Click here for a fully-worked example of the code in this blog post</a></p>
<p>GraphQL is a "query language for APIs". Rather than a web service exposing static, predefined REST endpoints, it instead exposes a <strong>single</strong> endpoint that is capable of:</p>
<ol>
<li>Receiving, interpreting and responding to incoming requests for data</li>
<li>Handling requests to manipulate (mutate) data</li>
<li>Expressing the capabilities of the server, through a queryable type system.</li>
</ol>
<p>GraphQL is itself a language-independent specification, with implementations existing in many programming languages. In this blog series, I will build a GraphQL server with NodeJS, using the "reference implementation" of a GraphQL server.</p>
<h2>Motivation</h2>
<p>While developing an application, I was assigned a task that required me to retrieve some information from the server. Unfortunately, the API endpoint we were already using did not provide this additional piece of required data. I had to call a different API endpoint, meaning another round trip to the server. On the client-side, I then had to write and maintain code to orchestrate multiple requests and stitching together the responses into a meaningful responses. When you consider handling the various failure modes, this becomes a reasonably complicated chunk of code, which isn't particularly efficient from a networking perspective.</p>
<p>The reason that we had two different APIs for the same domain object was largely a product of history, with each individual endpoint reaching out to a different system upstream and simply proxying the result back. Not particularly atypical within large enterprises.</p>
<p>It also meant that one of the API calls was <em>oversharing</em> - returning more information than the client actually needed, whilst the other was <em>undersharing</em> - because we had to go elsewhere for some extra properties.</p>
<p>One potential solution is to create a new REST endpoint that does all this for us, only returning the information that my application needed. Whilst this is a step in the right direction, removing the under/oversharing problem, reducing the round trips to one, and removing the need for the UI code, this isn't maintainable. The next time another app comes along, it may also have different data requirements.</p>
<p>GraphQL aims to replace multiple REST calls with a single call to an endpoint. You ask for only the data that you need, and the server responds with that data <strong>only</strong>. Responses are in JSON format, matching closely with the format of the query request. The GraphQL server is responsible for coordinating with upstream systems, databases etc, and responding appropriately. This creates a clear separation of concerns between client and server, whilst creating a single point of responsibility for callers of your server. This can be of great benefit in environments with high latency, or limited/sporadic network availability..</p>
<p><img src="https://samhogy.co.uk/assets/graphql/rest.png" width="400" height="412" /><img src="https://samhogy.co.uk/assets/graphql/graphql_resource.png" width="400" height="412" /></p>
<p>A query in GraphQL looks something like...</p>
<pre><code>{
developers {
name,
competencies {
name,
rating
}
}
}
</code></pre>
<p>...with the result taking the form...</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"data"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"developers"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Sam"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"competencies"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"JavaScript"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"rating"</span><span class="token operator">:</span> <span class="token number">8</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"C#"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"rating"</span><span class="token operator">:</span> <span class="token number">7</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Gary"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"competencies"</span><span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"C++"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"rating"</span><span class="token operator">:</span> <span class="token string">"8"</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Compilers"</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"rating"</span><span class="token operator">:</span> <span class="token number">8</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The 'types' in a GraphQL endpoint represent abstract domain entities. It shouldn't matter if your server has to build them from a database, an upstream service, or by combining several of these methods. That's an implementation detail. What should be exposed to <strong>your</strong> users is a consistent set of business objects. As such, GraphQL is type-aware. During creation, you specify the types of your objects, the fields on those objects, their types, and so on. In the example above, <code>developer</code> is a type, <code>name</code> is a field of the <code>developer</code> type. <code>competencies</code> is a field on developer, which has a type of <code>list of competency</code>. <code>rating</code> is a field on the <code>competency</code> type.</p>
<p>As the type system itself is queryable, your API has an aspect of discoverability to it. Not only that, the queries can be validated against the type system for correctness.</p>
<p>Conceptually, the 'graph' aspect of GraphQL is <strong>not</strong> a database. It is instead the mental model of how the entities in your domain are associated with one another. As an abstract concept, it doesn't map to how the data are stored. GraphQL is a generic mechanism to retrieve data in response to a query. Therefore, it conceptually sits at the same layer as your REST API in the server stack. GraphQL endpoints can happily sit alongside an existing REST API (it's not a competition!), with both interacting with the same underlying persistence/caching layers of your web service.</p>
<h2>The Case Study</h2>
<p>For the purpose of this blog series, I'm going to be defining a GraphQL API for a fictitious software consultancy. The API's purpose is for the administration of employees, the resourcing of projects, and the maintenance of a skill set of employees. Some of this will be security and permissions-protected too, so that certain functionalities can only be performed by users with specific permissions.</p>
<p>Let's first talk about the <strong>relationships</strong> between the types in the domain. There is a <strong>Developer</strong>, who will be assigned a <strong>Role</strong> at the company. They'll have a list of <strong>Competencies</strong> and may be assigned a <strong>Project</strong>. Hey, as we're talking about GraphQL, let's describe this as a graph!</p>
<img src="https://samhogy.co.uk/assets/graphql/consultancy_schema.png" width="800" height="306" />
<p>Initially I'll start simple, and from the ground-up. I'll define an in-memory database, and then build it out into using a SQL database. I may make mistakes along the way, but hopefully we'll learn a lot about how to build a GraphQL API!</p>
<h2>Setting Things Up</h2>
<p>I'll be using NodeJS for this series, so let's start by defining an express server, alongside the graphql layer:</p>
<p><code>npm install --save express express-graphql graphql</code></p>
<h2>Defining A Schema</h2>
<p>To define the GraphQL schema, you need to do two things:</p>
<ol>
<li>Specify your types, in GraphQL schema language</li>
<li>Define resolver functions to retrieve your data</li>
</ol>
<p>The schema is just a string, passed to the <code>buildSchema</code> function from the <code>graphql</code> package:</p>
<p><em>schema.js</em>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> buildSchema <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'graphql'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> getDevelopers<span class="token punctuation">,</span> getProjects<span class="token punctuation">,</span> roles <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./api'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> schema <span class="token operator">=</span> <span class="token function">buildSchema</span><span class="token punctuation">(</span>'</span>
<span class="highlight-line"> type Project <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">description</span><span class="token operator">:</span> String</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> type Competency <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">rating</span><span class="token operator">:</span> Int</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> type Developer <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> String<span class="token operator">!</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> String<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">competencies</span><span class="token operator">:</span> <span class="token punctuation">[</span>Competency<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">role</span><span class="token operator">:</span> Role<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">project</span><span class="token operator">:</span> Project</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">enum</span> Role <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token constant">GRAD</span><span class="token punctuation">,</span> <span class="token constant">DEV</span><span class="token punctuation">,</span> <span class="token constant">SENIOR</span><span class="token punctuation">,</span> <span class="token constant">LEAD</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> type Query <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">developers</span><span class="token operator">:</span> <span class="token punctuation">[</span>Developer<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">projects</span><span class="token operator">:</span> <span class="token punctuation">[</span>Project<span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line">'<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> rootValue <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function-variable function">developers</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getDevelopers</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function-variable function">projects</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token function">getProjects</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>The type system itself is quite easy to grasp. Exclamation marks represent that a field is non-nullable. Lists are represented by square brackets. Enumerated types are just a collection of strings.</p>
<p>The <code>Query</code> type is a special reserved type, which identifies the schema's root query. It represents fields that can be queried at the top level; the root; or a GraphQL query. In this case, I'm exposing a list of Developers and a list of Projects. The rootValue object provides data for both of these fields.</p>
<p>Both the built schema and the root value are exported from this module. These will later be passed to the GraphQL middleware. But for now, let's focus on filling in the data.</p>
<h2>Setting Up The Data</h2>
<p>The data are stored as in-memory objects in order to keep things simple.</p>
<p><em>database.js:</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> Competency <span class="token keyword">from</span> <span class="token string">'./competency'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> Developer <span class="token keyword">from</span> <span class="token string">'./developer'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> Project <span class="token keyword">from</span> <span class="token string">'./project'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> roles <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token string-property property">'GRAD'</span><span class="token operator">:</span> <span class="token string">'GRAD'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string-property property">'DEV'</span><span class="token operator">:</span> <span class="token string">'DEV'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string-property property">'SENIOR'</span><span class="token operator">:</span> <span class="token string">'SENIOR'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token string-property property">'LEAD'</span><span class="token operator">:</span> <span class="token string">'LEAD'</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> projects <span class="token operator">=</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span><span class="token string">'1'</span><span class="token punctuation">,</span> <span class="token string">'Facelift'</span><span class="token punctuation">,</span> <span class="token string">'Redesign of the UI'</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span><span class="token string">'2'</span><span class="token punctuation">,</span> <span class="token string">'Trader 2G'</span><span class="token punctuation">,</span> <span class="token string">'The second generation of the trading platform'</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Project</span><span class="token punctuation">(</span><span class="token string">'3'</span><span class="token punctuation">,</span> <span class="token string">'Market Data'</span><span class="token punctuation">,</span> <span class="token string">'Implementation of market data API'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">]</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> developers <span class="token operator">=</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Developer</span><span class="token punctuation">(</span><span class="token string">'1'</span><span class="token punctuation">,</span> <span class="token string">'Sam'</span><span class="token punctuation">,</span> <span class="token punctuation">[</span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span><span class="token string">'JavaScript'</span><span class="token punctuation">,</span> <span class="token string">'8'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span><span class="token string">'C#'</span><span class="token punctuation">,</span> <span class="token string">'7'</span><span class="token punctuation">)</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> roles<span class="token punctuation">.</span><span class="token constant">SENIOR</span><span class="token punctuation">,</span> <span class="token string">'1'</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Developer</span><span class="token punctuation">(</span><span class="token string">'2'</span><span class="token punctuation">,</span> <span class="token string">'Gary'</span><span class="token punctuation">,</span> <span class="token punctuation">[</span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span><span class="token string">'Java'</span><span class="token punctuation">,</span> <span class="token string">'6'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span><span class="token string">'C++'</span><span class="token punctuation">,</span> <span class="token string">'8'</span><span class="token punctuation">)</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> roles<span class="token punctuation">.</span><span class="token constant">LEAD</span><span class="token punctuation">,</span> <span class="token string">'2'</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token keyword">new</span> <span class="token class-name">Developer</span><span class="token punctuation">(</span><span class="token string">'3'</span><span class="token punctuation">,</span> <span class="token string">'Chris'</span><span class="token punctuation">,</span> <span class="token punctuation">[</span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span><span class="token string">'Java'</span><span class="token punctuation">,</span> <span class="token string">'7'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token keyword">new</span> <span class="token class-name">Competency</span><span class="token punctuation">(</span><span class="token string">'JavaScript'</span><span class="token punctuation">,</span> <span class="token string">'6'</span><span class="token punctuation">)</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> roles<span class="token punctuation">.</span><span class="token constant">DEV</span><span class="token punctuation">,</span> <span class="token string">'3'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">]</span><span class="token punctuation">;</span></span></code></pre>
<p>For the domain types, such as Developer, Project and Competency, I can define some ES6 classes. When a class is returned for a type in a GraphQL schema, any fields that match a class variable with the same name are auto-resolved. Additionally, if there's a method name with the same name as a field from the schema, it will be called whenever that field is requested.</p>
<p>For incredibly simple objects like the <code>Project</code> and <code>Competency</code> types, this means that all the work is done in the constructors.</p>
<p><em>competency.js:</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Competency</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">name<span class="token punctuation">,</span> rating</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>rating <span class="token operator">=</span> rating<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> Competency<span class="token punctuation">;</span></span></code></pre>
<p><em>project.js:</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Project</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> name<span class="token punctuation">,</span> description</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>description <span class="token operator">=</span> description<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> Project<span class="token punctuation">;</span></span></code></pre>
<p>The <code>Developer</code> class is more interesting, as it contains a dynamic and asynchronous resolver. To simulate some form of latency, I've implemented a custom method for returning a developer's assigned project, based on an id that is passed in. The GraphQL JavaScript implementation supports promises natively. When the return type of a field is a promise, the library will wait for all promises to resolve before sending a response to the client. That's as simple as wrapping a (synchronous) API call in a promise.</p>
<p><em>developer.js:</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> getProject <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./api'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">Developer</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">id<span class="token punctuation">,</span> name<span class="token punctuation">,</span> competencies<span class="token punctuation">,</span> role<span class="token punctuation">,</span> projectId</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>id <span class="token operator">=</span> id<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>name <span class="token operator">=</span> name<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>competencies <span class="token operator">=</span> competencies<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>role <span class="token operator">=</span> role<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>projectId <span class="token operator">=</span> projectId<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">project</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token keyword">new</span> <span class="token class-name">Promise</span><span class="token punctuation">(</span><span class="token parameter">resolve</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">resolve</span><span class="token punctuation">(</span><span class="token function">getProject</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>projectId<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> Developer<span class="token punctuation">;</span></span></code></pre>
<h2>Exposing the data to GraphQL</h2>
<p>There are some classes for the data, and a file that collects all of that data together. Next, I'll define an API file, whose job it is to export the raw data sets, but also defines some common query functions on this data. The <code>Developer</code> class uses one of these - to look up a project from a given ID:</p>
<p><em>api.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> developers<span class="token punctuation">,</span> projects <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./database'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getDevelopers</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> developers<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getProjects</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> projects<span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> <span class="token function-variable function">getProject</span> <span class="token operator">=</span> <span class="token parameter">id</span> <span class="token operator">=></span> projects<span class="token punctuation">.</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token parameter">p</span> <span class="token operator">=></span> p<span class="token punctuation">.</span>id <span class="token operator">===</span> id<span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Eventually, the in-memory database will be removed with a real database, but ideally this API file should remain as-is. Its purpose is simply to provide a degree of separation between the API surface from its underlying implementation.</p>
<p>Earlier, the <em>schema.js</em> file provided its root value object by calling some of the methods that are defined in the <em>api.js</em> file.</p>
<h2>Running the Server</h2>
<p>The main file will be responsible for setting up the express server on the spedcified port. It also has the <a href="https://github.com/graphql/graphiql">graphiql</a> interactive environment built-in, which allows us easily query the endpoint in a visual editor.</p>
<p><em>index.js</em></p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> express <span class="token keyword">from</span> <span class="token string">'express'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> graphQLHTTP <span class="token keyword">from</span> <span class="token string">'express-graphql'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> path <span class="token keyword">from</span> <span class="token string">'path'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> schema<span class="token punctuation">,</span> rootValue <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'./data/schema'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token constant">GRAPHQL_PORT</span> <span class="token operator">=</span> <span class="token number">8080</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> app <span class="token operator">=</span> <span class="token function">express</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line">app<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token string">'/'</span><span class="token punctuation">,</span> <span class="token function">graphQLHTTP</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">graphiql</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">pretty</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> schema<span class="token punctuation">,</span></span>
<span class="highlight-line"> rootValue</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> server <span class="token operator">=</span> app<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span><span class="token constant">GRAPHQL_PORT</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">GraphQL server running on http://localhost:</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span><span class="token constant">GRAPHQL_PORT</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>In the <code>package.json</code> for the project, let's define a script to start up the server, using <code>npm start</code>:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> ...</span>
<span class="highlight-line"> <span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"start"</span><span class="token operator">:</span> <span class="token string">"babel-node ."</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> ...</span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Next Steps</h2>
<p>It's very simple to create a GraphQL endpoint. In the next post, I'll swap out the in-memory database with a real database, to investigate how GraphQL performs and integrates with a more complete back-end stack. Later on, I'll add some more data as the fictitious consultancy grows as a business. I'll also allow for mutations to alter the server's data.</p>
Creating A General Election Slackbot2017-06-12T00:00:00Zhttps://samhogy.co.uk/2017/06/creating-a-general-election-slackbot/<p>With the surprise announcement of a General Election in the UK, I created a SlackBot to track the polling performance of the major UK parties. In this post, I'll show you how I created it.</p>
<p>I used NodeJS, with the help of some awesome npm libraries and got the application up and running in a few hours. The application is deployed to Heroku, which I'd never used before, but I found it incredibly easy to manage deployments.</p>
<img src="https://samhogy.co.uk/assets/electionbot.jpg" />
<h2>Handling Environment Variables</h2>
<p>When you register a new SlackBot through your Slack instance settings, you'll be presented with an API key. This is used to identify your bot and to allow for it to post to your Slack.</p>
<p>Locally I specified an <code>.env.local</code> file to hold environment variables that are required by my app. This is much more preferable to adding them to my real <code>PATH</code>. Furthermore, the file is included in the repository <code>.gitignore</code>, so that the secrets are not accidentally committed to the public repository. I'll talk later about how these are configured in Heroku. My <code>.env.local</code> file looks like:</p>
<pre><code>BOT_API_KEY=<my api key>
BOT_NAME=electionbot
</code></pre>
<p>The <a href="https://www.npmjs.com/package/env-cmd">env-cmd</a> npm package reads this file to set up the environment. When developing locally, the app is started via a <code>package.json</code> script:</p>
<pre><code>"start": "env-cmd ./.env.local node index.js",
</code></pre>
<p>Then, you can read any of these variables from within your node app:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">var</span> token <span class="token operator">=</span> process<span class="token punctuation">.</span>env<span class="token punctuation">.</span><span class="token constant">BOT_API_KEY</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">var</span> name <span class="token operator">=</span> process<span class="token punctuation">.</span>env<span class="token punctuation">.</span><span class="token constant">BOT_NAME</span><span class="token punctuation">;</span></span></code></pre>
<h2>Creating a SlackBot</h2>
<p>Creating a SlackBot involves setting up a WebSockets connection and interacting with Slack's "Real-Time Messaging API". This is quite low-level for a simple bot like mine, so I found a helpful npm package, <a href="https://www.npmjs.com/package/slackbots">slackbots</a> to manage this for me.</p>
<p>All you need to do is create a new instance, passing in your token and bot name:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">var</span> Bot <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'slackbots'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">var</span> mySlackBot <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Bot</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> token<span class="token punctuation">,</span></span>
<span class="highlight-line"> name</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>In the simplest case the bot object then allows for you to attach callbacks to events, alongside an easy-to-use API to post messages:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">mySlackBot<span class="token punctuation">.</span><span class="token function">on</span><span class="token punctuation">(</span><span class="token string">'start'</span><span class="token punctuation">,</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> mySlackBot<span class="token punctuation">.</span><span class="token function">postMessageToChanngel</span><span class="token punctuation">(</span><span class="token string">'general'</span><span class="token punctuation">,</span> <span class="token string">'hi!'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>I created a subclass, which would register callbacks and handle the lifecycle events I'm interested in:</p>
<ul>
<li><code>start</code> - Called when the bot goes live.</li>
<li><code>message</code> - Called when ANY message is posted to your Slack.</li>
</ul>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">class</span> <span class="token class-name">ElectionBot</span> <span class="token keyword">extends</span> <span class="token class-name">Bot</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token function">constructor</span><span class="token punctuation">(</span><span class="token parameter">settings</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">super</span><span class="token punctuation">(</span>settings<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>settings <span class="token operator">=</span> settings<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">on</span><span class="token punctuation">(</span><span class="token string">'start'</span><span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">.</span>onStart<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">on</span><span class="token punctuation">(</span><span class="token string">'message'</span><span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">.</span>onMessage<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">onStart</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// TODO: Startup Script</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">onMessage</span><span class="token punctuation">(</span><span class="token parameter">message</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// TODO: Handle Message</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>On startup</h2>
<p>On startup, id for the @electionBot user is calculated, which allows the bot to post into Slack as @electionbot:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">this</span><span class="token punctuation">.</span>user <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">.</span>users<span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">user</span> <span class="token operator">=></span> user<span class="token punctuation">.</span>name <span class="token operator">===</span> <span class="token keyword">this</span><span class="token punctuation">.</span>settings<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">;</span></span></code></pre>
<h2>Handling a Message</h2>
<p>As mentioned earlier, every message sent to your Slack instance will result in your <code>message</code> callback being called. Therefore, you'll need to apply some logic to work out if the message is actually something that is actionable by your SlackBot. I wrote a series of convenience functions to handle the following conditions:</p>
<ul>
<li><strong>The message is from a different user</strong>: <code>message.user !== this.user.id</code>;</li>
<li><strong>The message is a message (rather than any other type)</strong>: <code>message.type === 'message' && Boolean(message.text)</code></li>
<li><strong>The message mentions @electionbot</strong>: <code>message.text.indexOf(</code><@${this.user.id}><code>) > -1</code></li>
</ul>
<p>(Note the strange syntax there - Slack uses these identifiers to mark user ids, groups, direct messages etc.)</p>
<p>If all of these conditions are true, then a user has asked for @electionbot to perform some action. It will need to respond. This could take some time. To improve the UX while awaiting a response, I respond with a placeholder signal which indicates that @electionbot is in the process of replying:</p>
<img src="https://samhogy.co.uk/assets/electionbot_typing_response.jpg" />
<p>Unfortunately this is not a feature directly supported by the <code>slackbots</code> package API, but it does expose the underlying <code>ws</code> object. We can therefore achieve this by sending the following formatted message down the WebSocket.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">this</span><span class="token punctuation">.</span>ws<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'typing'</span><span class="token punctuation">,</span> <span class="token literal-property property">channel</span><span class="token operator">:</span> message<span class="token punctuation">.</span>channel<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Very simply, the SlackBot will perform its function if the message contains the text 'westminster'. If detected, then we'll post a response.</p>
<p>To put all this together:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">onMessage</span><span class="token punctuation">(</span><span class="token parameter">message</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">isFromSomeoneElse</span><span class="token punctuation">(</span>message<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token operator">&&</span> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">isMessage</span><span class="token punctuation">(</span>message<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token operator">&&</span> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">mentionsMe</span><span class="token punctuation">(</span>message<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">sendTypingResponse</span><span class="token punctuation">(</span>message<span class="token punctuation">.</span>channel<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>message<span class="token punctuation">.</span>text<span class="token punctuation">.</span><span class="token function">toLowerCase</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">indexOf</span><span class="token punctuation">(</span><span class="token string">'westminster'</span><span class="token punctuation">)</span> <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// TODO: Calculate polling data</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// TODO: Command not found</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">isFromSomeoneElse</span><span class="token punctuation">(</span><span class="token parameter">message</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> message<span class="token punctuation">.</span>user <span class="token operator">!==</span> <span class="token keyword">this</span><span class="token punctuation">.</span>user<span class="token punctuation">.</span>id<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">isMessage</span><span class="token punctuation">(</span><span class="token parameter">message</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> message<span class="token punctuation">.</span>type <span class="token operator">===</span> <span class="token string">'message'</span></span>
<span class="highlight-line"> <span class="token operator">&&</span> <span class="token function">Boolean</span><span class="token punctuation">(</span>message<span class="token punctuation">.</span>text<span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">mentionsMe</span><span class="token punctuation">(</span><span class="token parameter">message</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> message<span class="token punctuation">.</span>text<span class="token punctuation">.</span><span class="token function">indexOf</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string"><@</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span><span class="token keyword">this</span><span class="token punctuation">.</span>user<span class="token punctuation">.</span>id<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span> <span class="token operator">></span> <span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token function">sendTypingResponse</span><span class="token punctuation">(</span><span class="token parameter">channel</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span>ws<span class="token punctuation">.</span><span class="token function">send</span><span class="token punctuation">(</span><span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'typing'</span><span class="token punctuation">,</span> <span class="token literal-property property">channel</span><span class="token operator">:</span> channel <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Computing Average Polling Data</h2>
<p><a href="http://britainelects.com/polling/westminster/">Britain Elects</a> maintain an up-to-date list of polls, conducted by a variety of pollsters. Sadly, it's presented purely as tabulated data, rather than some queryable API. I used the <a href="https://www.npmjs.com/package/table-scraper">table-scraper</a> npm package, which visits a specified URL and returns an array of tables parsed from the returned HTML page.</p>
<p>A table is an array, with each item in the array being a row. Each row is an object, and you can access a given cell by referencing its header column name. So for a table with a header column called 'Lab', you can access the second row's 'Lab' cell by:</p>
<pre><code>table[1]['Lab']
</code></pre>
<p>My calculation takes the past eight polls, and computes a running average with a sample size of seven. The most recent polling data is returned as the current polling levels, but the <em>difference</em> between the two samples allows us to identify if there has been a significant movement in the polls.</p>
<p>A more realistic average would probably assign weights to each poll, but I figured the mean was good enough for our Slack experiment. I used <a href="https://www.npmjs.com/package/bignumber.js">bignumber.js</a> to ensure that the calculations do not suffer from floating-point precision errors, and <a href="https://www.npmjs.com/package/moment">moment</a> for date representation.</p>
<p>This returns an object of the following format:</p>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"data"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"con"</span><span class="token operator">:</span> <span class="token number">43</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"lab"</span><span class="token operator">:</span> <span class="token number">38</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"ldem"</span><span class="token operator">:</span> <span class="token number">8</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"ukip"</span><span class="token operator">:</span> <span class="token number">4</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"grn"</span><span class="token operator">:</span> <span class="token number">2</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"diff"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"con"</span><span class="token operator">:</span> <span class="token number">-1</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"lab"</span><span class="token operator">:</span> <span class="token number">2</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"ldem"</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"ukip"</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"grn"</span><span class="token operator">:</span> <span class="token number">0</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"meta"</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token property">"sample"</span><span class="token operator">:</span> <span class="token number">7</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"from"</span><span class="token operator">:</span> moment(...)<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token property">"to"</span><span class="token operator">:</span> moment(...)</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Posting the Response</h2>
<p>The result object above is formatted into a string:</p>
<pre><code>`Average of last ${response.meta.sample} polls (${response.meta.from} - ${response.meta.to}):\n` +
`${formatRow(response, 'con', ':conservative:')}\n` +
...
</code></pre>
<p>Where <code>formatRow</code> is a convenience function. It takes the <code>result</code>, a property <code>accessor</code>, and an <code>emoji</code> to display. It produces a string that may contain an up or down arrow, should the polls have moved. This function is called once for each party:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function">formatRow</span><span class="token punctuation">(</span><span class="token parameter">result<span class="token punctuation">,</span> accessor<span class="token punctuation">,</span> emoji</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> percentage <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>emoji<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>result<span class="token punctuation">.</span>data<span class="token punctuation">[</span>accessor<span class="token punctuation">]</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">%</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> change <span class="token operator">=</span> result<span class="token punctuation">.</span>diff<span class="token punctuation">[</span>accessor<span class="token punctuation">]</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>change <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>percentage<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">\t:small_red_triangle: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>change<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>change <span class="token operator"><</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>percentage<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">\t:small_red_triangle_down: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>change <span class="token operator">*</span> <span class="token operator">-</span><span class="token number">1</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> percentage<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>For posting messages back to the sender, we define a <code>post</code> function. It calls the bot api's own <code>postMessage</code> function, alongside the <code>as_user</code> param. This makes sure that the message within the group is attributed to @electionbot.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">post</span><span class="token punctuation">(</span><span class="token parameter">channel<span class="token punctuation">,</span> message</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">postMessage</span><span class="token punctuation">(</span>channel<span class="token punctuation">,</span> message<span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">as_user</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>To add a bit of flavour to the app, I'm also using Slack's custom emoji feature. Any emoji that are enabled by default, or added in your Slack settings, can be used in a response via the <code>:emoji_name:</code> syntax. My Slack app has custom emoji defined for each party.</p>
<p>Additionally, if a command cannot be interpreted, the SlackBot posts back a response:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token function">postDontKnow</span><span class="token punctuation">(</span><span class="token parameter">channel</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">post</span><span class="token punctuation">(</span>channel<span class="token punctuation">,</span> <span class="token string">':question: Sorry, I don\t understand.'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h2>Deploying to Heroku</h2>
<p>I don't keep my computing running all the time, but I would like for @electionbot to be always-available. Heroku is a cloud application platform that supports a wide range of programming languages, including Java, JavaScript (NodeJS), Python and Go.</p>
<p>Once you've created an account with Heroku, you'll need to download the Heroku CLI, install it, and link it to your account.</p>
<p>Heroku apps, upon deployment, look for a <code>Procfile</code> to identify startup tasks. This is instead of a <code>package.json</code> file. We specify that we'd like to start a node application.</p>
<pre><code>worker: node index.js
</code></pre>
<p>In your Heroku app's settings, it's possible to configure environment variables for your app. I copied them over as they're not checked in to the git repo (remember, <code>.env.local</code> is <code>.gitignore</code>d!).</p>
<p>There are two ways that you can deploy to Heroku. A Heroku app comes with a git remote, which you can directly push to:</p>
<pre><code>git remote add heroku <url.git>
git push heroku master
</code></pre>
<p>I used this method first in order to test that everything worked out correctly. However, pushing to two remotes (GitHub and Heroku) isn't particularly ideal. It is also possible to connect your Heroku and GitHub accounts together. This way, every time you push to GitHub, a Heroku deployment is triggered. There's also the option to trigger a manual deployment from a specific branch should you ever wish to perform a deployment of some experimental code. This is all accessible from the <strong>Deploy</strong> tab of your Heroku app.</p>
<img src="https://samhogy.co.uk/assets/electionbot_heroku.jpg" width="600" height="476" />
<h2>Conclusion</h2>
<p>You can <a href="https://github.com/sh1989/electionbot">take a look at the code</a> if you'd like to take a deeper dive. The real version of the app handles several commands, so the scraping logic has been extracted into a generic function. See <a href="https://github.com/sh1989/electionbot/blob/master/services/scraper.js">scraper.js</a> for the functions to compute the average, and <a href="https://github.com/sh1989/electionbot/blob/master/services/westminster.js">westminster.js</a> for an example of how to interact with this class. You'll notice that the real @electionbot can also return results for the Welsh Assembly, Scottish Parliament, and the Scottish voting intention for the Westminster Parliament!</p>
<p>In short, I found creating the SlackBot quite an enjoyable experience and it's served its function quite well over the past few weeks.</p>
Reusing Redux Reducers2017-04-12T00:00:00Zhttps://samhogy.co.uk/2017/04/reusing-redux-reducers/<p>Oftentimes in a Redux application we have a little bit of state that we'd like to be made re-usable. By state, I don't just mean the structure of some object, I'm also referring to the <em>management</em> of that state. This tends to happen as your application grows and then a requirement comes in to do something similar to what you've done previously. Very often you'll already have a nice bit of code that you'd like to re-use.</p>
<p>This recently happened to me, so I'd like to document how I go about making reducers re-usable in large-scale applications. Firstly, a brief recap.</p>
<h2>What's a Reducer?</h2>
<p>Recall that Redux applications contain a state object. The way that this state is modified is via a simple function: <code>nextState = update(prevState, action)</code></p>
<p>An action is typically an object with an identifying <code>type</code> property and a <code>payload</code>. The type name takes the form of a unique String. A sample action looks like this:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> sampleAction <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'SAMPLE_ACTION'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">payload</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">'Hello There!'</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>This <code>update</code> function is <em>reducing</em> the old state and the action into a new state object, which is why it's known as a <em>reducer</em> function. What you're actually wanting to re-use, therefore, are three things:</p>
<ol>
<li>The state model - the structure of the object corresponding to some application state</li>
<li>Some actions - which define how the state can be mutated</li>
<li>A reducer - which will perform the mutation on your state</li>
</ol>
<h2>A create Reducer function</h2>
<p>What I normally do is have a function which, when called, <em>returns</em> a reducer function. This can then be included in your reducer tree via a <code>combineReducers</code> call. In terms of actions - we need a unique name. To achieve this, the function accepts an <code>actionRoot</code> argument - it's just a string! Internally, the function creates some action strings by prepending this value to each action name. This makes them unique, but within the function, they can be reference by their variable names (in this case: <code>INCREMENT</code> and <code>DECREMENT</code>). The action names are also returned too, in case they're needed elsewhere.</p>
<p>This is how this looks:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">function</span> <span class="token function-variable function">createReducer</span> <span class="token operator">=</span> <span class="token parameter">actionRoot</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token constant">INCREMENT</span> <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>actionRoot<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">/increment</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token constant">DECREMENT</span> <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>actionRoot<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">/decrement</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">const</span> defaultState <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">val</span><span class="token operator">:</span> <span class="token number">0</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token function-variable function">reducer</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">state <span class="token operator">=</span> defaultState<span class="token punctuation">,</span> action</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">switch</span> <span class="token punctuation">(</span>action<span class="token punctuation">.</span>type<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token constant">INCREMENT</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">val</span><span class="token operator">:</span> state<span class="token punctuation">.</span>val <span class="token operator">+</span> <span class="token number">1</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token constant">DECREMENT</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">val</span><span class="token operator">:</span> state<span class="token punctuation">.</span>val <span class="token operator">-</span> <span class="token number">1</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">default</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> state<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token constant">INCREMENT</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token constant">DECREMENT</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> reducer</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> createReducer<span class="token punctuation">;</span></span></code></pre>
<p>Now, wherever we wish to use this state model, we can simply call the function!</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> createReducer <span class="token keyword">from</span> <span class="token string">'./createReducer'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> createAction <span class="token keyword">from</span> <span class="token string">'redux-actions'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token punctuation">{</span> <span class="token constant">INCREMENT</span><span class="token punctuation">,</span> <span class="token constant">DECREMENT</span><span class="token punctuation">,</span> reducer <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">createReducer</span><span class="token punctuation">(</span><span class="token string">'my-sample-counter'</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> incrementCounter <span class="token operator">=</span> <span class="token function">createAction</span><span class="token punctuation">(</span><span class="token constant">INCREMENT</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> decrementCounter <span class="token operator">=</span> <span class="token function">createAction</span><span class="token punctuation">(</span><span class="token constant">DECREMENT</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> reducer<span class="token punctuation">;</span></span>
<span class="highlight-line"></span></code></pre>
<p>In this example, I'm using the <code>redux-actions</code> library to create <em>action creator</em> functions based on the actions returned by my <code>createReducer</code> function, so that when called, they return a standard action object of that type. These can now be used in a redux <code>Connect</code>ed component via its <code>mapDispatchToProps</code> function - let's say so that when a user clicks a button, either my <code>INCREMENT</code> or <code>DECREMENT</code> action is dispatched. I'm also returning the reducer so that it can be included in the state tree.</p>
<p>You'll have probably noticed that this concept is very similar in nature to what I discussed in <a href="/2017/03/embracing-higher-order-reducers/">my article on Higher-Order Reducers</a>, and in yesterday's <a href="/2017/04/reusing-redux-forms-for-creating-and-editing/">article on reusing Redux Form components</a>. This shows the fantastic utility of composition!</p>
<h2>Conclusion</h2>
<p>As an application grows, you'll find yourself wanting to handle certain things in common ways - the state model for network requests, dialogs, and so on. This technique is a handy way of creating small modules that are ripe for re-use across your application. With this technique put in practice, you'll become familiarised with a common state model, and see a common set of actions that conform to a pattern in your dispatched action log.</p>
<p>Redux application states can become complicated to reason about. Reducing the cognitive overhead in doing so is imperative as your application grows in size and complexity. Overall, I think that this is an incredibly useful way of bringing down the cost of maintenance in a large-scale Redux project.</p>
Reusing Redux Forms for Creating and Editing2017-04-11T00:00:00Zhttps://samhogy.co.uk/2017/04/reusing-redux-forms-for-creating-and-editing/<p>I'm a big fan of the Open Source <a href="http://redux-form.com/">Redux Form</a> library - its primary purpose is to make handling form state in Redux applications simple. The general principle is that you define a Redux Form by specifying a React component with a bunch of <code><Field></code>s - each with a <code>name</code> property. Your form has a <code>name</code> too. What you end up with in your Redux state is a <code>form</code> object, with a sub-object for your form. Each <code><Field></code> is a connected component - it's hooked up to the state and will raise actions when the value changes. Validation is supported too.</p>
<img src="https://raw.githubusercontent.com/erikras/redux-form/master/logo.png" alt="Redux Form Logo" />
<p>One of the tasks I frequently run into is the need to have a form for creating (let's say, a user). Later, a requirement comes down to also support editing. We'd like to re-use the form we designed for creation, alongside all of the validation. How can we do this? It's actually incredibly easy!</p>
<p>Let's take a look at an incredibly trivial Redux Form, with some simple validation:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token keyword">import</span> React<span class="token punctuation">,</span> <span class="token punctuation">{</span> PropTypes <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'react'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">import</span> <span class="token punctuation">{</span> reduxForm<span class="token punctuation">,</span> Field <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'redux-form'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">UserForm</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> onSubmit <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>form</span> <span class="token attr-name">onSubmit</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>onSubmit<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>label</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span><span class="token class-name">Field</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>id<span class="token punctuation">"</span></span> <span class="token attr-name">component</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>input<span class="token punctuation">"</span></span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"> ID</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>label</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>label</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span><span class="token class-name">Field</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>name<span class="token punctuation">"</span></span> <span class="token attr-name">component</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>input<span class="token punctuation">"</span></span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span><span class="token plain-text">Name</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>label</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>button</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>submit<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token plain-text">Submit</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>button</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>form</span><span class="token punctuation">></span></span><span class="token punctuation">;</span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">validate</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">values</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> result <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>values<span class="token punctuation">.</span>id<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> result<span class="token punctuation">.</span>id <span class="token operator">=</span> <span class="token string">'Please enter a user id'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>values<span class="token punctuation">.</span>name<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> result<span class="token punctuation">.</span>name <span class="token operator">=</span> <span class="token string">'Please enter a name'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">return</span> result<span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> CreateUserForm <span class="token operator">=</span> <span class="token function">reduxForm</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">form</span><span class="token operator">:</span> <span class="token string">'createUser'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> validate</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">(</span>UserForm<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span></code></pre>
<p>Pretend for now that this is a more complicated form, with lots of custom form elements, business logic validation rules and looks stunning. If we want to re-use this for editing, then we'll want to retain all of that. How can we do it?</p>
<p>Firstly, note that the binding of the form component to the redux state is done via the <code>redux-form</code> decorator. This is where we give the form an identifying name and hook up the validation logic. This actually leaves the underlying form component entirely reusable - <strong>all we have to do is give the form a different name!</strong>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> EditUserForm <span class="token operator">=</span> <span class="token function">reduxForm</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">form</span><span class="token operator">:</span> <span class="token string">'editUser'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> validate</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span></code></pre>
<h2>Customising the Form</h2>
<p>Nice! Okay, let's now say that a requirements comes in that only <em>some</em> fields can be edited in your form. How can we implement this?</p>
<p>I'd introduce a property passed to the form component to identify the mode that we're using the form in. Something like:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line">UserForm<span class="token punctuation">.</span>propTypes <span class="token operator">=</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">mode</span><span class="token operator">:</span> PropTypes<span class="token punctuation">.</span><span class="token function">oneOf</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'create'</span><span class="token punctuation">,</span> <span class="token string">'edit'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">onSubmit</span><span class="token operator">:</span> PropTypes<span class="token punctuation">.</span>func</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<p>To set the prop and ensure that it's passed through to the form component correctly, we have to slightly change the code above. We introduce an intermediate component which renders the underlying form component:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">CreateUserForm</span> <span class="token operator">=</span> <span class="token parameter">props</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span><span class="token class-name">UserForm</span></span> <span class="token attr-name">{</span>
<span class="highlight-line"> <span class="token attr-name">...props,</span></span>
<span class="highlight-line"> <span class="token attr-name">mode</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">'</span>create<span class="token punctuation">'</span></span></span>
<span class="token attr-name">}</span> <span class="token punctuation">/></span></span><span class="token punctuation">;</span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">EditUserForm</span> <span class="token operator">=</span> <span class="token parameter">props</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span><span class="token class-name">UserForm</span></span> <span class="token attr-name">{</span>
<span class="highlight-line"> <span class="token attr-name">...props,</span></span>
<span class="highlight-line"> <span class="token attr-name">mode</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">'</span>edit<span class="token punctuation">'</span></span></span>
<span class="token attr-name">}</span> <span class="token punctuation">/></span></span><span class="token punctuation">;</span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> ConnectedCreateUserForm <span class="token operator">=</span> <span class="token function">reduxForm</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token operator">...</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">(</span>CreateUserForm<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">const</span> ConnectedEditUserForm <span class="token operator">=</span> <span class="token function">reduxForm</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token operator">...</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">(</span>EditUserForm<span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Other props that are passed to the ReduxForm components (the <code><Connected * UserForm></code> ones), such as <code>onSubmit</code>, will be passed through too, as I'm use property spreading - a neat feature provided in ES6 via the Object Spread operator.</p>
<p>Now, when we render the form (either <code><CreateUserForm /></code> or <code><EditUserForm /></code>) the mode is set to <code>create</code> or <code>edit</code> appropriately. We can use this prop to determine whether the <code>id</code> field is enabled:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">UserForm</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> mode<span class="token punctuation">,</span> onSubmit <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>form</span> <span class="token attr-name">onSubmit</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>onSubmit<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>label</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span><span class="token class-name">Field</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>id<span class="token punctuation">"</span></span> <span class="token attr-name">component</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>input<span class="token punctuation">"</span></span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token attr-name">disabled</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span> mode <span class="token operator">===</span> <span class="token string">'edit'</span> <span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"> ID</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>label</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>label</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span><span class="token class-name">Field</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>name<span class="token punctuation">"</span></span> <span class="token attr-name">component</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>input<span class="token punctuation">"</span></span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span><span class="token plain-text">Name</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>label</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>button</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>submit<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token plain-text">Submit</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>button</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>form</span><span class="token punctuation">></span></span><span class="token punctuation">;</span></code></pre>
<p>You could also do something like altering the form title, based on the <code>mode</code>:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">UserForm</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> mode<span class="token punctuation">,</span> onSubmit <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>form</span> <span class="token attr-name">onSubmit</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>onSubmit<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h2</span><span class="token punctuation">></span></span><span class="token punctuation">{</span> mode <span class="token operator">===</span> <span class="token string">'edit'</span> <span class="token operator">?</span> <span class="token string">'Edit'</span> <span class="token operator">:</span> <span class="token string">'Create'</span> <span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h2</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> ...</span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>form</span><span class="token punctuation">></span></span><span class="token punctuation">;</span></code></pre>
<h2>Conclusion</h2>
<p>This might be a simple example, but it's a good demonstration of the power of React and of component composition. As it's the decorator (the call to <code>reduxForm</code>) that controls where on the state model we look, everything underneath that is free to operate without knowledge of a state model. I think this is a great credit to the design of Redux Form.</p>
<p>One final point to make - how much you go down this route really depends on how close your create/edit forms turn out to be. I'd be happy with a bit of <code>if ... else</code> logic in a common component as long as it doesn't become too gnarly. But for me it strikes a decent balance between re-usability and customisation.</p>
Embracing Higher-Order Reducers2017-03-24T00:00:00Zhttps://samhogy.co.uk/2017/03/embracing-higher-order-reducers/<p>Here's a quick story about how I used <strong>Higher-Order Reducers</strong> to add a new feature to my D&D Initiative Tracker application, whilst keeping the code maintainable and reusable.</p>
<p>I was adding a feature allowing the user to clear any of their inputted data. It's quite a simple job - render a button, raise an action when the button is clicked and then handle the action in each reducer that contains user data. But this is duplication, as in each reducer you're effectively performing the same piece of work. This seems like a code smell and I started investigating ways to remedy this.</p>
<p>In functional programming terms, a higher-order function is a function whose return type is another function, or whose arguments can be functions. Languages that support this concept treat functions as first-class citizens - they're just another type that can be passed around, like an int, string or object. It follows that a higher-order reducer is one which returns a reducer, or accepts reducers as arguments.</p>
<p>One use of higher-order reducers is to wrap a reducer, to extend its functionality. Let's take a look at a code sample:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">persistable</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">defaultState<span class="token punctuation">,</span> wrappedReducer</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">(</span><span class="token parameter">state <span class="token operator">=</span> defaultState<span class="token punctuation">,</span> action</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>action<span class="token punctuation">.</span>type <span class="token operator">===</span> <span class="token string">'CLEAR_PERSISTENCE'</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> defaultState<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token keyword">else</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">wrappedReducer</span><span class="token punctuation">(</span>state<span class="token punctuation">,</span> action<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The outer function returns a reducer function, which can be included in our application via a <code>combineReducers</code> call, just like any other Redux reducer. We're passing two arguments to this outer function call - the default state, and a reducer we'd like to augment with extra behaviour.</p>
<p>A single reducer is returned, but now it handles an extra action - when <code>CLEAR_PERSISTENCE</code> is raised we return the initial default state. Other actions are passed through to the reducer we've wrapped. This is essentially the <a href="https://en.wikipedia.org/wiki/Decorator_pattern">Decorator Pattern</a> from OOP.</p>
<p>So let's see how we can wrap an existing reducer with our <code>persistable</code> reducer:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> defaultState <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">''</span> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">nameReducer</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">state<span class="token punctuation">,</span> action</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">switch</span> <span class="token punctuation">(</span>action<span class="token punctuation">.</span>type<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">case</span> <span class="token string">'SET_NAME'</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token operator">...</span>state<span class="token punctuation">,</span> <span class="token literal-property property">name</span><span class="token operator">:</span> action<span class="token punctuation">.</span>value <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">default</span><span class="token operator">:</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> state<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> persistableReducer <span class="token operator">=</span> <span class="token function">persistable</span><span class="token punctuation">(</span>defaultState<span class="token punctuation">,</span> nameReducer<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> persistableReducer<span class="token punctuation">;</span></span></code></pre>
<p>The concept of a persistable reducer is now entirely re-usable. We can call it multiple times, passing in a different reducer each time, to augment each of those with the common behaviour. Equally, the code to manage what happens exists in one place.</p>
Design, Data and Redux at Bristech2016-11-04T00:00:00Zhttps://samhogy.co.uk/2016/11/design-data-and-redux-at-bristech/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2016/11/04/design-data-and-redux-at-bristech.html">the Scott Logic blog</a></span></div>
<p>Yesterday I attended (and spoke!) at the 2016 Bristech conference, an event aimed at polyglots, innovators and the "tech curious". It was a full house at the Watershed with over 400 tickets sold. I had a great time at the conference, attending quite a few talks myself. What made the conference interesting to me was the diverse range of subjects covered - design, security, even physics! Presented here are some brief notes and comments from the day's proceedings.</p>
<h2>Create the right product! What happens before wireframing?</h2>
<p>(Carl Bettag and Dave Ellender)</p>
<p>This talk introduced me to the notion of design research - the very early discovery phase of a project that's normally conducted before we even have a complete idea of what we'd like to build. The technique involves asking questions, developing an understanding of mental models and building a picture of how the world works. We can then use this knowledge to inform us when we wish to take a concept and design a product in that space.</p>
<p>The example used was the development of an insulin pump for patients with diabetes, which is designed to be worn at all times. By discussing the problem with users they realised that there was a demand for devices that can be worn but does not impact upon the patient's ability to live a normal, healthy life. This was in contrast to the other products in the space, which were competing on the features and power of their devices. Having the conversations with the users resulted in a product that was user-friendly, customisable and more successful. By identifying the problem that mattered to the users, they were able to build a product better suited to their needs.</p>
<p>As an approach it's incredibly appealing, especially compared to the risk of having to fix key concepts at a later stage in the project lifecycle - something we all know is difficult to achieve, costly and time-consuming.</p>
<h2>Linking Data on the Web</h2>
<p>(Jen Williams)</p>
<p><code><a href</code> and you're done, right? Wrong! In this talk Jen introduced the idea of a web of linked data - which is something more than just putting data on the web. The concept was introduced by <a href="https://www.w3.org/DesignIssues/LinkedData.html">Tim Berners-Lee in 2006</a> (his <a href="https://www.ted.com/talks/tim_berners_lee_on_the_next_web?language=en">TED talk</a> on the subject is a very accessible primer). It involves enriching data to add meaning. Jen discussed the use of data description formats such as RDF to accomplish this, which enables humans and machines to understand more about the context of data. Search engines such as Google, Bing and Yahoo are heavily involved in this space.</p>
<p>Take for example a list of my speaking engagements, which could be uploaded to the web as a simple CSV data dump. Alternatively, in a linked data world this would be enriched with information about the subjects I'm speaking about, the cities I'm speaking at, the details of the event organisers and so on. These are linked through a vocabulary, which can be custom-built, but common sets do exist.</p>
<p>I thought this was a great talk, which was an eye-opener onto the seemingly-untapped abilities of our World Wide Web, if only we gave it a bit more of a helping hand!</p>
<h2>The Hitchhiker's Guide to Redux</h2>
<p>(Sam Hogarth - hey, that's me!)</p>
<img src="https://samhogy.co.uk/assets/shogarth_bristech.jpg" alt="Sam Hogarth, Speaking at Bristech" width="800" height="600" />
<p>I have been a keen follower of the evolution in front-end application architecture that we have witnessed over the past few years, from two-way binding MVC frameworks, towards the unidirectional style popularised by Facebook with Flux. I was therefore very keen to learn Redux, which reinterprets the unidirectional application update pattern.</p>
<p>What makes Redux particularly interesting is that by imposing some constraints upon development, you gain several benefits. The constraints are that:</p>
<ol>
<li>There is a single source of truth for your application state, as a single JavaScript object</li>
<li>The state is read-only, the only way to change it is by raising an action.</li>
<li>Updates to the state must be handled by pure functions.</li>
</ol>
<p>Firstly, the benefit is that your state update functions are simple, predictable and therefore easily testable. Not only that, but by forcing all state updates to enter the app through a single route, it is incredibly easy to gather an audit trail of changes to the app state.</p>
<p>Finally these constraints provide some excellent developer productivity tooling. Using hot-module reloading (from a tool like Webpack) and Redux's developer tooling, a developer can spot a bug and see the state changes that caused the bug. It can be fixed. Your application will instantly rewind, then replay the same set of state changes, to then verify the bug is fixed. This is known as time-travel debugging.</p>
<p>It was great fun speaking to a packed-full audience, who provided an excellent range of questions!</p>
<h2>Delightful UX for Distributed Systems</h2>
<p>(Mike North)</p>
<p>Mike covered one of the core problems of the distributed age - what happens when a user goes offline, or suffers poor network connectivity, or even switches from a mobile device a web interface? In today's world distributed systems are commonplace, our user experience therefore needs to address these concerns. With examples, Mike demonstrated how taking into account the distributed, asynchronous nature of his application in the core user experience led to a more fault-tolerant system. It also improved the more general user experience. The take-home message was four rules:</p>
<p>1 - Keep users in the loop. If there's some long-running job that needs to be performed, indicate to the users that the task is in progress. If a user is offline, let them know their data has not yet synced to the server.</p>
<p>2 - Loosely couple - reliability breeds trust. If your application speaks to a third-party, you risk being coupled to their uptimes and downtimes. In Mike's case, this caused a significant impact upon their user onboarding times. Their solution was to separate what needs to be done now, from what can be potentially done later.</p>
<p>3 - Limited offline support is okay - a key concept here, a mobile application during "offline" mode doesn't need to provide the bells and whistles experience. Dropping back to a limited feature set is acceptable to users. Determining what features can be temporarily disabled also helps identify the critical path in your application.</p>
<p>4 - Don't wait for "perfect" - the solution is always around the corner. Right now, service workers are looking increasingly promising for running tasks in the background for web applications and providing some offline support. But they aren't fully supported everywhere. Rather than waiting for the perfect solution, build something with the technology that exists now.</p>
<h2>Thank You</h2>
<p>Bristech was a huge success. It had excellent organisation, an interesting line-up of workshops and seminars and a wonderful set of attendees. I'd like to extend my thanks to the organisation team for hosting the event. If you're in the region, Bristech hold a yearly conference plus monthly meetups. I highly recommend paying them a visit!</p>
A Feast for Crows2016-06-26T00:00:00Zhttps://samhogy.co.uk/2016/06/a-feast-for-crows/<p>On Thursday I cast my vote early in the morning, then left to work confident that the UK would vote to Remain a member of the European Union. I was certain we'd do this with a vote share of 60-66%. That was until the Newcastle result came in and it became clear it was going to be a bad night for Remain.</p>
<p>The news has developed quickly and what follows are my thoughts on three key areas of the debate. There may be more to be said in future, about how ugly the campaign had become. Or that how we've created a polarised society where any major decision will thoroughly upset 50% of the population. Or that every promise made by the Leave campaign was rolled back and demonstrated to be completely false within a day, leaving a deep sense of betrayal for many.</p>
<h2>Identity</h2>
<p>I identify as a Geordie, British and European, not necessarily in that order. All of my international travel has been within Europe. I've worked for extended periods of time in other European countries. I've been a huge beneficiary of EU membership. I've lived in a region that has equally benefitted enormously from EU funding, with key infrastructure and tourist attractions developed.</p>
<p>I now feel as if part of that identity has been stripped from me. Even though I didn't vote for it, or that we haven't (yet) actually left the European Union. All because David Cameron added a manifesto commitment to appease his Euroskeptic back-benchers, never thinking it would have to be implemented. But then they won a majority. He will be known as the man who split two Unions unnecessarily.</p>
<p>I worry about the opportunities that I may have lost, the rights that could potentially be stripped away.</p>
<h2>Election</h2>
<p>Cameron decided to fall on his sword in response to the result, announcing he will not invoke Article 50. Instead he'd resign, allowing the next Prime Minister to begin negotiation. So enter Prime Minister Boris Johnson? No doubt we'll have a 'snap' General Election to try and increase the Tory working majority, but also to avoid the 'unelected' trap that Gordon Brown fell into. This election will be fought 100% over Europe, thus a strong message will be the key to electoral success.</p>
<p>The Conservative Party is left deeply divided, with its greatest asset removed. Whilst I'll never forgive Cameron for the actions of his Governments, his contribution to the perception of a stable and competent Conservative Party has been an election winner. Talk of a potential Tory split aren't far from the truth.</p>
<p>But closer to home, we're also seeing attempts to remove Jeremy Corbyn as leader of the Labour Party. Jeremy Corbyn won the leadership contest last year by a huge majority, so attempts to remove him will understandably be perceived as a snub. However in his tenure the Labour Party <em>lost</em> seats in the Council Elections. He has failed to capitalise on open goals such as Iain Duncan Smith's resignation, and been prone to own goals. He has been unable to stop the flow of support from Labour's traditional heartlands to UKIP. His contribution to the European Union debate was lukewarm at best, which is to be expected from a man with a published history of <a href="http://www.itv.com/news/2016-03-05/jeremy-corbyn-accused-of-trying-to-hide-eurosceptic-past/">opposition to the European Union</a>, a known tendency to stick to his principles, and who could barely contain his smile in an interview following the Leave vote on Friday. <a href="http://www.bbc.co.uk/news/uk-politics-36633238">Evidence has emerged</a> of his office undermining the Labour Remain campaign.</p>
<p>It is clear that we cannot continue to support an electorally unsuccessful and widely divisive leader. But I'm also worried that these actions will cause a split of the Labour Party, despite it being <a href="http://www.theguardian.com/politics/live/2016/jun/26/brexit-live-jeremy-corbyn-sacks-hilary-benn-tory-leadership?page=with:block-576fe35ce4b030d83eb4a784#block-576fe35ce4b030d83eb4a784">supported by the majority of Labour members</a>. But we must try and win the upcoming election, in order to win back the disenfranchised voters and ensure their voices are heard in Parliament.</p>
<h2>Opportunity</h2>
<p>Everything is up for grabs and in times of uncertainty, everyone decides to give their cause a pop. Already there are renewed calls for a <a href="https://www.rt.com/uk/348347-scotland-second-independace-referendum/">second Scottish Independence Referendum</a>. Sinn Fein are once again making noises about a <a href="http://www.independent.co.uk/news/uk/politics/brexit-northern-ireland-eu-referendum-result-latest-live-border-poll-united-martin-mcguinness-a7099276.html">United Ireland</a> - as both Scotland and Northern Ireland voted to Remain. Spain have declared an <a href="http://www.bbc.co.uk/news/world-europe-36618796">interest in Gibraltar</a>. Wales and Cornwall are seeking assurances that they will continue to receive the funding that the European Union grants them, despite their residents overwhelmingly voting to Leave. <a href="https://petition.parliament.uk/petitions/131215">There are calls for a second referendum</a>, with over 3 million signatories at the time of writing.</p>
<p>Nigel Farage hoped this vote was the "first brick in the wall" to come loose. He may not be wrong; far-right leaders across Europe are celebrating the UK result and <a href="https://www.theguardian.com/world/2016/jun/24/european-far-right-hails-britains-brexit-vote-marine-le-pen">calling for their own referendums on European Union membership</a>. The potential breakup of a political union that has provided stability and peace to a historically warring Continent is concerning, especially when some of the instigators are actual Nazis.</p>
<p>On the streets of the UK there are disturbing <a href="http://www.theguardian.com/politics/2016/jun/26/racist-incidents-feared-to-be-linked-to-brexit-result-reported-in-england-and-wales">reports of racism</a> towards minorities. Cries of "go home", demands to "fit in or fuck off". Children coming home from school in tears because of their family heritage. Not everyone who voted Leave is racist, but it is clear that the Vote Leave campaign made its primary argument on immigration. The language became incendiary. Nigel Farage used posters with alarming similarity to Nazi propaganda. He says that the referendum was won <a href="http://www.mirror.co.uk/news/uk-news/nigel-farage-claims-hes-won-8271149">"without a bullet being fired"</a> - but of course Jo Cox MP was assassinated outside of a library in her constituency by a man who later gave his name in court as <a href="http://www.bbc.co.uk/news/uk-36567005">"Death to traitors, freedom for Britain"</a>. It's as if certain people have interpreted the vote as a permission to be openly racist.</p>
<p>The political landscape has completely changed and the only emotion I consistently feel is fear. Fear of what I will lose, and of what's to come. We don't seem to have a Government, or an Opposition. The far-right are on the rise, and when everything comes out in the wash, we may not even have a United Kingdom left.</p>
Vote Remain to Protect the North East2016-06-11T00:00:00Zhttps://samhogy.co.uk/2016/06/vote-remain-to-protect-the-north-east/<p>Friends, Family,</p>
<p>In the upcoming Referendum, I will be proudly voting for the United Kingdom to REMAIN a member of the European Union. I'd like to explain why I have came to this conclusion.</p>
<p>Put simply, my opinion is that the North East has greatly benefited by partnership with the European Union. We should protect the economic development in our region by committing to this productive relationship.</p>
<h2>Investing in the North East</h2>
<p>In my lifetime I've seen Newcastle Quayside develop into a thriving tourist attraction. Many of the developments were paid for out of European Union regional development funds. I'm part of a blossoming software industry. The North East still has a strong manufacturing base - from Teesside petrochemicals to the Washington Nissan factory. In the case of Nissan, 70% of their trade is with the European Union. We're one of the most Eurosceptical regions of the country, but we have benefitted, and continue to benefit in huge ways from this partnership.</p>
<p>Not only will voting to leave the European Union affect the entire country, it will hit areas like ours disproportionately hard. Best economic estimates predict a minimum of a two-year recession. We've seen the effects that a recession has on areas like ours. Tory austerity cuts to the bone, and then onwards.</p>
<p>Our funding from the European Union (matched in some areas by UK Government Departments) has helped industries get off the ground. It's contributed money into the redevelopment of Gateshead College and New College Durham through the European Social Fund. We get more out than we put in. Advocates of leaving claim we'll control this money ourselves, but it's not exactly clear how. We know councils are worried about future funding cuts from the UK Government.</p>
<p>As part of our current arrangement, the North East of England punches well above its weight. We should vote to protect this and commit to developing our region, not devouring it through more austerity.</p>
<h2>No Clear Plan for Leaving</h2>
<p>No country has ever left the European Union. There can be no certainty on what happens next, but there are some likely scenarios that we can consider. Unfortunately, the politicians advocating Leave have been widely discredited on all fronts, unable to provide a clear plan for how the United Kingdom would operate outside of a European Union. They talk about creating trade deals with other countries - but we're already doing that. They talk about how the world will rally to our cause - despite clear indications from other nations that this will not be the case. They have claimed we can have our cake and eat it - that we will regain access to the Single Market without having to pay tariffs or comply with their regulation, but remaining an influential partner. Again, we've been told otherwise, by the very people who we'd be negotiating with.</p>
<p>All we're left with is blind, almost religious, belief that everything will be okay. Maybe it will, but we need to think about how long this would take, and what happens along the way.</p>
<p>I don't think it's a big ask for these people to give us a clear answer on their plan. It rings alarm bells that they cannot.</p>
<h2>Conclusion</h2>
<p>We're being asked to make a <em>huge</em> decision. There are no certainties - just probabilities, to which people will assign different weightings based on their own philosophies. The primary reason I wish to remain in the European Union is to protect the strong position the North East has forged. Of course, there are lots of secondary reasons too: in the short term the economic shock caused by leaving will leave lasting damage and will be exploited by the right-wing for further cuts. In the long term, it's unclear, but my best guess is we'd need a significant restructuring of our economy.</p>
<p>I have concerns about the fascist rhetoric that the Leave campaign has adopted. Immigration and the European Union are the dual scapegoats for all the ills (real, imagined or untrue) that the United Kingdom faces. Racism has not gone away, it's been deliberately exploited by politicians and the press to push a deeply Conservative agenda. We have to accept that we live in a globalised society. Attempting to roll back this progress is counterproductive. This rhetoric needs to be thoroughly rejected.</p>
<p>We've been given little of substance to give us a reason to leave. We need to acknowledge the positive impact European Union membership has had on our region.</p>
Further Thoughts on Redux2016-06-05T00:00:00Zhttps://samhogy.co.uk/2016/06/further-thoughts-on-redux/<p>Progress on my <a href="https://github.com/sh1989/initiative-tracker-redux">DnD Initiative Tracker app</a> has been steady. I've been working on it for about two months in my spare time. It's been a bit slow, but that's fine when you're learning.</p>
<p>The application has slightly grown, previously it had a pre-set encounter, with some players and enemies. Now, the application has the following feature set:</p>
<ul>
<li>The home screen shows your players and encounters</li>
<li>You can create a player (including level, HP and key attributes)</li>
<li>You can create an encounter, which has a name and a list of enemies</li>
<li>Starting an encounter navigates to a 'roll for initiative' screen</li>
<li>During an encounter you control damage, healing and death saving throws</li>
<li>After an encounter a summary screen is displayed.</li>
</ul>
<img src="https://samhogy.co.uk/assets/dnd_tracker_home.png" alt="D&D Initiative Tracker Home Screen" width="600" />
<img src="https://samhogy.co.uk/assets/dnd_tracker_builder.png" alt="Encounter Builder Screen" width="600" />
<img src="https://samhogy.co.uk/assets/dnd_tracker_encounter.png" alt="Encounter Runner Screen" width="600" />
<p>Building a relatively decent-sized application has taught me a lot about the Redux ecosystem. I'd like to follow up on my <a href="/2016/04/initial-thoughts-on-redux/">"Initial Thoughts on Redux"</a> post with some of the things that I have discovered along the way.</p>
<h2>Normalised state</h2>
<p>In my previous post I highlighted how you should work to keep your state as normalised as possible. As your application grows this becomes incredibly important. It reduces redundancy but also complexity. Flattening your state model means that changes to one particular section of your state are localised to one reducer. You can utilise a library such as <a href="https://www.npmjs.com/package/uuid">node-uuid</a> to generate uuids for your entities. Previously I was generating ids in a rather silly way:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">let</span> idCount <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function-variable function">addEncounter</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> id <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">ENC</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>idCount<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">;</span></span>
<span class="highlight-line"> idCount<span class="token operator">++</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'ADD_ENCOUNTER'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> id</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>If you were to persist your state, either to a server or local storage, a subsequent reload of the application would reset this <code>idCount</code> variable to 0, and you'll start getting id clashes. So design with the future in mind and use proper uuids:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> uuid <span class="token keyword">from</span> <span class="token string">'node-uuid'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function-variable function">addEncounter</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'ADD_ENCOUNTER'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">id</span><span class="token operator">:</span> uuid<span class="token punctuation">.</span><span class="token function">v4</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Selectors as an Interface</h2>
<p>The <a href="https://github.com/reactjs/reselect">reselect</a> library is primarily intended for Redux applications suffering from performance problems. Its idea is to create memoizable selectors - getters that retrieve data from your state tree. However, in of itself it is a useful tool to expose an more readable interface over your reducers.</p>
<p>Selectors are a chain of functions. They begin simply, accepting the state tree as an argument:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">getActiveEncounter</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">state</span><span class="token punctuation">)</span> <span class="token operator">=></span> state<span class="token punctuation">.</span>encounters</span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">e</span> <span class="token operator">=></span> e<span class="token punctuation">.</span>status <span class="token operator">===</span> <span class="token string">'active'</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">keySeq</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">getEncounters</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">state</span><span class="token punctuation">)</span> <span class="token operator">=></span> state<span class="token punctuation">.</span>encounters<span class="token punctuation">;</span></span></code></pre>
<p>Then they build up into more formal selectors. You specify, as arguments to a <code>createSelector</code> call, which selectors to accept as input. When they change, the resultant function is called:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">const</span> getCombatants <span class="token operator">=</span> <span class="token function">createSelector</span><span class="token punctuation">(</span></span>
<span class="highlight-line"> <span class="token punctuation">[</span>getActiveEncounter<span class="token punctuation">,</span> getEncounters<span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token punctuation">(</span><span class="token parameter">activeEncounter<span class="token punctuation">,</span> encounters</span><span class="token punctuation">)</span> <span class="token operator">=></span> encounters<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span>activeEncounter<span class="token punctuation">)</span><span class="token punctuation">.</span>combatants</span>
<span class="highlight-line"><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>Selectors can be called within your typical <code>mapStateToProps</code> function, but I've also found them useful if I need to do anything complicated with an asynchronous action creator. For example, when the user wishes to start an encounter, we to take the enemies and players, and transform them into <code>Combatant</code> objects. I do this as an action, but I use a selector to do the actual fetch work. This is all done asynchronously via <code>redux-thunk</code>, so that you have access to the state:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">import</span> getEncounterCombatants <span class="token keyword">from</span> <span class="token string">'./../selectors/getEncounterCombatants'</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function-variable function">startEncounter</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">encounterId</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">(</span><span class="token parameter">dispatch<span class="token punctuation">,</span> getState</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> state <span class="token operator">=</span> <span class="token function">getState</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> encounterCombatants <span class="token operator">=</span> <span class="token function">getEncounterCombatants</span><span class="token punctuation">(</span>state<span class="token punctuation">,</span> encounterId<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token function">dispatch</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">type</span><span class="token operator">:</span> <span class="token string">'START_ENCOUNTER'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">enemies</span><span class="token operator">:</span> encounterCombatants<span class="token punctuation">.</span>enemies<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">players</span><span class="token operator">:</span> encounterCombatants<span class="token punctuation">.</span>players<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">encounterId</span><span class="token operator">:</span> encounterId</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>You may not wish to use selectors for literally every piece of state access - indeed such a choice should be profiled in my opinion! - but thinking of selectors as a formal interface to your data increases the simplicity of accessing your state data. Especially when it's incredibly normalised!</p>
<h2>Separate your domain and UI state</h2>
<p>In addition to your domain state, comprising your business models, there's also the UI state. UI state that is transient, such as whether an element is focused, or what current working value of an <code><input></code> is. This data may be considered purely UI, so why bother tracking it outside of a React component? It's perfectly valid to store data outside of Redux's state tree, if you feel this is necessary. Alternatively, you may decide to track your UI state within the Redux state tree, in which case you'll have to expend a lot of effort plugging up <code>onBlahChange</code> handlers for every UI component. It makes for a rather complex React tree.</p>
<p>I had this issue with my application. I used a form to handle adding a player and an enemy. In one case I hooked up every input component to dispatch an action. In the other, I decided to not bother and simply gathered all of the data together in an action creator fired <code>onSubmit</code>. Thankfully I discovered the <a href="http://redux-form.com/">redux-form</a> library, which acts as a higher-order component, and does all the legwork for you. You declare the form's fields (and optionally its initial values), identify the form by a unique key, and wrap it all up:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token keyword">const</span> <span class="token function-variable function">AddEnemy</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">props</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">const</span> <span class="token punctuation">{</span> <span class="token literal-property property">fields</span><span class="token operator">:</span> <span class="token punctuation">{</span> name<span class="token punctuation">,</span> hp<span class="token punctuation">,</span> bonus <span class="token punctuation">}</span><span class="token punctuation">,</span> handleSubmit <span class="token punctuation">}</span> <span class="token operator">=</span> props<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">(</span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>form</span> <span class="token attr-name">onSubmit</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>handleSubmit<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token operator">...</span>name<span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>number<span class="token punctuation">"</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token operator">...</span>hp<span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>number<span class="token punctuation">"</span></span> <span class="token spread"><span class="token punctuation">{</span><span class="token operator">...</span>bonus<span class="token punctuation">}</span></span> <span class="token punctuation">/></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>button</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>submit<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token plain-text">Add</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>button</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>form</span><span class="token punctuation">></span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">;</span></span></code></pre>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> AddEnemyForm <span class="token operator">=</span> <span class="token function">reduxForm</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">form</span><span class="token operator">:</span> <span class="token string">'addEnemy'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">fields</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">,</span> <span class="token string">'hp'</span><span class="token punctuation">,</span> <span class="token string">'bonus'</span><span class="token punctuation">]</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">initialValues</span><span class="token operator">:</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">''</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">hp</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">bonus</span><span class="token operator">:</span> <span class="token number">0</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">(</span>AddEnemy<span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>In your root reducer, add in the redux-form reducer:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="highlight-line"><span class="token keyword">export</span> <span class="token keyword">default</span> <span class="token function">combineReducers</span><span class="token punctuation">(</span><span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token literal-property property">encounters</span><span class="token operator">:</span> encounters<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">enemies</span><span class="token operator">:</span> enemies<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">players</span><span class="token operator">:</span> players<span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token literal-property property">form</span><span class="token operator">:</span> formReducer</span>
<span class="highlight-line"><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>The library will handle dispatching actions to this isolated part of the state tree, so you don't have to.</p>
<p>There are other libraries that take this approach too, such as <a href="https://github.com/tonyhb/redux-ui">redux-ui</a>. It uses decorators to hydrate UI components with state from your reducers. These libraries touch upon the same idea - if you're going to use your state tree to store UI state, ensure that it's isolated from the domain parts of your state tree. Then when it comes to persisting your state tree, you can simply chop off the UI parts and only persist the domain logic.</p>
<hr>
<p>The application is not yet complete. There's a whole bunch of small things that I need to fix, ranging from small bugs to UI polishing. I have a few new features to add, such as taking the monsters from the 5th Edition System Reference Document and allowing users to add these to their encounters, rather than building afresh. This will then allow me to calculate encounter difficulties, based on the enemy challenge rating. I hope that adding new features will open up new avenues to explore within the Redux ecosystem.</p>
Initial Thoughts on Redux2016-04-05T00:00:00Zhttps://samhogy.co.uk/2016/04/initial-thoughts-on-redux/<p>I've started creating a sample application in Redux. You can follow the progress <a href="https://github.com/sh1989/initiative-tracker-redux/">on my github repo</a>.</p>
<p>It's a simple combat initiative tracker for Dungeons and Dragons Fifth Edition. Combat takes place in rounds, with players rolling an 'initiative' score to determine their order in the round. My application presents the ordered round list and cycles through each combatant, where they can do damage to other combatants.</p>
<p>Once I've finished the main part of the app I'll swap out the pre-fix combatants with what I'm calling an "onboarding" wizard - essentially a series of menus that lets a player add players and enemies. I plan on adding a server-side component too, so that the monster list can be based on real data.</p>
<p>There's enough in place at the time of writing to present some initial comments on Redux. I'll prefix everything that I say by saying that these opinions may change as I become more familiar with the library. But first, a quick overview on what's interesting about Redux, with a bit of history!</p>
<p>I use React and Flux in my day job and it's surprisingly effective. The project is six months old now and everything is holding up pretty well. We're considering moving our Flux code to Redux, so I thought I'd try it out.</p>
<h2>React</h2>
<p><a href="https://facebook.github.io/react/">React</a> only concerns itself with the 'view' in a typical MVC application, with its intent to be a pure functional transformation between a piece of state (i.e a JavaScript object) and a piece of DOM. This makes React fast, especially when it only applies the minimal necessary DOM changes to the real DOM, in response to a re-render call.</p>
<p>Commonly seen in React code is JSX, a syntax that effectively lets you write close to DOM markup, interspersed with JavaScript where necessary:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token function">render</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token punctuation">(</span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>section</span><span class="token punctuation">></span></span><span class="token plain-text">
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h1</span><span class="token punctuation">></span></span><span class="token plain-text">Round </span><span class="token punctuation">{</span><span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>round<span class="token punctuation">}</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h1</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>ul</span><span class="token punctuation">></span></span><span class="token plain-text"></span>
</span><span class="token punctuation">{</span> <span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>combatants<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">combatant</span> <span class="token operator">=></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span> <span class="token attr-name">className</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span><span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>currentPlayerId <span class="token operator">===</span> combatant<span class="token punctuation">.</span>id <span class="token operator">?</span> <span class="token string">'active'</span> <span class="token operator">:</span> <span class="token string">''</span><span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token attr-name">key</span><span class="token script language-javascript"><span class="token script-punctuation punctuation">=</span><span class="token punctuation">{</span>combatant<span class="token punctuation">.</span>id<span class="token punctuation">}</span></span><span class="token punctuation">></span></span><span class="token plain-text"></span>
<span class="highlight-line"> </span><span class="token punctuation">{</span>combatant<span class="token punctuation">.</span>name<span class="token punctuation">}</span><span class="token plain-text"></span>
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>ul</span><span class="token punctuation">></span></span><span class="token plain-text">
</span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>section</span><span class="token punctuation">></span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>I find this immensely powerful and satisfying. Some really disagree with melding your presentation logic in this way, preferring something like <a href="https://github.com/dominictarr/hyperscript">Hyperscript</a>, which would instead look like:</p>
<pre class="language-jsx"><code class="language-jsx"><span class="highlight-line"><span class="token function">render</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">return</span> <span class="token function">h</span><span class="token punctuation">(</span><span class="token string">'section'</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">h</span><span class="token punctuation">(</span><span class="token string">'h1'</span><span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>round<span class="token punctuation">)</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> <span class="token function">h</span><span class="token punctuation">(</span><span class="token string">'ul'</span><span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>combatants<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">combatant</span> <span class="token operator">=></span></span>
<span class="highlight-line"> <span class="token function">h</span><span class="token punctuation">(</span><span class="token keyword">this</span><span class="token punctuation">.</span>props<span class="token punctuation">.</span>currentPlayerId <span class="token operator">===</span> combatant<span class="token punctuation">.</span>id <span class="token operator">?</span> <span class="token string">'li.active'</span> <span class="token operator">:</span> <span class="token string">'li'</span><span class="token punctuation">,</span> combatant<span class="token punctuation">.</span>name<span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">)</span></span>
<span class="highlight-line"> <span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>I don't think it matters that much. I prefer to write what's going to be output, even if it means taking up some extra space with closing tags. Choose what grooves with you.</p>
<p>React is a brilliant little library for producing functional, composable and clean UI code that is performant at scale.</p>
<h2>Flux</h2>
<p><a href="https://facebook.github.io/flux/">Flux</a> aims to provide the big picture. It's a pattern rather a framework, as such there are many different implementations, not all of which use React as a view component. d3 or angular can do just fine!</p>
<p>Flux returns to the roots of MVC. It favours a unidirectional data flow through your application. That's a fancy way of saying that all changes to the application state, whether instigated from the UI, networking code, etc, must all be posted as actions. Actions are simply a JavaScript object, normally with a <code>type</code> property whose value is an ID (e.g <code>END_TURN</code>), and some other properties with data.</p>
<p>Actions are handled by a single 'bus' object, known as a dispatcher. In a Flux architecture your model objects are held in Stores - an object representing a particular domain in your application. Stores only have getters, no setters! In order for their state to be changed, they must register with the dispatcher. Doing so will let them receive actions and they can update their state appropriately.</p>
<p>When a store is updated as a result of an action, it raises some event that your UI code listens to, which triggers a re-render.</p>
<p>The hard part of Flux is figuring out when to split up a Store - they end up being a bit larger in size than what you're used to. There's no hard and fast rule, it's purely a judgement call for you to make.</p>
<p>I have found that Flux architectures are pleasant to work in. Provided that you stick to the pattern and ensure all state changes occur via actions, Flux can scale well and be tested easily.</p>
<h2>Redux</h2>
<p>Onto Redux. You can think of <a href="https://github.com/reactjs/redux">Redux</a> as an implementation of the Flux pattern, but likewise it constitutes a slight departure.</p>
<p>The key difference here is that instead of having many stores, there's one. It contains all of the application state, as a large JavaScript object. As there's only one store, you don't need an independent dispatcher. That becomes part of this monolithic store too!</p>
<p>But this isn't as bad as it looks. The job of a store is twofold: 1) it must store the application state, and 2) it must handle requests to change this state, via actions.</p>
<p>Redux <em>splits</em> these two responsibilities. Your singular store stores the application state. A new tree of components, called <strong>reducers</strong> handle the requests to change the state. A reducer is a function: <code>reducer(oldState, action) => newState</code>. Reducers can be arranged into a tree-like structure, so that your application can be effectively namespaced. Split your reducers by responsibility. By applying this separation, you get several benefits:</p>
<ul>
<li>The creation of a store is automatic, just give it your top-level reducers.</li>
<li>During development time, you can change a single reducer and have it <em>hot reloaded</em> into your running app, without having to rebuild the entire codebase! You can get some incredibly fast feedback, without having the state of the application reset or the browser refreshed. How cool is that?!</li>
<li>As actions are an audit trail of state changes to your application, you can <em>time travel</em>, undoing and re-applying state changes. Combined with hot reloading, you can discover a bug in a reducer, fix it, then replay the actions to verify it's fixed.</li>
</ul>
<p>So far the most difficult aspect for me at the moment is modelling my data in such a way that it allows for multiple reducers to act on it, without treading on each other's toes. Redux seems to advocate a heavily-normalized state object, so I'm going to look into normalizing my data even further.</p>
<p>On principle I'm fully on board with the advantages that Redux appears to offer. Now whilst at a small size, the usage of Containers with their <code>mapStateToProps</code> and <code>mapDispatchToProps</code> functions seems like total overkill, such abstractions are essential for larger applications.</p>
<p>What Redux doesn't handle, almost in any way, is how asynchronous actions should be handled. In fairness, neither does Flux. But several innovative ideas have formed in this area - in particular <a href="https://github.com/reactjs/redux-thunk">Redux Thunk</a>, <a href="https://github.com/yelouafi/redux-saga">Redux Saga</a> and <a href="https://github.com/raisemarketplace/redux-loop">Redux Loop</a>.</p>
<p>I'll try each of those for my onboarding wizard.</p>
Git Fu: Commit and Test Partial Changes2016-03-31T00:00:00Zhttps://samhogy.co.uk/2016/03/git-fu-commit-and-test-partial-changes/<p>I was adding a new feature to an existing code base, causing some fairly large changes along the way. I opened one file to make some changes, but noticed the file could also be re-written in ES2016. So that's what I did. But it would make sense for me to commit <em>just</em> that single file change. To do that, I'd need to make sure it works in isolation - how can I do that when there's other changes in progress?</p>
<p>Turns out, quite easily. Here's how.</p>
<ul>
<li>Add the files you want to commit with <code>git add</code></li>
<li>Stash the leftover unstaged changes with <code>git stash -k</code></li>
<li>Rebuild and re-test. If you have file watchers, this may even be automatic and very quick!</li>
<li>Now we're satisfied with that change in isolation, <code>git commit</code></li>
<li>Bring bring the rest of your changes back with <code>git stash pop</code></li>
</ul>
Public Perceptions of Parliamentary Procedure2016-03-16T00:00:00Zhttps://samhogy.co.uk/2016/03/public-perceptions-of-parliamentary-procedure/<p>That's a lot of P's in the title, but following yesterday's shenanigans with the Second Reading of the Investigatory Powers Bill, you'd be right to be P'd off.</p>
<p>Yesterday Parliament approved the second reading of the controversial legislation by 281-15, following abstentions by Labour and the SNP. Twitter was apoplectic. One of the most widely-spread images during the debate was the lack of attendance by Opposition parties during the second reading:</p>
<blockquote>
<p>Those empty benches, there? That's Her Majesty's Official Opposition during the reading of the #IPBill. Pathetic. pic.twitter.com/Og4dOuS1jO</p>
<ul>
<li><a href="https://twitter.com/richpollock/status/709774566931959808">@richpollock</a></li>
</ul>
</blockquote>
<p>Suffice to say this was not reported well:</p>
<ul>
<li><a href="http://www.theguardian.com/politics/2016/mar/14/lib-dems-attack-gutless-labour-surveillance-bill">Lib Dems attack "gutless" Labour over surveillance bill</a></li>
<li><a href="http://arstechnica.co.uk/tech-policy/2016/03/investigatory-powers-bill-labour-and-snp-to-abstain-from-parliamentary-vote/">MPs vote in favour of investigatory Powers bill after Labour, SNP abstain</a></li>
<li><a href="http://www.theregister.co.uk/2016/03/15/labour_abstain_snoopers_charter_vote_second_reading_investigatory_powers_bill/">Labour will create FUD and then abstain on UK Snooper's Charter Vote</a></li>
<li><a href="http://www.wired.co.uk/news/archive/2016-03/14/labour-surveillance-ip-bill-snoopers-charter">Labour to abstain from IP Bill over privacy concerns</a></li>
</ul>
<p>We've been here before. Back in July 2015, the Government tabled the controversal Welfare Reform and Work Bill in the middle of a brutal Labour leadership election. Labour attracted a lot of criticism for abstaining from the vote. What actually happened was that <a href="http://www.publicwhip.org.uk/division.php?date=2015-07-20&number=50">Labour proposed an amendment</a>, which had a high turnout, although not enough to defeat the the Government's majority. Acting Leader Harriet Harman's strategy was to then <a href="http://www.publicwhip.org.uk/division.php?date=2015-07-20&number=51">abstain from the second reading</a>.</p>
<p>We saw the <a href="http://www.jennihollis.org.uk/twitter_reacts_to_one_labour_mp_s_welfare_bill_abstention">same reaction</a> towards this strategy, indeed it allowed Jeremy Corbyn to position himself as the 'principled' candidate. It was instrumental in his resounding victory in the leadership contest. It didn't matter that Labour <a href="http://www.publicwhip.org.uk/division.php?date=2015-10-27&number=98">voted against the Bill en-masse during its third reading</a>, as at this point the damage was already done.</p>
<p>To understand why this happens, we need to look at how Bills are passed in Parliament.</p>
<img src="https://samhogy.co.uk/assets/commons-procedure.jpg" />
<p>When a Bill is introduced in the Commons, it goes through <a href="http://www.parliament.uk/about/how/laws/passage-bill/">several stages</a>:</p>
<ul>
<li><strong>The First Reading</strong> is the first stage of a Bill’s passage through the House of Commons - usually a formality, it takes place without debate.</li>
<li><strong>The Second Reading</strong> is the first opportunity for MPs to debate the main principles of the Bill.</li>
<li><strong>The Committee Stage</strong> is where detailed examination of the Bill takes place.</li>
<li><strong>The Report Stage</strong> gives MPs an opportunity, on the floor of the House, to consider further amendments (proposals for change) to a Bill which has been examined in committee.</li>
<li><strong>The Third Reading</strong> is the final chance for the Commons to debate the contents of a Bill.</li>
</ul>
<p>Following that it goes to the Lords, but we can stop here for now.</p>
<p>If you largely oppose a Bill and in all likelihood it will pass, then by abstaining at the second reading, all you are doing is ensuring that the Bill passes to the next stages of scrutiny. This is where the real powers of opposition come into force. Amendments can be tabled, debated and voted upon. You can exert better opposition by at least using the full process to water down the legislation. Should this not work, you can always vote to reject it at the Third Reading.</p>
<p>Of course, it's incredibly easy to spin this into something it's not. Smaller parties can take swipes at the larger parties for behaving differently. They can do this by playing on the public's general lack of understanding of the Parliamentary procedure to score points. If anything, this is apathy-inducing and not truly representative of reality. It makes for good headlines though.</p>
<p>Stella Creasey, who gave an excellent speech during the debate yesterday, clearly makes this point.</p>
<blockquote>
<p>Good to hear @andyburnhammp say the #IPbill not acceptable in its current form - this is 2nd reading so Ctte stage key to making change!</p>
<ul>
<li><a href="https://twitter.com/stellacreasy/status/709745301771436032">@stellacreasey</a></li>
</ul>
</blockquote>
<blockquote>
<p>So the SNP also abstaining on #IPBill in order to get into Ctte stage to see if can be amended - their spokesperson just said so…</p>
<ul>
<li><a href="https://twitter.com/stellacreasy/status/709757734120505346">@stellacreasey</a></li>
</ul>
</blockquote>
<p>So why didn't Labour and the SNP take the path of least resistance and vote to oppose at the early stage? The answer of course depends on the Bill in question, and your own definition of what Opposition is. Personally, opposition is not simply about block-voting 'no' in every case, it's also about ensuring that legislation that <em>does</em> pass is of a good quality. Many have pointed out that if Labour & SNP MPs had voted 'no' yesterday, then the Bill would have been blocked. Ambushes are always possible, but incredibly unlikely given the magnitude of this Bill.</p>
<p>The IP Bill is an extremely large Bill, with some good parts and some very bad parts. Overall I remain positive - opposition is still growing. A <a href="http://www.telegraph.co.uk/comment/letters/12194672/Letters-Voters-are-undecided-on-the-EU-because-they-dont-have-all-the-facts.html">letter to the Telegraph</a> from tech sector leaders begs for more Parliamentary time to fully understand the implications on the United Kingdom's digital economy. <a href="http://www.theguardian.com/world/2016/mar/14/investigatory-powers-bill-not-fit-for-purpose-say-200-senior-lawyers">200 senior lawyers described the bill as "not fit for purpose"</a>.</p>
<p>The Labour Party strategy has been effectively outlined by Keir Starmer:</p>
<blockquote>
<p>Surveillance legislation fit for the 21st century, which strikes the right balance between privacy, security and democracy is a prize worth fighting for, and Labour will work constructively with the government to achieve it. But for that to happen, the government needs to allow sufficient time for proper parliamentary scrutiny and, equally importantly, to shift position on a number of key issues. It can expect robust challenge from Labour where it fails to do so.</p>
<ul>
<li>http://www.theguardian.com/commentisfree/2016/mar/15/investigatory-powers-bill-labour-law</li>
</ul>
</blockquote>
<p>The Internet outrage machine, puts politics under the microscope. There are those who deem anything other than complete and total opposition as selling-out, and do very well by promoting extreme viewpoints. But it is also clear that it is highlighting some of the eccentricities about how our democracy works. Whether you think it's a good or a bad thing, you decide. But please make sure that you're in <a href="http://blogs.spectator.co.uk/2014/11/the-menace-of-memes-how-pictures-can-paint-a-thousand-lies/">full possession of the facts before believing that Twitter meme</a>.</p>
Should You Use Should In Tests?2016-03-15T00:00:00Zhttps://samhogy.co.uk/2016/03/should-you-use-should-in-tests/<p>It's a good question. To give the consultant's answer - it depends - but in most cases, I do not think it is particularly helpful.</p>
<p>No doubt you would have seen and perhaps written a test beginning with 'should':</p>
<p><code>it('should show placeholder text when user has not entered text', ...);</code></p>
<p>The use of should is a gripe for me, because every time I see it, my mind thinks "well, when should it <strong>not</strong> do that!". Nearly every time I question this during a review, eyebrows are raised. Am I being pedantic, petty, or just making sure that I have full a understanding of what I'm being asked to review? In most cases, the ambiguity was completely in my head. The code under test is always required to produce the behaviour that's being tested, under those conditions. But the use of 'should' made me doubt it.</p>
<p>What does the <a href="https://www.ietf.org/rfc/rfc2119.txt">RFC 2119 definition</a> of SHOULD have to say on the manner?:</p>
<blockquote>
<p>This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.</p>
</blockquote>
<p>The same document additionally defines MUST, so perhaps that is a suitable replacement?</p>
<blockquote>
<p>This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification.</p>
</blockquote>
<p>Well, it could work, as could SHALL. But can we do better?
What's the impact if we remove should altogether?</p>
<p>The first thing that happens is we need to change from a passive, to an <strong>active</strong> tone of voice:</p>
<p><code>it('shows placeholder text when user has not entered text', ...);</code></p>
<p>Not only have we saved characters, we've removed the ambiguity behind the test case. If the test fails, we know exactly why it's failing - the placeholder text failed to be rendered.</p>
<p>The convention of the 'should' prefix seems to originate with behaviour-driven-development, a notion I am a big fan of. In the past <a href="/2014/10/how-i-structure-unit-tests/">I've blogged about structuring tests</a>, such that they align with the behaviour of what's being tested. Tests communicate and demonstrate the intent of the code. Aim for tests to be unambiguous.</p>
<p>Too often I've seen hand-wringy, passive tests that are tightly-coupled to the particular implementation that they're meant to be testing. It's important to avoid that, else your tests end up testing that the code does what the code does, not what it is actually doing. You can break this trap by dropping 'should' from your test names. It forces you to <strong>write tests from the user's perspective</strong>. That's what truly matters here.</p>
<p>Plus the extra characters you save - I love me some horizontal space.</p>
MP Letter: Labour Fighting the IP Bill2016-03-05T00:00:00Zhttps://samhogy.co.uk/2016/03/mp-letter-labour-fighting-the-ip-bill/<p>I recently wrote to my MP (Alan Campbell, Lab, Tynemouth) regarding some of the concerns I have with the Draft Investigatory Powers Bill. Now that it is the Investigatory Powers Bill and working its way through Parliament, the time for action is upon us. My letter therefore focused on several key issues, rather than a more general request blanket opposition. My letter and its reply are published in full below.</p>
<hr>
<h2>Letter</h2>
<p>I would like to begin by wishing you a Happy New Year, in uncertain times for our Party. Today am I writing to express my concern at the Government's Draft Investigatory Powers Bill, both as a Tynemouth constituent and as a member of the North East's booming technology community.</p>
<p>The Draft IP Bill attempts to define the State's bulk and targeted data interception and acquisition abilities. This follows the expiration of existing legislation, a review of existing legislation from previous Governments, legal challenges at the State and EU levels over key pieces of legislation and public pressure following Edward Snowden releasing evidence of mass state and inter-state surveillance.</p>
<p>Clarification of existing powers is clearly needed. However, I remain unconvinced that the technical solutions outlined in this Bill are achievable, both technically and financially. The Government's are attempting to obfuscate several key issues. Additionally, the proposed 'double-lock' system of judicial oversight is half-formed at best.</p>
<p>As a Labour Party member, I believe effective opposition to this Bill will expose the myth that the Conservatives are strong on national security. We have an opportunity to present ourselves as the sensible and pragmatic party when it comes to national security issues.</p>
<p>I urge you to take action in the following ways:</p>
<p>(1) Ensure that the definition of an "Internet Connection Record" is outlined in Primary Legislation, such that it can be effectively scrutinsed by Parliament and relevant third-parties.</p>
<p>The current definition of an Internet Connection Record is incredibly unclear. The Home Secretary claims it is akin to an "itemised phone bill", i.e they will be able to know that you visited Facebook, but not which pages on Facebook one visits. But this fails to account for the fact that there will be items in your ICR log that you didn't visit, due to loading of third-party advertisments, or browsers pre-fetching link, or smartphone applications periodically "phoning home" to check for updates.
On a technical level, how this is to be implemented is uncertain. Different interpretations can vary the costs of compliance by an order of magnitude.</p>
<p>(2) Reduce the scope of mass surveillance powers in this Bill.</p>
<p>When a Government engages in mass, or 'bulk' interception, collection and retention of data, it is doing so on many individuals. This means that your personal information is being examined without you being suspected of committing a crime. It is an irrelevant distraction if this interception, collection, retention and analysis of data is automated or manual. Mass surveillance piles more hay onto an already insurmountable haystack. This is swamping our Intelligence Authorities and making them less effective at preventing crime and terrorism, rather than more effective.</p>
<p>(3) Remove attempts to weaken strong encryption.</p>
<p>Clause 189 obliges service providers to remove electronic protection that they apply to communications. This runs counter to the Home Secretary's claim that the Government support strong encryption. The strongest forms of encryption that we have do not require a service provider to hold a secret key - this is known as "end to end" encryption. Indeed, you do not need a service provider. But by inserting a clause obliging service providers to decrypt electronic protection, the end effect is to weaken the level of security in Britain, by forcing service providers to use weaker and breakable encryption technologie. Furthermore, this will only target law-abiding entities, whereas crimials will continue to use free and open source unbreakable encryption technology.</p>
<p>(4) Ensure that Judicial Oversight is based on a case's evidence, not just that the process has been followed correctly.</p>
<p>The Bill adds new powers requiring judicial oversight on any warrant issued. The Judge must verify the warrant process has been followed correctly. This is a great win for supporters of privacy and liberty. However, judges are not required to examine the evidence and independently assess whether the warrant is "necessary and proportionate". All they need to know is that the Home Secretary is confident the warrant is so. I hope you appreciate this is an easy process to subvert.</p>
<p>This Bill has been met with a strong coalition of opposition, including Liberty, the Open Rights Group, Apple, Google, Microsoft, Facebook, Yahoo and Twitter. I hope we in the Labour Party can join this coalition of opposition to ensure that this Bill does not weaken the security of the British public and businesses by enforcing historically-ineffective mass surveillance. It is necessary that, in the State's effort to ensure the security of its citizens by tackling crime and terrorism, it passes laws that are necessary, proportionate, are of sound technical nature and do not actively undermine the security of the British public. Sadly, this Bill does not meet such a test.</p>
<p>Thank you for your attention, I look forward to you response. I am happy to provide additional detail/clarifications, in person or via email, on any of the aforementioned issues.</p>
<hr>
<h2>Response</h2>
<p>Thank you for contacting me recently regarding the Government's draft Investigatory Powers Bill. I read your arguments with interest.</p>
<p>I appreciate that a number of organisations and campaigns, including the 'Don't Spy on Us' campaign, have expressed a number of concerns regarding proposed changes to surveillance and data retention laws.</p>
<p>Our intelligence and security services undertake vital, often unrecognised, work to protect our security and to counter the growing threats that we face, both internationally and domestically. I believe that the huge changes we have seen in technology have left our laws governing investigatory powers outdated, and it is important that the agenices and police have the appropriate up-to-date powers that they need to tackle terrorism, child sexual abuse, serious online crime, and to help locate missing people.</p>
<p>However, it is crucial that a new framework for providing these powers can command public trust by balancing strong powers with strong safeguards to protect privacy and long-held liberties.</p>
<p>There has been a great emphasis on the authorisation of interception warrants, which allow for the use of the most intrusive powers. I have long believed that the full legal authorisation of warrants by a judge would be a key test for this Bill, but unfortunately the Government's proposals for a 'double-lock' do not seem to match up to this.</p>
<p>I will therefore be pressuring the Government to amend the Bill so that it allows for a judge to review the evidence before choosing to authorise, or not, the approval of a warrant. This will provide the proper 'double-lock' that the Government has led us to believe they have provided.</p>
<p>I also believe that there need to be stronger safeguards on the powers in the draft Bill, including clearly defined thresholds justifying their use. This should include the police's access to people's internet connection records, and the Government should also consider limiting access to these records only to police officers of a specified seniority.</p>
<p>The draft Bill is now being considered by a Joint Committee of both Houses of Parliament, and it will of course be important to continue to scrutinise its measures in further detail. I hope the Government agrees to amend the Bill as it progresses through Parliament to build trust in this new legal framework.</p>
<p>Thank you once again for writing to me and for sharing your views. I can assure you that I will be following the debate around this issue and the progress of these proposals very carefully.</p>
<p>Yours sincerely</p>
<p>Alan Campbell MP
Tynemouth</p>
<hr>
<p>A willingness to campaign for full and proper Judicial Authorisation of warrants was one of my key goals with this letter. I'll be watching with anticipation to see how such an amendment is voted on. There's a chance that it could actually pass a vote.</p>
<p>Of course, my other points regarding encryption and Internet Connection Records were not addressed. This was to be expected - both points are highly technical in nature. However they're just as important in order to adequately specify the intent of the law. Now that this Bill is going through Parliament, more work is going to be needed here.</p>
<p>We're onto something with this one.</p>
The Apple and FBI Power Struggle2016-02-23T00:00:00Zhttps://samhogy.co.uk/2016/02/the-apple-and-fbi-power-struggle/<p>Below the surface of the Apple/FBI case is the next round in the perpetual struggle between government and corporate power. We need to talk about this. We're long overdue a good chat. As technology develops further, it changes what it means for a State to uphold its laws.</p>
<p>The FBI's requests are technically feasible. They have in their possession an iPhone 5c, discovered at the site of the San Bernadino shootings. It is a work-issued device, protected by a passcode, running iOS8. Having locked themselves out of the iCloud account through poor operation practice, the FBI are demanding that Apple now provide a custom iOS firmware modification. iPhones give you ten tries to guess the passcode before it completely wipes the device clean. The FBI want to try an unlimited amount of times without having to worry about the device erasing all the data. Apple are not being asked to create a 'back door', or to break their encryption. However they are being asked to considerably weaken the security of the device and they're not groovy with it.</p>
<p>The court order issued to Apple is backed by the 1789 All Writs Act. This is a rather fundamental piece of legislation without which US court orders are functionally useless. What matters most to the Apple/FBI case is a Supreme Court ruling from 1977: a court order must not place an "unnecessary burden" on compliance. Apple claim it is an unnecessary burden, the FBI claim it's not. Hence the deadlock.</p>
<p>Questions about whether the FBI actually need access to the device still exist. Apple actually helped the FBI retrieve a lot of information from previous iCloud backups - stuff that was not encrypted. Through Verizon, the FBI retrieved call and Internet metadata. They've been able to conclude there are no links to other terrorists organisations. What remains is of course unknown, but given what's already known, there's unlikely to be a hidden smoking gun. It's natural for a law enforcement agency to try to investigate every angle they have, so from their perspective they have a blank time frame that they want filled. A lack of cohesion can break a case. We'd expect the FBI to gather as much evidence as possible and cannot fault them for sticking to that goal.</p>
<p>Many who support the FBI with this case point out that Apple would be doing this as a one-off. There's no risk of the code leaking out, because it will simply be deleted once this specific case has been resolved. Apple have been accused of turning this into an all-or-nothing issue, or trying to engineer impunity for themselves. Likewise, the FBI have been accused of waiting for a case of particular political impact in order to make their claims for more powers. This may or may not be about setting a legal precedent in the United States law - reports certainly suggest the FBI have more iPhones they'd like access to. It's pretty clear that other countries will come knocking for access to this tool. Once a precedent is established, there will be expectations for similar tools for accessing other iDevices, including the more recent and secure iPhones.</p>
<p>This is the debate that now faces us. More and more of our lives are being stored online, housed in the data centers of private companies. There's a trust relationship - that our lieges will protect us from threats. Our mobile devices are the key to our lives. Break into those and you have everything you need to commit identity and financial fraud. The very protections we put in place to protect us if our phone is pickpocketed also impede law enforcement.</p>
<p>We cannot continue to pretend that it is possible to break security but only for the goodies and never the baddies. Such limitations are based in mathematics and not in software patents. People who do not understand the full implications will push for a compromise that simply cannot exist without causing the very things they want to protect us from. What is one person's compromise becomes another person's defect that can be exploited. Trust breaks down. What's left if we don't have trust? Apple have their global reputation at stake.</p>
<p>Apple/FBI exposes the Emperor's New Clothes for what it is - maybe governments do not wield as much power as they'd like to, or should do, despite the stories they like to tell us. When legislation has not kept up with reality, we face a large crisis of confidence in the democratic process. If we want to maintain a free, open and civil society we need to start discussing how we can practice effective law enforcement, with practical legislation for a truly digital age.</p>
<p><a href="https://storify.com/samhogy/apple-and-the-fbi">Further Reading collated with Storify</a></p>
The Insidious IP Bill2016-01-16T00:00:00Zhttps://samhogy.co.uk/2016/01/the-insidious-ip-bill/<p>The UK Government faced a lot of backlash as it mooted banning strong encryption. Then word started to spread around that this was not what David Cameron meant when he said that he will pass legislation "that does not allow terrorists to communicate with each other". Understandably, this has left everybody confused. So I decided to check the legislation.</p>
<p><strong>Note</strong> - When discussing specific legislative clauses, I use the following syntax: <strong>s189(4)(c)</strong> refers to section 189, subsection 4, paragraph c of the Draft Investigatory Powers Bill. Unfortunately it only currently exists in PDF form, so you'll have to scroll to the relevant sections yourself. Boo!</p>
<p>The <a href="https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/473770/Draft_Investigatory_Powers_Bill.pdf">Draft Investigatory Powers Bill</a> doesn't outright <em>ban</em> encryption, but it does certainly muddy the waters around the use of end-to-end encryption in the UK. s189 covers the "maintenance of technical capability". s189(4)(c) places:</p>
<blockquote>
<p>"obligations relating to the removal of electronic protection applied by a relevant operator to any communications or data".</p>
</blockquote>
<p>We have to dig a bit deeper to learn how to parse this. A 'relevant operator' can be</p>
<blockquote>
<p>"public postal services, or a telecommunications services".</p>
</blockquote>
<p>A 'telecommunications service' is:</p>
<blockquote>
<p>"any service that consists in the provision of access to, and of facilities for making use of, any telecommunication system (whether or not one provided by the person providing the service". - s193(11)</p>
</blockquote>
<p>If you provide a telecommunications system then you can be served with a notice under s189 obliging you to remove any encryption that your service adds to communications. Before the Secretary for State can issue a notice, they must assess both the technical feasibility and cost of complying with the notice as specified in s190(3). One would think that this would prevent a notice being served for the removal of strong encryption, but if it doesn't, it is possible to invoke the appeals process under s191. The Secretary for State may then consult both the Technical Advisory Board and Investigatory Powers Commissioner during appeals - who again will consider both the technical requirements and financial consequences of complying with a notice. During said review, you will be also be able to provide evidence to the Board and Commissioner, before the Secretary of State makes a decision to withdraw the notice.</p>
<h2>So what does this mean?</h2>
<p><a href="http://parliamentlive.tv/Event/Index/796f0216-13f6-4f85-9ed1-61e92c735f9f">Theresa May provided evidence to the Joint Committee on the Draft Investigatory Powers Bill</a>, in which she stated (17:07:00 onwards) that the Government are believers in strong encryption. This Bill is simply repeating former legislation, albeit turning some secondary legislation into primary legislation. Therefore, the legal changes introduced by this Bill with regards to encryption are nil. The Home Office doesn't care what type of encryption your service uses. They're not interested in setting up a key escrow service. All they care about is that if they serve a s189 notice, you better be able to remove encryption.</p>
<p>Of course you have the chance to appeal. If you're providing an end-to-end encrypted service, it is unclear if the cost of switching to a less secure encryption method will be defined as a reasonable financial cost of complying. s185 and s186 cover payments towards certain compliance costs, or developing systems enabling compliance, offering grants, loans, investments. Unfortunately, it does not provide financial cover for complying with <em>technical capability notices</em>. So in the current form, you're on your own. There's hope that the Technical Advisory Board (as defined in s183) will provide a pragmatic approach, given it is to include parties who would have obligations under this Bill.</p>
<p>I find this logic quite insidious. Any sensible compliance department will err on the side of caution and press to ensure that any encryption that their service provides can be removed, just in case they are served with a s189 notice. Some big names whose headquarters are outside of the UK might claim this law doesn't apply to them and some smaller UK services may not bother, silently hoping that the UK government never comes knocking. But on the whole, such legislation doesn't outright ban end-to-end encryption, it just makes it incredibly risky for a service to implement. In effect, they are spreading the culture of fear onto telecommunications service providers, whilst legally coercing them to weakening encryption for everyday users.</p>
<p>Finally, as I explained in my <a href="/2015/11/the-realities-of-encryption/">previous post on encryption</a>, you're still engaging in a game of legal whack-a-mole, chasing new services as they pop up. This is a bit of a mess.</p>
Newcastle Skeptics 20152015-12-13T00:00:00Zhttps://samhogy.co.uk/2015/12/newcastle-skeptics-2015/<p>You may know that in 2014 I took a role as co-organizer of <a href="http://www.meetup.com/Newcastle-Skeptics-in-the-Pub/">Newcastle Skeptics in the Pub</a>. Will and I inherited a well-organised group from Danny, Richard and Emma - all whom unfortunately had to resign at the same time. Since then we've recruited Christine to round off the team.</p>
<p>We have bigger plans for next year, but first, a look into this year. The general take-home points are that our group size has doubled (in terms of regular attendees), <a href="https://soundcloud.com/sitp-ncl">we've launched a podcast</a>, forged a good working relationship with the best venue in Newcastle and pushed the boat out in terms of the topics we've addressed.</p>
<p>January began with an excellent talk on a difficult subject by Dr Margaret McCartney. Talking about death is tough, but it's true that we are living longer. Dr McCartney discussed palliative and social healthcare, aimed at improving quality of life during end-of-life treatment.</p>
<p>Dr. Richard Shelley took on the banking system in February, back by popular demand from 2014's Open Mic Night. Richard was speaking on behalf of Positive Money, a group whose intentions are to increase fairness in the banking system - a goal they believe is achieved through education. We found out how money is made and how crashes happen. Alternatives to quantitative easing were covered, such as investment in public infrastructure.</p>
<p>Rebecca O'Neill, a founder of Dublin Skeptics in the Group, confessed her experiences as a shopworker in a famous health food shop. She gave the other side of the coin to some of the claims that are made about miracle cures/supplements, and discussed why alternative therapy pervades in the public consciousness.</p>
<p>We'd worked for several months behind the scenes to organise a General Election debate in April. We recruited Dr. Nick Randall, Head of Politics at Newcastle University (on the recommendation of my partner) to chair a debate between candidates for Labour, the Conservatives, the Liberal Democrats, the Green Party and UKIP. Alongside our participants was the well-respected Labour MP for Newcastle Central, Chi Onwurah. We had over 100 attendees and started regularly hosting Skeptics at the Tyneside Irish Centre. This allowed us to begin podcasting our events too.</p>
<iframe width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/96919421&color=ff5500&auto_play=false&hide_related=false&show_comments=false&show_user=true&show_reposts=false"></iframe>
<p>In May our Edinburgh Fringe season began, kicking off with Ash Pryce (from Edinburgh Skeptics fame) and his popular show 'How to Talk to the Dead'. Ash has been touring the country with this show. It delighted a record audience size with a skeptical look back to Victorian spiritualism, but also a healthy dose of entertaining magic tricks. Look out for this in a town near you.</p>
<p>Lady Luck was on our side in June. We were able to nab Suw Charman-Anderson, a founder of the Open Rights Group and of Ada Lovelace Day, while she was over in the UK. I recall learning about Ada Lovelace back in my very first Computer Science lecture at Newcastle University and thinking how underappreciated she is. Suw sees Ada Lovelace as a powerful figurehead for women in STEM. Her talk provided a historical look into the life of Ada, and why she set up Ada Lovelace day.</p>
<iframe width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/210034077&color=ff5500&auto_play=false&hide_related=false&show_comments=false&show_user=true&show_reposts=false"></iframe>
<p>Our second Edinburgh Fringe show featured Chris Coltrane, a standup comedian I've been a fan of since 2009, but had unfortunately never seen live. Chris has a history with the skeptics movement, having previously overdosed on homeopathy tablets during the 10:23 campaign. His show brought joy to us during a very dark time. If you're in London, check out Lolitics. Otherwise,
<a href="http://www.chriscoltrane.co.uk/2015---left-wing-propaganda-machine.html">you can view the show here.</a></p>
<p>Another coup for us in August, as we recruited James Welch, the Legal Director of Liberty. James demolished the new Government's alarming proposition to scrap the Human Rights Act and replace it with a British Bill of Rights. Although this is not something happening in the 2015/16 Parliamentary Period, it's a subject that has historically been abused within the press. James' talk was a fantastic antidote.</p>
<iframe width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/219103492&color=ff5500&auto_play=false&hide_related=false&show_comments=false&show_user=true&show_reposts=false"></iframe>
<p>September marked our Fourth annual Open Mic Night - one which proved to be remarkably existential. We discussed whether Intelligent Design is a Science, whether mind and matter are on different planes of reality, the difference between dieticians and nutritionists (there is a <strong>big</strong> difference!) and how our own cognitive dissonance affects our skeptical abilities.</p>
<p>Another alternative month for us in October! We invited Margaret Corvid, a New Statesman contributor and professional dominatrix. The UK passed a set of misguided, ill-informed 'porn panic' laws in 2014, whilst enjoying the box office hit Fifty Shades of Grey. Margaret's talk mythbusted public perception of BDSM and the nature of consent. I am absolutely gutted we could not get an audio recording of this talk - the only downside to an utterly brilliant event.</p>
<p>In November we returned the the Bridge Hotel and invited urban explorer Dr Bradley Garrett to talk about trespass in the UK. Brad has climbed the shared, explored the hidden stations of the London Underground and provided an account of his travels with startling and stunning photography. His talk was philosophical, focusing on the history of trespass and the future of our cities.</p>
<iframe width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/232729533&color=ff5500&auto_play=false&hide_related=false&show_comments=false&show_user=true&show_reposts=false"></iframe>
<p>To round off the year, we always host a religious talk. This year we invited the founder of Faith to Faithless, Aliyah Saleem, to discuss her experiences of leaving Islam. Faith to Faithless are a support group for those who wish to leave a faith. Her talk focused on the implications of doing so, such as impact in the community and family life. I was particularly impressed at how Aliyah provided an extremely broad, yet empthatic, view of the topic.</p>
<iframe width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/236932259&color=ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false"></iframe>
<p>I've really enjoyed organising Skeptics in the Pub alongside Will and Christine. We set ourselves a go of having a 50/50 gender split in our talks, thinking this would be incredibly difficult. The Skeptics circuit can be quite white, male and middle-class at times. Turns out, it's not so hard after all. As mentioned above, we have big plans for 2016, some of which I'll elaborate on in the coming weeks. <a href="http://www.meetup.com/Newcastle-Skeptics-in-the-Pub/">I hope you come along for the ride!</a></p>
The Realities of Encryption2015-11-21T00:00:00Zhttps://samhogy.co.uk/2015/11/the-realities-of-encryption/<p>You can send someone an encrypted message, without sharing a secret such as a password in advance. The encryption is effectively unbreakable by today's technology when used correctly.</p>
<p><a href="https://gnupg.org">The software libraries to do this</a> are out there, for free. They are not centrally distributed, via an 'app store' under the control of an overlord like Apple, Google or Microsoft. The source code can be obtained and inspected to ensure it has not been meddled with. If it has, you can modify, build and distribute a new version.</p>
<p><a href="https://otr.cypherpunks.ca/">There is software used for chats</a> that abstract much of this process from the end user. All they need to know is that their chat is encrypted. Furthermore, the encryption periodically changes, so that if one communication exchange is broken, it does not grant the attacker access to all their historic and future communications.</p>
<p>Encryption has been around for centuries. Simple ciphers used in playgrounds, such as A=X, B=Y, C=Z are encryption algorithms. Fundamentally, it's just mathematics - technology simply makes it fast. The popular encryption algorithms out there today are not trade secrets, or proprietary. It's published work, for all to use. Technology itself is not inherently good or evil. Is a hammer good or evil? It can hammer a nail, or cave somebody's face in. Encryption is just another technology, used both for good and evil. Not everyone who encrypts has malicious intent.</p>
<p>Where do we want encryption to be used? We would not want our credit card details, health records or bank statements to be visible to anybody who can <a href="https://en.wikipedia.org/wiki/Firesheep">install a web browser plugin</a>. Encryption is a vital piece of our digital information infrastructure. Compromising how encryption works compromises the integrity of online business. How could we have confidence in online shopping if it became trivial for an attacker to redirect your purchases to a different address, or to spy on your credit card details? HTTPS, an encrypted transport protocol, enables us to have confidence that nobody can tamper with messages in transit and trust the authenticity of the websites we visit. We rely on encryption to protect us, keep our secrets confidential and our digital economy secure.</p>
<p>Attempts to capitalise on the recent atrocities in Paris have focused on weakening encryption, even though <a href="https://theintercept.com/2015/11/18/signs-point-to-unencrypted-communications-between-terror-suspects/">current reports indicate the attacks were arranged over SMS</a>, an entirely unencrypted medium. With 'backdoors', the intention is for a government to be able to decrypt communications.</p>
<p>Unfortunately, this is futile. A government may be able to coerce particular companies into weakening the encryption they use. The effect will be to stop determined bad guys only from using that service. Banning a free and open source software library will start a game of whack-a-mole, as there's no central distribution site to serve notice on. As the code is open for anyone to copy and modify, backdoors can be removed. Stronger encryption algorithms can be added to the libraries' suite. Backdoors are for everybody, not only for friendly folks. They may remain secret, but people actively hunt for them. <a href="https://en.wikipedia.org/wiki/Dual_EC_DRBG">A suspected backdoor</a> within the DUAL_EC_DRBG encryption algorithm was recently found in the OpenSSL encryption suite. Even if you were to ban an encryption algorithm, there are plenty of others to be used. There's also the point that bad guys don't follow the law.</p>
<p>The benefits that we truly need in a digital age allow for enemies to communicate with total secrecy. Law and order types might feel this is a bleak conclusion. Their attempts to reassert control will be ineffective against their true targets, but would harm innocents by eroding basic trust. This is simply the reality in which we live online.</p>
Destroying Monoliths at DDDNorth2015-10-27T00:00:00Zhttps://samhogy.co.uk/2015/10/destroying-monoliths-at-dddnorth/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2015/10/27/destroying-monoliths-at-dddnorth.html">the Scott Logic blog</a></span></div>
<p>Saturday marked the fifth annual DDD North conference, held once again at the University of Sunderland. DeveloperDeveloperDeveloper! events are volunteer-organised conferences where members of the community propose and vote on the agenda. ScottLogic took part this year as a Platinum-level sponsor. We had many fantastic conversations with developers from all across the country, plus our squishy rugby ball giveaways were very popular!</p>
<img src="https://samhogy.co.uk/ssets/dddnorth.png" alt="DDDNorth logo" width="250" height="188" />
<p>Luckily, I managed to slack off from our recruitment stand and attend some of the great talks. As always I like to write up my impressions and pay compliments to the speakers. Overall this is one of the most diverse lineups that I've seen at a DDD event. This year's prominent topic was microservice architectures, that have been rapidly taking the developer world by storm. But there were also talks on sketchnoting, fun things to do with Raspberry Pi 2s, Deep Learning, DevOps and Usability. As the agenda is decided upon by community votes, it's always interesting to see which of the many competing ideas out there stand out from the crowd. Hopefully in the future we'll see fewer monolithic applications. Anyways, onto the talks:</p>
<h2>How I became Less Terrible at JavaScript - Steve Higgs</h2>
<p>At the time that I moved from C# into JavaScript I felt underwhelmed at the lack of basic tooling support baked into the language. Over time I started to love that as it has allowed JavaScript to prosper with innovation and experimentation. Nowadays I feel overwhelmed at the variety! But once you realise that there are key steps that you need to perform in a JavaScript toolchain, your choices become manageable.</p>
<p>In this talk, Steve introduced a modern-day JavaScript build toolchain. Webpack was used to facilitate modular JavaScript development, ESLint for style and error checking, Babel for transpilation so that we can utilise the language benefits of ES2015, and Gulp as a build tool to drive the entire process. Phew! The focus here wasn't on how to use this particular set of tools (although you did pick that up through osmosis), it was more about what each tool was doing to your code and why that is useful. Alternatives such as Browserify, Grunt and JSLint were mentioned to demonstrate that in JavaScript, there is no One True Path. Steve even touched on the controversial topic of CSS within JavaScript, something we'll see more of with the trend towards isolated front-end components.</p>
<p>JavaScript is such a rapidly-evolving environment that even experienced developers will learn something new by exploring the bleeding edge.</p>
<h2>Why SOA? - Sean Farmar</h2>
<p>Sean reminded us of the big ball of mud. It's what code becomes if not regularly curated, pruned and honed. It's a product of team scaling, product growth and feature creep, under release pressures. Coupling is why this happens, but not all coupling is equal. There are many types of coupling:</p>
<ul>
<li>What you depend on (efferent)</li>
<li>What your dependents are (afferent)</li>
<li>Time (temporal)</li>
<li>Deployments and endpoint locations (spatial)</li>
<li>Protocols and platforms</li>
</ul>
<p>You're not going to remove all types of coupling, but Sean covered ways of minimizing this coupling. For example, with asynchronous messaging, publish/subscribe, not hard-coding endpoints and using a "share contract and schema" mantra. Following from this Sean busted several distributed computing myths and assumptions. Your network is not secure, bandwidth may not be a problem at the moment, but when you scale it will be. For each myth, Sean described patterns for mitigation.</p>
<h2>Monoliths to Microservices: A Journey - Sam Elamin</h2>
<p>A nice complementary talk to Sean's SOA talk. Sam's focus wasn't to present another microservices 101 talk. Instead this was a de-brief on an effort to decouple a monolithic application and what was discovered along the way.</p>
<p>Sam mentioned that all developer conferences should have a failure track. As an industry, we're unique in our ability to fail fast and learn from our mistakes. This is a great suggestion, hindsight is 2020 and there is an opportunity to learn a lot. Sam gave pragmatic advice on how to get started with microservices, if you're already living in a monolithic codebase. Don't bother re-writing everything now. Instead, the next time you need a new feature, microservice it! Over time your code will gradually migrate to a more distributed architecture. Sam's team now have 290 microservices!</p>
<p>The example that Sam used was adding a new payment provider (let's say PayPal) to an existing application. Along the way, Sam discussed many microservice patterns mentioned earlier in Sean's talk, such as the use of pubsub and asynchronous communication and keeping your service contracts small. Sam also impressed the audience with his team's bespoke error handling system, which tracked failure rates and if a threshold was hit, would terminate a service and flag the on-call developers of an issue.</p>
<h2>React for .NET Developers - Macs Dickinson</h2>
<p>React is currently my favourite JavaScript view engine. It's the first since Knockout that has felt right to me. React is fast but in my opinion its USP is how it encourages you to think about data flow through your application. Consider the view layer of your application as a functional transformation of the model layer, with a tiny bit of view-specific state. React is good at solving just that specific problem, rather than trying to do everything, poorly.</p>
<p>Macs' talk aimed to introduce React to .NET developers, most likely for use within the client-side of their ASP.NET applications. Macs justified this by looking into ASP's long history and seeing an increasing demand for rich, interactive client-side functionality, with fast load times. Admittedly, a trend not unique to .NET. Macs provided a comprehensive introduction to React, which you can do because its functionality is so incredibly specific. From creating and rendering a React component, he introduced the scary-looking-but-totally-not-scary JSX, looked at React component properties and internal state. What most impressed me was Macs' demonstration of how fast React can be, by rendering thousands of numbers (and applying CSS styles) on-screen and changing them at customisable intervals.</p>
<p>Macs showed how well React works in Visual Studio. It's nice! I enjoyed learning about <a href="http://reactjs.net/">ReactJS.NET</a> - a tool for server-side JSX compilation in ASP.NET MVC, so that applications can benefit from server-side rendering. Hooray SEO! Finally, TypeScript 1.6 supports JSX. Macs showed that React and TypeScript in Visual Studio is a pleasant setup. A great talk to end the day.</p>
<h2>Summary</h2>
<p>For the fifth year running, DDD North has been a massive success, reaching record attendance. It is now claimed to the be the largest conference of its type in the country. Such success is down to the organisers (thanks, Andy, Linda and co!) and speakers, all of whom donate their time. They are driven by a passion to build a thriving community focused on education and integration. We all have a million things to be doing on a weekend, but if possible, I'd highly recommend popping along to next year's event. I guarantee you'll learn something new and make new friends.</p>
The Darkest Timeline2015-05-10T00:00:00Zhttps://samhogy.co.uk/2015/05/the-darkest-timeline/<p>Well, this is depressing.</p>
<img src="https://samhogy.co.uk/assets/evilSam.jpg" alt="Evil Sam" />
<p>Billed as the most unpredictable election of modern times, it certainly produced an unpredictable result. The results were expected around the negotiating table, not the ballot box. In the end the voters returned the first Conservative majority in two decades.</p>
<p>It's a slim majority, but functional. The largest transfer of wealth to the privileged few, sorry, the long-term economic plan, can continue. Indeed, no longer encumbered by the Liberal Democrats, it can be accelerated.</p>
<p>I want to talk about how the pollsters got it wrong, how the media and Conservative party spread fear throughout the electorate, how Labour's message failed, how just under 2/3rds of the electorate didn't vote for the party that won a majority. I want to talk about this, but right now I can't find the energy. There's so much hurt and there's more to come. A close friend's disabled father burst into tears when he saw the news. On social media I've seen people publicly worry about how they are going to pay the bills over the next five years - and these are people in full-time employment. I've witnessed family members struggle to get the help they were entitled to, after they suffered from severe medical conditions. I've seen those entitlements rolled back. You're only one step away from this: one accident, one genetic timebomb.</p>
<p>We're in for a tough five years. Elections are games; real life is not. Lives are at stake; <a href="http://www.independent.co.uk/voices/comment/scrap-the-human-rights-act-and-keep-ttip-heres-what-you-voted-for-and-will-get-with-a-tory-government-britain-10235374.html">your human rights are in the firing line</a>. As I write this, there are <a href="http://www.theguardian.com/uk-news/2015/may/09/anti-austerity-protesters-take-to-uk-streets-after-tory-election-victory">protests on the streets of the capital</a>. Politics isn't about turning up to the polling station and ticking a box every half-decade. It's about standing up for your beliefs every day. This truly is the <a href="http://community-sitcom.wikia.com/wiki/Darkest_Timeline">Darkest timeline</a>. <a href="https://dogmic.files.wordpress.com/2011/11/dark-timeline.jpg">Print out your Evil Abed beard</a> and hold your loved ones extra close tonight.</p>
Parting of the Ways2015-04-27T00:00:00Zhttps://samhogy.co.uk/2015/04/parting-of-the-ways/<p>They say confession heals the soul, but I feel no better by admitting that in 2010 I voted Liberal Democrat. Back in the previous election I concluded that after 13 years of Government it was time for a change. Having principles, I'd never bring myself to vote Conservative and so I turned elsewhere in protest. I felt a mix of conflicting emotions following the formation of the Conservative/LibDem coalition. Firstly, pride. The party I voted for ended up in power! Following that came a deep, lingering sense of betrayal - one which would grow into disappointment, frustration and anger.</p>
<p>To list all the faults with the Coalition is a blog post in itself. In short, they have dismantled the NHS. They have starved the poorest in our society of financial support, with record numbers now relying on food banks, forced the unemployed to take part in unpaid labour and cult-like behavioural counselling, forced the sick and disabled to be examined and demeaned by non-medically-trained, corrupt, outsourcing companies. They violently suppressed protest, creating a more draconian state by rushing through data retention bills under false "emergency" situations. They have oversaw, indeed facilitated, one of the largest transfers of wealth, from poor to rich, in history. Neither the Conservatives or Liberal Democrats are fit to govern for a second term.</p>
<p>Their actions hurt. The Lib Dems didn't win where I voted, but in some way that vote counted. In some way they were doing what they were doing in my name. My family have suffered greatly under this Government as a direct result of their policies. What pushed me through was the thought that Labour would get back in next time and start to correct the excesses of this Government. Under Ed Miliband, the New Labour cohort have been de-prioritised and causes such as the abolition of the bedroom tax, a re-commitment to a publicly funded NHS and a crackdown on tax avoidance have been adopted as core principles of a future Labour government. These ideas came not from Labour, but from protestors on the streets. Don't let anybody tell you that protest doesn't work. For this, Labour have been punished in the media. Once again they were in the pockets of the Union Barons, with Neil Kinnock level floundering and no credible alternatives. Labour have been squeezed out of the debate as the Overton Window was dragged rightwards. Remarkably, the public are considerably to the left of Labour when it comes to transport, energy and economic policy, leaving many dissatisfied with the supposed Prime Minsiter in waiting.</p>
<p>Labour are neck and neck in the polls with the Conservatives. To some degree this always happens around election time, but the key question remains - why aren't Labour way out ahead? After five years of pain and damage, how come a Labour majority isn't effectively guaranteed? The simple answer is the SNP. If it weren't for the SNP this would be an open and shut election - instead for almost every Labour gain, they'll lose a Scottish seat to the SNP.</p>
<p>It is almost a certainty that a second election in a row will produce a Parliament with no overall control. It can no longer be said that First Past the Post
produces "strong, stable governments", as the No2AV campaign argued. It will be 23 years since the Conservatives won an overall majority at an election and a decade for Labour. Multi-party governments in some form or another, are here to stay. Minor parties such as UKIP and the Greens (and to some degree the LibDems) will continue to claim their vote share is not reflected by their seat share. Their arguments will increasingly ring true. Despite the public rejecting the Alternate Vote, we may still get a better voting system. This time, by necessity, not convenience.</p>
<p>It is truly a parting of the ways. Conservative and Labour dominance is now longer absolute, it comes with conditions and minor party support. But the key question still remains - which of the two will gain the most amount of seats? The Conservative "coalition of chaos" narrative has barely shifted the polls. Major policy announcements have had negligible impact. David Cameron announcing he won't serve a third term, followed by criticisms of the Conservative's "dirty" campaigning and rumors that his heart isn't in the game have stirred dissent within the Tory rank and file. The Game of Thrones begin. Cameron's barely holding on.</p>
<p>This time around, I'm supporting Labour. The deck has been stacked in favour of the rich and powerful. It's time to re-balance. Labour seem genuinely interested in doing so, but they're going to need external influence to ensure this happens. A Labour government with Ed Miliband as Prime Minister can begin to pull the Overton Window leftwards, introducing new ideas to the electorate in a gradual manner. It's time to put the pressure on and campaign for the heart and soul of the nation.</p>
New Year Update2015-01-19T00:00:00Zhttps://samhogy.co.uk/2015/01/new-year-update/<p>I've kept my head down over December, so sorry for that! Things are a bit up in the air at the moment, so I have a few updates to report:</p>
<h2>New Year, New Job</h2>
<p>I've spent just under a year at Nonlinear Dynamics, but in February I'll be moving back to Scott Logic to take up a role that I'm rather excited about. It's going to push me slightly out my comfort zone too. I've spent a few weeks re-acquainting myself with Knockout JS and Angular JS. Your mileage may vary, but I've always found I learn better by building things. My partner has asked me to make her an application for use in her classroom, so I'm writing it in Knockout and then I'll convert it to Angular JS.</p>
<h2>Newcastle Skeptics in the Pub</h2>
<p>Will and I are aiming to go big with Skeptics this year. We have a goal to drive up average attendance to 50 people, which is a roughly 30% increase on current average. With that in mind, we're in the process of confirming the remaining events of the first half of the year. The latter half of the year is in early planning stages too. So far: later-life health care, how the financial system works, secrets of working in a health food shop, (hopefully) a debate of North-Eastern General Election Candidates and how to speak to the dead!</p>
<h2>By-Election Game Development</h2>
<p>At the beginning of December, I made a change to By-Election that completely broke the game. Actually, that's an overstatement, it was just a bit unbalanced. But it did make me consider what direction I'd like the game to go in. There are rough ideas in my head for the next version, which hopefully will be prototyped by the time of the next Newcastle Playtest meetup. Right now it's looking like the game will become more focused on the interaction between players. I have no qualms turning it into a "one-shot" RPG if it seems like it's heading that way.</p>
<h2>Running</h2>
<p>Last September I picked up an injury (plantar fascitis). It's still nagging me but I'm starting to get back into the running saddle. Sadly I'm unable to run very far or very fast. Some of this is psychological, I don't want to re-injure myself by over-doing it. That's what caused this injury in the first place! So I'm sticking to about 2 miles at 9 min/mile pace, then sowly build back up to 5k, then 5 miles. Once I'm there, I'll focus on improving speed. I think this means I'll not be doing the Great North Run this year. It's a shame I can't do three in a row, but the risk of over-training from a sedentary four months to 13 miles at 8 min/mile pace is too great.</p>
<p>This is already to be shaping up to be a busy year!</p>
How I Structure Unit Tests2014-10-15T00:00:00Zhttps://samhogy.co.uk/2014/10/how-i-structure-unit-tests/<p>I've been asked how I structure my unit tests. As I think the style that I've converged on through practice is a bit unusual in the .NET world, I thought my answer warranted a blog post.</p>
<p>My philosophy is that tests are a form of living documentation of your code. If I need to learn about some code, the tests are the first place I go to. I want them to be simple, communicative and representative of real-world usage. When I'm thinking about a chunk of code that I want to test, I want to ensure that the tests describe the intent of the code, without being tightly coupled to a particular implementation. I struggled with this notion literally for years - how can you do that?!</p>
<p>Let's dive into an example. At the moment I'm building an event cache for a view model, which is hooked up to an underlying event source, like a web service. When the view model is on-screen, it simply forwards events from the source. When the view model is off screen and an event is produced, the cache remembers that an event is produced and will forward it as soon as the view model is re-shown.</p>
<p>So how do we communicate the intent? We consider the thing under test as a black box and test its public API. This keeps our tests representative of real-world usage, but also limits our knowledge to what is publicly exposed. This means that we test by controlling inputs and making assertions about the outputs. This translates, in English, to:</p>
<ul>
<li>When source fires event and view model is activated
<ul>
<li>Cache forwards event immediately</li>
</ul>
</li>
<li>When source fires and view model is deactivated
<ul>
<li>Cache does not forward event
<ul>
<li>And view model is then reactivated
<ul>
<li>Cache forwards event produced while deactivated immediately</li>
<li>Cache forwards further events from source immediately</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li>When multiple events produced from source with a deactivated view model
<ul>
<li>Cache does not forward event
<ul>
<li>And view model is then reactivated
<ul>
<li>Cache forwards most recent event produced while deactivated immediately</li>
<li>Cache forwards further events from source immediately</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>What's appealing about this is that this reads like a specification document. We are using scenarios as a method of grouping pieces of behaviour. This can be directly translated into a suite of unit tests! The mapping is simple - a scenario becomes a test class, a behaviour becomes a test method. NUnit and MSTest both support the notion that a public inner class, annotated appropriately, will be picked up by the test runner and executed.</p>
<p>So in code this will look like:</p>
<pre class="language-cs"><code class="language-cs"><span class="highlight-line"><span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestFixture</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">CacheWhenDeactivatedSpec</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestFixture</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">When_Source_Fires_Event_And_View_Model_Is_Activated</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Forwards_Event_Immediately</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestFixture</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">When_Source_Fires_Event_And_View_Model_Is_Deactivated</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Does_Not_Forward_Event</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestFixture</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">And_View_Model_Is_Then_Reactivated</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Forwards_Event_Produced_While_Deactivated_Immediately</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Forwards_Further_Events_From_Source_Immediately</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestFixture</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">When_Source_Fires_Multiple_Events_And_View_Model_Is_Deactivated</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Does_Not_Forward_Event</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestFixture</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">And_View_Model_Is_Then_Reactivated</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Forwards_Most_Recent_Event_Produced_While_Deactivated_Immediately</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">[</span><span class="token attribute"><span class="token class-name">TestMethod</span></span><span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token return-type class-name"><span class="token keyword">void</span></span> <span class="token function">Cache_Forwards_Further_Events_From_Source_Immediately</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token range operator">..</span><span class="token punctuation">.</span> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The first thing I do when refactoring legacy code is write a new set of tests in this style. Every time I've done this I've noticed the original test code was missing test cases. If you're writing your tests according to the typical <code>MethodUnderTestScenario_Expectation</code> style, it's very easy to forget a test case. Perhaps this is because you're grouping tests by <code>MethodUnderTest</code>, not by scenario. In the style of testing described above, you have the ability to use the collapse class ability of your IDE to hide away scenarios that you don't care about at the moment. Indeed, folding everything down to method declaration gives you, with a bit of C# syntax, <em>exactly the same bulleted list as I described above!</em></p>
<p>Don't just rush into your enterprise codebase and start making this change. I've seen plenty of hand-wringing over this method in Java and .NET land, although it seems pretty common in languages like Ruby because of RSpec. That said, there are .NET tools like SpecFlow which try and follow this line of thinking, but presented in a slightly different way. Try it, your mileage may vary, but it works well for me.</p>
Great North Run 20142014-09-10T00:00:00Zhttps://samhogy.co.uk/2014/09/great-north-run-2014/<p>I decided to enter this year's Great North Run after completing the course last year, just shy of my target time of 1hr 45. I wasn't expecting to be selected in the ballot, but I was! This year's event was last Sunday, so this post is a quick debrief of the event.</p>
<img src="https://samhogy.co.uk/assets/samGNRFinish2014.jpg" title="Look I finished!" width="200" height="302" />
<p>Let's start with a data-dump from my Garmin Forerunner 410 for both races:</p>
<h2>Overall results</h2>
<table>
<thead>
<tr>
<th>Category</th>
<th>2013</th>
<th>2014</th>
</tr>
</thead>
<tbody>
<tr>
<td>Time</td>
<td>1:46:35</td>
<td>1:45:45</td>
</tr>
<tr>
<td>Avg Speed</td>
<td>7.4mph</td>
<td>7.5mph</td>
</tr>
<tr>
<td>Avg Pace</td>
<td>8:06 min/mi</td>
<td>8:00 min/mi</td>
</tr>
</tbody>
</table>
<h2>Splits</h2>
<table>
<tr>
<th>Split</th>
<th>2013</th>
<th>2014</th>
</tr>
<tr>
<td>1</td>
<td>7:52</td>
<td>7:42</td>
</tr>
<tr>
<td>2</td>
<td>8:38</td>
<td>7:41</td>
</tr>
<tr>
<td>3</td>
<td>7:54</td>
<td>7:50</td>
</tr>
<tr>
<td>4</td>
<td>8:17</td>
<td>8:02</td>
</tr>
<tr>
<td>5</td>
<td>8:05</td>
<td>8:09</td>
</tr>
<tr>
<td>6</td>
<td>7:51</td>
<td>7:59</td>
</tr>
<tr>
<td>7</td>
<td>7:47</td>
<td>7:48</td>
</tr>
<tr>
<td>8</td>
<td>7:59</td>
<td>7:53</td>
</tr>
<tr>
<td>9</td>
<td>8:21</td>
<td>8:18</td>
</tr>
<tr>
<td>10</td>
<td>8:04</td>
<td>8:15</td>
</tr>
<tr>
<td>11</td>
<td>8:26</td>
<td>8:25</td>
</tr>
<tr>
<td>12</td>
<td>8:23</td>
<td>8:27</td>
</tr>
<tr>
<td>13</td>
<td>8:04</td>
<td>7:59</td>
</tr>
<tr>
<td>14</td>
<td>7:34</td>
<td>6:52</td>
</tr>
</table>
<h2>Commentary</h2>
<p>Overall I ran faster and shaved 30 seconds from last year's time. I aimed to run at an average pace of 8 minutes per mile, and hit exactly that. Having a bit of knowledge of the course topology from the previous year helped out quite a lot, especially because it was so hot! I was doing my best to hide in the shade wherever possible, which normally meant darting across the road onto a path obscured by trees. That actually meant I ran a further 0.07 of a mile further than last year! It did mean that I knew where to hold back and reserve energy and where to go for it. I think this accounts for most of the laps where I was slower in 2014 than 2013.</p>
<p>After reaching the 12 miles point last year, I was virtually out of gas. This year, by holding back just a bit, I had enough energy to pick up the pace and actually finish with a sprint! I was really happy about being able to do that!</p>
<p>1:45:45 is still 45 seconds shy of my target time, but given the heat I'm willing to consider this year's event a success. In training runs on a roughly comparable course, I've hit 1:42:59, so I know under the right conditions I can do better.</p>
<p>However, the bad news is that I seem to have quite seriously sprained my foot, so walking is incredibly painful. I think this happened because of trying to run in such a way to avoid re-injuring my ankle. A few weeks back I tried to run around a dog on the Whitley Bay boardwalk and landed on the side of my foot. That cut into one of my training runs, and I had to hobble home after just half a mile! As a side rant, dogs are supposed to be on leads while on the boardwalk during summer, so if you disregard that law you're a bad person and are responsible for injuries. I should probably nag the council, because I've been also bitten by dogs there, who were not on their leads.</p>
<p>Should I enter next year? Bupa events are expensive, the Great North Run costs about £50 to enter, but it is incredibly tempting to go for three in a row and get under 1:45. If I focus on improving my half marathon time, I could aim to beat my Auntie's half-marathon personal best of 1:39. But I reckon that will involve some quite serious lifestyle changes, primarily dietary. It's doable if I start training now.</p>
If You Don't Like It, Leave2014-08-04T00:00:00Zhttps://samhogy.co.uk/2014/08/if-you-dont-like-it-leave/<p>Ever voiced a criticism of a product, service or idea and promptly received the reply: "If you don't like it, leave?"</p>
<p>It's the ultimate in free-market thinking, to reduce your thoughts and feelings down to an in/out binary decision, then exercise that in silence. Just leave. Don't buy it. Don't use it. Walk away and stop complaining. The argument is bollocks and it only requires a slight pause for thought to realise why it's so.</p>
<p>I'm going to use Google as an example. They're truly a pervasive company and in my opinion a privacy nightmare. They know what you're searching for, what websites you visit, the contents of your emails, your physical location, your contacts and communications. Almost every part of your being, they can know. It hasn't always been that way, but the walls between their services have been slowly, ever so slowly, eroded.</p>
<p>It's a big buy-in for what are a fantastic range of products. But is it too much? Let's say you find the privacy controls that Google present to you unacceptable. After all, their controls only extend to the public, not to what Google can internally do with your data. So you decide enough is enough and "just leave". That means you're going to have to:</p>
<ul>
<li><strong>Stop using Google products</strong>. No more Google searches, storing documents in Drive, using Gmail or Calendar. No more Maps, StreetView. Switch to a different browser, you can't use Chrome. Find a different language translation service. Give up your Android phone and stop watching videos on YouTube.</li>
<li><strong>Stop using Google products!</strong> Google do more than that, however. You're going to have to make sure the websites you use don't use Google Analytics, or Google Web Fonts. You'll have to take your personal websites out of Google's search results and also any other third-party website that mentions you.</li>
<li><strong>STOP USING GOOGLE PRODUCTS.</strong> Block all network connections to Google. Seriously, you've left. That means you can't send email to, or receive email from, Gmail users. Straight into the spam bin with them.</li>
</ul>
<p>Ok, this is a pretty ridiculous example, but it makes the point sufficiently. To exercise your right to not use the Google ecosystem is, in reality, functionally impossible. All you can do is reduce your scope of usage to a level you deem acceptable, or campaign to change those practices you oppose.</p>
<p>This is important. Environments which encourage those who hold criticisms to remain silent and/or go away are by definition exclusionary and discriminatory. If I take the nuclear option above, it erects a massive wall around a large proportion of the Internet that I'm excluded from. Now consider a fictional social network where sexist imagery is frequently shared by its userbase. Would we be happy if a user raised this concern, only to be told to "just leave" the site? Of course we wouldn't accept that.</p>
<p>"Just leave", like any of these simple statements that some see as a panacea of wisdom, hold true in a very limited set of circumstances. You can choose to leave a supermarket if you think they're too expensive, but it's not as easy to emigrate from a country if you disagree with the Government's policy. Instead, we should create environments where people are free to voice opinions, for them to be listened to and if necessary adopted. We cannot assume that the users of our service approve of everything we do, because they're still using the product. We cannot use our dominance or privilege to bully out alternate viewpoints. We have to work together to survive.</p>
Breaking the Build2014-07-22T00:00:00Zhttps://samhogy.co.uk/2014/07/breaking-the-build/<p>If you work in software development, you'll be used to having source control store your code in a central location. A continuous integration program will pick up new code changes, compile your application, run your tests and produce a new version of your application. This encourages developers to keep the build 'green', to test their code locally before putting it out to the wider organisation (or world).</p>
<p>What happens when the build fails? Ther are many reasons for failure, from accidental to reckless. Many teams have adopted the practice of applying a penalty, of some form, to the developer who breaks a build. A quick check on <a href="http://stackoverflow.hewgill.com/questions/211/426.html">this (archived) Stack Overflow question</a> shows that this practice is pretty common. I'll quote some answers:</p>
<ul>
<li>"We have a bee outfit... You have to wear it until the end of the day."</li>
<li>"Take No Prisoners. I maintain a nerf arsenal. Revert the check-in and aim for the face."</li>
<li>"USB foam dart gun. It is the geek way!"</li>
<li>"A giant cardboard cutout of Jar-Jar Binks"</li>
<li>"We sometimes use fingerpointing"</li>
<li>"We have a rubber chicken that hangs outside the cubicle of the person who most recently broke the build".</li>
<li>"WALL OF SHAME"</li>
<li>"We have just started throwing shoes at the offender"</li>
<li>"Wear a dress for a day"</li>
</ul>
<p>You may call this a laugh, office banter whilst getting a serious message across. <strong>I call it ritual humiliation</strong>, because that's what it is. Verbal humiliation, social humiliation. In some cases, what could be reasonably classed as assault. That's not what you expect from professional, inclusive environments. I do not stand for it. You should not either. Based on the responses above, I'm glad the question was deleted from StackOverflow.</p>
<p>Builds break. When it happens, all you need to do is make sure the developer who broke it fixes it right away. Help them out if it's something non-trivial. Make it a learning experience - perhaps this wasn't some reckless check-in, but an accidental oversight. They could have forgotten to save their .csproj file when adding a new code file, so everything works locally but fails on the build server. If it was reckless and this is a frequent occurrence from that person, have a private conversation with them, but don't humiliate them. We should demand better than that. Your CI server is another line of defence and sometimes it's going to flag build failures for you.</p>
<p>We can do better. Here are a couple of suggestions that I have, purely from my experience.</p>
<h2>Gated Check-ins</h2>
<p>When you push your code to the server, it doesn't actually 'accept' the change unless a build passes. Should a build fail, you'll get a report and the change is rejected. Gated check-ins will ensure that you don't have a broken build and/or revert commits scattered throughout your code history. Furthermore, you can make this part of your code review approval process! <a href="https://code.google.com/p/gerrit/">Gerrit</a>, an open source code review tool, will handle this process for you.</p>
<h2>Feature Branching</h2>
<p>A developer may want to push code to the server, to back it up, or allow someone else to work on it. That may be experimental, in progress code. We don't want a build failure for that, nor do we want that code on master. The solution is to create a feature branch and commit your work-in-progress changes to that branch. It can be re-synced with the master branch periodically to avoid diverging too much from other changes. When it's ready, it can be merged back into the master branch. You may still want a CI build to be generated for every commit to that branch, just so your team-mates have a snapshot of the branch status. But, it doesn't matter if that build fails! All that matters is that the build passes when it comes to re-integrate with master. The Atlassian toolset, <a href="https://www.atlassian.com/software/stash">Stash</a> for code hosting and reviews, plus <a href="https://www.atlassian.com/software/bamboo">Bamboo</a> for builds, works very nicely here. Just like Github, you can create 'pull requests' to re-integrate a feature branch back into master. Bamboo can be configured to build feature branches. When it comes to accepting and merging the pull request, the two tools integrate, so if the 'tip' of the feature branch doesn't have a working build against it, then you can't merge.</p>
Organising a Skeptics in the Pub Talk2014-07-19T00:00:00Zhttps://samhogy.co.uk/2014/07/organising-a-skeptics-in-the-pub-talk/<p>The Skeptics in the Pub movement is fantastic. The idea is simple, organize a pub meeting to talk about subjects of a skeptical nature. That's fairly broad, and deliberately so! You'll hear talks about quackery, actual proper science, ROBOTS, technology, politics, sociology, all the -ologies actually. Earlier in the year I took over organisational duties for the Newcastle group with my friend Will. Since then, we've been arranging for speakers for the remainder of the year. Being new to all this, but also tech-heads, we decided that we needed a way to keep track of all the things we need to do to make an event happen.</p>
<p>I've previously written about <a href="/2013/08/issue-tracking-for-my-new-house/">how I use Trello as a simple to-do list for my house</a>, so I decided to use it here too.</p>
<p><a href="https://trello.com/b/3UxUwEKU/event-checklist-template">This is our "Event Checklist Template" board</a>.</p>
<img src="https://samhogy.co.uk/ssets/skeptics_trello_board.jpg" alt="Newcastle Skeptics in the Pub event template Trello board" width="648" height="282" />
<p>For each month, we take a copy of this board. That will give us a new board with all of the cards copied over, so each month starts with a blank slate. Now rather than keeping this as a "to do, doing, done" board, I instead created three categories:</p>
<ul>
<li>Things we need to do before the event can happen</li>
<li>Things we need to do on the day</li>
<li>Things we need to do after the event</li>
</ul>
<p>You can copy a board through the menu sidebar:</p>
<img src="https://samhogy.co.uk/ssets/skeptics_trello_copyboard.jpg" alt="How to copy a board in Trello" />
<p>Once everything is marked as done, we'll close the board.</p>
<p>It's admittedly a pretty simple system, but that's why it works in my opinion. It makes sure that things that are easy to forget (such as checking next month's venue booking) actually get done. If you organise a Skeptics in the Pub event, feel free to take a copy of our board and put it to use in your own group!</p>
MP responds to DRIP Bill Concerns2014-07-16T00:00:00Zhttps://samhogy.co.uk/2014/07/mp-responds-to-drip-bill-concerns/<p>Late last week, the UK Government announced what they called 'emergency legislation' - the <a href="http://www.parliament.uk/business/news/2014/july/mps-debate-data-retention-bill/">Data Retention and Investigatory Powers Bill</a>. This was timetabled to be passed through the Commons yesterday (Tuesday 15th), and the Lords today and tomorrow (Wednesday 16th, Thursday 17th July). This is the final week of Parliament sitting before their summer recess.</p>
<p>To keep this brief, in April the European Court of Justice ruled the Data Retention Directive, specifically with regards to retention of communications data, was illegal and a breach of fundamental human rights. In response, the UK Government felt it necessary to legislate. They argued they needed to act to "clarify" the Regulation of Investigatory Powers Act, that this is doing nothing than "retaining the status quo", it "breaks no new policy ground", whilst ensuring "the police, the law enforcement agencies and the security and intelligence agencies have the capabilities that they need to protect the public and keep us safe". (All quotes, Home Secretary Theresa May during the second reading of the Bill)</p>
<p>I emailed my MP, Alan Campbell (Labour, Tynemouth) to voice my particular criticisms about this bill, which are:</p>
<ul>
<li>
<p>There is not enough time for Parliament to properly scrutinise this bill, because it's being rushed through Parliament. It was agreed in secret by all parties and put to the floor at the last minute. They've had since April for this to be debated in Parliament, instead, they specifically chose this strategy.</p>
</li>
<li>
<p>The Government's argument that this is preserving the status quo is not true. Clauses 4 and 5 specifically extend powers beyond what is defined in RIPA. The Government might have "assumed" these powers based on the original legislation, but they're new in terms of being explictly defined in law. This is a position consistent with an <a href="http://www.slideshare.net/EXCCELessex/open-letter-uk-legal-academics-drip">open letter published by several of the UK's leading academics</a>:</p>
<blockquote>
<p>"DRIP is far more than an administrative necessity; it is a serious expansion of the British surveillance state"</p>
</blockquote>
</li>
<li>
<p>The Government are choosing to act not to honour the ECJ decision, but to make what we were doing illegally, legal. They're completely ignoring it.</p>
</li>
<li>
<p>The 'sunset clause', the expiration of the legislation in 2016, is easily subverted by more primary legislation, so is nothing more than a promise.</p>
</li>
</ul>
<p>Here's the reply I recieved, posted verbatim:</p>
<hr>
<p>Thank you for writing to me regarding the government’s emergency legislation on communication data and interception.</p>
<p>As a result of a recent judgement by the European Court of Justice, the police and intelligence agencies are in danger of losing vital information which is used in 95% of serious and organised crime investigations as well as counter terror investigations and online child abuse.</p>
<p>In order to prevent this, UK legislation needs to change to be compliant with EU law. If these changes are not made, the police are likely to suddenly lose vital evidence this summer.</p>
<p>The Government has come forward with emergency legislation and in considering the Government's proposals, I believe it is essential to maintain the security of our citizens and also ensure people’s privacy is protected.</p>
<p>Serious criminal investigations and counter terrorism intelligence operations must not be jeopardised. That is why I support this emergency legislation which I accept is designed solely to protect existing capabilities.</p>
<p>However, given the limited Parliamentary time to discuss emergency legislation Labour has ensured that the Government's legislation is temporary and it will expire in 2016. This will require the Government and Parliament to properly consult on and consider longer term proposals next year.</p>
<p>Labour has also now secured agreement to our proposal for a major independent review of the legal framework governing data access and interception (the RIPA review we called for earlier this year) in the light of the huge changes in technology. As I have previously argued, in the wake of the Snowdon leaks and the concerns raised about whether the legal framework has failed to keep up with new technology, there is a clear need for wider public debate about the right balance between security and privacy online, a review of powers and stronger oversight.</p>
<p>This review will enable longer term questions and concerns to be properly dealt with and debated in time for new legislation. Changes will then follow. Labour has also called for and secured further safeguards to restrict the ways in which communications data and intercept can be used to prevent misuse.</p>
<p>I share some of your concerns about rushed nature of these proposals being presented to Parliament, and I am pleased that Labour has been able to secure safeguards within the limited timetable. I strongly support the temporary legislation it would be far too damaging to the fight against serious crime, online child abuse and counter terrorist intelligence to suddenly lose these capabilities now.</p>
<p>Thank you again for your letter on this very important issue.</p>
<p>Best wishes</p>
<p>Alan Campbell MP
Tynemouth</p>
<hr>
<p>I must admit to being incredibly frustrated by the response. It seems my MP has accepted, without question, the government and Labour Party spin on the Bill. The law is not being changed to be "compliant with EU law". You are not ensuring "privacy is protected" by compelling ISPs, telephone networks etc to store <em>all</em> communications data, ready to be handed over upon request. That kind of blanket retention was deemed illegal, this (re-)legalizes it. Indeed there is a balance to be struck between security and liberty, but this bill subverts that balance entirely, by going against the Court's decision. We cannot allow those who make the law to be its final arbiters. The legislation being "temporary" is academic; as we've seen with the "limited timetable" of this bill, passing something off as an emergency is an easy way to push primary legislation through Parliament. The same could be done to remove the sunset clause.</p>
<p>Passing this bill is an abuse of Parliamentary procedure, sets a dangerous legal precedent and is in breach of human rights. As Tom Watson MP says, it is a stitch-up.</p>
<p>This is how freedoms are eroded. Not in great chunks, but DRIP by DRIP. </badpun></p>
Pandemic in a Box2014-06-16T00:00:00Zhttps://samhogy.co.uk/2014/06/pandemic-in-a-box/<p>Pandemic is one of my favourite board games. It's sufficiently complex to warrant strategic thinking, but it's also accessible enough to be taken to a party and everyone will enjoy it.</p>
<p>Thanks to last Christmas and this year's birthday, I've acquired both expansions: On the Brink and In the Lab. I sometimes take the game to a friend's house to play and I didn't like the idea of having to transport three boxes for one game. Is it possible to combine all of the material into one box?</p>
<p>The answer is yes, with a bit of trickery.</p>
<p>Both expansions add some new mechanics to the base game, requiring new components (meeples, counters, buildings etc), cards and boards. That all has to go in. Additionally, On the Brink gives you five petri-dishes to store your disease cubes. In the Lab provides you with five plastic vials, to replace the wooden ones from the core game. Both of these changes are purely cosmetic, but why wouldn't you include them?!</p>
<img src="https://samhogy.co.uk/ssets/pandemic_components.jpg" alt="All the game pieces, plus rulebooks and boards, in the core box" />
<p>An overview of what's included: all three rulebooks, the core and lab boards, plus a single copy of the bioterrorist character sheet.</p>
<img src="https://samhogy.co.uk/ssets/pandemic_box_close_up.jpg" alt="A close of of the game components in the core box, using the On the Brink box insert" />
<p>A close-up of everything combined in one box. I took the plastic insert from On the Brink (which had slots for vials and petri-dishes) and swapped it with the cardboard insert from the core box. That way everything is stored in the core box! :) There were too many cards in the player deck to stack up in their own row, so some went on top of the epidemic deck. I can live with that. All other cards, such as the lab cards, roles, events and rules, are in the third row. The blank cards are not included.</p>
<img src="https://samhogy.co.uk/ssets/pandemic_below_insert.jpg" alt="The leftover items: the lab tokens, a meeple, a research station and the epidemic / outbreak trackers, are placed in plastic bags and stored under the inserts." />
<p>Sadly and annoyingly, not everything fits into the plastic insert. The epidemic and outbreak trackers don't really have a place. I had one extra meeple and research station that could not fit into the allocated slots. Plus, all the counters from In the Lab had no home. I decided to cheat here, and bag them up, then place them under the insert. This requires some skilled placement, or you can draw three crosses on the box with a pencil, like I have, so that you know roughly where they go.</p>
<img src="https://samhogy.co.uk/ssets/pandemic_box_closed.jpg" alt="The box closed. It doesn't fully close, but isn't too far from closing completely" />
<p>The box, when closed, doesn't fully shut. That'll be the bulk of the extra game board from In the Lab. The fit is good enough, however, to be transportable. To be extra careful you could always wrap it in a hairband or elastic band.</p>
<p>It would be nice if Z-man games released a larger box, which would fit all of the game components, but as there's next to no chance of that happening, this is an acceptable compromise to me. I hope it helps you too!</p>
E3 20142014-06-08T00:00:00Zhttps://samhogy.co.uk/2014/06/e3-2014/<p>E3, the games industry's largest conference, is due to start this week. We're expecting big announcements from all the console manufacturers, plus the major third parties (Activision, EA, Ubisoft, etc). Last year was awesome as the next-gen consoles were announced. But going into this year's gig, we haven't really had many next-gen exclusives, with most new titles still being released on the previous generation's hardware.</p>
<p>That of course is with the exception of Nintendo. The media narrative is that Nintendo have had an awful first year and are doomed to oblivion. I bought a Wii U at Christmas last year when there was an Amazon Lightning Deal. So far I've been really pleased with the console. It's primary use has been as a Netflix device, but now that I own Super Mario 3D World and Mario Kart 8, that's going to change! I've also got Lego City Undercover and Zombi-U to play. The lineup is actually pretty strong compared to the other consoles. I'm hoping that sales will pick up now Mario Kart is out. The "Luigi Death Stare" meme seems to be doing wonders for their marketing too. I'm actually really pleased with how wonderful of a console the Wii U is. Nintendo Land and Wii Play U demonstrate some interesting mechanics using both the new gamepad and your standard Wii controller. I'm hoping that, with time, some of these ideas are fleshed into full games. The fidelity of the graphics are amazing, too, especially in Mario Kart.</p>
<p>I didn't buy into the Xbox or Playstation this generation and I don't intend to. Last year I invested in a gaming PC, which is still very much capable of handling whatever's thrown at it. So for this reason, I'll focus on the top 5 things that I want to see from the Nintendo side of thigns at this year's E3:</p>
<h2>Nintendo's take on Minecraft</h2>
<p>Sandbox games are all the rage at the moment, thanks to Minecraft. I'd love for a Nintendo take on this idea. Animal Crossing's out so I wouldn't expect to see a new release there, but potentially something to do with creating levels from the SNES Super Mario World? It's one of my all-time favourite games, and already heavily-modded on the 'net.</p>
<h2>Metroid</h2>
<p>My foray into the Metroid universe started late, basically with Fusion (GameBoy Advance). I loved it, bought Zero Mission (a GameBoy remake of the original) and then Prime on the Gamecube, which is probably my highlight of the series. I couldn't really get into its two sequels and eventually got rid of them. My Wii U has a Virtual Console version of Super Metroid (SNES) waiting to be played, once I have some free time.</p>
<p>It would be fantastic to see more Metroid games, either 3DS or Wii U - I don't care!</p>
<h2>Pets for your Mii</h2>
<p>Seriously, Nintendo, just do it. You can have this idea for free. Providing you don't call them 'pii's, because that will be stupid.</p>
<h2>Majora's Mask 3D</h2>
<p>Ocarina of Time is one of my favourite games of <em>all time</em>. Seriously, it's that good. But I always liked its brother, Majora's Mask, more. It's just so dark and it's soundtrack is... haunting. I have it for the GameCube as part of the Master Collection disc that came with the Wind Waker. Incidentally I also have the Wind Waker HD remake and A Link to the Past sitting on my Wii U waiting to be played. If they announced Majora's Mask on the 3DS it would be an instant purchase!</p>
<h2>Hyrule Online</h2>
<p>I'm going to have to thank my friend Will for pitching this idea, but OH GOD YES. Hyrule Online, Nintendo's MMORPG. Shut up and take my money, my life, take everything. This would ruin me and I'd enjoy every minute of it.</p>
<p>I expect the hit to miss ratio here to be pretty poor. In fact, 0 out of 5. Nintendo will most likely announce GameCube games on the Virtual Console and lead with Super Smash Brothers. Oh well, only a few more days to go!</p>
Getting Functional at DDDNorth2013-12-05T00:00:00Zhttps://samhogy.co.uk/2013/12/getting-functional-at-dddnorth/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2013/10/18/getting-functional-at-dddnorth.html">the Scott Logic blog</a></span></div>
<p>Last weekend I had the pleasure of attending this year's <a href="http://dddnorth.co.uk/">DDDNorth</a> conference at Sunderland University. DDD events are organised by the community, so anyone is free to submit a session. The community votes on which sessions they would like to see at the event. This year's event took on a distinctly functional feel - advocates of wisdom of the crowds, feel free to interpret this as a sign of things to come. I'd like to reflect upon the excellent sessions I attended.</p>
<img src="https://samhogy.co.uk/ssets/dddnorth.png" alt="DDDNorth logo" width="250" height="188" />
<h2>TDD - Where did it all go wrong? - Ian Cooper</h2>
<p>Ian Cooper took a retrospective look at the practice of TDD, going back to the original source, Kent Beck's book. Why is it that test frameworks do not often deliver on the grand promises of TDD? Test suites built upon brittle foundations become an obstacle. Adverse effects of brittle test frameworks includes adding new code, rather than refactoring because it would "break the tests".</p>
<p>Ian's delve into recent history comes up trumps. Kent was not necessarily talking about testing code in isolation, but testing behaviours in isolation. The key point to remember is that you should be able to change the exact implementation of your code during the "refactor" step of TDD, without breaking the tests. Make it run, make it right.</p>
<p>This then becomes an architectural issue: recognizing that brittle tests are tightly coupled to the code they are testing. Ian's introduced <a href="http://alistair.cockburn.us/Hexagonal+architecture">hexagonal architecture</a> as a way of handling this situation. The focus is on clear separation of concerns across API boundaries. Keeping the domain model separate from the details of the UI, persistence or networking strategy means that code is reusable but testable. The code to glue your domain model, via its APIs, to a particular implementation detail lives in a separate adapter entity. Thus, switching to a command-line UI running an in-memory database for testing purposes is simple.</p>
<p>Ian's talk was a great start to the day. Hexagonal architecture reminded me to be wary of allowing domain logic to cross your API boundary into adapters. Keep it nicely wrapped up and you can write maintainable test cases against it.</p>
<h2>Scaling Systems Architectures - Kendall Miller</h2>
<p>Scaling - the ability to perform and cope under an increasing workload. Kendall Miller is adept at breaking down complex topics into transferable chunks of knowledge, with enthusiasm and theatrics. The audience loved his amusing anecdotes about vanishing heavy machinery and an interpretation of the 'SQL Walk of Shame'. How much work is actually involved to say "Hi, Bob!" at the top of a webpage? Quite a lot, it turns out!</p>
<p>Scaling systems comes down to four core concepts; the first three which aid scalability, the fourth hampers it:</p>
<ul>
<li><strong>Asynchrony</strong> - taking work out of the critical path and doing it later.</li>
<li><strong>Caching</strong> - Do less work! Authoritative information is time consuming. Oftentimes it is not needed. Software and hardware deploy multiple levels of caching, simply to avoid costly operations.</li>
<li><strong>Distribution</strong> - get as many people to do the work as you can. Where possible, utilise parallelism to scale outwards, across multiple systems. This also applies to geographical distribution, of databases, servers and processes.</li>
<li><strong>Consistency</strong> - the more shared state that a system shares, the harder it is to scale out. Shared states require locks. Waiting on locks wastes processor cycles that could be used for more important work.</li>
</ul>
<p>From these four pillars, it is possible to deploy scaling strategies. Kendall's pragmatic advice for the developer looking to scale a system is to tailor your strategy to your business requirements - don't go looking for "one size fits all" solutions. Kendall's in-depth knowledge of real-world examples helped solidify this concept. For example, Amazon operates on a global scale. It can geographically distribute its database of orders, order number generation and order processing geographically. There is no nasty singleton living somewhere on one Amazon server, generating unique monotonically-increasing order numbers.</p>
<h2>WTFP?! Functional Programming and Why You Should Care - Grant Crofton</h2>
<p>Grant's talk focused on the concepts of functional programming, with examples in F#. The functional programming paradigm has a long history - with roots in 1930s lambda calculus. This style is becoming popular once again. Languages like Scala and Clojure, both of which run on the JVM, are reaching a sufficient level of confidence in order for <a href="http://www.thoughtworks.com/radar">ThoughtWorks</a> to advise adopting them in production applications.</p>
<p>Where functional programming stands strong is with a declarative approach to programming. Problems are described closer to the problem domain, with the language dealing with the implementation particulars. Grant introduced several concepts, such as expressions, parallelism, immutability and higher-order functions. For each section, a problem was addressed in two ways. First, Grant presented an imperative approach in C# and compared this to the corresponding F# declarative implementation. Sometimes, just for kicks, Grant would show an imperative solution in F#, just to show it is possible and to demonstrate how well it interacts with C#.</p>
<p>As a means of quick introduction to functional programming and the F# language, this talk was particularly useful. Unfortunately, I didn't get the opportunity to see Phil Trelford's "F# Eye for the C# Guy" talk, but from fellow conference-goers I heard that these were two great complementary sessions.</p>
<h2>Tyrannosaurus Rx - John Stovin</h2>
<p>Reactive Extensions (Rx) is a .NET library used to simplify the handling of asynchronous, event-driven programs. Rx is built upon the classic "Gang of Four" Observer pattern, where you have an observer wrapping a data source. Subscribers are notified when updates to the source occur and react accordingly. Rx allows for LINQ-style operations (such as filtering) to be applied in response to subscription notifications. Where concurrency is needed, Rx integrates tightly with schedulers. Once again, the functional, declarative approach shines through.</p>
<p>John's talk focused on covering the basics of Rx, explained succinctly with a trusty whiteboard and <a href="http://rxwiki.wikidot.com/marble-diagrams">marble diagrams</a>. The session was very use-case oriented, which is an effective way of demonstrating the benefits of a library. One particular example concerned sampling and interpreting a user's mouse movements, which was scaled to support clicking and dragging.</p>
<p>With hindsight, I would have loved to see how an example of how Rx integrates with C#5's async and await mechanisms; but overall I'm impressed with how easy reactive extensions are to use and how simple they make your code. In fact, when you then see the old way of doing things - manually hooking up event listeners with delegates, maintaining state and so on - it just seems dated. Rx seems to fit perfectly. Rx now has a life of its own outside .NET land - the library has been open-sourced and is frequently ported to other languages.</p>
<h2>You've Learned the Basics of F# - What's next? - Ian Russell</h2>
<p>Ian's talk took a deep dive into F#, to show how it can aid developers quickly create scalable applications. This was a fantastic end of day session, building upon the core concepts from Grant's session. The focus was on two particular concepts:</p>
<ul>
<li>
<p><strong>Type Providers</strong> are a quick, easy and flexible way of pulling content out of SQL, JSON, CSV, XML and much more. The functional paradigm suits this particularly well, providing an elegant means of selecting, filtering and operating on data pulled from a source. Type Providers integrate well into Visual Studio, exposing via Intellisense types, properties and methods pulled directly from the data source!</p>
</li>
<li>
<p><strong>The MailboxProcessor</strong> is an asynchronous message queue and handler which runs on its own thread. Any other thread can post messages to the mailbox processor agent, which may also reply back. MailboxProcessor is thus a very lightweight means of inter-process communication, requiring little boilerplate code. To demonstrate the usefulness of this functionality, Ian created a simple example of sending 100,000 asynchronous messages to the MailboxProcessor agent. Following on from this, the example was scaled up into a full chat room service!</p>
</li>
</ul>
<h2>Summary</h2>
<p>DDD events are always high quality. This year's DDDNorth was no exception! My thanks go out to Andrew Westgarth and co for organising the event. As a developer, DDD events around the country are a great way to delve into concepts you may not normally have the time to investigate, network with developers around the country and eat far too much food at the Geek Dinner. I always come home brimming with ideas, but also with great insights on everyday practice. I highly recommend that you come along next year - remember to say hi!</p>
On Running a Code Club2013-12-04T00:00:00Zhttps://samhogy.co.uk/2013/12/on_running_a_code_club/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2013/12/04/on-running-a-code-club.html">the Scott Logic blog</a></span></div>
<p>Since September, a colleague (Pat Armstrong) and I have been taking part in a local school's after-school <a href="https://www.codeclub.org.uk/">Code Club</a>, <a href="http://www.scottlogic.com/news/20131203-certificates-awarded-to-code-club-whiz-kids/">backed by Scott Logic</a>. This is a nationwide, non-governmental initiative to aid children, aged 9-11, to learn programming. It's operated purely on a voluntary basis and is free for a school, library or community centre to set up. In this post, I'd like to discuss my personal experiences from the initiative and provide some tips I found useful during the sessions.</p>
<h2>What is Code Club?</h2>
<p>Code Club lessons are structured into terms, each with nine lessons. The terms are further subdivided into 'levels' of difficulty. At the end of each level, the children are awarded a certificate to recognise their achievements.</p>
<p>Term One focuses on <a href="http://scratch.mit.edu/">Scratch</a>, an educational tool which combines a programming language and development environment. Using Scratch it is incredibly easy to create games, animations and interactive stories, which is perfect for whetting the appetites of budding programmers! Children are able to pick up Scratch very quickly - it needs little exploration and results are immediate.</p>
<p>Code Club lessons are extremely well-designed. Instructions are clear, concise and age-appropriate. Students are reminded when to test, when to save, and there are extension activities to really test the understanding of a project. <a href="https://www.codeclub.org.uk/about">Sample lessons are available online.</a></p>
<h2>My Top 5 tips on Running a Code Club</h2>
<ol>
<li>
<p><strong>Anticipate, log, and feedback issues.</strong>
Follow the <a href="http://military.wikia.com/wiki/7_Ps_(military_adage)">7 P's of the military</a> religiously. Before each lesson take some time to follow the lesson plans yourself to identify potential issues. For example, I asked myself: are some of these instructions tricky? Did <em>I</em> make a mistake? Is it easy to make a common mistake at this point? Jot these down, along with the solutions. If the lesson introduces a new programming concept, come up with some spiel to explain it. Explaining the notion of a variable to a bunch of kids off-the-cuff is tricky! I overcame this with a heavy focus on examples. I also maintained a diary to track the progress of the students and the details of any issues we encountered. In one specific case discovered a bug in the lesson code! Code Club host plain text versions of their <a href="https://github.com/CodeClub/scratch-curriculum">lessons on Github</a>, which gives you an easy way to provide feedback. You don't have to be meticulous with your book-keeping, but reporting on your experiences makes the lessons better for everyone!</p>
</li>
<li>
<p><strong>Use the check-boxes!</strong>
Each instruction in the lesson plans has a check-box to the right of it. Make sure the children use them! When you're diving into a piece of new functionality it's very easy to lose track of your place. Give each children a pen and hail the importance of keeping track of work. We didn't quite make it through one lesson a week (some of which is down to the nature of an after-school club session), so having the students record their progress meant they could get straight back into the action at the next session.</p>
</li>
<li>
<p><strong>Encourage problem solving.</strong>
It's easy to grab the mouse and show the student the solution when they get stuck. That works well in the initial stages, providing you carefully explain why your change fixes the issue. In the initial sessions I tried my best to do this, but as the students progressed to the later stages I began to take a hands-off role. By this point they have some Scratch experience under their belts, so it's worth giving them a chance to figure it out for themselves. One way of achieving this is to ask the students to read the code aloud. Sometimes they'll spot the issue straight away ("oh, I've got less than when I actually meant greater than!"). I don't tell them we call this <a href="https://en.wikipedia.org/wiki/Rubber_duck_debugging">rubber-duck debugging</a>! However, if that doesn't work, I instead try to encourage problem solving by explaining the requirements, sometimes with other examples.</p>
</li>
<li>
<p><strong>Recruit helpers.</strong>
Some students will finish lessons before others. Luckily this is anticipated! The Code Club material contains extension challenges to add an extra level of difficulty. I found that some of the challenges were quite tricky (one required completely changing the perspective of a game), but effectively cement the knowledge developed over a lesson. It can be great fun to get students who finish these challenges to help others! A key concept in programming is communication - including verbal communication. Understanding requirements, talking through solutions and helping others are all vital skills that should be encouraged within the club. Not only that, but you develop a community atmosphere in which everyone helps each other complete the work. After all, it's not a competition.</p>
</li>
<li>
<p><strong>Have fun!</strong>
Code Club can be hard work, so take a step back and enjoy it! Give the students time to play the games they create, especially when they need two players! The students will quickly come up with new ideas and begin implementing them! Students will also want to put their own spin on the games. Wherever possible, allow them to do so. It's not always possible to deviate from the lessons, such as with Fish Chomp (collision detection is performed on a fish's teeth, which is not possible on all sprite images), so be aware of where the students will have to be more constrained. If they're having fun, you'll be having fun and they're learning!</p>
</li>
</ol>
<h2>The Developer's Perspective</h2>
<p>Programs in Scratch are created by dragging and dropping 'blocks' of functionality into a stage. The shape of a block affords its function - iterative and conditional blocks literally enclose their contents. That way it's easy to see exactly what code will be executed under certain conditions. Interactive components, known as sprites, are added to a stage. These can be styled with 'costumes', respond to messages and events (such as mouse clicks) and can contain their own scripts.</p>
<img src="https://samhogy.co.uk/assets/scratch.png"/>
<p>Scratch comes in two forms, both of which are available for free online. Version one is a desktop application and version two is both a desktop and web application. Scratch 2 also has some new features that the Code Club material doesn't cover, such as function blocks'. There's an ongoing effort to <a href="https://github.com/CodeClub/scratch-curriculum/issues/37">convert the lessons to Scratch 2 over on GitHub</a>, should you wish to help out. Until the Scratch 2 conversion has finished, I advise sticking to Scratch 1.4 as there are some subtle differences between the two platforms.</p>
<p>The very essence of possessing a general-purpose computer is its ability to be programmed, unlocking another form of creative expression. Passing this knowledge to the next generation and seeing such a rapid, passionate uptake of the information is something I've found to be extremely rewarding. Indeed, I'd go as far as saying this is the most rewarding experience I've had in my professional career. Computing, specifically programming, is becoming an integral part of education - next year will see a new <a href="https://www.gov.uk/government/publications/national-curriculum-in-england-computing-programmes-of-study">Computing curriculum</a> introduced in the UK. If done correctly, this can have profoundly positive effect on the capabilities of our future society. Being a small part of that is exciting and something I can't recommend enough!</p>
Assessing Android Fragmentation2013-10-18T00:00:00Zhttps://samhogy.co.uk/2013/10/assessing-android-fragmentation/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2013/10/14/assessing-android_fragmentation.html">the Scott Logic blog</a></span></div>
<p>Android targets a wide user base. It runs on budget to high-end phones and tablets, digital cameras and even refrigerators! In its short lifetime, several major versions have been shipped. This proliferation has been described as the 'fragmentation' of the platform.</p>
<p>Looking at the numbers; <a href="http://opensignal.com/reports/fragmentation-2013/">OpenSignal</a> detected 11,868 distinct Android devices this year, up 200% from 2012. Gingerbread (an Android version that is 2 ½ years old), still accounts for 30% of devices. Whilst these numbers can be considered rock-solid, they're only really painting a picture of what’s intended by design - Android is fragmented. It's a feature, not a bug!</p>
<p>What truly matters is the impact this has on developers and product owners. Fragmentation is a problem if applications are difficult to develop, maintain and market. By looking at some of the key attributes and supporting mechanisms of the platform, we can assess whether Android's fragmented nature is a strength or weakness.</p>
<h2>Handling different screen sizes</h2>
<p>With such a wide variety of devices, how is it possible to make sure your app looks great everywhere? The solution is to design with this in mind. Responsive design involves adapting the UI such that it is optimized for the user’s advice. Layouts displayed individually on a phone may be combined into a multi-column layout on tablets. Adaptations apply at the micro level too, with flexible margins for content to scale and fit a new boundary, and images displaying at an appropriate resolution.</p>
<img src="/assets/android/responsive_android_layout.png"/>
<p>Android supports, indeed encourages, this style of design. Images, layouts, colours and strings are all loaded dynamically. Apps can therefore provide multiple versions of an asset for different situations and let the system choose the most appropriate resource, based upon the device configuration. It’s possible to load a resource based on screen size, OS version, screen orientation, language - or a combination of these!</p>
<p>What’s great about how Android approaches this solution is that it is flexible. You can buy into as much or as little as your app needs to work. If you want your app to look the same everywhere, it can. Android does its best to help you out. This flexibility also extends to when you need to tweak something for a particular form factor - the scope of the change is limited.</p>
<p>Part of getting a responsive design right is <strong>rigorous testing</strong>. This is especially essential when there is code that reacts differently based upon the device configuration. Using Android’s emulator, developers can simulate a device according to some custom specification - such as screen size, hardware and OS version - and test the app. There is a risk bugs can be introduced, causing behaviour to differ between devices. You do not need to test every possible combination, but rather focus on well-established form factors and then think about edge cases.</p>
<p>Overall, responsive design somewhat future-proofs your application. If a new 15" tablet is released tomorrow, your UI will scale and arrange itself appropriately so that it looks good. There is no need for you to release multiple versions of an application, one to target phones, another for tablets and so on. Admittedly, creating and testing responsive layouts incurs a larger development cost, but this is somewhat mitigated by the wealth of tooling support available and the benefit of being flexible enough to look great on all devices.</p>
<h2>Android OS Version Changes</h2>
<p>Between Gingerbread and Ice Cream Sandwich, Android did a lot of growing up! Honeycomb was a tablet-only version of Android, including a new visual style, the Fragments API (a key component in creating multi-column layouts) and the Action Bar design pattern. These changes were made available to all devices with Ice Cream Sandwich.</p>
<p>Alone, these additions are welcome, but not useful to apps which would like to support versions of Android before Honeycomb, too. To avoid this predicament, Google packages up re-implementations of these new APIs as part of its <a href="http://developer.android.com/tools/support-library/index.html">Android Support Library</a> package. This is compatible with older versions of Android and mirrors the native APIs. Developers can very quickly hook it into their applications to bypass lots of platform-specific behaviours and UI inconsistencies.</p>
<p>Not all API changes can be backported. A simple example is when APIs are added to support a new hardware feature. Furthermore, there have been bugs found in the Support Library. However, the support library significantly reduces the cost of fragmentation, by providing consistency in looks and behaviours between platforms. Developers aren’t left to completely re-implement core Android elements!</p>
<p>New OS versions are stricter than previous versions. This is normally done for security or efficiency reasons. For example, a change was made recently to further restrict costly operations during UI layout. Following the strict guidelines can benefit all versions - especially so for older devices where processor speeds and memory are lower.</p>
<p>In the case where APIs are superseded by new OS versions, you have a choice on the course of action to take:</p>
<ol>
<li>
<p><strong>Always use the old API.</strong> This is the easiest option, as it involves no extra code. However, it does mean that your app does not take advantage of new features available in the new API.</p>
</li>
<li>
<p><strong>Use the most appropriate API.</strong> Add some code to detect which version a device is running, and use the most appropriate API. This technique is often used to handle small behavioural differences between platforms, but can be extended to full features. A consequence is that features introduced in new APIs will not be available to incompatible devices, so the functionality of your app diverges by OS version.</p>
</li>
<li>
<p><strong>Reimplement the new API.</strong> Just like the Android Support Library, you can opt to use the native API where it is supported, and provide your own implementation where it is not. This involves the largest cost in terms of development, but comes with a great benefit of cross-OS-version consistency.</p>
</li>
</ol>
<p>Taking this decision on a feature-by-feature basis is most appropriate, as no "one size fits all" solution exists. Assessing the business value of each feature can often determine the method to overcome the platform inconsistency.</p>
<p>Android handles OS differences primarily through its supporting infrastructure, such as the Android Support Library. The resourcing system encountered previously also plays a role (it is possible to provide different UI styles for different OS versions). However, it is inevitable that version differences can result in bespoke solutions, which come at a cost.</p>
<h2>Security</h2>
<p>One final factor to consider is how fragmentation affects security. Security considerations are fourfold:</p>
<ul>
<li>
<p><strong>Security vulnerabilities in a manufacturer’s custom code.</strong>
Device manufacturers ship lots of additional material with the stock Android code. Samsung and HTC, for example, add their own UI layer and pre-installed applications to create a consistent look and feel for their range of Android devices. Alongside this are bootloaders and device drivers. Vulnerability in this code is the responsibility of the manufacturer and not the core Android team; although they do work in close collaboration when such vulnerabilities are found. Working around these issues can be extremely costly and tricky, although they are a rare occurrence.</p>
</li>
<li>
<p><strong>Security mechanisms shipped in new Android versions.</strong>
New versions of Android do not ship to all devices. The consequence is that new security mechanisms, such as application verification, are restricted to the devices that can support the new version, and receive an update. Of course, this risks introducing behavioural inconsistencies between platforms, requiring either workaround code, or updating code to follow new best practice. Implementing such changes comes at a cost of added development.
Furthermore; existing mechanisms are strengthened in newer OS versions. Typically reacting to these changes involves making a configuration change to your application that was not mandatory on earlier versions, but is in newer versions. These changes are typically safe, recommended and easy to implement.</p>
</li>
<li>
<p><strong>From an enterprise perspective - Device Management.</strong>
Device administration is supported in Android from Froyo onwards - which is pretty much every device on the market. Enterprises may wish to further ‘lock down’ devices to provide a further level of security on corporate devices. New features were added to this API in Honeycomb, so if your company has lots of pre-Honeycomb devices, be aware that they are more open to exploitation.</p>
</li>
</ul>
<p>Following security developments in the Android community and documented <a href="http://developer.android.com/training/articles/security-tips.html">security best practices</a> will limit exposure to vulnerabilities. If your requirements determine security to be a Very Important Thing (such as if you are handling sensitive information or money), consider hiring a penetration tester. There's a cost associated with good security and the fragmented nature of Android certainly increases this cost.</p>
<h2>Conclusion</h2>
<p>Fragmentation is a feature with bugs. There is a wealth of support available to handle Android’s fragmented nature, some of which is built-in to the framework. External solutions, such as the Android Support Library, are also available to speed up development within this environment and helps apps behave consistently.
When fragmentation issues do pop up, a tailored cost-benefit analysis approach must be taken to ensure that the right decision is taken. Factor into analyses the time involved to fix an issue, versus the amount of affected devices and the impact of the fix.</p>
<p>Android embraces fragmentation. You should too! A more open operating system results in more innovation and a wide uptake of devices. Embracing fragmentation means your application is ready to handle whatever device the market produces, and works to a high degree of quality. Maximising the amount of devices your application is compatible with ensures you are well-placed in an increasingly lucrative marketplace.</p>
Running 20132013-09-16T00:00:00Zhttps://samhogy.co.uk/2013/09/running-2013/<p>Since my last post about running, I’ve taken part in three major racing events: the Great North 10k, the Edinburgh Marathon Relay Race and finally yesterday’s Great North Run. Whilst not attempting to run competitively in the solo races, in all three I’ve competed with myself to get the best time possible.</p>
<p>First up was the Glory Leg of the Edinburgh Marathon Relay in May. The relay race takes place simultaneously with the marathon, over the same distance, but split into four legs. A different team member runs each part and there are changeover points throughout the course. Each of the legs is a different length, with the Glory Leg being the shortest distance. On one hand this is a blessing - you don’t have to run considerable distance and get to cross the finish line, on the other hand, you need to make sure it’s a fast effort. My work entered two teams, one for the Newcastle office and the other for the Edinburgh offics. In previous years a Scott Logic team had won the corporate challenge category, but this year the focus was on being the best office. On the day, we (Newcastle) won both on time and first to cross the finish line. I chased down my Scottish counterpart over the four miles, finally overtaking him on, quite literally, the last corner before the finish line!</p>
<p>The next race was in June. The Great North 10k was where I made all the mistakes you’ll expect of a racing newbie; setting off too fast, not settling into a consistent pace and not remembering the course elevation. I ran the first mile very quickly. The last mile begins with a very steep hill, which sapped most of my energy, but culminates with a lap of Gateshead Stadium. I mustered enough strength to sprint finish, to get a time of 45 minutes. Bupa’s events are well organised at the starting line – based on an estimated time you’re assigned a group of similar ability. For this race I was in Orange, the set of runners to depart right after the Elite and Fast Running Clubs groups. My mistake was to start off right at the start of the group, allowing the crowd to push me along at an unsustainable pace.</p>
<p>Finally, yesterday’s Great North Run. Prior to this I had ran a few 10 mile routes, and two 13 mile routes. My predicted time was 1hr 45, and I could achieve this in practice. I knew it would be tough to sustain this in the race, given the crowds and the fact it’s a tougher course. There’s plenty of uphill, including my new mortal enemy, the mile 11 hill. That said, there’s plenty of downhill too, but it simply doesn’t feel like you’re going downhill. 13 miles is a real slog, especially in the poor weather conditions of yesterday. In no way does that compare with a flyover from the Red Arrows! After comparing my times with my colleagues, I was pleased to have came first on raw time, but second on age categories. I know for next time I need to put more training in beforehand. Hopefully this will pull down that time even further! Having the atmosphere of a massive running event certainly helped inspire me during the toughest parts. Raising £400 for the British Heart Foundation was a nice side-effect too. Once again, many thanks to the incredible generosity of my friends, family and colleagues.</p>
<img src="https://samhogy.co.uk/ssets/samGNRFinish.jpg" alt="Me at the finish of the Great North Run" />
<h2>Tools and Tech</h2>
<p>I’ve been a faithful member of Runkeeper since 2010 – the Android app is great for entry-level and its web interface is brilliant for organising activities, planning routes and tracking your progression. I built up quite a wealth of data, which I wanted to keep. However, I have always been reluctant to take my mobile phone out running with me. I also wanted to gather more accurate GPS data and heart rate data. The Garmin Forerunner 410 was the ideal choice.</p>
<img src="https://samhogy.co.uk/ssets/garmin_forerunner_410.jpg" alt="Garmin Forerunner 410 GPS Watch" />
<p>It’s a mid->entry-level GPS watch, which comes with a Heart Rate Monitor as an optional extra. The watch is fairly advanced – you can program routes, have a “virtual partner” race alongside you (the watch tells you how far in front or behind you are) and set intervals. Results tend to be accurate and I’ve never had any problems with heart rate monitor connectivity. On the negative side, it can take a while to find an initial GPS lock. I’ve also found that at the start and end of a run, sometimes during also, I get ‘spikes’ on my GPS data where my pace speeds up to faster than Mo Farah, or slower than a Tortoise. However, I’m prepared to tolerate this. I export my data from the Garmin Connect website as a .tcx file and upload them into Runkeeper. Runkeeper will strip these extraneous data points. Once an activity is uploaded, you’ll need to edit it and assign it a predefined course, should you want to track multiple runs over the same course. In future I’d like to automate this.</p>
<h2>Overall Performance</h2>
<p>This year I’ve been very happy with my running progression. I don’t intend to take part in any more racing events this year other than Parkruns. Instead I will focus on improving my 5-10k pace. Garmin handily collates all of your personal bests, which I’ll display for posterity:</p>
<img src="https://samhogy.co.uk/ssets/running-times-2013.png" alt="Best 1 mile(6:07), 5k(22:34), 10k(46:12) and half-marathon(1:44:36) times 2013" />
<p>For me, these are solid times which reflect a personal journey from a sedantry lifestyle into one which is quite fit. It’s a great sense of achievement, but also an opportunity to relax, enjoy yourself and meet new people. Oh, and if you like technology, buy new toys ;)</p>
Issue Tracking for my New House2013-08-24T00:00:00Zhttps://samhogy.co.uk/2013/08/issue-tracking-for-my-new-house/<p>Moving into a new house is quite a difficult process. In addition to filling out countless change of address forms, playing the game of “who will give me something I can use to prove my address without requiring proof of address”, buying and building lots of <a href="http://www.youtube.com/watch?v=IUPu_ipbVB0">IKEA</a>; there’s also all the unforeseen issues that crop up. As a software developer, I’m used to having a backlog of defects and improvements to work from. What if I could apply the same system to the new house? And so… TO THE INTERNET!</p>
<p>I first noticed that <a href="http://www.flourish.org/blog/">Francis Irving</a> had been using <a href="https://github.com/frabcus/house">Github for a home issue tracking system</a> back in Spring 2012. After having <a href="https://github.com/frabcus/house/pulls?direction=desc&page=1&sort=created&state=closed">a good trololol at some of the pull requests</a> made, I realised that I’d prefer something more private. Github’s nice, but most of the functionality won’t be needed. Furthermore, there’s no way to prioritise issues on Github; something I’d quite like to do.</p>
<p><a href="https://twitter.com/paul_edwin">@paul_edwin</a> suggested <a href="https://trello.com/">Trello</a> as an alternative. What a fantastic system! Trello has the concept of cards, which can be placed on lists. Colour-coded labels allow cards to placed into categories. A board (containing your lists of cards) can be private and can be shared with named members. There’s no tricky configuration needed – everything is completely self-explanatory. Nice!</p>
<p>I created three lists to create a <a href="http://en.wikipedia.org/wiki/Kanban_%28development%29">Kanban</a>-like system. A fourth list for long-term projects sits outside the day-to-day tasks, for when we have the time, motivation and budget to handle them.</p>
<img src="https://samhogy.co.uk/ssets/trello-house.png" alt="trello-house" width="640" height="300" />
<p>A secondary reason for wanting an issue-tracking system is that it can also act as a note-taking system. For example, we need a new oven light bulb. We have the make and model of the old bulb listed on the card description. Luckily, Trello also has an Android app, so you can view and edit your boards on the move. I can check which type of light bulb I need whilst in the shop! The Android app is very pretty. Long-pressing a card allows you to place it onto a list. Lists are displayed in all their full-screen glory, with left and right swipes navigating between your lists.</p>
<img src="https://samhogy.co.uk/ssets/trello-android.png" alt="trello-android" width="168" height="300" />
<p>I think we’ll be sticking with Trello for the foreseeable future, at least until a system that can be self-hosted comes along. If you’re looking for a no-nonsense way of dragging your house to do list into the 21st Century, I can’t recommend Trello highly enough. Thanks once again to Paul for recommending this to me and making my life easier!</p>
DDD East Anglia App: Published!2013-06-26T00:00:00Zhttps://samhogy.co.uk/2013/06/ddd-east-anglia-app-published/<p><a href="https://play.google.com/store/apps/details?id=uk.co.samhogy.dddeastanglia&feature=nav_result#?t=W251bGwsMSwxLDMsInVrLmNvLnNhbWhvZ3kuZGRkZWFzdGFuZ2xpYSJd">My scheduling app for DDD East Anglia has landed on the Google Play Store!</a></p>
<p>I’ll continue my blog series about creating this application once the conference is over and done with. As for the next few days, I’ll be taking care of any bug fixes that may be necessary.</p>
DDD East Anglia App, Part 2: The Database2013-06-17T00:00:00Zhttps://samhogy.co.uk/2013/06/ddd-east-anglia-app-part-two/<p>Yesterday I got a basic Android app skeleton, with a customised action bar, set up. Today, I’m moving onto the data layer.</p>
<h2>Data</h2>
<p>Using the sessions list from the DDD East Anglia website, I compiled two JSON resources – one for speakers, another for sessions.</p>
<h3>Speakers.json</h3>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> “speakers” <span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> “name”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “bio”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> "thumbnail_url”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “id”<span class="token operator">:</span> “<span class="token number">1</span>”</span>
<span class="highlight-line"> <span class="token punctuation">}</span><span class="token punctuation">,</span></span>
<span class="highlight-line"> …</span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<h3>Sessions.json</h3>
<pre class="language-json"><code class="language-json"><span class="highlight-line"><span class="token punctuation">{</span></span>
<span class="highlight-line"> “slots”<span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> “start_time”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “end_time”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “sessions”<span class="token operator">:</span> <span class="token punctuation">[</span></span>
<span class="highlight-line"> <span class="token punctuation">{</span></span>
<span class="highlight-line"> “room”<span class="token operator">:</span> “<span class="token number">1</span>”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “title”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “description”<span class="token operator">:</span> “”<span class="token punctuation">,</span></span>
<span class="highlight-line"> “speaker_id”<span class="token operator">:</span> “<span class="token number">1</span>”</span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"> <span class="token punctuation">]</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The DDD East Anglia schedule list uses a gravatar URL for each speaker who has supplied an image, which I add directly into the JSON. Sessions are arranged into time slots, where each session identifies the room it is in and the id of the speaker running the session. I’ve made the decision to add in registration, lunch and tea breaks as sessions within a time slot, for consistency. In code, I’ll add some functionality to determine whether a session is compulsory (if a slot contains only one session), or optional (ie – an actual tech talk). The <strong>res/raw</strong> directory can be used to contain resources like JSON files. The system will take the name and compile it into the auto-generated R class. I’ll be able to access the JSON files using <strong>R.raw.sessions</strong> and <strong>R.raw.speakers</strong>.</p>
<h2>Constructing the Database</h2>
<p>The first time my application is launched, I intend to parse the JSON and store it into a <code>SQLiteDatabase</code>. This is a fairly lightweight relational database which any Android application can create.</p>
<p>To create a custom <code>SQLiteDatabase</code>, create a subclass of <code>SQLiteOpenHelper</code>:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">DatabaseHelper</span> <span class="token keyword">extends</span> <span class="token class-name">SQLiteOpenHelper</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token keyword">static</span> <span class="token class-name">String</span> <span class="token constant">DATABASE_NAME</span> <span class="token operator">=</span> “ddd<span class="token operator">-</span>ea<span class="token operator">-</span><span class="token number">2013.d</span>b”<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">private</span> <span class="token keyword">static</span> <span class="token keyword">int</span> <span class="token constant">DATABASE_VERSION</span> <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token class-name">DatabaseHelper</span><span class="token punctuation">(</span><span class="token class-name">Context</span> c<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">super</span><span class="token punctuation">(</span>c<span class="token punctuation">,</span> <span class="token constant">DATABASE_NAME</span><span class="token punctuation">,</span> <span class="token keyword">null</span><span class="token punctuation">,</span> <span class="token constant">DATABASE_VERSION</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token annotation punctuation">@Override</span></span>
<span class="highlight-line"> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">onCreate</span><span class="token punctuation">(</span><span class="token class-name">SQLiteDatabase</span> database<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// Insert here</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Remember to declare in your manifest file that you’d like to write to storage:</p>
<pre class="language-xml"><code class="language-xml"><span class="highlight-line"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>uses-permission</span> <span class="token attr-name"><span class="token namespace">android:</span>name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>android.permission.WRITE_EXTERNAL_STORAGE<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span></span></code></pre>
<p>Next, it’s time to formalise the structure of the database. I’ll have a table for speakers, another for slots, and another for individual sessions. It’s important to give each table a primary key with the name <code>_ID</code>, using the system String <code>BaseColumns._ID</code>. This will allow me to easily create foreign key references, but also bind a database row to a list view later on. In my sessions table, <code>speaker_id</code> and <code>slot_id</code> are foreign key references to the speakers and slots table.</p>
<h3>Speakers</h3>
<ul>
<li>_ID: integer primary key autoincrement</li>
<li>name: text not null</li>
<li>bio: text</li>
<li>thumbail_url: text</li>
</ul>
<h3>Slots</h3>
<ul>
<li>_ID: integer primary key autoincrement</li>
<li>start_time: text not null</li>
<li>end_time: text not null</li>
</ul>
<h3>Sessions</h3>
<ul>
<li>_ID: integer primary key autoincrement</li>
<li>location: text not null</li>
<li>title: text not null</li>
<li>description: text not null</li>
<li>speaker_id: integer, foreign key references speakers(_ID)</li>
<li>slot_id: integer, foreign key references slots(_ID)</li>
</ul>
<h3>Session_Selections</h3>
<ul>
<li>_ID: integer primary key autoincrement</li>
<li>slot_id: integer, foreign key references slots(_ID), unique on conflict replace</li>
<li>session_id: integer, foreign key references sessions(_ID)</li>
</ul>
<p>That <code>Session_Selections</code> table could do with some explaining. It is used to track any sessions that a user has added to their schedule. For each slot a user can choose at most one session. The <code>UNIQUE</code> constraint will trigger a conflict if the same <code>slot_id</code> is inserted a second time into the database. With SQLite you can choose what happens on conflicts. Selecting <code>ON CONFLICT REPLACE</code> will delete the existing entry for that slot_id, and replace it with the new entry.</p>
<p>The <code>onCreate</code> call in your <code>SQLiteOpenHelper</code> subclass is a good opportunity to create your tables. I also use this function to load up the JSON files and parse them into the database. SQLiteDatabase has an <code>execSQL</code> method which you can use to create the tables. As this deals with Strings, I decided to formalise my table and column names using interface fields. This means I can access them in code and not have to worry about spelling mistakes causing a SQL syntax error:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">interface</span> <span class="token class-name">Tables</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">SPEAKERS</span> <span class="token operator">=</span> <span class="token string">"speakers"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">SLOTS</span> <span class="token operator">=</span> <span class="token string">"slots"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">SESSIONS</span> <span class="token operator">=</span> <span class="token string">"sessions"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">SESSION_SELECTIONS</span> <span class="token operator">=</span> <span class="token string">"session_selections"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"><span class="token keyword">public</span> <span class="token keyword">interface</span> <span class="token class-name">SpeakersColumns</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">SPEAKER_NAME</span> <span class="token operator">=</span> <span class="token string">"name"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">SPEAKER_BIO</span> <span class="token operator">=</span> <span class="token string">"bio"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">String</span> <span class="token constant">THUMBNAIL_URL</span> <span class="token operator">=</span> <span class="token string">"thumbnail_url"</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>An example of the SQL create table syntax:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line">db<span class="token punctuation">.</span><span class="token function">execSQL</span><span class="token punctuation">(</span><span class="token string">"CREATE TABLE "</span> <span class="token operator">+</span> <span class="token class-name">Tables</span><span class="token punctuation">.</span><span class="token constant">SPEAKERS</span> <span class="token operator">+</span> <span class="token string">" ("</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">BaseColumns</span><span class="token punctuation">.</span>_ID <span class="token operator">+</span> <span class="token string">" INTEGER PRIMARY KEY AUTOINCREMENT,"</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">SpeakersColumns</span><span class="token punctuation">.</span><span class="token constant">SPEAKER_NAME</span> <span class="token operator">+</span> <span class="token string">" TEXT NOT NULL,"</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">SpeakersColumns</span><span class="token punctuation">.</span><span class="token constant">SPEAKER_BIO</span> <span class="token operator">+</span> <span class="token string">" TEXT,"</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">SpeakersColumns</span><span class="token punctuation">.</span><span class="token constant">THUMBNAIL_URL</span> <span class="token operator">+</span> <span class="token string">" TEXT)"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<h2>Parsing the JSON</h2>
<p>The Android SDK includes the org.json JSON parsing library, and for a relatively-small set of JSON files, this is more than acceptable to use as a parser. A word of caution – this library builds an in-memory representation fo a JSON file, so for larger files could be a bit cumbersome. Either way, it’s important to do this loading on a background thread. Keeping I/O off the main thread is a good idea – if the main thread ‘hangs’ for more than 5 seconds, Android will terminate your application with an ‘Application Not Responding’ message.
To parse the speakers data, first I need to create a SQL String which will be executed for each row of data I’d like to insert:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token class-name">String</span> sql <span class="token operator">=</span> <span class="token string">"INSERT INTO "</span> <span class="token operator">+</span> <span class="token class-name">Tables</span><span class="token punctuation">.</span><span class="token constant">SPEAKERS</span> <span class="token operator">+</span> <span class="token string">" ( "</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">SpeakersColumns</span><span class="token punctuation">.</span><span class="token constant">SPEAKER_NAME</span> <span class="token operator">+</span> <span class="token string">", "</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">SpeakersColumns</span><span class="token punctuation">.</span><span class="token constant">SPEAKER_BIO</span> <span class="token operator">+</span> <span class="token string">", "</span></span>
<span class="highlight-line"> <span class="token operator">+</span> <span class="token class-name">SpeakersColumns</span><span class="token punctuation">.</span><span class="token constant">THUMBNAIL_URL</span> <span class="token operator">+</span></span>
<span class="highlight-line"> <span class="token string">" ) VALUES (?, ?, ?)"</span><span class="token punctuation">;</span></span></code></pre>
<p>As I’m inserting multiple rows, I’d like to wrap all of this up into a transaction. This makes the entire procedure atomic, in the sense of it fails or it doesn’t. A transaction can be started using <code>database.beginTransaction();</code> When the database operation is complete, call <code>database.setTransactionSuccessful();</code> If something bad happens, then a <code>SQLException</code> will be thrown. Therefore, it’s important to call <code>database.endTransaction();</code> in a finally block. If the end transaction call occurs without <code>setTransactionSuccesful();</code> being called first, the entire operation will be rolled back:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line">database<span class="token punctuation">.</span><span class="token function">beginTransaction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token class-name">SQLiteStatement</span> s <span class="token operator">=</span> database<span class="token punctuation">.</span><span class="token function">compileStatement</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token comment">// do database operations</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">setTransactionSuccessful</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">endTransaction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Firstly, I’ll read the JSON data from the raw resource into a String by constructing an <code>InputStream</code>:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line">database<span class="token punctuation">.</span><span class="token function">beginTransaction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token class-name">SQLiteStatement</span> s <span class="token operator">=</span> database<span class="token punctuation">.</span><span class="token function">compileStatement</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token class-name">InputStream</span> is <span class="token operator">=</span> ctx<span class="token punctuation">.</span><span class="token function">getResources</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">openRawResource</span><span class="token punctuation">(</span><span class="token class-name">R</span><span class="token punctuation">.</span>raw<span class="token punctuation">.</span>speakers<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">BufferedReader</span> reader <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">BufferedReader</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">InputStreamReader</span><span class="token punctuation">(</span>is<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">final</span> <span class="token class-name">StringBuilder</span> builder <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">StringBuilder</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token class-name">String</span> line <span class="token operator">=</span> <span class="token keyword">null</span><span class="token punctuation">;</span> <span class="token punctuation">(</span>line <span class="token operator">=</span> reader<span class="token punctuation">.</span><span class="token function">readLine</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">!=</span> <span class="token keyword">null</span><span class="token punctuation">;</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> builder<span class="token punctuation">.</span><span class="token function">append</span><span class="token punctuation">(</span>line<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token class-name">String</span> json <span class="token operator">=</span> builder<span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token comment">// Do database operations</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">setTransactionSuccessful</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">IOException</span> ex<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">Log</span><span class="token punctuation">.</span><span class="token function">d</span><span class="token punctuation">(</span><span class="token string">"IO"</span><span class="token punctuation">,</span> <span class="token string">"Failed to read R.raw.speakers"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> is<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">IOException</span> ex<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">endTransaction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>It’s worth noting that you should always close your <code>InputStream</code> in a finally block, and that this itself can throw an <code>IOException</code>!</p>
<p>Now that we have a String representation of the JSON, it’s time to do the parsing and insert each speaker into the database! The org.json library makes this fairly easy – each JSON object corresponds to a <code>JSONObject</code> instance. Elements inside an object can be accessed by name and it’s possible to iterate over a <code>JSONArray</code>. For each speaker, I use the compiled SQL statement and call a <code>bindX()</code> method to bind a database column to the appropriate value. When I’m done, I call execute on the <code>SQLStatement</code>. A word of warning – <code>SQLiteStatement</code> starts its index from 1, not 0! This is in contrast with <code>Cursor</code>, which is used to retrieve data, which indexes from 0. Sigh.</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line">database<span class="token punctuation">.</span><span class="token function">beginTransaction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token class-name">SQLiteStatement</span> s <span class="token operator">=</span> database<span class="token punctuation">.</span><span class="token function">compileStatement</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token class-name">InputStream</span> is <span class="token operator">=</span> ctx<span class="token punctuation">.</span><span class="token function">getResources</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">openRawResource</span><span class="token punctuation">(</span><span class="token class-name">R</span><span class="token punctuation">.</span>raw<span class="token punctuation">.</span>speakers<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">BufferedReader</span> reader <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">BufferedReader</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">InputStreamReader</span><span class="token punctuation">(</span>is<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">final</span> <span class="token class-name">StringBuilder</span> builder <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">StringBuilder</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token class-name">String</span> line <span class="token operator">=</span> <span class="token keyword">null</span><span class="token punctuation">;</span> <span class="token punctuation">(</span>line <span class="token operator">=</span> reader<span class="token punctuation">.</span><span class="token function">readLine</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">!=</span> <span class="token keyword">null</span><span class="token punctuation">;</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> builder<span class="token punctuation">.</span><span class="token function">append</span><span class="token punctuation">(</span>line<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token class-name">String</span> json <span class="token operator">=</span> builder<span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>json <span class="token operator">!=</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">JSONObject</span> obj <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSONObject</span><span class="token punctuation">(</span>json<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">JSONArray</span> speakers <span class="token operator">=</span> obj<span class="token punctuation">.</span><span class="token function">getJSONArray</span><span class="token punctuation">(</span><span class="token string">"speakers"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token class-name">JSONObject</span> speaker<span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">int</span> i <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator"><</span> speakers<span class="token punctuation">.</span><span class="token function">length</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> speaker <span class="token operator">=</span> speakers<span class="token punctuation">.</span><span class="token function">getJSONObject</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> s<span class="token punctuation">.</span><span class="token function">clearBindings</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> s<span class="token punctuation">.</span><span class="token function">bindString</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> speaker<span class="token punctuation">.</span><span class="token function">getString</span><span class="token punctuation">(</span><span class="token string">"name"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> s<span class="token punctuation">.</span><span class="token function">bindString</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">,</span> speaker<span class="token punctuation">.</span><span class="token function">has</span><span class="token punctuation">(</span><span class="token string">"bio"</span><span class="token punctuation">)</span> <span class="token operator">?</span> speaker<span class="token punctuation">.</span><span class="token function">getString</span><span class="token punctuation">(</span><span class="token string">"bio"</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token string">""</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> s<span class="token punctuation">.</span><span class="token function">bindString</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">,</span> speaker<span class="token punctuation">.</span><span class="token function">has</span><span class="token punctuation">(</span><span class="token string">"thumbnail_url"</span><span class="token punctuation">)</span> <span class="token operator">?</span> speaker<span class="token punctuation">.</span><span class="token function">getString</span><span class="token punctuation">(</span><span class="token string">"thumbnail_url"</span><span class="token punctuation">)</span> <span class="token operator">:</span> <span class="token string">""</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> s<span class="token punctuation">.</span><span class="token function">execute</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"></span>
<span class="highlight-line"> <span class="token comment">// Do database operations</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">setTransactionSuccessful</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">IOException</span> ex<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token class-name">Log</span><span class="token punctuation">.</span><span class="token function">d</span><span class="token punctuation">(</span><span class="token string">"IO"</span><span class="token punctuation">,</span> <span class="token string">"Failed to read R.raw.speakers"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token keyword">try</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> is<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">IOException</span> ex<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">endTransaction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token punctuation">}</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>The method for parsing sessions and slots is similar, but slightly more advanced. As each slot contains an array of sessions, I basically add a slot into the database using a different call, which returns the primary key of an newly-inserted row:</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token keyword">int</span> slotId <span class="token operator">=</span> database<span class="token punctuation">.</span><span class="token function">executeInsert</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span></code></pre>
<p>This is needed because a session requires a foreign key reference to the slot it is in. After a slot insertion, I iterate over the array of sessions for that slot, and insert those into the database normally. If a slot only has one session - such as a lunch break, I’ll map this <code>slot_id</code> and <code>session_id</code> as a user-selection in the <code>session_selections</code> table. If there’s more than one session for a slot, I don’t add anything into this table - I’ll leave that up to the user to decide!</p>
<h2>Upgrading your database</h2>
<p>You can increment the version number of your database as part of an application update, and your <code>SQLiteOpenHelper</code> will handle this by calling <code>onUpgrade</code>. Should this happen, I opt to dump the database and re-build it. To the organisers of DDD East Anglia – don’t go changing any sessions!</p>
<pre class="language-java"><code class="language-java"><span class="highlight-line"><span class="token annotation punctuation">@Overridepublic</span> <span class="token keyword">void</span> <span class="token function">onUpgrade</span><span class="token punctuation">(</span><span class="token class-name">SQLiteDatabase</span> database<span class="token punctuation">,</span> <span class="token keyword">int</span> oldVersion<span class="token punctuation">,</span> <span class="token keyword">int</span> newVersion<span class="token punctuation">)</span> <span class="token punctuation">{</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">execSQL</span><span class="token punctuation">(</span><span class="token string">"DROP TABLE IF EXISTS "</span> <span class="token operator">+</span> <span class="token class-name">Tables</span><span class="token punctuation">.</span><span class="token constant">SPEAKERS</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">execSQL</span><span class="token punctuation">(</span><span class="token string">"DROP TABLE IF EXISTS "</span> <span class="token operator">+</span> <span class="token class-name">Tables</span><span class="token punctuation">.</span><span class="token constant">SLOTS</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">execSQL</span><span class="token punctuation">(</span><span class="token string">"DROP TABLE IF EXISTS "</span> <span class="token operator">+</span> <span class="token class-name">Tables</span><span class="token punctuation">.</span><span class="token constant">SESSIONS</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> database<span class="token punctuation">.</span><span class="token function">execSQL</span><span class="token punctuation">(</span><span class="token string">"DROP TABLE IF EXISTS "</span> <span class="token operator">+</span> <span class="token class-name">Tables</span><span class="token punctuation">.</span><span class="token constant">SESSION_SELECTIONS</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"> <span class="token function">onCreate</span><span class="token punctuation">(</span>database<span class="token punctuation">)</span><span class="token punctuation">;</span></span>
<span class="highlight-line"><span class="token punctuation">}</span></span></code></pre>
<p>Now I have an SQLite database populated with data from some JSON files. I’m still debating on the best way to retrieve this data from the database. Shall I deal with my database directly, or indirectly through a <code>ContentProvider</code>?</p>
DDD East Anglia App, Part 1: Initial Setup2013-06-16T00:00:00Zhttps://samhogy.co.uk/2013/06/ddd-east-anglia-app-part-one/<p>I’m attending <a href="http://dddeastanglia.com/">DDD East Anglia</a> at the end of the month. I thought I’d create an Android scheduling application, so that conference attendees can view a list of sessions and store which ones they’d like to go to. Although in my day job I develop an Android charting library, I have no experience pushing an actual application out into the world through Google Play. This gives me the perfect opportunity to remedy this!</p>
<p>Using the ADT version of Eclipse, I created a new Android Application Project. As part of the initial project setup wizard, you have the option of adding a custom icon. I took the image of King’s College Chapel from the DDD East Anglia logo and increased the brightness in Paint.NET until it was all white. I then imported this into the wizard, on one of the stock backgrounds. You can customise the colour of your background, too.</p>
<img src="https://samhogy.co.uk/ssets/android/ic_launcher.png" alt="DDD East Anglia launcher logo" />
<h2>The Action Bar</h2>
<p>Android applications from Honeycomb-onwards come with an ActionBar. This is a strip at the top of an application displaying the logo and title. I want the main application screen to have three tabs:</p>
<ul>
<li>The schedule, containing the user’s selected sessions.</li>
<li>A twitter feed displaying the #dddea and #DDDEastAnglia hashtags.</li>
<li>General information about the venue, such as location and organiser contact details.</li>
</ul>
<p>The Action Bar is perfect for this – tabs are displayed within the ActionBar if there is room, such as on a 7” or 10” device. When there is no room, the tabs move down onto the next line. This behaviour happens out of the box – no complex configuration required!</p>
<p>ADT also has a pre-configured template to create an Activity with three fixed Action Bar tabs. For now, I auto-generated me up a main activity. The default layout displays the tab number on each tab. That’s good enough until I actually have data to display.</p>
<p>I’d like to target as many devices as consistently as possible. Developers frequently complain about Android being a fragmented platform that is cumbersome to develop for. As I recently mentioned at my “Android Strokes for iOS Folks” talk, the framework actually helps you out quite a lot. But not with the Action Bar. Or at least, not yet. Google announced that the Android Support Library, which contais lots of helper classes for backporting new framework features to older platforms, will soon include a backwards-compatible Action Bar. As that’s not released publicly yet, I’ll be using an open-source version – <a href="http://actionbarsherlock.com/">ActionBarSherlock</a>.</p>
<p>Including ActionBarSherlock is as simple as downloading the source, and importing it into Eclipse as an Android Library Project. Next, you specify in your project’s properties to use ActionBarSherlock as a library project.</p>
<p>ADT generates multiple different styles.xml files for you, to ensure that the base theme for your app is consistent with the underlying framework version. ActionBarSherlock can remove all of this boilerplate, as it provides a custom implementation of styling for later platforms, so you can delete all those styles and replace it with:</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Theme.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Theme.Sherlock.Light<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> <item name=<span class="token string">"android:actionBarStyle"</span>>@style/ActionBar.Solid.Dddeastanglia</item></span>
<span class="highlight-line"> <item name=<span class="token string">"actionBarStyle"</span>>@style/ActionBar.Solid.Dddeastanglia</item></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span>
<span class="highlight-line"></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ActionBar.Solid.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Widget.Sherlock.Light.ActionBar.Solid<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> …</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span></code></pre>
<p>You need to point your manifest to use this theme:</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>application</span>
<span class="highlight-line"> <span class="token attr-name"><span class="token namespace">android:</span>name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>uk.co.samhogy.dddeastanglia.DDDEastAngliaApplication<span class="token punctuation">"</span></span></span>
<span class="highlight-line"> <span class="token attr-name"><span class="token namespace">android:</span>icon</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@drawable/ic_launcher<span class="token punctuation">"</span></span></span>
<span class="highlight-line"> <span class="token attr-name"><span class="token namespace">android:</span>label</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@string/app_name<span class="token punctuation">"</span></span></span>
<span class="token attr-name"><span class="token namespace">android:</span>theme</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Theme.Dddeastanglia<span class="token punctuation">"</span></span><span class="token punctuation">></span></span></code></pre>
<h2>Styling the Action Bar</h2>
<p>The ActionBar is also a great branding opportunity for your app. I’d like for the background colour to match the green colour scheme of the website. I can easily do this using the <a href="https://android-ui-utils.googlecode.com/hg/asset-studio/dist/index.html">Android Asset Studio</a> – an open-source set of web tools. You can create action bar styles for the framework or ActionBarSherlock implementation. It’s as simple as specifying what colours you’d like each element to take. When you’re done, all the necessary resources are zipped into a folder structure mimicking your res project directory, so including this theme is as simple as unzipping in that location.</p>
<p>As I’m using a dark Action Bar colour with Holo.Light as my underlying theme, some of the default icons and text don’t look right. Black text on a dark green background is hideous, but can be fixed by manually tweaking the styles.xml:</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ActionBar.Solid.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Widget.Sherlock.Light.ActionBar.Solid<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> ...</span>
<span class="highlight-line"> <item name=<span class="token string">"android:titleTextStyle"</span>>@style/ActionBar.TitleText.Dddeastanglia</item></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ActionBar.TitleText.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@android:style/TextAppearance<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> <item name=<span class="token string">"android:textColor"</span>>@<span class="token property">android</span><span class="token punctuation">:</span>color/white</item></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span></code></pre>
<img src="https://samhogy.co.uk/ssets/android/dddea_titletext.png" alt="Action Bar title text, black and white." />
<p>This fixes up my Action Bar title text just fine, but I noticed that in landscape mode, when the Action Bar tabs can be fit on the Action Bar (rather than on a different row), I have the same problem. So, how can I make the tab text white in landscape mode, but black in portrait? Easy!</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Theme.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Theme.Sherlock.Light<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> ...</span>
<span class="highlight-line"> <item name=<span class="token string">"actionBarTabTextStyle"</span>>@style/ActionBar.TabTextStyle.Dddeastanglia</item></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span>
<span class="highlight-line"></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ActionBar.TabTextStyle.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Widget.Sherlock.ActionBar.TabText<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> <item name=<span class="token string">"android:textColor"</span>>@color/tab_text</item></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span></code></pre>
<p>This is referencing a <em>@color</em> resource. There are two variants of this:</p>
<p>in values: <color name="tab_text">@android:color/black</color>
in values-sw320dp-land: <color name="tab_text">@android:color/white</color></p>
<p>At run-time the style is resolved to use the correct resource! But wait, what’s up with the ‘sw320dp’ part in that folder name? On devices with a width smaller than 320 device-independent pixels, there is not enough room to move the action bar tabs into the action bar, so they will stay in the split mode. In this instance, I need the text to stay black. So, adding ‘sw320dp’ means that the white colour is only used on devices in landscape mode, whose width is larger than 320dp. Admittedly, it was trial and error with emulators to get that figure correct, but it is important to give each platform the best experience possible.</p>
<p>One more very subtle issue. The system icon set for Holo.Light uses a dark grey to contrast with the (expected) light gray backgrounds of the application. The ‘up’ icon, which is a left-facing caret displayed to the left of the application logo in the Action Bar, will be hard to see. To remedy this, I downloaded the Holo.dark icon set from the <a href="http://developer.android.com/design/downloads/index.html">Android Design web page</a>. I placed the new logo in the appropriate <strong>drawable</strong> folder for each variant of the ic_up png. All you need to do is tell your style to use this resource, not the standard system resource:</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Theme.Dddeastanglia<span class="token punctuation">"</span></span> <span class="token attr-name">parent</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@style/Theme.Sherlock.Light<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="highlight-line"> <item name=<span class="token string">"homeAsUpIndicator"</span>>@drawable/ic_up</item></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span></code></pre>
<img src="https://samhogy.co.uk/ssets/android/dddea_upicon.png" alt="Action Bar up icon" />
<p>Here’s what the application looks like in both orientations:
<img src="https://samhogy.co.uk/ssets/android/dddea_landscape.png" alt="Landscape view of app, with tabs inside action bar" />
<img src="https://samhogy.co.uk/ssets/android/dddea_portrait.png" alt="Portrait view of app, with tabs below action bar" /></p>
<p>That’s not too bad at all, especially seeing as most of this is auto-generated and only a bit of style-wrangling was needed. Tomorrow, I’ll talk about setting up the database.</p>
Viral Ingress2013-05-10T00:00:00Zhttps://samhogy.co.uk/2013/05/viral-ingress/<p>The most recent Ingress report alludes to new scanner technology for the near future – entities known as the Jarvis Virus and the ADA Refractor. Both of these are deployed to an enemy portal to effectively ‘flip’ their alignment over to your faction.</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/wMBpl3sRZlg?rel=0" frameborder="0" allowfullscreen></iframe>
<p>Not much is known about these new mods (herein known as viruses for simplicity), but from the initial proposal, I am rather skeptical about them bringing a positive impact to the game, and I’ll like to propose an alternative.</p>
<h2>What’s Wrong</h2>
<p>These viruses appear to be too close to a god weapon, undermining all of the recent work encouraging people to group together to defend/attack, as detailed in my <a href="http://samhogy.co.uk/2013/05/ingress-socialize-to-win/">previous Ingress post</a>. If well-linked portals can be simply flipped, then the incentive to level up portals to very high levels isn’t as strong – the enemies will just take them. Chances are the portal’s links will decay when a portal is flipped, as it seems unreasonable for the virus mod to travel to all linked portals and flip their alignment too!</p>
<p>What this does create, should a player store up lots of virus mods, is the ability to almost effortlessly create a high-level portal farm. Unless there is a significant XM cost to deploying the virus (which can be remedied using power cubes anyway), then it will be possible for an entire team to quickly-change a few high level portals, farm them until they burn out and then rampage around a city. This might be fun in the sense of the back-and-forth action it creates, but in a game where attacking has always been favourable to defending, this does nothing to address that balance. Having an ‘all-or-nothing’ weapon seems too powerful.</p>
<h2>My Suggestion</h2>
<p>A virus mod should have a level associated with it, just like resonators, bursters and power cubes. When it is deployed to an enemy portal, it attacks that portal (rather than instantly flips) and then <strong>spreads</strong> along the portals links! When it gets to those portals, it <strong>attacks them too</strong>! The strength of the attack is determined by the level of the virus mod. As it travels down links, the attack strength decays proportional to the length of the link, such that linked portals close by will suffer lots of damage and further away portals take less.</p>
<p>The effect of this is to avoid the ‘one man army’ player who can roll into town with lots of virus mods, flip a lot of portals and them level them up further. Instead, it brings a tactical dynamic to the game, as defenders need to be aware of calculating an optimal link density around a portal. Right now, the incentive is to defend by linking portals as much as possible. But, if that very defensive action could be turned against the faction if taken too far, you’ll end up with a far more interesting challenge.</p>
Ingress, where the only winning move is to socialize2013-05-02T00:00:00Zhttps://samhogy.co.uk/2013/05/ingress-where-the-only-winning-move-is-to-socialize/<p>I’ve been playing <a href="http://www.ingress.com/">Ingress</a> for roughly two months. So far I’ve reached level 5, I’m now 50% of the way to level 6. One of the things I’ve wanted to do is write about the mechanics of game balance in Ingress, but that post has been sitting as a draft for a few weeks. Something has since then came along and changed the picture quite considerably!</p>
<p>Recently, a <a href="https://plus.google.com/u/0/103320655754019011706/posts/1RJhPYjDoo4">game-changing update</a> came into force: portals linked to other portals are more difficult to destroy. I think this is a fantastic move, despite being in a city completely dominated by the opposing faction! (Boo, Smurfs).</p>
<p>As you progress through the levels of Ingress, you start to care less about the ownership of an individual portal and more about the spatial distribution of portals. Higher-level players take on a social role of weakening portals so that low level players can finish them off and reap the AP. Higher-level players think about creating pop-up portal farms – that is, capturing lots of portals close together and hacking them until they burn out (which is four hacks per four hours on each portal).</p>
<p>The problem with this approach is that as you get to higher levels there is little incentive to creating links and fields, unless you’re one of the dedicated players who will try to establish fields the length of a country just for fun. Destroying links and fields gives the opponents more AP. Once you get to level 8, the max level, there’s no incentive to link because you don’t need the AP! Previously, adding lots of links and fields to a portal added no value to your team, instead they act as a big advertising billboard for the enemies – COME GET ME!</p>
<p>Now, highly-linked portals have higher defensive capabilities, meaning they can be exploited for weapons by your team. Combined with another update, that firing a weapon requires you to expend some XM, this means that there is an incentive for a team to build and maintain highly-linked portals. After all, in the storyline this is a global competition to create fields, yet the end game discouraged this! Historically Ingress has always been in favour of the attacker – even a well-stocked level 6 player could decimate an entire town. This redresses the balance a bit.</p>
<p>Of course, attackers have a new item too. A power cube can be obtained from hacking a portal. They essentially act as a battery, restoring XM up to a certain value. So if you’re planning a big attack, you have the opportunity to recharge yourself a few times. Don’t leave home without them!</p>
<p>Earlier, I mentioned that this was a good idea even in one-sided environments. Newcastle-Upon-Tyne is heavily controlled by the Resistance. Their portals are well-linked and well-maintained:</p>
<img src="https://samhogy.co.uk/ssets/ingress-blue-domination-624x606.png" alt="A map of the city of Newcastle-Upon-Tyne, overlaid with the corresponding Ingress portals. The region is 90% blue." width="300" height="291" />
<p>Any attack on their portals is fruitful, but temporary. Due to the fact that Ingress favoured attackers, we have slowly managed to pull ourselves up through the levels. We can now nearly give as good as we get, but there’s a few of us who still need to level up a bit more to be effective. How did we do this? By <strong>organising</strong> and <strong>socializing!</strong> These have always been the goals of Ingress – you’re a faction, after all! Asides from the benefit of playing a game outdoors and discovering lots of hidden gems within your city, you get to meet new people and take point in a spontaneous group experience. This is why we play games, not for the predictability, not even for the outcomes, but for the experience. Seeing a Resistance member failing to defend a portal being attacked by 5 or 6 Enlightened is extremely funny and rewarding.</p>
<p>The only way to play this game, is to socialize. Organize days to create pop-up farms, or to take down a well-linked enemy portal. Use your invites to recruit more members to your faction, then use your higher-level players to help them level up as quickly as possible. Submit new portals to stretch the enemy resources a tiny bit further out their comfort zone, and enjoy the company of your team mates!</p>
ORGCon North - with impromptu me!2013-04-18T00:00:00Zhttps://samhogy.co.uk/2013/04/orgcon-north-with-impromptu-me/<img src="https://samhogy.co.uk/ssets/ORGConNorth.jpg" alt="ORGCon North Logo" width="300" height="97" />
<p>Last Saturday, I attended the first <a href="http://www.openrightsgroup.org/blog/2013/orgcon-north-summary">ORGCon North event</a>, hosted by the Open Rights Group. Speakers included John Buckman (chair of the EFF), Loz Kaye (leader of the UK Pirate Party), lots of other excellent people and… me!</p>
<p>The latter half of the conference was reserved for <a href="http://en.wikipedia.org/wiki/Unconference">unconference</a> slots. This was a new idea to me, but the general principle is that sessions are proposed by regular conference-goers. There’s no obligation to actually chair a session you propose, but putting forward some idea of how the session should be ran is quite important. So, what are unconference sessions about? Maybe you have something to show, or some knowledge to share? Maybe you just want to hold a discussion, gather ideas? This is the opportunity! If you proposal is appealing, others can vote on it – the highest-voted sessions are given some time and a room!</p>
<p>I proposed a session on good password security principles and it got accepted! I thought that it would be appropriate, given the recent high-profile exploitations of sites such as LinkedIn and Evernote, to talk about the severe implications of password breaches. You don’t only run the risk of losing your own personal information – there’s a lot more at stake. If you’re a journalist, your exclusive scoop may be stolen, or even the identities of confidential sources leaked.</p>
<p>In the session, we discussed how using the same password everywhere is not appropriate. Using that strategy in today’s disparate online environment, one breach can cascade into several. However, this moves the problem onto remembering all your passwords. Given the increasingly complex security requirements this problem is difficult! So, we talked about the merits and flaws of <a href="https://lastpass.com/">LastPass</a>, <a href="http://keepass.info/">KeePass</a> and <a href="https://www.pwdhash.com/">PwdHash</a>. There was no clear winner here, although some preferred how PwdHash doesn’t keep a centralised list of all your passwords.</p>
<p>Bruce Schneier described three types of authentication: something you are, something you know, or something you have. With this in mind, I introduced the concept of two-factor authentication, such as <a href="http://www.yubico.com/products/yubikey-hardware/yubikey/">YubiKey</a> and <a href="http://support.google.com/accounts/bin/answer.py?hl=en&answer=180744">Google Authenticator</a>, as a means of further restricting access to online accounts.</p>
<p>A brief diversion into the differences between http and https (it’s not just a golden padlock) led into an unnervingly-common case where insecure http calls can be made over https connections. Some browsers block this by default, others will in later versions. The implications of this have been <a href="http://www.troyhunt.com/2013/04/5-ways-to-implement-https-in.html">detailed wonderfully by Troy Hunt</a>.</p>
<p>The session then came to a close! The one major point that, due to time constraints, I didn’t get to mention was that once the problem of cracking passwords becomes too difficult, the bad guys move the problem onwards and turn to social engineering. Vigilance is required in addition to good passwords.</p>
<p>I’m very thankful to the attendees for contributing so much to the conversation and I hope they all came away having learnt something new! If you’re thinking about developing your public speaking skills – I can highly recommend proposing an unconference session!</p>
Castle Panic2013-02-04T00:00:00Zhttps://samhogy.co.uk/2013/02/castle-panic/<p><a href="http://firesidegames.com/castlepanic.html">Castle Panic</a> is a co-operative tower defense board game, in which players have to defend their castle from a horde of monsters. It’s a fantastic game with enough charm and simplicity to appeal to children and enough strategy to appease adults.</p>
<p>I discovered this game when it was featured on an episode of Tabletop, a web series hosted by Wil Wheaton. Since then I picked up the game at Travelling Man and it’s been on my desk at work since. Occasionally I play board games on a lunch break with some colleagues and Castle Panic is a favourite.</p>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/MpuTGWFkBYs?rel=0" frameborder="0" allowfullscreen></iframe>
<h2>How do I play it?</h2>
<p>Your castle is positioned at the middle of the board. Four concentric circles span outwards, with the circle being divided into six arcs. Each arc is coloured either red, green or blue. Monsters progressively move towards the castle. Should they reach the centre, they will destroy a wall of your castle, or a tower! The game ends when all the monsters are killed, or when all the castle towers have been demolished.</p>
<p>To ward off the monsters, players have archer, knight and swordsmen cards. These are used to hit enemies within a certain ring. These are also coloured, so they are only effective in that colour. The strategy of the game involves trading cards between players to make sure the monsters can be slain whilst minimising damage to your castle.</p>
<p>The difficulty is increased by boss monsters (which, when summoned, have some devastating effect upon the board), or by the fun giant boulder mechanic. When played, a giant boulder starts in an arc and rolls towards a castle tower, knocking down everything in its way! This can sometimes produce hilarious results, with multiple monsters wiped out and the boulder casually passing through the castle! Not all monster tokens are actually monsters – some will attempt to disrupt your strategy by re-arranging the monster tokens on the board, or even summoning more monsters.</p>
<p>For a game whose rules can be picked up in literally one turn, there’s an excellent mix of luck, depth and sheer panic to make this an excellent family or party game and is truly worth adding to any board game collection. There are several variants of the rules to allow for differing gameplay, so there is plenty of mileage should you get tired with the original rules.</p>
<p>I already have the <a href="http://firesidegames.com/thewizardstower.html">Wizards’s Tower expansion</a> on order. In equals measures I’m excited and terrified to play it!</p>
PropertyCross2012-12-21T00:00:00Zhttps://samhogy.co.uk/2012/12/propertycross/<p>Today, the open source project I have been contributing to has been ‘released’! <a href="http://propertycross.com/">PropertyCross</a> was founded by my colleagues <a href="http://www.scottlogic.co.uk/blog/chris">Chris Price</a> and <a href="http://www.scottlogic.co.uk/blog/colin">Colin Eberhardt</a> to help developers choose a cross-platform mobile application development framework. We’re aiming to be to the <a href="http://addyosmani.github.com/todomvc/">TODOMvc</a> for mobile development. Check out <a href="http://www.scottlogic.co.uk/blog/colin/2012/12/introducing-propertycross-helping-you-select-a-cross-platform-mobile-framework/">Colin’s introductory post</a> for an overview of the 14 applications we’ve developed so far.</p>
<img src="https://samhogy.co.uk/ssets/PropertyCross.png" alt="PropertyCross" width="600" height="186" />
<p>So far, I’ve contributed the <a href="http://propertycross.com/xamarin/">Xamarin Android</a> implementation and as a Christmas coding project I’m developing the <a href="http://propertycross.com/native/">native Android</a> version. We’re supplying native implementations of an application on each framework to act as a benchmark for the cross-platform implementations.</p>
<p>This is the first open-source project I have contributed to. There are many articles describing why working on open-source projects is beneficial. Personally I’ve found that it breaks you out of your comfort zone, presents the opportunity to learn new technologies, processes and is a great way of building reputation within the community. I’ve had the opportunity to learn git, improve my Android skills and delve into Xamarin’s Mono for Android framework.</p>
<h2>Contribute!</h2>
<p>Contributions are welcomed – we don’t bite! <a href="https://github.com/colineberhardt/PropertyCross/">We are on Github</a>. In my opinion this is a space which is going to move very quickly – frameworks which are popular now may not be in a year, <a href="http://www.w3.org/News/2012#entry-9667">html5 is now finalized</a> and on top of that the mobile operating systems are evolving rapidly. Opportunities like this are a great chance for developers to diversify their skills.</p>
Open Rights Group, North East: Our first event!2012-11-03T00:00:00Zhttps://samhogy.co.uk/2012/11/open-rights-group-north-east-our-first-event/<p>I am a member of the <a href="http://www.openrightsgroup.org/">Open Rights Group</a>, a campaign organisation promoting civil liberties, transparency and openness. I’ve been working closely with <a href="http://milenapopova.eu/">Milena Popova</a> on establishing a North East working group, where members and non-members alike can meet to freely discuss issues, hear talks from interesting speakers and collaborate on future campaigns.</p>
<p>Our first event is penciled in at the Star and Shadow Cinema (Stepney Bank, Newcastle, NE1 2NP) from 4-6pm on <strong>Sunday 11th November 2012</strong>. If you have the time free, I’d love to see you there!</p>
<p>For now, here’s some useful links that it’s worth keeping an eye on:</p>
<ul>
<li><a href="http://www.facebook.com/ORGNortheast">Facebook</a></li>
<li><a href="http://www.twitter.com/ORGNorthEast">Twitter @ORGNorthEast</a></li>
<li><a href="http://lists.openrightsgroup.org/listinfo/org-northeast">Mailing List signup page</a></li>
</ul>
<p>We’re looking for people to speak at future events or help out in any way. If this is something you’d like to do then please contact me and we can start a discussion.</p>
Dispatches from DDDNorth 22012-10-15T00:00:00Zhttps://samhogy.co.uk/2012/10/dispatches-from-dddnorth/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2012/10/15/dispatches-from-dddnorth-2.html">the Scott Logic blog</a></span></div>
<p>Last weekend I traveled to Bradford for the second <a href="http://www.dddnorth.co.uk/">DDDNorth</a> - a community-organised developer event which featured some excellent speakers. DDD events are held all over the country throughout the year and are supported by local users groups. I highly recommend actively seeking out and attending these events, they are a great way of picking up new ideas and insights into technologies you don't get a chance to normally play around with. I'd like to now report on the five talks that I attended through the course of the day.</p>
<img src="https://samhogy.co.uk/ssets/dddnorth.png" alt="DDDNorth logo" width="250" height="188" />
<h2>Lions and tigers and hackers! Oh my! - Phil Winstanley</h2>
<p>In my opinion this was a great opening talk from Phil. My only complaint was that this was not a keynote! Phil's talk, bursting full of anecdotes from a wide range of domains, demonstrated the urgency of the problem. We need to get better at security. To do that we need to start thinking seriously about security from day one.</p>
<p>We're moving more of our lives into the virtual - our money, our data and even our identities. This is a honey pot of unimaginable size, unknowable consequences, and is increasingly targeted over modern, more secure operating systems. Not only that, we are connecting disparate networks together and accessing content with mobiles devices. This introduces new security threats. The threat comes from hackers, organized criminals, rival companies and even governments.</p>
<p>Phil mentioned Microsoft's <a href="http://www.microsoft.com/security/sdl/discover/default.aspx">Security Development Lifecycle Process</a>, designed to be a robust complement to standard models of software development. This is now deployed in all areas of Microsoft and Phil provided some stories to show how this has benefitted the company. As developers, we should appreciate that security is not a product, it is a process and a policy. Security is an essential, every day task. We should, at all stages in the software development lifecycle, analyse our products from a security point-of-view by performing threat modelling, discussing privacy, removing deprecated code and devising incidence response plans.</p>
<h2>Test all the things or maybe not! - Will Charles</h2>
<p>As promised, this was a pragmatic talk about TDD. Typically, code examples demonstrating TDD are so simple and detached from reality that they do not scale well for use in production code. Adding the extra complexity, with mock object frameworks for example, can often hinder test quality. Will's talk focused on the tips and tricks he has found help apply the principles of test-first development in the domain of a complex system.</p>
<p>Fundamentally, we add tests to code to demonstrate the correctness of our code, which has some demonstrable business value. This has caused people to advocate 100% code coverage and for code coverage to be seen as some form of quality metric. Put simply, code coverage is a guide and does not necessarily correlate with some concept of value. Focus instead on testing the areas of code which deliver business 'value' - this is going to be code that is complex, likely to change and likely to contain bugs.</p>
<p>Make your tests work for you. Meaningful test names provide an abstract overview of what unit of value you are testing. Using a domain-specific-language by encapsulating arrange, act and assert stages into meaningful method names goes a long way into making your test code 'production-quality'.</p>
<p>Finally, Will revisited Michael's Feathers' concept of sprouting as a means of introducing unit tests into legacy code. He has found using this technique identifies potential points of change and areas of responsibility in the code. These are areas ripe for refactoring and once you have tests in place, they can be used as a means of evolving the legacy code. Overall, this was the most practical talk of the day in my opinion and lead quite nicely into the next talk...</p>
<h2>Bdd - Look ma, no frameworks! - Gemma Cameron</h2>
<p>Gemma's talk focused on a very raw form of behaviour-driven development, which rings very true and was widely appreciated by the session attendees. Behaviour-driven development centres on a shared understanding through the discussion of examples. It is a marriage of test-driven and domain-driven design, not with a focus on a particular framework, but on the ongoing conversation between developers, testers, business analysts and product owners.</p>
<p>Gherkin syntax and tools such as Fitnesse and Specflow are often heralded as a pinnacle of requirements capture and developer-business coordination. Very often they're just a starting point to kick off conversations. Gemma argues, like Will, that we should aim to use domain-specific-language when capturing our requirements in an effort to eradicate areas of ambiguity and capture requirements in a pure, readable and (potentially) executable form.</p>
<p>The take-home message for me that conversations should be the primary focus. Us developers should work at getting better at this - we will be respected more for it. You may choose to use tools and frameworks to capture the requirements and examples which result from these conversations, but we should never become slaves to them. Systems are complex because problems are complex and there is value in solving them. The nature of this complexity is such that the solution cannot be skimmed from the surface. We need to delve deep to truly understand and manage the complexity. Doing this effectively requires various areas of expertise and is a collective experience. From this process we derive good-quality software which meets the expectations of the product owners.</p>
<h2>JavaScript sucks and it doesn't matter - Rob Ashton</h2>
<p>JavaScript should come with a warning - it will make you lose hair, and the hair you don't lose will turn grey. Many of the humorous examples demonstrated in the infamous <a href="https://www.destroyallsoftware.com/talks/wat">WAT?</a> video are largely to do with JavaScript's approach to scoping and type coercion. Rob argued that this makes JavaScript a fairly easy target for naysayers, but these issues don't really matter. Tools such as JsLint (and it's less authoritarian cousin JsHint) will point out where you have went wrong. There has been a large developer focus on integrating these tools into development environments and continuous integration builds - as an example, my colleague Luke Page has released a <a href="https://jslint4vs2010.codeplex.com/">Visual Studio plugin for JSLint</a>. By doing so, developers can get immediate feedback of potential errors and take action to correct these.</p>
<p>An interesting tangential point was made about Typescript. As a concept it has been <a href="htt://scottlogic.com/blog/2012/10/02/typescript-versus-closure">well-received</a> - but it has its limits. Greenfield development may benefit more from the Typescript than a project which essentially 'glues' multiple frameworks together. This is certainly worth some detailed research and is great fodder for a blog post or even a future DDD talk! (Hint, hint, etc).</p>
<p>Testing is possible in JavaScript, without the need for browser automation! There are tools such as Zombie and PhantomJS which simulate a browser and work extremely well for integrating into a continuous integration environment. Rob's talks are always entertaining but remain insightful and practical. From this talk I have learned a lot more about the JavaScript landscape, and to move as quickly as possible to the fifth stage of JavaScript denial: Acceptance. JavaScript's here to stay, let's make the best of it.</p>
<h2>Event-Driven Architectures - Ian Cooper</h2>
<p>Finally, Ian Cooper delivered a comprehensive, accessible talk on an entirely new concept to me: event-driven architecture, or service-oriented architecture. At an extremely distilled level, this is where an application is sliced into logical parts and a service is aligned with each individual part. These logical divisions correspond to some piece of business capability and are exposed to the outside world via a data contract. This boundary must be explicit and well-defined. JSON and XML are commonly used as a means of describing the data contract.</p>
<p>Inter-service communication is achieved through events, and Ian took us on a whistle-stop tour on the various techniques that you can use to implement event dispatching and processing. He then progressed onto more complex aspects, such as how to maintain consistency between services and how to handle erroneous situations through the use of sagas, orchestration services and more event processing. Ian also talked in some depth about how to best deploy caching and cataloguing data as a means of improving the overall efficiency of an event-driven system.</p>
<p>For someone new to this kind of architecture, my overall impression is that this is an extreme form of loose-coupling. Only schema and data contracts are exposed at boundaries - not types. Everything inside a service is entirely encapsulated, right down to the data storage mechanisms (only one service has write-access to a specific logical area of data storage). This naturally gives us the means to perform unit and integration testing. However, there is certainly a large degree of complexity in the interoperation between services that requires an extensive amount of coordination to overcome. I need to make a point of learning more about this architecture!</p>
Template: The Exuberant Title Top Pattern in PowerPoint2012-09-17T00:00:00Zhttps://samhogy.co.uk/2012/09/template-the-exuberant-title-top-pattern-in-powerpoint/<p>I’ve recently read <a href="http://www.amazon.co.uk/Presentation-Patterns-Techniques-Crafting-Presentations/dp/0321820800/ref=sr_1_1?ie=UTF8&qid=1347915012&sr=8-1">Presentation Patterns</a>, a ‘gang of four’-style book containing dos and donts for creating presentations. One of the patterns I was particularly impressed by was the exuberant title top. It sounds simple – a title is displayed in the centre of a slide.</p>
<img src="https://samhogy.co.uk/ssets/ett-starting-position-1024x575.png" width="300" height="168" title="PowerPoint slide, with the title at the top, with bullets underneath." />
<p>On click, the title moves to the top of the page, leaving room for some content to be presented. The overall effect is that the content pushes the title to the top of the page.</p>
<p>It is fairly simple to implement this pattern if you are not concerned with the look of the slide if it is printed in handout format. When the slide is displayed as a handout, it is expected that the title will be displayed in its final position with the other content directly underneath. This means that the view in the Designer should match the final state of the slide.</p>
<img src="https://samhogy.co.uk/ssets/ett-final-position-1024x575.png" width="300" height="168" title="PowerPoint animation pane screenshot describing the aforementioned animation sequence." />
<h2>Implementation Guide</h2>
<p>Personally, I found the method presented for PowerPoint difficult to follow. One step specifies you to set a duration to 0, which does not seem to be possible in PowerPoint 2007, 2010 and the 2013 preview. So, I have found what I believe to be a far more effective method, which requires one less click to execute. Here’s how to do it:</p>
<ul>
<li>Create the title, in its top position.</li>
<li>Add a ‘Disappear’ exit animation to the title and set it to <strong>start with previous</strong>. This will mean the animation starts instantaneously.</li>
<li>Add a motion path animation. This is a downwards path and moves the title to the centre of the slide. Set this animation to <strong>start with previous</strong>, and manually change its duration to 0.01 seconds. This seems to be the smallest value that you can set a duration to and although it is not an option presented in the Duration dropdown menu, you can just type the value in.</li>
<li>Add an ‘Appear’ entrance animation to the title and set it to <strong>start after previous</strong>. This means that the animation to the starting position is hidden from the user.</li>
<li>Add a motion path animation. This should be the exact opposite of the ‘down’ motion path from step three. Set this animation to run <strong>on click</strong>. This gives the presenter the opportunity to talk about the title, before introducing the related content.</li>
<li>Add the related content to the slide.</li>
<li>Add an ‘Appear’ entrance animation to the related content. Set the duration to whatever value you feel comfortable with (I like the very fast option). It should <strong>start after previous</strong>. This can alternatively be replaced with a ‘Fade’ animation.</li>
</ul>
<img src="https://samhogy.co.uk/ssets/ett-standard-sequence.png" width="214" height="259" alt="PowerPoint animation pane screenshot describing the aforementioned animation sequence." />
<p>If you’d like to display each line (or bullet point) of your related content individually, this is possible using the advanced animation menu. In the ‘Text Animation’ tab, set the “Group text” value to “1st Level Paragraphs”.</p>
<img src="https://samhogy.co.uk/ssets/ett-advanced-options.png" width="382" height="328" alt="PowerPoint screenshot showing how to set an animation 'after previous'." />
<p>This results in each bullet point / line having its own separate (albeit related) animation entry in the animations pane. From here it is possible to configure whether you’d like each bullet to appear on click, or after a fixed duration.</p>
<img src="https://samhogy.co.uk/ssets/ett-advanced-sequence.png" width="216" height="270" alt="PowerPoint animation pane screenshot showing each bullet point animated individually." />
<h2>Look, free download!</h2>
<p>I have made both of these implementations available for download at this link. Included in this template are the two implementations described above. It should be the case that the slide you want to use can be copied and pasted into your own presentations. Enjoy!</p>
Is the Dashboard Pattern Dead?2012-09-10T00:00:00Zhttps://samhogy.co.uk/2012/09/is-the-dashboard-pattern-dead/<div class="crosspost"><span>🔗 This article is a cross post from <a href="https://blog.scottlogic.com/2012/09/10/is-the-dashboard-pattern-dead-2.html">the Scott Logic blog</a></span></div>
<p>Android application design has experienced radical change over the last two years. It has transitioned into an OS that can be executed on mobile devices, tablets and now television sets. In this blog post, I'd like to discuss what could be behind the most obvious user interface layout changes - the decline of the dashboard pattern and the emergence of several alternatives.</p>
<h2>The Dashboard Pattern</h2>
<p>At Google I/O 2010 the Twitter and iosched apps were published with the intention of demonstrating recommended UI design patterns. One of these was the dashboard - a grid of icons serving as a top-level menu.</p>
<img src="https://samhogy.co.uk/assets/android/dashboard.png" alt="Twitter Dashboard Pattern" />
<p>New users to an application are 'oriented' by a dashboard; they are shown the functionality that is provided by the app. However, this has a short-term positive effect before it becomes a hindrance. Dashboards can have a limiting factor, as more obscure functionality may not be discovered. There is an extra menu to navigate through to switch between top-level activities - 'sideways navigation' is not supported. It is possible for users to forget what they were trying to do when they return to the dashboard screen. Our brains have a short-term memory which can be hard to access after crossing a boundary. I often have an idea whilst walking and as soon as I walk through my front door I forget it!</p>
<p>Applications that once used the dashboard, such as Facebook, Twitter, Google+ and Evernote have since switched to different layout techniques. Why is this?</p>
<h2>The Design Guide</h2>
<p>The design guide was published at the beginning of this year. It explains in depth the principles behind Android application design, such as the use of fragments, the action bar and how to structure applications. Interestingly, it does not advocate using a specific UI pattern for navigation, instead the focus is on providing advice to let developers make the right choice for their application. The advice it gives is:</p>
<blockquote>
<p>"Avoid navigation-only screens and instead let people get to the meat of your app right away by making content the centrepiece of your start screen."</p>
</blockquote>
<blockquote>
<p>"Choose layouts that are visually engaging and appropriate for the data type and screen size."</p>
</blockquote>
<blockquote>
<p>"Minimize perceived navigation effort by keeping your apps shallow."</p>
</blockquote>
<p>Whilst this advice does not advocate the use of a particular design pattern, it is implying that developers should not use the dashboard pattern, which violates all of these principles. So, what are the alternatives that are rising in its place?</p>
<h2>Grid Layout</h2>
<p>Menu items are presented as cells within a grid. Cells in the grid may take up more than one column or row, which presents an irregular layout but allows for one item to be given prominence over others in the grid. It is typical to see vertical or horizontal scrolling present on a grid, but not both.Grid layouts are arguably very similar to the dashboard, in that they present a grid of menu items. Where grids differ however is with their ability to give prominence to particular items. Google Play is the most well-known example of this layout pattern, using a grid to separate out navigation to its four stores (apps, books, movies and music) and using other entries to provide 'deep links' into the application. For example, in the screenshot below there is a link to 'games' and to featured content.</p>
<img src="https://samhogy.co.uk/assets/android/googleplay.png" title="Google Play Grid" />
<p>Another application which uses this layout is Songkicks, a concert-planning application utilising the user's music library and location. This information is also used to personalise the content within the grid layout.</p>
<img src="https://samhogy.co.uk/assets/android/songkicks.png" title="Songkicks Grid" />
<p>Technically this form of layout is still a dashboard, albeit with the added ability of scrolling and more scope for personalisation. Many of the negatives associated with the dashboard are shared - this is still a navigation-only screen affording no 'sideways' navigation between menu items. Whilst the room for personalisation and the ability to add deep links do recognise the need to bring content forward, this can only be considered a baby step. Grids with few items, such as in the Songkicks example above, could be considered as a waste of space. Grids with many items are not suitable as top-level menus, given the design guide advice to keep application structure shallow.</p>
<h2>Sliding Drawer</h2>
<p>Applications designed according to this technique do not allocate an individual screen to a top-level menu. Instead, one of the items in the menu is considered the 'main' activity and this is shown on startup. To access the menu, the user performs some action. This causes the menu to slide in from screen-left, pushing the current activity to the right but keeping it slightly visible.</p>
<img src="https://samhogy.co.uk/assets/android/google+.png" title="Google+ Sliding Drawer" />
<p>This pattern is extremely popular at the moment. It is implemented in many of the top applications including Google+, Facebook, Spotify and Evernote. This pattern represents a good compromise between the need for a top-level hierarchy and conforming to the design guide advice. Content is made centre-stage, with the navigation elements only being displayed when the user wants to interact with them. This certainly avoids navigation-only screens and aids keeping applications shallow. Additionally, it is possible for this style of menu to contain more items than a dashboard due to the presence of a scrollbar, making this technique useful for applications such as Facebook which have, by necessity, a large number of top-level menu items.</p>
<p>In terms of the number of interactions needed to navigate between two child items of a menu, this technique and the dashboard are identical. Firstly, the menu has to be displayed, followed by an item selection. The key difference is that the menu and the current activity are displayed simultaneously, which mitigates the boundary-memory issue.</p>
<p>The biggest drawback to this navigation technique is the inconsistency between the myriad of implementations, specifically regarding the width of the sliding drawer and how to show it. Integration with the action bar is the main difference in implementation. Facebook and Spotify replace the action bar logo with a button to launch the sliding drawer (<img src="https://samhogy.co.uk/assets/android/facebookdrawer.png" title="Facebook Sliding Drawer Button" />). Evernote displays the sliding drawer when the action bar logo is pressed, whereas Google+ and YouTube utilise the concept of navigating 'up': the action bar logo displays the 'up' icon, which launches the sliding drawer when pressed (<img src="https://samhogy.co.uk/assets/android/google+back.png" title="Google+ Action Bar Up Button" />). Swiping left to display the sliding drawer is also a nice, complementary technique that is implemented in YouTube, Spotify and Evernote.</p>
<p>Using 'up' to display the sliding drawer is semantically correct and in my opinion the implementation which should be standardised upon. It would be nice for some consistency, at least!</p>
<h2>Fragments and the Action Bar</h2>
<p>To improve the usability of Android and to better support multi-pane tablet layouts, Honeycomb introduced the concepts of Fragments and the Action Bar. Fragments are reusable UI elements which are hosted in a parent Activity. Think of a master-detail view as two fragments inside one containing Activity. The Action Bar is a persistent row at the top of the screen showing application branding and acting as a dock for navigation elements and common action shortcuts. The Action Bar was introduced in the same Google I/O presentation as the dashboard and has been incorporated into the framework, unlike the dashboard!</p>
<p>This pattern aims to use fragments to focus on presenting content from each of the top-level menu items. Where possible these are displayed together on one large screen to provide an information-dense application. When it is not possible to display the fragments together, each fragment is made its own full-screen tab page and a series of tab page headers is added to the Action Bar, or on a separate row underneath the main action bar.</p>
<p>The prime example of how this layout technique works is the iosched 2012 application. There are three top level items: an agenda, a live social network feed and a list of available conference sessions. Tablets display three side-by-side views:</p>
<img src="https://samhogy.co.uk/assets/android/ioschedlandscape.png" title="iOSsched Fragment Layout Landscape" />
<img src="https://samhogy.co.uk/assets/android/ioschedportrait.png" title="iOSsched Fragment Layout Portrait" />
<p>Twitter has a similar, although less complete, implementation of this pattern. It does not yet have a tablet-optimised layout and its customised header style is too closely-linked to iOS. On Android it is common to use text rather than icons as tab headers.</p>
<img src="https://samhogy.co.uk/assets/android/twitter.png" title="Twitter Split Action Bar" />
<p>Gmail also uses a similar layout structure for a master-detail view. Tablet devices display the inbox and a selected email side-by-side. On smaller devices the inbox and email are displayed on separate screens. However, in this configuration 'sideways' navigation is supported as the user can swipe left or right to read other emails without returning to the inbox screen. Utilising shortcuts like this mitigates boundary memory loss, by removing the need to go through the boundary.</p>
<p>In a similar manner to the sliding drawer technique, this method presents the application content to the user without needing navigation-only screens. Navigation elements are only shown on screens which cannot display multi-pane layouts. On the other hand if the navigation is needed it cannot be hidden like a sliding drawer can be. There is a limit to the amount of tab headers which can be displayed, thus limiting the scope of this solution to applications with small numbers of top-level items. Therefore this technique will result in shallow application structures.</p>
<h2>Conclusion</h2>
<p>Good Android design optimises layouts for devices with various different screen sizes. To achieve this it is advisable to arrange UI elements into reusable fragments, incorporate the action bar and make the prime content of the application discoverable. Clearly the dashboard is no longer fit for purpose. Grid layouts should be used with caution due to their similarity to the dashboard pattern.</p>
<p>There is commonality between the sliding drawer and fragments/action bar approaches. They are both trying to solve the problem of making content the centrepiece of an application. The two approaches represent different trade-offs between the discoverability of content and navigation.</p>
<p>The use of fragments and tabs on the action bar is the closest fit to the advice given in the design guide, although practically speaking this is only suitable when the number of top-level items is small. On smaller devices the tab pages take up a second 'row' onscreen and this eats into the already-limited screen space. For applications which many top-level items (such as Facebook), it is more practical to utilise the sliding drawer technique, preferably the YouTube implementation (uses the Up button and gestures). I'm calling this a draw.</p>
5 Questions about Porn Blocking2012-09-02T00:00:00Zhttps://samhogy.co.uk/2012/09/5-questions-about-porn-blocking/<p>The UK Government is considering whether to restrict access to adult content online using an opt-out system. In this blog post I am going to present my opinion on the matter not by making a moral argument, but by treating the blocking system as a <strong>security system</strong>. Using this approach I will analyse the proposed security system to glimpse at the effectiveness of such a measure, were it to be implemented.</p>
<p><a href="http://schneier.com/">Bruce Schneier</a>, a renowned security expert, presented a five-step plan for performing a security analysis of a system in his book Beyond Fear. This will be used as the structure of this post. The steps are:</p>
<ul>
<li>What assets are you trying to protect?</li>
<li>What are the risks to these assets?</li>
<li>How well does the security solution mitigate those risks?</li>
<li>What other risks does the security solution impose?</li>
<li>What costs and trade-offs does the security solution impose?</li>
</ul>
<p>At the moment, there are only glimpses of exactly how this will be implemented. The approach suggested by an <a href="http://www.claireperry.org.uk/downloads/independent-parliamentary-inquiry-into-online-child-protection.pdf">independent Parliamentary inquiry</a> is for a “network-level ‘opt-in’ system, maintained by ISPs, that delivered a clean internet feed as standard but allowed them to choose to receive adult content”. Confusingly, the report does not specifically define what should be blocked. For the purposes of this analysis I am going to take this solution of an ISP-level blocking system for pornography only. Note that under current UK law ISPs are required to block illegal websites when notified. Such content is therefore outside of the scope of this analysis.</p>
<h2>What assets are you trying to protect?</h2>
<p>We are trying to protect, or indeed preserve, the ‘innocence’ of children.</p>
<h2>What are the risks to these assets?</h2>
<p>We are concerned with exposing children to content that is considered harmful. That is, content which children may find upsetting or are not capable of fully understanding.</p>
<h2>How well does the security solution mitigate those risks?</h2>
<p>This solution prevents users subscribed to the blocking filter from accessing the content of websites listed on the block list. As this is implemented at the ISP level, such restrictions apply to all devices using a particular internet connection.</p>
<p>The effectiveness of this solution depends upon several key aspects:</p>
<ul>
<li>The coverage of the block list. It is important for the detection mechanisms to be adequate to prevent content from falling through the block, known as a ‘false-negative’. This can never be reduced to zero, but should be kept minimal.</li>
<li>The regularity at which the block list is updated must be sufficient so that coverage can be maintained.</li>
<li>Blocking only takes place when a customer is subscribed to the blocking list. In the situation where a family household chooses to opt-out of the blocking system, all users will of that connection will have unrestricted access.</li>
<li>Traffic which is encrypted in a manner designed to mask the destination website from the ISP will not be affected by the filter. Anonymity tools, such as Tor, proxies and Virtual Private Networks can bypass this solution. Such technologies have legitimate use cases so to simply restrict access to these is outside the scope of this security solution.</li>
</ul>
<h2>What other risks does the security solution impose?</h2>
<ul>
<li>The risk of too many false positives. This often happens when keywords are scanned and used to determine whether a web page should be blocked. Often it is difficult to accurately understand the context in which certain keywords are used and there is evidence that access to content relating to sexual health has been blocked. The risk here is that we risk unnecessarily restricting access to information which should be made available to encourage safe sex.</li>
<li>Mission creep. This is where the blocking filters begin to restrict access to more types of content than originally intended. This risk can be mitigated though transparency, preferably in the form of independent scrutiny. A clear definition of what should be blocked is needed. Ideally this definition should form part of the Bill itself and can only be amended by further legislation requiring a vote in Parliament.</li>
<li>In the case of content being blocked unnecessarily, it is important for an adequate appeals procedure to be established. Under the current mobile data blocking system appeals are dealt with on an ISP-level basis and <a href="http://blocked.org.uk/">there are documented cases showing that lifting a block is a difficult thing to do</a>.</li>
<li>There are additional risks regarding the particular implementation of the block list. Off-the-shelf solutions tend to come with their own block lists that are protected by intellectual property rights. As this will be a publicly-mandated level of blocking, ultimate control of the block list should remain in public ownership. This is especially important because there will be several instances of the blocking system – one at each ISP. They will need to be kept in sync for overall confidence in the mechanism to remain high. Adequate protections need to be placed such that it is only possible for the bill payer to unsubscribe from the blocking service. If a teenager can impersonate a parent and have the block lifted the entire system is ineffective.</li>
<li>Finally, there is a risk that this creates a false sense of complete security in the situation where this mechanism is the only protection put in place.</li>
</ul>
<h2>What costs and trade-offs does the security solution impose?</h2>
<ul>
<li>There is the cost of establishing and running the blocking systems at each ISP. This may be government-funded, or recuperated from customers as an additional charge or one-time price rise for their Internet services.</li>
<li>There is further financial cost towards establishing and running the appropriate auditing and appeals bodies. Depending upon the solution this may be the role of the government (through a quango body or the courts), or part of a self-regulation solution between a group of ISPs.</li>
<li>As a society, we are making the fundamental trade off that it is necessary to restrict access to particular types of content, effectively a form of state censorship.</li>
<li>Consider again the family household. Adult members of the family who choose not to opt-out of the blocking system for the benefit of their children are willingly restricting the types of information they can access.</li>
</ul>
<h2>Conclusion</h2>
<p>This mechanism alone is not a silver bullet. It is easily circumnavigated using well-known methods and relies upon a blocking service which has wide coverage, limits the number of failures and is responsive to change. If the Government implements this system they should not remain content. They should continue in efforts to educate and inform the public about risks associated with the Internet.</p>
<p>By adding security in depth, that is another layer of security in the form of a home blocking system, the overall security <strong>solution</strong> can improve. There will be the option for false-positives to be caught and successfully blocked by the second system. But, implementing security in depth adds its own layer of complexity and will probably be susceptible to the same avoidance techniques as the ISP-level filters.</p>
<p>A large concern remains, which is the potential lack of transparency. The Government have not yet discussed how the public will remain informed about this system. For public confidence to remain high it is important for the Government to prove that the block is effective. Without transparency the Government is asking for blind trust, which makes a mockery of the current line of allowing parents to make informed choices.</p>
<p>Without the adequate transparency measures in place I cannot support this system. In this instance I feel that the risk to society (the loss of holding the Government to account) outweighs the benefits of a security system that is only moderately effective and can instead be implemented through off-the-shelf solutions at the home level.</p>
Running2012-08-02T00:00:00Zhttps://samhogy.co.uk/2012/08/running/<p>I have previously tried to take up running with various degrees of success. The first time round, in 2010, I ran with some colleagues on a lunch break and they were too fast for me. After a few months of apathy, I tried again, running a 1 mile lap of a local park regularly after work. However, my interest soon subsided after moving house.</p>
<p>At the beginning of the year some colleagues and I decided to start running on lunchtimes in an ‘unfit’ running club. Our initial attempt was not very good. We ran 2 miles in 24 minutes! The course was quite hilly and we needed to stop regularly. We stuck to it and have ran almost every Tuesday and Friday since. It’s now August and I can run 5k in 24 minutes, 5 miles in 43 minutes and I’ve just today recorded a 6:53 1 mile time trial score.</p>
<p>I feel a lot healthier for running. I’ve definitely lost weight, feel less stressed and picked up a hobby in the process. By no means can I run competitively – I’m not interested in that. Working as a software developer you tend to spend a lot of time sitting at a desk. Getting some exercise and a bit of fresh air really helps to focus your mind.</p>
<p>Running is admittedly a tough hobby to pick up. You need to control your breathing, learn how to pace yourself and keep your mind at bay. Running with friends will encourage you to run further and faster, but going out alone is a different ball game. At every opportunity your mind will tell you to stop, go home and eat some bacon. Overcoming the faux-pain, the real pain and clearing a hurdle (such as 5k or 5 miles) will really help to show you can do it!</p>
<p>There is no need for running to be competitive, about going further, being fastest or obsessively logging routes, times and your heart rate. Indeed there are myriad of races, but there are also non-competitive, free, events such as the Parkrun. Typically held on a Saturday morning across the country, these are fun and popular. Running is about doing something that will make you feel better, has long term health benefits and doesn’t cost a penny. Try it and you’ll have fun, I promise!</p>
An Online Privacy Toolkit2012-03-13T00:00:00Zhttps://samhogy.co.uk/2012/03/an-online-privacy-toolkit/<p>In my joint talk for Newcastle Skeptics in the Pub, we argued that data is a commodity. Your personal information should be treat as such. There may be times where you consent to services accessing your information, however there are other times in which you don’t. Instead, your data is taken unless you opt-out, or secretly uploaded to a server without “unambiguous” consent.</p>
<p>This blog post serves as a utility belt; a toolkit of web browser extensions and mobile phone applications I find useful to use in order to ensure I can remain in control of as much of my information as I can.</p>
<h2>Search Engines</h2>
<p><a href="http://duckduckgo.com/">DuckDuckGo</a> is an alternate search engine that has been gaining traction in the last year or so. It is completely anonymous, in that it does not track or log your searches, it is encrypted as standard, and does not <a href="http://dontbubble.us/">filter your results according to your worldview</a>. In comparison to Google, DuckDuckGo is not quite there yet, although I use it as my day-to-day search engine and rarely have to revert to Google. The !bang synatx is amazing, too!</p>
<h2>Behavioural Tracking</h2>
<p><a href="https://ublockorigin.com/">uBlock Origin</a> is a browser extension originally developed for Firefox, but also available on Chrome. This extension will stop ads from being displayed in web pages, and also from phoning home and tracking data.</p>
<p>I also have <a href="http://www.ghostery.com/">Ghostery</a> installed, which does very much the same thing as uBlock, but is a lot more transparent. Its options menu also makes unblocking items possible with relative ease. Ghostery is fairly robust and works on Firefox, Chrome, Opera, Safari (including Safari iOS!) and Internet Explorer. It’s worth it.</p>
<p>There are some other alternatives out there that I haven’t tried, namely the <a href="http://www.abine.com/dntdetail.php">Do Not Track Plus</a> extension, which works in a similar manner to the above extensions. Furthermore, there is an experiment into embedding opt-out to web tracking as part of the underlying HTTP protocol, by specifying a “<a href="http://donottrack.us/">Do Not Track</a>” header. The linked site specifies how to enable this for each browser, although not everyone is willing to support this yet. Hopefully, that will change.</p>
<p>Surprisingly, an effective way to stop a lot of tracking is to simply <strong>log out</strong> of a service after using it. This is not just closing the tab, but actually logging out. When you’re logged in, even if the tab is closed, any tracking cookies on your machine from that service can identify your browsing habits to your user account.</p>
<p>If you’d like to read about the general principles behind web tracking, I have found <a href="http://netograph.com/overview">this visualisation tool</a> to be remarkably effective.</p>
<h2>Social Networking</h2>
<p>Part of the issue that I have with social networking is that the privacy policies are constantly changing. Keeping on top of the changes is difficult, especially as the trend for default settings is to <a href="http://mattmckeon.com/facebook-privacy/">push more of your data into the public sphere</a>. I’ve found that Lifehacker offer the best, always-up-to-date guides for <a href="http://lifehacker.com/5813990/the-always-up+to+date-guide-to-managing-your-facebook-privacy">Facebook</a> and <a href="http://lifehacker.com/5827683/a-guide-to-google+-privacy-and-information-control">Google+</a>. Bookmark and check back every month. My rule of thumb is to assume the default privacy settings are crap, and therefore I make a point to tighten them up as soon as I sign up to a service.</p>
<h2>Mobile Phone Privacy</h2>
<p>The permission systems on mobile phone operating systems are wildly different. Android provides an “opt-in” mechanism, where applications have to declare they want to access certain functions on your device, such as accessing the Internet or viewing your contact lists. This information is provided to the user at the time of installation, and can be viewed at any time from the application settings menu. Apple, on the other hand, prefer this to be hidden from the user and ask users to put faith in Apple’s rather draconian and opaque approval system to weed out any bad eggs. That is not always the case.</p>
<h2>LBE Privacy Guard</h2>
<p>If you have a rooted Android handset (it’s quite <a href="http://www.androidauthority.com/rooting-for-dummies-a-beginners-guide-to-root-your-android-phone-or-tablet-10915/">easy to do</a> and I recommend taking the time to do so purely for this), then I strongly advise installing <a href="https://play.google.com/store/apps/details?id=com.lbe.security.lite&feature=more_from_developer#?t=W251bGwsMSwxLDEwMiwiY29tLmxiZS5zZWN1cml0eS5saXRlIl0.">LBE Privacy Guard</a>. This runs as an always-on service, and allows you to revoke permissions for an application either once and for all, or whenever the app tries to access a feature. For example, I have LBE Privacy Guard set to always reject access to my contact list for the likes of Facebook and Twitter, but I allow LBE to prompt me when Google Maps requires access to my GPS.</p>
<h2>Security</h2>
<p>Last but certainly not least, I’d like to mention that maintaining good security is essential in order to remain in control of your information. Not only should you have a <a href="http://www.passwordmeter.com/">good, strong, password</a>, you should only use this for one login. Having many passwords for many services makes stealing your identity extremely difficult, but it also makes your life more difficult as you have to remember the damn things. Well, you don’t. Services like <a href="https://lastpass.com/index.php">LastPass</a> (which works on literally everything) keeps your passwords secure and handles logging in for you. It only requires one master password, and synchronises everything across multiple devices.</p>
<p>Do you connect to the Internet using a wireless connection? If you do, make sure the wireless router is encrypted. Nowadays, most open networks are at pubs and coffee shops. They actually have an extra layer of protection behind them known as a VPN, which requires log in details. However, for the standard out-of-the-box router, these services are not available. Take it as a given that if your home wireless router is unencrypted, someone is stealing your internet. You can be held liable for their actions. It seems as if some basic encryption is enabled by standard on wireless routers now, but if yours is open you can look at <a href="http://www.wikihow.com/Secure-Your-Wireless-Home-Network">this guide</a> to get things encrypted.</p>
<p>There was a media uproar over the discovery of the <a href="http://codebutler.com/firesheep?c=1">Firesheep</a> extension. This displayed any logged-in users for the most popular online services, and, at the click of a button, lets you <strong>log in as them</strong>. This only works on completely unencrypted networks and is totally illegal, but the reason for the controversy was that the tool made the process so user-friendly! I’ve never discovered the use of Firesheep in the wild, but the Blacksheep extension will notify you if someone is using Firesheep on the network you’re connected to.</p>
<p>You can encrypt web traffic. This is used for online payments so your credit card information cannot be stolen, but is also being enabled for more and more websites as computing costs come down. Most webmail services (such as Gmail) use HTTPS by default. Likewise, logged-in Twitter users get the benefit of security. Plus, it is possible to enable HTTPS for many other services, such as Google and Facebook. The <a href="https://www.eff.org/https-everywhere">HTTPS Everywhere</a> extension from the Electronic Frontier Foundation will take care of encrypting as much web traffic as it can. This extension is simply <strong>essential</strong>.</p>
<p>Finally, the <a href="https://www.torproject.org/">Tor project</a> will encrypt all of your web traffic. How this works is beyond the scope of this article, however the general principle is that it adds several layers of encryption to your web traffic, and your data packets perform less predictable hops around the intertubes. This technology has the capability to break through government-level filters, and as such is commonly used by journalists, human rights activists and protesters in less-than-friendly circumstances. Tor can take a bit of technical know-how to set up, however the <a href="https://www.torproject.org/projects/torbrowser.html.en">Tor Browser Bundle</a> is your easiest bet. If you need the same functionality on your mobile phone, get a rooted Android device and install <a href="https://guardianproject.info/apps/orbot/">Orbot</a>.</p>
<h2>Conclusion</h2>
<p>Well, this blog post was longer than I expected, but the web is a dynamic place with multiple protocols, services and methods of access that to reliably provide you with a means of keeping oneself protected requires a fairly in-depth analysis. There’s still a lot more that can be said on this subject, namely how to truly remain anonymous online, but for the general users, the above should be sufficient.</p>
A Skeptical Approach to Online Privacy2012-02-27T00:00:00Zhttps://samhogy.co.uk/2012/02/a-skeptical-approach-to-online-privacy/<p>I will be taking part in the Newcastle Skeptics in the Pub <a href="http://newcastle.skepticsinthepub.org/Event.aspx/913/SITP-Open-Mic-Night">Open Mic Night</a> event, which is being set up as part of the <a href="http://www.newcastlesciencefest.co.uk/">Newcastle Science Festival</a> (8th – 15th March). This ten-minute talk is tentatively titled ‘A skeptical approach to online privacy’, and will be performed in conjunction with my university friend Will. We’ll be briefly covering the historical aspects of privacy, and how this has changed over time alongside our ideas of freedom of expression. Furthermore, we’ll be looking into the future to see where our attitudes could lead us.</p>
The Unofficial St James' Park Stadium Name Generator2012-02-21T00:00:00Zhttps://samhogy.co.uk/2012/02/the-unofficial-st-james-park-stadium-name-generator/<p><a href="http://samhogy.co.uk/nufc">The Unofficial St James' Park Stadium Name Generator</a></p>
<p>Subverting Mike Ashley’s sponsorship ideas with <strong>programming</strong>.</p>
Hello world!2012-02-21T00:00:00Zhttps://samhogy.co.uk/2012/02/hello-world/<p>Welcome to my site! I am a Newcastle-based software developer, currently working as a consultant in the financial sector. This blog will mostly cover software development, my programming hobbies outside of work (such as Android and games development), and also some fun stuff along the way.</p>