# Boring tools win: five tests before a small team buys automation > Knight Capital, the Post Office and CrowdStrike show the cost of automation nobody can read or stop, and five tests can expose it before a contract is signed. - Published: 2026-10-18 - Author: 28creation Desk - Format: Analysis - URL: https://28creation.com/articles/boring-tools-win/ ## The short read **Knight Capital lost more than $460 million in about 45 minutes, after 97 warning emails nobody acted on.** - Knight Capital, the Post Office and CrowdStrike failed the same way: automation nobody could read or stop. - People stop checking systems that are usually right; in one study they caught only 33% of misses. ### What you can do with this > Feed any workflow tool a missing field, a duplicate record and a wrong file format before you sign. If it does not stop and tell a named person, keep it away from money. --- On August 1, 2012, a trading router at Knight Capital sent millions of unwanted orders into the US stock market in about 45 minutes, and the firm lost more than $460 million. The [SEC's order](https://www.sec.gov/litigation/admin/2013/34-70694.pdf) traced it to nothing exotic: old code nobody had deleted, a manual update nobody checked, and 97 warning emails nobody acted on. That pattern runs through two other public failures, and it scales down to a ten-person office. So a small company buying workflow software should pick the predictable, legible tool over the clever one. Legible means a person can see what the tool did and why. The automation that does damage changes without asking, hides how it reached its answer, or is right so often that people stop checking. Below are the three cases, the research on how people supervise machines, and five tests to run on any tool before you sign. ## Knight Capital lost $460 million to code nobody could see Knight's router, called SMARS, still carried a feature named Power Peg that the firm had [stopped using in 2003](https://www.sec.gov/litigation/admin/2013/34-70694.pdf). The code stayed on the servers, "present and callable." In 2012, Knight wrote new code for a New York Stock Exchange retail program and reused an old switch: a flag that used to turn Power Peg on. Starting July 27, Knight copied the new code onto the router's eight servers in stages. A technician missed one of them. Knight had no second person review the work, and no written procedure required one. On August 1, orders carrying the reused flag reached the eighth server and woke Power Peg up. From 212 small customer orders, it produced more than 4 million executions in 154 stocks. By the time it stopped, Knight had taken on positions worth about $3.5 billion long and $3.15 billion short. The warnings were there. From 8:01 a.m., an internal system sent 97 emails flagging an error called "Power Peg disabled" to a group of staff before the market opened at 9:30. Knight had not designed them as alerts, and staff generally did not read them. There was no procedure to halt the router when it misbehaved, either. One attempted fix, removing the new code from the seven healthy servers, made things worse. The SEC later fined Knight $12 million on top of its losses. None of those failures called for a smarter system. They called for a change someone could verify, a warning that reached a person, and a way to stop. Swap the trading router for an invoice approval flow and the shape is familiar: an old rule nobody deleted, an edit made by one person, error emails landing in a shared inbox. ## The Post Office treated a system nobody could inspect as proof The UK Post Office began rolling out its Horizon IT system to branches in about September 1999. Bugs in it could show losses in a branch's accounts that never happened. The [public inquiry](https://www.postofficehorizoninquiry.org.uk/volume-1-post-office-horizon-it-inquirys-final-report) found that some Post Office staff "knew or, at the very least, should have known" this, yet the organization "maintained the fiction that its data was always accurate." The people running the branches, called subpostmasters, had almost nothing to argue with. In 2019 the High Court found that their transaction logs were available for only 42 days, later 60, and could be printed on till rolls ([Horizon Issues judgment](https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-judgment.pdf), paragraph 999). With so little data, they "simply could not identify apparent or alleged discrepancies and shortfalls, their causes," the judge wrote. He called the Post Office's stance "the 21st century equivalent of maintaining that the earth is flat." The inquiry's first report, published in July 2025, estimates that about 1,000 people were prosecuted and convicted on Horizon evidence. Many thousands of people were held responsible, wrongly, for losses that were illusory. The scale was national, but the mechanism is ordinary. When the people using a system cannot see why it says what it says, the system wins every argument by default. In a small company the argument is smaller: a customer disputing a late fee, an employee questioning a payroll figure. You still need to show what happened. ## Systems that are usually right teach people to stop checking Before the third case, a prediction. In a 1993 experiment, people juggled a tracking task and a fuel-management task by hand while an automated system watched engine gauges for them. Sometimes the automation missed a fault. Guess what share of those misses people caught when the automation had been steadily reliable all session. It was 33%. When the automation's reliability varied over time, people caught 82%, according to a 2010 review by [Parasuraman and Manzey](https://doi.org/10.1177/0018720810376055). When watching the automation was their only job, detection was near perfect. The trouble starts when other work competes for attention, which describes most small businesses on most days. The review has worse news for anyone counting on training. Complacency showed up in experts as well as novices and "cannot be overcome with simple practice." Automation bias, following a machine's advice against other evidence, "cannot be prevented by training or instructions." In one simulator study the review cites, 75% of pilots followed an automated checklist that told them to shut down the wrong engine. With a paper checklist, 25% did. ## CrowdStrike shipped an update on the strength of past successes On July 19, 2024, at 04:09 UTC, CrowdStrike sent a configuration update to its Falcon security software for Windows. It pulled the update at 05:27 UTC, according to its [preliminary review](https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/). Windows machines running version 7.11 or later of the software that were online in those 78 minutes, and received the update, crashed. Microsoft [estimated](https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/) that 8.5 million Windows devices were hit, "less than one percent of all Windows machines." The bug was small. A detection template defined 21 input fields, while the code feeding it supplied 20, and the new content was the first to read the 21st ([root cause analysis](https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf)). The tests had used a wildcard in that field, so they never touched it. Why did it ship? CrowdStrike's review names earlier testing, "trust in the checks performed in the Content Validator," and "previous successful" deployments of the same kind. That is the complacency result, at company scale. A process that had worked several times was trusted to work again. The fixes CrowdStrike announced are the boring ones. Its root cause analysis concludes that each such update "should be deployed in a staged rollout," starting with a canary, meaning a small group gets it first. Customers can now choose "where and when" these updates land. ## Clever steps belong inside a boring frame For this piece, clever mostly means an AI step, such as a model that reads incoming email or drafts a reply. Those steps are useful, and by design they carry both risks above. OpenAI's own [documentation](https://developers.openai.com/api/docs/guides/advanced-usage) says its chat completions "are non-deterministic by default (which means model outputs may differ from request to request)." It also warns that determinism may change "due to necessary changes OpenAI makes to model configurations on our end." Read that as a buyer. The same input can produce a different output, so explaining a past run gets harder, which was the Horizon problem. And the behavior can shift on the vendor's schedule, which was the CrowdStrike problem. None of this argues for avoiding smart tools, and predictable has real costs. Rule-based tools need someone to write and maintain the rules. They also handle messy input badly: a customer who buries the order number in the third paragraph of a free-text email will beat most rules. Staged rollouts cost speed as well. CrowdStrike's fast-track content exists so its software can respond to new threats "at operational speed," and a canary stage slows that down. Some of the time, waiting is the risk. Our view is that the trade is worth making for anything that touches money, customers or staff records. Let the clever part do the reading, then hand its result to a plain, logged step a person can inspect. Require a human yes before anything leaves the building. Put the cleverness in a box with clear edges, and make the box easy to open. ## What this means for you: five tests before you sign It is tempting to spend a trial clicking through features. Spend one afternoon of yours trying to break the tool instead. Each test below maps to one of the cases above. 1. **Break it on purpose.** Feed one real workflow a missing field, a duplicate record and a file in the wrong format. A pass means the tool stops and tells a named person, somewhere that person actually looks. Knight's 97 emails show what a fail looks like. 2. **Find the stop button.** Ask how to pause every run in one action, what happens to runs already in flight, and whether resuming processes anything twice. Knight had no halt procedure, and its attempted fix made the damage worse. 3. **Have a stranger explain a run.** Pick a run from last month. Ask a colleague who was not part of the setup to say what came in, which rule fired and what went out, using only the tool's own history. Then ask how long that history is kept. Horizon's branch logs lasted 42 days at first. 4. **Ask who changes it, and when.** Find out what the vendor can change without you, whether you get release notes, and whether you can try an update on a copy before it reaches live work. For any AI step, ask which model version runs and whether you can pin it. 5. **Clean out the old.** Replace one rule and confirm the old one is deleted rather than switched off. Then ask for a list of every active automation. Power Peg sat unused for nine years before it woke up. A tool that passes all five has earned a look at its feature list. One that fails any of the first three should not touch money or customers, however good the demo was. Knight's warnings arrived between 8:01 and 9:30 that morning, 97 of them, each naming the broken feature. The cheapest test on this list asks whether your tool would have put the first one in front of someone who reads. ## Sources - U.S. Securities and Exchange Commission, [Order in the Matter of Knight Capital Americas LLC](https://www.sec.gov/litigation/admin/2013/34-70694.pdf), Release No. 70694, October 16, 2013. - Post Office Horizon IT Inquiry, [Volume 1 of the final report](https://www.postofficehorizoninquiry.org.uk/volume-1-post-office-horizon-it-inquirys-final-report), July 2025. - High Court of England and Wales, [Bates and Others v Post Office Ltd, Judgment No. 6, "Horizon Issues"](https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-judgment.pdf), [2019] EWHC 3408 (QB), December 16, 2019. - R. Parasuraman and D. H. Manzey, [Complacency and Bias in Human Use of Automation: An Attentional Integration](https://doi.org/10.1177/0018720810376055), Human Factors 52(3), 2010 ([free copy](https://depositonce.tu-berlin.de/items/0c787d91-6021-480b-a333-009e0d1d0092/full)). - CrowdStrike, [Preliminary Post Incident Review](https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/), July 24, 2024. - CrowdStrike, [External Technical Root Cause Analysis, Channel File 291](https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf), August 6, 2024. - David Weston, Microsoft, [Helping our customers through the CrowdStrike outage](https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/), July 20, 2024. - OpenAI, [API documentation: Advanced usage, reproducible outputs](https://developers.openai.com/api/docs/guides/advanced-usage).