Skip to main content

Featured

Plumbing Business Startup Cost: I Rebuilt the 1-to-5 Van Ladder From Its Own 19 Rows

The short version: "How much does it cost to start a plumbing business" has no single answer, and a kit that gives you one number is lying by format. So I unzipped this one, rebuilt its one-to-five-van ladder from its own 19 line items in Python, re-priced all five worked calls from their inputs, and ran its checker. The ladder footed to the cent, the five calls came back identical, and the one honest thing the headline number hides is that half of it is cash you keep, not money you spend. Every row of the ladder footed when I rebuilt it from the 19 line items, and the break-even landed on 52.53 calls — which is 53 whole calls, not 52. The problem: a trade business with no written prices If you are a service plumber or a lead tech about to go owner-operator, the skills are not the problem. You can read a leak, trace a drain, sweat a joint and leave a clean cabinet behind. Almost nobody in that position has ever priced a job on paper. So the first quotes are guesses. A nu...

Your Eval Suite Is Theater Until a Non-Zero Exit Code Stops the Merge

Your Eval Suite Is Theater Until a Non-Zero Exit Code Stops the Merge

Every team that ships an LLM feature has an eval suite somewhere. Almost none of them have an eval gate. The difference is one thing: when accuracy drops, does the build fail, or does a number get pasted into a Friday update while the change merges anyway? I have watched a 0.71 accuracy line scroll past six reviewers without a single comment, because nothing in the pipeline had the authority to say no.

So I went looking for the smallest possible piece of software that can say no, ran it from a clean unzip on my own machine today, and wrote down exactly what it does and where it stops being useful. The tool is AIEval CLI, it costs a dollar, and what you are buying is an exit code — not a model, not a dashboard, not a judge. This post is an account of measured behavior, not a review, because it is a product I sell.

Disclosure: AIEval CLI is my own product, sold on Gumroad. I earn the sale price if you buy through this link, and I built it for my own eval loops. What follows is what the bundle actually printed when I ran it — not a third-party review.

Get AIEval CLI — $1.00 USD (price read from the live listing on 2026-11-22 immediately before writing this, from two independent fields on the page: product:price:amount=1.0 and the JSON-LD offer at 1.0 USD): https://nandminc.gumroad.com/l/vmocdo

What you get: a 10,127-byte Python file plus a bash wrapper, MIT licensed — scaffold an eval suite, score recorded answers against expected ones, and exit non-zero when accuracy falls below the threshold you set. Python 3.8+. No install, no dependencies, no API key, no network.

The specific failure it targets

There is a shape of eval rot that is easy to miss because nothing crashes. You have prompts with expected answers. You have a script that records model output. Someone runs the comparison by hand, sees 94%, feels fine. Then a prompt change lands that quietly breaks the twenty prompts nobody re-read, the true number is 78%, and the 94% is still on the slide.

What is missing is not intelligence. It is a threshold that is written down in a file, checked by a program, and expressed as a process exit status so a CI step can read it. That is a boring thing to own, and boring is the point: a gate that always runs the same way is worth more than a dashboard that disagrees with itself.

What is actually in the package

I downloaded the file a buyer gets, not the working copy on my disk. The zip is 10,833 bytes and contains exactly eight files:

  • eval-harness.py — 10,127 bytes, 275 lines. This is the entire program.
  • ai-eval — 506 bytes of bash. It only resolves its own directory and execs python3 eval-harness.py "$@", so it works from any cwd.
  • README.md 7,963 bytes, QUICK-START.md 3,362 bytes, LICENSE 1,065 bytes (MIT).
  • examples/example-suite/ — config.json (125 B), prompts.jsonl (198 B), answers.jsonl (122 B): three prompts that pass, so the shipped example prints 100% out of the box.

The import list of that 275-line file is argparse, datetime, json, os, sys and nothing else. I grepped the source for requests, urllib, socket, http, api_key, openai, anthropic and telemetry: zero occurrences of every one. It genuinely cannot call a model or phone home, because it has no code that could. That is easy to verify yourself before you trust a single number it prints, which is the only reason to buy a tool this small.

What I measured running it

Clean unzip into an empty temp directory, Python 3.12, no environment variables set. These are actual outputs, trimmed only where the report repeats itself.

1. The shipped example, real mode

python3 eval-harness.py run examples/example-suite --responses examples/example-suite/answers.jsonl prints three PASS lines, then:

  Accuracy: 100.00%  (3/3 passed)
    Gate min_accuracy=0.80: PASS

ALL GATES PASSED

and exits 0. The bash wrapper form produces byte-identical output, which is a small thing that matters if a CI step invokes ./ai-eval instead of python3.

2. init scaffolds a suite you can actually edit

init my-suite writes config.json and prompts.jsonl with three example rows and prints the next three steps. The config is a name, an evaluator string, a metrics list, and a gates object holding min_accuracy: 0.8. Running init a second time refuses rather than clobbering: Error: directory 'my-suite' already exists. with exit 1. Small thing, correct thing.

3. Demo mode tells you it is demo mode, out loud

run my-suite without --responses prints:

  MODE: demo evaluator (simulated responses — accuracy below is NOT a
        measurement of any model; pass --responses to score real output)

then reports 100% anyway, because demo mode fabricates a response guaranteed to contain the expected answer. ai-eval test ends with Smoke test PASSED (demo evaluator; no model was called). If a vendor's eval screenshot is 100% and has no MODE: line above it, you are looking at the wiring, not the model.

4. The gate actually fires

I recorded one answer out of three on purpose. It scored 0/3, printed Missing responses: 2 (counted as failures), Gate min_accuracy=0.80: FAIL, GATE FAILED, and exited 1. Malformed JSONL exits 1 with the line number (bad.jsonl:2: not valid JSON); a missing answers file exits 1 with the OS error. Nothing silently degrades into a passing build.

The scoring rule is one function, so read it

This is the whole metric, verbatim from the bundle:

def _score_answer(response, expected):
    if not expected:
        return True
    r = " ".join(str(response).split()).lower()
    e = " ".join(str(expected).split()).lower()
    return e in r

Lowercase, collapse whitespace, substring. That is deliberate and it is also the tool's sharpest limitation, so I ran the failure cases instead of trusting the README table. With a four-prompt suite and a 0.80 gate I recorded answers that should have failed:

  • expected yes, response Yesterday it was. — PASS. A false positive, because yes is inside yesterday.
  • expected not found, response 404 means Not\nFound — PASS, whitespace normalisation doing its job.
  • expected empty string, any response — PASS, by design, for prompts with no single right answer.
  • expected paris, response Lyon. — FAIL, correctly.

Result: 75.00% (3/4 passed), Gate min_accuracy=0.80: FAIL, exit 1. Three of four verdicts came from substring behavior alone, which means short expected tokens like yes and no will flatter your model. Choose long, distinctive expected strings, or the number you gate on is partly a coincidence.

Where it breaks, in order of how much it will hurt you

  • A missing response is a failure, not a skip. If your recorder crashes halfway, the score collapses instead of quietly scoring the rows it got. The tool prints how many responses were missing so you can tell a recorder bug from a model regression — but you will read that line after the failed build, not before.
  • max_avg_latency_ms is accepted and inert. The harness never times a call, so it prints Gate max_avg_latency_ms: NOT ENFORCED with an explanation rather than ignoring the key. Measure latency in your own recorder.
  • Substring scoring is not regex, not exact match, not semantic. No embeddings, no judge model. If you need graded correctness, this is the wrong tool and it does not pretend otherwise.
  • Single process, no parallelism. A 5,000-prompt suite is 5,000 fast string comparisons and a lot of JSON parsing.
  • It does not record answers for you. The {"id","response"} JSONL is your job, in whatever language talks to your provider.
  • Preserve the exec bit if you move files. The zip stores ai-eval as mode 755; copy it around as plain data and you get permission-denied instead of a CLI. Call python3 eval-harness.py and the wrapper's permissions stop mattering.

Wiring it into CI

Two steps, one of which you already have:

- name: Record answers
  run: python3 scripts/record_answers.py --suite evals/my-suite --out answers.jsonl
- name: Gate on accuracy
  run: python3 eval-harness.py run evals/my-suite --responses answers.jsonl

The second step fails the job on a gate miss because the exit code is non-zero. That is the entire integration. Your recorder is the only write you own, and it is also the only place a model gets called — which keeps the secret and the scoring in different processes on purpose.

Who it is NOT for

If you want a platform with a prompt registry, run history, tracing, human review queues and an LLM judge, you need an eval product, not a $1 script, and there are good ones. If your team cannot be relied on to record answers into a JSONL file, this tool will sit unused. If your accuracy claim depends on semantic equivalence — "the model said it a different but correct way" — substring scoring will tell you it failed, and you will spend an afternoon arguing with a tool that is technically right about the strings it was given. And if you need latency percentiles, cost accounting or per-token attribution, none of that is here and no number of retries will make it appear.

Honesty block

Stated plainly, and it applies to every number above: example figures are illustrative planning inputs — not market rates and not earnings. The prices and byte counts in this post were read from the live listing and the actual files on 2026-11-22 and will drift. No legal, tax, insurance, licensing, veterinary, automotive-repair, or financial advice is given here. No CRM, map, API or invoicing integrations are claimed or shipped — the bundle has one Python file, one bash wrapper and an example suite, and that is the whole of it. There are no ratings, no testimonials and no customer figures in this article because I have none to cite, and a tool that says no for a living should not manufacture social proof.

If you run it and something contradicts this post, email nandminc2020@gmail.com with the command, the exit code and the first lines of stderr. The gate is the product; being wrong about what it gates is not something I want in writing.

Comments