AI governance
The useful question is not whether I have a policy. It is what my pipeline does when the model is wrong.
I have shipped 38 AI systems since July 2023 and 25 are in production. What follows is the machinery that decides whether one of them goes out, the artifacts it leaves behind, and the case where that machinery passed something it should not have. Every claim here points at a file in this repository, because governance you cannot inspect is just a paragraph.
The release gates
These are the gates my demo factory runs on every build. Each one is a written contract a separate agent executes, and each one writes an artifact I can go back and read. A gate whose output nobody can audit is a vibe.
Runs on every build before anything else. Fails on API keys or tokens, on any real person's name, email, address or personal data, on confidential client data, and on hardcoded author filesystem paths. Fictional sample data is expected and passes. The cheapest model on the roster runs it, so it runs on everything, every time.
Artifact: portfolio/_logs/<slug>.sanitize.log
A real browser drives the actual primary action, with selectors derived from reading the file rather than guessed. Zero console errors and zero uncaught page errors, on load and during interaction. Warnings are allowed; errors are not.
Artifact: portfolio/_test/<slug>.spec.mjs
The same flow is run again with the network cut. It must still produce output and still throw nothing. Anything model-backed has to degrade to a deterministic bundled result rather than break in someone's hands. This is the gate that decides whether a failure is graceful or a blank screen.
Artifact: portfolio/_logs/<slug>.qa.log
The deterministic path runs twice with a reload in between, and the key output must be byte-identical. If I cannot reproduce an output I cannot debug it, and if I cannot debug it I should not have shipped it.
Artifact: portfolio/_test/<slug>.spec.mjs
A screenshot is compared against the real product's own design tokens, read from that product's source rather than from memory. It exists to stop a demo drifting into a prettier thing than the product it claims to represent.
Artifact: portfolio/_shots/<slug>.png
The part that matters
The first run of the factory came back 18 of 18 green, in a single cycle. Clean on secrets, clean on the user simulation, design matching the real product tokens. I wrote that number down and it was, in the narrow sense, true.
Then I ran a deeper audit that did not trust the gates: drive each demo with a real browser and diff what it displays against what it actually did. Two demos were labelling their output groq · livewhile making no model call at all. The output was pre-baked. Every gate had passed it, because no gate was asking “is this label telling the truth?” — they were asking whether output appeared, whether it was deterministic offline, whether any secret leaked. All fine. All beside the point.
That is the failure mode I care most about, and it is not a model failure. It is a measurement failure: the system was reporting a state it was not in, and my checks were shaped so they could not notice. A demo that lies about being live is a small version of a production system that lies about being confident.
The fixes were specific. One demo got rewired to make a real call and score it against ground truth. Others: a clipboard crash, a CTA blocked by an overlay, simulated output that now says (sim) on its face, honest labels replacing aspirational ones. External web fonts stripped so the pages are genuinely self-contained rather than nearly. Four demos were left explicitly pending with a written report each, rather than being quietly marked done.
What I changed afterwards was not the demos. It was the question the gate asks. A gate that only checks whether output exists will pass a convincing lie every time.
Governance in action
Describing a control and demonstrating one are different claims. These four run the real logic in your own browser: a consent gate that refuses, a deterministic validator that contains no model, a release gate that blocks a committed secret, and a retention toggle that shows what is kept when a session ends.
Nothing leaves this page. No network request, no storage, no analytics — every rule below is plain JavaScript evaluated in your tab, on synthetic data. The build fails if a request fires while these are driven, because a promise like that is worth what the test behind it is worth.
Most demos show the happy path. This one defaults to the refusal, because what a system does when it is not allowed to proceed is the only part worth trusting.
Refused — the session does not run
Consent record present and current — blocked
No record found for this participant. The gate stops here — it does not fall through to a default.
Teach-back completed — not evaluated
Not evaluated. Once a gate blocks, later checks do not run and cannot overturn it.
Scope matches the requested action — not evaluated
Not evaluated.
Refuse — blocked
Session refused and the refusal is logged with the rule that caused it. A human is notified; the system does not retry.
The first eval layer contains no AI at all. These three rules are plain functions running in this tab; edit the text and watch them fire.
Blocked by 1 deterministic rule — no model was consulted
Required disclosure present — passed
Care-sector output must carry its disclosure. A missing one is a compliance defect, not a style note.
No absolute-outcome language — failed
A model will happily promise an outcome. A regex will not let it.
Numeric claim is internally consistent — passed
"Five of the four" is the kind of thing fluent text hides and arithmetic catches instantly.
An LLM judge can run after this, and it is useful for the things rules cannot see. It runs second, and it cannot overturn a deterministic failure.
A simplified version of the fence that runs on every change. Pick a diff — three of the four are meant to be stopped, and the gate names which check stopped them.
+ const client = new Client({
+ apiKey: 'sk-live-9f2c4a7e11b8', // committed by mistake
+ });Release blocked by: Secret scan
Secret scan — blocked
A live-format API key appears in the diff. Blocked before it can reach a remote.
Destructive-action classifier — passed
No irreversible operations.
Claim-of-done requires evidence — passed
Change is covered by the existing suite.
Retention off is the production default. Toggle it and watch the record change shape — with retention off the verbatim fields are absent from the page, not merely hidden.
Structured summary only — there is no transcript to leak
The three fields that appear when you toggle retention on are not on this page right now. That is the difference between minimisation and redaction.
Hard lines
A rule with no enforcement is a preference. Each of these is checked by something that runs, not by me remembering.
Every demo runs on fictional sample data. Real products operate on separate, isolated data paths that the demos cannot reach. During one build the scanner caught a placeholder that read like a real person's name and blocked it until it was replaced.
Enforced by: the secret and PII gate, on every build
The model API key lives only in a gitignored environment file and in the host's environment. It is in no demo, no registry entry, and no committed file. Model calls from the browser go through a same-origin proxy so the key never travels to a client.
Enforced by: a same-origin proxy route plus the secret scan
One same-origin call to my own proxy, and nothing else. No CDNs, no remote web fonts, no analytics, no remote images. One build wanted a real mapping library; real map tiles need external requests and real geography, so it became a self-contained abstract tracker instead.
Enforced by: the secret and PII gate flags any external URL
If output did not come from a model call, it does not get to look like it did. This is now a hard line specifically because a previous version of the factory broke it and the gates did not notice.
Enforced by: the second-pass runtime audit, which is why it exists
Cost is a governance decision
Spending a frontier model on a regex scan is not rigour, it is waste, and waste is what makes teams quietly stop running the checks. I route by what the job actually needs.
| Role | Tier | Why |
|---|---|---|
| Orchestrator | Frontier | Planning, dependency and loop control, resolving conflicting reviews, final audit synthesis. Highest reasoning, fewest calls. |
| Builder | Mid | The coding workhorse. Reads real product source and reproduces design and flow. Many run in parallel, so cost per call matters. |
| Verifier | Mid | Writes and runs the browser simulation and interprets what a failure actually means. Needs real reasoning about errors, not pattern matching. |
| Design reviewer | Mid | Compares a screenshot against real tokens, and escalates genuinely ambiguous calls upward rather than guessing. |
| Scanner | Cheapest | Regex, secret and PII scanning, registry validity. Rote work with a clear right answer. Cheap enough that it runs on everything. |
The principle: reserve the expensive reasoning for judgment, parallelise the mid-tier for building and verifying, push rote checks down to the cheapest thing that succeeds. A deterministic script beats any model at the jobs a script can do, and it never has an off day.
Restraint
I run accuracy evals against labelled ground truth on the systems that need them. Those numbers are a quality instrument, and the moment they become a marketing surface I will start optimising the number instead of the system. They are also easy to quote out of the context that makes them meaningful. So they inform what I ship and they do not go on a page. If you are evaluating me and you want to see them, ask and I will walk you through them properly.
Two of the production systems sit in the care sector, so PHI handling under HIPAA is a design constraint from the first commit rather than a retrofit: separate isolated data paths, access control and audit logging on the paths that touch it, and no real personal data in anything demonstrable. What you will not find on this site is a “HIPAA certified” badge, because no authority issues one. There is no certifying body. A vendor claiming that badge is telling you they have not worked under the rule. When there is a signed BAA or a completed third-party assessment behind a claim, the right move is to name that artifact and who signed it, which is a stronger thing to say than a badge anyway.
Honest gaps
Next step
If your organisation needs the monitoring, the provenance trail, the red-team pass, or the independent review that I have just told you I do not have on my own work, that is precisely the function I want to build for you. I would rather start that conversation from an accurate picture than a badge.