Agent shipping checklist

A pre-flight list. Each item keeps only pitfall + judgment. Full cases, diagrams, and the printable quick table ship with Mortal’s Compass.

Six stages · check them off in order before you build

Pick the scene
Data & permissions
Design & tools
Eval & acceptance
Launch & fallback
Cost & iteration

How to use it

Check “Pick the scene” first. If the first three boxes don’t stick, don’t start building—you’ll mostly ship another chat box.
Pilot one workflow at a time. Finish the related group before you grant write access.
Leave marks for next-week you. Tag the items that actually saved you as “must check next round.”

18 items in six groups.

1. Pick the scene

1.1 Are you changing the underlying workflow, or bolting on another Q&A box?

You moved at least one of: the right data, a cross-org boundary, or when humans review—and you know whose approval you hit, plus who owns data and audit.

Pitfall: One agent promising two jobs makes acceptance fight itself.
Judgment: Write “what we won’t do” first. Ship only when success/fail is machine-readable.

1.2 After something breaks, can it restart itself—or does a human have to click Start again?

Triggers are events / state changes, not “someone remembered to click.”

Pitfall: Mimicking a human clicking the old UI is RPA; the flywheel never spins.
Judgment: Imagine everyone takes a week off—if token use collapses, AI is still a tool, not a process participant.

1.3 Can success fit in one sentence, and be checked by a machine or script?

You can write what “done” looks like, and whether failure should stop or escalate.

Pitfall: Skip this and every later “feels fine” is unreliable.
Judgment: If you can’t measure it, you can’t automate it—define metrics and eval before stacking models.

2. Data & permissions

2.1 Is the knowledge base ready—or are you still “feeding” the model a pile of historical queries?

Definitions, table/field meaning, and business rules are explicit; key knowledge is human-readable and machine-retrievable.

Pitfall: Dumping historical queries wholesale into the model barely helps—and can hurt.
Judgment: Ready the definition asserts, code tables, and canonical docs before swapping models.

2.2 Are orchestration / execution / external three separate trust planes?

Orchestration only builds tasks and human gates; execution stays in the minimal directory/tool set; every external input is untrusted.

Pitfall: Giving the orchestrator “full worker rights” for unattended runs is the all-or-nothing trap.
Judgment: Treat external input as untrusted; even small-team inbox RAG should isolate sources and ban auto-fetching outbound links.

2.3 Does the “cleared to proceed” signal live somewhere the caller can rewrite?

Trust results stay in an in-process loop (return codes / memory state / controlled artifacts)—not comments, chat messages, or other forgeable channels.

Pitfall: Writing “OK to merge” in a comment lets anyone with comment rights forge a gate.
Judgment: Gate signals must never sit on a channel the caller can write.

3. Design & tools

3.1 Are you designing the work environment—or still polishing the instruction manual?

Prompts get lighter; tools, skills, memory, and context get heavier. Fixed extract/clean → scripts or schemas (stable); open exploration → free agents (flexible).

Pitfall: Noun fights hide the implementation; silent call ceilings pretend to be exits.
Judgment: Sample quality before scale. Ask first: does this step need stable or flexible?

3.2 Are control / orchestration / execution layers actually separate?

Chat only takes commands and reports; orchestration owns routing, monitoring, budget; real work runs in an isolated environment.

Pitfall: Spawning code or prod writes straight from chat will blur every boundary.
Judgment: Clear layers unlock pre-check gates and security audits.

3.3 Does a skill remember context—or only list steps?

Skills narrow filters and boundaries the user already stated; they are not another process manual.

Pitfall: Aggregate accuracy can look fine on both sides while dropping the time window and ticket id already said in chat.
Judgment: When short replies must carry prior context, skills need context narrowing—and tests for it.

4. Eval & acceptance

4.1 Does the loop have an owner, a deadline, and an acceptance definition?

Who owns it, when it’s done, and what “pass” means are written down; success criteria prefer machine-readable.

Pitfall: No owner / clock / acceptance turns weekly meetings into risk collectors.
Judgment: If you can’t write “what done looks like” before launch, you aren’t automating.

4.2 When accepting AI deliverables, can the owner explain without the AI?

On assumptions, tradeoffs, failure boundaries, and acceptance criteria, the owner can answer.

Pitfall: Treating “lightweight” as skipping architecture—docs look complete, humans can’t tell the causal story.
Judgment: Technical closed loop ≠ business closed loop; the owner must narrate the middle layer.

4.3 Have you listened to single failures, not only aggregate scores?

After a major change, a human has flipped through a batch of traces; the first failure is noted and folded back into regression.

Pitfall: Clients, networks, and auth eat most of the time; many teams stop at the first sample set.
Judgment: New sessions + new samples turn a demo into engineering acceptance.

5. Launch & fallback

5.1 Is the default “soft fence + human-tagged gate”—hard sandbox only when opening to the outside?

Before untrusted input enters orchestration: wrap with a fence, neutralize dangerous instructions, and only dispatch after a human clearance tag.

Pitfall: Without an explicit bar, gut feel replaces acceptance.
Judgment: Default conservative at launch; clearance signals are machine-readable and out of the caller’s rewrite reach.

5.2 After a config change, did you validate and confirm the process is actually listening?

Schema/config validates; the service is really listening; deleted agents left no orphan channel accounts.

Pitfall: Thinking a “disable” flag is enough—schema errors take the process down, or orphan channels remain.
Judgment: Physically separate preview and production stores; after changes, validate + confirm listen.

5.3 When humans step outside orchestration, do the final owner, escalation path, and machine-readable success/fail signals remain?

Final owner, which errors must escalate, and one-click kill are written down.

Pitfall: Hiding behind “the AI said so”; nobody is on the hook.
Judgment: Wrong words from an external agent still land on the company—write “when a human must take over” first.

6. Cost & iteration

6.1 Are tokens spent on real workflow work—not idle residency?

Wake on events; orchestration can run near-zero-token pre-checks before waking an expensive execution model.

Pitfall: A rising bill isn’t always usage—it can be silent retries on failing tools.
Judgment: Instrument tool errors before swapping models; event-driven + pre-check, then on-demand wake.

6.2 Did you run one real business path yourself before stacking scaffolding?

At least one real input has run end-to-end before you abstract skills / multi-tenant / graphs.

Pitfall: Half a day of skills, ADRs, and multi-tenant—and still no real draft delivered.
Judgment: One end-to-end delivery first; reuse second.

6.3 Have requests you’ve seen twice become skills—and only then cron?

Path: manual runs a few times → harden as a skill → cron only when automation is worth it.

Pitfall: A full tool belt ≠ capability; unused tools still burn context every turn.
Judgment: Audit never-called tools first; harden after you’ve seen it twice—don’t live forever in one-off manual work.

Get the full edition with Mortal’s Compass

That’s the public short version.

The full edition adds: redacted practice cases, public case sources, three ink-and-line diagrams (trust planes / dev·prod isolation / silent-retry cost loop), and a printable “stage × key questions” quick table.

Subscribe to Mortal’s Compass to get the full edition. No paywall for now—subscribers can use it directly.