AI · Engineering

Adding AI features without shipping nonsense

Most AI features that disappoint do not fail because the model was not good enough. They fail because of everything wrapped around the model call, and almost none of that is the interesting part anyone wants to work on.

The model is rarely the problem

When a feature that uses a model goes wrong in production, the cause is usually mundane. A response came back in a shape the code did not expect. A prompt was edited six weeks ago to fix one case and ended up breaking another. Nobody can reproduce the failure, because the exact prompt that produced it was never recorded. The output went straight into a customer record with nothing checking it.

These are not model problems. They are integration problems, and they are the same class of problem you already know how to solve. You would not accept an HTTP API that returns free-form text where you asked for JSON. You would not ship a payment integration with no validation on the callback. AI features deserve the same discipline, and mostly do not get it because the demo worked on the first try.

Four ways this goes wrong

1. Free text where you needed a structure

Asking a model for prose (plain text) and then trying to deduce (parse) meaning out of it with string matching is the most common mistake we see. The prompt should ask for the exact structure your system already accepts, and the system should refuse anything that does not match. If your feature needs a title, a summary and three tags, ask for those fields and validate them. Do not ask for “a blog post” and hope.

2. Prompts that drift away from the data

A hand-written prompt is documentation that nothing enforces. Someone adds a field to the schema, forgets the prompt, and the model keeps producing the old shape. The fix is to generate the prompt from the schema rather than writing it beside the schema. Then the two cannot disagree, because there is only one source.

3. No human in the loop where it matters

There is a large difference between a model that drafts and a model that commits. Drafting is low risk: a person reviews, edits, approves. Committing is high risk: the output becomes truth in your database or a message to your customer. Decide which one you are building. Most good AI features are drafting tools with an obvious review step, and that is not a weakness.

4. One provider wired through the codebase

If a vendor name appears in twenty files, changing model is a rewrite and negotiating power sits entirely with the vendor. Put the provider behind one interface. Then the model is a configuration decision, and you can move when pricing changes or a better option appears.

What to do instead

1. Define the output shape first. Write the schema your system will accept before you write a prompt. The schema is the contract; the prompt is just how you ask for it.
2. Generate the prompt from the schema. One source of truth. Regenerate when the schema changes.
3. Validate every response. Reject anything that does not match, and log the rejection with the input that caused it. A malformed response should be a handled event, not an incident.
4. Return drafts, not commits. Give the person a review step, and make applying the result undoable.
5. Keep the provider behind an interface. One seam, one place to change.
6. Cap the cost. Timeouts, limited retries, and a log of what each feature actually spends. A runaway loop should be boring, not expensive.
7. Keep your test cases. A handful of inputs with known-good outputs lets you tell whether a prompt change improved anything or merely changed it.

When not to use a model at all

A surprising share of requested AI features should not be model calls. Categorising a record into one of six known buckets, when the rules are known and stable, is a lookup. Validating a phone number is not a language task. Extracting a date from a structured field is parsing.

Models are worth their cost where the input is genuinely unstructured, the space of correct answers is wide, and a human would otherwise be doing the work by hand. If you can write the rule, write the rule. It is faster, cheaper, deterministic, and it does not hallucinate on a Sunday. We regularly talk clients out of a model call for exactly this reason, and we would rather do that than bill for something you did not need.

The prompt is an artifact, not a paragraph

The most useful shift in this whole area is to stop treating the prompt as prose that somebody wrote once, and start treating it as a file your build produces. If your system has a schema, you already know every field the model has to return, which ones are required, what the valid enum values are and what the defaults should be. That is the entire contents of the instruction. Generate it.

Four things become possible the moment the prompt is generated:

• It cannot drift. Change the schema and the instruction changes with it, in the same commit.
• It can be fingerprinted. Record a hash of the source schema it was built from, so you can tell whether a deployed prompt matches the code it is supposed to serve.
• It can be regenerated in CI. A schema change that forgets to regenerate the skill becomes a failing build rather than a mystery in production.
• It can be handed to your users. This is the part most teams miss, and it is the one that changes the commercial picture.

Shipping the skill instead of building a chat box

A generated skill file is a plain markdown document that you give to an AI assistant. It describes the schema in terms the model can follow: every block type, which fields are required, which have defaults, what nests inside what. If you hand that to an assistant your customer already pays for, it can produce content your system will accept on the first attempt, with no integration work on your side at all.

We do this in BlogForger. A generator reads the shared schema files and writes a skill document covering block types, including those that nest, along with the two object schemas for posts and emails. It records a fingerprint of the schema it was built from and is regenerated whenever those files change. Users download it from the sidebar and hand it to their own assistant. A second skill file covers something different: it teaches an agent to drive a workspace over the public API, adding contacts to a list, setting merge fields and queueing sends using a scoped key pair.

How we build this

We built this pattern in our own product. BlogForger. has a model drafting layer with a provider registry that resolves a model per request, prompts generated from the data schema so output stays in step with the block types the editor actually understands, drafts validated as structured blocks rather than free text, and a review step before anything is applied. It also ships the two downloadable skill files described above.