Grounded covenant review: putting an LLM inside an approval workflow people actually trust

If you live in an HOA and want to repaint your house, replace a fence, or put up a shed, you file an architectural review request. A volunteer committee then compares what you proposed against the community's architectural guidelines, which is a PDF that runs anywhere from four pages to ninety, and votes.

Two properties of that process shape every technical decision that follows. First, the guidelines are per community. There is no universal rulebook to train against or embed once. Second, the output is a decision that affects someone's property, delivered by volunteers who can be challenged on it. A confident, unsourced answer is worse than no answer.

Committees were taking days to weeks per submission, mostly spent hunting through the PDF for the clause that applied. That is the task worth automating: not the judgment, the lookup.

The review result feeds a database record and a UI, so it has to be a schema, not a paragraph. The common approach is to ask for JSON in the prompt and parse what comes back, which means writing defenses against markdown fences, preamble text, and trailing commentary.

Instead the request declares a single submit_review tool and forces it:

["tools"]       = new JsonArray { tool },
["tool_choice"] = new JsonObject {
    ["type"] = "tool",
    ["name"] = "submit_review"
},

The model cannot reply with prose. Its only move is to call that tool, and the tool's input_schema is the contract. Reading the result is finding the tool_use block and deserializing its input straight into a C# type. There is no parsing layer, and therefore no parsing layer to debug at 2am.

The first field in the schema is assessable, a boolean, paired with reason_if_unassessable. The instruction attached to it is explicit: if there is no relevant guideline, or the submission is missing information needed to judge it, set assessable to false and say why. Do not guess.

This matters more than any other part of the design. A homeowner submits a request to install a generator, and the community's guidelines simply never mention generators. The failure mode without an explicit refusal path is not that the model errors. It is that the model produces a plausible, well-written, entirely invented covenant citation, and a volunteer committee, reading quickly, forwards it to a homeowner.

Giving refusal a first-class slot in the schema makes "I cannot tell" a success case that the UI can render, rather than an outcome the model has to fight the format to express.

Every review sends that community's full guideline document. Across a season of submissions for one community, the same tens of thousands of tokens go over the wire repeatedly, and they are the overwhelming majority of the request.

Prompt caching fixes this, but only if the content is ordered for it. A cache applies to a prefix, so anything reusable has to come before anything that varies. The guidelines are stable per community; the submission text and its photos change every time. So the content array is assembled stable-first, with the cache breakpoint set on the last guideline block:

The savings are measured, not assumed. Every response's usage block carries cache_creation_input_tokens and cache_read_input_tokens, and both are persisted with the review alongside the ordinary input and output counts. Cache hit rate per community is a number in the database, so a regression in prompt assembly shows up as a cost change rather than as a surprise on the monthly bill.

Guideline documents arrive as PDFs of wildly varying provenance. Some are clean exports with a real text layer. Some are scans of a photocopy of a document from 1998.

The client prefers extracted text and falls back to sending page images only when a document has no usable text layer. Vision handles the scans, which is the capability that makes the feature possible at all for older communities, but it is materially more expensive per page and slightly less reliable on dense clause text. Submission photos, being actual photographs of fences and paint swatches, always go as image blocks.

Two features call the API and they are not the same problem.

• Covenant review is high stakes, low volume, and requires reasoning over a long document to produce a cited finding. It runs on the frontier model.
• Channel summarization condenses a discussion thread so a reviewer can catch up. It is low stakes, higher volume, ephemeral, and never persisted. It runs on the faster, cheaper model with a 1k token ceiling.

Picking one model for the whole application means either overpaying for summaries or under-resourcing the decision that actually matters.

LLM calls are network calls, and the interesting part is which failures are worth retrying. Rate limits and server-side errors (429, 500, 502, 503, 529) get up to four attempts. Everything else, notably a malformed request or a bad key, fails immediately, because retrying a 400 just turns one error into four.

Backoff honors the server's Retry-After header when present and otherwise doubles, capped at twenty seconds, with random jitter so that a burst of submissions does not retry in lockstep. Timeouts and transport exceptions are treated as transient, but a caller's cancellation is not, so a user navigating away stops the work instead of retrying it.

The API key lives in Azure Key Vault and is read through the same configuration path as the database credential. The client exposes an IsConfigured check, which is what makes the next part possible.

The feature is opt-in per community. A management company can pilot it with one association before enabling it across a portfolio. Where it is not enabled, or where no key is configured, the review workflow behaves exactly as it did before rather than erroring, which means the AI path can never take down the core product.

The output is framed throughout as a draft. The model produces flagged issues with citations and a proposed decision letter; the committee votes. That is not hedging, it is the correct division of labor. The model is good at finding the clause on page 47 that mentions fence height. It has no standing to decide what happens to someone's property.

Guidelines are currently sent whole. That is the right call while documents are tens of pages and the cache absorbs the cost, but a community with a two hundred page document plus decades of amendments will eventually justify real retrieval over the document set rather than a full-context send.

The larger gap is evaluation. Correctness today is judged by committees accepting or editing the draft, which is a real signal but a slow and lossy one. The version of this I would build next starts with a labeled set of historical submissions and their actual outcomes, so that a prompt change can be scored before it ships instead of after.