AWS reports 100% document classification on 20 files. Treat that as a pilot signal.

AWS published a three-agent document-classification design on August 18, 2026 that combines text reasoning, visual similarity, and a validation agent. In AWS's test, the system classified all 20 insurance documents correctly, while Bedrock Data Automation classified 70% correctly.

That result is useful evidence for trying the design, but it is not a measured production error rate. The set had only three classes and just two examples in one of them. This article explains what the architecture adds, what the reported numbers support, and how to decide whether your document queue needs all three agents.

What did AWS actually test?

AWS's implementation post describes a set of 20 documents: 8 policies, 10 affidavits, and 2 miscellaneous files. The comparison reports aggregate accuracy for four approaches.

Scroll sideways to see all 3 columns.

Approach	Reported accuracy	Reported average time
Amazon Textract plus keywords	25%	2.88 seconds
Amazon Comprehend plus entities	25%	3.26 seconds
Amazon Bedrock Data Automation	70%	25.7 seconds
Three-agent system	100%	23.3 seconds

AWS calls this a limited evaluation and says production accuracy may vary with larger and more diverse document sets. It also says the run did not use fine-tuning or custom classification logic. That makes the comparison a useful implementation result under the published conditions, not a general ranking of the four services.

The class mix matters. A system can score 20 out of 20 and still have no tested evidence for the many layouts, scan qualities, languages, revisions, and edge cases that did not appear in those 20 files. The two miscellaneous examples are especially thin support for predicting behavior across an open-ended catch-all class.

Why use three agents for one classification?

The first specialist is a Document Analysis Agent running Claude Haiku 4.5 on Amazon Bedrock. It reads up to the first 3,000 characters in the sample implementation and returns one of three labels with a confidence value and reasoning.

The second specialist is a Vector Similarity Search Agent. It converts the first PDF page into a Titan Multimodal Embeddings vector, retrieves the three nearest examples from a FAISS index, and uses their labels as evidence. This lane is meant to capture structure: form layouts, tables, and formatting patterns that can distinguish documents with similar legal vocabulary.

The Validation Agent calls both specialists, compares their outputs, and returns the final class, confidence, justification, and a requires_human_review flag. The sample tells the orchestrator to favor textual evidence for content-rich documents, favor vector evidence for visually distinctive formats, and escalate contradictory evidence.

Those dependencies explain why the design can outperform a text-only or extraction-oriented baseline on documents where meaning and layout both matter. They also create more moving parts: two model-assisted analyses, an embedding index built from known examples, an orchestrator, logging, threshold behavior, and a human-review path.

Architecture should follow the observed failure. If a text baseline already separates your invoice, claim, and correspondence classes at the required service level, a second analysis lane may add latency and operating work without changing the routing decision. If the misses cluster around forms that share vocabulary but use reliably different layouts, visual similarity has a specific job.

What does 100% accuracy not tell you?

On this set, 100% means 20 correct classifications out of 20 reviewed files. It does not estimate how often the system will be wrong after document sources change, a template is revised, a low-quality scan arrives, or a new subtype enters the miscellaneous queue.

The published page does not describe an independent train-and-test split, repeated runs, a confidence-calibration study, or a production drift period. It also does not publish a per-document dollar cost. BaristaLabs therefore would not use this result to promise an error rate, set a straight-through-processing target, or approve a production budget.

AWS separately says Haiku 4.5 averaged 19.3 seconds per classification at 93% confidence. The comparison table reports 23.3 seconds for the multi-agent system, and the page does not reconcile the two timing figures. More importantly, a model-produced confidence of 93% is not the same as 93% measured accuracy or a 93% probability of correctness unless the score has been calibrated against observed outcomes.

A human-review threshold needs that calibration. If cases below 0.8 go to an analyst, test how many actual errors appear above and below 0.8 on representative files. Also measure how much of the queue the threshold sends to review. A threshold that catches errors but routes most documents to people may be safe without meeting the automation goal.

How should a business test whether the visual lane earns its place?

Start from the queue, not the reference architecture. Assemble a sanitized corpus that preserves the document sources, class proportions, scan quality, page counts, languages, template versions, and costly edge cases the operation actually receives. Keep an untouched holdout that is not used to write prompts, choose examples, or tune the FAISS index.

Run a simple text baseline first. Record per-class precision and recall, review volume, latency, and the operational cost of each error type. A policy sent to the wrong compliance queue may carry a different cost from a miscellaneous attachment held for review, so aggregate accuracy alone is not enough.

Add the visual-similarity lane only when the baseline's reviewed misses show that layout or structure contains useful evidence. Then rerun both designs against the untouched holdout and across repeated invocations. The comparison should show whether the extra lane fixes the targeted errors without creating unacceptable new ones.

Before straight-through routing, calibrate the escalation threshold against analyst decisions and preserve a reject path for unknown classes. In production, watch class prevalence, reviewer overrides, low-confidence volume, latency, and index freshness. A newly introduced form can change nearest-neighbor behavior even when the models and prompts stay fixed.

AWS recommends PII redaction, topic constraints, model-invocation logging, and escalation for uncertain cases. Those are useful controls, but logging sensitive insurance documents can itself create retention and access obligations. Decide which inputs, outputs, similarity matches, and reasoning fields are necessary for audit before enabling full request-response logging.

When is the three-agent design justified?

The design is a credible pilot candidate when text and layout carry different, measurable classification signals; a wrong route has enough financial or compliance cost to justify added latency and review; and the team can maintain representative examples, an untouched evaluation set, and a monitored human-review path.

A simpler classifier remains the better choice when classes are already separable by stable text rules, document volume is too low to support evaluation, or nobody owns the exception queue. The decision is not whether three agents sound more capable. It is whether the added lane fixes a documented error pattern on files the system did not tune against.

AWS's 20-of-20 result clears the bar for investigation. Your own holdout, calibrated escalation rule, and production drift record have to clear the bar for automation.

Source

• AWS Machine Learning Blog: Implement vector-prompt document classification using Amazon Bedrock, published August 18, 2026.