Building Robust AI Intelligence Processing Systems: Lessons from the Front Lines

While testing an AI system I built to process intelligence feeds, I found that it could produce a strong analysis and still lose important reports. The model would pick the most urgent event from a batch, analyze it well, and leave other useful information out. The system then advanced its processing state as though the whole batch had been handled.

Fixing that meant changing how reports moved through the system, how duplicates were identified, and when a report counted as processed.

Why I started with batches

Batch processing seemed like a sensible approach. Send several reports to the model in one call, and it could connect information across sources, spot patterns, and produce a broader strategic assessment. Fewer API calls also meant lower costs.

The custom model I developed was trained to act as a strategic analyst. Its prioritization logic focused on kinetic events, pivotal developments, and emerging threat patterns. Given several reports, it concentrated on the most critical intelligence.

That was useful for deciding what deserved immediate attention. It also made it easy to overlook what the output left behind.

How useful reports disappeared

During production testing, I found a flaw in the way the system tracked its progress. It would collect reports A, B, C, and D, generate a synthesis of the most important material, and mark the newest report, A, as processed. Reports B, C, and D were then permanently lost from the processing pipeline, even if they contained information the synthesis had never addressed.

The model was doing the prioritization it had been trained to do. The surrounding system was treating that selective analysis as evidence that every report had been handled.

Consider a hypothetical batch: one report covers a military engagement, another a significant diplomatic development, and a third a new cyber threat vector. The model focuses on the military engagement and produces an excellent analysis. If the system then moves past the entire batch, the diplomatic and cyber reports disappear with it.

Their information may be valuable and unique. Being less urgent than another report is no reason to discard it.

Deciding what is actually a duplicate

Processing every report individually still leaves a problem: intelligence feeds repeat themselves. The system needs to avoid redundant outputs while recognizing meaningful changes in a developing situation.

My initial duplicate detection was too aggressive. Reports about the same topic or geographic region could be treated as redundant, even when they described new deployments, another country entering a conflict, or an escalation in scope or scale.

Take another hypothetical example. Yesterday’s report said, “Country X increases military posture.” Today’s says, “Country Y deploys naval assets to the region.” Both concern the same conflict, but the second introduces a new actor and action. In the scenario described here, it represents an escalation worth reporting. That judgment depends on context, including the potential for escalation and the significance of the new actor.

I worked with Claude and Gemini to explore different architectures. The approach I settled on gave each report its own analysis, checked for genuine duplication, and explicitly looked for material developments.

I tightened the duplicate criteria around reports that:

• Cover the same event without new developments
• Repeat earlier analysis without adding context
• Describe the same incident without new actors, actions, or implications

The system also needed to recognize developments such as:

• New actors entering an existing situation
• Escalation in tactics, weapons, or geographic scope
• Changes to the timeline or a new phase of an ongoing situation
• Additional geographic areas being affected
• Changes in threat levels or alert status

These checks gave the system a more useful basis for comparison than a shared topic or location.

The revised processing flow

The final design retained the full analytical model but handled one report at a time:

1. Generate an analysis of the new report.
2. Run the enhanced duplicate check.
3. If the report contains a material development, process it and produce an output.
4. If it is a genuine duplicate, skip the output but retain the report for possible reprocessing.
5. Update the processing state only after a successful output.
6. Move to the next report.

That fifth step matters. Evaluating a report and deciding to skip it should not mark it as processed. A report judged redundant at one point may deserve another look when circumstances change. Retaining it makes that possible.

I also made the duplicate check default to “not similar” if it failed. For this system, the cost of occasionally producing similar analyses was lower than the cost of suppressing a critical development. A failed check therefore allowed the report to continue through processing.

What the change cost, and what it fixed

Individual processing requires more computing resources than batching. In return, each report gets its own analytical attention. An unrelated report can no longer cause it to be dropped, and a failure on one report does not affect the others. The system records progress after each successful output rather than advancing past a whole group.

In production, the revised approach delivered four improvements:

• Zero data loss, with every report receiving an individual evaluation
• Fewer false duplicate matches, allowing material developments through
• Better coverage of important reports that would previously have been overshadowed by a more urgent item
• More reliable processing, because incremental state updates prevented large-scale reprocessing

The additional processing cost bought better control over what the system handled, what it skipped, and what remained available to revisit.

Where else this applies

The same problem can arise anywhere a system prioritizes an information feed. News analysis needs to distinguish a repeated story from a developing one. Security monitoring needs to separate routine alerts from genuine escalation. Financial intelligence needs to catch material changes in ongoing situations, and research synthesis needs to preserve complementary findings.

In each case, a useful summary can leave valuable information out. The processing design needs to account for that possibility.

For intelligence systems where missing a report can have serious consequences, I favor giving each report its own evaluation and using careful duplicate detection to control repetition. Batching still offers efficiencies and opportunities for synthesis across sources, but those benefits have to be weighed against the risk of losing information.

As AI systems take on more consequential information flows, tracking each report becomes more important. Every report needs a clear outcome, skipped material needs to remain available when context changes, and the system’s record of progress needs to reflect work it has actually completed.