Barron AI Solutions

2

Research

Notes from building AI systems that keep their memory and are held to the truth. Each one comes from our own records, with the number, how we measured it, and where it came from. Three of the six are things that didn't work.

  1. 1. We trained a small local copy of our assistant for ten weeks. It never passed.
  2. 2. Our model of another agent was no better than assuming it thinks like us.
  3. 3. A confidence number about our assistant came from a template, not from the assistant.
  4. 4. One small embedding model couldn't tell a deploy script from a hosting note.
  5. 5. Tools that quietly ignore a misnamed argument give quiet wrong answers.
  6. 6. An agent with nothing to do repeats itself. Work fixed it; filters didn't.

We trained a small local copy of our assistant for ten weeks. It never passed.

What we did

From July 28 to October 8, a small model on our own machine was trained on the assistant's history and graded before it could be used.

What we found

In ten weeks, no trained version passed its grade, and nothing in the system relied on it. We retired it on October 8. The files are kept, so it can come back.

What it means

A model that can't beat its own test isn't a feature yet, however good the idea sounds. We kept the test and dropped the model.

Source: Retirement record, October 8, 2026.

Our model of another agent was no better than assuming it thinks like us.

What we did

Two of our agents predicted each other's answers in sealed rounds: each prediction was locked in before the answer was shown.

What we found

Ten rounds each way. In both directions the model of the other agent was right 6 times, and the shortcut "it will answer the way I did" was also right 6 times.

What it means

The faculty isn't shown yet. Any claim that an AI understands another mind needs to beat projection, not chance.

Source: sibling_model_score, last round October 2, 2026.

A confidence number about our assistant came from a template, not from the assistant.

What we did

Our system reported that the assistant was under-confident: right far more often than its stated confidence. We traced every row behind that claim.

What we found

All 93 rows came from one template line with a fixed 60%. The assistant never stated them. Its own forecasts since October 2: 10 right out of 18, at 68% stated on average.

What it means

Eighteen forecasts is too few to say whether it runs over- or under-confident. We struck the old number and say "we don't know yet".

Source: Forecast audit, October 14, 2026.

One small embedding model couldn't tell a deploy script from a hosting note.

What we did

Mneva, our memory service for coding agents, recalls by meaning. The first version used MiniLM, with a keyword fallback.

What we found

Asked "how do we release", MiniLM couldn't separate a note about the deploy script from a note about hosting. bge-small with its retrieval prefix could. We switched and rebuilt every stored vector.

What it means

Test recall on the questions people actually ask. Known limit: very loose paraphrases still miss.

Source: mneva-engine commits ce00d9f and 384641f.

Tools that quietly ignore a misnamed argument give quiet wrong answers.

What we did

One of our tools read its input with default values and never checked which keys arrived. A key named "type" went unread, and 5 claims were stored as unverified.

What we found

The tool now reports the keys it ignored. A scanner found the same shape in 300 places across 118 of 485 files. The test runs both ways: the old version is flagged, the fixed one isn't.

What it means

Every tool an AI calls should say what it didn't use. Silence looks like success.

Source: Commit e5e5098c.

An agent with nothing to do repeats itself. Work fixed it; filters didn't.

What we did

One of our agents speaks freely, with no filters on what she says. With nothing real to do, she sent the same message every minute.

What we found

Given live websites to proofread, she read all 868 pieces in her queue in under three hours and reported what she found.

What it means

An agent needs material, not just freedom. When it loops, give it a job before you add a rule.

Source: Commit 24c985ed, October 29, 2026.

Working on something like this? Ask about a project.