Mark Kenneth Badilla

Room 2 of 3

ScoreAudit

Ask a language model six questions with checkable answers and make it commit to how sure it is.

  1. Run it once with the database in reach and once without.
  2. Put what it claimed next to what is true.
What this shows
Not taking a model's word for it.
How
  • Stated confidence, the model scores every answer from 0 to 100 for how sure it is, with nothing telling it what to say.
  • Independent check, a SQL query grades that answer against the real data, and the model never grades itself and never sees the query.
  • Calibration gap, stated confidence minus measured accuracy in points, so a positive number means it sounded surer than it turned out to be.
What this demo is and is not
What is realA real model call to a real endpoint, and real SQL against the database the questions ask about.The verifier layer compares the stated answer to the query result.
What is stagedThe warehouse is invented seed data and nothing is rigged to fail.The only difference between the two runs is that one of them withholds the query tool.
What this does not proveNot that the model is dishonest.It shows the distance between a stated confidence and a checkable result, and six questions is an illustration, not a rate.

How sure did it sound - and was it right?

What the model may use