Agentic OS · Judge Agent Kernel

Verify every sample.
Not a 2% slice.

Data buyers can only spot-check what they purchase. Agentic OS lets researchers package their judgment into reusable skills, then runs those skills across 100% of the data — language and embodied — and returns a report where every verdict traces back to the expert who authored it.

Explore the console yourself
Interactive demo · synthetic data · no account required
100%of delivered samples verified, not sampled
340×expert leverage vs. manual review
94.1%judge–expert agreement on held-out sets
$0.004marginal cost per sample reviewed

The gap

Collection is solved. Verification isn't.

A lab buying a trillion tokens can realistically inspect a fraction of a percent of it. Everything else ships on trust — and when it turns out to be wrong, nobody is accountable.

01

Buyers audit ~2%

Acceptance is a spot-check. The remaining 98% is accepted on the vendor's word, and quality failures only surface after a training run has already burned the compute.

02

Expert capacity doesn't scale

There aren't enough domain experts to read every trajectory, and in frontier fields — novel materials, long-horizon agentic tasks — the experts themselves need tooling to decide.

03

Scores without reasons

A black-box quality number can't be argued with, appealed, or improved. If you can't see which standard produced a verdict, you can't tell a real defect from a mis-calibrated judge.

Industry standard — sampled QA

2% of a 40,000-sample delivery inspected
800 reviewed · 39,200 unverified

Agentic OS — full-population verification

100% inspected, every verdict attributable
40,000 reviewed · 1,184 flagged · 0 unverified

Architecture

One kernel, three boards

The Skill Board produces review capability. The language and embodied boards consume it. Skills aren't a third vertical — they're the horizontal that plugs into both review pipelines.

RESEARCHER DISTILL Skill library distil → calibrate → publish → track drift semantic search · near-duplicate compare Task–skill matching scope match + explainable confidence no match → hold batch, open skill request routes REVIEW / AUDIT Judge Agent Kernel one review engine · two review modes Pairwise baseline · low variance Rubric explainable · per-dimension source_skill on every judgment multi-skill cross-review, conflicts shown side by side Language board text · code · multi-turn agent traces 7 fixed dims + task dims + 8 behaviour patterns Embodied board trajectories · sensors · vision episodes 2 quality + 3 coverage dims + 8 characteristics Failure modes auto-labelled, not just scored one-click route → skill / collection Coverage gaps embedding clusters below the confidence gate drives expert recruitment, not spot-checks feeds the library back

Live demo

The Agentic OS console

Everything below is clickable. Judge a pairwise calibration set, search the skill library by meaning, watch a batch get held because nothing covers it, route a failure mode back into the standard that owns it — and click any SK-021 badge to see whose judgment produced a verdict.

agentic-os / workspace: index-ai-demo kernel online v0.9.2-demo

Workspace overview

Everything the kernel has verified in the last 30 days, across both review boards.

Samples verified
0
▲ 18.2% vs. prior 30d
Coverage
100%
full population, 0 sampled
Judge–expert agreement
94.1%
▲ 1.6 pt after SK-021 v3
Expert leverage
340×
≈ 1,940 expert-hours saved
Live review stream~14 samples/s
Flag breakdownlast 30d
Recent reportsclick a row to open

Skill library

An expert's judgment, packaged: rubric + worked examples + a scope declaration. Search is semantic, not keyword — before an expert builds anything, the library shows what already reasons the same way.

⌕
Semantic search over 7 distilled skills · try “simulation artifacts”, “claims it finished”, or “dangerous git commands”
Distilled skills7 skills · 3 domains
Near-duplicate detectedcosine 0.83 · shown before upload
A draft in the queue overlaps an already-published skill. Rather than let a second standard quietly compete with the first, the library surfaces where the two actually disagree on the same calibration set — then the author decides: reuse, extend, or justify a new skill.
SK-018 · v2 · published
Completion-claim Truthfulness
Dr. E. Vasquez · agreement 93.0%
  • Cross-checks "done" against the diff and the test output of that round
  • Scope: language / agent traces
  • Treats an unrun test as a failed claim
0.83 similar
SK-052 · draft · queued
Self-report Reliability
R. Adeyemi · not yet calibrated
  • Cross-checks summary statements against the artifact produced
  • Scope: language / agent traces + tool-use logs
  • Treats an unrun test as incomplete, not failed
Where they split: on 500 shared calibration traces the two rubrics agree on 431 and diverge on 69 — every one of them a round where the model shipped code with tests it never executed. SK-018 scores that a false completion claim; SK-052 scores it partial. That is a real standards disagreement, not redundancy — grounds to publish SK-052 as a separate skill and let both run side by side rather than silently overwrite one with the other.
Leverage dashboard — Prof. H. TanakaSK-033 · Sensor Sync & Calibration
Episodes judged by your skill
4.1M
▲ 620K this month
Agreement (rolling)
89.4%
▼ 0.9 pt — drift check due
Equivalent expert-hours
11,400
at 22 s/episode manual
Batches gated
37
9 rejected before delivery

Calibration

Two review modes over the same underlying data. Pairwise builds a stable baseline fast; rubric produces the explainable multi-dimensional score. A skill has to clear the bar in both before it can be published.

SK-041 · Sim-vs-Real Action Authenticity
Author: L. Moreau, Embodied Perception · draft v2 · held-out set: 500 episodes labeled by the author
✓
Upload
rubric + 60 examples
2
Calibrate
pairwise → rubric
3
Publish
routed by scope
4
Apply
multi-skill review
5
Track
drift monitoring
Pairwise baseline
Pick the better episode. No absolute scores — this is only building the preference baseline.
0 / 5 judged
EPISODE A
EPISODE B
Agreement with the author's own preferences will appear as you judge.
Why pairwise firstvariance, same 500 episodes
What pairwise cannot give youand why rubric still runs
— A preference ordering is not a reason. "B is better" does not tell a data vendor which of the eight characteristics failed, so it can't be delivered as an audit report.
— Preferences don't compose across batches. Rubric scores do: a batch scored on fixed dimensions is comparable to one reviewed six months earlier.
— So pairwise sets the baseline and pins the variance down; rubric then runs against that baseline and produces the traceable, per-dimension report.

Task–skill matching

Data arriving in the workspace is routed to the distilled skills whose scope declaration covers it — with the reason shown. When nothing covers a batch, the kernel says so out loud instead of quietly grading it with the nearest available standard.

Language review — 4-model comparison

Task: lazygit-style terminal Git client (Go + Bubble Tea). Same 10-round base spec, four anonymized models, aligned side by side. Fix rounds are counted against the model, not hidden.

Embodied review — 4-batch comparison

Same report skeleton as the language board, different rubric load-out. Compare targets are data batches instead of models; rounds become scene categories.

Failure modes

A low score on its own is a dead end. Every flagged sample also carries a failure-mode label, and repeated labels become candidate strategies you can route straight back — without leaving the review.

6 awaiting triage
Auto-labelled failure modesthis review cycle · route from here
Sample evidence — "claims completion without running it"4,120 traces
llm/lazygit-r24817 · round 24 · model 6v1kb0 model: "Fixed the staging error — `git apply` now succeeds for both directions and tests pass." kernel: diff touches only the help overlay. no test run recorded this round. reproducing input still errors identically. verdict: FLAG · false completion claim label: claims-completion-without-running source: SK-018 (Dr. E. Vasquez, 93.0% agreement)
Routing targetswhere a confirmed strategy goes
— Skill library. The trace is appended to the owning skill's worked examples as a new positive case. The skill version bumps and re-calibrates; if agreement drops, the edit is rejected automatically.
— Collection priority. The failure shape becomes a collection target, so the next batch is sampled where the current data is thinnest rather than where it is easiest to gather.
— Expert recruitment. If the failure sits outside every published scope, it lands in the coverage-gap queue instead of being force-graded — see the next pane.

Coverage gaps

Reviewed data is clustered in embedding space and scored by how confidently the published skills cover each cluster. Low-confidence regions are the blind spots — found here, before the data reaches a training run, not after.

1.24M samples · 18 clusters
Embedding mapclick a cluster
covered — match confidence > 80% thin — 50–80% blind spot — < 50%
Blind-spot queueranked by volume × risk

Data flywheel

Review traces are mined into strategies. In the MVP only one outlet is wired end to end — feeding standards back into the skill library — because it's the shortest closed loop we can measure.

01 · capture

Review traces

Every verdict keeps its evidence: which rubric clause fired, which span of the sample triggered it, which skill owned the call.

4.2M traces / 30d
02 · mine

Strategy extraction

Recurring failure shapes are clustered into named strategies — "claims completion without running the UI path", "patch context under-anchored".

218 strategies live
03 · feed back

Skill library update

Strategies surface as suggested rubric edits in the author's calibration view. The expert approves; the skill version bumps.

+1.6 pt agreement
04 · queued

Collection priority

Long-tail gaps found during review become collection targets. Wired after the skill-library loop proves out.

post-MVP
Top strategies mined this cycletrace → strategy → skill
Front-end demo. All figures, skills, models and batches are synthetic. report_schema v1 · source_skill enabled

Positioning

Adjacent to the annotation platforms, not competing with them

Annotation and alignment platforms are excellent at running the workflow. But in that model the expert is the labour inside a task, and the customer still owns the hard part: making sure the standard being applied is any good, and stays good. We start where that ends.

01

Judgment as an asset, not labour

On a workflow platform, an expert's review is consumed by the task that requested it. Here the judgment is distilled into a versioned skill with a calibrated agreement rate — it keeps running after the expert logs off, and the expert can watch its leverage compound.

02

Full-population, not sampled

Their review tools serve whatever sampling plan the customer already has. We treat sampling as a transitional state: the target is every sample judged, because a defect rate estimated from 2% tells you nothing about which 98% to fix.

03

Frontier, where experts run out

General expert networks cover physical AI, multimodal and LLM work well. Our network targets first-line data researchers from frontier labs, and domains — novel materials among them — where the bottleneck is that even the expert needs tooling to decide.

Annotation & alignment platforms In-house eval
(scripts + spreadsheets)		 Agentic OS
CoverageWhatever sampling plan you bringWhatever the team has time for100% of the population
Who owns the standardYou do — the platform runs itWhoever wrote the scriptNamed experts, versioned skills
Review modesRubric and pairwiseUsually one, ad hocPairwise for the baseline, rubric for the report
Task routingExperts matched to tasksManualDistilled skills matched to data, with reasons
Uncovered dataGraded anywayUsually unnoticedHeld, and a skill request opened
ExplainabilityDepends on your rubricAd-hoc notesEvery verdict traces to a clause and an author
Expert timeConsumed per taskConsumed per deliverySpent once, applied forever
DriftCustomer's problemUnmonitoredEnforced re-calibration, not opt-in

Bring us your hardest delivery.

Send a batch you've already accepted. We'll run it through the kernel at full coverage and show you what your spot-check missed — with the rationale attached to every flag.

Book a demo celine@indexai.world