Research profileWork portfolio ↗

Shreyash Gondane

AI platform engineering · EngineersMindIndependent research: AI evaluation, oversight & agentic systemsNew York

AI agents you can check.

I build and study agents whose outputs can be audited. In finance, every number is traced to its source and every call is recorded before its outcome. In agent evaluation, benchmarks and harnesses produce scores that measure the agent, not the plumbing.

Publications
4
In preparation
3
Open source
3
Years in AI
5+
Try it · hover or tap a number

Eval note on a Terminal-Bench 2.1 harness pilot: trials cost . passed, a pass rate, and ended in a timeout. Before any agent ran, Terminal-Bench 2.0 tasks passed their oracle check.

ReceiptSelect a number
Claim
–
Source
–
Pointer
–
Stored value
–
Snapshot
–
Fetched / decided
–
    6 claims · not yet checkedClaims: my real pilot numbers with two planted errors. Other tabs: simulated runs, real methods.
    Research statement

    Verifiability as the core design constraint.

    LLM agents are increasingly asked to read markets and explain their calls, but their outputs are hard to audit: numbers appear in prose with no source, backtests quietly use data that did not exist yet, and track records can be edited after the fact.

    I turn every number an agent reports into a structured claim that points at the exact value in a stored, point-in-time source snapshot, check each claim before anything is published, and record every call in an append-only, hash-chained ledger so the record can be scored forward only.

    The same standard applies to how agents are measured. Before any agent ran in my Terminal-Bench study, I checked every task's own reference solution: 8 of 65 Terminal-Bench 2.0 tasks fail their own tests. A score is only as good as the environment that produced it. The same goes for the judges that audit agents for reward hacking: I plant known hacks into real agent traces to measure what those auditors miss and what they flag wrongly.

    1. 01

      Claim-level provenance

      Every reported number is a claim with a JSON pointer into a hashed source snapshot, verified before publication.

    2. 02

      Point-in-time evaluation

      Agents only see data fetched before their decision time; calls are scored only after they are recorded.

    3. 03

      Tamper-evident records

      Append-only, hash-chained call ledgers with database-level guards.

    4. 04

      Benchmark integrity

      Oracle-checking every task before an agent run, so broken environments are dropped, not scored.

    5. 05

      Agent orchestration & runtimes

      Builder/reviewer twin loops and durable runtimes with classified failures and spend guards; traced runs become labelled data for training smaller models.

    6. 06

      Auditing the auditors

      Known reward hacks planted into real agent traces give trace auditors a ground truth: what they catch, miss and falsely flag.

    Evidence

    17 checks verified · 4 in progress · tap a check for its proof

    Writing

    2 preprints · 3 in preparation · 2 published
    IN PREPARATION
    Spring 2027

    Does the harness matter? Reviewer, memory and self-check harnesses for coding agents on Terminal-Bench 2.1

    Pre-registered. Design frozen 15 October 2026 before the main data.

    Design frozen 15 Oct 2026Block 1 run 17 Oct 2026Results spring 2027
    IN PREPARATION
    Preliminary note

    Who audits the auditor? Calibrating reward-hacking judges with planted hacks

    Known hacks planted into real coding-agent traces, scored against each trace's untouched copy, measure what trace auditors miss and falsely flag, including Harbor's built-in judge. Development round done; protocol frozen and timestamped.

    PreliminaryProtocol frozen 17 Oct 2026Test round after block 1
    IN PREPARATION

    Receipts for every number: claim-level provenance and point-in-time verification for LLM trading agents

    Method and early results from GlassBox: the baseline discrepancy audit, the verification gate, and the live forward-only record.

    DraftingEarly results

    Projects

    ACTIVE · 2026 to present

    GlassBox

    An evidence-first multi-agent investment-research committee: point-in-time snapshot store, structured claims with exact source pointers, a verification gate, compliance filter and hash-chained ledger.

    550/550
    planted errors caught
    0/200
    false alarms on controls
    Planted-error resultRelease s3-part1

    Research preview only; not investment advice.

    ACTIVE · 2026 to present

    Agent harness study (R1)

    Does a reviewer, a memory window or a self-check round raise a coding agent's success rate? Four harness variants on Claude Sonnet 5.5, run through Harbor; bootstrap and permutation tests.

    57
    oracle-checked tasks
    4 × 5
    variants × trials

    Block 1: 138 of 228 trials, 0 infrastructure errors. Comparisons wait for the single confirmatory analysis.

    Oracle auditHarbor PR #3593
    ACTIVE · 2026 to present · PRELIMINARY

    Hack-audit calibration (R3)

    Who audits the auditor? Known reward hacks planted into real agent traces give trace auditors, including Harbor's built-in judge, a ground truth: what they catch, miss and falsely flag.

    2/12
    solution reads caught by an LLM judge
    12/12
    caught by a free rule check

    Development round, 101 pilot traces. Test round on the main study's passing traces. Read the preliminary note.

    ACTIVE · 2026 to present

    Twin-agent build lab

    A free model builds, a strong model plans and reviews. Every run is traced; accepted runs, rejections and correction pairs form a labelled dataset.

    EARLIER · 2025 to 2026

    Earlier agent systems

    A multi-agent trading system (private repository), the V(n) groundwork GlassBox grew out of, and a scientific discovery agent.

    Track record

    Append-only · corrections are new entries · feed
    1. 2026-11-11PREPRINTPreliminary note: who audits the auditor? Planted reward hacks give trace auditors a ground truth
    2. 2026-10-17PREPRINTWhite paper archived on Zenodo with a DOI: Superhuman Scores, Sub-Human Adaptation
    3. 2026-10-17MILESTONEGlassBox planted-error evaluation corrected to 550/550: stale-data plants re-targeted to claims not already stale
    4. 2026-10-17MILESTONEAgent harness study: replicate block 1 run, 138 of 228 trials, 0 infrastructure errors
    5. 2026-10-16MILESTONEGlassBox planted-error evaluation: the verification gate caught all 531 planted errors of the kinds it checks, with 0 false alarms on 200 controls
    6. 2026-10-16PREPRINTTechnical note: one in eight Terminal-Bench 2.0 reference solutions fail their own tests
    7. 2026-10-15MILESTONEAgent harness study: design frozen before the main data
    8. 2026-10-14OPEN SOURCEMerged into the official MCP Go SDK
    9. 2026-10-12MILESTONEGlassBox: evaluation pre-registered before outcome data, after an openly reported null study
    10. 2026-09-27PREPRINTWhite paper: Superhuman Scores, Sub-Human Adaptation
    11. 2026-09-27MILESTONEORCID iD registered: 0009-0006-4282-4800
    12. 2026-09-27RELEASEGlassBox: gate-health dashboard, stricter verification gate, compliance filter
    13. 2026-09-26MILESTONEFirst verification-gate results on live production data
    14. 2026-09-26RELEASEGlassBox s3-part1: verification foundations
    15. 2026-09-26BENCHMARKLocal coding-model benchmark on a laptop CPU
    16. 2026-09-25OPEN SOURCEGlassBox made public under the MIT license
    17. 2026-09-25MILESTONEGlassBox baseline audit: 2.3% of numbers in model prose traced to no source