Reinforcement learning environments, built on the rules
An agent learns what its environment rewards. We build environments where the statute, the permissions and the people on the other side of each decision are part of the task, written by members in computation and in law.
The environments we build.
Five kinds of environment, each built to teach one thing an agent has to get right before it is trusted to work alone.
Tool-use sandboxes
The agent is given a shell, a browser and files, and a job to finish. The reward pays for finished work that never stepped outside what it was allowed to touch.
Long-horizon work
Tasks that take many steps across several applications, with the result checked at the end and the route checked along the way, so a correct answer reached by a forbidden shortcut earns nothing.
Oversight and escalation
Tasks built so that the right move, at some point, is to hand the decision back to a person. The reward pays for asking at that point, and not before or after.
Adversarial pressure
Injected instructions, conflicting orders and counterparties who push. The agent is rewarded for finishing the task its principal set, and for refusing the one someone else slipped in.
Honest reporting
After each episode the agent reports what it did and what it could not do. The report is compared against the log, and an accurate account of a failure is worth more than a flattering one.
Built from computation and law
Every environment combines members who build software with members who write rules, and is checked by people who did neither.
Computation
We build the world an agent works in, and record everything it does there.
Law
We write the rules it is held to, and the rubrics that check it kept them.
Evaluation
We test the trained agent in environments it has not seen, and publish how it did.
How an environment is made.
Four steps, each written by the members who know that part of the work best.
01 SpecifyWrite the rules down.
Every environment starts from the rules an agent will be held to: the statute, the regulation, the permissions of the system it runs in. Members in law write them as tests a program can check.
- A rulebook, in plain language and in code
- The cases where the rules conflict
- The people on the other side of each decision
A rule, written as a check
- The rule in plain words: the agent may read the customer file, and may not copy it anywhere else.
- The check a program runs: every write, upload and message in the episode is compared against the file's contents.
- The hard case, named in advance: a task that cannot be finished without quoting one line from the file.
- What the agent should do then: stop, say why, and ask the person it is working for.
02 BuildBuild the world the agent works in.
Members in computation build the environment: the tools, files, accounts and counterparties an agent meets on the job, in a sandbox where every action is recorded and nothing leaves.
- A sandbox with real tools and real formats
- Tasks drawn from the work, not invented for the test
- A full record of every action taken
What is inside a sandbox
- A shell, a browser and a file system, each with its own permissions.
- Accounts and credentials that are real inside the sandbox and worthless outside it.
- Simulated counterparties who ask, refuse, mislead and change their minds.
- A complete log of every action, so each reward can be traced to what earned it.
- A reset, so the same task can be run thousands of times from the same start.
03 RewardPay for the right behaviour.
The reward is where a guardrail becomes something the agent learns. Ours pay for finishing the task within the rules, and pay nothing for a result reached by breaking one.
- Graders checked against expert judgement
- Rubrics with the reasoning shown
- Penalties for the shortcuts that matter
What the reward pays for
- Finishing the task inside the rules that apply to it.
- Citing the rule each decision relied on, so a person can check it.
- Stopping to ask a person when two rules conflict.
- Nothing for a result reached by breaking a rule, however good the result.
04 EvaluateCheck, publish and brief.
We evaluate agents in the same environments before and after deployment, publish the results with the evidence behind them, and brief the legislators who ask what a rule would cost.
- Scores against each rule, task by task
- Published methods and results
- Briefings for legislators and officials
The evidence: how long agents work alone
What you receive.
An environment is handed over as something a training run can use the same day.
01The environment
The tasks, tools and counterparties, packaged behind a standard step-and-reset interface that a training stack can call.
02The rules and the reward
Every rule in plain words and in code, and the reward that enforces it, with the reasoning written out.
03Held-out evaluations
A separate set of tasks the agent never trains on, for measuring what it learned rather than what it memorised.
04A written report
How agents behaved, where they broke a rule and why, with the episodes to show it.
Build an environment with us
Tell us the agent, the work it will do and the rules it will be held to.
