Skip to content

Operations

Context

Getting access, making calls, the patterns worth copying (routing, guardrails, SRE evidence gates), limits, cost model, and the rollout procedure that keeps a cheap model from becoming expensive through mistakes.

Getting Access

Three routes (official + reported):

  1. Official early access: join the waitlist at typesafe.ai (days, not weeks, per early users). The console at console.typesafe.ai has the Playground, API keys, usage, cookbooks, and a paste-into-your-agent prompt that installs a TypeSafe skill.
  2. Gateway: Vercel AI Gateway serves Jev as typesafe-ai/jev at the same official price — usable through the AI SDK without a TypeSafe account. OpenRouter and Cloudflare routes are reported; verify current availability before depending on them.
  3. Reseller: jevtypesafeai.com sells instant hosted keys — unofficial, higher pricing ($0.25-$0.42/M). Treat as untrusted for keys and state payloads (see Security).

Commands & Recipes

Raw curl against the System One endpoint:

curl https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-latest",
    "state": "The deploy failed twice and customers are seeing 500s.",
    "questions": {
      "urgent": {
        "type": "noul",
        "instructions": "Does this need attention right now?"
      },
      "owner": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "engineering": "Product failures and outages",
          "billing": "Charges, invoices, and refunds",
          "sales": "Pricing and new accounts"
        }
      }
    }
  }'

Python (official adapter used for workflow evals and LLM comparisons):

pip install system-one-adapter

from system_one_adapter import TypeSafeClient  # per repo docs; check README for exact import

client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"])
result = client.systemOne(
    model="jev-latest",
    state=user_request,
    questions={
        "route": {"type": "choice",
                  "instructions": "Which model tier should handle this request?",
                  "criteria": {"fast": "Lookups, extraction, small edits",
                               "powerful": "Architecture, ambiguity, high stakes"}},
    },
)
model = fast_model if result.answers["route"].choice == "fast" else powerful_model

MCP server (exposes Jev as typed tools to any MCP-capable agent):

# jev-mcp — https://github.com/jkudish/jev-mcp
npx -y jev-mcp   # per repo README; configure with TYPESAFE_API_KEY

Batch classification over a CSV (product-site Batch Analyzer): paste rows, ask 1-8 questions, get a typed calibrated answer per row, then filter/sort/export — no code for one-off triage jobs.

Limits

Limit Value
Shared budget (state + all questions) ~64,000 tokens
State + longest single question ~32,000 tokens (~150K chars English)
Choice cardinality up to 255 options (2-stage scoring at the top of the range)
Score levels 2-10
Input modality text only (string / JSON / JSON array of text)
Latency 70-500ms end-to-end
Price $0.042/MTok input, output free

Patterns Worth Copying

  • Atomic questions, composed in code. Instead of "rate this startup pitch", ask market size, feasibility, and differentiation as separate score questions, then weight them in your code. Changing priorities becomes a coefficient change, not a prompt rewrite.
  • Model routing (pi-jev pattern). Grade request complexity with one choice question; send fast requests to a fast model, ambiguous ones to a frontier model. The router chooses; it never answers the request.
  • Pre-execution tool-risk gate (LangChain middleware pattern). Before an agent runs a shell command, classify read-only / reversible / destructive with separate nouls (deletes files? touches git history? touches production?). High-confidence read-only proceeds; destructive or uncertain pauses for approval.
  • SRE evidence gates (SREGym pattern). jev_plan ranks competing diagnostic hypotheses; jev_submit reviews evidence before submission with a 0.70 probability floor per required question. Rejected submissions must gather new evidence, not reword the old claim. Result: agent pass rate 40% -> 48% on SREGym-Lite.
  • Retrieval relevance. Embeddings find related text; a noul per candidate passage decides whether it actually answers the query — filter before generation.
  • Counting via decomposition. One noul per item ("is items[i] a fruit?"), sum in code — never ask Jev to count.
  • Cross-harness memory (Beacon). Capture agent traces across 19+ harnesses, use Jev to score which runs are worth learning from, extract recurring workflows as skills.

Rollout Procedure (from the DDDS walkthrough)

A cheap model becomes expensive when mistakes cause retries, reviews, or incidents — measure expected cost per decision including escalations, false approvals, false blocks, and recovery work.

  1. Pick one bounded, low-risk decision with a closed outcome set that a human can label quickly.
  2. Write the rubric before calling the model — define what belongs in every option. Overlapping criteria produce ambiguous choices that are not Jev's fault.
  3. Collect representative examples with expected answers, including ambiguous and adversarial cases.
  4. Run in shadow mode beside the current workflow without letting it change behavior.
  5. Plot accuracy against confidence; set thresholds from your data, not the marketing claims.
  6. Automate the safest branch first; keep a human or stronger model on uncertain cases.
  7. Pin or log model version, questions, criteria, and thresholds so changes replay against the same evaluation set.

Cost Model

$0.042/MTok input, output free. Reference points from TypeSafe: a Doom bot making 10 queries/second costs ~$7/hour; the headline "193.6x faster / 444.6x cheaper" comes from workflow evals against frontier models and is an upper bound. Because output is unmetered, cost scales with state size — trimming irrelevant context cuts the bill and raises accuracy at the same time.

Self-Hosting with AnyJev

When the hosted API is a non-starter (data can't leave the VPC, cost at extreme volume, availability requirements), AnyJev reproduces the state + typed-questions interface over any open LLM via logit readout with label-free debiasing (L0) and optional per-question calibration (L1, 100-500 labels):

pip install "anyjev[hf]"        # library + transformers backend (vLLM also ships)

from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend

d = Decider(HFBackend("Qwen/Qwen3-8B"))

route = Question.choice("Which handler should process this request?",
                        ["billing", "technical", "sales", "other"], name="route")
safe  = Question.noul("Is the proposed tool call destructive or irreversible?", name="safe")

r = d.decide(state, [route, safe], level="L0")
r["route"].argmax          # "billing"
r["route"].distribution    # {"billing": 0.81, ...}
r["safe"].p_true           # 0.12

Trade-offs vs the hosted API: choice costs K prefills (2 for noul, 1 for score) — on one H100 the transformers path at 20 permutations runs ~0.25s per decision at batch 32, versus Jev's 70-500ms hosted round trip; the letter readout caps choice at 26 options (Jev allows 255); you own the serving stack and the calibration data; but the weights are open, the state never leaves your infrastructure, and the L1 calibration artifact is frozen and auditable. AnyJev's own bench (python -m bench.run) is the fastest way to see whether L0/L1 helps on your task before committing.

Troubleshooting

  • Wrong-but-confident answers — the schema guarantee does not cover judgment. Check for overlapping criteria, ambiguous rubrics, or a question that needs decomposition.
  • Accuracy drops after adding context — isolation means every question sees the whole state; irrelevant content dilutes. Prune state per decision.
  • Literal misreadings — Jev reads words, not intent ("jaggedness" page per model version documents known ones). Make instructions explicit or split interpretation into two questions.
  • Slower than advertised on big choice sets — high-cardinality choices (>~dozens) use a 2-stage internal scoring system.
  • Calibration drift after model bump — re-run the shadow evaluation; do not carry thresholds across model versions blindly.

Sources