Skip to content

Security

Context

Jev inverts the usual AI security conversation: schema safety is absolute, judgment safety is not. The real security surface is (1) the gap between "cannot hallucinate" and "can be wrong", (2) calibration drift in automated decisions, (3) a hosted-only closed-weights supply chain with an active reseller ecosystem, and (4) sending your state — which is often your users' data — to a third-party API.

Output Integrity: Schema Safety ≠ Judgment Safety

  • What is guaranteed: the response cannot violate the declared output schema. There is no representable "malformed value" outcome — TypeSafe calls the 0% type-error rate structural ("mathematically impossible" to falsify), not an empirical measurement. Downstream code will never see an undeclared option, a string where a number was declared, or a hallucinated key.
  • What is not guaranteed: that the winning option is correct. Jev can select a wrong valid option with high confidence — the DDDS walkthrough's billing/technical example won by 6 points with 0.18 confidence, which is exactly the case the confidence field exists to catch.
  • Engineering consequence: every automated branch needs an abstention policy scaled to consequence. Small-consequence + high-confidence -> act; middling -> confirm or escalate to a stronger model; low -> human. Thresholds live in code, are versioned like code, and must be derived from your own accuracy-vs-confidence plots, not from marketing numbers.
  • Calibration drift is the silent failure mode. RLCD training aims to make confidence track accuracy, but that relationship is empirical and per-distribution: it can break when the model version changes, when questions/criteria change, or when input traffic drifts. Re-measure after any of those; SREGym's failure case (a repair accepted as complete without testing the durability invariant) is the canonical example of confident-but-wrong at the boundary.

Supply Chain

Jev is closed-weights and hosted-only — you are trusting TypeSafe's service, plus whatever route you take to it.

Artifact Risk posture Control
api.typesafe.ai (official) Primary trust anchor; early-access service, pricing possibly subsidized (their own disclosure) Pin model version IDs (jev-1.13.0, not jev-latest) for replayable behavior; log the versioned ID in every response
Gateway routes (Vercel typesafe-ai/jev; OpenRouter/Cloudflare reported) Adds the gateway as a second trust anchor with its own logging/retention posture Verify the gateway's data-retention terms independently; confirm route authenticity before sending sensitive state
jevtypesafeai.com (UNOFFICIAL) Community demo + reseller selling "instant hosted keys" at 6-10x official pricing; name mimics the vendor Do not send sensitive state or keys through it; it is not TypeSafe. Official domain is typesafe.ai
Ecosystem packages (jev-mcp, awesome-jev lists, pi-jev routers) Third-party code at varying maturity Review before wiring into agent loops; do not assume a clone inherits Jev's calibration
AnyJev (Nokia, Apache-2.0, self-hosted) You own the stack: model weights, calibration labels, serving infra; L1 artifacts are frozen and auditable Standard model-supply-chain hygiene on the underlying LLM; re-run calibration checks on every model or distribution change
  • No official offline story: no self-hosted or weights-export path exists from TypeSafe; availability, rate limits, and deprecation policy are the vendor's to change. Keep the interface shim thin (one client module). The credible escape hatch is AnyJev (Nokia, Apache-2.0): the same interface over your own open LLM, state never leaving your VPC — at the cost of K prefills per choice, a 26-option cap, and owning the calibration data yourself. AnyJev's L1 artifacts freeze the prior they were fit with, which matters for auditability; its L1 calibration does not survive distribution shift, the same drift discipline as above applies.
  • No known incidents or CVEs as of 2026-09-23 — the product is weeks old. This reflects age, not audit depth. TBD — re-check before production reliance.

Privacy and Data Flow

  • State is your data, exfiltrated by design. Every call sends the full state to TypeSafe's hosted service — for triage/moderation use cases that means user content, tickets, resumes, or logs. Classify what flows through: the ~64K-token budget is generous enough to over-share by accident. Send the minimum state per decision (which also improves accuracy).
  • Retention posture: not independently documented as of 2026-09-23 — TBD. Before wiring regulated data (HR screening, support tickets with PII), get written retention/processing terms from TypeSafe; the ZDR configurations in Zero Data Retention are the pattern to demand.
  • Questions and criteria are configuration, but sensitive ones: your choice criteria encode business taxonomy and policy. They transit the same channel; treat rubric leakage as a (minor) business-logic exposure.
  • Downstream flow: Jev outputs feed code branches — an automated rejection or escalation is a decision about a person (resume scoring, moderation). Keep humans on the low-confidence path and log the probability with the decision for auditability.

Enforcement Boundary: Scores Inform, Never Enforce

Jev is well suited to screening for prompt injection, policy violations, and risky tool calls — and unsuitable as the enforcement mechanism. Its score should gate nothing by itself: permissions, sandboxes, allowlists, and tests must enforce exact rules. The correct composition is Jev as a fast semantic pre-filter in front of hard enforcement (high-confidence read-only -> proceed to the permission check; anything destructive or uncertain -> pause), so a miscalibrated or adversarially steered score can only change latency and friction, not authority. Remember the state itself is attacker-reachable text: a user who can influence state can attempt to steer the judgment — the SREGym and LangChain patterns mitigate this with evidence requirements and middleware confirmation gates, not with trust in the score.

Threat Model Summary

Threat Vector Impact Likelihood Control
Confident wrong decision Ambiguous rubric / distribution drift Bad automated outcomes (misc routing, wrong approval) Medium Abstention thresholds from your own calibration plots; shadow mode first
Calibration drift Model version bump, question edits, traffic shift Thresholds silently stop meaning what they meant Medium Pin version IDs; re-run shadow eval on every change
Reseller/key theft jevtypesafeai.com and similar lookalike sites Key or state leakage to unknown third party Low-Medium Official domain only (typesafe.ai); treat resellers as untrusted
State over-collection Generous 64K budget invites dumping full context PII/regulatory exposure to hosted API Medium Minimal-state per decision; written retention terms for regulated data
Score steering Attacker-controlled text in state Semantic filter says "safe" for malicious input Medium Jev pre-filters, hard enforcement decides (permissions, allowlists, sandboxes)
Vendor lock-in / availability Closed weights, hosted only, early access Service or pricing change breaks automated branches Medium (early) Thin interface shim; versioned questions/criteria for replay; LLM fallback path
Overlapping criteria Bad rubric design Ambiguous categories misrouted as model error High (design-time) Rubric review before calls; criteria are program logic — test them

Agent-Loop Placement Guidance

Where Jev sits in an agent loop changes its risk profile. Three placements, ordered by increasing caution:

  1. Advisory (lowest risk): Jev ranks, sorts, or labels — a human or the LLM consumes the output as context (SREGym's jev_plan test-ranking; Beacon's trace scoring). A wrong answer wastes attention, not authority.
  2. Gated decision (medium): Jev's probability is one input to a branch that also checks hard conditions (confidence floor + evidence requirements + confirmation prompts). SREGym's jev_submit is the reference design: every required question must clear 0.70, and rejection forces new evidence rather than rewording.
  3. Autonomous enforcement (highest risk): Jev's output alone triggers consequential actions. This is the placement to avoid — combine with hard enforcement (permissions, allowlists, sandboxes) so a miscalibrated or steered score can add friction but never authority.

Additional rules of thumb for agent builders: give Jev the smallest state slice per decision (accuracy and privacy improve together); keep separate Jev questions per risk dimension instead of one composite judgment (deletion vs git-history vs production reachability); and log every gate outcome with its probability so incidents can be replayed against the versioned model that made them.

Monitoring and Audit Hooks

  • Log the versioned model ID from every response (the response reports which version answered) alongside the probabilities and the branch taken — this is what makes decisions replayable and audits possible.
  • Accuracy-vs-confidence is the health metric. Plot it per decision type on live traffic; a widening gap between predicted confidence and observed accuracy is the earliest signal of drift.
  • Re-run the shadow evaluation on every change — model version, question wording, criteria, thresholds, or input mix — before promoting thresholds.
  • Track cost per decision end-to-end (calls + escalations + false approvals/blocks + recovery), not just token spend.

Sources