TypeSafe Jev vs. Pepper X
We benchmarked TypeSafe AI's System One judgment model (Jev, using the default jev-latestmodel, which resolved to 1.13.0 during the tournament run) out of the box against Tongits.io's reigning Reinforcement Learning champion (Pepper X, run 6 iteration 712). Here is how a general judgment primitive compares against Pepper X, selected at iteration 712 from a 750-iteration, 3 million-hand PPO self-play run.
Tournament Scorecard
100 Seed Pairs (200 Hands)fee units/hand (+EV against league opponents)
vs 22.0% for Pepper X (+3.0% lead)
3x fewer burns than Pepper X (3.0%)
36.2% forced turns resolved locally in benchmark actor
Seed-Paired Telemetry Scorecard
Identical card deals evaluated under standard league rules and betting multipliers.
| Performance Dimension | TypeSafe Jev | Pepper X (RL) | Net Delta |
|---|---|---|---|
Hand Win Rate Percentage of deals won outright via Tongits, accepted Fight wins, or lowest deadwood at draw deck exhaustion. | 25.0% | 22.0% | +3.0% absolute |
Sunog / Burn Rate Near-zero instances of being trapped with zero melds during opponent Fight calls, avoiding sunog dues penalties. | 1.0% | 3.0% | -2.0% (3x Fewer Burns) |
Average Deadwood Lower terminal point count remaining in unmelded cards, protecting against late deck showdowns. | 20.3 points | 25.5 points | -5.2 points (Tighter Hand) |
Net Money Margin Expected profit per deal. Both models are profitable (+EV), with Pepper X extracting higher margin from aggressive Fight challenges (+0.660 vs +0.260 fee units/hand, -1.01σ). | +0.260 fee units / hand | +0.660 fee units / hand | -0.400 fee units |
Fight Challenge Rate Jev strategically folds when uncompetitive to evade showdown and burn dues, maintaining a 1.0% burn rate. | 6.7% accept | 23.1% accept | Disciplined Fold Defense |
Decision Latency Cloud API round-trip time vs compiled on-device neural inference graph. Well under the 30-second human turn clock. | 224 ms (mean) | 8 ms (local ONNX) | +216 ms |
Actor Fast-Path Bypass Turns with only 1 legal action (such as forced deck draws when Chow is illegal) are resolved instantly in benchmark actor code, saving network calls. | 36.2% of turns | N/A (Local Model) | Zero API Overhead |
Money and playstyle metrics are emitted by the evaluator script; latency and forced-move bypass figures are recorded from the Jev actor during the same 100-pair tournament run.
Tournament Methodology and Setup
How the 3-player tables were structured to ensure rigorous, reduced-variance statistical comparisons between TypeSafe Jev and Pepper X by controlling for starting-deal luck.
3-Player Duplicate Tournament Format
Identical cards played twice to control for starting-deal luck.
In a 3-player card game, pitting two models at the same table introduces deal luck: whoever happens to receive stronger cards in a single deal wins. Instead, each deal seed is played twice. In Match 1, Jev occupies the test seat. In Match 2, Pepper X takes the exact same seat with the exact same starting cards and deck order. This controls for starting-deal luck while allowing subsequent game trees to branch naturally based on model choices.
Identical League Opponents in Both Games
The other two seats at the table are occupied by matching AI league personas.
For every deal seed, the remaining two seats are filled by league opponents sampled deterministically from the 12 Tongits personalities (such as Mang Tonyo, Rosa, Pedro, or Maria) and the default heuristic profile. Jev and Pepper X face the exact same opponents in identical seats across both duplicate matches.
Deterministic Fast-Path Routing (36.2% Bypassed)
Turns with only one legal move are executed instantly without cloud calls.
When a player is in the Draw phase and cannot chow the discard, drawing from the deck is the only legal action in Tongits rules. Jev never skips playing a turn; rather, the benchmark adapter recognizes that only 1 legal move exists, bypasses the cloud API, and executes the move immediately in local actor code (0ms). Jev is queried only when genuine strategic choices exist.
Live Betting and Penalty Stakes
Evaluated under real live multipliers, carryover pots, and sunog penalty dues.
Hands are scored with full game-server financial rules: base pot awards, streak multipliers, and penalty dues for sunog (burned) players. This measures whether a model actually generates net positive money margin rather than just winning hands.
Executive Findings: Tactical Profile vs. Economic Margin
In Tongits, winning money is distinct from merely winning hands. High payouts stem from avoiding costly burns (sunog), seizing favorable Fight opportunities, and holding low deadwood points when the draw deck runs dry.
Pitted against Pepper X across seed-paired duplicate deals with identical 12-persona league opponents and rotating dealer positions, TypeSafe Jev demonstrated a distinct tactical profile: it outperformed Pepper X on proxy metrics (hand win rate, deadwood control, and burn avoidance), while Pepper X retained the higher net economic margin (+0.660 vs +0.260 fee units per hand, -1.01σ) through aggressive Fight challenges. The benchmark demonstrates a compelling zero-shot tactical baseline rather than an overall champion replacement:
- Higher Hand Win Rate: Won 25.0% of tournament hands outright compared to 22.0% for Pepper X (+3.0% lead).
- 3x Fewer Sunog Penalties: Sustained only a 1.0% burn rate (versus 3.0% for Pepper X), avoiding sunog in 99% of rounds through disciplined fold defense.
- Tighter Deadwood Control: Maintained an average deadwood of 20.3 points compared to 25.5 points for Pepper X.
- Real-Time Fluidity: Averaged 224ms per decision via structured API judgments, with 36.2% of turns resolved instantly via local benchmark actor code.
How It Works: System One Decision Engine
TypeSafe Jev represents a modern paradigm in game AI: composable, typed semantic judgment primitives integrated directly into high-throughput card logic.
System One Judgment Primitive
Fast, typed semantic choices instead of verbose text generation.
Unlike classical LLMs that generate free-form text or require multi-turn chain-of-thought parsing, TypeSafe Jev evaluates structured board state and returns typed choices. The adapter validates the returned choice against the legal-action map and uses a deterministic legal fallback for unknown or failed responses.
Zero-Shot Domain Generalization
No pre-training on Tongits datasets; driven purely by semantic understanding of rules.
While Pepper X came from a 750-iteration PPO self-play run spanning 3 million simulated hands, with checkpoint 712 selected as the champion, Jev exhibited strong competitive fundamentals and hand-level discipline out of the box solely through structured state representation and rule criteria.
Deterministic Fast-Path Routing
Skipping model calls when only one legal action exists cuts latency and API cost by over 35%.
In card games like Tongits, players frequently face turns with only a single legal move (e.g. forced draw when Chow is illegal, or single-card final plays). By evaluating legality masks in the engine adapter before invoking the cloud model, redundant network round trips are eliminated.
Compact Partial-Information State Encoding
Compact JSON representation capturing visible melds, top discard card, card counts, and pot carryovers.
The adapter translates complex engine states into clear contextual attributes: card deadwood values, opponent meld visibility, top discard card, and streak pot multipliers. This provides the model with compact partial-information state without exposing hidden opponent hands.
// 1. Instant bypass for forced single-action turns (saves 36% of calls)
if (ctx.legal.length === 1) {
return ctx.legal[0];
}
// 2. Structured imperfect-information state encoding
const summary = {
gamePhase: state.turnPhase,
roundTurn: state.turnCount ?? 1,
deckRemaining: state.deck.length,
topDiscardCard: topDiscard ? formatCard(topDiscard) : null,
mySeat: {
deadwoodScore: ctx.deadwood,
handCards: me?.hand.map((c) => ({
card: formatCard(c),
deadwoodValue: cardValue(c),
})) ?? [],
exposedMelds: me?.exposedMelds?.map((m) => m.cards.map(formatCard)) ?? [],
hasCalledSecretMeld: (me?.secretMelds?.length ?? 0) > 0,
},
opponents: opponents.map((op) => ({
seatId: op.id,
cardCount: op.hand.length,
exposedMelds: op.exposedMelds?.map((m) => m.cards.map(formatCard)) ?? [],
hasExposedMelds: (op.exposedMelds?.length ?? 0) > 0,
})),
matchPoolPotUnits: carry?.poolUnits ?? 0,
streakHolder: carry?.streakPlayerId ?? null,
};
// 3. Formulate candidate legal actions as a TypeSafe Choice question
const { criteria, keyToIdx } = buildActionCriteria(state, seatId, ctx);
const response = await client.systemOne({
state: summary,
questions: {
action: choice(
'Evaluate the current board state and pick the best strategic action...',
criteria
),
},
});
const chosenKey = response?.answers?.action?.choice;
if (chosenKey && keyToIdx.has(chosenKey)) {
return keyToIdx.get(chosenKey);
}
return ctx.legal[0]; // Deterministic fallbackArchitectural Profile: Foundation Judgments vs. Reinforcement Learning
Comparing a zero-shot System One model against a 712-iteration PPO reinforcement learning champion across win rate, deadwood management, burn defense, and net economic return.
Semantic Grounding vs Blind Float Vectors
Processes a raw 349-float array with no prior knowledge of numbers, suits, or meld math. Must learn basic card arithmetic from scratch via millions of simulated self-play transitions.
Evaluates structured JSON state with built-in semantic understanding of card values, deadwood totals, and game phases inherited from pretraining.
Asymmetric Risk Calibration and Loss Aversion
RL policies frequently succumb to reward traps: winning marginal challenges can reinforce aggressive play, blinding the policy to catastrophic sunog penalty dues.
Inherently understands minimax loss aversion. When deadwood is uncompetitive, folds to protect against showdown losses and burn penalties, achieving a 1.0% burn rate (3x fewer burns).
Conceptual Generalization in Huge State Spaces
In a massive state space with billions of possible deal combinations, unseen board configurations can trigger out-of-distribution policy noise or suboptimal decisions.
Smoothly interpolates across novel deals using conceptual abstractions like late-game urgency, flank threat, and hand connectivity rather than fixed state-action tables.
Structured API Judgments Without Generative Hallucination
Fast (8ms on-device ONNX) but rigid and brittle to rule variations or multi-model upgrades.
Avoids the wordiness, hallucinated cards, and multi-second delays of conversational LLMs. Evaluates calibrated choice probabilities in a single structured API judgment (~220ms API latency).
Tactical Playstyle Profile and Trace Hypotheses
Qualitative trace observations and measured aggregate telemetry examining how Jev differs from conventional neural models.
Qualitative trace review suggests Jev often prefers drawing from the deck over exposing early melds with unproven combinations.
Sampled traces indicate Jev frequently holds completed melds in hand rather than exposing them on turn 1.
Jev accepted only 6.7% of Fight challenges (versus 23.1% for Pepper X), choosing to surrender 1 unit when deadwood is uncompetitive.
Sampled traces suggest Jev often sheds costly deadwood while preserving more connected cards for potential melds; the benchmark directly measures the resulting terminal deadwood, not each discard rationale.
Coming Soon: Special "Jev Boss" Bots in the AI Arena
Following this benchmark confirmation, we will be introducing special limited-edition Jev Boss bots to the Tongits.io AI Opponents roster. Players will be able to challenge Jev directly in high-stakes tables with dynamic reasoning dialogue, unique persona badges, and coin bounty rewards.