{question}\nA. {A}\nB. {B}\nC. {C}\nD. {D}\nE. {E} and read by a 5-way sequence
classifier whose classes are the option letters — exactly the format used at training time.
Everything runs in your browser; no server, no queue, nothing sent anywhere.
| Model | Params | Role | Runs as |
|---|---|---|---|
| Scratch-base transformer written from scratch in pure PyTorch — no transformers, no pretrained weights |
651,909 | comparison | hand-written JavaScript port, 2.6 MB float32 weights |
| DeBERTa-v3-small (fine-tuned) | 141,898,757 | ensemble, weight 0.70 | ONNX int8 via Transformers.js |
| RoBERTa-base (fine-tuned) | 124,649,477 | ensemble, weight 0.30 | ONNX int8 via Transformers.js |
Fine-tuning recipe (both readers): all 2,000 labelled rows, 3 epochs, AdamW
lr 2e-5, weight decay 0.01, batch size 8, max_len 160 fixed padding,
float32, linear schedule with 50 warmup steps of 750, non-finite-gradient guard.
Final train loss 0.0028 (DeBERTa) and 0.0022 (RoBERTa).
From-scratch architecture: token embedding (3,016-token word-level vocabulary built
from the training corpus) + fixed sinusoidal positional encoding → 2 ×
pre-LN block {multi-head self-attention with 4 heads, hand-written scaled dot-product;
position-wise FFN Linear→GELU→Linear} → masked mean-pool over non-PAD tokens
→ 5-way linear head. d_model=128, trained 12 epochs with
OneCycleLR(max_lr=3e-4, pct_start=0.1).
Its JavaScript port here reproduces the PyTorch logits to a maximum absolute difference of
9.5e-7 over 200 real questions, with 200/200 identical top-1 predictions.
bert-base-uncased fine-tuned on the same split reached only 0.6233.
That inversion is the finding, not a win: the benchmark rewards memorisation capacity.GroupKFold grouped on the option set. Roughly 59 accuracy points of any
"near-perfect validation" number is leakage.