Chollet on X: What the ARC Semi-Private Eval Means

Explanation of why ARC uses a semi-private evaluation set — tasks not publicly available at test time — to prevent overfitting to the test distribution. Chollet explains that any publicly released benchmark becomes a training target: researchers fine-tune models on it, which inflates scores without reflecting genuine capability improvement. The semi-private eval is designed to measure actual generalization by ensuring models cannot be pre-trained on the test tasks. This design choice is philosophically important: it prioritizes measurement integrity over convenience and reflects Chollet's overall position that benchmark design is the crux of AI progress measurement.