Chollet on X: Why LLMs Cannot Solve ARC Through Scale Alone
AI Summary
Thread explaining why the ARC benchmark is specifically designed to resist the type of performance improvement that scaling data and parameters provides. Chollet argues that LLMs improve at tasks similar to their training distribution, but ARC tasks are explicitly designed to be out-of-distribution — each puzzle is novel by construction. Training on more ARC tasks would help on those specific tasks but would not improve performance on new novel tasks (which is the whole point). This is a clear illustration of why "more data + bigger model" cannot solve ARC: ARC is measuring something that is not improved by those interventions.
Original excerpt
Core argument: ARC is out-of-distribution by design. Scaling improves performance on near-distribution tasks. ARC specifically excludes near-distribution tasks. Therefore scaling cannot solve ARC.
This is the clearest statement of why ARC serves as a genuine "AGI detector" rather than another solvable benchmark.
Frequently asked questions
What is "Chollet on X: Why LLMs Cannot Solve ARC Through Scale Alone" about?
Thread explaining why the ARC benchmark is specifically designed to resist the type of performance improvement that scaling data and parameters provides. Chollet argues that LLMs improve at tasks similar to their training distribution, but ARC tasks are explicitly designed to be out-of-distribution…
Who wrote "Chollet on X: Why LLMs Cannot Solve ARC Through Scale Alone"?
"Chollet on X: Why LLMs Cannot Solve ARC Through Scale Alone" was written by François Chollet. It is curated in the François Chollet vault on Burn 451, which covers agi evaluation & arc-agi.
How can I read more content from François Chollet?
The complete François Chollet reading list is available at burn451.cloud/vault/francois-chollet. Each article includes an AI-generated summary so you can decide what to read in seconds. Connect the Burn 451 MCP server to Claude or Cursor to query all François Chollet articles as live AI context.
Can I use "Chollet on X: Why LLMs Cannot Solve ARC Through Scale Alone" with Claude or Cursor?
Yes. Install the burn-mcp-server npm package and connect it to Claude Desktop, Claude Code, or Cursor. Once connected, your AI can search and reference this article and the full François Chollet vault in real time — no manual copy-paste required.
28 more articles in this vault.
Import the full François Chollet vault to Burn 451 and build your own knowledge base.
Content attributed to the original author (François Chollet). Burn 451 curates publicly available writing as a reading index. For removal requests, contact @hawking520.