Tool Suggestions
Google PLE: An 85-Task Benchmark That Tests AI Agents on Real Engineering Work
Google's AI and Infrastructure team, with Mercor, released Physical-Engineering's Last Exam, or PLE, on November 3: a benchmark of 85 long engineering tasks where an AI agent must produce a real deliverable, not answer a textbook question. Twenty-nine tasks are public, with grading tools and an evaluation container, alongside an open agent competition and leaderboard at ple-bench.com. The headline result is humbling: every tested model scored below 50% on the private test set.
The tasks span four disciplines: 30 mechanical, 27 electrical, 18 civil and 10 structural. A mechanical task might ask for a cooling loop design and a clash-free three-dimensional equipment hall; an electrical task, sizing a switchboard and producing its cable schedule; a civil task, laying out a campus with grading and utilities; a structural task, designing a roof deck. Each assignment runs in a containerized project environment built on Mercor's Archipelago framework, mirroring what a practising engineer receives on the job: an average of 61 files per task, 5,170 files across the benchmark, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and superseded document revisions. Agents get no internet access and up to six hours.
What counts as passing
The grading is the point. Success is measured through deliverables that must satisfy engineering codes, physical constraints and, for design work, cost or performance objectives. Points for cost efficiency are awarded only after a design clears every hard safety and feasibility check, which is the rule that separates this benchmark from most agent evaluations. Partial credit is real but capped: the strongest configuration satisfied 91% of individual constraints on average, yet only 41.3% of its deliverable items passed every hard safety and feasibility check required to earn objective credit. In engineering, a design that meets most constraints but fails one safety check is not a mostly-good design. It is a design you cannot build.
That gap between "mostly right" and "safe to build" is exactly what PLE is designed to expose. Most benchmarks reward the average; this one punishes the single missed constraint, because that is how physical engineering works. An agent that sizes 91% of a switchboard correctly and gets the fault rating wrong has not done 91% of the job.
The results so far
Google tested 14 frontier models across seven publicly available agent harnesses: Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands and Pi, running each task three times per configuration. On the 56-task private test set, the best native-harness configuration was Claude Opus 5.5 in Claude Code at 48.3%, followed by GPT-6 Astra at 45.8% and Gemini 4 Argon at 40.6%. The strongest configuration averaged 27 million tokens and 63 minutes per task, costing $17.18 per task at list prices.
One numerical discrepancy deserves a plain note. The research paper reports a high of 50.6% for Claude Opus 5.5 in Claude Code across all 85 tasks, while the November 3 blog reports 48.3% on the hidden test set. Neither source explains the difference, so the two figures should not be treated as interchangeable: 48.3% is the private-set number, 50.6% is the all-tasks paper number. Also note the timeline honestly: the paper was first submitted on October 21 and revised on October 30. November 3 is the public release of the tasks, grading tools and competition, not the paper's first publication.
How the tasks were built
Forty-four tasks were expert-authored end to end: practising engineers in Mercor's network created the project folders, prompts and grading rubrics based on real professional engineering tasks, and licensed engineers wrote and validated each task. The other 41 were co-authored: practising engineers designed the initial task, and an agent pipeline assembled the task's environment and grader from licensable engineering standards, in an adversarial loop where a Creator agent constructs a task and a Solver agent tries to break it.
For builders, the practical value is immediate. The 29 public tasks, grading scripts and evaluation container are downloadable, the competition runs against the hidden test set, thirteen Google-sponsored models can be run for free on the competition servers, and prize details are coming soon. If you build agents for technical work, this is the first benchmark that grades them the way a licensed engineer would: not on whether the answer sounds right, but on whether the thing can be built safely.