Introducing Physical-Engineering’s Last Exam for agents
---
Today, the Google AI & Infrastructure team is announcing ????????-???????????'? ???? ???? (???), a new benchmark and open competition, developed in collaboration with Mercor, and built to test a frontier model’s ability to complete tasks in mechanical, electrical, civil and structural engineering.
To help the research community measure and improve how agents handle physical engineering work, we are releasing 29 public tasks, their grading tools, and an open competition leaderboard at ple-bench.com.
Frontier labs and the broader research community use model examinations to test AI’s progress in completing tasks in a specific discipline. Humanity's Last Exam measures a model’s general knowledge. Agents' Last Exam measures a model’s ability to complete digital professional work. ????????-???????????'? ???? ???? measures a model’s ability to complete work that helps build the physical world. Physical engineers design skyscrapers, power grids, and computer chips we rely on, yet we could find no existing benchmark that tests whether frontier models can do that work end to end.
Why textbook benchmarks fall short for physical engineering
Frontier language models now score above 90% on many academic benchmarks. However, those benchmarks mostly test textbook-style questions, single tool calls, or small, self-contained design problems. For AI developers and researchers building agents for physical engineering, existing benchmarks do not show whether an agent can produce safe, code-compliant, and cost-effective designs under a licensed engineer's review and control. As the demand for infrastructure development outpaces financing, there is a growing opportunity for agents to support engineers working to close this gap.
Inside PLE: Real-world tasks built with practicing engineers
PLE closes this evaluation gap with 85 long-horizon tasks across four core disciplines: mechanical (30 tasks), electrical (27 tasks), civil (18 tasks), and structural (10 tasks) engineering.
Instead of answering a single prompt, an agent works inside a containerized project folder — built on Mercor’s Archipelago environment — that mirrors what a practicing engineer receives on the job. Each task holds an average of 61 files (5,170 total files across the benchmark), including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models, and document revisions. The agent must reason through these files, find the governing building codes and physical limits, run calculations, and generate production-ready deliverables that a licensed reviewer would review and approve.
In collaboration with experts from Mercor and licensed practicing engineers from industry, we built PLE using two methods:
*
Expert-authored tasks (44 tasks): Practicing engineers created the project folders, prompts, and grading rubrics based on real professional engineering tasks.
*
Co-authored tasks (41 tasks): Practicing engineers designed the initial task and an agent pipeline assembled the task's environment and grader from licensable engineering standards. Engineers then validated and increased the difficulty of every task assembled by the agent.
All frontier models score below 50%
We tested 14 frontier models on PLE using seven publicly available agent harnesses (Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands, and Pi). Because PLE evaluates both code compliance and cost efficiency, agents only earn points if their design passes every hard safety check first. Our baseline results show a wide gap between current models and real-world engineering requirements:
*
High headroom: All frontier models score below 50% on our private test set, with the highest-performing model on its native harness reaching a mean score of 48.3%.
*
Hard safety gates: While the top-scoring agent configuration satisfied 91% of individual constraints on co-authored tasks, only 41.3% of its deliverables passed every hard safety and feasibility check required.
*
Model vs. harness impact: Choice of model drives performance far more than the harness (up to a 50-point spread across models, versus a median 3-point spread across harnesses for the same model), though harnesses significantly affect run cost.
Test your agent and join the open competition
Evaluating AI agents on realistic project files, strict safety rules, and material costs is a practical step toward building helpful tools for physical engineers. Here is how you can get started with PLE today:
*
Check out the full technical breakdown in our PLE paper on arXiv.
*
Download the 29 public tasks, grading scripts, and evaluation container from our GitHub repository.
*
Bring your own agent setup and submit it for evaluation on our private test set at ple-bench.com—with 13 Google-sponsored models running free on our servers and prize details coming soon. 🔗 Google IA
https://cloud.google.com/blog/products/ai-machine-learning/introducing-physical-engineerings-last-exam-for-agents/?utm_source=dlvr.it&utm_medium=blogger
