Eleven tasks from contract work

OpenAI and contract-management platform Ironclad assembled 11 research tasks drawn from legal, commercial and procurement work. The agent must do more than click through software: it has to turn company requirements into forms, templates and approval paths, then verify that the finished process behaves correctly across different situations. The team scored each task against 8 to 50 criteria and used synthetic materials derived from public contracts in SEC EDGAR, not private customer agreements.

On this evaluation, GPT-6 Astra at Max reasoning averaged 55.0%, versus 41.6% for GPT-5.6 Sol at High reasoning. Estimated times were 19.2 and 37.0 minutes per attempt respectively. These are simulated estimates for 11 tasks, not measured customer time savings.

Evaluating a complete workflow

The tasks assess complete workflows, including exceptions and control rules, against predefined criteria. OpenAI describes the work as a research prototype and emphasises specialists’ role in complex contract tasks.

Sources