Factual Lead
A new testing walkthrough designed to evaluate contract AI against changing terms and incomplete evidence provides a structured approach using 12 synthetic contract families and 24 specific questions. Conducted as a preliminary browser pilot on September 20, 2026, the evaluation tested general-purpose applications including ChatGPT and Gemini using 36 source documents and invented contract rules. According to the research details, the walkthrough aims to assess how well artificial intelligence handles document versions, amendments, and missing signatures without inventing citations.
What Changed
Testing methodologies for legal and contractual artificial intelligence have evolved to address complex scenarios involving temporal evolution, multi-document amendment histories, and missing evidentiary proof. Specifically, a teaching pack containing 36 source documents and 24 questions was established using invented contract rules across 12 synthetic contract families.
On September 20, 2026, a preliminary browser pilot was executed to test general-purpose ChatGPT and Gemini browser applications. Each of the 24 questions was submitted once to both applications, yielding a total of 48 completed responses. During the pilot, ChatGPT operated under High effort settings with an initial GPT-5.6 Sol selection that reverted to Latest in fresh chats. Gemini was evaluated while displaying Pro, with version 3.1 Pro visible in its mode menu.
Context
Commercial legal documents rarely exist as single, static files. They are constantly updated through amendments, superseding clauses, schedules, and exhibits that change obligations over time. Useful contract AI must reliably connect a specific clause to the correct service, date, and document status. Systems that uniformly default to always choosing the latest document or always selecting a signed amendment fail when confronted with conditional questions that depend on historical timelines or superseded provisions.
Broader academic and industry research highlights similar challenges in document retrieval and temporal logic. For example, FiscalQA Pro studies the retrieval of date-applicable versions within French tax law. Simultaneously, TIDE examines temporally evolving documents and amendment timelines embedded within customs instruments. The newly released walkthrough builds upon these conceptual challenges by testing AI performance against controlled, multi-document scenarios.
Technical and Business Explanation
The evaluation mechanism relies on structured test cases embedded within 12 synthetic contract families, which incorporate 36 source documents and 24 distinct questions. Each family tests specific logical constraints, such as identifying whether a clause was modified by a later agreement or whether a prerequisite condition was formally executed.
During the September 20, 2026 pilot, AI-assisted review found that the expected substantive conclusion was present in all 24 responses generated by both ChatGPT and Gemini. Furthermore, neither application invented a cited clause ID, indicating a high degree of precision in document referencing during this specific test run.
However, subtle differences emerged when handling incomplete evidence. A notable example occurred in the missing-signature case designated as F12.Q01. In this scenario, both ChatGPT and Gemini recognized the missing evidence within their explanatory text and actively requested execution evidence. Despite this shared recognition, their structured decision outputs diverged: ChatGPT returned a decision value of “answer” with a null content field, whereas Gemini returned a structured decision of “insufficient_evidence.”
Sector Impact and Implications
Evaluating legal technology against changing contract terms carries significant implications for corporate legal departments, contract management software vendors, and compliance teams. Enterprise users cannot rely solely on simple keyword searches or basic retrieval-augmented generation systems that ignore amendment histories.
As organizations deploy tools to parse complex vendor agreements and master service agreements, rigorous testing frameworks help quantify how well an AI model distinguishes between active provisions, expired terms, and pending amendments. Accurately categorizing missing evidence—such as distinguishing between a fully executed contract and one lacking necessary signatures—prevents downstream operational and legal errors.
Limitations and Attribution
It is essential to note several strict limitations regarding the pilot findings and test materials. The answer key remains provisional, and both the pilot’s methods and its operational limits are published alongside its findings. The findings do not evaluate enterprise platforms such as Sirion, nor do they establish broader production reliability for mission-critical deployments. Crucially, neither the test key nor the final scoring has received independent specialist sign-off from certified legal experts.
Attribution for these testing methodologies and preliminary observations is drawn from published technical walkthrough materials detailing the 12 synthetic contract families and the September 20, 2026 browser pilot.
What to Watch Next
Observers of contract automation and legal technology should monitor how evaluation frameworks adapt as foundation models update their reasoning architectures. Key areas to watch include:
- The formal peer review or specialist sign-off of provisional scoring keys.
- Expansion of synthetic contract testing families to encompass broader regulatory domains and multi-jurisdictional rules.
- Integration of standardized temporal logic benchmarks, similar to FiscalQA Pro and TIDE, into commercial contract lifecycle management systems.
- Deeper examination of structured output consistency, particularly how models handle edge cases involving missing signatures and ambiguous amendment hierarchies.