Summary

• Ask the toolkit is a public-beta assistant drawing on about 40 departmental guidance pages.
• Defra's published evaluation met four of six performance thresholds and missed two on quotation and source grounding.
• The service cannot change government records or execute autonomous administrative tasks.

The Department for Environment, Food and Rural Affairs has launched a public-beta AI assistant for navigating its digital guidance, publishing detailed information about the model, hosting, testing and limitations before wider operational experience is available.

Ask the toolkit opened on 27 October and provides short answers based on roughly 40 pages of departmental AI guidance. Users can ask questions in ordinary language, with links to source pages and a referral to human staff where the material does not address a request.

Defra’s algorithmic transparency record identifies Claude Sonnet 4.6 through Amazon Bedrock in the AWS London region. It says the service passed four of six pre-launch performance thresholds, while missing two that concern quoting expected rules and grounding answers in the specified sources.

A limited assistant, not an agent

The application is restricted to its supplied guidance material, rather than freely searching the public web. It receives a question alongside the toolkit content and up to four previous turns from the conversation, then produces a short answer and relevant links.

A separate application check compares any quoted text with the cited source page and removes quotations that cannot be matched. This reduces fabricated exact quotations, but does not guarantee that the quoted rule is the best one for a question or that it has been interpreted correctly.

Where the source material does not support a reliable answer, the system is instructed to acknowledge the limitation and refer the user to a human team. That design is intended to avoid unrestricted answers generated from the model’s general training information.

The assistant has no tools to amend government records, browse outside websites or operate an autonomous workflow. Its purpose is information retrieval and explanation, not decisions about policy, people or services.

Four passes and two missed standards

Defra tested the application on 19 October using a fixed question set, running repeated assessments with Claude Sonnet 4.6 and using another model to judge parts of the output against six criteria.

Four thresholds were met: correct answer-status classification, refusal of specified unsuitable questions, sufficient answer completeness and avoiding fabricated quotations. The department reports successful performance against the minimum requirement in those categories.

Two more demanding tests were missed. The assistant selected the expected rule in only 16 to 19 of 22 relevant cases, and its answers were judged grounded in named sources in 87 to 90 of 100 cases. Both thresholds required every applicable answer to pass.

The results are significant because users may regard a fluent explanation as authoritative. An answer can quote a real passage and still omit context or point to a less relevant rule, particularly where different parts of guidance address related situations.

Defra also compared a subset of automated evaluations with human judgements and reported inconsistent agreement between reviewers. The published record acknowledges that automated scoring is not a perfect measure of correctness, especially for source grounding and completeness.

Further manual tests included simulated prompt attacks and checks for inappropriate refusals. Some limitations remained after mitigation, including at least one incorrect answer and the system sometimes assuming users had access to internal departmental services.

Hosting, privacy and operational controls

The service uses a website and backend application hosted on Defra’s Core Delivery Platform, connecting to AWS inference infrastructure. The London region specifies where the model service operates, not that every relevant supplier or technology component is domestically owned.

Users are told not to submit personal or sensitive information. Routine monitoring is designed to track response categories, errors and costs rather than preserve every user’s question, although text can be supplied to staff when a user actively files a problem report.

Defra’s data protection officer concluded that a full impact assessment was unnecessary for the intended use because the service was not designed to collect personal data. Security and platform-readiness processes were separately completed before the public beta opened.

A daily safety limit stops the application after 600 model calls. Some responses require a second call to correct an invalid structure, so that figure should not be treated as a guarantee of 600 completed user conversations.

Load tests involved 20 simultaneous questions and produced average responses of several seconds in two runs. They provide evidence about a defined test scenario, not confirmed performance under sustained public demand.

What the disclosure makes possible

Publication of the model version, deployment arrangements, evaluation methods and missed thresholds gives readers information usually absent from an ordinary software launch. Future changes can be assessed against an explicit pre-launch baseline.

Users still need to check relevant source guidance when an answer has consequences for their work. Defra says the service will monitor errors and problem reports after launch, making live operational evidence the next source of reliability information.

The algorithmic transparency record establishes both the application boundaries and its weaknesses. It does not show that four passed checks cancel out two missed accuracy requirements or that a beta assistant is ready to make autonomous government decisions.

The case illustrates why separate quality measures are useful. A model can refuse an unsuitable request reliably while still selecting the wrong rule for a legitimate question, meaning a single overall percentage could obscure failures relevant to users.

An assistant constrained to a fixed document set can simplify source auditing, but guidance itself changes. Operators must ensure that new and withdrawn pages are reflected promptly and that links continue to point to authoritative versions.

Accessibility and user testing also matter after technical validation. People may phrase a policy question differently from the benchmark examples, and the service will need to handle unfamiliar wording without inventing exceptions or suggesting inappropriate action.