Ground it

One tool swapped. The agent does not read 3,000 documents; it asks a knowledge base that does.

In this order. Step 2 is where most of the time goes; step 7 is the number she asked for.

1. Start from Project 01

   Same Foundry project, same agent, same instructions file. Create an Azure AI Search service (Basic tier is enough) and a Blob Storage container called policies.

   The Project 01 agent still answers from its three PDFs. Nothing is broken yet.
2. The documents

   Generate 3,000 synthetic policy documents from a script: 40 product lines, 5 regions, 3 years of revisions, and a handful of deliberate contradictions where a newer document supersedes an older one. Upload them to the container.

   3,000 blobs. You can name three that contradict each other and say which one is current.
3. Knowledge source and knowledge base

   In Azure AI Search, create a knowledge source pointing at the container (indexed, with the indexer's execution environment left default for now) and a knowledge base over it. Give the Search service a system-assigned identity with Storage Blob Data Reader on the container.

   The index reports about 3,000 documents and the indexer run shows zero failures.
4. Swap the tool

   Remove file search from the agent. Add the Foundry IQ knowledge tool pointing at the knowledge base.¹ Give the agent's identity Search Index Data Reader on the service. Change one line of the instructions: "cite the document title and page for every policy statement."

   "Is water damage covered on an AquaGuard washer sold in Ohio in 2025?" returns an answer with a document title and page, and the document is the current one.
5. Make it refuse

   Ask about a product that does not exist. Then ask about a real product with a question none of the 3,000 documents answers. Add to the instructions: "if the knowledge base returns nothing relevant, say so and offer a person; never answer from general knowledge."

   Both questions get "I could not find that in our policies" and a handoff offer, not a guess.
6. Evaluation set

   Write 100 questions with the expected document for each, including 15 that should get a refusal. Half from the support team's real inbox, rewritten; half adversarial: superseded policies, near-duplicate product names, a region that does not exist.

   A CSV with 100 rows: question, expected document, expected behaviour.
7. Measure the hit rate

   Run all 100 through the agent with tracing on. For each, record whether the cited document matched the expected one. Run the Foundry groundedness evaluator over the answers. Write the two numbers down.

   You have a hit rate, a groundedness score, and a list of the misses. Now run the identical set a second time. Ours moved from 5/15 to 9/15 with nothing changed, so a single run is not a measurement; two runs of the same set tell you how much of your number is noise before you start attributing changes to your fixes.⁴
8. Fix the worst miss, measure again

   The misses cluster on superseded documents outranking current ones. The filter does not live on the knowledge source, which accepts little more than an index name; it goes on the retrieve call, as filterAddOn inside knowledgeSourceParams. Index a boolean you can filter on, pass it per request, and rerun the same fifteen questions.⁴

   "knowledgeSourceParams": [{
     "knowledgeSourceName": "policies-ks",
     "kind": "searchIndex",
     "filterAddOn": "superseded eq false"
   }]

   Ours went from 5/15 and 9/15 unfiltered to 15/15 twice with the filter, and the average references per question dropped from 13 to 7. Two numbers moved: accuracy up, and the amount of text the model had to read down.

python measure_hit_rate.py
             no filter: current doc ranked first  5/15 ( 33%), avg 13 refs/query
   no filter, 2nd pass: current doc ranked first  9/15 ( 60%), avg 13 refs/query
   filterAddOn applied: current doc ranked first 15/15 (100%), avg  7 refs/query
 filterAddOn, 2nd pass: current doc ranked first 15/15 (100%), avg  6 refs/query

33 then 60 with no change at all: that gap is the noise floor of a single run.

9. Tear it down

   azd down

   Then delete the Search service and the storage account by hand; azd down only removes what azd made.

   The resource group is empty. Today's cost is under $12.

Cause each one on purpose. Retrieval fails quietly; these make it loud.

• The confident wrong document A superseded 2023 policy outranks the current 2025 one, so the answer cites a real document, quotes it accurately, and is still wrong. This is not an edge case: unfiltered, the superseded revision came first in two thirds of our questions on one pass. Fix: a filterable field in the index and filterAddOn on the retrieve call, not a sentence in the prompt asking for the newest one.⁴
• The empty index The indexer ran, reported success, and indexed nothing, because the Search service's identity had no Blob reader role. Fix: check the document count, not the run status.
• General-knowledge leakage Remove the "never answer from general knowledge" line and ask about a competitor's warranty. Watch it answer helpfully and wrongly. Put the line back.
• The citation that is not there The agent cites "page 4" of a two-page document. The model invented the page. Fix: pass the retrieved chunk's real reference through, and instruct it to cite only what the tool returned.
• The role you forgot The agent's identity lacks Search Index Data Reader; the tool returns a 403 the agent reports as "I could not find that." Which is the wrong kind of honest. Fix: make tool errors distinguishable from empty results.
• Cost of curiosity Agentic retrieval runs several searches per question. Read the Search and token cost of your 100-question run, then multiply by a real week's volume.

Foundry IQ, or plain Azure AI Search as a tool?

A plain search tool runs the query you give it. Foundry IQ plans several, checks the results, and tries again, which is what makes it survive vague customer questions. The cost is more searches per turn and a preview label. This build takes the preview because the hit rate is the point.

Indexed, or remote knowledge sources?

Indexing Blob content gives you ranking, filters and version metadata. Remote sources such as SharePoint are queried live and stay where they are. Policies are documents you own; index them.

Refuse, or answer with a warning?

"I could not find that" loses a customer who could have been helped. "Here is my best guess, please confirm" invents warranty terms. For warranty, refuse and hand off; the cost of a wrong answer is a refund.

One run, or several?

Agentic retrieval plans its own queries, so the same question set does not return the same thing twice. Two identical unfiltered passes scored 33 and 60 per cent for us. Any change smaller than that gap is invisible in a single run, which means a number you took once cannot tell you whether your fix worked.

Measure with a hundred questions, or a thousand?

A hundred gives you a number with a wide margin and a list of misses you can read in an afternoon. A thousand gives a tighter number and a list you will not read. Start at a hundred; grow it when the misses stop clustering.

The number, twice

Hit rate and groundedness before and after the version filter, with the date.

One miss, explained

A question, the wrong document it cited, the right one, and the index change that fixed it.

The repo

The document generator, the evaluation CSV, the index definition, and the scoring script.

One sentence for the résumé

"Grounded a support agent in 3,000 documents using a Foundry IQ knowledge base on Azure AI Search, with citation enforcement, a refusal path, and a measured retrieval hit rate tracked across index changes."