Internal knowledge lives in too many places at once: Drive, Notion, SharePoint, Slack, PDFs, CRM notes, and the heads of senior people. A knowledge base agent gives a team one place to ask a question and get a source-backed answer. This playbook walks the four decisions that determine whether it earns trust or quietly gets abandoned. Work through them in order before any building starts.
Step 1 — Define the audience
Start with one team. Support, sales, operations, HR, and finance ask fundamentally different questions, and a corpus built for all of them serves none of them well. Do not build a generic "company brain" first.
Readiness test: you can name five questions this team asks every week. If you're reaching for vague categories instead of concrete questions, scope isn't ready.
Common mistake: starting with the broadest audience because it feels more ambitious. Broad scope is exactly what makes the agent vague and unmeasurable.
Done right when: the agent has one team, and you could predict most of what they'll ask it on day one.
Step 2 — Choose approved sources
List the sources the agent is allowed to treat as ground truth:
- Policy documents
- SOPs
- Product docs
- CRM or ERP records
- Internal wiki pages
- Approved spreadsheets
Every source is a promise that the agent will answer with that document's authority. If the source is wrong, the agent is wrong — faster and more often. Audit each one before indexing: Is it current? Does it have an owner who keeps it current? Does it contradict another source on the list?
Common mistake: indexing two documents that describe the same process differently. The agent will flip-flop depending on which one it retrieves, and grounding can't save you — it will faithfully cite the wrong policy.
Done right when: every source has an owner, and no two sources contradict each other on the same fact.
Step 3 — Decide how answers should behave
A useful knowledge agent should:
- Cite sources — every answer links back to its document for one-click verification
- Say when it doesn't know — "I don't have an approved source for that" beats a confident guess
- Respect permissions — if a user can't see a document, the agent can't read from it on their behalf
- Escalate ambiguity — route genuinely unclear or high-stakes questions to a human
- Refuse unapproved sources — answer from the approved list or don't answer
The goal is trust, not theatrical confidence. One confident wrong answer about a refund or compliance rule teaches the whole team to stop believing the agent — and an agent no one believes is just expensive infrastructure.
Done right when: the agent's "I don't know" rate feels slightly too high to you. That's the correct setting at launch.
Step 4 — Launch narrow, then let usage teach you
Start with a focused group from your one chosen team. In the first weeks, review the logs weekly and watch three things:
- Questions the agent couldn't answer → your missing documents
- Answers that were weak or hedged → your unclear sources
- Questions you didn't anticipate → your real scope vs. the imagined one
Every unanswered question is a precise instruction about what to add next.
Common mistake: launching broad to "get more data." Wide launch buries the signal; a narrow one makes every gap obvious.
Done right when: the team reaches for the agent reflexively instead of pinging each other — and the weekly gap list is shrinking.