A good knowledge base should sometimes say “I don’t know”

During a training session I showed a simple experiment. I added a source to a notebook with made-up information about the colour of a company’s headquarters. Then I asked about that colour. The tool answered on the basis of the note I had added.

Limiting a knowledge base to the right sources is part of the quality of its answers.

Next I cleared the conversation, switched that source off and asked the question again. This time I was told that the available materials did not contain the answer.

The second result was just as important as the first. It showed that in this trial the tool respected the boundary of the available material. It did not try to rescue the conversation with any old colour.

Why a knowledge base needs a boundary

When working with your own document base, this is often exactly what we need. If we ask about the team’s arrangements, the terms of a service or a procedure, a generally plausible answer can be useless. We want to know what follows from specific materials.

Tools such as Gemini Notebook, formerly known as NotebookLM, let you choose the sources used in a conversation. That is a practical form of controlling the scope. It does not guarantee correctness, though. The tool can still misread a document, miss an exception or draw too far-reaching a conclusion. And if the source itself is wrong, quoting it faithfully will not fix reality.

How to prepare and test a knowledge base

That is why, when building a knowledge base, I would start with the quality of the collection: current versions, clear names and removing documents that should no longer apply. A large number of files is not the goal. An old set of terms added next to the new one can complicate the answer instead of enriching it.

A useful test looks like this: prepare a question whose answer is in the sources, a question that requires combining two passages and a question the materials do not allow you to answer. Check all three results, together with the references to the documents.

No answer in the last case is the desired behaviour. It lets you notice the gap and decide who should fill it. In a company this may mean writing down a procedure or passing the case to a person authorised to decide.

The colour experiment was deliberately trivial. It was easy to see which source I was switching on and which effect I was switching off. With more serious documents we need the same transparency, even if checking will take more work.

What this means for a company

Before giving employees such a tool, it is worth preparing questions it should answer as well as questions outside the documentation. If the system fills in missing facts, you need to correct the range of sources or the way it is used.

Sources: Google, adding and choosing sources, Google, references to source passages.