“Skill created” does not mean “it works”

During an exercise on creating skills I asked participants for one specific thing: “run the skill you have prepared in a new conversation”. Definitely a new one.

Creating a skill is not the same as confirming that it works.

The reason is simple: in the chat where we create the instruction, the model already knows our requirements. It has them in the conversation history. If it does the task well, we still do not know whether it is using the saved procedure or simply what we told it a moment ago.

That is a problem with the test. The result may look correct even though we have not checked the most important thing: whether the solution also works outside the context in which it was created.

A new conversation is not yet a laboratory

A new conversation reduces this particular problem, but it is not a perfectly clean laboratory. The tool may still use memory, account instructions or project settings. That is why it is worth noting what was supposed to be available and which materials the result was meant to rely on. For a team rollout you also need to check how it works for the target user.

The second step I showed during the training was even simpler: open the saved instructions and see what was actually created. A skill’s description may sound great. The procedure itself may, however, leave out a limitation, misunderstand the task or contain an assumption we never wanted to make.

Three test cases

For testing I would prepare three cases. The first is an ordinary one, with complete data. The second has an important piece of information missing. The third is a situation in which the procedure should refuse to decide or pass the question to a person.

If a skill is meant to answer questions about terms and conditions, it is easy to check the answer to a point that is in the document. Just as important is a question about something the document does not cover. Will the tool recognise the gap, or will it add a plausible-sounding rule?

The word “created” confirms that an operation was performed in the application. Only trials on known cases show whether we have a useful procedure. It is worth finishing the creation of a skill with exactly this check before we start relying on it for further tasks.

What this means for a company

The simplest acceptance test for such a procedure is a test in a new conversation on two or three cases: a typical one, an incomplete one and a deliberately problematic one. The team should know what it considers a correct answer and in which situations automatic action should stop.