Does AI training actually work? Good feedback from participants is not enough

Ten out of eleven people named KREDO as the most important thing they took away from my AI training. I could call that a success, talk about how effective my method is and leave it there. Except that would be going much too far.

Remembering something does not yet mean you can use it

KREDO is my method for building prompts, based on five elements: Context, Role, Editing, Action, Reference (the letters come from the Polish words).

I teach it because I want to give participants techniques, not just ready-made instructions to copy. I want them to be able to solve new problems on their own, including ones they did not practise in class.

When, at the end of one session, I asked what the most important thing they had learned was, ten people out of eleven wrote “KREDO” in the chat.

It is an interesting observation. It shows that one element of the training stuck with the participants at least well enough for them to name it at the end of the session.

What I do not know is whether, the next day, they could apply the method in their work, recognise the mistakes in a vague prompt and get a better result without my help. I did not measure that.

And that is exactly why I believe that any discussion of how effective training is should go beyond the satisfaction survey.

What I think should happen during training

I hold a few beliefs about designing sessions that I have developed while working with groups.

Above all, I do not want participants to spend most of the time listening passively. So I bring in exercises, experiments, tasks in pairs and challenges that require independent thinking.

Sometimes participants try to break an AI agent built by another pair. At other times they fix a prompt, watch what it produces and look for ways to improve it.

I also try to use problems that come up in the room. If a participant brings a real task from their work, that example can turn out to be more useful than a scenario prepared in advance.

I do not give up on humour either. Stories, memes and simple associations help me hold the group’s attention. I want participants to experience something, not just sit through another explanation.

Just as important to me is the ability to respond to what is happening during the session.

If a group of accountants has come to learn how to use AI in practice, I see no point in dragging out a lecture on neural networks and model temperature. Especially if the participants stopped listening long ago.

So my approach is based on tailoring the content, active exercises and watching the group closely.

But that is still a description of how I work, not proof that it works.

What I would like to measure before saying my method works

For some time I have been thinking about how to check the results of my training reliably.

I am not interested only in whether the participants enjoyed it. I would like to know what they can actually do after the session that they could not do before.

I see room for a simple before-and-after measurement.

For example, before the session a participant could be given a task that requires writing instructions for AI. After the training they would solve a comparable task, and both results would be assessed against the same criteria.

You could look not only at the quality of the model’s final answer, but also at whether the participant can recognise a poor result and improve their instructions.

For now this is an idea for a measurement, not a study that has been carried out.

I would like to have such data, because I know that a good atmosphere during training does not automatically mean that skills have grown.

What does this mean for a company ordering training?

If I were responsible for choosing AI training for employees, I would not limit myself to comparing programmes.

I would ask the trainer how participants will practise on their own, whether there will be examples from their daily work and how the trainer will respond if it turns out the group needs a different pace.

I would also ask how you will be able to tell that the training has had the intended effect.

There is no single perfect recipe for every group, at least as far as I am concerned. I am still experimenting, observing and looking for better solutions myself.

I also know that participants appreciate engaging sessions. One of them once told me: “I didn’t believe something could be interesting for 8 hours.”

But in the future I would like the results of skills measurements to be just as strong an argument.

Because a company does not buy eight hours of entertainment or a presentation about artificial intelligence. It buys the chance to teach employees something that will be useful to them.

And that should be possible to check.