zlacker

Our team has found success in treating skills more like re-usable semi-deterministic functions and less like fingers-crossed prompts for random edge-cases.

For example, we have a skill to /create-new-endpoint. The skill contains a detailed checklist of all the boilerplate tasks that an engineer needs to do in addition to implementing the logic (e.g. update OpenAPI spec, add integration tests, endpoint boilerplate, etc.). The engineer manually invokes the skill from the CLI via slash commands, provides a JIRA ticket number, and engages in some brief design discussion. The LLM is consistently able to one-shot these tickets in a way that matches our existing application architecture.

>>jgmedr+(OP)
How do you test these skills for consistency over time, or is that not needed?

>>moored+ml
The same way you'd test a human following written instructions over time.

Check the results.

>>moored+ml
My experience has been that if the skill is broken down into a function, possibly paired with a validator in another stage, you're at 99.9% deterministic.

I have not yet tested this at scale but give me six months.

>>jgmedr+(OP)
So the only difference between slash custom command and agent skills is that they can be invoked only when needed instead of stuffing the whole markdown file? I’m trying to understand how is this different from what we already have in markdown files.

>>sagarp+yJ2
Correct. It helps by not distracting your LLM with a prompt that is, in X% of the cases, irrelevant to the task at hand.

However, when you DO need to do something special (like create a new endpoint), the LLM knows where to get more info on this.

Kinda like a library of „how to“ books.

>>pizzaf+hR
Deal! I will follow up with you in 6 months.