You might as well just write instructions in English in any old format, as long as it's comprehensible. Exactly as you'd do for human readers! Nothing has really changed about what constitutes good documentation. (Edit to add: my parochialism is showing there, it doesn't have to be English)
Is any of this standardization really needed? Who does it benefit, except the people who enjoy writing specs and establishing standards like this? If it really is a productivity win, it ought to be possible to run a comparison study and prove it. Even then, it might not be worthwhile in the longer run.
It’s also related to attention — invoking a skill “now” means that the model has all the relevant information fresh in context, you’ll have much better results.
What I’m doing myself is write skills that invoke Python scripts that “inject” prompts. This way you can set up multi-turn workflows for eg codebase analysis, deep thinking, root cause analysis, etc.
Works very well.
I'm very curious to know the size & state of a codebase where skills are beneficial over just having good information hierarchy for your documentation.
codex + skills finetunes Qwen3-0.6B to +6 on humaneval and beats the base score on the first run.
I reran the experiment from this week, but used codex's new skills integration. Like claude code, codex consumes the full skill into context and doesn't start with failing runs. It's first run beats the base score, and on the second run it beats claude code.
That said, it's not a perfect comparison because of the Codex model mismatch between runs.
The author seems to be doing a lot of work on skills evaluation.
I am very interested in finding ways to combine skills + local models + MCP + aider-ish tools to avoid using commercial LLM providers.
Is this a path to follow? Or, something different?
To be clear, I'm suggesting that any specific format for "skills.md" is a red herring, and all you need to do is provide the LLM with good clear documentation.
A useful comparison would be between: a) make a carefully organised .skills/ folder, b) put the same info anywhere and just link to it from your top-level doc, c) just dump everything directly in the top-level doc.
My guess is that it's probably a good idea to break stuff out into separate sections, to avoid polluting the context with stuff you don't need; but the specific way you do that very likely isn't important at all. So (a) and (b) would perform about the same.
So yeah, I agree that it's all just documentation. I know there's been some evidence shown that skills work better, but my feeling is that in the long run it'll fall to the wayside, like prompt engineering, for a couple of reasons. First, many skills will just become unnecessary - models will be able to make slide decks or do frontend design without specific skills (Gemini's already excellent at design without anything beyond the base model, imho). Second, increased context windows and overall intelligence will obviate the need for the specific skills paradigm. You can just throw all the stuff you want Claude to know in your claude.md and call it a day.
https://xcancel.com/ben_burtenshaw
I'd like a user writeable, LLM readable, LLM non-writable character/sequence. That would make it a lot easier to know at a glance that a command/file/directory/username/password wasn't going to end up in context and being used by a rogue agent.
It wouldn't be fool proof, since it could probably find some other tool out there to generate it (eg write-me some unicode python), but it's something I haven't heard of that sounds useful. If it could be made fool/tool proof (fools and tools are so resourceful) that would be even better.
My guess is that the standardization is going to make its way into how the models are trained and Skills are eventually going to pull out ahead.
0: https://vercel.com/blog/agents-md-outperforms-skills-in-our-...
Having a super repo of everyone else's slop is backwards thinking; you are now in the era where creating written content and verifying it's effectiveness is easier than ever.
I wish they arranged it around READMEs. I have a directory with my tasks and I have a README.md there - before codex had skills it already understood that it needs to read the readme when it was dealing with tasks. The skills system is less directory dependent so is a bit more universal - but I am not sure if this is really needed.
This standardization, basically, makes a list of docs easier to scan.
As a human, you have a permanent memory. LLMs don't have it, they have to load it into the context, and doing it only as necessary can help.
E.g. if you had anterograde amnesia, you'd want everything to be optimally organized, labeled, etc, right? Perhaps an app which keeps all information handy.
https://claude.com/blog/context-management
> Context editing automatically clears stale tool calls and results from within the context window when approaching token limits.
> The memory tool enables Claude to store and consult information outside the context window through a file-based system.
But it looks like nobody has it as a part of an inference loop yet: I guess it's hard to train (i.e. you need a training set which is a good match for what people use context in practice) and make inference more complicated. I guess more high-level context management is just easier to implement - and it's one of things which "GPT wrapper" companies can do, so why bother?
A +6 jump on a 0.6B model is actually more impressive than a +2 jump on a 100B model. It proves that 'intelligence' isn't just parameter count; it is context relevance. You are proving that a lightweight model with a cheat sheet beats a giant with amnesia. This is the death of the 'bigger is better' dogma
Yeah, WWW is really just text but that doesn't mean you don't need HTTP + HTML and a browser/search engine. Skills is just that, but for agent capabilities.
Long term you're right though, agents will fetch this all themselves. And at some point they will not be our agents at all.
Plus, as has been mentioned multiple times here, standard skills are a lot more about different harnesses being able to consistently load skills into the context window in a programmatic way. Not every AI workload is a local coding agent.
Long term you're right though, agents will fetch this all themselves
It's not "long term", it's right now. If your docs are well-written and well-organised, agents can already use them. The most you might need to do is copy your README.md into CLAUDE.md.
Programs and data are the basis of deterministic results that are accessible to the llm.
Embedding an sqlite database with interesting information (bus schedules, dietary info, or a thousand other things) and a python program run by the skill can access it.
For Claude at least, it does it in a VM and can be used from your phone.
Sure, skills are more convention than a standard right now. Skills lack versioning, distribution, updates, unique naming, selective network access. But they are incredibly useful and accessible.
You don't want to give an English description of how to compress LZMA and then let the AI do it token by token. Although that would be a pretty good arduous methodical benchmark task for an AI.
I haven't done a formal study, so I can't prove it, but it seems like I get better output from agents if I tailor my English more towards the LLM way of "thinking".
Their reasoning about it is also flawed. E.g. "No decision point. With AGENTS.md, there's no moment where the agent must decide "should I look this up?" The information is already present." - but this is exactly the case for skills too. The difference is just where in the context the information is, and how it is structured.
Having looked at their article, ironically I think the reason it works is that they likely force more information into context by giving the agent less information to work with:
Instead of having a description, which might convince the agent a given skill isn't relevant, their index is basically a list of vague filenames, forcing the agent to make a guess, and potentialy reading the wrong thing.
This is basically exactly what skills were added to avoid. But it will break if the description isn't precise enough. And it's perfectly possible that current tooling isn't aggressive enough about pruning detail that might tempt the agent to ignore relevant files.
Splitting the docs into neat modules is a good idea (for both human readers and current AIs) and will continue to be a good idea for a while at least. Getting pedantic about filenames, documentation schemas and so on is just bikeshedding.
So if you want to do this, the current workaround is basically to have a sub-agent carry out tasks you don't want to pollute the main context.
I have lots of workflows that gets farmed out to sub-agents that then write reports to disk, and produce a summary to the main agent, who will then selectively read parts of the report instead of having to process the full source material or even the whole report.
For example, if you've just joined a new team or a new project, wouldn't you like to have extensive, well-organised documentation to help get you started?
This reminds me of the "curb-cut effect", where accommodations for disabilities can be beneficial for everybody: https://front-end.social/@stephaniewalter/115841555015911839
Claude is programmed to stop reading after it gets through the skill’s description. That means we don’t consume more tokens in the context until Claude decides it will be useful. This makes a big difference in practice. Working in a large repo, it’s an obvious step change between me needing to tell Claude to go read a particular readme that I know solves the problem vs Claude just knowing it exists because it already read the description.
Sure, if your project happened to already have a perfect index file with a one-sentence description of each other documentation file, that could serve as a similar purpose (if Claude knew about it). It’s worthwhile to spread knowledge about how effective this pattern is. Also, Claude is probably trained to handle this format specifically.
Making your docs nice and modular, and having a high-level overview that tells you where to find more detailed info on specific topics, is definitely a good idea. We already know that when we're writing docs for human readers. The LLMs are already trained on a big corpus written by and for humans. There's no compelling reason why we need to do anything radically different to help them out. To the contrary, it's better not to do anything radically different, so that new LLM-assisted code and docs can be accessible to humans too.
Well-written docs already play nicely with LLM context.
To overly programmer-brain it, a slash command is just a skill with a null frontmatter. This means that it doesn't participate in progressive disclosure, aka Claude won't consider invoking it automatically.
Now if a format dominates it will be post trained for and then it is in fact better.
IMO it's great if a plugin wants to have their own conventions for how to name and where to put these files and their general structure. I get the sense it doesn't matter to agents much (talking mostly claude here) and the way I use it I essentially give its own "skills" based on my own convention. It's very flexible and seems to work. I don't use the slash commands, I just script with prompts into claude CLI mostly, so if that's the only thing I gain from it, meh. I do see other comments speculating these skills work more efficiently but I'm not sure I have seen any evidence for that? Like a sibling comment noted I can just re-feed the skill knowledge back into the prompt.
The whole point of LLM-based code execution is, well, I can just type in any old language it understands and it ought to figure out what I mean!
A "skill" for searching a pdf could be :
* "You can search PDFs. The code is in /lib/pdf.py"
or it could be:
* "Here's a pile of libraries, figure out which you want to use for stuff"
or it could be:
* "Feel free to generate code (in any executable programming language) on the fly when you want to search a PDF."
or it could be:
* "Solve this problem <x>" and the LLM sees a pile of PDFs in the problem and decides to invent a parser.
or any other nearly infinite way of trying to get a non-deterministic LLM to do a thing you want it to do.
At some level, this is all the same. At least, it rounds to the same in a sort of kinda "Big O" order-of-magnitude comparison.
On the other hand, I also agree, but I can definitely see present value in trying to standardize it because humans want to see what is going on (see: JSON - it's highly desirable for programmers to be able to look at a string representation of data than send opaque binary over the wire, even though to a computer binary is gonna be a lot faster).
There is probably an argument, too, for optimization of context windows and tokens burned and all that kinda jazz. `O(n)` is the same as `O(10*n)` (where n is tokens burned or $$$ per second or context window size) and that doesn't matter in theory but certainly does in practice when you're the one paying the bill or you fill up the context window and get nonsense.
So if this is a _thoughtful_ standard that takes that kinda stuff into account then, well, great! It gives a benchmark we can improve and iterate upon.
With some hypothetical super LLM that has a nearly infinite context window and a cost/tok of nearly zero and throughput nearing infinity, you can just say "solve my problem" and it will (eventually) do it. But for now, I can squint and see how this might be helpful.
This is different from swagger / OpenAPI how?
I get cross trained web front-end devs set a new low bar for professional amnesia and not-invented-here-ism, but maybe we could not do that yet another time?
I can't speak to what the exact split is or what is a part of post training versus pre training at various labs but I am exceedingly confident all labs post train for effectiveness in specific domains.
I claimed that OpenAI overindexed on getting away with aggressive post-training on old pre-training checkpoints. Gemini / Anthropic correctly realized that new pre-training checkpoints need to happen to get the best gains in their latest model releases (which get post-trained too).
Of course any LLM can write any script based on a document, but that's not very deterministic.
A good example is Anthropic's PDF creator skill. It has the basic english instructions as well as actual Python code to generate PDFs
That said, I reckon that’s actually what this project is trying to lean into. It looks like it's just standardising where those instructions live (the SKILL.md format) so tools can find them, rather than trying to force a new schema.
Fair play to them for trying to herd the cats. I think there's an xkcd comic for this one somewhere.
Aspects of it will be similar but it trends to disruption as it becomes clear the new paradigm just works differently (for both better and worse) and practices need to be rethought accordingly.
I actually suspect the same is true of the entire 'agent' concept, in truth. It seems like a regression in mental model about what is really going on.
We started out with what I think is a more correct one which is simply 'feed tasks to the singular amorphous engine'.
I believe the thrust of agents is anthropomorphism: trying to map the way we think about AI doing tasks to existing structures we comprehend like 'manager' and 'team' and 'specialisation' etc.
Not that it's not effective in cases, but just probably not the right way to think about what is going on, and probably overall counterproductive. Just a limiting abstraction.
When I see for example large consultancies talking about things they are doing in terms of X thousands of agents, I really question what meaning that has in reality and if it's rather just a mechanism to make the idea fundamentally digestable and attractive to consulting service buyers. Billable hours to concrete entities etc.
In other words, if you run an identical prompt, one with skill and one without, on a test task that requires discovering deeply how your codebase works, which one performs better on the following metrics, and how much better?
1. Accuracy / completion of the task
2. Wall clock time to execute the task
3. Token consumption of the task
A skill is essentially just a markdown file, containing whatever instructions you want, possibly linking to other markdown files and/or scripts to avoid context pollution.
What skills give you is autodiscovery. You need to somehow tell the agent that documentation exists and when it should be looked at, and that's exactly what the skills standard does. It's a standardized format for documentation that harnesses can automatically detect and inform agents about, without them having to do many useless calls on every single turn to see if there are any skills present.
we wrote a blog on getting agents to write CUDA kernels and evaluating them: https://huggingface.co/blog/upskill
It _can_ run a PEP723 script without any specific setup (as long as uv and python are installed). It will automatically create a virtual environment AND install all dependencies. All with a single command without polluting the context with tons of setup.
In terms of experience, I’ve noticed that agents don’t always use skills the way you want; and I’ve noticed that they’re pretty good at browsing existing code and docs and figuring things out for themselves.
Is this an example of “the bitter lesson”? That’s conjecture, but I think pretty well-founded.
It could well be that specific formats for skills work better because the agents are trained on those specific formats. But if so, I think it’s just a local maximum.
Claude Code and others have some extras, such as the ability for the main agent to put them in the background, spawn them in parallel, and use tool calls to check on the status of them (so basic job control), but "poor mans sub-agents" only requires the ability for the coding agent to run an executable the equivalent of e.g. "claude --print <someprompt" (the --print option is real, and enables headless use; in practise you'd also want --stream-json, set allowed tools, and specify a conversation id so you can resume the sub-agents conversation).
And calling it all "summarising" understates it. It is delegation, and a large part of the value of delegation in a software system is abstraction and information hiding. The party that does the delegation does not need to care about all of the inner detail of the delegated task.
The value is not the summary. The value is the work done that the summary describes without unnecessary detail.
This is true for MCP as well. You could just describe a bunch of command line tools in AGENTS.md and tell the LLM when and how to call them. It would simply take more effort to set up, at least for some tools.
This is where a comparison in productivity would return a meaningful result: how much does it make it easier to set up things like that.
I think the main conflict in this thread is whether skills are anything more than just structuring documentation you were lacking in your repo, regardless if it was for Claude or Steve starting from scratch.
If we're just pattern matching to adjacent memes that might provide insight, I'd also throw "sufficiently smart compiler" into the mix. Like, yes, in theory as the compiler gets better you shouldn't have to worry about implementing random optimizations yourself, but in practice you do.
In theory, you just need normal docs and a sufficiently smart LLM and agent harness can use them, but in practice there's still benefit in organizing them a certain way to more directly manage the context window yourself.
I see this with Cursor all the time with tools. Cursor will stop editing files in the editor and use the command line to echo edits into a file. It's so frustrating.
The "bitter lesson" only applies if the model makes the agent redundant. We aren't there yet. Agentic loops are just software engineering on top of CS constructs; they help current models produce better results.
Could models eventually internalize the logic used in Claude Code / Codex / OpenCode / Aider? Maybe. But for now, keeping that complexity in the agent is more energy-efficient. Even if complex agents eventually get replaced by simple loops, these standards save tokens and time today. That’s worth something.
In the way that Swagger / OpenAPI is for API endpoints, but most of the "skills" you need for your agents are not based on API endpoints
I explicitly tell it about the skills and that it should load them when the context feels correct.
```prompt.md
Company Name codebase ...
# Skills
Use the company specific skills that look like `company-*`. Load them once per conversation if they seem relevant to what you are working on.
```
```SKILL.md
---
description: Company TypeScript libraries and conventions
trigger: Writing or reading TypeScript in Company services ---
# company-ts
```
Why not just extend the OpenAPI specification to skills? Instead of recreating something that's essentially communicating the same information?
T minus a couple years before someone declares that down-mapping skills into a known verb enumeration promotes better skill organization...
Because the descriptions aren't API specs and the things described aren't APIs.
Its more like a structure for human-readable descriptions in an annotated table of contents for a recipe book than it is like OpenAPI.
Because approximately none of what exists in the existing OpenAPI specification is relevant to the task, and nothing needed for the tasks is relevant to the current OpenAPI use case, so trying to jam one use case into a tool designed for the other would be pure nonsense.
It’s like needing to drive nails and asking why grab a hammer when you already have a screwdriver.
As humans we need to specialise. Even though we're generalists and have the a priori potential to learn and do all manner of things we have to pick just a few to focus on to be effective (the beautiful dilemma etc).
I think the basic reason being we're limited by learning time and, relatedly, execution bandwidth of how many things we can reasonably do in a given time period.
LLMs don't have these constraints in the same way. As you say they come preloaded with absolutely everything all at once. There's no or very little marginal time investment per se in learning anything. As for output bandwidth, it also scales horizontally with compute supplied.
So I just think the inherent limitations that make us organise human work around this individual unit working in teams and whatnot don't apply and are counterproductive to apply. There's a real cost to all that stuff that LLMs can just sidestep around, and that's part of the power of the new paradigm that shouldn't be left on the table.
Agent Skills isn't a spec for how information is presented to the model, its a spec whose consumer is the model harness, which might present information made available to it in the format to the model in different ways for different harnesses, or even in the same harness for different models or tasks, considering things like the number and size of the skill(s) available, the size of the model context, the purpose of the harness (is it for a narrow purpose agent where some of the skills are central to that purpose?), and user preference settings.
The site itself has two different main styles of integration for harnesses described ("tool based" and "filesystem based"), but those are more of a starting point for implementers that an exhaustive listing.
The idea is that skill authors don't need to know or care how the harness is presenting the information to the model.
(1) providing a bash tool with direct access to the filesystem storing the skills to the model,
(2) providing read_file and related tools to the model,
(3) by providing specialized tools to access skills to the model,
(4) by processing the filesystem structure and providing a structure that includes the full content of the skills up front to the model.
And probably some other ways or hybrids.
> It increases benchmarks a few points now but what's the point in standardizing all this if it'll be obsolete next year?
Standardizing the information presentation of skills to LLM harnesses lets the harnesses incorporate findings on optimization (which may be specific to models, or at least model features like context size, and use cases) and existing skills getting the benefit of that for free.
That difference alone likely accounts for some not insignificant discrepancies. But without numbers, it's hard to say.
And if you're just making docs and letting your models go buck wild in your shell, doesn't an overspecified docs structure ruin the point of general purpose agents?
Like, a good dev should be able to walk into a codebase, look at the structure, and figure out how to proceed. If "hey your docs aren't where I was expecting" breaks the developer, you shouldn't have hired them.
Feels like a weird thing to take "this is how we organize our repos as this company" and turn that into "this is an 'open standard' that you should build your workflows around".