zlacker

>>moored+(OP)
This stuff smells like maybe the bitter lesson isn't fully appreciated.

You might as well just write instructions in English in any old format, as long as it's comprehensible. Exactly as you'd do for human readers! Nothing has really changed about what constitutes good documentation. (Edit to add: my parochialism is showing there, it doesn't have to be English)

Is any of this standardization really needed? Who does it benefit, except the people who enjoy writing specs and establishing standards like this? If it really is a productivity win, it ought to be possible to run a comparison study and prove it. Even then, it might not be worthwhile in the longer run.

>>iainme+Qb
We’re working with the models that are available now, not theoretical future models with infinite context.

Claude is programmed to stop reading after it gets through the skill’s description. That means we don’t consume more tokens in the context until Claude decides it will be useful. This makes a big difference in practice. Working in a large repo, it’s an obvious step change between me needing to tell Claude to go read a particular readme that I know solves the problem vs Claude just knowing it exists because it already read the description.

Sure, if your project happened to already have a perfect index file with a one-sentence description of each other documentation file, that could serve as a similar purpose (if Claude knew about it). It’s worthwhile to spread knowledge about how effective this pattern is. Also, Claude is probably trained to handle this format specifically.

>>ashdks+N41
To clarify, the bit where I think the bitter lesson applies is trying to standardize the directory names, the permitted headings and paragraph lengths, etc. It's pointless bikeshedding.

Making your docs nice and modular, and having a high-level overview that tells you where to find more detailed info on specific topics, is definitely a good idea. We already know that when we're writing docs for human readers. The LLMs are already trained on a big corpus written by and for humans. There's no compelling reason why we need to do anything radically different to help them out. To the contrary, it's better not to do anything radically different, so that new LLM-assisted code and docs can be accessible to humans too.

Well-written docs already play nicely with LLM context.

>>iainme+A71
Is your view that this doesn’t work based on conjecture or direct experience? It’s my understanding Anthropic and OpenAI have optimized their products to use skills more efficiently and it seems obviously true when I add skills to my repo (even when the info I put there is already in existing documentation).

>>ashdks+Sh2
Hmm, that’s a good question! I think a bit of both.

In terms of experience, I’ve noticed that agents don’t always use skills the way you want; and I’ve noticed that they’re pretty good at browsing existing code and docs and figuring things out for themselves.

Is this an example of “the bitter lesson”? That’s conjecture, but I think pretty well-founded.

It could well be that specific formats for skills work better because the agents are trained on those specific formats. But if so, I think it’s just a local maximum.

>>iainme+033
I had a kind of visceral distaste for all of this rules, skills etc stuff when I first heard about it for similar reasons. This generalized text model can speak base64 encoded Klingon but a readme.md isn’t good enough? However given the reality of limited context windows, current models can’t consider everything in a big repo all at once and keep coherent. Attaching some metadata to the information that tells the model when and how to consider it (and assisting the models with tooling written in code to provide the context at the right time) seems to make a big difference in practice.