zlacker

>>goranm+(OP)
> very few open-source LLMs explicitly claim they intentionally support structured data, but they’re smart enough and they have logically seen enough examples of JSON Schema that with enough system prompt tweaking they should behave.

Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al that interact with the internals of the model (https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliabl...). The structured outputs can be arbitrary grammars, for example, not just JSON (https://github.com/outlines-dev/outlines#using-context-free-...).

>>twelft+08
> Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al...

Yes, but you should also instruct the model to follow that specific pattern in its answer, or else the accuracy of the response degrades even though it's following your grammar/pattern/whatever.

For example, if you use Llama-2-7b for classification (three categories, "Positive", "Negative", "Neutral"), you might write a grammar like this:

```

root ::= "{" ws "sentiment:" ws sentiment "}"

sentiment ::= ("Positive" | "Neutral" | "Negative" )

ws ::= [ \t\n]*

```

But if the model doesn't know it has to generate this schema, the accuracy of classifications drops because it's trying to say other things (e.g., "As an AI language model...") which then get suppressed and "converted" to the grammar.

>>behnam+v9
This can sometimes be fixed with a few-shot example for in-context-learning.

But you are right that the model can go off the rails if it is being forced too far from where its 'happy place' is, especially for smaller models.