zlacker

>>goranm+(OP)
> very few open-source LLMs explicitly claim they intentionally support structured data, but they’re smart enough and they have logically seen enough examples of JSON Schema that with enough system prompt tweaking they should behave.

Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al that interact with the internals of the model (https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliabl...). The structured outputs can be arbitrary grammars, for example, not just JSON (https://github.com/outlines-dev/outlines#using-context-free-...).

>>twelft+08
Yeah, JSON mode in Ollama, which isn’t even the full llama.cpp grammar functionality, performs better than OpenAI for me at this point. I don’t understand how they can be raking in billions of dollars and can’t even get this basic stuff right.

>>buggle+Ku
llama.cop Performs better than what?

3.5 Turbo, one of the 4.0 models, API or App?

>>Whitne+AC
JSON mode and function-calling with a JSON schema in the OpenAI API.

>>buggle+nD
Right, but which model?

It makes a huge difference.

>>Whitne+vM
I’ve been using OpenChat 3.5 1210 most recently. Before that, Mistral-OpenOrca. Both return JSON more consistently than gpt-3.5-turbo.

>>buggle+hN
gpt-3.5-turbo is not the benchmark

>>airstr+tP
I don’t know what point you’re trying to make. They also return JSON more consistently than gpt-4, but I don’t use that because it’s overkill and expensive for my text extraction tasks.

>>buggle+V01
Because people have different interests and want to hear your results for different reasons.

Some want to consider results relative to cost, and some are interested only in how it compares to SOTA.

>>Whitne+Fa1
I mean, sure, but the parent should also just explicitly state what it is they were asking or claiming. I’ve answered every question asked. Making vague declarations about something not being “the benchmark,” while not stating what you think “the benchmark” should be, is unhelpful.