My understanding/experience is that LLM performance in a language scales with how well the language is represented in the training data.
From that assumption, we might expect LLMs to actually do better with an existing language for which more training code is available, even if that language is more complex and seems like it should be “harder” to understand.
Additionally just the ability to put an entire language into context for an LLM - a single document explaining everything - is also likely to close the gap.
I was skimming some nano files and while I can't say I loved how it looked, it did look extremely clear. Likely a benefit.
The characteristics of failures have been interesting: As I anticipated it might be, an over ambitious refactoring was a train wreck, easily reverted. But something as simple as regenerating Android launcher icons in a Flutter project was a total blind spot. I had to Google that like some kind of naked savage running through the jungle.
As others said, the key is feedback and prompting. In a model with long context, it'll figure it out.
This does fill up context a little faster, (1) not as much as debugging the problem would have in a dynamic language, and (2) better agentic frameworks are coming that “rewrite” context history for dynamic on the fly context compression.
This isn't even true today. Source: heavy user of claude code and gemini with rust for almost 2 years now.
This isn't really true. LLMs understand grammars really really well. If you have a grammar for your language the LLM can one-shot perfect code.
What they don't know is the tooling around the language. But again, this is pretty easily fixed - they are good at exploring cli tools.
Getting the Doom sound working on it involved me setting there typing "No I can't hear anything" over and over until it magically worked...
Maybe I should have written a helper program to listen using the microphone or something.
(Which is true - it's easy to prompt your LLM with the language grammar, have it generate code and then RL on that)
Easy in the sense of "it is only having enough GPUs to RL a coding capable LLM" anyway.
Eventually AIs will create their own languages. And humans will, of course, continue designing hobbyist languages for fun. But in terms of influence, there will not be another human language that takes the programming world by storm. There simply is not enough time left.
In the long term I expect it won't matter - already GPT3.5 was able to reason about the basic semantics of programs in languages "synthesised" zero-shot in context by just describing it as a combination of existing languages (e.g. "Ruby with INTERCAL's COME FROM") or by providing a grammar (e.g. simple EBNF plus some notes on new/different constructs) reasonably well and could explain what a program written in a franken-language it had not seen before was likely to do.
I think long before there is enough training data for a new language to be on equal grounds in that respect, we should expect the models to be good enough at this that you could just provide a terse language spec.
But at the same time, I'd expect the same improvement to future models to be good enough at working with existing languages that it's pointless to tailor languages to LLMs.
That's when I smelled that things were getting a little crazy.
I still experience agents slipping in a `todo!` and other hacks to get code to compile, lint, and pass tests.
The loop with tests and doc tests are really nice, agreed, but it'll still shit out bad code.
This is such a silly thing to say. Either you set the bar so low that "hello world" qualifies or you expect LLMs to be able to reason about lifetimes, which they clearly cannot. But LLMs were never very good at full-program reasoning in any language.
I don't see this language fixing this, but it's not trying to—it just seems to be removing cruft
I don't know what to say. May experience does not match yours.
The point is that your statement about the ability to do RL is wrong.
Additionally your response to the Deepseek paper in the other subthread shows profound and deliberate ignorance.
https://github.com/benj-edwards/atari800-ai https://github.com/benj-edwards/bobbin
GPU poor here though...
To quote someone (you...) on the internet:
> More generally, don't ask random people on the internet to do work for you for free.
That GRPO works?
> Group Relative Policy Optimization (GRPO), a variant reinforcement learning (RL) algorithm of Proximal Policy Optimization (PPO) (Schulman et al., 2017). GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources. By solely using a subset of English instruction tuning data, GRPO obtains a substantial improvement over the strong DeepSeekMath-Instruct, including both in-domain (GSM8K: 82.9% → 88.2%, MATH: 46.8% → 51.7%) and out-of-domain mathematical tasks (e.g., CMATH: 84.6% → 88.8%) during the reinforcement learning phase
Page 2 of https://arxiv.org/pdf/2402.03300
That GRPO on code works?
> Similarly, for code competition prompts, a compiler can be utilized to evaluate the model’s responses against a suite of predefined test cases, thereby generating objective feedback on correctness
Page 4 of https://arxiv.org/pdf/2501.12948
> here is no RL for programming languages.
and
> Either RL works & you have evidence
This is just so completely wrong, and here is the evidence.
I think everyone in this thread is just surprised you don't seem to know this.
Haven't you seen the hundreds of job ads for people to write code for LLMs to train on?
I manage a team of interns and I don't have the energy to babysit an agent too. For me, gpt and gemini yield the best talk-it-through approach. For example, dropping a research paper into the chat and describing details until the implementation is clarified.
We also use Claude and Cursor, and that was an exceptionally disruptive experience. Huge, sweeping, wrong changes all over. Gyah! If I bitch about todo! macros, this is where they came from.
For hobby projects, I sometimes use whatever free agent microsoft is shilling via VS Code (and me selling my data) that day. This is relatively productive, but reaches profoundly wrong conclusions.
Writing for CLR in visual studio is the smoothest smart-complete experience today.
I have not touched Grok and likely won't.
/ two pennies
Hope that answers your questions.