I assume that it's trickier than it seems as it hasn't happened yet.
One could use the LSP errors to remove those completions.
While we have millions of LOCs to train models on, we don’t have that for ASTs. Also, except for LISP and some macro supporting languages, the AST is not usually stable at all (it’s an internal implementation detail). It’s also way too sparse because you need a pile of tokens for even simple operations. The Scala AST for 1 + 2 for example probably looks like this,
Apply(Select(scala, Select(math, Select(Int, Select(+)))), New(Literal(1)), Seq(This, New(Literal(2))) etc etc
which is way more tokens than 1 + 2. You could possibly use a token per AST operation but then you can’t train on human language anymore and you need a new LLM per PL, and you can’t solve problem X in language Y based on a solution from language Z.
Agreed, but that could be generated if it made a big difference.
I do completely take your points around the instability of the AST and the length, those are important facets to this question.
However, what I (and probably others) want is something much, much simpler. Merely (I love not having to implement this so I can use this word ;) ) check the code with the completion done (so what the AI proposes) and weight down completions that increase the number of issues found from the type-checking/linting/lsp process.
Honestly, just killing the ones that don't parse properly would be very helpful (I've noticed that both Copilot and the DBX completers are particularly bad at this one).