zlacker

>>Josely+(OP)
This is a bit off topic to the actual article, but I see a lot of top ranking comments complaining that ChatGPT has become lazy at coding. I wanted to make two observations:

1. Yes, GPT-4 Turbo is quantitatively getting lazier at coding. I benchmarked the last 2 updates to GPT-4 Turbo, and it got lazier each time.

2. For coding, asking GPT-4 Turbo to emit code changes as unified diffs causes a 3X reduction in lazy coding.

Here are some articles that discuss these topics in much more detail.

https://aider.chat/docs/unified-diffs.html

https://aider.chat/docs/benchmarks-0125.html

>>anothe+Pf
Lazy coding is a feature not a bug. My guess is that it breaks aider automation, but by analyzing the AST that wouldn't be a problem. My experience with lazy coding, is it omits the irrelevant code, and focuses on the relevant part. That's good!

As a side note, i wrote a very simple small program to analyze Rust syntax, and single out functions and methods using the syn crate [1]. My purpose was exactly to make it ignore lazy-coded functions.

[1]https://github.com/pramatias/replacefn/tree/master/src

>>empora+jN
It sounds like you've been extremely lucky and only had GPT "omit the irrelevant code". That has not been my experience working intensively on this problem and evaluating numerous solutions through quantitative benchmarking. For example, GPT will do things like write a class with all the methods as simply stubs with comments describing their function.

Your link appears to be ~100 lines of code that use rust's syntax parser to search rust source code for a function with a given name and count the number of AST tokens it contains.

Your intuitions are correct, there are lots of ways that an AST can be useful for an AI coding tool. Aider makes extensive use of tree-sitter, in order to parse the ASTs of a ~dozen different languages [0].

But an AST parser seems unlikely to solve the problem of GPT being lazy and not writing the code you need.

[0] https://aider.chat/docs/repomap.html

>>anothe+N31
Really great article. Interestingly I have found that using the function call output significantly improves the coding quality.

However for now, I have not run re-tests for every new version. I guess I know what I will be doing today.

This is an area I have spend a lot of time working on, would love to compare notes.