The reason I have yet to publish a book is not because I can't write words. I got to 120k words or so, but they never felt like the right words.
Nobody's giving me (nor should they give me) a participation trophy for writing 120k words that don't form a satisfying novel.
Same's true here. We all know that LLMs can write a huge quantity of code. Thing is, so does:
yes 'printf("Hello World!");'
The hard part, the entire reason to either be afraid for our careers or thrilled we can switch to something more productive than being code monkeys for yet-another-CRUD-app (depending on how we feel), that's the specific test that this experiment failed at.