I can understand having an LLM trained on previous inquiries made via email, chat or transcribed phone calls, but a general LLM like ChatGPT, how is that going to be able to answer customers questions? The information ChatGPT has, specific to Chevrolet of Watsonville can't be anymore than what is already publicly available, so if customers can't find it, then maybe design a better website?
In this particular case they screwed up the implementation.
Besides, what makes you think that it’s ineffective? Any reason to believe that the chat bot was bad in fulfilling legitimate user requests? FYI, someone making it act outside of its intended purpose affects only that person’s experience.
It’s a DAN attack, people are having lots of fun with this type of prompt engineering.
It’s just some fun in the expense of the company paying for the API. The kind of fun that kids in the early days of the web were having by hacking websites to make it say something funny - just less harmful because no one else sees it.