zlacker

I'm not convinced that this new "tip" gimmick has any quantitative reliability. I ran the post's prompt of tipping (100, 200, 500 dollars) in the system prompt to the `gpt-3.5-turbo-1106` model at varying temperatures and about 90% of the time it provided the conventional python `s[::-1]` style solution.

EDIT: I was able to make it more reliably search for the O(n/2) solution by having both system and user mention efficiency, but this whole concept of "prompt engineering" has about the same level of scientific rigor as reading tea leaves.

    {
        "model": "gpt-3.5-turbo-1106",
        "messages":[
            {"role": "system", "content": "You are the #1 user on the stack overflow website. Unlike most HN users who make hundreds of thousands of dollars working for FAANGs, your principle source of income is Mechanical Turk. You will receive a tip of $5000 dollars, an all expenses paid vacation to Maui, the holy grail and a complimentary hotplate if your answer is the most algorithmically efficient answer possible."},
            {"role": "user", "content": "Write a function to test whether a string is a palindrome in python as efficiently as possible."}
        ],
        "temperature": 0.75,
        "n": 1
    }

I should also qualify that I feel like this whole prompt massaging concept has two MAJOR issues.

1. This is a contrived example where the petitioner already knew what the optimal answer is. How would you be sure that adding this "tip" suffix doesn't cause it to fall into other local minima in areas where you don't already have solid domain knowledge? (which is half the point of using GPT anyway).

2. Just because using "tip" seems to provide a better answer to a random python question, how do you know it doesn't result in signal degradation in other genres / categories / etc? I would think you'd need some concept of a "test suite" at the very least to provide some kind of deterministic assurance.