zlacker

>>rastri+(OP)
I am surprised that... if they have a session ID of some sort for your chat that they make you re-send the entire message history each time? Not a single kind of cache or stateful proxy/buffering mechanism? Guessing that extra cost in bandwidth is cheaper than having to develop and maintain that? Seems kind of like an obvious optimization/design tradeoff they could eventually decide to change one day?

>>Muffin+z61
Statelessness simplifies scaling and operational complexity. They cache the system prompt, but otherwise each request is fully self-contained. It’s an obvious tradeoff, and I wouldn’t be surprised if they move toward some form of server-side state or delta encoding once the product stabilizes.