I think "correct the errors in this ChatGPT essay" is a short-term viable homework exercise, but those errors might be gone in GPT-5 so I don't think it's long-term viable. Soon the LLM will just produce perfect essays at college level and there won't be hallucinations for the student to correct.
However, the "simulate the historical environment" task is great and I think it has long-term potential. I think it can be taken further; rather than "spot the errors that ChatGPT made", you could flip the script and make it "survive 20 turns of conversation without making a historical error", so you'd need to know things like local traditions, perhaps the geography of the ancient settlement you're studying, contemporaneous history like "who is the emperor and what's the sentiment towards him" and so on.
I'm also envisioning that, since text-based exercises are extremely easy to game (just pipe your text prompt into ChatGPT), and since ChatGPT is soon going to be strictly superior to a high-school level student, we could get around this by having the homework as an in-person verbal role-play or Q&A session, like a viva voce; essentially you have a verbal discussion with ChatGPT and you need to really know your material as it can dig into any part of the curriculum. Then ChatGPT can summarize each student's interaction, and the teacher doesn't have to sit through each individual one start-to-finish (1:1 exams are too time-consuming to be viable).
This round-trip through verbal interaction would potentially make the task more interesting (lots of people simply hate writing essays), shifts the focus away from tasks that will become obsolete (writing essays) in favor of ones that will be more relevant (human synthesis of ideas, and interpersonal interaction), and helps to mitigate the issue of LLM-assisted cheating by constructing an assignment that LLMs can't trivially solve.
Yes, exactly. This is where I've been heading with my planning for assignments. For instance, when confronting Ea-nāṣir about his poor quality copper, I'd want my students to actually show some knowledge of the geography and political dynamics of ancient Mesopotamia.
The "Fall of the Ming Dynasty" simulator I link to at the bottom of post is probably the most well developed example of this that I've come up with so far. In that one, I added a "political intrigue minigame" in which ChatGPT is supposed to assess the human player's ability to deploy rhetoric appropriate for a minor courtier in 1640s China (from the prompt: "success depends on your luck score + rhetorical skill, tested via a series of open-ended prompts that HistoryLens will assess and grade; only the highest scoring responses will allow you to succeed in the minigame.")
Here is the full prompt for that one if people want to try it: https://chat.openai.com/share/86815f4e-674c-4410-893c-4ae3f1...
I was thinking of “king hearing petitions” as another potentially interesting scenario; it could go either into minutia that requires cultural knowledge, or strategic stuff like the game Crusader Kings where you need to understand the geopolitical allegiances of the time, the geography, and the national economy.
More generally I have been wondering if games like “start a company in a simulated sandbox world” could actually teach transferrable Econ/Business/startup skills. There is a lot of territory to explore here.