The tasks: scrape all the selko articles and then generate fi-> en translation tasks in the form of phrases (using an LLM). Same thing except en->fi. Free form task generation where I tend to use HS for articles (so like, adult Finnish). And then one I'm working on now, Finnish transcription from auto-generated speech, from selko again.
A lot of my evaluation is based on embedding + cosine distance. LLM's are truly bringing a golden age for language learning.
I was planning on doing a similar thing but my wife didn't seem terribly enthused by the current SOTA with Finnish text generation. Sometimes I do ask GPT-4 to "kerro minulle jotain kiinostavaa [aiheesta]" and throw the generated longform text into a single Anki card to read later on. The style definitely feels different, and it hallucinates a lot, but hey, it's still net beneficial on the margin.
Let's keep in touch, it's always good to meet fellow Finnward folk. My email is in my bio if you ever need anything.
I'm pretty happy with translation, even with GPT-3.5. I haven't used it for native text generation. Happy to keep in touch :D.