zlacker

>>Josely+(OP)
Haha of course this news comes just after I wrote a parser for my ChatGPT dump and generate offline embeddings for it with Phi 2 to help generate conversation metadata.

>>luke-s+it
so far you can't search your whole conversation history, so your tool is relevant for a few more weeks. is it open source?

>>singul+Yx
I'll share the core bit that took a while to figure out the right format, my main script is a hot mess using embeddings with SentenceTransformer, so I won't share that yet. E.g: last night I did a PR for llama-cpp-python that shows how Phi might be used with JSON only for the author to write almost exactly the same code at pretty much the same time. https://github.com/abetlen/llama-cpp-python/pull/1184 But you can see how that might work. Here is the core parser code: https://gist.github.com/lukestanley/eb1037478b1129a5ca0560ee...

>>luke-s+e41
The ChatGPT dump format is not intuitive so I used a tree search algo to print it up to a defined depth level then gave ChatGPT 4 the extract and you it what the expected output parts were.

>>luke-s+2E6
* and told it what the expected output parts were.