The backend is written in Rust, and the frontend is written in TypeScript with SvelteKit. The API is based on gPRC, using Buf Connect on the TS side. On the storage side, it’s using Postgres + Qdrant (self-hosted on dedicated servers) + R2. The reason I am self-hosting on dedicated servers are mostly the high CPU and RAM requirements for the main project which would otherwise make it cost prohibitive.
For the LLM parts (which are central to the main project), I built my own language-agnostic framework and platform to facilitate building LLM powered services. This is based on some lessons learned while working on Google’s Bard and other LLM projects that preceded Bard (those were ultimatelly killed — long story :)).
The framework and platform are currently private, but I might open it up in the future. To give you a glimpse:
The main premise is that building apps that leverage LLMs should not be that different from building regular apps. The main difference is the need for configurability and observability.
Each component (e.g. LLM call, Vector DB lookup, etc.) is just a regular RPC method, with well defined schema for its input & output.
The platform allows you to configure each method, which just means specifying values for some subset of its inputs.
The platform also tracks and stores the inputs and outputs of all the RPC calls and sub-calls. This is invaluable during development to understand and debug the methods, and in production to collect data for evaluation & training.
Thanks to the well defined input & output schemas, all of this can be done universally, rather than creating a bespoke solution for each method. Here are some examples:
- The “LLM” building block: https://www.loom.com/share/bc6fa6b27298420c82fcacca5f84d096
- Higher level JSON → Format → LLM building block: https://www.loom.com/share/d09d38d2c316468fa9f38f5b386fc114
- Example a complex trace from a summarization method: https://screenbud.com/shot/61dc59a6-5c73-4610-8168-753b79062...
Example code for a simple retrieval augmented chat bot in TypeScript and Rust https://gist.github.com/Palmik/42940e3887d6446244f8b74ce0243... — there’s still a lot of room for polish and boilerplate removal, but it gets the job done already.
Happy to answer any questions.
