More than one session is too many session

I have been using my personal AI assistant for a few weeks and it is working really well, after I figured out which models to deploy, how to exactly words the various tools and forcing at least one tool call for most interactions. There are some smaller annoyances, but overall things are working mostly as expected.

To have a bit more of a personal touch and to make sure Endirillia is not just a brain dead chat bot, I added a user profile and facts. This part turned out to be surprisingly effective, way more than I expected. And it ensure that requests such as "when does G2 play the next time?" does not end in her asking "Which game?", because the user facts are queried for keywords and the top three results are injected into the prompt.

This happens automatically whenever the system does not have anything to do. A go-routine checks for new messages in the session table that are not marked as fact_processed and sends a batch request to the LLM to extract important facts. There is a maintenance go-routine running at 2am that goes through the whole table and compresses it, as well as invalidates old facts that changed. So if Endirillia once thought I prefer Aela as a companion in Skyrim but I correct her at 2:02am that I usually have Serana as a companion the Aela fact will be gone. (Serana best girl, fight me on that!)

Additionally there is a user profile which is injected to each request. It is generated after the facts maintenance routine is done and it creates a short profile about me. Not as detailed as facts, but a standard part of the prompt such as "The user is a software engineer that thinks all programming languages suck, but he mostly writes Python and go these days and he likes when JavaScript is badmouthed". The main system prompt is something along the lines of "make sure to respond in a way that matches the users preferences". (This is obviously made up and not that fine-grained, but it shows the idea.)

Sessions

You know the standard layout of basically any LLM based chat application. There is a main window that looks like a message window and a sidebar with previous chat sessions that can be continued at any time. A simple layout and it's very easy to manage context like this. It looks very familiar. Like iMessage or WhatsApp or any email app beside mutt.

I think in the context of coding agents this layout works very well. You might have different sessions for different features and instead of writing multiple markdown files to disk or re-iterating what is currently going on, you simply pull up the past session.

But is this actually a good idea for an assistant? Actually, I do not think so.

I mostly want a continuous conversation with the assistant and I am more likely to reference things that happened within the last 10 messages or 10 minutes then coming back a day later to continue a conversation. So why not design the whole system around the idea of a sliding window? Context will not get lost, as all messages are stored and for some facts are derived. A RRF process can pick up what is relevant and inject it in the prompt. This keeps the conversation "flowing" visually and is far more pleasant.

Sadly to get there I have to rebuild the persistence layer and UI, basically from the ground up. I can keep a few of the components, but most of it needs to go away. I might regret this and revert the changes at some point, but for now I will continue to work on this implementation and see if this makes Endirillia feel more like someone to talk to and less like an email program with advanced auto complete and web search. To be clear: The goal is not to pretend Endirillia is a person, sentient or any of the other delusional LLM / pseudo-AGI stuff people get so hung up on. I would rather describe it as rethinking the UX.

Process

The rewrite of Endirillia's chat interface and session management is underway. I should be able to finish it next week and get started on testing if the thesis actually holds up or if I am running into the completely wrong direction.

The calendar server is mostly spec complete and might not only confirm to the caldav RFCs but also WebDav! I got some neat ideas, so it made sense to implement the whole spec while I am at it. I am currently trying to get the caldav test suite to run properly in a CI workflow, which might be trickier than implementing the RFCs. I might add CardDav support as well as next step, which should round out the little service.