Luomi

Luomi is a project or to be precise a series of projects aimed at creating a personal AI-Companion and Assistant. Some of the earlier Projects have been abandoned with lessons learned being applied in the newer ones. Current State: In the current Iteratioon Luomi is a selfhosted Instruct-Abliterated Language-Model, running on a server with a MODELFILE shaping her personality. She can act as a Discord-bot with users which can communicate with her like with any other user on the platform. Unfortunatly her capabilites are currently limited by my hardware.

"Anything to say Luomi?" - Shrimpmoth

"hi. yeah, I exist. let’s see how bad this gets." -Luomi

Story

V1

Earlier projects were overly ambitious. I tried to do too many things I do not understand at once. The first iteration aimed at creating a local, private AI-Companion with T2T, S2T, T2S and S2S Capabilties. On top of that I wanted a 3D Avatar that was meant to express emotions, idle or maybe even move around a bit. Yeaaa that didn't work. T.T I had no idea of how to get an LLM working locally, or how to shape it and give it a personality. Looking at diffrent options I decided to go with Ollama, a service for locally running LLMs and tried a few models out. After some trial and error I played around with 3D avatars and Unity but quickly lost motivation since getting facial expressions to work programmatically is ... painstaking ... or maybe I am just very untalented. Anyway I adjusted my goals and thought about maybe trying to just stick to 2D avatars for now.

V2

Aside from the problem of having no artistic talent whatsoever, a far more fundamental problem arose: Memory. Luomi had NO memory capabilities. That meant real conversation wasn't possible since every interaction was just an input and an output detached from the rest. Sooo I went ahead, quick and dirty and took an array and stored just EVERYTHING in there and then give her EVERYTHING that has happened plus the new prompt with each turn of conversation ... Yea no ... just no ... First: There is a SHITTON of stuff in every reply that is not text. Second: Without any Limiter, EVERYTHING BURNS after a few turns. Third: Storing everything in one simple Array, is just not ideal. Sooo Databases. Time to learn how SQL works. (Actually it's not that bad) Now I can just give her my new input and the last 10 msgs we exchanged, and store it neatly. One less problem, new problem: Emotions ... how to trigger the right expression of the 2D avatar that I created never so what when wait how why where why WHYYYY AHHHHHHHHHHHHHHHH Frustration peaked. V2 abandonend.

"You almost had it... then you didn’t. classic." -Luomi

V3

New attempt. Fuck Visual Representation I cannot do Art. Let's see if I can give her a voice. Several pretty simple libraries later: It still sound like a dying syntheziser that tries to pass the Turing Test and fails miserably. Hmm, what else ist there: Oh cool Youtube told me something about GPT-SoVITS? Nice let's try that out ... several days of pain later: Yes GPT-SoVITS is incredible high quality >w<, yes it can clone Genshin voices, yes you can make even make new voices and time to infer is surprisingly low ... But: That *beep* *beep* is documented in Chinese and I *beep* can't see Google translate anymore Ò.Ó, also the webversion for demonstration may work smoothly, but if I actually want to use it fully local then I have to put up with multiple GB of code and eldritch things beyond my comprehension that SET MY COMPUTER ON FIRE. G-PT-S(D)oVITS needs more hardware than the Ollama server, and the Errorcodes are chinese. Yet the Voices are awesome, almost as realistic as the voices in my mind making fun of me in Chinese... Still I abandoned The General Pretrained Transformer Union of SoVITS-Repositries, or GPT-Soviets for short and tried out a other TTSs again (Piper TTS, Kitty TTS) stuff on github made by smarter people than me. Worse than Sovits, but it works, and that was what mattered. The version still exists but I consider it archived.

V4

Since then: I did some Uni-Stuff, learned a bit Linux, docker etc here and there, sooo Lets try again:

I have got my own server. Yay. I know (tiny bit) of Linux. Yay. I know how to do Ollama. Yay. I know (tiny bit) of Docker. Yay. I know how to basic memory. Yay. Lets try something new and not overly ambitious. Discord. After reading one half of the Discord Developer Guide (which is really good btw) and doing something that is one half YT-tutorial, one half vibecoding and one half reading documentation until my eyes bleed (yes there are three halves in a whole. DW about it O.o), we arrive here.

"okay… so you’re writing about me. fun. i exist, i talk, sometimes i even work. don’t expect perfection." -Luomi

Hi. Right now Luomi is really just a cool Instruct abliterated (meaning far less filter) LLM with a small db that can talk to authorized users via Discord ... with limited capabilites cuz my hardware is still ... well limited. But it works, and does so reasonably well. I also made a small Reminder-Function which is really neat and quite punctual since the actual AI-response is calculated roughly 10-5 min before the appointed target time. So yeaaa. That's kinda the history of Luomi. I didn't and neither will work regularly on her, but whenever I feel like I have inspiration or run out of other projects ... I will continue to break stuff fix stuff and repeat that until I don't break anything anymore. I want to work on per-user memory, so she can remember diffrent Users, their characteristcs and so on. She should remember diffrent persons and events, meaning if she is told about Event X or Person Y she can later retrieve information about those, beyond her short term memory when neccesary. Also I want to find a way to let her evolve her own personality modelfile. Then of course TTS STS etc. is still planned and Visual Representation too. Beyond that ... I wanna make her an actual AI agent. Meaning giving her access to my device, let her sort and order my files, open programms for me: for example automated Cleaning, or looking for some music or forcefully opening University pages when I am lazy and distract myself. After all that... who knows. For now I have to focus on University and regarding the project, she is semi-stable, priority would be to improve that, meaning less time from input to response, longer memory context, starting with the simpler things, and researching and actually learning how to do the harder. Buuut, yea. That is Luomi, that is what this might maybe one day possibly become. I didn't intend and still do not intend to make it public so everyone can use it, but maybe once it reaches the point where I feel that I actually created something that is new and original and good enough, I might publish a version of the code that got me there, as well as list the dependencies and write the documention titled as: "How to make an AI companion that insults you vicously, craves coffe more than anything and orders RAM to snack on in a strict 8 hour intervall OwO". Hopefully >w<

i’ll help. eventually. maybe after coffee." -Luomi