I've been busy living and just realized I've lost faith in most platforms I've used so far. I've been running Claude CLI on an isolated VM in Proxmox, tried the limits of running smaller local models on consumer grade hardware, tried out Rool and a few others. Got curious of Grok Bot but want to find a more sovereign approach, before suddenly finding myself stuck with a new subscription.
What are you guys using, if running agents locally or through API on local machine?
I use
llama_cppas a local model server andpi-agent(using this docker container as an inspiration) as the default framework for agentic stuff. For reproducible pipelines I often just write a python script (or if it is unimportant, have an LLM write the python script.)If you want to run at scale, you may want something like
vllmInteresting, thanks. Been wanting to try
llama_cppbut didn't have the hardware at the time to run any kind of productive models. Not sure I do now either, but they seem to compress better and better.What factors made you choose the approach you did?
I won't pretend I'm a dev, so maybe I could use some sort of framework to start with, unless the docker container you linked does just that?
What you described is exactly why I built https://nymbot.ai with an anonymous mode that allows sending your credits via Cashu’s blind vouchers to a throwaway keypair, so the messages can’t be linked from the purchased credits.
Thanks, I'll keep that in mind.
What did you use for translation of the site? Quite disorienting in swedish, but made more sense in english.
Thanks for letting me know about the Swedish language, and apologies for that. I used the IndicTrans2 model lol. Sad to hear it is not as good of a model as they say. Could you please help describe in more detail what made it disorienting? Your feedback is most appreciated