pull down to refresh
Wanted something really lightweight to play around with for a 16gb machine.
Anything heavier you recommend?
Turns out it was much easier to set up than I had anticipated, so eager to try a few other ones.
And yes, I can feel a world of difference using such an old model as I got used to the proprietary flagship models. Feels almost retro, even though it's only been 2 years.
Ollama has these in their library:
gemma4:e4b(gemma4:12bcould work but e4b works better on my 16GB macbook than any 12b ones)qwen3.5:9b
Reporting back, using qwen3.5-9b locally, this has been a pretty useful experience. I've used it for all these little questions you want to ask an LLM rather than googling them, but don't want to have a record of them online. E.g., medical questions, etc.
There is a sense of relief that starting last week, ChatGPT does not know as much as it used to about me. Now I need to convince/train my wife, who routinely uses it for personal advice. And I need to get to a larger model/better hardware so I can also start switching away from the proprietary models for my work-related stuff. Also, deleting old records, but that's just to make me feel good. They probably already recorded it in several locations.
Small steps.
That's awesome progress!
If you want to share the interface with your family, you could opt to get a Mac Studio rather than a macbook, and serve the local model through a web interface with either llama.cpp serve or vLLM Playground over LAN/tailscale if you wanna use it outside?
Hardware is crazy expensive right now though.
Neat. Will report back after trying.
qwen2.5:14bis a valid answer! I wonder one thing tho: why Qwen 2.5? It's from September 2024.