pull down to refresh
Bigger argument for local is not leaking data, if hardware gets reasonable again that'll favor open-weights again.
Yep. You're right about cost. Hard to justify just on basis of token pricing.
I got my rig (AMD AI 390) mid-2025 when prices were reasonable, so $1900 for 96GB RAM, don't think its worth it at double or triple the price.
But like you said, the big benefit is privacy. I've moved my accounting setup to be all "beancount" based (python text based accounting program).
So I can drop CSV files from my bank into my local LLM and have it transpose all the entries into beancount format.
Absolutely you can do it programmatically - and that is in fact what my LLM does for me: (a) when new CSV format is encountered it writes a beancount importer to handle that specific file format, and (b) intelligently understands the descriptions to categorize the entry and update that rule in the categorize.py script if it doesn't exist (so PHO/GRILL it understands is a restaurant, etc)
Those are generally one-time task. However the real use is being able to chat with the LLM about hypotheticals....ways to become more tax-efficient, etc.
I was leaning that way too but if the token costs keeps coming down such as they have I'm not so sure. Gork 4.6 is literally the cheapest, bigger issue is I can't just easily plug a Cursor API key in everywhere I would an OAI compatible key.
Also keep flip flopping on buying hardware, first the local models were too shit to justify it, now that they're good the hardware is still expensive but the closed models are getting cheap.
Would need a $7500 box to run GLM 5.3-Flash correctly, and can't reasonably expect to amortize that over years because who knows what the next gen will need.
Bigger argument for local is not leaking data, if hardware gets reasonable again that'll favor open-weights again.