pull down to refresh

122 sats \ 5 replies \ @magnolia_mayhem 16 Oct \ on: Local models.. lived experience AI

Things keep crashing with me. Hermes init with Gemma4:12b loads and then throws a

⚠️  API call failed (attempt 1/3): InternalServerError [HTTP 500]
   🔌 Provider: custom  Model: gemma4:12b
   🌐 Endpoint: http://127.0.0.1:11434/v1
   📝 Error: HTTP 500: an error was encountered while running the model: error: task id = 0, error: decode() failed: vk::Queue::submit: ErrorDeviceLost
   ⏱️  Elapsed: 70.35s  Context: 2 msgs, ~4,515 tokens
⏳ Retrying in 2.9s (attempt 1/3)...
⚠️ custom stream drop (EmptyStreamError) after 48.5s — reconnecting, retry 2/3
⚠️  API call failed (attempt 2/3): InternalServerError [HTTP 500]
   🔌 Provider: custom  Model: gemma4:12b
   🌐 Endpoint: http://127.0.0.1:11434/v1
   📝 Error: HTTP 500: Post "http://127.0.0.1:45355/tokenize": EOF
   ⏱️  Elapsed: 126.54s  Context: 2 msgs, ~4,515 tokens
⏳ Retrying in 5.8s (attempt 2/3)...
⚠️ custom stream drop (EmptyStreamError) after 40.3s — reconnecting, retry 2/3
⚠️  API call failed (attempt 3/3): InternalServerError [HTTP 500]
   🔌 Provider: custom  Model: gemma4:12b
   🌐 Endpoint: http://127.0.0.1:11434/v1
   📝 Error: HTTP 500: Post "http://127.0.0.1:45089/tokenize": EOF
   ⏱️  Elapsed: 178.05s  Context: 2 msgs, ~4,515 tokens
❌ API failed after 3 retries — HTTP 500: Post "http://127.0.0.1:45089/tokenize": EOF
   💀 Final error: HTTP 500: Post "http://127.0.0.1:45089/tokenize": EOF
 ─  ⚕ Hermes  ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

     API call failed after 3 retries: HTTP 500: Post "http://127.0.0.1:45089/tokenize": EOF``` 

And then I get mad and go eat some chocolate.
write
compose
107 sats \ 1 reply \ @optimism 16 Oct
11434

Tried serving it with llama.cpp and not ollama?

reply
150 sats \ 0 replies \ @magnolia_mayhem 17 Oct

I've considered it, but I only have a few minutes a day for tinkering and other things have taken presence.

reply
130 sats \ 2 replies \ @RDClark OP 16 Oct

When Hermes starts up, it sends a massive wall of instructions to the AI all at once (around 4,500 tokens). Your graphics card tries to process that entire chunk in one giant bite, takes too long, and your computer thinks the graphics card froze—so it resets it (ErrorDeviceLost).

Once that happens, the AI engine crashes in the background. Attempts 2 and 3 (EOF) only fail because Hermes is still overwhelmed

To fix it, you just need to tell Ollama to take smaller bites (num_batch) and give the model a larger memory window (num_ctx) so it doesn't choke on startup.

reply
25 sats \ 1 reply \ @magnolia_mayhem 17 Oct

Is this under ~/.ollama or some other location?

reply
5 sats \ 0 replies \ @RDClark OP 17 Oct

You actually don't need to go into ~/.ollama or hunt for any folders at all:

  • For Step 1: Just open your terminal and paste those commands right where it opens. Ollama automatically saves the new model inside ~/.ollama behind the scenes for you.
  • For Step 2: Hermes keeps its settings in its own folder (~/.hermes/config.yaml), or you can just type hermes model in the terminal to switch to gemma4-hermes:12b without editing any files manually.
reply