Issue with Core Lightning on Umbrel: app not "Starting..."
59 sats \ 8 comments \ @a68dd96af9 19 Apr lightning
Hey there,
I've an umbrel node with CLN, and since few days is refusing to start. I've restarted the node multiple times but the Core Lightning app continue being in a "Starting..." state!
Anything I could try from the terminal with ssh commands?
write
preview
related posts
5 sats \ 1 reply \ @Wumbo 19 Apr
Is Bitcoin Core fully Synced?
reply
0 sats \ 0 replies \ @a68dd96af9 OP 19 Apr
Yes fully synced and running. Can I get the logs of core-lightning somehow?
I've v23.08-hotfix-1 running
reply
5 sats \ 1 reply \ @nym 19 Apr
Did you upgrade to Bitcoin Core 27?
reply
6 sats \ 0 replies \ @a68dd96af9 OP 19 Apr
No I still running umbrel 0.5.4 and no updates are available for Bitcoin core, currently on v26.0.0
reply
0 sats \ 0 replies \ @a68dd96af9 OP 20 Apr
SOLVED! Today the node just restarted working correctly 🤷 Don't ask me how!
reply
0 sats \ 2 replies \ @a68dd96af9 OP 19 Apr
this is the log from umbrel:
core-lightning

Attaching to core-lightning_tor_server_1, core-lightning_c-lightning-rest_1, core-lightning_app_proxy_1, core-lightning_app_1, core-lightning_lightningd_1, core-lightning_tor_1
app_1               | 2024/04/19 16:28:29 socat[23994] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_1               | 
app_1               | Waiting for lightningd
app_1               | 2024/04/19 16:28:30 socat[23999] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_1               | 
app_1               | Waiting for lightningd
app_1               | 2024/04/19 16:28:30 socat[24004] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_1               | 
app_1               | Waiting for lightningd
app_1               | 2024/04/19 16:28:31 socat[24010] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
app_proxy_1         | yarn run v1.22.19
app_proxy_1         | $ node ./bin/www
app_proxy_1         | [HPM] Proxy created: /  -> http://10.21.21.94:2103
app_proxy_1         | Waiting for 10.21.21.94:2103 to open...
c-lightning-rest_1  |     at Socket.<anonymous> (/usr/src/app/lightning-client-js.js:80:23)
c-lightning-rest_1  |     at Socket.emit (node:events:513:28)
c-lightning-rest_1  |     at emitErrorNT (node:internal/streams/destroy:157:8)
c-lightning-rest_1  |     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
c-lightning-rest_1  |     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
c-lightning-rest_1  |   errno: -111,
c-lightning-rest_1  |   code: 'ECONNREFUSED',
c-lightning-rest_1  |   syscall: 'connect',
c-lightning-rest_1  |   address: '/root/.lightning/bitcoin/lightning-rpc'
c-lightning-rest_1  | }
lightningd_1        | Core-Lightning starting
lightningd_1        | 2024-04-18T19:04:22.308Z UNUSUAL plugin-clnrest.py: Killing plugin: timed out before replying to getmanifest
lightningd_1        | 2024-04-18T19:04:22.308Z **BROKEN** plugin-clnrest.py: Plugin marked as important, shutting down lightningd!
tor_1               | Apr 19 07:03:47.000 [notice] While not bootstrapping, fetched this many bytes: 112505 (consensus network-status fetch); 18363 (authority cert fetch); 193674 (microdescriptor fetch)
tor_1               | Apr 19 07:59:14.000 [notice] No circuits are opened. Relaxed timeout for circuit 725 (a Measuring circuit timeout 4-hop circuit in state doing handshakes with channel state open) to 60264ms. However, it appears the circuit has timed out anyway. [4 similar message(s) suppressed in last 6000 seconds]
tor_1               | Apr 19 13:03:47.000 [notice] Heartbeat: Tor's uptime is 18:00 hours, with 9 circuits open. I've sent 11.13 MB and received 21.83 MB. I've received 0 connections on IPv4 and 0 on IPv6. I've made 25 connections with IPv4 and 0 with IPv6.
tor_1               | Apr 19 13:03:47.000 [notice] While bootstrapping, fetched this many bytes: 679228 (consensus network-status fetch); 9294 (authority cert fetch); 12000310 (microdescriptor fetch)
tor_1               | Apr 19 13:03:47.000 [notice] While not bootstrapping, fetched this many bytes: 177763 (consensus network-status fetch); 31419 (authority cert fetch); 287566 (microdescriptor fetch)
tor_1               | Apr 19 13:28:26.000 [notice] No circuits are opened. Relaxed timeout for circuit 1027 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60264ms. However, it appears the circuit has timed out anyway.
tor_1               | Apr 19 14:59:33.000 [notice] No circuits are opened. Relaxed timeout for circuit 1103 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60264ms. However, it appears the circuit has timed out anyway. [1 similar message(s) suppressed in last 5520 seconds]
tor_1               | Apr 19 16:08:39.000 [notice] No circuits are opened. Relaxed timeout for circuit 1188 (a Hidden service: Uploading HS descriptor 4-hop circuit in state doing handshakes with channel state open) to 60264ms. However, it appears the circuit has timed out anyway. [1 similar message(s) suppressed in last 4200 seconds]
lightningd_1        | Can't recover from plugin failure, terminating.
reply
10 sats \ 1 reply \ @nikotsla 19 Apr
It's important to know what was the step you do to get to this state, do you remember anything unusual?
lightningd_1 | 2024-04-18T19:04:22.308Z **BROKEN** plugin-clnrest.py
I have no idea of Umbrel, but I will try to check one component at a time... according to that line, there is some issues with a CLN plugin, possible a connection issue. Is there any way to start the components one by one?
1- Start BTC core 2- Check all ports are listening 3- Start CLN
reply
0 sats \ 0 replies \ @a68dd96af9 OP 19 Apr
Thanks for the details, I was trying to find the exact code to restart specific apps by ssh but I could not find it. It must be something like:
  1. sudo ~umbrel/core-lightning stop
  2. sudo ~umbrel/core-lightning start
Can anyone confirm?
reply