A story about a missing channel
Last week a friend opened a channel to my lightning node. Usually, this should be an easy process. This time, it was not. My Umbrel restarted at the same time as his open channel transaction was confirmed on-chain. When my node was back online, it did not show the open channel. On my friend’s node, there was no trace of the channel.
After some time, my friend discovered that his node reverted to a state 4h before the channel opening transaction. He found a hint in Ride The Lightning that the open channel transaction existed and already had many confirmations. But he had no option in the GUI to do anything with the channel.
I started looking for solutions on my end and found different options. Reindexing, setting up a new node, and recovering from a backup or using a lightning wallet like Blixt or BlueWallet. From these options, reindexing sounded like an easy step.
I sshed into my umbrel and executed:
sed -i "s/\[Application Options\]/\[Application Options\]\nreset-wallet-transactions=true/g;" ~/umbrel/app-data/lightning/data/lnd/lnd.conf
While reindexing, my Umbrel restarted. I did not notice. At some point, I went to the settings page of my Umbrel, searching for any hint on the progress, and saw that the uptime was too short for the time the Umbrel was already running. Docker ps seemed to confirm the restarting loop. So I had a look at the log files:
sudo ~/umbrel/scripts/app compose lightning logs --tail=100
From the log file, two things were clear: Lnd was in an endless restarting loop, and the following error message followed by a trace log was not a good thing.
panic: freepages: failed to get all reachable pages
I went back to search for possible fixes but didn’t get many results. I decided to follow the tips from Umbrels troubleshooting guide. Maybe it was something stupid that I was not seeing.
Simple Solutions that Work 99% of the Time
I flashed a new sd-card with the current version (same as I had installed anyways). I made sure to confirm that I had the right power supply attached. I double-checked all cables. Umbrel booted, but it ended in the same endless loop with the same error. So I decided to remove the “reset-wallet-transactions” option and rebooted. Again, the same result.
sed -i "s/reset-wallet-transactions=true//g;" ~/umbrel/app-data/lightning/data/lnd/lnd.conf
Setting up a new node is not a thing I wanted to do. I have this planned for summer as the last step of migrating the few remaining self-hosted services from my small Raspi farm to a more powerful mini computer. Performance and stability have been real issues with these Raspis. At this point, experimenting with Blixt sounded like the easier way to continue, but for some reason, I was still going through the search result as I found this tweet:
Introducing Automatic Channel Recovery in our Lightning Node app
Passphrase: check, up to date channel backup file: check. Now, this sounds like the easiest solution to my problem. I logged into my Umbrel and tried removing the lightning app. Of course, this would not work as I had 5 or 6 other lightning apps installed; after uninstalling them, removing the lightning node went without problems. I rebooted the Umbrel just in case and then reinstalled the lightning node. After entering the passphrase, the system prompted for a channel recovery file, but I could not upload or select a file. This was only possible after reindexing completed (around 12h later). I selected a backup file, and all previous channels were found and started to close as expected, but the missing channel was, well… still missing.
Back to square one. I revisited the original transaction, this time on mempool, and guess what I discovered: mempool has very nice little labels for channel openings or closings. And what else did I see there? You guessed it: my friend opened a channel to the wrong address. I double-checked on 1ml, and there was the channel in all its beauty.