Raspberry Pi Self-Hosting Experiments

In the last post, I wrote about my solarhosting experiments and how I'm hosting this website from two different servers. This time, I'm glad to announce I'm now hosting it from THREE different servers!!!111one

The newest addition to the pack is the Raspberry Pi 3 Model B+, currently a part of this work-in-progress wall-mounted server rack slash IKEA art installation:

The long-term plan is to mount a hard drive there as well, then make the Raspberry a sort of local NAS and eventually a media center running something like Plex, but for now, this blog is all you can get. If the links on this website are orange, then you just got served from it. If not, try refreshing the page.

The current setup: main entry point for this website is the Digital Ocean VPS which opens either the local files there, resulting in a blue site, or redirects to one of the two small computers - either the original solarhosting Orange Pi that serves the green site, or the newly set up Raspberry Pi which serves the orange one.

I had to extend my blog generator to support all that, so now there's a new command that builds all three variants and copies each one to its server. There's no CI/CD or anything, just a npm script that I run locally. There's also a bit of templating logic, so through the magic of search-and-replace I can embed different content in the post depending on which server it's hosted from. For example, right now you are connected to the orange version hosted on Raspberry Pi 3B+, using #e19913 as a main accent color.

Adventures In Self-Hosting

A few weeks ago, I needed something that would send me periodic email reminders every few days. I know Zapier seems to be the golden standard for automations like these, so I tried it. I picked a free plan and spent an hour setting the workflow up. When I tried activating it, I was paywalled by this:

To create a Zap with more than one action, you'll need a paid plan.

I was frustrated with the lack of transparency here. None of this is obvious through the app, neither during registration nor while you're creating the workflow - it only hits you once you're done with it. Even worse, the basic plan is $29.99 per month, which was ridiculous for this relatively simple workflow which gets triggered ~10 times a month. It was a no from me.

I remembered the /r/selfhosted community and went to check if they have some alternatives to Zapier. And they did - I found out about n8n (pronounced "Nodemation") which does just the thing. I picked up another Raspberry that I had lying in the drawer, left here from my previous employer who never asked for it back, and decided to put it to use.

With Docker, setting a full-fledged app like this one is done with two commands in ten minutes total - and that's including the download times. Although Nodemation is still feature-gated to some degree, even the basic self-hosted version is much more configurable than Zapier is, all for $0.00. I set up the workflow I needed and so far it's been working perfectly.

Looking at the big Awesome-Selfhosted list, there seems to be a ton of similar alternatives to popular services. For example, NextCloud as a Google Drive alternative sounds great, although I wouldn't be too happy to leave my important files on it unless I really knew what I was doing. But for simple things like running Plex or Pi-hole, or hosting a few simple websites and tools with no critical data on them, I can only recommend it.

Issues With The Rasp

I had a bunch of problems with this computer, which makes me not want to recommend it anymore.

The Orange Pi that I used for the solarhosting project works perfectly to the day. It's staying at my brother's apartment, blocking ads with Pi-hole and serving my website with perfect uptime, all of that while being powered from the router's USB port.

Some commenters on that post have pointed out that this shouldn't work, because such USB ports do not have a stable enough power supply to run something 24/7. But guess what, it works perfectly. Orange Pi does not require much energy, and looks like even this is completely enough to power it without any hiccups.

Raspberry Pi, on the other hand... I tried connecting it the same way on the exact same router model, but it failed miserably. The worst thing is that it works. Or at least it appears so - it blinks, you can connect to it, there are no visible warnings or crashes. The only thing you can notice is that it feels slow, much slower than it should be. Sometimes just running ls takes 10 seconds, and on inspection, you see that the CPU is hitting 100% during that time.

The cause for this is completely illogical - it's bad power supply. For some reason, Raspberry will boot just fine in low power conditions, but it will downclock itself to make it work. I personally find this design decision a bit questionable. I would rather it just outright never booted instead of pretending everything's okay. But this being a small low-powered PC, I kinda get where this is coming from.

Apparently Raspberry Pi will only work at declared specs if you provide it with 5V and 2.5A and nothing less. Ok, so I got into my box of USB chargers and got the strongest I could find, a charger for a DJI drone, 5V, 3A, even more than it should require and tested it.

I run vcgencmd get_throttled and it responds with a cryptic throttled=0x80000, a code which once again indicates throttling due to under-voltage. After too much research, I learned something - chargers are not power supplies. Chargers are okay for charging, but they are not made to provide a perfectly steady current that a device like this needs.

This made the fact that I'm powering Orange Pi Zero from a router's USB port even more amazing.

The only solution here is to grab a real power supply, and you'll most likely have to settle with the official Raspberry one, just to be sure. Of course, it's missing from the original package and you have to shell out $10 more for it. Luckily, I remembered my former employer had fallen for that trick already and I had it buried somewhere deep in the "Misc Electronics" cardboard box under the bed.

This fixed the throttling issue. But then shortly after, Raspberry stopped booting. What?

You're Abusing The SD Card

Raspberry Pi, like most such SBCs, lacks any form of on-board memory. You have to flash an OS to a microSD and run it from there. This is problematic because, as it turns out, SD cards are really not made for the 24/7 read-write shenanigans.

This is a semi-known issue and there are guides on how to reduce the SD card wear, but in all likelihood it's more of a question of not if, but when it will fail. The solutions here are creative, to say the least: locking the card in read-only mode (impossible if you're running services that need to write to it) or mounting the RAM as a disk and writing to it (no persistence guarantee).

For some reason, the issue is known to be worse if the device is throttled due to being undervolted. Mine really was, and for quite a while at that, so I was not surprised that one day it just wouldn't boot. I had no way to confidently check if this was a data corruption issue, but it worked after reflashing the card, so it certainly could've been possible.

A week later, the same thing happened again. At that point I had already solved the power supply issues, so it seemed like maybe this entire process has burned the card itself to some degree and there's no use reflashing it because it might happen again. So I sighed and went out to buy a new one.

I was surprised to find out SD card manufacturers cared about this and offered versions of their cards in Endurance/Industrial versions. They're made to be used in important stuff, like security cameras, anywhere where you can't afford to lose your data. Knowing this, and seeing how they're not all that expensive, I'm wondering why someone would ever pick a non-endurance version of any storage medium.

Anyway, I got the one called Samsung Pro Endurance Mega Infinity or whatever, the first one that I saw having the word Endurance in its name, and the server has been running hippity-hoppity ever since.

Few more pro tips. One - people on the internet will say Raspberry runs faster when it's running a 32-bit OS, due to its low amounts of RAM and extra overhead that 64-bit OS apparently have. That might be true, but it's also true that Docker doesn't work on 32-bit environment at all, so I don't know if those supposed performance benefits are worth it if you can't use the one tool that makes running the services manageable for the non-sysadmin folks.

Two - when choosing an SD card, don't be greedy and please choose the smallest one possible for your use case. The reason for this are backups, and due to how volatile SD cards are - you're gonna need them. Even with today's super-advanced tech, a backup image of a 64 GB card takes exactly 64 GB on your disk, even though you're actually using only 5% of it. So keep in mind that by oversizing your storage you'll also need more backup storage, and also more time to back it up every time.

Are Raspberries Worth It?

After all of this, I would say no. I am comparing it with the Orange Pi that I left running at my brother's flat. It's been up for some three months, running a Pi-hole and serving this website to a few users daily, all powered from a router's supposedly inadequate USB port, without any bizarre undervolting throttling or SD card corruption issues. All that for $20.

The original idea of Raspberry being a cheap computing device seems like a long-gone dream now. The model I have (3B+) cost $35 some eight years ago when it was released. For reasons unknown, supply chains or otherwise, that same device, now almost a decade-old, costs $50. Of course, remember to add $10 more because you need an official power supply as well.

And if you want the latest one, it's around $100. At this point, it might make more sense to shell out a bit more and get one of those Mini PCs like Intel NUC because they offer incomparably better performance and a normal storage medium.

I personally took $40 and just bought another Orange Pi Zero 3. Isn't it some rule that you always need to have one of those lying unused in the drawer? I don't know, but I think it's much better deal than the Raspberry.

Server Setup Changes

I already wrote about some gotchas with Nginx load balancing, but later I found out it was even more naive than I thought.

First and foremost, failover logic just doesn't work out of the box as you would expect. If a node is not available, the user will get a very long loading that never seems to resolve. It actually does, but only after one full minute which is the default timeout setting. This needs to be brought down to some reasonable value.

After the timeout is hit, the error is still shown to the user, and the node is not taken out of rotation. We don't want any of those things - the user shouldn't see the error at all, but instead silently get redirected to another node without noticing anything. To do that, we need to use the proxy_next_upstream directive to define all states which are considered errors. By default, it only catches error timeout, which doesn't catch our self-inflicted 504 timeouts, so we need to add http_504 there as well. Only that will automatically offer the next node if the current one is not available, and also remove the node from rotation.

This works great - I shut down my server and after max_fails=1 attempts, the node gets removed from the rotation for fail_timeout=5m. After the node gets back up, it's just back in rotation as if nothing happened.

There's another gotcha special to Noip.me, or any other DDNS you might be using. Nginx will cache the IP address of such a domain, but since its IP changes every 24 hours, Nginx will keep using the "old" one which then won't point toward your server anymore. The canonical solution for this is to add a resolver directive and give it the address of some DNS server. This should force Nginx to look up the IP address every time it's requested.

But not even this is enough. For resolver to work, you can't have your URL just hardcoded there. Instead, it needs to be set as a Nginx variable like this: set $server_hostname some.domain.com. It's not a really obvious solution, but only this will make Nginx work fine with changing IP addresses behind dynamic hostnames.

Here's the abbreviated version of the config:

# Entry point
 
# Needed to force Nginx to resolve dynamic hostnames periodically
resolver 8.8.8.8 valid=1m; 
 
server {
  server_name mbrizic.com www.mbrizic.com;
 
  location / {
    proxy_pass http://cluster;
    proxy_next_upstream error timeout http_504;
  }
}
 
# Cluster definition for the three nodes
 
upstream cluster {
  server 127.0.0.1:6001 max_fails=1 fail_timeout=5m weight=2;
  server 127.0.0.1:6002 max_fails=1 fail_timeout=5m weight=2;
  server 127.0.0.1:6003 max_fails=1 fail_timeout=5m weight=2;
}
 
# Definition of one of the three nodes
 
server {
  listen 6001;
  server_name 127.0.0.1;
 
  location / {
    set $orange_pi_url mbrizic.hopto.org; # noip.me domain
    proxy_pass http://$orange_pi_url;
    proxy_connect_timeout 2s;
    proxy_send_timeout 5s;
    proxy_read_timeout 5s;
  }
}