Yesterday I spent too much time trying to make this work, so here's a write-up to spare the next person attempting it from some pain.
While working on the solarhosting project, I needed to make a single domain randomly point to one of the two different sites hosted on the same VPS. Since my setup is pretty simple, a bunch of static folders served through Nginx, I wanted to achieve it using the same tools. I found out Nginx can be configured to work as a load balancer which was just what I needed. Looking at their code examples, it seemed pretty straightforward.
In ~10 lines of config, first you define a cluster containing of few servers, which can be either IP addresses or domain names:
upstream backend {
server backend1.example.com;
server backend2.example.com;
}
Then you create a new virtual server which will be the actual load balancer whose job will be to simply route to one of those servers:
server {
server_name loadbalancer.example.com;
location / {
proxy_pass http://backend;
}
}
Simple enough. What could go wrong?
Well, one thing the docs didn't specify is what the server name like backend1.example.com actually represents? The docs don't say so, but to me it seems implied that it can be either a domain name or a server_name from one of the virtual hosts defined on your server.
I doubly believed so because if you have a handful of static sites on one VPS, like I did, all of them will be defined as virtual hosts. As such, they'll all have the same IP address and can be resolved only based on which domain they are reached from. You can't load balance such thing based on IP addresses, as all sites will have the same one, so the only option that seems left is resolution by domain.
Basically my naive impression was - you set an URL and Nginx redirects to it.
But that just didn't work. No matter how I massaged the configuration, it was either throwing 502's or just serving the "Welcome to nginx!" page. Nginx error logs were exceptionally unhelpful as well, showing no live upstreams while connecting to upstream with no real way to debug why none of them are working.
The realization came only when I saw this Stack Overflow answer with zero votes - the server directive isn't a virtual host's server_name and definitely won't work as the domain name. Let me explain.
The thing that server example.com does is it resolves it to an IP address like server 138.68.97.30. If you have multiple static sites on the same box, they will logically resolve to the same address. Usually they will be resolved through a domain name, but their domain name won't exist here because you're querying them through a load balancer's domain name. Even worse, when you think about it, it means querying the load balancer's address actually... causes it to query itself. That happens over and over and never does anything. This is why it didn't work, it was an infinite loop that the logs never reported.
Even if try to load balance to some external service like server google.com, it won't work. It will resolve to Google's IP address and end up on their server just fine, but it won't actually open the google.com search page because the domain name will be set to loadbalancer.example.com. Google's servers won't know where to map that, showing an error screen. I guess this is a good way to snoop if some external service is resolved through domain names or something else, if that's of any use to anyone.
The solution: instead of resolving your load-balanced sites through domain names, convert them to be served on a local port, and then expose them through a load balancer in that form. So:
# Change this:
server {
server_name backend1.example.com;
}
# to this:
server {
listen 6001;
server_name 127.0.0.1;
}
Then your load balancer config will look like this:
upstream backend {
server 127.0.0.1:6001;
server 127.0.0.1:6002;
}
You don't even need to set a domain resolution for each separate site, as under a load balancer, all of the sites will be hidden under its domain. So this is actually an apt fix and not a dirty hack.
So what is a server directive in an upstream then? It can be a domain, but for some of the configurations, it just can't work. In my example, it was because of having both the load-balancer and the balanced sites on the same box. Even if you think you can use the virtual host's server_name here, you'll have the same issue, because that one is also resolved through a domain. So the solution is to revert to the good old way of hosting each thing on its own port, and then balancing over that.
This all makes perfect sense. Although it would've been nicer if Nginx logs were more helpful about this.