NGINX doesn’t forget!

As part of our migration from VMs to OpenShift, I added an NGINX reverse proxy in front of our services, also running in OpenShift. This provided us with the ability to redirect some of our traffic to new services running on OpenShift as and when they are migrated and available.

NGINX Basic config looked something like this:

    location /api {  
        proxy_pass   {{ .Env.API_ENDPOINT }} ;  
    }  

So, any traffic coming into /api would be reverse proxied to what was defined in API_ENDPOINT environment variable.
For some services this was set to an OpenShift service name, for others it was set to our legacy infrastructure.
Our services running on VMs are all running behind a WIP load balancer, so if the service goes down on one VM it is removed from the load balancer, or we could switch to COB servers transparently.

However, after adding NGINX into the mix, if one service went down the clients started getting Bad Gateway errors.
The reason for this is that NGINX will resolve domain names to IPs while loading the configuration, and cache them until you restart/reload it.

The way our load balancer works is DNS Round Robin, so each time we try and resolve the IP of our service, it will return one of the servers ip addresses for the client to connect to.

So it looked like NGINX was starting, resolving the ip address of our load balancer to one of our backed servers and always used that.

This also means that with 3 pods running NGINX in ECS, each resolving to a different server – but only using 3 out of the 5 Prod servers behind the load balancer.

To address this we added a variable as this forces NGINX to resolve the variable and not use a cached value.
For this to work we had to add a resolver to NGINX configuration.  By default the resolver will cache the result as per the TTL, but there are options to overwrite this if required.

server {  
    resolver {{ .Env.DNS_SERVER }};  
    set $api_endpoint {{ .Env.API_ENDPOINT }};  
  
    location /api {  
        proxy_pass    $api_endpoint;  
    }  

To set the DNS_SERVER at container startup we add the following to our startup script:

export DNS_SERVER=$(cat /etc/resolv.conf |grep -i '^nameserver'|head -n1|cut -d ' ' -f2)  

Please let me know if you have a better way to do this.