Our Services consisted of 4 services running on one physical server which had 32 CPUs.
The largest cloud server at the time was 8 CPU, so there was only one option:
32 divided by 8 = 4, so lets move to 4 cloud servers!
Services had followed the Developer Principles and was Event Driven.
Completed builds would fire an event in the API Service to trigger downstream Deliveries, as illustrated below.
API going green indicates a Build Completed Event which triggers a Delivery on the Delivery Service.
Lets see how that scaled when we started multiple instances of the services:
A Build Completed Event fired by API is received by all 4 Delivery services, which all start to process the same delivery, and they get into an error state!
The Delivery service had another existing problem, triggered deliveries would be stored in memory until they were processed. In the event that the Delivery service went down or had to be restarted, all these delivery requests were lost!!
To address both of these problems we added a Persistent Queue backed by our Mongo DB:
Now a Build Completed Event on API Service will add a Delivery request to our queue, which will then get picked up by ONLY ONE Delivery service, which could be running on ANY of our servers.
Delivery services can now be restarted without losing deliveries or causing downtime.
Scheduled Batch Jobs
We have also added some nightly batch processing jobs to our Services. These are defined in our application and uses the Mongo DB to control scheduling and locking.
They will run automatically on any Server running the APIs service, which means they are not tied to one Server. This is important for redundancy and COB.
All servers are created equal, and they are equally disposable!
There are no special master servers, killing any server should have the same minimal impact!
