Last weekend we made a very significant, albeit transparent, change to the back end – artifacts are now archived to Object Storage instead of SAN on the master TeamCity node. There are two advantages to this:
- Object Storage costs about 1/10th of the ongoing price of replicated SAN
- There is also no up-front cost to reserve the space as there is with SAN, S3 cost it is purely consumption-based
- In the last week alone we reduced our SAN consumption by around 1.5TB, and expect to ultimately recoup about 25TB – about $100K/yr!
- Build agents write directly to the storage rather than uploading to the TC master node and having it written to SAN from there
- This removes the extra hop through the TC master node (faster)
- It also reduces the load on the TC master node (better performance for everyone)
YMMV, but we are seeing the “Publishing Artifacts” step reducted from minutes to seconds.
Spot-checks of a few projects show significant improvements in the archiving step at the end of a build!
This is the history for just the archiving step on one project which had very variable upload times – worst case was over 90 minutes! (nearly 6000s). Look at it since the 1st Feb weekend…it is not only much faster, more importantly it is now far more consistent.
Another sample build:
Before:
Publishing artifacts (3m:54s)
Publishing artifacts (1m:18s)
Publishing artifacts (2m:44s)
After:
Publishing artifacts (6s)
Publishing artifacts (11s)
Publishing artifacts (6s)
Publishing artifacts (9s)
Similarly for another build:
Before:
Publishing artifacts (14m:09s)
Publishing artifacts (9m:08s)
After
Publishing artifacts (2s)
Publishing artifacts (4s)
