Infrastructure setup and open-source scripts to train 70B model from bare metal
In this technical retrospective (2024), @thejash shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
This is such a valuable piece. I've learned so much reading it! And your open-source code is great as well.
Some open questions I have: 1) Why did you choose to setup your own cluster? How was the experience with your cloud partner regarding faulty machines / switches? 2) What were your considerations choosing the cluster architecture that have proven the most valuable ? (apart from the all2all comms) 3) Can you share a bit more about your logging infra apart from the fact that it was Loki based? 4) What necessitated the use of a local docker registry? did you use other images apart from nvidia-container-runtime?
Thanks!
I wonder if it's possible for a huge number of hobbyists to team up and train a model together in a distributed manner like seti@home or folding@home. Or does this kind of workload not really lend itself to that approach?
Those things were of course characterised by the ability to spread the work into pretty self-contained work packages. Not sure if that can be done with model training.
Honest question: why is there so much PC hardware in the mix here? Why don't we have PCI + infiniband backends with GPUs and a little tiny orchestrating ARM controller and just let them all coordinate with each other? Is it just "momentum" from previous designs and/or lack of "market" for specialized GPU controllers?
This was discussed on the Latent Space podcast a few days ago: https://www.latent.space/p/llm-training-2024 ↗
That was a good episode, worth listening to for hearing justifications behind some of these decisions.
This post focuses on one cluster that had 4,092 H100 GPUs spread across 511 computers, with eight GPUs to a computer
Am I right in understanding, that's over $100 Million worth of GPUs?
I wonder what/when/if any of this will be within the realms of an enthusiast with a gaming-pc budget.
In the span of a few months, with a small team of researchers and engineers, we trained a 70B parameter model from scratch on our own infrastructure that outperformed zero-shot GPT-4o on reasoning-related tasks. Using our cluster for high performance training meant that every component — InfiniBand, Ethernet, GPUs, and the nodes themselves — had to work perfectly. If even a single one of the over 12,000 connections was a little flaky, it could slow down the entire training run.
We're sharing open-source scripts and an end-to-end guide for infrastructure set-up that details the process of making everything work perfectly, and ensuring that it stays that way.
This is one of a three-part toolkit on training a 70b model from scratch. The other two sections focus on evaluations and CARBS, our hyperparameter optimizer; you can find them here: https://imbue.com/research/70b-intro/ ↗
Thoughts and questions welcome! :)