Pre-configured machines from most companies I found online were either 2x the cost of parts and/or with inferior critical components that would have slowed machine learning performance and increased cost to run. (This article is a work in progress)
I ended up testing very assumption in this article with benchmarks - public or self-generated. If the goal was to prototype with GPT2 and other data, I need space and fast processing speed. The Cifar10 example on a 2013 13in maxed-out Macbook Pro with integrated graphics ran at about 250 calcs per second on the CPU. A 15in maxed-out MacBook Pro reached 500. Running the same code on the GPU of a Vast.ai rented 2080Ti machine reached 6000 calcs per second, while a dual 2080Ti did the same at around 8000 (apparently splitting calculation between bridged NVIDIA 2080Ti cards is not as efficient as a solo card). Finally when I configured and built the machine described below with machine learning considered in every detail the same Cifar10 training benchmark ran at whopping 13,000-16,000 calculations per second! The desktop cost around $4600 and has capacity for 3 more GPUs while it runs cool, quiet, and with less power demands due to efficient case airing. Going with aws or gcp would be harder for me when my 2013 MBP had only 200gb of hard drive space left, which was insufficient for the datasets required to train. This desktop is designed for prototyping of deep learning and NLP projects that can then be fully trained in the cloud later when needed.
The final configuration ended up being:
PROCESSOR: AMD THREADRIPPER 2095
GPU: EVGA NVIDIA 2080ti
MOTHERBOARD: AORUS EXTREME (built in 10gb ethernet handy for an external storage array)
HARD DRIVE: 1tb m.2. SAMSUNG EVO
STORAGE: 6tb HDD @ 7200RPM
MEMORY: 2x 16Gb CARDS AT FREQUENCY 2933, which is best by spec for rhe peocessor. Overclocking can cause glitches and that's the last thing needed with a long running deep leadning training excercise. More cards could be added later.
CASE: be quiet 900 - excellent air throughout and also quiet. The large case size helps reduce fan power needs and long term cost to run experiments.
(this page is a work in progress)