Modernization Monday is a field guide to what it actually takes to modernize your infrastructure for AI, published every other Monday. Each installment takes one piece of the puzzle and breaks it down in plain terms. This is post four of twelve. Twelve is holding, but only because I cheated. The GPU Deep Dive and Introduction don’t count. Enjoy post four. Not six. Four.
Over the last two installments we picked the engine. Memory, precision, interconnectivity, and power told us which GPU fits the model and vice versa. The GPU Deep Dive put those numbers to work but left us hanging. We know how to choose the engine, but we now need a vehicle.
In real life, don’t we usually choose the vehicle first and assume that the engine will do the job? Huh? Interesting… wonder if there is a correlation here?
By itself, a GPU works about as well as an engine sitting on cinderblocks. Just as engines come in many sizes, the vehicles on the lot for GPUs range from a system you can hold in one hand to a rack that weighs as much as a pickup truck (or a hippo, for those who were digging the zookeeper analogy). For this first compute installment, let’s take a walk around the lot and see what is for sale, who builds it, and why nearly everything on the NVIDIA partner lot seems to come from a similar set of blueprints.
Two questions from the first GPU post are still on the board: how many, and from whom. To begin to answer these questions, we’ll start with a walk around the lot and see what is for sale. By the end of this installment, we’ll be well on the say to finding our answers. By the end of Compute Part 2, we’ll have them.
As our tagline demands, we'll start small with the mopeds and work our way up to the freight train out back, with a stop in the back corner of the lot where the wrong purchases end up. As we walk through, we’ll keep note of when we feel finance and facilities' blood pressure rising.
How many? From whom? One lot, one engine family. The train is parked out back because it doesn’t fit in the lot.
The mopeds and motorcycles: phones and laptops
Freeze. Why are we talking about phones and laptops? This seems ridiculous for a series on AI infrastructure and AI Factories. Aren't we starting too small?
Yes, the lot starts here. Your phone already runs small AI models on its own chip for photos, dictation, and summaries, and it does so without asking anyone’s permission. That is the moped. One rider, goes anywhere, with not a single thing in the trunk. A laptop with a real GPU in it, the gaming and workstation laptops now being sold as AI PCs, is the motorcycle. You are getting a beefier engine, still built for one or two riders, still not much room for cargo. This is where most people consume AI. It is worth noticing that most of the tokens they end up generating from these devices are produced somewhere else. The meter from the Tokenomics post is running in someone else’s AI Factory. Nobody runs a factory from a motorcycle. Developers do live on them between trips to the lot, which is why the next vehicle matters.
Blood pressure check: Finance expensed it without reading the receipt. Facilities has one in their pocket and couldn’t care less how much power it consumes as long as it is charged up. If it generates too much heat, they’ll switch pockets.
The cars: the AI desktops and workstations
Let’s go off on a brief tangent and speak ecosystems. A developer with a DGX Spark on their desk can pull down an open-weights model, build an application, and fail or succeed at it quickly and cheaply. A solid proof-of-concept can be imagined (Think Big), invested in without breaking the bank (Start Small), and moved on from quickly without worrying about a sunk investment (Fail Fast). This can all be done in the same ecosystem, testing security, leveraging proven guardrails, and scrutinizing functionality in a repeatable proving ground (Discipline Matters). A DGX Spark, either direct from NVIDIA or built by an OEM partner, runs the same operating system (OS) and the same Compute Unified Device Architecture (CUDA) as a 4,000-pound GPU beast. Are there differences? Of course, but the core is the same. Build an app on the desk, pilot it on a server, promote it to a rack in the data center, and scale without starting over at each stage. The box on the desk is a scale model of the factory.
Does this mean every container built on a Spark can simply be exported and deployed in the datacenter? Not quite because not every container is interchangeable. Spark uses an Arm-based CPU, so moving workloads from Spark to an x86-based server may require rebuilding containers, validating architecture-specific dependencies, and tuning performance. The CUDA ecosystem provides a common foundation, but it does not eliminate differences between platforms.
It is the ecosystem that is the commonality. Deployment of production workloads still requires real engineering effort.
You may not look like much, kid, but you got it where it counts.
The compact on the desk. Same CUDA as the racks in the data center.
Again. We are starting small. Very literally. The compact car on our lot is NVIDIA's DGX Spark. It is about 2 x 6 inches and weighs roughly as much as a guinea pig (new animal, check). In that little frame it employs 128 GB of memory shared between a Blackwell GPU and an Arm CPU. One driver, cheap to run, parks anywhere, and the fastest way to learn a route before you buy anything with a cargo bed. In Deep Dive terms, that is enough bowl to hold the Qwen3-32B model at full 16-bit precision with room left over for a table of friends, and Sparks can be physically cabled together to take on larger jobs.
The sedan is a workstation with a 96 GB RTX PRO 6000 Blackwell Workstation Edition GPU. It is the car everyone with a license can drive.
The station wagon is, fittingly, the DGX Station. Still the same family as the compact and sedan but there is a lot more room in the back. A B300-class GPU with 252 GB of HBM plus 496 GB of CPU memory.
The usual workstation builders sell versions of all the above.
Now the tokenomics caveat, because, well… there always is one. The Spark's memory is big for a desktop, but it is not fast compared to the rest of the family, and the GPU Deep Dive taught us that bandwidth is chewing speed. Using the simplest memory-bandwidth calculation, an 8-bit Qwen3-32B model at Spark's 273 GB/s delivers a theoretical estimate of approximately eight tokens per second for a single conversation. Actual performance depends on the inference engine, model implementation, context length, and optimization. Fine for one developer learning the software and testing functionality of an application. Hopeless for a department needing to host a production workload. Breakfast for one, and no friends are coming over unless they are very patient.
Blood pressure check: Finance shrugs on the expensive Spark desktop. “You sure you need this?” email may have been written. Facilities didn’t even notice a Spark was purchased until seeing one on someone’s desk. The department head gets a call from finance for the DGX Station. The DGX Station’s listed 1,600 watts and required 20-amp circuit get a walk in from the Facilities team.
The work van: the PCIe server
Most of the GPU servers in the enterprise are still standard servers with two to eight GPUs plugged into the same PCIe slots we have been using for networking adapters and RAID controllers for years. The GPUs are typically air-cooled, but liquid-cooled configurations are available. You’ll find L40S, H200 NVL, and the 96 GB RTX PRO 6000 Server Edition in these servers. NVIDIA’s RTX PRO Server platform includes eight-GPU designs with both liquid and air configurations.
These are typically easier facilities projects then what you will see to come, but each configuration will need to be accurately interpreted in order to not make an expensive mistake. For instance, eight 600 W GPUs add up to 4.8 kW before adding everything else in the server. That isn’t a light lift. The familiar PCIe slots do not guarantee a familiar PDU and power bill.
The trade-off with PCIe servers, compared to their more powerful big sisters, is interconnectivity. Typically, GPUs in PCIe slots must pass notes over, you guessed it, PCIe. In some configurations, like the H200 NVL builds, GPUs can pass notes over an NVLink bridge. This is still not the same level of communication you get with an NVL8 fabric which is coming up next. Spoiler alert! Here is another one: Verbal Kint is Keyser Söze! If you haven’t seen that movie by now... I just don’t know what to say.
That makes the PCIe server the right vehicle for work that fits inside a single GPU or one that occasionally spans a few. Think inference on models that fit in a single card, retrieval-augmented generation (RAG), computer vision, virtual desktops, digital twins, and lighter fine-tuning methods like LoRA. It is a poor vehicle for a job that needs eight GPUs talking at memory speed. The work van hauls a great deal but not compared to a semi.
Blood pressure check: Finance signs. Facilities asks which rack, checks the PDUs, and cooling capacity. A quick glance at the power bill. If those numbers come back ok, he remains unconcerned and like that… *poof*... he’s gone...
The semi: The Eight-GPU D/HGX Server
This brings us to the workhorse most seen on the AI roadways. Eight GPUs joined at the hip, all in one NVLink domain. The NVL8, sold as DGX by NVIDIA and built around HGX by everyone else, a distinction we shall untangle next time. Around 300 pounds of muscle with an appetite for 15 kilowatts of power. NVIDIA’s DGX B200, for example, weighs in at 313.9 pounds and requires 14.3 kW. This is the semi, and it is where most enterprise AI Factories begin for the same reason most freight moves by truck. It carries a serious load, runs on roads that already exist, and you can buy a fleet of them (if you can afford them and, lately, get ahold of them).
Eight SXM GPUs are manufactured on a single baseboard with NVSwitch, so all eight tightly communicate and work as a team for software to divide work across. One tractor, one trailer, one load. Around that SXM baseboard the server OEM adds host CPUs, memory, storage for staging data, and power supplies that pull as much from the PDUs as a dozen homes. The server also includes high-speed network adapters that connect it to other GPU nodes, along with interfaces for storage, management, and application traffic. NVIDIA BlueField DPUs may be used for infrastructure services such as security enforcement, storage acceleration, and multi-tenant isolation. We’ll go into more of this later. Not this post later or next post later. But… later later.
The NVLink domain is the group of GPUs that is connected by a fast, internal fabric. Add a second semi and you have another NVLink domain with a network connecting them. That is why eight GPUs in one NVLink domain and eight GPUs scattered among other types of servers are not interchangeable even when the total specs of GPU, memory, storage, and cooling read similarly.
Current Blackwell nodes can be air-cooled with enough airflow in the facility. This is often helped along by rear-door heat exchangers (RDHX), if the data center’s usual hot and cold aisle containment can’t quite get the job done. Just keep in mind that RDHX often need separate supporting infrastructure like Cooling Distribution Units (CDU) and manifolds, a subject also for later later. There are multiple liquid-cooled options from mature OEMs. The HGX Vera Rubin generation of NVL8 leverages liquid cooling for the baseboard. With these systems, you must match the OEM’s site requirements with your rooms actual capabilities. Often, this must be done with a walkthrough with the partner or vendor to get right. Don’t expect your facilities people to be experts in system requirements.
The fine-tuning job from the Deep Dive called for a semi. It wants an eight-GPU node to start, with time-to-result deciding how many you put on the road and how you team them together.
I personally have multiple customers who bought one or many of these and assumed they could turn them on in their usual facility. They couldn’t. I have seen these servers shipped, delivered, racked, and not powered on for months, sometimes over a year. I had one customer who powered one on in a data center, and the people upstairs complained about the noise, so they had to turn it back off. Running these types of systems requires in-depth planning and expertise.
Delivered, racked, and a year into waiting on the building. The wasted nail, with the lights off.
With that said, these are incredibly performant machines in a mature ecosystem. We have customers with hundreds of these in production. Has anyone heard of memory shortages, supply chain issues, and massive data center builds? These are why.
Blood pressure check: Finance clears their throat. Facilities asks about airflow, then asks again, not believing you. You give them the power requirements and they start web browsing, because they still don’t believe you. You mention checking the raised floor and racks, because the servers are about 300 pounds in 10 rack units, and they stare at you sideways. “… and you want to put how many in a rack? Even if the floor can handle it, this room qualified for 2 kW rack and you want 60 kW!”
Three hundred pounds is a respectable deadlift. It is also one server, and four are going in the rack. Bring a server lift.
The freight train: the rack-scale NVL72 system
You read the title right. This isn’t a server. It is an entire system, with its own internal network built in. In a rack-scale system like the GB300 NVL72 the right vehicle is no longer on the lot. It is on the rails out back, because it is too advanced to use regular roads. It is purpose-built to provide something better. Eighteen compute trays, each carrying two Grace CPUs and four Blackwell GPUs, plus nine NVLink switch trays, are coupled into this train. 72 GPUs acting in tandem with 36 CPUs acting as the conductors. Power arrives through power shelves and a copper busbar, because ordinary power strips can no longer do the job. A train draws from the rail instead of pulling over at gas stations. The whole thing is liquid-cooled because there is no other way to absorb the 120 to 140 kilowatts of heat generation a single Blackwell rack generates.
Vera Rubin NVL72, the 4,000-pound rack the first GPU post compared to a hippo, pushes that over 200 kW. A train moves what no fleet of trucks can, but it only goes where someone has laid rail. That is the facilities conversation in one sentence. The floor, the power in the room, and the facility water loop with its cooling distribution units (CDUs) and manifolds are the rails. This is the vehicle for the most demanding jobs. Training or fine-tuning very large models and serving frontier-scale open models are a fit for this train. The models we previously named as almost always multi-GPU (DeepSeek, the large Qwen mixture-of-experts variants, Llama 4, etc.) are right at home in the NVL72. If your jobs do not need it, a fleet of semis is cheaper per GPU, air-coolable, and a far lighter facilities project.
A good practice here is to buy the smallest NVLink domain your biggest single job requires, then scale out. The biggest mistake we see people make with the NVL72 is bringing facilities in after one of them has been procured. These systems are absolutely not plug-and-play in a normal facility. We’ll go into the supporting equipment one of these needs down the line.
Blood pressure check: Finance and Facilities are both out in the parking lot having a moment. Facilities may actually be questioning past and future career choices. Finance is trying to figure out why. There is no more to this sentence. Just why.
The train. Without rails in place, you are going nowhere. Where you going? Nowhere! (Say it with a thick Boston accent to land the reference.)
The power of the ecosystem
Notice what connects these systems. The compact, the van, the semi, and the train can all be run by the same human (yes, training is required for the more advanced vehicles). All of them run the similar software and the same CUDA libraries. A container and model built on the compact can be ported to the train (with some modification), and a person comfortable on one is comfortable on the other. Moving everyone on the train by compact car would take a traffic jam of them and might never finish, not because the job is foreign to the compact but because it is just too big of a job. What changes is what each vehicle is capable of and what is needed to support it: how much is coupled into it, how much power and cooling the facility must deliver, and how much the floor must hold. Two of those three are facility questions, which is why the second half of this series exists. The workload picks the vehicle. Facilities decides whether it can be turned on and safely used.
The back corner of the lot: the money pit of despair
Every dealership has a back corner where the wrong purchases end up, and the AI Factory version is expensive. Until now, our vehicles have shown the scale of systems, from a compact car to freight train. Here, let’s focus on some specialty vehicles for specific types of work. The Tokenomics post gave us two vehicles for two kinds of jobs, the muscle car and the cement truck, and it is time to park a school bus next to them. A cement truck carries one specialized load to a job site. Training and fine-tuning are cement-truck jobs. Enormous workloads. Not a lot of users. A school bus carries a lot of people, each in their own seat, along a scheduled route, and top speed is not the point. Get the job done for everyone safely and consistently. Inference for a department is a school-bus job. Hundreds of conversations, each user needing a seat, and a modest, steady speed.
The muscle car is what you get when you buy the engine and forget the cargo. For instance, HPC requires efficient 64-bit precision not the latest and greatest performant GPU.
The back corner. Nothing here is broken, but it is all wasted on the wrong workload.
Where am I? The Pit of Despair! Don’t even think… clears throat … don’t even think about trying to escape… (Anyone? Deep cut here.) This equipment is expensive and hard to get. If you need one thing and purchase another, you can waste a lot of money here.
Needed a school bus, bought a cement truck. I’ve had this conversation a few times. The use case is an internal assistant for a hundred users. Customer wants it to be built on a training-grade system because "we might fine-tune someday." Ummm. Don’t do that. “Why aren’t we getting the ROI we expected? This is very expensive.” Yeah.
The training-grade system doing lighter inference delivers at a terrible cost per token, because most of it sits idle. Tokenomics called idle GPUs one of the most expensive ways in the world to produce nothing. Nothing is broken or non-functional. In fact, the system screams and users are happy. But funding that could have gone to something else is simply gone.
Needed a cement truck, bought a fleet of vans. The fine-tuning job gets quoted. The final, approved bill shows a price for a stack of PCIe servers. The GPU count matched and the price was millions lower than the competing quote for NVL8s. Why should we spend more on the HGX when these other ones have GPUs with the same name and the same count? This is the architect's seven-million-dollar problem from the Tokenomics post. The payload is one 512 GB load that requires riding in one coupled vehicle. Ever see one of those semis carrying pre-assembled houses? Ever see one of those houses balanced precariously on the roofs of eight vans with individual drivers? I haven’t either. PCIe slots pass notes at a fraction of NVLink speed. A fine-tuning job would run so slowly it effectively wouldn’t work. Finance was happy because nothing was wasted on paper. Everything was wasted in practice. This just won’t work. Then, after it fails, you still need to go buy the right-fit solution. Then Finance is no longer happy.
Needed a cement truck, bought a muscle car. The research team that says 'AI' but actually needs FP64-heavy scientific computing. Let’s go out and buy the newest inference-optimized system because the keynote petaflops are enormous. The Deep Dive's label on the B300 read "Not a significant source of FP64." HPC needed the full 64-bit cement truck and got a very fast car instead. No place to put the load. I guess you could try to move some cement with a Mustang. I wouldn’t bank on a solid resale value afterward though. This is the exact example the Tokenomics post used, and it happens all the time.
No time for hurt feelings
That is the correlation I so playfully pondered up top. Most of us buy the vehicle first because we feel it is a good fit and we trust the engine will be adequate. Sometimes it is. Sometimes it is close enough. We do this in the real world because we can visualize things and then use our natural common sense to make close-enough decisions. As adults, we spend a lot of time driving so our first-glance instincts on what we need are naturally acute.
In the GPU world, on this lot, the engines and vehicles can look the same but be vastly different. It is extremely easy to oversimplify and then make what we feel are common-sense decisions and promptly waste a lot of money. This series went engine first on purpose. The workload picks the engine, the engine picks the vehicle, and the vehicle picks the facility. Dial the combination lock from the introduction in that order and it opens. Get any number wrong or put the right numbers in the wrong order and the lock just won’t budge. You won’t get the result you are looking for.
There is also another way to waste money on this lot. Ever debate buying a new car for a three-day vacation? Maybe a rental is a better idea. Yeah. Don’t buy what you don’t need, and if you aren’t sure what you need, definitely don’t buy something expensive. We’ll jump into this in more detail in the next one.
Pick your load first
You know the vehicles, from the moped in your pocket to the train out back, and you know the back corner where the money goes. One rule keeps you out of that corner: figure out the purpose of the system before you walk the lot. Two questions from the board are still open, how many and from whom, and the second one has an interesting answer. Next up! Who actually builds these things, why five quotes for the same server look identical, and the biggest question of all, whether the fleet and the garage should be yours or someone else’s.
Think big. Start small. Fail fast. Discipline matters.
If your team is working through this right now, it is exactly the kind of conversation we have every day. Get in touch and we can talk through where you are.
Next time: Compute, part two. Who builds it, and should it be yours?