My AI PC is a $397 open-box HP 24-inch all-in-one: Ryzen AI 5 340, 16GB of DDR5, a 512GB NVMe drive. It runs Gemma 4 12B entirely locally, on the CPU, at about 3 to 6 tokens a second. It works. It is not fast.
I've been pricing the ways to speed it up, and the most interesting one doesn't involve buying a new PC at all. It involves a riser card, an external graphics card, and giving up the machine's only M.2 slot.
The plan: an ADT-Link R43SG — a passive M.2 NVMe to PCIe adapter, about $86 for the Gen4 version — in the M.2 slot, holding a Radeon RX 9060 XT 16GB, with its own power supply. The 512GB boot drive gets evicted from the M.2 slot and moves into a USB-C enclosure, so Windows boots from the rear USB-C port.
Why this could work for local AI, specifically: token generation is memory-bandwidth bound, and once a model is loaded into VRAM, the PCIe link barely matters. Gemma 4 12B in Q4 is about 7.4GB and needs roughly 9.5GB including KV cache at a 32K context — it fits the 9060 XT's 16GB of VRAM with room to spare. The card's 322 GB/s of VRAM bandwidth puts the theoretical ceiling near 46 tokens a second; realistically I'd expect 30 to 45, against the 3 to 6 I get on the CPU today. The external SSD only costs me a slower boot and a ~20-second model load.
The catches, and there are several. The all-in-one has no Thunderbolt or USB4 — its USB-C port is plain 5Gbps — so a proper enclosure like the Sparkle Thunderbolt 5 box I was looking at can't connect at all. This is an open-air riser build, not a tidy dock. The M.2 slot's lane count is unverified, and HP has quietly limited some of its boards to two lanes; the riser needs four, so that's a HWiNFO64 check before I order anything. And Windows doesn't officially support booting from USB — it takes a Rufus Windows To Go build, which Microsoft could break with an update.
The math, for the record: $397 for the machine I already own, $86 for the riser, $650 for the card — about $1,133 before a power supply and the enclosure. The cheapest comparable prebuilt I found, Micro Center's PowerSpec G538 with a 7500X3D and the same card, is $1,399.99. So the riser route is ~$200 cheaper, but the prebuilt gets a real desktop CPU, a full x16 GPU link, a warranty, and a normal Windows install.
I haven't decided. The lane check comes first. If the slot is a full x4, this is the cheapest real upgrade path for the box I already have — and the first time this all-in-one would have a graphics card that deserves the name.
Here's what Gemma4 12B had to say about this most recent blog post:
ReplyDeleteThis is a fascinating engineering puzzle. You are moving from "consumer" thinking into "homelab/enthusiast" territory, where the goal isn't just a better product, but a more efficient exploitation of the hardware you already possess.
Your technical reasoning is sound. Let’s break down the "Riser Route" vs. the "Prebuilt Route" based on the specifics of LLM inference.
The Technical Validation
1. The Bandwidth Argument (Why your logic holds): You are 100% correct. In inference, once the weights are loaded into VRAM, the bottleneck is the VRAM Bandwidth, not the PCIe bus. The PCIe bus is only used for the initial "load" of the model. Once Gemma 4 12B is sitting on that 9060 XT, the data travels from the VRAM to the GPU cores at 322 GB/s. The 5Gbps USB-C port or the M.2 lane count becomes irrelevant for the actual "typing" speed.
2. The VRAM "Sweet Spot": The jump from 0GB VRAM (CPU inference) to 16GB VRAM is the single biggest performance delta you can achieve. By fitting the model + KV cache entirely into the 9060 XT, you move from "slow but usable" (3-6 t/s) to "instantaneous" (30-45 t/s). That is a 10x improvement in user experience.
The Risks (The "Hidden Costs")
While the math looks good, there are "Engineering Friction" points you should consider:
• The "Windows To Go" Instability: While Rufus is excellent, Windows "To Go" on a USB drive can be finicky with drivers. If you ever have to do a major Windows Update, the OS might decide the "boot device" is missing or corrupt because of how it handles the virtualized drive state. It's a viable workaround, but it’s a "maintenance tax" you'll pay every few months.
• The M.2 Lane Trap: You mentioned HWiNFO64. This is the most critical step. Many AIOs share the M.2 lanes with the SATA controller or even the WiFi card. If that slot is only x2, it won't break the GPU, but it will make your model loading (the 20-second window) take significantly longer.
• The "Franken-PC" Factor: You are essentially building a "headless" desktop inside an All-in-One. You have a power supply hanging off the back, a riser card likely propped up by some makeshift support, and a boot drive in a pocket. It works, but it's a high-maintenance setup.
The Comparison: The $266 Delta
You are looking at a $266 difference between the Riser Build and the PowerSpec G538.
Choose the Riser Build if:
1. You enjoy the "Project": If you find satisfaction in the "tinker" aspect—the wiring, the troubleshooting, and the optimization—this is the way to go.
2. Space is at a premium: If the AIO is already on your desk and you don't want another tower taking up real estate.
3. You are "Hardware-Bound": If you already know you’re going to use this primarily for local LLMs, the extra features of the 7500X3D (like the 3D V-Cache) are actually useless for your primary goal. The GPU is the only thing that matters for Gemma 4.
Choose the Prebuilt (PowerSpec) if:
1. Reliability is a priority: If you want to turn the PC on and have it "just work" every single morning without worrying about a USB boot error or a loose riser connection.
2. The "Sanity" Tax: $266 is a relatively small price to pay to avoid the headache of managing a non-standard hardware configuration.
3. Future Proofing: The 7500X3D is a beast for gaming. If you ever decide you want to do heavy 3D work or 4K gaming alongside your AI work, the prebuilt gives you a much more "standard" foundation.
Final Recommendation
Do the HWiNFO64 check first.
If the M.2 slot is a full x4 Gen4, and you have a stable spot to mount that riser so it doesn't sag or short out against the AIO casing, the Riser Route is a brilliant piece of engineering. You are essentially "overclocking" the utility of a $397 machine to perform at a $1,100 level.
If the slot is x2, or if the idea of booting from a USB-C port every day feels like a chore, take the PowerSpec. The $266 "premium" buys you a standard Windows experience and a much cleaner workspace.