-What do you think is the most cost efficient bang for your buck hardware to run AI locally in 2026? From what I've read, multiple GPUs will pool memory together in running local LLMs together unlike SLI/CrossFire days. Probably 2 x RTX 5060 Ti 16GB will be the cheapest poor man's setup?
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
AI LLM hardware discussion thread
- Thread starter David-Duc
- Start date
depending on what you do-need size wise, used single 3090 or M2 era 24GB mac mini would be alternative...
a single 3090 still have more memory bandwith than 2 modern but 128 bits gpu, blackwell will offer "native" modern FP4 support and other bonus, but can be a bit more complicated setup.
24gb used mini is the cheapest (to buy and to run), but slowest, give you a full machine that can do other things.
a single 3090 still have more memory bandwith than 2 modern but 128 bits gpu, blackwell will offer "native" modern FP4 support and other bonus, but can be a bit more complicated setup.
24gb used mini is the cheapest (to buy and to run), but slowest, give you a full machine that can do other things.
sparks are nice for DEV in the cuda env, but lot of money for that very limited level of bandwith if you only interested in running things, depend quite a bit what you want to do.
One general tips, would be to try renting ont he cloud (5060, 3090, etc... are the type you can) to run your exact flow to see the actual difference
One general tips, would be to try renting ont he cloud (5060, 3090, etc... are the type you can) to run your exact flow to see the actual difference
- Joined
- May 18, 1997
- Messages
- 58,293
Not sure about that, but would be a nice setup for the memory footprint.
I run models on my 7900XTX (24GB), and it is super speedy. 16GB would be somewhat limiting.
For going 256GB, i would look Apple m3 ultra 256gb price vs 2x DGX spark...
The memory bandwith will be quite higher on apple here, ~800 GB/s, no FP4 support of course, Spark get more interesting if you know in advance you will use only nvpf4 models. and apparently now support have been added, but would verify to be sure.
Used Mac ultra 256GB seem in a similar price range, $8000 ish dollars.
The memory bandwith will be quite higher on apple here, ~800 GB/s, no FP4 support of course, Spark get more interesting if you know in advance you will use only nvpf4 models. and apparently now support have been added, but would verify to be sure.
Used Mac ultra 256GB seem in a similar price range, $8000 ish dollars.
Have you tried multiple GPUs running ollama or similar? From what I can find on the internet, 2x16GB will give you 32GB just like a single 32GB GPU so the only difference (hypothetically) between an RTX 5090 and 2x 5060 Ti 16GB is speed or token/s. What I want to know is if there's any hidden gotcha in doing this? PCI-E bandwidth becoming a problem? Anything like the old day with SLI where the rendering on both card not matching up and one would have to play with Alternate/Split Frame...
I'm not a fan of small and dumb models, even if they are fast. Deepseek v4 Flash 0731 NVFP4 with 1M context and 46 tok/s decode and some 1100 tok/s pre-fill just works.
Shame the pre-fill isn't much to write home about on Apple devices.
- Joined
- May 18, 1997
- Messages
- 58,293
No.
True enough, with that many * maybe that use case specific, better just:
https://aws.amazon.com/ec2/instance-types/mac/
If you can, actually try them before pulling the trigger.
I have no interest in any Apple products personally.
I have 4 Sparks on my desk and next week a MikroTik CRS504 will arrive so I finally can turn them into a 4-node cluster instead of the current 2-node clusters.
We shall see if 100G is fast enough for the interconnects, if not I will add one more switch at a later date to double the bandwidth.
I have 2x3090s running qwen 3.6 coder 30b. It's okay for pet projects and some minor code suggestions, but it's so much worse than something like fable or opus. It's fun but ultimately not that useful. If I get it to run openclaw or hermes and have it build stuff in unity over like 6 hours it can knock out a small pet project or feature if I just let it go. Some of it's useful, some of it is not.
![Zarathustra[H]](/data/avatars/m/9/9298.jpg?1723407683){kind=avatar}
- Joined
- Oct 29, 2000
- Messages
- 42,357
I think this is desired performance / budget dependent.
I did some research on this, and from a tokens/s per dollar perspective of the solutions that had enough VRAM to run a decent model, I came to the conclusion that the AMD Instinct MI210 was quite the bang for the buck. But that doesn't mean it was cheap...
I did some research on this, and from a tokens/s per dollar perspective of the solutions that had enough VRAM to run a decent model, I came to the conclusion that the AMD Instinct MI210 was quite the bang for the buck. But that doesn't mean it was cheap...
- Joined
- Dec 19, 2005
- Messages
- 17,792
Funny how dual Sparks was my suggestion at the very top of this thread.
To me there is not an affordable hardware for LLM. If you go with the old route of buying 10x A100 for $1500 a pop and a tall server blade, then you are easily looking at $30.000, which is out of my price range and you would still only get 480-640GB of VRAM.
New Mac Studio with M5 Ultra can be preordered. 256 GB max for now, 512 GB in October.
https://www.apple.com/mac/compare/?modelList=Mac-mini-M6,Mac-mini-M5-Pro,Mac-Studio-M5-Ultra
https://www.apple.com/shop/buy-mac/...core-cpu-80-core-gpu-256gb-memory-1tb-storage
Sadly no 138 GB option on the new mini.
https://www.apple.com/mac/compare/?modelList=Mac-mini-M6,Mac-mini-M5-Pro,Mac-Studio-M5-Ultra
https://www.apple.com/shop/buy-mac/...core-cpu-80-core-gpu-256gb-memory-1tb-storage
Sadly no 138 GB option on the new mini.
A nice $12300 for the 256GB model with 4TB of storage.
It will be interesting to see if the pre-fill speed has improved at all with this generation.
That is a pretty interesting option. $8208 over 3 years, for 256G + 4tb, similar specs to 2x Sparks.
You are correct, this setup seems to be the best bang for the buck right now if you want useful work. "Dev kit" or not.
Prefill is what kills the performance on the mac. Also performance doesn't seem to scale well with multiple macs.
I like the cut of your jib. To me this stuff started getting useful recently, around Opus/Sonnet so it doesn't make sense to bother setting this up or spending any money unless you can get lots of useful work out of it. From what I hear from software devs Deepseek v4 Flash is supposed to be somewhere between the 2. I'd rather wait for useful work slowly than get trash quickly.
Speaking of 128 GB Strix Haloo machines - how are the laptops compared to the mini desktops?
Asus has the Z13 and HP has one, too. Of course they don't list the TDP they are running the CPU at. Are they full powered like the desktops? The laptops sell at the same prices or even below.
Asus has the Z13 and HP has one, too. Of course they don't list the TDP they are running the CPU at. Are they full powered like the desktops? The laptops sell at the same prices or even below.