• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Cooling an AMD Instinct MI210 for local inference

Zarathustra[H]

Zarathustra[H]

Extremely [H]
2FA
Joined
Oct 29, 2000
Messages
42,351
Does anyone else have one of these bad boys?

mi210.jpg


What are you doing about cooling?

The Instinct MI210 doesn't have a fan, it is passively cooled. It is open in the back, and the design intent is that it is to be installed in a server chassis, and the fans in the fan wall are supposed to blow through and cool the GPU.

I figured, well, I'm going to use my Supermicro 4U chassis, so I am probably good, but just to err on the safe side, why not use a fan shroud with an 80mm fan.

Well, it turns out, the commonly available fan shrouds don't fit in my 4U case. They make the card too long and it interferes with the fan wall...

PXL_20260816_220035655.PORTRAIT.jpg


So I figured, screw it. I need server air flow, I have server airflow, I'll just install it naked....

PXL_20260816_231556079.jpg


Well, that was a no-go.

As the system booted, the amdgpu driver in linux immediately alarmed of an overheat condition and shut down the system....

This, despite the fact that I significantly upgraded the airflow in my Supermicro chassis by replacing the stock fan wall wit its small (and loud) 80mm fans with a custom one consisting of three beefy 120mm Noctua iPPC-3000 PWM fans.

I guess not all server air flow is created equal.

I suspect that the servers this card originally went into just had crazy full blast fan profiles, and probably used some sort of shroud to direct fan wall air straight into the GPU.

So I got creative.

I cut pieces out of a plastic cutting board, and taped them together with HVAC foil tape to create my own low profile shroud I could attach to a beefy 80mm fan (I went with a San Ace 80mm server fan) zip tied to the carrier handle in the back.

It just barely squeezed in, but it was a huge improvement.

At the ~45-50% fan speeds Supermicro sees it fit to use by default, it was reading 54C at idle. Still not great, but at least not a forced shutdown.

I noticed that at the stock fan speeds Supermicro uses for the FanB header I used when in "Optimized" fan mode, the San Ace fan was only operating at ~2200rpm.

FanA and FanB are Supermicro headers usually used for storage array cooling, so I changed the fan profile to "Heavy I/O" which boosted it up in speed (and volume) of the fan.

1786934122351.png


So, with the San Ace 80mm fan at about 6000rpm, it makes a little bit more noise than I prefer, but at least the GPU gets decent idle temps now.


Code: 
amdgpu-pci-c700
Adapter: PCI adapter
vddgfx:      725.00 mV
edge:         +44.0°C  (crit = +99.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
junction:     +44.0°C  (crit = +100.0°C, hyst = -273.1°C)
                       (emerg = +105.0°C)
mem:          +36.0°C  (crit = +94.0°C, hyst = -273.1°C)
                       (emerg = +99.0°C)
PPT:          39.00 W  (cap = 300.00 W)
sclk:         800 MHz
mclk:           2 GHz


I understand that about 45C idle is typical for these things in typical OEM server installs.

Unfortunately in my rush to get the server back up, I forgot to take any pictures of the ghetto shroud. I'll have to take a picture of it so everyone can revel in its ghettoness.

It's certainly the most expensive hardware I have ghetto modded to date!

I still need to set up my VM with vLLM so I can load test it. When I do, I'll share the temps here.
 
Last edited:
the static pressure drop across the server cards is insane. have Claude code you up something that does full-RAM-sized bf16 GEMMs or spin up ComfyUI and have it generate some long videos with Minimax-H3 and keep an eye on the HBM temps as you tune cooling, that is a $4000 card so I'd err on the side of caution and try not to thermal cycle it too hard. the core is the same as its ever been but those HBM2 stacks are fragile (the hard limit is 85C which is 20C lower than the core)
 
Just get two 15-25krpm 40x40x28-38mm fans on the open edge of the card. You need a shedload of high pressure directed airflow. In your case all air will simply go around it.
 
mashie said:
Just get two 15-25krpm 40x40x28-38mm fans on the open edge of the card. You need a shedload of high pressure directed airflow. In your case all air will simply go around it.
Click to expand...

That sounds.... LOUD 😅

I do have a dedicated server area, but I'm not sure I want fans that are audible through multiple sets of doors 😅

The ghetto shroud on the 80mm server fan seems to be working this far. I'll see if it holds up to stress testing.
 
bwang said:
the static pressure drop across the server cards is insane. have Claude code you up something that does full-RAM-sized bf16 GEMMs or spin up ComfyUI and have it generate some long videos with Minimax-H3 and keep an eye on the HBM temps as you tune cooling, that is a $4000 card so I'd err on the side of caution and try not to thermal cycle it too hard. the core is the same as its ever been but those HBM2 stacks are fragile (the hard limit is 85C which is 20C lower than the core)
Click to expand...
I'm honestly not 100% sure what to expect from my loads.

Unlike how these were likely used when new (for training) where they saw constant high matrix math loads, I suspect for local inference the high heat load will be very short and bursty during pre-fill. During evaluation it should be mostly RAM bandwidth limited, and run cooler.

No idea what to expect from a heat perspective.
 
Last edited:
Zarathustra[H] said:
That sounds.... LOUD 😅
Click to expand...
Maybe try to find pressure values more than RPM ? maybe you can reproduce that pressure with larger fans and shrouds.

If you are willing to use a pci riser, maybe a gpu cooling "tube" with a push fans and a pull fans at each end, going for 120mm high pressure fans, gpu in the middle, need to be quite air thight too.

pre-fill should be pick gpu heat, inference peak HBM2 heat ? probably still some under 75 watt type for the memory or something like that.
 
LukeTbk said:
Maybe try to find pressure values more than RPM ? maybe you can reproduce that pressure with larger fans and shrouds.

If you are willing to use a pci riser, maybe a gpu cooling "tube" with a push fans and a pull fans at each end, going for 120mm high pressure fans, gpu in the middle, need to be quite air thight too.

pre-fill should be pick gpu heat, inference peak HBM2 heat ? probably still some under 75 watt type for the memory or something like that.
Click to expand...

The plan right now is to test with real world loads and se rig my ghetto shrouded 80mm San Ace server fan will do the trick, and then decide what to do after that.

I spent some time on it last night, but found myself in ROCm/Python/PyTorch/vLLM dependency hell.

Per the vLLM install guide it looks like I am going to have to install a generic docker container, and then compile a specific version of vLLM inside it that is able to properly communicate with ROCm.

Fun times. I'll probably work more on it tonight...
 
Zarathustra[H] said:
The plan right now is to test with real world loads and se rig my ghetto shrouded 80mm San Ace server fan will do the trick, and then decide what to do after that.

I spent some time on it last night, but found myself in ROCm/Python/PyTorch/vLLM dependency hell.

Per the vLLM install guide it looks like I am going to have to install a generic docker container, and then compile a specific version of vLLM inside it that is able to properly communicate with ROCm.

Fun times. I'll probably work more on it tonight...
Click to expand...
If you want to stay Linux, I would just drop Ubuntu on the machine, Ollama, and Hermes if you want to play with Agentic to keep it simple out of the gate. Or go full Win11 and use Hermes Desktop which is awesome. You are going to find that trying to force your way down the ROCM path with local is going to be HUGE disappointment. You have modify almost all these open models to use ROCM. Vulcan is the norm for backend runtime. At your stage I would not set anything up for thinking it is going to be that way "forever." Start small, with easy configs. That way when you decide you want to do it differently, which you will, then you have not spent tons of time on configuring your base. My two cents, you may need change.
 
FrgMstr said:
If you want to stay Linux, I would just drop Ubuntu on the machine, Ollama, and Hermes if you want to play with Agentic to keep it simple out of the gate. Or go full Win11 and use Hermes Desktop which is awesome. You are going to find that trying to force your way down the ROCM path with local is going to be HUGE disappointment. You have modify almost all these open models to use ROCM. Vulcan is the norm for backend runtime. At your stage I would not set anything up for thinking it is going to be that way "forever." Start small, with easy configs. That way when you decide you want to do it differently, which you will, then you have not spent tons of time on configuring your base. My two cents, you may need change.
Click to expand...

Interesting.

Thanks for the warning. In all of my googling I never came across anything but the ROCm approach.

I had been under the impression that the model itself couldn't care less about the backend.

I had been under the impression it worked like this:
GPU -> ROCm -> vLLM/Ollama/etc -> Model files.

It had been my impression that in this model, as long as vLLM/Ollama can support the model, and can communicate with ROCm and as long as ROCm sees the GPU, all is good.

I'm going to give it a try either way, building a docker container and compiling vLLM for Docker and then see how it goes. If it breaks, it breaks.

I appreciate the heads up though!
 
So, I finally got it running by building the container, and finding a different community version of the model that actually worked properly with vLLM.

and I'm.... Disappointed?

Throughput looks pretty impressive when running 10 simultaneous prompts as the vLLM benchmark does
Code: 
============ Serving Benchmark Result ============ 
 Successful requests:                     10         
 Failed requests:                         0          
 Benchmark duration (s):                  8.91       
 Total input tokens:                      5120       
 Total generated tokens:                  1280       
 Request throughput (req/s):              1.12       
 Output token throughput (tok/s):         143.64     
 Peak output token throughput (tok/s):    186.00     
 Peak concurrent requests:                10.00      
 Total token throughput (tok/s):          718.22     
 ---------------Time to First Token---------------- 
 Mean TTFT (ms):                          2203.07    
 Median TTFT (ms):                        463.59     
 P99 TTFT (ms):                           4984.83    
 -----Time per Output Token (excl. 1st token)------ 
 Mean TPOT (ms):                          32.81      
 Median TPOT (ms):                        33.55      
 P99 TPOT (ms):                           35.04      
 ---------------Inter-token Latency---------------- 
 Mean ITL (ms):                           32.81      
 Median ITL (ms):                         32.37      
 P99 ITL (ms):                            42.68      
 ==================================================

But lightly "threaded" (for lack of a better term) loads are nowhere near where I had predicted they would be. A standard single prompt is only giving me about 44 tokens/s

I'm not sure if I made a beginner mistake, or if there is something wrong with my setup.

At least the temperature is reasonable. When I loaded it up with the benchmark, the hottest I saw the GPU get was about 55C.
 
Zarathustra[H] said:
So, I finally got it running by building the container, and finding a different community version of the model that actually worked properly with vLLM.

and I'm.... Disappointed?

Throughput looks pretty impressive when running 10 simultaneous prompts as the vLLM benchmark does
Code: 
============ Serving Benchmark Result ============
 Successful requests:                     10        
 Failed requests:                         0         
 Benchmark duration (s):                  8.91      
 Total input tokens:                      5120      
 Total generated tokens:                  1280      
 Request throughput (req/s):              1.12      
 Output token throughput (tok/s):         143.64    
 Peak output token throughput (tok/s):    186.00    
 Peak concurrent requests:                10.00     
 Total token throughput (tok/s):          718.22    
 ---------------Time to First Token----------------
 Mean TTFT (ms):                          2203.07   
 Median TTFT (ms):                        463.59    
 P99 TTFT (ms):                           4984.83   
 -----Time per Output Token (excl. 1st token)------
 Mean TPOT (ms):                          32.81     
 Median TPOT (ms):                        33.55     
 P99 TPOT (ms):                           35.04     
 ---------------Inter-token Latency----------------
 Mean ITL (ms):                           32.81     
 Median ITL (ms):                         32.37     
 P99 ITL (ms):                            42.68     
 ==================================================

But lightly "threaded" (for lack of a better term) loads are nowhere near where I had predicted they would be. A standard single prompt is only giving me about 44 tokens/s

I'm not sure if I made a beginner mistake, or if there is something wrong with my setup.

At least the temperature is reasonable. When I loaded it up with the benchmark, the hottest I saw the GPU get was about 55C.
Click to expand...
Might not be out of line for that beast, what model are you trying to run? Temps seem decent anyways.

Also, are those 3x of the Asus M.2 cards I spy in that rig?
 
SamuelL421 said:
Might not be out of line for that beast, what model are you trying to run? Temps seem decent anyways.
Click to expand...

It's currently running an 8bit quantized version of Gemma4:26b.

It's an MoE model, but it's performing more as if it were dense which is a little confusing.

Based on how it was running CPU only on my EPYC (~207GB/s RAM bandwidth) at 24-48 tokens/s I was expecting this cars to be about is faster since it has 1600GB/s RAM bandwidth...


SamuelL421 said:
Also, are those 3x of the Asus M.2 cards I spy in that rig?
Click to expand...

Yup. I wanted to get a lot of NVMe storage in this server. It currently has 18 NVMe drives at 4x lanes each. 12x in the 4-way risers, two on the motherboard and two more Optanes attached to an 8x SlimSAS port.
 
Last edited:
Zarathustra[H] said:
Based on how it was running CPU only on my EPYC (~207GB/s RAM bandwidth) at 24-48 tokens/s
Click to expand...
207GB/s for a small model (MoE under 4GB for expert) give you what theorical max speed or about 50-55 token/s ?, reaching 48 is really good.

Maybe when it is that small, it decide to put all the model instead of just expert in the available vram ? and treat it virtually like a full 26b model each time, you would go down to around 60 token max speed on the 1,600 GB/s.

Is it possible to use Llama.cpp instead of vLLM ?

That said agentic/multi agent loop work can look a bit more like simultanous work than single and more were local make sense (single prompt thing is virtually free right now online) and that type of pro hardware for doing it has well.
 
LukeTbk said:
207GB/s for a small model (MoE under 4GB for expert) give you what theorical max speed or about 50-55 token/s ?, reaching 48 is really good.

Maybe when it is that small, it decide to put all the model instead of just expert in the available vram ? and treat it virtually like a full 26b model each time, you would go down to around 60 token max speed on the 1,600 GB/s.

Is it possible to use Llama.cpp instead of vLLM ?

That said agentic/multi agent loop work can look a bit more like simultanous work than single and more were local make sense (single prompt thing is virtually free right now online) and that type of pro hardware for doing it has well.
Click to expand...

To be clear, I am getting ~44 tokens per second for non-parallel prompts on the AMD MI210 HBM2 GPU with 1600GB/s VRAM bandwidth.

On the EPYC CPU with 207GB/s RAM bandwidth I was seeing about 24 tokens per second.

So, I am getting a little over 1.8x performance from just under 8x the memory bandwidth, which is really quite perplexing.

This is on Gemma4:26b, by the way, which is a 26B total weight MoE model with ~3.8B active parameters.

And I can't seem to figure out if:

1.) There are inherent efficiency/ scaling issues with configurations like these that I just missed due to being a noob

2.) Gemma4:26b is just a poor match for this hardware

3.) If my configuration (ROCm/vLLM) is screwed up. (It took several tries to get it working, eventually compiling/building my own docker image with ROCm and vLLM in it.)

4 ) If there is something wrong with the hardware.

That said, the standard vLLM benchmarking script (results posted above) shows that the GPU can REALLY tear through the model when given some parallelism, but at lower parallelism it seems to be severely under-utilized.

My intended use was to set up a chat and general purpose AI model with some privacy emphasis for the house. I appreciated the ability to use some parallelism (maybe 2-4 parallel prompts) but being able to support higher levels of parallelism is kind of wasted on my usable models where only a handful of people will be using it. I'm not supporting a 50-person org 😅
 
pioruns said:
https://sleepingrobots.com/dreams/stop-using-ollama/

Don't get vendor-locked. Use llama-cpp.
Click to expand...

Ollama is good for setting something up for testing. I understand why they are popular. They take a highly convoluted and fragile process of configuring docker images that contain the likes of ROCm and vLLM with the specialty TRITON and AITER modules needed to make the most of the system and instead provide a setup that "just works".

But yeah, for my final setup, I moved away from Ollama.
 
So,

I think part of this performance issue is due to the ROCm/vLLM Docker image I built. It appears to be missing the AITER kernel, which is apparently key to optimizing for performance on AMD's professional hardware.

Not sure how much fixing this is going to help, but I am going to give it a try.
 
FrgMstr said:
LM Studio is SO easy...
Click to expand...

That is the usual trade-off isn't it? Ease of use, vs vendor lock in (and potential privacy/telemetry issues.)

I'd ideally want a system which completely lacks any integration with any online service.

I insist on managing everything locally and manually. I don't want any hooks into Hugging Face or Ollama's online library or anyone else.
 
Last edited:
Zarathustra[H] said:
That is the usual trade-off isn't it? Ease of use, vs vendor lock in (and potential privacy/telemetry issues.

I'd ideally want a system which completely lacks any integration with any online service.

I insist on managing everything locally and manually. I don't want any hooks into Hugging Face or Ollama's online library or anyone else.
Click to expand...
llama-cpp solves this, I already said it two posts above. ollama creeps into online/cloud model, because that's their end game, they want money.
llama-cpp on the other hand, is 100% offline, and it works with any GGUF model, no vendor/platform locking. I use llama-cpp and I've integrated it into many local services, like writing e-mails, interpreting logs, daily stock exchange summaries and portfolio analysis on Telegram, two-way chat in Telegram, code generation, and more. Everything offline, if internet goes down tomorrow, or some company goes bust, my setup is not affected. It's perfect for tinkering, everything runs on my offline GPU model.
 
pioruns said:
llama-cpp solves this, I already said it two posts above. ollama creeps into online/cloud model, because that's their end game, they want money.
llama-cpp on the other hand, is 100% offline, and it works with any GGUF model, no vendor/platform locking. I use llama-cpp and I've integrated it into many local services, like writing e-mails, interpreting logs, daily stock exchange summaries and portfolio analysis on Telegram, two-way chat in Telegram, code generation, and more. Everything offline, if internet goes down tomorrow, or some company goes bust, my setup is not affected. It's perfect for tinkering, everything runs on my offline GPU model.
Click to expand...
Yup. And at at this point, you can use a harness to automagically set all of those up without worrying about "easy". Swap between llama/vllm/sglang just by telling Hermes to whenever i want to try a new model or configs.
 
pioruns said:
llama-cpp solves this, I already said it two posts above. ollama creeps into online/cloud model, because that's their end game, they want money.
llama-cpp on the other hand, is 100% offline, and it works with any GGUF model, no vendor/platform locking. I use llama-cpp and I've integrated it into many local services, like writing e-mails, interpreting logs, daily stock exchange summaries and portfolio analysis on Telegram, two-way chat in Telegram, code generation, and more. Everything offline, if internet goes down tomorrow, or some company goes bust, my setup is not affected. It's perfect for tinkering, everything runs on my offline GPU model.
Click to expand...
Yeah, I considered going with llama.cpp, but I read somewhere that vLLM was better at managing multi-user loads, juggling users context windows and such, and I figured I'm setting this thing up for the whole household, so that may not be a bad thing to have, but it has certainly turned out to be the more intense setup.
 
Last edited:
Zarathustra[H] said:
I'm honestly not 100% sure what to expect from my loads.

Unlike how these were likely used when new (for training) where they saw constant high matrix math loads, I suspect for local inference the high heat load will be very short and bursty during pre-fill. During evaluation it should be mostly RAM bandwidth limited, and run cooler.

No idea what to expect from a heat perspective.
Click to expand...
if it is anything like the nvidia stuff figure max power during prefill for a couple minutes at a time, 80w during decode, and max power generating images/videos
 
Zarathustra[H] said:
So,

I think part of this performance issue is due to the ROCm/vLLM Docker image I built. It appears to be missing the AITER kernel, which is apparently key to optimizing for performance on AMD's professional hardware.

Not sure how much fixing this is going to help, but I am going to give it a try.
Click to expand...

So I have done more research here, and have a few failed attempts.

My first attempt was building the docker image per these instructions. It works, but it lacks AITER and TRITON which are kind of key for performance enhancements.

I then tried the rocm/vllm image, and it was too old to run Gemma4's MoE models.

After that I tried rocm/vllm:nightly

This is bleeding edge, but did not work for me. The reason being that for whatever reason, the version of AITER included in that docker image only supports FP8, not INT8 (it is supposed to support every quantization scheme under the sun, but for some reason this one doesn't) Someone suggested that maybe that's because modern vLLM images are focused on MI300+ series of cards which support FP8.

I could run an FP8 model with AITER, but then I'll have software emulation of FP8, which would slow things down even slower than running without AITER at all.

I could get rid of this problem and just run a GGUF 8_0 model in llama.cpp, but then I give up all of the good stuff in how vLLM handles multiple simultaneous users and managing context.

I think I might try a llama.cpp instance and see how it performs. If it performs similarly in concurrent prompts as my existing working vLLM build does, then I'll just stick with that for now. If llama.cpp performs better, then I know I really need to pursue getting a version of vLLM with AITER that supports INT8 working, tand that will probably mean monkeying with the internals of docker images, compiling and linking the AITER module manually. 🙄

This will be fun...
 
Nobu said:
AITER doesn't work on mi2xx or earlier generations, only mi3xxView attachment 821087View attachment 821088
"AITER (AMD Instinct Triton Extensions for ROCm) rejects gfx90a " (mi200 etc)
https://github.com/davetha/mi210-llm-stack/blob/main/guides/moe-expert-cache-vllm.md
Click to expand...

Well that certainly explains why I haven't been able to get it working 😅

I don't know how I didn't come across this in any of my reading.

It appears that a lot of the guides and articles I have come across just assume you have newer hardware without explicitly stating it.

Thanks for the heads up. While I am a little disappointed, at least I won't be wasting any more time on this.

I did notice one thing I can probably do to speed up my performance significantly though. I realized I accidentally am using context quantized to FP8, which would involve emulation. After work today I am going to try switching the context quantization setting to INT8 (and if that doesn't work, - reluctantly BF16) and see how much that helps.
 
Zarathustra[H] said:
Well that certainly explains why I haven't been able to get it working 😅

I don't know how I didn't come across this in any of my reading.

It appears that a lot of the guides and articles I have come across just assume you have newer hardware without explicitly stating it.

Thanks for the heads up. While I am a little disappointed, at least I won't be wasting any more time on this.

I did notice one thing I can probably do to speed up my performance significantly though. I realized I accidentally am using context quantized to FP8, which would involve emulation. After work today I am going to try switching the context quantization setting to INT8 (and if that doesn't work, - reluctantly BF16) and see how much that helps.
Click to expand...
Not to send you down another rabbit hole, but here is something to check out: https://github.com/davetha/aiter-cdna2

Full disclosure - I no longer have any modern AMD cards to test so I couldn't say if this works or not. My Epyc server/workstation/thing is similar to yours but I've been strictly using ampere parts for the past year or so.
 
I decided to stop fighting the software stack in vLLM, and am now running llama.cpp in server mode.
Performance is much improved:

Code: 
prompt eval time =     471.21 ms /   979 tokens (    0.48 ms per token,  2077.63 tokens per second)
eval time =    1024.95 ms /    86 tokens (   12.06 ms per token,    82.93 tokens per second)
total time =    1496.16 ms /  1065 tokens

It seems like llama.cpp is a lot more robust. I really like the memory paging feature in vLLM, but it isn't worth the performance impact, IMHO.
 
Zarathustra[H] said:
I decided to stop fighting the software stack in vLLM, and am now running llama.cpp in server mode.
Performance is much improved:

Code: 
prompt eval time =     471.21 ms /   979 tokens (    0.48 ms per token,  2077.63 tokens per second)
eval time =    1024.95 ms /    86 tokens (   12.06 ms per token,    82.93 tokens per second)
total time =    1496.16 ms /  1065 tokens

It seems like llama.cpp is a lot more robust. I really like the memory paging feature in vLLM, but it isn't worth the performance impact, IMHO.
Click to expand...
Well done!
 
pioruns said:
Well done!
Click to expand...

Yeah, it wasn't quite as straight forward as I had hoped.

First, llama.cpp refused to compile due to dependency conflicts. My original VM was Debian 13 (since I have been trying to get Ubuntu out of my life) and I tried to manually remediate the dependency conflicts and just couldn't make it work. In defeat I moved to Ubuntu 26.04 LTS, and had the same problem there.

In the end I had to install the older Ubuntu 24.04 LTS in order to get it to work.

Once I finally got llama.cpp to work, I had several problems with the models I found on huggingface.

The community 8bit quants of Gemma4:26b available on Hugging Face were utter trash. The models were messing up reasoning tags and formatting left and right, were entering frequent loops (one - amusingly - on its own just decided to repeat the phrase "loop-de-loop" over and over again ad infinitum) and were generally acting completely insane.

Best I can tell these were quantized using either llm-compressor or the scripts in llama.cpp before those packages fully understood the new MoE structures inside of Gemma4:26b, thus messing them up in the process.

I wound up having to grab the full BF16 version of googles vanilla Gemma4:26b .safetensors model, and manually convert it to a GGUF and quantize it down to 8_0. Now all seems to work well though, but fighting this and learning along the way took me most of Friday evening after work, and most of the day on Saturday.

But I did just discover that this broke the vision capability of the model. Apparently when running direction from .safetensors models you don't have to take any additional steps, but when using GGUF models you need to manually extract and trigger something called "mmproj" layers, which I didn't realize.


Sadly I only realized this after I deleted the full BF16 model. Normally I archive these things on my NAS, but I figured "I'll never need that again" and just deleted it. So today I am downloading the model again in order to extract the mmproj so I can get the vision capability back.

Wish me luck!

PS.

FWIW, if anyone else is having a difficult time maintaining decent download speeds when grabbing massive files from huggingface using traditional tools such as wget or curl, I have found that the command line tool aria2c is a godsend.
 
Last edited:
Well. Turns out the mmproj component of gemma4 is not evenly divisible by whatever number it needs to be evenly divisible by to make an 8_0 version, so I am going to have to run it at full BF16... I hope everything still fits in VRAM...

If anyone is curious, here are my resultant files:
Code: 
alpaca@alpaca-gpu:/usr/share/llama-models/google$ ls -lh
total 27G
-rw-rw-r-- 1 alpaca alpaca 1.2G Aug 23 22:10 gemma-4-26b-mmproj-BF16.gguf
-rw-rw-r-- 1 alpaca alpaca  26G Aug 23 05:38 gemma-4-26b-q8_0.gguf

And here is my resultant systemd startup script:
Code: 
[Unit]
Description=Llama Server - Gemma 26B (Multimodal)
After=network.target

[Service]
Type=simple
User=alpaca
WorkingDirectory=/home/alpaca
Environment="LD_LIBRARY_PATH=/opt/rocm/lib"
ExecStart=/usr/local/bin/llama-server \
    --model /usr/share/llama-models/google/gemma-4-26b-q8_0.gguf \
    --mmproj /usr/share/llama-models/google/gemma-4-26b-mmproj-BF16.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    --n-gpu-layers 99 \
    --parallel 3 \
    --ctx-size 262144 \
    --cont-batching \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --kv-unified \
    --temp 1.0 \
    --top_k 64 \
    --top_p 0.95

Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
 
Zarathustra[H] said:
Yeah, it wasn't quite as straight forward as I had hoped.

First, llama.cpp refused to compile due to dependency conflicts. My original VM was Debian 13 (since I have been trying to get Ubuntu out of my life) and I tried to manually remediate the dependency conflicts and just couldn't make it work. In defeat I moved to Ubuntu 26.04 LTS, and had the same problem there.

In the end I had to install the older Ubuntu 24.04 LTS in order to get it to work.

Once I finally got llama.cpp to work, I had several problems with the models I found on huggingface.

The community 8bit quants of Gemma4:26b available on Hugging Face were utter trash. The models were messing up reasoning tags and formatting left and right, were entering frequent loops (one - amusingly - on its own just decided to repeat the phrase "loop-de-loop" over and over again ad infinitum) and were generally acting completely insane.

Best I can tell these were quantized using either llm-compressor or the scripts in llama.cpp before those packages fully understood the new MoE structures inside of Gemma4:26b, thus messing them up in the process.

I wound up having to grab the full BF16 version of googles vanilla Gemma4:26b .safetensors model, and manually convert it to a GGUF and quantize it down to 8_0. Now all seems to work well though, but fighting this and learning along the way took me most of Friday evening after work, and most of the day on Saturday.

But I did just discover that this broke the vision capability of the model. Apparently when running direction from .safetensors models you don't have to take any additional steps, but when using GGUF models you need to manually extract and trigger something called "mmproj" layers, which I didn't realize.


Sadly I only realized this after I deleted the full BF16 model. Normally I archive these things on my NAS, but I figured "I'll never need that again" and just deleted it. So today I am downloading the model again in order to extract the mmproj so I can get the vision capability back.

Wish me luck!

PS.

FWIW, if anyone else is having a difficult time maintaining decent download speeds when grabbing massive files from huggingface using traditional tools such as wget or curl, I have found that the command line tool aria2c is a godsend.
Click to expand...
Did you configure a HF token? I always max out my 1Gig connection from them.
 
mashie said:
Did you configure a HF token? I always max out my 1Gig connection from them.
Click to expand...
I don't sign in or use tokens.

I also don't use any of the automated download tools. I don't allow anything to have direct contact with the outside world. It's air-gapped. I download the files manually and copy them in.

I don't trust these people/organizations in the slightest.
 
Based on FrgMstr speaking pretty highly of Laguna S 2.1 I decided to add it as a Code Review / Reasoning / Logic check secondary model running on the CPU of the EPYC.

Being on the CPU the pre-fill will be brutal, but the EPYC has almost as much memory bandwidth as a Nvidia Sparc unit, so eval happens at an OK rate. You wouldn't want to chat with it real time, but it is usable to pop in a lump of code, ask for a code review and have it run in the backgrund.

Code: 
total duration:       20.001859968s
load duration:        104.253558ms
prompt eval count:    303 token(s)
prompt eval duration: 5.004488s
prompt eval rate:     60.55 tokens/s
eval count:           203 token(s)
eval duration:        14.889402s
eval rate:            13.63 tokens/s

So I use Gemma4:26b on the GPU with 2000+ tokens/s pre-fill performance and 80+ tokens/s eval rate.

Then I use Laguna S 2.1 (Q4_K_M) running on the CPU for additional background tasks where more logic is required at 60+ tokens/s pre-fill and almost 14 tokens/s eval.

Chances are I'll change my mind at some point, but this seems to do the trick for right now.

Hindsight being 20-20, the MI210 may not have been the amazing bang for the buck I expected for my use case, but I am making the best out of it, and honestly, the performance is quite great.

On the rare occasion I generate code (usually just some automation script in python for the home lab) my workflow will be something like this:

Chat with the fast model and come up with ideas and write some sample code. When I think I have what I want, grab that code and pop it into Laguna running on the CPU give it the premise and tell it to do a detailed code review, fix errors and make improvements, and then go to lunch or something.
 
Last edited:
You must log in or register to reply here.
Back
Top