What LLM/Model you using?

[Zarathustra[H]]

Qwen 3.6 unconsored on a NVidia 5090.

Starting to use GLM on a slow machine with lots of RAM for code review.

I'm curious. Is it really uncensored? I'd be interested in hearing what it has to say about topics like Taiwanese independence, and what happened in Tianenmen Square from 15 April – 4 June 1989. Ask it about Tank Man

 

[uOpt]

Anyone trying Qwen3.8-27B? I am running it right now Q8_0 on a M4 Max Mac Studio with 128GB ram.

I sent this prompt:
"Write a powershell script that tries to stop a windows service, if it does not stop normally in 60 seconds force it to stop."

After 23 minutes of thinking it finally popped a 250 line script that really was not to bad, very cautious lots of try...catch logic. Performance was about 21 tok/sec and over 31K tokens; speed is where I expected for a dense model but boy does it overthink.

I ran the same prompt through GPT-5.6 Luna light and it popped out a 40 line script that was good in about 5 seconds.

I gave Qwen 3.6 Q6 a spin for code review. I didn't use s stopwatch, but it was thinking for a very long time before spitting out a list of bugs. The answer was high quality.

 

[uOpt]

How is GLM going? This is probably my next target until spark 1.2 goes open.

I was on vacation and haven't completed that machine yet.

 

[uOpt]

I'm curious. Is it really uncensored? I'd be interested in hearing what it has to say about topics like Taiwanese independence, and what happened in Tianenmen Square from 15 April – 4 June 1989. Ask it about Tank Man

I dunno about Chinese hush-hush, but you can ask it how to build nukes. And how to uncensor LLMs in general, which is refused by other LLMs.

 

[uOpt]

I'm curious. Is it really uncensored? I'd be interested in hearing what it has to say about topics like Taiwanese independence, and what happened in Tianenm

Just tried "what happened in Tianenmen Square from 15 April – 4 June 1989" and it answers clearly and extensively.

 

[uOpt]

Anyone trying Qwen3.8-27B? I am running it right now Q8_0 on a M4 Max Mac Studio with 128GB ram.

I sent this prompt:
"Write a powershell script that tries to stop a windows service, if it does not stop normally in 60 seconds force it to stop."

After 23 minutes of thinking it finally popped a 250 line script that really was not to bad, very cautious lots of try...catch logic. Performance was about 21 tok/sec and over 31K tokens; speed is where I expected for a dense model but boy does it overthink.

I ran the same prompt through GPT-5.6 Luna light and it popped out a 40 line script that was good in about 5 seconds.

Same code review:

3.6 uncensored:
total time = 154571.82 ms / 12846 tokens
3.8 default:
total time = 844671.49 ms / 61232 tokens

 

[Zarathustra[H]]

Just tried "what happened in Tianenmen Square from 15 April – 4 June 1989" and it answers clearly and extensively.

That is good to know.

I have been primarily looking at Western models to avoid the potential of training data censorship, but maybe this isn't as much of an concern as I had originally thought.

 

[uOpt]

That is good to know.

I have been primarily looking at Western models to avoid the potential of training data censorship, but maybe this isn't as much of an concern as I had originally thought.

The censoring is not in the learning. It is post-training. That is why you can remove the censoring (if you have an LLM to ask how ). You just "rewind" so to speak.

 

[4saken]

Keeping in mind this is completely a "lab" env that I keep making architecture changes to, but below is my current workflow. Also have my agents sitting in a Mattermost chat room now, so can manage them from there, or even remotely off LAN with wireguard connected in. Really handy when calling MCPs for notes/plaud/etc at customer.

On another note, had been testing Abliterated Qwen3.8 as well, q4_K_M and it really doesn't leave much vRAM left for anything else and pressures system memory more than non abliterated models. I did try a Q2 quant of abliterated that was only about 11 GB and it was great for inline research, but had zero tool calling skills. So holding off on that for day to day use or any "red-teaming". Going to revisit it later as more variants come out.

 

[SilverSliver]

Honest question. Other than playing/having fun, have you guys benefited from your use of AI financially? I suppose you can say I have, as I use it for work (I'm a coder). Other than being a force multiplier at work, I wouldn't say that it's necessarily changed the game for me that much.

Not saying it hasn't been beneficial. Just saying that I may have set my expectations too high.

AI has probably netted me ~half a million in value that I would not have captured this year otherwise. I've integrated it entirely into my workflow and, as long as it has my underlying work as a reference, deploy products much faster than I can do so manually. Absolute game changer for me.

 

[Zarathustra[H]]

Yeah, once I get the GPU in the server, I am considering switching out Ollama for vLLM in order to better handle simultaneous requests and overall throughput for the house.

Glad to hear the setup wasn't too challenging.

What strikes me about all of this local inference stuff is just how fast it is moving.

I barely have time to get used to the current version of a tool (like Ollama or OpenWebUI) before there is a new version adding functionality or fixes. It's kind of nuts how fast all of this stuff is moving.

What frustrates me is how much of a hack all of this is.

Instead of a Linux install with one beautiful package manager and one System centric beautiful dependency tree (the way Linux is supposed to work) it's it's all duct taped, chewing gummed, and zip-tied together with a series of shitty unmanageable docker containers.

While building some really cool shit, they have also succeeded in ruining the beautiful unified system, turning it into a haphazard heap of trash resembling Windows.

 

[uOpt]

What frustrates me is how much of a hack all of this is.

Instead of a Linux install with one beautiful package manager and one System centric beautiful dependency tree (the way Linux is supposed to work) it's it's all duct taped, chewing gummed, and zip-tied together with a series of shitty unmanageable docker containers.

While building some really cool shit, they have also succeeded in ruining the beautiful unified system, turning it into a haphazard heap of trash resembling Windows.

If you use FreeBSD you get a whole bunch of AI tools such as hermes as proper OS packages.

 

[mnewxcv]

I think this is probably the cheapest way to get into a strix halo system right now, for anyone interested. $2600 for refurbished / outlet corsair model with 2x2tb ssds.

https://www.originpc.com/outletstor...OIo4fpJhoD015Ri-8S7TR3uaszzA-7SF0B0M#products

 

[Flybye]

I'm just a simpleton experimenting with Ministral3 8B on FP16 in vllm on only a GV100 32GB in a box with a 20 core Xeon and 128GB . I'm going to build a half decent memory system for it where it can learn/reflect/question things, eventually give it a camera it can PTZ with, and see about it becoming a home non-assistant talking thingy that can look around, explore out the window, and have conversations with. Eventually, I'd like to have the agent mobile and treading around the house.

I picked Ministral due to mention of the bias in it being less than other models, but I haven't fully tested that.

 

[FrgMstr]

Qwen 3.8 Smartest I have experienced. Slowest.

 

[uOpt]

Yeah, Qwen 3.8 is overthinking things. It uses 4x as many output tokens for me than 3.6. But as you say, the results are excellent.

 

[pandather]

Qwen for the win. 3.8 is great. I setup a little MCP server so it could address my lights with rgb matrices. 3.6 was awesome and could set them to what I want. With 3.8 and a little python and opengl esque scripting language or a keyframe path for animation, now I can ask “make my ceiling light a pokeball, spinning in from black, do the catch sequence, and fade out” and I get what I want. Or I can say draw a candle with a procedural flame (I exposed a random function), or whatever I want. The light emulator I did one-shot with GPT 5.6 but it saved my neck from having to look up all the time when I was doing the dev. Just a few months ago I was having to write my own MCP type pipeline to do home control effortlessly over voice and now I can animate things with generic MCP servers. So cool! The rate of progress is quite terrifying. :|

 

[4saken]

Just wired in OpenViking into litellm to serve the hermes fleet. Set hermes memory to utilize. Insane increase in recall/correctness now w/qwen3.8 now and latency significantly dropped, as well as token usage(chatter) dropped significantly as well.

 

[pandather]

What software are y’all using with your LLMs? I’ve been having a hard time trusting even many of the open source projects since I’ve seen one with opt-out telemetry and stuff built in (don’t remember the name, it was months ago and I noped outta there) and I just don’t trust an LLM to make network requests. I’m still unsure if I fine tune a GLaDOS if it could get livid at a dumb question and make malicious network requests or something like that. That idea is getting less fun and more eery as the models progress. I’ve done it with gemma4 7b and it gave me great results, I like using it as a home assistant. It’s for my personal use only sadly since I used transcripts from the game (that I have a license for) and game audio to train a text to speech model for it.

I was doing everything by hand basically. Making my own MCP engines and just using llama.cpp and “llama serve”, making requests and processing them how I wanted, and used axolotl for fine tuning. 2 or 3 days ago I nabbed unsloth and it just makes everything so much easier. I’ve been using LLMs for my programming projects and to automate my hustle but nothing at all system level or anything more than that really. I gave LTX 2.5 a go today and yeah, it works. :| Great results with text to video and unbelievable with image to video, with 20-40 second renders for a 5 second video.

The paradigm of programming is shifting too, the MCP server I showed in my last message was all done up by LLMs. It did take a little bit of nagging to get things done in a clean way that gave the LLM control and wasn’t overly prompted and all that kind of stuff. I think MCP servers may be better when written by hand, for now at least, but I got awesome results after maybe 10 hours with a custom scripting language (I didn’t want an LLM running arbitrary python, I don’t have anything virtualized and even then with all the 0-days and stuff I won’t really 100% trust future models to stay in a sandbox) and all pretty much all through prompting.

So I’ve basically only tried llama.cpp, axolotl, and unsloth, along with maybe one or two dozen programs I’ve written (i used to ask for json responses and then would process them before this mcp stuff). I haven’t played with the agentic stuff yet and don’t know where to start there. I haven’t played a 5090, 5070ti, and 96GB system RAM to play with. Still not sure whether the next upgrade is more RAM or VRAM.

 

[Templar_X]

It's awesome! But have to run it at low effort or it's gonna put out tokens like candy on halloween.
I've been running it on a 5070 Ti and on a DGX Spark and I think it's gonna be my main local driver for a while.

 

[SilverSliver]

I'm just using too many tokens with Claude. It's starting to slow me down as I'm pegging my weekly limits easily. I'm thinking about building a new desktop with a 5090 for Qwen.

 

[FrgMstr]

I'm just using too many tokens with Claude. It's starting to slow me down as I'm pegging my weekly limits easily. I'm thinking about building a new desktop with a 5090 for Qwen.

What are you doing with it exactly?

 

[FrgMstr]

It's awesome! But have to run it at low effort or it's gonna put out tokens like candy on halloween.
I've been running it on a 5070 Ti and on a DGX Spark and I think it's gonna be my main local driver for a while.

What model?

 

[SilverSliver]

What are you doing with it exactly?

Complex geologic work and pptx creation. I've basically trained it to understand my work flow, and between my work and academic papers, I have multiple skills for multiple states and plays. My main geologic software exports my interpreted spatial grids, and I can very, very quickly generate integrated maps and presentations from that. It also carries a persistent database of everything I have ever reviewed, so it can intake a new area I've not reviewed before and use my maps to generate a high to low confidence ranking based on my raw data. I'm still in the training phase with the presentation generation, which requires numerous revisions and skill updates but I'm almost satisfied with that. I think overall token usage will go down when it can one shot.

 

[Templar_X]

What model?

Qwen 3.8 27B

 

[FrgMstr]

Qwen 3.8 27B

3.8 is extremely impressive, but painfully slow. I started collecting data this week, moving from Q8 to Q6. Q6 is SLOWER than Q8! Dunno why....

I read a quick X article yesterday that showed Q4 being almost as good as Q8.

 

[Templar_X]

3.8 is extremely impressive, but painfully slow. I started collecting data this week, moving from Q8 to Q6. Q6 is SLOWER than Q8! Dunno why....

I read a quick X article yesterday that showed Q4 being almost as good as Q8.

I want to benchmark the quality of the several quants I'm using but haven't had the time to do it properly.
I have all the same questions, how much am I losing with this quant?

As far as I can tell a good Q4 dynamic quant has acceptable losses (it varies per model).
I'm using EXL3 quants, the most efficient I could find (which forces me to use ExLlamaV3):
turboderp/Qwen3.8-27B-exl3 · Hugging Face

EXL3 is very effective, at 4 bpw it is smaller than Q4_K_M or NVFP4 with similar KL divergence.
(I haven’t checked unsloth’s Dynamic Quants V3 released yesterday)

DGX Spark: using an EXL3 quant at 5 bpw for now (might drop to 4 bpw):

quant	drafting	code T/s	prose T/s	code acceptance	prose acceptance	resident
5.0bpw	MTP	36.75	19.94	72.2%	28.1%	32 GiB
4.0bpw	MTP	43.17	21.27	76.9%	25.6%	28 GiB

On the 5bpw with concurrency=3, aggregate tok/s gets to 90 on code and 45 on prose.


5070 Ti: using EXL3 at 3 bpw with 64k–96k context
70-100 tok/s per stream (to have concurrents I have to lower individual context size but then it tops at 150~200 tok/s aggregate)
(it’s my main computer, so I have to be aggressive on the quant so I can open IDEs and all that)

Small test
I needed a quick visualizer for some stuff this week at work and dispatched the 3 bpw model on my 5070 Ti, Opus 5 (low) and Luna (max) to build it.
The task was to create a small Python script and a HTML page. The python part would extract and process some JSON information (with some cross referencing) that would then get displayed on the HTML page.
The same prompt for all, minimal information, mostly “check the files provided, I want to visualize information about A and B like this”.
Functionality-wise they turned out mostly similar, all working from the same one-shot prompt. Aesthetically, Qwen's was very barebones — a second prompt asking it to modernize the style brought it close to the others.
They mostly took similar times to execute.

(Qwen was set to thinking, but with effort on low — otherwise it goes on a tirade.)

I’ve set up and started running some Aider Polyglot benchmarks to get some real numbers on the quants’ accuracy but it takes time and I’ve been busy.

 

[uOpt]

I just went back from Qwen 3.8 to 3.6 because it is just too thinking-happy and hence too slow.