How AI will fail

[LukeTbk]

Targeting is a simple match for simple things like user location. It doesn't use neural nets at all.

And the historic of all users action, scrolling and what not., it is custom to the user.

And you have meta advantage+/advanced google affair that are generative, they generate hundreds on alternative ads, test them and go with the tops one according to target profiles.

Content suggestion is not something consumers pay for.

I do not imagine they directly pay for it but almost... (if we of course mean by customer here the ads buyers), that said Google has now 325 millions paid subs accross service a lot from youtube premium, not sure how big it would be without content suggestion that hook them.

The cost is higher than the revenue for those 2, and my list was for profitable applications only.

Anthropic just had a positive ebidta quarter, https://www.wsj.com/tech/ai/mind-bl...rofitable-quarter-7edbf2f4?eafs_enabled=false and it depend what we mean by cost higher than revenues and subsidizing, anthropic mostly stopped subsidizing (as you said limits and what you can do for free being replaced more and more by api that charge by usage).

But they have 70-80% margin on the inference itself, they will probably go back to burning a bit of money everything included and being profitable overall last quarter the result of having an hard time buying enough compute for training, but the the new thing, their revenues exploded so much they have an hard time spending it all, as much as they want to do so.

Same story. Waymo uses humans to intervene, which is very costly.

growing fast, profitability is something they can see already, but Tesla FSD, many chinese one do not use paid humans intervention but push it to the users, John deer let the farmers intervens and do not do it themselve, more predictable fleet system for the walmart has really low intervention cost

 

[1.1.2.3.5...]

Wait, what are you claiming is done for autonomous tractors? (JD isn't unique to autonomous tractors).

Most precision ag applications are not FSD. But most major brands are advertising fully autonomous machines within ~5 years. Well see what actually happens.

Open question is not around capability but around liability. If you use a human in the loop, it just reduces the liability exposure.

 

[LukeTbk]

Wait, what are you claiming is done for autonomous tractors? (JD isn't unique to autonomous tractors).

That they let the farmer interven with their phone when there is an issue, instead of having remote john deer employee doing it a la waymo, is that false ? i.e. a lot of semi automous system use the clients as the person in charge, they do not pay employees to do so, so those FSD fees have giant raw margin once R&D-training are amortized, waymo is a bit more of an execption in that field, but their ability to charge more for rides than human driven service show that profitability should be quite possible.

 

[1.1.2.3.5...]

That they let the farmer interven with their phone when there is an issue, instead of having remote john deer employee doing it a la waymo, is that false ? i.e. a lot of semi automous system use the clients as the person in charge, they do not pay employees to do so, so those FSD fees have giant raw margin once R&D-training are amortized, waymo is a bit more of an execption in that field, but their ability to charge more for rides than human driven service show that profitability should be quite possible.

edit: edit: edit:
https://www.businesswire.com/news/h...utonomous-Tractors-in-American-Sugar-Industry

Things are changing so fast these days it's really hard to know what's happening across different businesses. So basically ignore my comments above. I just don't know anything for sure.

 

[stuffmonger]

Tesla's taxi has a human sitting in the passenger seat. Yeah, much self-driving, such wow. Camera only systems suffer from phantom braking, and won't be as reliable as LIDAR. Zoox and Waymo are the only ones who have a chance of succeeding.

Autonomous is well defined. Fully autonomous means level 5. Conditions are still debatable. E.g. weather, area restrictions.

Farming robots are usually programmed to go in a straight line, then turn by waypoint. This isn't reliable enough as 1.1.2.3.5... said, evidenced by the job listings for remote operators/supervisors.
Since a tractor could be millions of $, farmers need a solid proof that the system is at least as reliable as a human driver. There is no such proof.

 

[LukeTbk]

edit: edit: edit:
https://www.businesswire.com/news/h...utonomous-Tractors-in-American-Sugar-Industry

Things are changing so fast these days it's really hard to know what's happening across different businesses. So basically ignore my comments above. I just don't know anything for sure.

yup, John Deere is not the one doing it, farmer (or here the farm group getting together to do it, centralizing it a bit for their group member) is taken care of human intervention. For small independant farm, it get be the family with their phones.

 

[LukeTbk]

Tesla's taxi has a human sitting in the passenger seat. Yeah, much self-driving, such wow.

Tesla used to charge $6000 for FSD upfront, now $99 a month and the human intervention cost is past to the buyer.

Chinese self driving in some section, as an human but that can work or sleep, we are talking about revenues here, not if it is level 4 or 5, the fact they can already charge good money at lower level is a good sign for them, not a bad one.

 

[1.1.2.3.5...]

Being in the industry you would think I would know a lot more, but I am not customer facing and everything changes so quickly I don't want to make any specific claims. But I will say, devils in the details here.

 

[DukenukemX]

Not sure how limiting it is, Coreweave does not really know what their clients are doing with their compute with quite good encryption, that why even secret obsessed big pharma research create and use custom models on them. Cloud AI is a large sector that goes from using ChatGPT to running your own model on coreweave computers with a lot of controls.

One example is when I ask Grok to make images of middle-eastern leaders having a missile up their rears and it told me it can't do that Dave. Which shows me who exactly is running the show when you can't criticize them through basic images. From a pure art standpoint, I'd rather have a LLM any day of the week. Then there's the misinformation which Google and Gemini are really proactive about it. It's gotten so bad that I get better results with Bing. Then there's special interest groups who are trying to use AI to change peoples perspective of them. Not that I find any real use having AI answer me questions that it can't do, but if I wanted J.A.R.V.I.S. then I'd run it on a RPI. A very powerful RPI.

Gotta love that she thinks people trust AI. Isn't that cute.

 

[LukeTbk]

One example is when I ask Grok to make images of middle-eastern leaders having a missile up their rears and it told me it can't do that Dave.

that 2 different things, there is no LLM (that I can think of) that you can run at home but not on the cloud and I can easily imagine a local Apple AI with many of those limitations. Cloud vs local is just where the model is ran, not which is it.

 

[1.1.2.3.5...]

that 2 different things, there is no LLM (that I can think of) that you can run at home but not on the cloud and I can easily imagine a local Apple AI with many of those limitations. Cloud vs local is just where the model is ran, not which is it.

Well sure. But there are specific cases where the cloud doesn't work as well imho. For example I have an extremely large corpus of ebooks on technical things that I was to use as part of a rag setup. I could maybe figure that out, but much easier and cheaper to do that locally.

Additionally I want a smaller voice to text model that runs constantly and sends prompts to my larger llm. So i can talk with the system while it's working on things.

Technically all doable over tcp and iot clients, but not ideal.

 

[LukeTbk]

Well sure. But there are specific cases where the cloud doesn't work as well imho. For example I have an extremely large corpus of ebooks on technical things that I was to use as part of a rag setup. I could maybe figure that out, but much easier and cheaper to do that locally.

yup or dlss is better to be next to the data, cars use a lot of locals (they have a lot of video data and spotty connexion) robots will has well.... for ebook it would be relatively easy to upload them on your AWS or others virtual machine, static data is easy to make local to the model in the cloud, it is more realtime continous flow of local data where local become more interesting... has for much cheaper it could depends on the electricity cost and upfront investment for the hardware, it could be hard to beat deepseekv4 pricing.

But often will LLM you have text (very small data) relative to the compute to be made, which made cloud an easy to be used.

 

[1.1.2.3.5...]

yup or dlss is better to be next to the data, cars use a lot of locals (they have a lot of video data and spotty connexion) robots will has well.... for ebook it would be relatively easy to upload them on your AWS or others virtual machine, static data is easy to make local to the model in the cloud, it is more realtime continous flow of local data where local become more interesting... has for much cheaper it could depends on the electricity cost and upfront investment for the hardware, it could be hard to beat deepseekv4 pricing.

But often will LLM you have text (very small data) relative to the compute to be made, which made cloud an easy to be used.

These are books with a copyright that should not be in the parameters of LLMs, but we know they are to some extent.

But even if the model has that knowledge parameterized, it cannot cite the book. I want to be able ask my model something about say category theory and give me an exact citation. If LLMs are doing that without author's permissions, that's a really big problem.

 

[stuffmonger]

These are books with a copyright that should not be in the parameters of LLMs, but we know they are to some extent.

Meta torrented books from libgen to train llama. Turns out you can get paid for being a pirate

I want to be able ask my model something about say category theory and give me an exact citation

Index your books with vector embeddings, and use it as a RAG tool for an LLM to use. Deep research mode can do it, but it's expensive and very niche. Worst case, it's not hard to set up locally with Postgres and DeepSeek.

 

[LukeTbk]

These are books with a copyright that should not be in the parameters of LLMs, but we know they are to some extent.

But even if the model has that knowledge parameterized, it cannot cite the book. I want to be able ask my model something about say category theory and give me an exact citation. If LLMs are doing that without author's permissions, that's a really big problem.

I am not sure what this has to do with running your LLM at home or in a AWS session (or any other cloud arrangement) with the RAG of the books on the remote drives of that session if you want.

Not that using an remote deepseek API for a local RAG of the books would be any issues.... (and there are cheap cloud model for doing those), it will be small data versus compute for modern internet speed regardless.

I think one element that happen when people talk about how much cheaper it is to run their models at home instead of using the cloud, they compare the price of using very expensive model (claude, opus, GPT 5.5, etc...) with running way more limited model at home, if you compare how much it will cost to build something able to run Deepseekv4 284 billion versus using the remote API with the better 1.6 trillion full model, once you count electricity cost and amortization... you need quite the usage to save any money.

 

[DukenukemX]

I think one element that happen when people talk about how much cheaper it is to run their models at home instead of using the cloud, they compare the price of using very expensive model (claude, opus, GPT 5.5, etc...) with running way more limited model at home, if you compare how much it will cost to build something able to run Deepseekv4 284 billion versus using the remote API with the better 1.6 trillion full model, once you count electricity cost and amortization... you need quite the usage to save any money.

I don't think anyone could build something that would compete against what data centers have in them, but they don't need to. Also, the electricity cost is overblown since the machine clearly won't run full tilt 24/7. You could get a Panther Lake machine that draws 1.5W power while idling and put in some RTX GPU and let it idle all day long. Or just shut it off and send a magic packet to wake it up?

 

[1.1.2.3.5...]

You're suggesting I could deploy this in aws and just use their cloud hardware to run it, and yes, that's all true. I don't think the system running with an always on voice to text llm would be very cheap. I'll admit my circumstances are a little bit of a unicorn because I managed to get all my hardware for dirt cheap (relatively speaking).

To run the 27b qwen model I use the minimal ec2 node would likely be a 4xl40 or a10g. We are talking ~285 a month for continuous usage. I could try to grt fancy and build a load balancer that turn it on or just log into the aws cli and launch it when i want to. But that's a lot of work and it sucks.

I have been running 2x3090s with nvlink and my total gpu cost was ~2300 bucks since 2020. We are looking at 50 / month for my local costs.

 

[FrgMstr]

You're suggesting I could deploy this in aws and just use their cloud hardware to run it, and yes, that's all true. I don't think the system running with an always on voice to text llm would be very cheap. I'll admit my circumstances are a little bit of a unicorn because I managed to get all my hardware for dirt cheap (relatively speaking).

To run the 27b qwen model I use the minimal ec2 node would likely be a 4xl40 or a10g. We are talking ~285 a month for continuous usage. I could try to grt fancy and build a load balancer that turn it on or just log into the aws cli and launch it when i want to. But that's a lot of work and it sucks.

I have been running 2x3090s with nvlink and my total gpu cost was ~2300 bucks since 2020. We are looking at 50 / month for my local costs.

I got a buddy working on a voice to text input for his local setup as well. Will let you know his solution when/if he gets there. He is running Strix Halo as well, so whatever his solution, it should not be too resource intensive to run.

 

[stuffmonger]

Whisper is pretty good for speech to text, but it won't work with a wake word. Personally, I don't like voice UI, and prefer typing. Although, I was impressed with SesameAI's demo, which had a model that could be interrupted. The problem was that the LLM they used was small to be real time, and it was relatively dumb. The voice part was very human with sighs, and pauses.

 

[FrgMstr]

Personally, I don't like voice UI, and prefer typing.

Exackery. I do not want that on my desktop either...

 

[1.1.2.3.5...]

Whisper is pretty good for speech to text, but it won't work with a wake word. Personally, I don't like voice UI, and prefer typing. Although, I was impressed with SesameAI's demo, which had a model that could be interrupted. The problem was that the LLM they used was small to be real time, and it was relatively dumb. The voice part was very human with sighs, and pauses.

All fair, but i want my Jarvis!!!

 

[stuffmonger]

Sorry, no Jarvis in stock. Best I can do is Ani, Grok AI waifu.

 

[LukeTbk]

I don't think anyone could build something that would compete against what data centers have in them, but they don't need to. Also, the electricity cost is overblown since the machine clearly won't run full tilt 24/7. You could get a Panther Lake machine that draws 1.5W power while idling and put in some RTX GPU and let it idle all day long. Or just shut it off and send a magic packet to wake it up?

if it does not run 24/7, it become harder and harder to amortize....

Deepseek v4 flash api cost:
https://api-docs.deepseek.com/quick_start/pricing

$0.14 input, $0.28 output per millions, cache virtually free.

If you run at 60 token per second and you bought a $1600 machine to do thing at home, you need if electricity was free youwould need about 8,000 millions tokens to pay for it, at 60 token per seconds that's 37,000 hours or 1,540 days

To run the 27b qwen model I use the minimal ec2 node would likely be a 4xl40 or a10g. We are talking ~285 a month for continuous usage. I could try to grt fancy and build a load balancer that turn it on or just log into the aws cli and launch it when i want to. But that's a lot of work and it sucks.

It would be more using deepseek flash api for what the 27b qwen model do, would have been probably cheaper than running 3090rtx even at good price, you do not really need the virtual machine and to put the RAG on it... still have a local cheap laptop do what the non LLM part of the machine is doing.

A lot of local AI hardware stuff tend to be people that love building computers, setuping them and so on, finding an use case/project to do in my experience, a bit like our Plex server stuff before.

 

[1.1.2.3.5...]

How about ai powered NPCs? I would imagine gamers want a highly responsive npc experience and running a 7b llm on the cpu (should be trivial for zen 6 with avx and 24 core to get over 100 tokens / sec) and that would give the game a lot more life. You could use cloud ai for that, but now you're giving people a different experience based on their ISP latency etc....

 

[stuffmonger]

How about ai powered NPCs? I would imagine gamers want a highly responsive npc experience and running a 7b llm on the cpu (should be trivial for zen 6 with avx and 24 core to get over 100 tokens / sec) and that would give the game a lot more life. You could use cloud ai for that, but now you're giving people a different experience based on their ISP latency etc....

View: https://store.steampowered.com/app/3730100/Whispers_from_the_Star/

This game did quite well. It made the input lag believable with the space premise. No need for that anymore with something like Sesame.

 

[LukeTbk]

How about ai powered NPCs? I would imagine gamers want a highly responsive npc experience and running a 7b llm on the cpu (should be trivial for zen 6 with avx and 24 core to get over 100 tokens / sec) and that would give the game a lot more life. You could use cloud ai for that, but now you're giving people a different experience based on their ISP latency etc....

latency would not come from the ISP outside extreme case, but the AI server queing/overload of demand during peak hours, under 20ms ping is quite common and the data could be quite small, people after all play very fast twitch competitive games with a server between them.

Having the gamers pay for it with its local hardware and not having to manage anything using local AI will be a popular solution regardless of latency here.

running a 7b llm on the cpu (should be trivial for zen 6 with avx and 24 core to get over 100 tokens / sec)

9950x tend to go under 10token per second on text generation on the Qwen 8B/7B type of model I think ? They could have a lot of power but would they not face the hard limit of dual ddr5 bandwith ? seem it would even with 8000mhz ddr5 be an hard cap around 20tk/s regardless of how powerfull the cpu is. Very small model like that could run at over 400 t/s on the cloud and end up having much better latency before enough response get usable than CPU.

 

[1.1.2.3.5...]

latency would not come from the ISP outside extreme case, but the AI server queing/overload of demand during peak hours, under 20ms ping is quite common and the data could be quite small, people after all play very fast twitch competitive games with a server between them.

Having the gamers pay for it with its local hardware and not having to manage anything using local AI will be a popular solution regardless of latency here.


9950x tend to go under 10token per second on text generation on the Qwen 8B/7B type of model I think ? They could have a lot of power but would they not face the hard limit of dual ddr5 bandwith ? seem it would even with 8000mhz ddr5 be an hard cap around 20tk/s regardless of how powerfull the cpu is. Very small model like that could run at over 400 t/s on the cloud and end up having much better latency before enough response get usable than CPU.

Nonsense. There's nothing keeping a 7b from 100s of tokens / s on zen 6. Larger cache, sea of wires interconnects, 24 cores, wider avx decode port, dual imc and faster ddr5. (this is all not confirmed but that's currently what the rumor and leaks are suggesting)

 

[LukeTbk]

Nonsense. There's nothing keeping a 7b from 100s of tokens / s on zen 6. Larger cache, sea of wires interconnects, 24 cores, wider avx decode port, dual imc and faster ddr5. (this is all not confirmed but that's currently what the rumor and leaks are suggesting)

memory bandwith would... they could reach well over 100s of token doing a queue batch size of different task, but memory bandwith set maximum token speed for a model size. That why a Nvidia Spark can have a lot more tflops but cannot be faster than an apple M4 usually, because memory bandwith is the same.

Or did those things changed a lot ? if you have smaller sub-agents, then sure. (CPU with only dual channel will stay a bit under ~100 GB/s here)

If there 6 NPC, you can do a batch size to use more of the CPU and go in the over 100 token/s, but each individual will be limited by the bandwith and going to a max around 20s... no ? and that what will determine how fast an NPC react, its individual task time not the global throughput.

That why a spark has a lot of tflops but its 275gbs limit it quite a bit versus a gpu with gddr7.

 

[1.1.2.3.5...]

memory bandwith would... they could reach well over 100s of token doing a queue batch size of different task, but memory bandwith set maximum token speed for a model size. That why a Nvidia Spark can have a lot more tflops but cannot be faster than an apple M4 usually, because memory bandwith is the same.

View attachment 808526

Or did those things changed a lot ? if you have smaller sub-agents, then sure. (CPU with only dual channel will stay a bit under ~100 GB/s here)

If there 6 NPC, you can do a batch size to use more of the CPU and go in the over 100 token/s, but each individual will be limited by the bandwith and going to a max around 20s... no ? and that what will determine how fast an NPC react, its individual task time not the global throughput.

That why a spark has a lot of tflops but its 275gbs limit it quite a bit versus a gpu with gddr7.

You only need 1 llm to interact with at a time. The npcs will effectively be the same model with different agent scripts/notes. MTP and cache size should help a lot here. Will see when it comes out but my money says it will have no issues running a 7b class model.

 

[stuffmonger]

I bet most people don't pay much attention to NPC dialog. It makes more sense for a therapy app. This means it needs to run on a phone, so everything needs to be remote, which is expensive.

 

[1.1.2.3.5...]

I bet most people don't pay much attention to NPC dialog. It makes more sense for a therapy app. This means it needs to run on a phone, so everything needs to be remote, which is expensive.

On a game like Fable, the NPC interaction is great even scripted. Add something with more depth? yes pls

 

[LukeTbk]

when they can take them to actually script them, they will be good (and better than LLM), it is for the many that are super generic that it could work.

An in between could be nice, it use the script but change things a bit (tone and some wording) depending of the current exact situation (if you are bloody-injured vs fine, things that the NPC just saw-went through, that cannot be all scripted in advance).

 

[kram182]

I want AI to deliver real-time chaos effect weather systems - a butterfly flapping its wings on one side of the (world) map affects particles etc that lead to a rain storm on the other side of the (world) map

 

[stuffmonger]

On a game like Fable, the NPC interaction is great even scripted. Add something with more depth? yes pls

In Postal 2, the fun bits of NPC interaction are visual, and mostly non-verbal. Kicking, tasing, decapitating, setting them on fire, pissing on them.

 

[FrgMstr]

How about ai powered NPCs? I would imagine gamers want a highly responsive npc experience and running a 7b llm on the cpu (should be trivial for zen 6 with avx and 24 core to get over 100 tokens / sec) and that would give the game a lot more life. You could use cloud ai for that, but now you're giving people a different experience based on their ISP latency etc....

That was the reason I went down the RPG path first. Had it working. Just was not compelling the way I built it, and quite frankly, I had not "upped" my coding LLM at that time.

 

[AVATARAT]

How about ai powered NPCs? I would imagine gamers want a highly responsive npc experience and running a 7b llm on the cpu (should be trivial for zen 6 with avx and 24 core to get over 100 tokens / sec) and that would give the game a lot more life. You could use cloud ai for that, but now you're giving people a different experience based on their ISP latency etc....

Responsive NPC with AI model in games?
Disgusting!

 

[Nobu]

Not gonna say it'd end badly, because models have improved since, but it has been done before. it was...interesting.

Don't think I would like a generative AI for dialogs/interactions in game. I think they could and would break immersion. But at the same time, sometimes that can be fun too. I guess it depends on the game.

 

[LukeTbk]

Don't think I would like a generative AI for dialogs/interactions in game. I think they could and would break immersion.

Their absence can break the immersion has well, take Baldur Gates 3 with everybody talking with a voice over except the main characther, with generative AI now they could have went around the challenge to have to record so many possible voice because of the custom characther creation.

Text-to voice is a different form of course than generating what is being said... but it could also tweak a main script to customize to the specific in-game situation, to reflect better what is going on, often cut-scene dialogue will not match the current injury level of everyone or does make sense in the context of what just actually happened, it could be one of its strenght, worse dialogue but way more immersed into the situation, because it took everything that just happened and currently going into account.

 

[Ididar]

Their absence can break the immersion has well, take Baldur Gates 3 with everybody talking with a voice over except the main characther, with generative AI now they could have went around the challenge to have to record so many possible voice because of the custom characther creation.

Text-to voice is a different form of course than generating what is being said... but it could also tweak a main script to customize to the specific in-game situation, to reflect better what is going on, often cut-scene dialogue will not match the current injury level of everyone or does make sense in the context of what just actually happened, it could be one of its strenght, worse dialogue but way more immersed into the situation, because it took everything that just happened and currently going into account.

The dream sounds awesome. But, as with everything, the devil is in the details. If the characters start acting way out of context or with different personalities to the script then it will ruin the game for a lot of people. But, you're right, that at times some NPC's seem oblivious to what is going on. So, I think in this case we'd be trading one good thing for another good thing and might not get a gain overall. I think whatever they do has to be done slowly so it doesn't ruin the game in the process. But, given the current climate I expect some major screwups before they sort it out. The whole "go big and break things" is the modern technical mantra.

 

[LukeTbk]

But, given the current climate I expect some major screwups before they sort it out. The whole "go big and break things" is the modern technical mantra.

It depends, one could view about the current context, AAA cost a lot, extremelly risk averse and the complete opposite in general of go big and break things.

Very small games will use those firsts, break them a lot of course, but I do not imagine we need to worry too much that the Call of Duty, Elder Scrolls of the world will try to go big with pushing unproven enveloppe, that was more a 80s/90s mantra than the modern one in games.

Modern technical mantra tend to be you can pick a controller and start to play a game without reading a manual because of how similar to what you already played in the past, with everything working fine.