I am so happy I haven't written a line of C++ in like 15 years. Absolutely disgusting language. Every time I look up one of their new standards, I'm like how does anyone keep all of this in their head (usually on top of stuff like boost, etc.)? No wonder LLMs are a thing.
C++ is a decades-long mistake that spawned other misguided directions in the entire software industry. We're still trying to undo many of the bad ideas, but I'm afraid even Rust has too much of C++ in it, not only in terms of syntax but mentality and culture. LLMs are making things worse by automatically generating code in such bloated languages, where people have less need to even look at the horrific spaghetti inside. The whole thing needs to be reconsidered from the ground up, from first principles, by actual creative thinking human beings with lessons of hindsight.
> Gall's Law: A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
One-shotting is a bit of a red herring, imo. Let's say I wanted to build a "Super Dario" meme platformer. I would do some research, find a platformer on Codepen or Github that "feels" good, tweak it to my liking, change the sprites, and voila.
I constantly have to tell agents to just "look it up online" and "don't hallucinate your own components" because people have already done this a million times. Ironically, being more lazy could make these models more useful.
So you're saying that it's not fair to rate a coding model on its ability to code, and instead the best way to use it is to tell it to find existing human-written code online rather than generate actual code on its own?
If you're testing how good LLMs are at compressing information, then I think that's a fair test. Personally, I don't really think that's where their strength comes from (especially considering how much more useful local models that are orders of magnitude smaller than Claude/OpenAI-tier models have gotten). In other words, we already have a "super-intelligence"—it's called the internet, so just use the darn thing.
Strange, I use M3 and Sonnet a lot, they never hallucinated. Gemini used to make up some function names, but it was in long context windows. My AGENTS.md does specific they should use context7 when unsure about a library, but still…
Pangram doesn't work, and I wish people would stop treating it as gospel (but the AI/anti-AI grift is real). Here's a fun paradox: I can literally tell ChatGPT: "Say X" and it will say "X"—so that's a case where content is both AI generated and not. What if it changes a few words? Moves some sentences around? Where does something go from human- to AI-generated? (This is the classic Sorites paradox.)
Pangram tries to look for common patterns (rule of three, em dashes, etc.) but these are heuristic methods and not to be taken as gospel. There is no provable method to make a distinction between AI and human-generated other than the fact that AI-generated text tends to reek of pseudo-intellectual undergrad with a thesaurus.
Pangram does work, in the specific sense that when it says something was AI authored it is vanishingly unlikely that it was written by a human (who was not deliberately trying to write like an AI), and IMO getting people to recognize that we actually do have a decent solution in this space now is pretty important if we want the Internet to remain a place for humans and not just bot swarms.
> rule of three, em dashes, etc
You appear to be misinformed about how Pangram specifically works, it is not based on pattern detection of that sort. I recommend reading their whitepaper, it's a pretty understandable explanation of exactly how they trained their classifier.
I just tried it, created an account, and wrote a few sentences about how my day went. These sentences got classified as AI assisted, so clearly their classifier doesn't work that well.
It's trivial to see how many people think Pangram is absolute trash[1] (because it is).
> You appear to be misinformed about how Pangram specifically works, it is not based on pattern detection of that sort. I recommend reading their whitepaper, it's a pretty understandable explanation of exactly how they trained their classifier.
I did read their paper (which is, by the way, very scant on details), and they trained their classifier in the laziest way possible: here's a chunk of "human-written" text and here's a chunk of "AI-written" text, put them in the right bucket, and do this a zillion times. Literally zero sophistication. Also: what do you think "pattern recognition" is, if not a "classifier"?
Windows has its own library built-in: NLS. There have been a lot of back-and-forth about whether to use NLS or ICU in dotnet on Windows because it resulted in different behavior between Windows and other platforms. I think they ended up using ICU by default from .NET 5 on.
This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.
Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.
I'm using Kokoro for a fun little side-project browser-based game I'm working on. It's legitimately super good for being only 85mb (for the wasm version) or 300mb (for the webgpu version).
"Same tick" is a misnomer for a few reasons. First of all, games use UDP, which is basically a "fire and forget" protocol (which means packets get dropped routinely). Second of all, realtime games use some interpolation/prediction to make up for latency (and aforementioned dropped packets).
So it's sort of a "relativistic" temporal system, not a linear "oh now you're at t=1, now you're at t=2" kind of timeline. And there's all kinds of complicated ways you create concensus between multiple clients, between server and clients, etc. (A lot of this remains an active research area.)
A tick is the smallest amount of time the server does its "work" in. It does not mean, as the person I'm replying to was implying, that clients are 100% always lined up, because a game is played (locally) at much higher update rates.
So a server, of course, does send updates (player position, etc.) every "tick," but that doesn't matter. Even assuming zero dropped packets (suppose we're playing over TCP), it would feel like shit (stuttering, rubber-banding, pop-in, jittery physics, etc.) to play a game over a ~60ms latency that updates ~60 times a second vs other people that also are ~60ms from the server, so game engines do a lot of interpolation and servers are in charge of concensus. Hence, it's a bit of a misdirection to say: "can't you just send everyone the right player data every server update?" because servers obviously already do that (and that's not really the hard part, anyway).
Local interpolation and remote concensus is the hard part, and games handle this differently. In CS, for example, two players cannot kill each other (with guns) simultaneously. Valve's engine requires that someone always wins a gunfight (which sometimes can feel random). However, I would argue that feels way better than, e.g. in Halo, where you can headshot each other (and both players die), which feels dumb and frustrating.
So when building these servers, there are lot of tradeoffs to consider (a lot of which might change the feel of the game).
In the context of this fog of war thread when we're talking about pop in and peekers advantage. The relevant bit is to make the game fair for the both of them. For FOW its trivial to make the decision to reveal both players to each client during the same tick. Yes I'm aware of all the other knobs and gameplay possibilities. Ive got a shitload of hours and followed pro scenes for a long time, none of that matters to this fog of war anti cheat program.
> to reveal both players to each client during the same tick
And I explained why this is not feasible given modern network topology. Players move around maps and peek around corners in between these "tick updates" the client gets from the server; therefore, we have to use clever methods of interpolation and prediction as well as server consensus to ensure that gameplay is smooth (e.g. models can't just pop in out of nowhere, movement feels good) while also being as fair as possible (e.g. we don't favor one player's view over another's).
The fog of war does this by revealing early. Counter Strike devs decide the interpellation implementation and if it's balanced. As I said. The goal of the anti cheat program isn't to change the game balance
This has been done before in both 1.6 as well as Source. I helped with some of these implementations back in the late-2000s when I was playing professionally and I even tried to kickstart an anti-cheat hardware solution about a decade ago[1].. spent way too much time working on some of these problems. The main issue with occlusion was slightly increased latency, visual jitter because of interpolation (especially around corners), and a few other more technical problems[2]. It's good enough for public servers, but not tenable in serious competition.
Cheating has always been a problem in FPSs, and it likely won't go away. That's why premier competitions have always been on LAN.
[2] Hard to fully obfuscate audio sources, hard to obfuscate hitboxes since you still need them for collision checking (e.g. if a grenade bounces off an enemy player behind a wall—the server does not do all physics for all clients), and this is on top of the engine itself sometimes requiring actual entities, so you're stuck with these dummy entities in memory, and so on.
I was on the other side of this in League of Legends. They used to have packets leak information to the client about player position. The classic examples are for stuff like particle emission or spell usage. Well at some point they decided that they had enough and did a pretty major rework of their netcode to only send player positions when you were supposed to have vision on them (plus a tiny extra radius). That absolutely wrecked the determinism of the game, as small jitters would cause skill shots to hit you before they even drew in your game, players would become invisible as the packets arrived out of order. Particle emitters would bug out.
This stuff i way harder than people imagine. I think League eventually got it somewhat figured out, but it took a couple of years from what i recall.
I wonder why similar methods haven't been employed in MMO servers to help curb botting/cheating there. The impact of jitter and loss of smoothness in a tab-target game like WoW would be minimal (even in PvP), because it's always had a noticable level of artifacts like rubberbanding.
I don't know how it'd be a huge advantage in MMOs, but in Ragnarok the bots used a re-implementation of the client and custom servers tested for differences and culling of entities by the original game client to ban bots (as they'd try to interact with entities that should've been culled)
I haven't played WoW, but I played other third-person MMORPGs, and usually by design you can freely rotate the camera and see behind walls. The "fog of war" is simply a distance around the player. So they do implement this, it's just not nearly as complicated to implement as in an FPS.
WoW has a lot of mechanics involving line of sight (eg a spell you begin casting when the target was in LOS will fail when the casting time finishes if the target moves out of LOS) and positioning (AOE spells, spell ranges, NPCs/objects you can interact with only within a certain range, among many others)
Because what do you think matters more in an online game:
A. Smooth and consistent client experience, where bullets hit what you aim at (client-side prediction) where aimbots and wallhacks work.
B. Jittery/laggy client experience, where aimbots still work, but wallhacks are disabled?
You can only choose one option.
Generally, everyone agrees "A" is the best option and cheaters will be dealt with at game time. It's annoying, but that's the cost of online video games.
It isn't an unsolvable problem though, there's only jitter because the implementation isn't very good
Implementing a more conservative anti wallhack cheat where player positions start streaming in slightly earlier still significantly cuts down on the efficacy of wallhacking, while entirely avoiding the jitter problem. Characters in CSGO move at a fixed speed, so you can calculate exactly how many ticks in advance you need to start sending that data in, before they will become visible to another client around a corner - and add a margin
There's also no excuse for sending player positions through a smoke for example - the server should be performing visibility culling
With the two combined you could cut the utility of wallhacks by 80%, in a way that would be completely unnoticeable. The real reason that this has never happened is that valve's investment into anticheat has always been pretty minimal compared to what's necessary for it to be effective (VAC has generally been the least effective anticheat). They're a small company which largely develops steam as a platform, not VAC or anticheat solutions for games
They’d need to either poach folks from other firms or go for a third party solution.
Cursed timeline: Valve pays Epic for good EAC build on CS2 + whatever encryption solution they have for Fortnite executable, 99.9% P2Cs disappear day 1
Also even if it worked perfectly, there'd still be other practical ways to cheat. In a way it's better to detect cheats than prevent them, because you waste a lot more of their time sticking them into a pool of cheaters and can also undo it. Though afaik Counterstrike tells you when you're low-trust.
The many tradeoffs involved are not trivial, this can only feasibly work well on LAN.
This is the reason why Valorant is the least playable among all competitive shooters if your internet is anything lesser than Google campus fiber, ironically in spite of having even-slower-than-CS movement physics on its side to mask the problem.
Riot conveniently cherry picks the best case scenario and handwaves the actual technical tradeoffs in their smug "we solved peeker's advantage!" engineering blog posts that are really just barely-disguised monorail Gish gallop.
Yup, I just reread the post you are talking about (https://www.riotgames.com/en/news/peeking-valorants-netcode) and it really seems like just endless fluff and no real solutions. Reading this post is almost unbearable and it all just comes down to this part:
> We built Riot Direct, our own internet backbone, to minimize network routing delays and processing time across the internet.
> We’re standing up VALORANT servers around the world, ultimately aiming to deliver 35ms ping to 70% of our player base.
> We optimized our servers to provide a smooth 128 server tickrate to all players.
> We optimized the VALORANT game client to run at 60FPS on most machines from this decade and higher framerates for players with high refresh rate monitors.
> We run clients and servers running with minimal buffering, targeting one buffered frame of movement data for clients and an average of half a frame of movement data on servers.
This seems like basic stuff that every other game does.
This must have been CS 1.4 or 1.5 times. I had terrible internet. 36k modem times, super high (ish - 150+ms at the very least, been a while, don't quote me, maybe it was way higher even?) ping to a server that had all the custom maps I loved.
I got accused of cheating.
I didn't.
Simply a result of CS/HL netcode. I was able to "run circles" around others. Something in that version favored my local client view of things I suppose. I literally just went around corners and if there was an enemy I circled them while driving bullets into them w/ my P90. They died without even really seeing me (from what they were saying). On my end everything looked smooth and normal.
This tactic never worked all that well on servers where I had the usual <120ms ping. I guess never mind at <15ms :)
Valve also implemented this on CS:GO back in 2015 [0], and enabled by default in competitive servers, so I would consider it absolutely tenable (FACEIT, the platform used by competitive variants, had their own hand-rolled SMAC implementation, before using the 1st party solution, albei,t that solution was buggy). Why Valve didn't port this over to CS2, I will never know.
Always confuses me why people speak so authoritatively on topics they aren't versed in. PVS culling is not even remotely comparable to occlusion culling, mainly because wallhacks are not relevant accross the map; in fact they are only useful when opponents are always well into your PVS range.
FYI: there are also some clever ways to get around PVS culling (mostly by inferring opponent position based on other indicators, like gunfire).
PVS-based culling is very different from the kind of fog of war system in discussion here. The cases that these FoW systems purports to deal with would all be "pass" cases for PVS culling.
PVS culling solves the informational/strategic advantage aspect provided by wallhacks, but not the pre-aiming reaction-time advantage in a peeker vs holder scenario.
I'm an engineer and data professional interested in team-building, consulting and architecting data pipelines. At Edmunds.com, I worked on a fairly successful ad-tech product and my team bootstrapped a data pipeline using Spark, Databricks, and microservices built with Java, Python, and Scala.
At ATTN:, I re-built an ETL Kubernetes stack, including data loaders and extractors that handle >10,000 API payload extractions daily. I created SOPs for managing data interoperability with Facebook Marketing, Facebook Graph, Instagram Graph, Google DFP, Salesforce, etc.
More recently, I was the CTO and co-founder of a crypto gaming startup. We raised over $6M and I was in charge of building out a team of over a dozen remote engineers and designers, with a breadth of experience ranging from Citibank, to Goldman Sachs, to Microsoft. I moved on, but retain significant equity and a board seat.
I am also a minority owner of a coffee shop in northern Spain. That I'm a top-tier developer goes without saying. I'm interested in flexing my consulting muscle and can help with best practices, architecture, and hiring.
Would love to connect even if it's just for networking!
An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says:
> temperature 0.1 — low, supposedly nudging the model toward deterministic outputs
This is not correct (and is briefly touched on later in the piece when he sets temperature to 0), temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution).
In theory, temperature 0 does make the LLM deterministic.
Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample (using the actual formula that works for non-zero values would cause a zero division).
However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.
>in theory theory, temperature 0 doesn't really exist.
It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check that directly chooses argmax instead of calculating the limit of some function that includes 1/T quotients. But setting T to zero is in both, theory and practice, turning the usual probability function into greedy sampling.
> Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function.
In pure math, it does not always do that. It becomes a dirac-delta comb with equal weight on every maximum. There can be more than 1 maximum. Setting the temperature to zero turns into greedy sampling, but greedy sampling is not necessarily deterministic as you can have multiple equally optimal options.
That is not a problem for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal. That's why greedy sampling generally produces deterministic output for LLMs. The real gotchas are elsewhere (like with batch inference as we've seen with earlier GPTs). But unlike what the earlier comment says, this is a non-issue mathematically.
> That is not a problem for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal
Any two tokens ending up with the exact same logit is very unlikely, but not impossible; and as the number of output tokens grows, the odds that it will happen eventually gets higher and higher.
I suppose, to ensure determinism, rank by logit then token ID, so you still have a deterministic winner even if occasionally two tokens get precisely identical logits.
You aren't looking for a random set of tokens that have the exact same logit, you are looking for the largest n tokens to have the exact same probability.
This is exceedingly unlikely, as training will only push one of them up for any individual sample. There are likely some pathological situations that could end up with that situation, maybe, but it is pretty unlikely in a general case.
Exactly, consider the scenario where laws are at play and violating them could cost companies thousands. Recently my father received a 'request for address' letter addressed to me at his nursing home, the building has always been a nursing home, and he's also in his mid-70s. That's very obviously a violation of the Fair Debt Collection Practices Act. Imagine the implication of this if the law firm in questions used an AI-assisted data enriching product to find this information. That SaaS company is not only liable to that one law firm but every law firm who uses their software. Its potentially a federal class action lawsuit.
My point is, deterministic logic matters in certain circumstances 100% of the time. Forcing the LLM to make something unlikely is not good enough because a series of mistakes could very quickly bankrupt the company.
>My point is, deterministic logic matters in certain circumstances 100% of the time. Forcing the LLM to make something unlikely is not good enough because a series of mistakes could very quickly bankrupt the company.
If your argument is that the danger of equal values being selected inconsistently breaks determinism, that's a trivial problem to solve.
Any non-infinite precision numbering system by definition is at the limits of it's precision when equal values occur. If you need to order such values you can extend the precision and add on a deterministically unique tiny value (position, order encountered, etc.) . Your original value stays in the same precision range but they are now unique.
It's usually more likely that you want to sacrifice a little precision for determinism so you can quantise to allocate the range where you apply the unique ID
For example if you had an array of 256 fp32 values but you required them to be unique, you can lop off 8 bits of mantissa and replace it with its index in the array, Every value is then unique.
Granted token dictionaries make for some fairly hefty indexes now, but the principle applies in general, it's easily solvable if you are prepared to spend some precision or do some extra calculation.
> for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal.
In one thinking trace of 10k tokens, with fp16 or bf16 logits, I don't reckon a collision is rare? There are only 65k floating point numbers with that accuracy. And an agent can quickly rake up 100k tokens, so while not every token will have such a collision of equiprobable logits, it is not rare.
> It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check that directly chooses argmax instead of calculating the limit of some function that includes 1/T quotients.
I don't understand the distinction you're drawing. A Dirac delta function is a "simple if check".
The point is that the case T=0 doesn't just "exist" as a special code branch - it is still well defined mathematically without any change to the output function. What the above comment refers to with the extra "if" check is just a limitation of computers not liking to divide anything by zero, even if the actual function exists and is well behaved at zero. It is not some weird or special theoretical construction.
In almost all forms of math, the value n/0 is undefined. It's definitely not infinity, for two reasons - depending on the value of n, it can be negative; and neither info nor -inf are numbers, so they can't be the result of an equation (unless you look at transfinite equations).
What you can do in math is talk about the limit of a series of fractions as the denominator approaches 0, and that's where you get some relation to infinity or -infinity. But the limit can also be any other number, if the numerator also gets closer to 0; or it can not exist, if the function oscillates.
It's not positive or negative infinity. It is simply undefined. Math has many conventions, and you can define your own convention that it does equal some flavor of infinity, but that is only a convention, and not a universal one.
That's not the problem, and this is not just pedantry. It's just not correct to say that n/0 = inf, nor even to say that positive_n / 0 = inf, in any normal math context.
For example, if you accepted that n/0 = inf just like n/1 = n, then you'd conclude that n/0 + 3 = inf + 3 = inf, so n/0 + 3 = n/0, so 3 = 0. Or you'd want to do weird things like asking what is sin(inf).
Which is the case with softmax function, as for T=0 you end up with a fraction that either becomes 0/0 or inf/inf [0]. So you do need branching as floating point arithmetic is not gonna get you there.
[0] except for weights that are exactly 0
edit: thinking more about it, one could always express the softmax formula in ways that this could work with floating point arithmetic but it would be very inefficient and sort of pointless
> Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0.
That's not how limits work. As the temperature goes to 0, the rest goes to 0. That's it. The "almost-but-not-quite" is part of the "goes to".
Let's say f(x) = 3x+1. It's a continuous function. If we let x go to 10, f(x) goes to 31. Not "almost-but-not-quite 31". No, to 31. (If you don't have a continuous function then it's the same argument, but less intuitive to illustrate.)
Even if it's deterministic that doesn't mean it isn't arbitrary. I can achieve determinism at any temperature by saving the seed. But that wouldn't make rejects feel much better knowing that if a bit was flipped in an arbitrary seed they would be scored differently.
> However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs.
Exactly. While I’m assuming this won’t be news for most here, for those that are still new and/or curious about some more explanation on e.g. the floating-point imprecisions, see this nice article: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware one), and if you where to swap out that with something deterministic you'd get the same results every time (barring non-determinism in other parts of the OS/drivers/maybe even hardware).
But theoretically, the output of every LLM is seed-driven (or could be if you wrote the software to isolate it) just like any computer software. It's just none of the software written (even llama.cpp AFAIK) chooses to support stable-seeding due to the changes in stuff like CPU/Vulkan/CUDA/Metal differences making it difficult to make consistent.
They could though! Hopefully one day someone implements it into the mainstream LLM-engine software and it gets exposed in the APIs serving the models. It'd do a lot to show folks the "internals" of these models.
It's probably due to the fact that it's a cloud service. You have no guarantee that your next request will go to the same machine. So even with an identical seed, and temp 0 you might get different hardware and hence different accuracy/noise in the floating point operations.
How can there be noise in floating point operations? I could buy like completion order for parallized batches i.e. adding a+b+c is different from a+c+b etc.
IEEE-754 doesn't mandate exact results for functions like exp(x). It mandates things like "within 2 ULP of the true answer." Hardware vendors are free to implement these functions in any way that meets the error tolerance.
Batching order, as you mentioned, matters a lot, and for any heavily optimized kernels it will change from one machine to the next. You also have the choice of backend numerical library from, e.g., different OS versions. There are floating-point bugs from time to time, especially in GPUs. Many operations (like transcendentals) are usually given a couple bits of wiggle room in the result. Another program executing could have changed the floating-point rounding mode on one device. More aggressive ML optimizers might automatically apply various forms of reduced precision to the requested high-level operation. If you have enough optimizations enabled, you might non-deterministically get compiled instructions like fmadd so that any one build of your library is deterministic (excluding other ideas mentioned above) but different machines with different builds (because of a staged rollout, different architectures, engineering mistakes, etc) can have different outputs. And so on.
While the IEEE 754 standard ensures that individual basic operations are deterministic and strictly bounded, it does not guarantee that an entire program will yield bit-identical results on all CPUs.
CPUs and their execution environments introduce subtle hardware variations, architecture choices, and compiler optimizations that break bit-level consistency.
Parent is correct - the math is very deterministic if you can guarantee it’s running repeatedly on the same machine and you’re not processing “random” requests in parallel. The compiler is irrelevant because once the code is generated it’s not getting recompiled and thus isn’t a source of non determinism (and generally if you don’t touch the math the compiler will consistently emit the same underlying machine code).
This sub-thread was about cloud environments, where different requests may be served by different hardware. And it's in fact very likely that there will be a mix of different hardware from different vendors, in any particular LLM cloud for now.
It is, after all, a fundamentally voltage-based process, and the logical “no-man’s land” is chosen to limit the likelihood of a weak component producing faulty logic, but it’s impractical to run through the set of all possible starting states and to verify that after an unbounded number of clock steps the machine reaches a predictable end state on all of the devices being manufactured.
Stable seeding is not enough. A lot of modern, fast compute kernels are nondeterministic. Floating point multiplication/addition is not strictly associative and e.g. reductions can combine results from different threads in different orders (e.g. through atomic ops). You can write kernels to be deterministic, but it is generally less efficient.
They are only non-deterministic when you’re doing batching and a kernel ends up running across a “random” set of token streams. If you’re only processing one user’s request, they’re very much deterministic.
I did large scale tests temp 0 and there was still randomness with the same prompt inputs coming in.
I did this with several model apis.
GPU processing is not going to be the same from what I read but also the AI backend is doing a lot of fancy batching resulting in another layer of randomness.
It is not deterministic because the order of computations in a typical multithreaded system is not deterministic and also because when combined with the devil that is IEEE754, it gets even less deterministic.
If you make an exact integer implementation and run with temp=0 it's deterministic.
You don't even need temperature 0, just make a random seed for the sampler part of the input and then its deterministic as a function of the input.
But running autoregressive models at temp=0 tends to expose pathological behavior, because the training process produces a function with a lot of gain so its prone to feedback on its own noise.
> However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run
The implementation does not often differ run by run.
> The implementation does not often differ run by run.
If you use a cluster, or even multiple clusters, and they have non-identical hardware, then two consecutive runs could end up being routed to nodes having different GPU models with slightly different floating point behaviour, or even software differences (e.g. newer GPU offers some feature usable to speed up calculations which older model lacked; same code can use the feature when it is available, fall back to slower alternative if it isn’t). The larger your scale, the greater the odds it will happen
To be clear, temperature 0 is deterministic and will produce the same output for exact duplicate inputs, across all seed choices.
Provided:
* If it’s MoE we are talking about, that the duplicate inputs are for the whole batch (yes, your batch neighbours can impact your choice of experts. Blergh.)
* Your kernels are deterministic
* There’s no system wide effort switch that responds to, e.g. work load across the cluster (for a thinking model)
Upshot:
Temperature 0 is not deterministic in probably any existing cloud infra, but it could be for edge inference pretty reliably.
To your quibble on 0.1 being more deterministic - I think it’s a pretty fair summary - we’re going to sample much more from the ‘temp 0’ answer at 0.1 than we would at temp 0.9, no?
Even then it's deterministic in the way a hash function is deterministic. Change one letter and you can get a completely different output. What people actually want is something continuous.
Agreed on the desire for continuous behavior. That said, in a modern LLM, is this hash analogy accurate? I would be surprised if a single letter changed most zero temp force ranked outputs.
E.g:
“Where is the Eiffel Tower Located? One word only.”
“Where is the Effel Tower located? One word only.”
“Where is the Eiffel Tower located? One wor only.”
I’d be very surprised if those got different answers from even a small local model at temp 0.
The entire chain will be affected from the different tokenization on down. Even if it lands in roughly the same semantic area, it doesn't mean it will land there with anything like the same syntactic selections. Anywhere there were multiple near-tokens could easily select a different route based on even minor fluctuations in the starting conditions. It's chaotic.
I believe there's some data that they will have different responses if the names signify different cultural / race / gender affiliations. Here be dragons.
The whole problem of text understanding is a problem of reasoning under uncertainty, that is, you can't really be sure which witch people are talking about all the time. A person you might hire might be successful or unsuccessful at the role, no matter what hiring process you use. Two people might look at the same resume and come to the same conclusions. Two patients with the same symptoms and clinical presentation might have different diseases, etc.
I don't buy the story that the old AI died primarily due to the cost of knowledge base maintenance [1], but rather the lack of a universal system of reasoning over uncertainty.
For me it's a running gag that Spock was always saying things like "Captain, we have a 21% probability of surviving this mission" when Bayes teaches us your probability distribution has a probability distribution, "we have a β(5,1) chance of surviving this mission" is more like it.
To that end it wouldn't be too crazy to run a resume through that machine 100 times and look at the probability distribution of the score.
[1] then again I am the kind of maniac who will sort images on a tablet lying in bed until my visual system malfunctions
My favorite recent example was submitting a resume for a job that was almost a word-for-word description of my current title and job at a similarly sized company. Within 24 hours, I got the rejection, and several days later, a recruiter reached out to let me know that my profile looked like a great match for the role and wanted to schedule an intro call.
I remember applying for one position that was around building care and management systems for pregnant mothers - EHR, practice management, claims benefits, etc., all of which I had over a decade of experience in. "Name something that might stand you out from the crowd." "In addition to all this I've also delivered 12 babies as a paramedic". Twenty minutes later "we are looking for candidates whose experiences and skill sets are more closely aligned with the role we are looking to fill".
A distribution with all probability mass on one outcome is deterministic, so in principle, setting temperature to 0 _should_ result in deterministic outputs. There are a few reasons it might not, but I don't think any of these apply when running a local model like the author did.
> so in principle, setting temperature to 0 _should_ result in deterministic outputs
It is a common misconception, but it is not true even in principle. If I have 2 or more logits which are equal to the maximum of my logits, I will sample uniformly random from them with any temperature, even zero. Sampling from softmax([1, 0, 1]) is still stochastic at temperature 0, because the limit is to sample uniformly from the first or the last element.
Anyway: "GPUs don't do deterministic matrix multiplications" is the biggest source of randomness in LLMs. GPUs put the associativity of the sums in matrix multiplications in arbitrary order, and this has a huge impact on the logits coming out of the neural network.
> "GPUs don't do deterministic matrix multiplications" is the biggest source of randomness in LLMs.
But this isn't a fundamental property of LLMs, it's just an implementation detail. It's pretty obvious that if you evaluate the matrix multiplications correctly and deterministically sample from the highest-probability outputs, you will have a deterministic LLM.
It may be an implementation detail, but in practice, if the only way to get a deterministic output is to run on the CPU, then it's not going to be usable.
You don't have to sample uniformly. You could take the lowest index of all maxima.
But yeah, the main source of randomness is non-deterministic matmul, and temperature does nothing with it
The matrix multiplication is only deterministic for sparse-dense products under these settings:
> torch.bmm() when called on sparse-dense CUDA tensors
And it's not listed under the operations that raise an exception otherwise, so I'm not sure the docs promise that dense-dense matrix-matrix products are deterministic.
Setting the temperature to 0 should give deterministic results but that's not any better - it's just hiding the huge variance by only taking one sample.
There are. If the kernels are nondeterministic (e.g. timing issues) there are minor changes between runs, on a single system, even with eager decode enabled (typically what temperature=0 achieves).
I mean the easiest explanation would be that the model harness doesn't always take the most likely token but does top-k sampling or similar. temperatur just means that probabilities get more and more equalized, boosting the chance that an unlikely token gets picked. but even with temp 0 you could have 0.8 T1, 0.19 T2, ... and sometimes sample T2
No, this can't happen at temperature 0. The formula defining temperature-adjusted softmax isn't strictly defined at 0, but taking the limit (in the case where all logits are distinct) results in probability 1 being placed on the largest logit. Samplers will typically special case temperature 0 and pick the most likely token at each step.
This is a very authoritative answer that should be more nuanced and caveated as implementation-dependent. In some cases, repetition penalties take precedence over sampling; top_k and top_p can also be handled before or after the temperature step. In other cases, `0` is turned into like 1e-10 or some super tiny float value (which can drift if you do any arithmetic with it). Routing, quantization, etc. can also have an effect on sampling. And yes, in some cases, setting temperature to 0 can mean "pure greedy decoding" which makes the decoder about as deterministic as it can get.
A more spikey distribution exactly makes the distribution closer to deterministic. That's not the point though. Even in greedy (deterministic) decoding, it is still a black box though that reacts in ways ways that are unpredictable to the inputs. Switching one word around might lead to different scores for example.
Yeah, this is the forest that the people arguing about math trees are missing. It doesn't matter that the algorithm is deterministic if the algorithm passes the input through a cryptographic hash function to make a yes/no decision. The result may be perfectly reproducible and still non-sensical in its distribution with respect to its input domain.
Several of my claimed AI-expert colleagues repeat this as though it's gospel. I've heard "set the temperature to 0 so we get consistent results" more times that I can count.
I imagine it's much like game-developers saying: "Set a fixed seed so the player gets consistent results."
Yeah, it can work, but it is subject to so many potential pitfalls that you can't casually assume it will. It's a property you have to actively design-for and rigorously test to be sure the system can deliver it for some particular scenario.
> temperature is not some kind of "deterministic" switch, but rather it affects the sampling distribution (which becomes more "spiky"—but is still very much a distribution).
You're correct. The confusion arises because we use the word "non-deterministic" when we mean "probabilistic".
Small refinement: the underlying model isn’t stochastic at all. The forward pass is a deterministic function of the weights and input, it just produces a probability distribution over the next token. The stochasticity is an optional sampling step layered on top, not something inherent to LLMs. Greedy/argmax decoding (or temperature 0) makes the whole thing deterministic.
So “purely stochastic” overstates it a bit: the distribution is computed deterministically, and you choose whether to sample from it or not.
There are more layers to this problem, if we want to get into the details. The LLM is defined in terms of floating point operations, and those are not actually fully deterministic, on most hardware and in most performant implementations.
IEEE 754 only specifies precision requirements for certain operations, not precise bit patterns (e.g. for exponentials). So, at least in principle, the same hardware performing the same operation could produce different results at different times, as long as they are close enough to the theoretical answer. I'm not sure if any hardware actually works like this.
IEEE 754 also specifies that many of the basic arithmetic operations are not associative - so any reordering (which is common when batching multiple queries at the same time) will introduce indeterminacy from the perspective of your own query (that is the result for your query will change depending on what other query happens to be processed at the same time, which is not under your control).
Finally, even if we take the case when a query is processed alone, and even if one particular hardware is completely deterministic, the result will be different on different hardware - which can again look like non-determinism if you're sending your query to a load balancer.
So, the math for LLMs is deterministic in theory, but implemented with non-deterministic approximations & optimizations in practice, and their results are then normally used only as a probability distribution to be sampled from.
Every time people point out a limitation or constraint of LLMs, I see a comment that is to the effect of “but humans…”. I don’t understand why this comparison is relevant to this particular thread. Is it just an amusing similarity?
I think it often useful to push the conversation down "we built a system for humans that dealt with this, what from that is or is not applicable for agents in the same context"? Humans randomizing resume review for screening is pretty known; I've seen companies try to fight it with things like hiding information, panel reviews, etc - it's unclear to me how effective those would be for agents (honestly, it was unclear how effective those were for humans). I was depressed about the hiring process before we had AI screening and I remain depressed about it.
It may seem trite but the point is that if separate humans were assigned the same task the LLM was here the results would be similarly non-deterministic.
Indeed: LLMs do tasks that would otherwise be assigned to humans. So when pointing out deficiencies in LLM performance they should be compared to the alternative, which also isn't perfect.
We expect computers to be consistent on the other hand. A calculator will always give you the same answer unless some chip gets struck by a particle. LLMs are on computers and should be fairly consistent too.
Yeah but daily tools have lots of complexity which appears as non determinism (if we are thinking only UX, not actual determinism). For example, try moving an image in the word doc. I have been using MS word my entire life it seems, still don't know what the rules are lol.
What's even worse, different humans have different weights.
If you train two different LLMs and replace what data they "see" in batch n, that doesn't affect the data they see in batch n+1, or any further batches. In LLMs, you can introduce "noise" into the training process, but that noise doesn't really compound.
Humans learn from experience, not from data, and their experiences at age n shape what experiences they seek (and hence train on) at age n+1. A small amount of "noise" injected into their "training", let's say hearing a group of friends discuss a movie while their identical tween goes to the bathroom, can compound into them watching that movie, which can compound into them forming an identity around that genre, and so on, until they're two completely different people, trained on completely different "data mixtures".
The same person is not going to give you three different answers within span of minutes. Especially when nothing fundamentally has changed. People might or might not update their views depending on their biases.
I'm pretty sure the personality tests are created specifically for the reason that a single person can have fundamentally (or conflicting) beliefs about himself in a matter of minutes. You can say "I am honest person" and the next minute you can say "I never lie" - and both cannot be true for an average person.
There is evidence that children will oscillate between understanding and not understanding while learning topics. Philip Sadler at Harvard published about this but i can't find the paper im thinking of on his google scholar. too many papers!
but moreover, to verify a test item you need to make sure that peopel will select the same answers under teh same conditions at different times. people generally forget the specific questions they were asked if you ask them the same questions a month later so being able to get them to answer the same way each time is important. it is assumed the people have some static knowledge of a topic in this scenario.
If you want to consider a statistical examination of how people answer tests and how we assess knowledge and other things in people through surveying you can read about item response theory and rasch analysis.
how did they account for sampling bias? a judge might leave easier cases for after lunch. people with control over their schedules usually ease themselves back into it after breaks.
1.0 is "natural units". If your energy corresponds to nats, you should be using temperature 1.0. If your energy corresponds to bits, you should be using temperature ln(2) ~= 0.7. The optimization pressure is
max nats = max entropy + energy / temperature
Why might energy correspond to bits or nats? Imagine your goal is to play as many interesting games of chess as possible in a tournament. This implies you have to keep winning. If you look at the RL environment from the right perspective, you can turn it into optimizing bits or nats.
If RL was used to train the model, the model will have been trained on its own sequences. Those will have been generated with a temperature of 1.0. They must be, otherwise you would get a premature collapse or explosion of your entropy if the temperature was respectively lower or higher.
After that RL step, you want to stick to the RL distribution, and so keep a temperature of 1.0. Other temperatures will drive the model out-of-distribution.
That is why the sampling step for agents or thinking LLMs are usually kept at a temperature of 1.0.
It really depends on the application does it not? I'm not an LLM guy, but for creative tasks like storytelling wouldn't you want a higher temperature usually? Happy to gain insight from anyone with experience here :)
Heavily depends on the model architecture and the implementation though, I don't think you can say what values are better than others without first specifying those, otherwise it's straight up guessing, ironically.
Willing to be corrected but I believe this type of automated resume filtering is illegal. Not saying it never happens but my understanding is it is not typical.
I don't know for sure, but I would be surprised if it was illegal in my particular US state. You might be able to argue the AI has inherent biases that introduce illegal discrimination in the hiring process, but my understanding is winning I case like that would be very difficult, especially since most employers are very cagey about their hiring process and why they mades a decision.
In situations when you get hundreds of applications for one open position (real market now), whatever reduces your pool to the size a human can handle, works. You can preserve some diversity metrics in the process. This particular filtering is rather primitive, but LLM as a first filter can definitely do the job. You may burn less tokens than the hourly rate of your HR and it will be fairer than just dumping 50% of unread CVs in trash.
Great until someone realises you’ve filtered out minority groups from the application process (most developers are men so maybe the LLM decided they’re the best fit, but you’ll never know exactly why it screwed your over) and you suddenly have an expensive lawsuit
LLMs are DEI-aware, as over past few years, their vendors all had various high profile news stories with their models and their default biases, so it's more likely they'll heavily discriminate in favor of minority candidates, not against them. Still, in both cases it would indicate whoever is operating the system is doing a really, really lazy job. It's really not hard to test and supervise LLMs on tasks where they give you mere 2-10x leverage, and prompt adherence today is much better than it was 3 years ago.
Workday is facing a class action over claims their system is ageist. And another very large company dropped LLM use from their filtering, after finding it strongly preferred men over women, even when the females were objectively stronger resume candidates.
What „not so smart“ person would filter minority groups out of the process in 2026? It‘s more likely that 90/10 gender disbalance will be converted to 60/40 or even 50/50. Diverse teams are more fun and stable.
this happened a decade ago when a US courted tried to make sentencing decisions via ML. it was easialy demonstrated that the training data was flawed because the justice system was flawed so the data it was trained on was weighted against minorities because it oversampled because you know, police routinely oversample and poverty for es oversampling
nonetheless, people will defend history as perfect and say those samples, like nepo babies, are "perfect".
Under GDPR, you have the right to request manual processing whenever personal data is processed automatically to make a decision about you that has "significant impact". Not being hired seems like it would qualify.
