Who is buying these at 70% of new pricing given the sky high likelihood of them being shot? Maybe it's safe to buy from small labs that went under quickly, but I can't imagine a cluster that has been operating near its thermal limits for a couple years fetching that kind of resale.

Why would they be shot? Unlike the consumer cards that are basically factory overclocked to look good on benchmarks, the datacenter GPUs are designed to run at full tilt 24/7 and survive for years.

According to the article these cards have a 9% annual failure rate.

>The number traces to Meta’s Llama 3 technical report, which documented 419 unforeseen disruptions across 16,384 H100s over 54 days of training, of which 148 were GPU failures and 72 were HBM3 memory failures.

From an annualized number on the llama 3 training report. would be interesting to see if we have a better idea given that we're already on rubin.

It depends entirely on the time-to-failure distribution though whether used cards are a good deal or not. Often this kind of hardware has a bathtub shaped hazard rate, actually getting burned in cards may mean you get the weeded out solid specimens, and forgo the lemons.

Most hardware has an exponential (memoryless) failure distribution in practice. The bathtub curve is a myth.

A bathtub curve is also memoryless though, and the back portion can be exponential. Consider two sub-populations of the same hardware. The first is small, evaded QA, and will fail early. Failures follow an exponential fall off. The second is much larger and failures rise exponentially with age.

Also consider mixing in "was dropped during shipping" or "was stored improperly".

How is a bathtub curve memoryless?

My mistake, it seems I misunderstood what OP meant by that. Ignore that semantic detail and my position still stands. You've got a mix of subpopulations and many of the failure modes are definitely time dependent in some manner at least in the real world.

No worries. I just thought I was misunderstanding something.

Where is this exponential thing coming from?

Why would hardware failure raise exponential with linear scale? Doesn't make sense

> Where is this exponential thing coming from?

That's what you get from a constant failure rate per unit of time.

It's saying that hardware fails in the same way that Carbon-14 dating works.

Lets hope they add more context to it.

Most if not all hardware failure on my side is def not memoryless besides the one lightning damage thingy but thats not hardwares fault.

A 9% (static) failure rate, implies an average lifespan of 11 years...

With a median lifespan of 7.35 years, although I doubt that 9% per year remains constant as the number of years increases.

Given the decades of consumer and enterprise GPUs often being identical or near identical hardware, it'd be interesting to see if there's any evidence of this actually being true.

Binning can take in a supposedly uniform stream of chips and produce different tiers on the output.

It was true of crypto GPUs too, although mostly people picking them up for gaming. Always seems high to me too but if you can get any guarantee of them not being on fire when they were pulled the bathtub curve keeps you pretty safe, thermal limits are limits for a reason.

crypto gpus didnt run at 100% power draw so the strain wasn't that massive

I bought a RTX 3070 off a miner when Eth went to proof of stake.

It was clean, cheap, and is still going strong for daily gaming.

In its working life it was undervolted and probably cooled better than in my rig.

I got an entire Prebuilt PC when POW ended from a miner. AMD 5950x, 64gb ram, 3090, 2tb SSD, all for $1700. This was 4 years ago when the 3090 was $1500 by itself, and with minor upgrades it's still going strong today.

It's wild to think that the system now is worth at least as much as I paid for it then if not much more than that. I saw a similar one going for $2500.

Why weren't crypto gpus running at 100%?

To advertise the maximum possible performance numbers, CPUs and GPUs massively shoot up the power draw for that final few percentage points. If you are worried about actual calculations/$, cutting back on the power will save your energy bill for practically no loss.

When I tested "ECO" mode on my AMD CPU, performance was ~97% of regular mode, and temperatures dropped 5-10C. I think ECO mode drops TDP from something like 100W->65W. Cheaper and cooler to run for basically no cost.

If you are bitcoin mining or selling GPU capacity, you are incredibly conscious of your energy bill, so it only makes sense to optimize the power draw.

Last point should also be for llm. Not just crypto

Not true. A B200 "only" draws 1 kW. At 16¢ USD / kWh that's $1400 / year. When the hardware costs multiple tens of thousands you might be looking at a yearly electricity bill of ~3% of that. And that's before accounting for the datacenter itself.

The difference is that much of the hardware crypto was running on was dirt cheap in comparison. Add to that a lot of crypto was being run on discount gear in cobbled together setups. Some people were even stealing electricity which often involved installing large clusters of relatively cheap devices in places without proper cooling.

Capex != Opex

Your are measuring two very different things.

And you are only considering electricity bills for the GPU. Higher TDP also means more heat which means more cooling.

It's not really easy to make an off hand comment about costs. A 30-40% decreaae in electricity usage and heat output for a fleet of GPUs definitely makes a massive difference to your opex bottom line.

The datacenter will already have been built with an explicit cooling budget. Moving 1 watt of heat requires substantially less than 1 watt to do the work. So you've got a constant multiplier against something like a 9% annual failure rate and an order of magnitude difference in cost.

And the change in failure rate and technical advances are a de-facto change in actual depreciation vs. accounting depreciation.

Electricity gets converted to heat which has to be removed through the cooling system.

Performance per Watt is the key metric for data centers full of GPUs.

That just shows up as a constant factor, accounted for at build out. Consider if each watt dissipated required an additional watt. That's clearly an absurd overestimate yet it would "only" double power costs making the example $2800 / year. Underestimate the B200 at $30k USD and you've got 10.5 years of operation before you achieve parity as a fairly unrealistic worst case estimate. Now consider that at a 9% failure rate the average lifespan is only ~11 years. So electricity (both operation and cooling) is pretty much guaranteed to be less than half your total operating cost no matter how you slice it.

Most crypto mining on GPUs would use 100% of memory bandwidth, but only a fraction of the compute available. This is a consequence of ASIC resistance of their mining algorithms -- custom silicon can only offer a modest benefit over GPUs if the hard part is memory bandwidth.

So does llm inference. You're lucky if you hit 40% of the advertised flops.

Right, but datacenter GPUs optimized for LLM training/inference would have a bandwidth:compute ratio scaled to that workload.

No they don't.

Then where are the HDMI ports on Nvidia's current data center GPU product lines?

Why not? Seems like they would be poorly optimised?

AI optimised cards is the biggest cashcow for nvidia. They are definitely doing custom silicon. And Google et al have cards that don't even pretend to be able to do graphics.

There is more than one workload in AI. Inference for llms is memory constrained on even a single card. Training for llms is memory constrained on the level of racks. In both cases you hardly ever see more than 40% of advertised flops used.

So the question is: why are they designing AI cards so unbalanced?

Why is that true? Can’t you just make more stuff parallel and shrink the ASIC chips accordingly?

No - inability to do so is part of the design of a good cryptographic hash, quite explicitly.

https://en.wikipedia.org/wiki/Avalanche_effect

Most were powerlimited to a degree to get the most hash/watt out of them.

we over clocked and under volted... the goal was to lower power and run the clocks as fast as possible.

Honest question. Does silicon wear out due to high temperatures, or is it more like lightbulbs where it wears out from thermal cycles? Does it really wear out at all?

It’s mostly due to higher temps resulting in faster ion migration, which can cause a breakdown in the structure of transistors, as well as increased wear on the conductors (though this is rarely a dominating factor). Thermal cycling can also cause cracking, which is also a problem. Though there are lots of reason chips fail due to heat, from the wire bonds on pads getting too hot to increased leakage current at high temps.

For a long time it was assumed it wouldn't wear out and tbh if you look how long it takes, how rare it is, its not a real issue...

besides what intel did with Raptor Lake. This series had massive issues with oxidiation. Was the first time ever i became aware of this issue on scale.

Nontheless there are papers out there that silicon can degenerate and does.

I feel like this is not as important as people make it out to be.

When I tried warp I was convinced that was where the industry was going (agents as terminal replacement), but it felt a bit too heavy to me so I haven’t been using it lately. Still think all things will converge on terminal and browser replacement.

(1) be naturally gifted, (2) avoid falling down wells of distraction, (3) be lucky. Don’t sleep on (3), it’s easy to call it capitalizing on opportunity in hindsight when it was honestly just luck.

I'd also add (4) be incredibly curious about lots of things; (5) surround yourself with other smart, curious, and committed people who have a culture of critiquing ideas; and (6) devote a lot of time to deep thinking.

(8) be good in counting things, (h) be consistent in your thinking, (10) have a good memory, (11) be good in counting things, (12) refrain from making silly comments

luck in terms of natural intelligence and focus. But generally people can approve their ability to avoid falling down the "wells of distraction". This year I set time filters for apps on my phone and its made a stark difference. Even though I can turn off "work mode" to get around it, that little reminder has saved me hundreds of hours as I usually just put down the phone when I see it.

Person 1 made every deck exactly the same way though.

I guess Person 2 could run some simple validations of what Person 1 initially did.

Then again, Person 2 could have also replaced all of Person 1s decks with their own.

Person 1 would make exactly one of each possibility. They're just deciding which cards go in which pile/box.

You may be referring to PayPal as one of those companies, but in case you’re not: Elon did not found Tesla.

He didn't found PayPal either. He founded X.com, and they happened to merge with Confinity who owned the brand PayPal. Confinity existed before X.com (and had better brand recognition with PayPal), and if Musk would've had it his way, PayPal as brand wouldn't exist post-merge; it would have been called X.com.

I said "for some definition" because according a legal settlement he made with the others, he did found it.

You do not know with any degree of certainty what their motivations are. In fact, the utter disdain they demonstrate for certain US citizens dismantles your entire claim.

Then frankly you weren’t paying attention. This was all laid out in Project 2025 and there were plentiful ties between it and Trump despite his deflection.

People will absolutely lose their labs over this, please stop spreading lies either intentionally or from a place of ignorance.

I agree with the other guy, usually funding is dispersed in months or years at a time, so this shouldn't impact anyone in a meaningful way (unless they're doing gain of function research, in which case their lab will likely stop receiving funding).

Oh good, it’s just grant proposals so only some people will lose their labs due to the idiotic politicization of the NIH. This pushes back study sections by months even if they do resume on Feb 1. You can’t just schedule x leading researchers to review a grant on a whim, it’s planned months in advance.

Except middle America, where the populace would find a reason to be distrustful of a straight up cure.

I have a family member who buys supplements off YouTube video recommendations because they straight-up cure cancer.

I really with hen would be more distrustful