"Anthropic and OpenAI generate a lot of revenue with relatively few employees – an estimated $9M and $5.5M in revenue per employee (RPE), respectively. If either company were to go public, it would have a higher RPE than any public tech company on Forbes’ Global 2000 list." https://epoch.ai/data-insights/revenue-per-employee-ai-compa...
This assumes they do not have to increase prices to be profitable, and that they will continue to have customers when customers can switch to open models at similar performance.
As an analogy, Uber could crank up rates after the VC growth play was over to stoke revenue and profits because they have a duopoly with Lyft. LLM consumers can switch to Kimi models fairly trivially today, and whatever the frontier open model landscape looks like later. Model training and development is expensive, self hosted inference on open models not so much.
(a component of my work is currently building scaffolding so our organization can swap out commercial inference providers for on prem inference infra to derisk against the eventual rug pull when the math gets icky for LLM providers, while consuming as much subsidized tokens as we can until then, when it makes sense to use tokens for work)
The question will be whether customers can switch.
Can you install a near-SOTA model on a cluster in a data center? Of course. Compliance and operations are the sticking points. I work in healthcare IT, and it's amazing how tight the data compliance requirements are. I can't have someone in Canada look at prod data. If we told hospitals that we were handing off PHI/PII to Chinese models, they'd end our relationship due to the long history China has of hacking Western networks and computers. They don't care how open and cheap things are.
Then, you have to keep up-to-date on the latest technology and right-size things in a very fluid market. If you sign a contract for hosting the model on a data center that's running what the SOTA is now in hardware, and someone comes through with a data center hardware or software product that makes that data center contract a disadvantage (maybe it's too expensive and the other party won't budge on the price), you might have to factor that into your offering's price, and that could put you at a disadvantage in your marketplace.
Google, MS, etc. all want to leverage the cloud model to make this be less of an issue for you, for a price. They have the ability to update you with the SOTA stuff in the data centers, because they're the ones driving that SOTA. They can say they host in the US and develop most of their stuff in the US.
Will that be enough of a moat?
Probably not for the levels of spending that are happening now, but over the long term, probably.
How long will a SOTA model be necessary? If day to day work can be achieved on an open weight model, the most evaporates overnight.
Look at any computer in a big company. It isn't the fastest on the market, nor will it have the most RAM or largest monitor or fanciest keyboard. It is good enough at a good enough price point. Once it becomes possible and cheaper to host your own good enough open weight models, with all the benefits of keeping data internal to the company, then the big providers are cooked, so to speak.
> How long will a SOTA model be necessary? If day to day work can be achieved on an open weight model, the most evaporates overnight.
Depends on the advantage it gives people and marketing of that advantage. You'd be surprised at how overpowered the average workplace laptop is. Each company I've been at has had at least some people who do non-technical roles using high-end hardware. Why? Because the account executive wants the fast machine and they get what they want.
You can apply the same to GenAI. Humans are notoriously bad at estimating actual needs when it comes to resource consumption. Best to have it and not need it than need it and not have it, especially if your competition just shelled out for SOTA.
And that's not even taking into consideration regulatory and customer concerns about where the AI you're serving requests with came from.
Execs getting what they want doesn't mean they let everyone have the same thing. It's far more likely that everyone else is using lesser equipment.
We have already seen tech workers at big name companies get whiplash from "leaderboards showing people using the most tokens!" as a good thing one month to being pressured to using fewer tokens a month later.
My primary role is cybersecurity in a regulated entity in a regulated industry, I am highly confident it is straightforward to do so based on work accomplished in only a couple of weeks. Stand up a router, stand up a Kubernetes cluster if you don't have one, stand up the necessary VMs and compute for serving inference. Two pizza team, in my experience.
Customers can switch (although we can argue the speed and pain of doing so), and the speed at which they do will be a function of cost efficiency and demonstrable value (imho). A recent example of this is Broadcom and VMware [1], for example. When motivated, it can be done. If there is no objective, measured value being delivered, the spend will be cut. If the value delivered is measured, it will be enabled at a lower cost through cost optimization measures (ie self hosting) [2].
This is all to say: there is no moat, the revenue of inference providers is volatile and not assured in any measure. Caveat emptor.
Meanwhile, in real companies, you have to wait 2 months or more to access an API endpoint in preprod.
To setup a cross business kubernetes cluster will take 2 years with unknown results.
On Cloud, in Switzerland, you need to call Microsoft when you need new resources, so much for agility and minute infrastructure provisioning, and I heard the same for AWS.
> Meanwhile, in real companies, you have to wait 2 months or more to access an API endpoint in preprod.
> To setup a cross business kubernetes cluster will take 2 years with unknown results.
Do you seriously believe those times will not go down 95% if the CEO pushes for it to get done yesterday because it will save the company millions in expenses?
You get it. Given sufficient incentives, processes and systems become potentially more malleable, and hard requirements can become optional. Speed is a function of appetite, will, and resources.
There's a difference between "It's straightforward to do" and "I can convince a customer to sign a contract allowing us to do it."
If the second one were as easy as the first, I wouldn't have to be online at 9:00 to deploy stuff to prod tonight; the team in India would handle it. But customers write into the contracts that only US-based employees interact with prod systems. No amount of cajoling will get them to change their minds; they have data sovereignty, international telecommunications treaties, and HIPAA compliance to worry about. So I'll be pressing buttons tonight.
Could you swap out Anthropic or OpenAI or Google or whoever's models for Kimi? Yes. They're like other software these days, they're modular. What isn't modular is regulatory and geopolitical concern.
