I've heard something along the lines of "Claude is like a compiler: source code is the new object code, you don't look at that anymore" many times.
And I don't really think this is true. Compilers are usually deterministic, and whilst we can find edge cases, it's nothing like an AI agent writing all the code for you.
I think you have two choices, given the Claude is a code generator and not a compiler: (a) you review most or all of code to make sure it makes sense, or (b) you trust but verify via a strong test suite, potentially also created by Claude.
The problem with (a) is that you lose a lot of the speed-up. The problem with (b) is that you have no human oversight and the code may be incomplete, badly designed, or plain wrong.
Currently we review all code because correctness is extremely important to what we do, but that comes at a cost.
I don't know what the answer is here, in general. Does trust build over time? Do the models just get so good we can trust them to make zero mistakes?
Part of the value of LLMs and humans is nondeterminism. Pair nondeterministic output with strictly verified results (proper tests) and you can create a useful working system.
Humans can't build a system perfectly, and agents definitely can't. And agents (like humans) will build a different system every time even with the same prompt. Even if it's just trivial differences like array vs linked-list, there are still differences.
The value in having executable code is that it won't change its behavior unless you deliberately modify it. I don't think there are any shortcuts. Models will get smarter and smarter and make fewer mistakes, but you'll still want to produce real source code to execute because it's consistent (you can sell it as a product, set it and forget it, etc.), and most importantly:
running real code is orders of magnitude faster than having an AI either run the process "manually", or have it write the code out multiple times.
If you have a product intended for humans, you're gonna need humans to do acceptance testing, just like now. All the software engineering principles you learned (did you learn them? now's as good a time as any!) are still just as applicable now as they were 10 years ago.
Maybe just a very rigorous version of the Turing test? Modern LLMs can superficially simulate conversation but it's trivial to force them into revealing their non-human like intelligence.
They've been "patched" since but all models fail basic tests like "Should I walk or drive to the car wash which is 100 feet away" by recommending you walk.
So you'd just ask questions that require theory of mind, abstract and common sense reasoning, causal inference, learning novel rules, transferring knowledge novel situations, recognizing ambiguity, etc.
If an alien lands on Earth and learns English, would you deem it non-intelligent if you can tell it apart from a human in conversation?
I think we should consider slime mold intelligent, and realise that it's a spectrum. Path finding is AI. There are probably forms of intelligence we have yet to discover.
If an alien landed we could decide whether it seems to have a human-like intelligence or not. It could be incredibly intelligent but very non-human-like.
Can you give me one example that works on Claude right now?
I'm never sure whether this indicates "no reasoning present" or you've just hit an odd behaviour in the AI such that its reasoning fails. For example, you present a problem in a way that's dissimilar to the way problems are presented in its training set. That doesn't mean it's not reasoning, just it can only reason correctly in some circumstances.
The models are continually patched with training and post-training. All you have to do is find an area they haven't patched yet, and they'll be just as stupid. I run into deep technical examples every day where they fail in the most basic ways no human ever would.
I'm pretty sure most people building these models would admit they don't operate as human-like intelligences? It's baffling that anyone thinks they are.
I get what you're saying but this is kind of a semantic game.
These LLM models/agents absolutely do not reason in the sense that humans do, so you're quietly redefining the word.
You can say of course decide to call them an "alien kind of intelligence" that "reasons" but you could just as reasonably say that calculators are an "alien" kind of intelligence that "reasons" about math differently than us.
And what stops what AIs do from being "reasoning"? What's the elusive magic fairy dust of reasoning that humans put into their napkin notes, but AIs neglect to put into their chain of thought scratchpads?
Do you have a RealReasoningBenchmark, perhaps, that can reliably tell apart that fake mass produced token-flavored AI reasoning from the real, organic, 100% natural human reasoning?
If you had access to a bunch identical copies of me that couldn't communicate with each other, you'd be able to find many questions I would give stupid answers to. I suspect I'd come out of it looking worse than an LLM.
But immitating others is about the only thing genAI does. Sometimes "others" is a 'programmer', sometimes "others" is an 'artist', but regardless, it still does it poorly.
Not OP, but I'd settle for something that actually learns, instead of being a static pile of linear algebra. Pretending it learns because you change the input (context) doesn't count.
I think a big question is whether any of these labs can produce a model that is _ahead_ of Anthropic and OpenAI.
A related question is how much they're dependent on the APIs of Anthropic and OpenAI to achieve their results - whether through distillation or other uses.
If these models are derivative of Anthropic/OpenAI I would expect performance to be more narrow and progress to be limited.
We use US models for everything in practice, but we are looking at open-source models right now. So it may just be a turn of phrase hiding the reality. I can't imagine anywhere near 80% are relying on open-source as their primary models.
I don’t agree. I think businesses including SaaS are not primarily their tech, but are mostly composed of other functions: sales, marketing, customer success, product management, strategy etc. The value is actually more in these other components than in the code.
For some businesses yes, but for many no, the software and the infrastructure are all that matters and hopefully they don’t touch what works and don’t add features no one asked for.
Agree the functions aren't primarily their tech, but all of their activities ends up serving a business outcome, usually manifested in code (with much distilled or cut along the way). Marketing, PM, and strategy aren't directly shipping slides or docs to production - but they should get distilled into code that ships to fulfill the activity's productivity (decisions, features, etc.).
Sure, but code isn't the only barrier that's been lowered. A competent entrepreneurial team can also scale their sales, marketing and customer success workflows with AI.
I'm not saying business is easy now. I'm saying that a small team of smart people can do a whole lot more with the tools available to them. You still need strategy and vision (or the ability to copy someone else's playbook) but execution is a lot cheaper.
reply