If you’ve ever read about modern physics, you know your high school physics math won’t get you very far. Books by Stephen Hawking and Brian Greene contain descriptions that you have to take on faith. Without the math, all that science may as well be science fiction. Computing has reached a similar place with artificial intelligence. Unless you know a lot about bits of architecture in software called “transformers,” you have to listen to sales pitches by tech CEOs and try to separate the big deals from bullshit.
Transformers were popularized in a now-iconic 2017 paper by eight Google scientists called “Attention Is All You Need.” It’s important to know a few things about them: They parse a whole sentence at once; they convert words into interchangeable chunks of code in a way that makes AI good at analogy, but at least so far, bad at counting the number of r’s in ‘strawberry;’ and, importantly, they don’t seem to be running into problems with how they scale. So far, every time that people have dumped a pile of data into them, they have shown small progress on the way to staggering breakthroughs. So far.
Scaling is the big challenge of almost all technology. Batteries can only get so big before they grow less efficient, and since the beginning of mobile phones, our pocket wizards have been limited by the batteries more than the software. For more than a decade, they could have done a lot more if the charge didn’t need to last 10+ hours. It’s common in software platforms, too, and you might remember early Twitter and YouTube blowing up before they could handle the volume.
Transformers don’t have a comparable problem, at least not publicly. There has always been another breakthrough. And tech companies are investing trillions into generative AI, leading one Goldman Sachs researcher to wonder, what trillion dollar problem is it going to solve? What payoff could justify such a massive investment? Some believe the whole field is on the verge of collapse; Ed Zitron recently referred to the moment as the subprime AI crisis. Not every cool new tech product turns a profit.
This line of thinking can be persuasive, but the people publishing this stuff aren’t the ones spending trillions. There’s a strong signal that those who know the most about AI are putting the most on the line. Microsoft avoided big investments into augmented reality and crypto blockchains, and they are dumping a fortune into OpenAI, the makers of the chatbot ChatGPT. It’s not irrational to wonder if Microsoft knows something that Goldman Sachs does not.
So what is a savvy outsider to make of this mess? What’s between believing tech CEOs on press tours and pure skepticism? Actually, we can just wait a few months.