Robert Važan

How I failed to invent ChatGPT

Believe it or not, I was on the right track 15 years ago. By that time, I already gained several fundamental insights that explain performance of modern language models. I also got a lot of things wrong and I was working within too narrow hardware constraints, so I inevitably failed. Given that even today many people, including people working on language models, fail to understand what the models are about and why they work, I think it would be a good idea to reevaluate my experience with the benefit of hindsight.

So here's what I got right at the time:

What I got wrong 15 years ago:

It is now clear why my efforts couldn't possibly succeed 15 years ago. But the world got there eventually even without my help. I think that in the end it does not matter who was first. What matters is that the technology is eventually opensourced, optimized, democratized, and integrated into everything. And I am optimistic we are heading in this direction.