A little more than 30 years ago, I wrote a Master’s Thesis that describes a framework for building a semantic model that could represent a set of requirements so completely, concisely and precisely that code, even an entire application’s worth of code, could be automatically generated from that model. It’s title is “A complete semantic modeling approach to representing the knowledge to be encapsulated in a system life cycle support environment.” It’s the kind of title you give to a book when you can’t think of a good title. (The insomniac reader is referred to here.)
The context of my thesis was modeling the semantics of something called a life cycle support environment (LCSE). LCSEs manage the artifacts in a software project throughout their life cycles, from the maintenance of code in a source code repository to how the artifacts are built, deployed and operated. Several years after my participation in our work in this area at the University of Houston – Clear Lake under the leadership of Dr. Charles McKay, LCSEs as we had envisioned them hadn’t been widely adopted by the mainstream software development community, and I thought that maybe this was another promising path of thought that ultimately wouldn’t pan out. Until it did. A notion of LCSEs exists today. It’s called DevOps. The form that DevOps has today is different from we had envisioned, but the fundamental ideas are there: the ability to scale services up and down in response to changes in an uncertain environment, the ability to deploy application updates with no down time, and the ability to support 24/7/365 operations.
The idea of “complete semantic modeling” is that we can take some input text like a set of requirements, convert it into a deep semantic representation, and translate that representation into some other form, like the text translated into another language or source code. This wasn’t exactly a totally original thought; it basically mirrors the surface structures and deep structures in Noam Chomsky’s transformational grammars that he wrote about in the 1960s.
Whereas transformational grammars were intended to describe languages, my thesis attempted to show how they could be used to do stuff like generate code. Alas, like many theses, mine may have been as wide as the Pacific Ocean, but it was only a half-inch deep. It described a framework, but did not implement it. Later, I made an attempt or two to move towards an implementation, but didn’t succeed.
So it looked like my thesis was another promising path of thought that ultimately wouldn’t pan out. Until it did. A notion of complete semantic modeling exists today. It’s called a Large Language Model transformer. At a high level, a transformer looks like this:
The encoder takes input text and generates a deep semantic representation of it (in the form of hyper-dimensional word vectors). The decoder takes that deep semantic representation and generates output text that can be a response to a query posed in the input text, or a translation of that input text into some other form.
This is roughly the same idea that’s expressed in my thesis, but implemented using concepts I couldn’t have conceived of. I’m not going to tell you that my thesis had anything whatsoever to do with LLM transformers, or the word vectors that serve as their semantic representations. It almost certainly did not. I would be shocked if any of the trailblazers in this field (or anyone else, for that matter) have even heard of it.
What I find interesting about all this is that when these seemingly new ideas like LLMs come about, there is a long lineage of thought behind it. We can trace the thinking that led to DevOps back at least 40 years, and the train of thought that led to LLMs goes back more than 60 years. We can trace artificial neural nets back to early statistics work that was done more than 200 years ago.
Throughout all of that time, there were probably many false starts, or roads that didn’t lead to the intended destination. But the ideas persisted, and others picked them up and tried different approaches, until after many iterations, success was achieved.
When I expressed my dream of automated text transformations through semantic models, I had not heard of word vectors, feed-forward neural networks, or self-attention mechanisms. The LLM transformers that are today’s engines of semantic modeling are superior by orders of magnitude to any implementation I could have come up with, and are of a form that I couldn’t have imagined 30 years ago. They have applications beyond my wildest dreams. Perhaps my thesis was a false start, one of those roads that didn’t lead to fruition. If so, I’ll take it. I’d like to think that I’m somehow a tiny part of what led to the LLM revolution, even if such a thought is a delusion of grandeur.
New, groundbreaking ideas find their way into the mainstream from time to time. When one does, take a moment to think about the lineage of thought behind it, and the many contributors to that idea, most of whom will remain unmentioned. And if you have such an idea, pursue it. Perhaps you won’t be the one who develops the next new thing that takes the world by storm, but chances are a future iteration of your idea will, and you’ll be amazed by how marvelous it will be.