The Unexamined Model Is Not Worth Training

Tags: academia, research

« Previous post: Some Comments on “Magnifica …

Just as Socrates said that “The unexamined life is not worth living,”, so the unexamined model is not worth training (in machine learning). We often do it anyway but we really shouldn’t.¹

Many machine learning researchers, I myself being no exception, have a soft spot for what one may call unexamined engineering practices.² One of these is our propensity for joining together a bunch of different components, optimistically referring to them as a novel architecture, and then go on training said architecture. If it works—great, it must have been an act of genius invention. It if does not work—duh-doy, it was an obvious mistake anyway. Sometimes, to make it work, we start tinkering with it until it does. We then declare this to be a win, write it up, and move to the next project. But after the excitement settles, we often find that we were wrong about some key assumptions of our model.

For instance, consider a message-passing graph neural network. Its main idea is to take a graph with node attributes and, by exchanging messages with neighboring nodes, create a per-vertex and, ultimately, a per-graph representation of the graph, which may be gainfully used for downstream tasks like classification. There is absolutely nothing intrinsically wrong with this approach; in fact, numerous interesting architectural choices enable adjusting this to a variety of research domains.³

But—there is always a but—a closer look shows that such graph neural networks rely on the graph even when the graph contains spurious information that is not required for solving a task.⁴ Fair enough, we might say, because that is what we would expect them to do, being graph neural networks. Somewhat paradoxically, message-passing graph neural networks also show themselves to be incapable of leveraging important properties of a graph. For instance, their ability to detect a cycle is roughly determined by how many layers such a network has, or, equivalently, how many rounds of message passing it performs.⁵ The situation becomes even weirder for some datasets, which, upon further inspection, turn out to be best processed by ignoring the graph structure altogether or, more generally, using a different graph than the input graph.

So, what is going on here? Are we, like the alchemists of yore, mixing things together in the quest for the Philosopher’s stone artificial general intelligence? Partially, yes. We unfortunately tend to not examine our models (and our data—but that’s a different story) as seriously as we should. Instead, we are content if we get the “best” numbers in comparison to all other models. Whether the task that we “solved” is hard or not, whether the other models are well-trained or not, whether our model is actually leveraging some of the problem structure to solve the task or not—all of these things become secondary to the warm glow of “reaching SOTA,” i.e., reaching the fabled “state of the art.”

I sometimes observe myself giving in to this feeling, and I would liken it to the obsession of a somewhat strange butterfly collector. Instead of meticulously trying to collect different species and studying them, the collector only wants “something new” for their collection. There is no rhyme or reason to it, and the collection is not ordered in any way, but yet the collector continues…

If we do not make a conscious effort to address it—sometimes fighting against our very own proclivities and (misplaced) incentives—we end up believing the wrong things about our models. I want to close this post with a particularly surprising case. In recent work, my group and I studied the behavior of neural networks operating on manifolds, i.e., specific topological domains. Much to our surprise, it turns out that, at least for “classical” topological properties like homology groups, there are no discernible improvements in performance between graph neural networks and architectures that perform message passing on higher-order structures like manifolds.

However, what looks like a pretty bad result at first is actually good news in disguise: It shows that there is a new research gap to address; it also indicates that we may be—at least when it comes to these topological properties—focusing on the wrong model paradigm. Without diving deeper into how models process such data, we would have never uncovered this fascinating phenomenon.

This serves as a good lesson for me and I vow to examine things more closely moving forward.