You have a great research idea. You run some experiments and get promising results. You think, “Great! I’ll have a research paper in no time!” You run some more experiments, and results start to look a bit strange. You try to dig into the issue more, but end up with even more conflicting findings and new questions. A few weeks turns into a few months until eventually you realize the conference deadline is only two weeks away! What will you do? Was all of your effort a waste?
In the recent past, I was working on a research project focused on representation learning. I had achieved initial promising results and was in the midst of continuing to try to improve the representation learning objective, under the assumption that there were still benefits to be gained by learning a better representation. I made numerous attempts at changing various components of my algorithm, but over the course of two months ultimate performance wouldn’t budge. I then decided to try running a test: I would devise an “ideal” representation based on my own knowledge of the task and give this ideal representation directly to my model. Certainly, an ideal handcrafted representation should improve performance! Lo and behold, the performance with this oracle representation was no better than my initial promising results. It was now embarrassingly clear that I had already solved the “representation learning” aspect of my problem and whatever remained to solve was orthogonal to representation learning. Looking back, I wish I could have those two months back….
