An Opinionated Guide to ML Research

Oct 4, 2025

Here are some of my takeaways from John Schulman’s An Opinionated Guide to ML ResearchOpens in a new tab.

How to Develop a Good Taste:

  • Read a lot of papers, think about them critically, and discuss them with experienced researchers.
    • Interestingly, Geoffrey Hinton gives a different kind of advice: “Read the literature but don’t read too much of it. Read a bit of literature and notice something that you think everybody is doing wrong” (Source: Heroes of Deep Learning: Andrew Ng interviews Geoffrey HintonOpens in a new tab). This to me shows that developing good taste can involve both deep engagement with the literature and the ability to identify overlooked opportunities.
  • Collaborate with others working on similar problems.
  • Seek advice from experienced researchers when choosing problems to pursue. They may suggest many ideas, but ideas are cheap. Your job to decide which ones are worth pursuing and how to execute them effectively.
  • Think about:
    • Conceptual contribution: Would my contribution offer a new way of thinking about a phenomenon by improving, extending, or challenging how the parts of an existing theory fit together (e.g., by introducing a new variable), what the theory takes for granted (its underlying assumptions), and when and where the theory applies (read moreOpens in a new tab).
      • Ask yourself:
        • What is new? (What is the novel insight?)
        • So what? (Why is it important? What implications does it have?)
        • Why so? (What are the justifications?)
        • Well done? (Is it complete and not brittle?)
        • Why now? (Is it timely?)
        • Who cares? (Is it relevant to broader audiences?)
    • Generalizability: Would my empirical results transfer to other settings, contexts, or populations?

Goal-Driven Research vs. Idea-Driven Research (Work on a Goal-Driven Research):

  • Goal-driven research begins with a desired outcome or target capability (e.g., make X work for the first time). This helps the researcher to identify problems and ask questions that might help them get closer to the goal. The process involves experimenting with a variety of existing methods from the literature; when these prove insufficient to achieve the goal (i.e., solves the problems you identified or answers the research questions), the researcher develops new methods that improve upon them.
  • Idea-driven research begins when reading an existing method (e.g., a paper on how to do X) inspires an idea for how to do X even better. The researcher then starts a project to test this new idea.
  • The biggest difference is where your perspective comes from. With goal-driven research, you are better situtated to develop a unique perspective from the start, while idea-driven research tends to keep your perspective similar to that of the existing literature (and the community).

Drawback of Idea-Driven Research:

  • Since your perspective may be similar to that of others reading the same literature, there is a high risk of being scooped or duplicating someone else’s work.

Drawback of Goal-Driven Research:

  • When you interpret your goal too narrowly, you risk pursuing domain-specific solutions that achieve the desired outcome but contribute little to broader scientific knowledge. It is important to look for generalizable solutions (e.g., the goal is to … in a way that is general and could be applied to other problems).
  • When you set the wrong goal. A goal should be high enough (i.e., in a way that has an important and long-term vision) so that you could do several incremental work towards achieving that goal.
    • Complexity must be justified by impact. For example, a minor improvement over the baseline (which is often the case if you are working on an incremental idea) should come from a method that is straightforward to implement.