Terminology:

  • ML = machine learning
  • PL = programming languages

The fields of ML and PL view the world from different perspectives.

<system 1 vs system 2>

ML is probabilistic and good at making suggestions, but has no guarantees built-in about properties of their predictions

PL techniques are rule-based (type systems, formal verification, etc) and guaranteed correct, but often do not scale to real world scenarios (production languages like C++, not enough of a sound solid base with nice properties to start with, need to build toy languages)

<link to Lucas’ talk at PLDI about GitHub Copilot>

It’s common for PL people to be skeptical of ML techniques, and vice versa.

However, one in which I learned on my current team is learning about how ML and PL can help each other.

<PL: good training data for ML, known oracles like type checkers and theorem provers>

<ML: allow producing lots of suggestions>

<Cite Prof Mayur Naik’s talk, slides about system 1 vs system 2>

View ML and PL not as in competition (actively bad) or orthogonal (neutral fields going in their own direction) but a positive feedback loop: ML helps PL be better at helping ML, etc

  • ML supports constrained decoding: can limit the vocabulary for the next token to autoregressively regress.

  • Maieutic prompting: a way to turn soft predictions from LLMs into more confident facts via clever self-consistency checking. Also enables getting confidence intervals on predictions, by recursively applying the approach to justify explanations and their negations. A good example of turning soft predictions into harder facts (not guaranteed, but quantified higher confidence).

  • Consideration: models are good at learning correct syntax, robust against typos. Want to use PL for the biggest value-add, e.g. add data that provides the greatest value and is most difficult for model to learn by itself. (Consider adding an 2D plot of difficulty vs value).

After I joined this team, it took me a year to feel comfortable. … The nature of the work felt so different: much of ML (these days, at least my work) is transforming data and running command line invocations.

However, talking to expert researchers helped me feel better over time: learning about what makes them excited helped me become excited and productive.

And then I learned about the connections between my past and current work, how ML and PL can help each other. connecting the dots

Meta point: having multiple areas of expertise is not zero sum, total knowledge is more than sum of the parts, interesting insights and connections are waiting to be discovered – the best of both worlds

Good researchers can dive in and find interesting aspects of any subject!