I keep coming back to one strange, wonderful idea: if you treat a protein like a sentence and show a language model enough of them, it starts to understand biology nobody ever explicitly taught it. No structures, no labels — just sequence after sequence, and somewhere in the middle of all that, the model picks up the rules.

How they actually work

The recipe is almost embarrassingly simple. Hide a few amino acids, ask the model to guess what belongs in the gaps, and repeat that across hundreds of millions of sequences from every branch of the tree of life. To fill in the blanks well, the model has to absorb the constraints evolution has been enforcing all along — which positions can drift, which really can't, and how far-apart parts of a chain quietly depend on one another.

Formally, training minimizes a masked-language-modeling loss over the masked positions M\mathcal{M}:

LMLM=−∑i∈Mlog⁡pθ ⁣(xi∣x∖M)\mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log p_\theta\!\left(x_i \mid x_{\setminus \mathcal{M}}\right)

where xix_i is the amino acid at position ii and x∖Mx_{\setminus \mathcal{M}} is the sequence with the masked residues held out. Minimizing it is just "get better at guessing" — but the only way to guess well is to internalize the grammar.

A protein sequence is a sentence written in a twenty-letter alphabet — the model's job is to learn the grammar evolution has been editing for four billion years.

Why I care

Once you have a model that "gets" proteins, a lot of stubborn problems suddenly have a foothold:

  • Drug discovery — guessing how a candidate molecule will latch onto a protein
  • Protein engineering — designing proteins that don't exist yet but probably should
  • Understanding disease — asking what a single mutation actually breaks

Trying it yourself

The best part is you don't have to train any of this.1 Pulling embeddings out of ESM-2 is about a dozen lines:

import torch
from transformers import AutoTokenizer, AutoModel

# Load a pre-trained protein language model
tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t33_650M_UR50D")
model = AutoModel.from_pretrained("facebook/esm2_t33_650M_UR50D")

# A protein sequence (hemoglobin subunit alpha)
sequence = "MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHF"

# Tokenize and pull per-residue embeddings
inputs = tokenizer(sequence, return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state

Those embeddings are the whole point: a numeric fingerprint of each residue, shaped by everything the model learned about how proteins are allowed to exist. Feed them to a small downstream model and you get surprisingly far on structure, function, and stability — without ever touching a crystal.

Notes

  1. ESM-2 was pre-trained on roughly 65 million unique sequences from UniRef. Someone else already paid that bill; you get to start from the embeddings. ↩