Learning by Comparison

How do you teach a model what images mean when you have no labels to tell it? SimCLR (Chen, Kornblith, Norouzi, and Hinton, 2020) gave an answer so clean it became the textbook example: teach it by comparison.

The core move

Take one image and make two different random views of it — crop it two ways, jitter the colors, add a little blur. These two views are a “positive pair”: they came from the same picture, so the model should map them to nearby points in feature space. Every other image in the batch is a negative: it should be pushed away. That’s the whole objective. Pull together what belongs together, push apart what doesn’t. The supervision isn’t labels — it’s the structure you created by augmenting.

Formally this is the NT-Xent contrastive loss, with a temperature knob that controls how hard the model leans on the toughest negatives. Practically, a batch of N images becomes 2N views, and each view’s negatives are simply the rest of the batch — no separate negative-mining machinery required.

Two details that carry the result

• Augmentation choice is not a detail — it’s the method. The standout combination is random cropping plus color distortion. Either alone is weak; together they force the model to find what’s invariant about an object across wildly different views. Contrastive learning wants stronger augmentation than supervised learning does.
• Throw away the head you trained. SimCLR passes features through a small projection network before computing the loss, then discards it and keeps the representation from before the projection for downstream use. That earlier representation transfers over 10% better — the projection head seems to soak up information the loss would otherwise destroy. Later methods kept this trick.

Why it landed

A big ResNet-50 trained this way hit 76.5% on ImageNet under linear evaluation — matching a fully supervised ResNet-50, with no labels used during pretraining. With just 1% of labels for fine-tuning, it beat AlexNet while using 100× fewer labels.

The bill comes in batch size: SimCLR leaned on batches up to 8192 (over 16,000 negatives per positive) and long schedules, which is expensive, and much later work chipped away at that dependence. But the idea outlived the cost. To learn what things are, you don’t always need someone to name them. You need enough examples of “same” and “different” — and the discipline to compare.

References

• Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations (SimCLR). ICML.