Masked Swingers

Harnessing Data Augmentation to Advance Autoencoders
for Self-Supervised Learning

Anthony Fuller1 Scott C. Lowe1 Daniel G. Kyrollos3
Graham W. Taylor1,4★ Evan Shelhamer1,2★ James R. Green3★

1Vector Institute 2University of British Columbia 3Carleton University 4University of Guelph ★Joint last author

Diagram: an image of a pineapple is augmented and masked two ways. Each view is encoded separately, the two encoders' CLS tokens are exchanged, and each decoder reconstructs its own view's masked patches.
Masked Swingers exchanges CLS tokens between two independently augmented and masked views of the same image before reconstruction, encouraging the encoder to learn a view-agnostic summary.

Abstract

Self-supervised learning (SSL) drops the need for annotations by training on inputs alone, which makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches.

In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms MAE by +3–5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error −64% relative to MAE on three new state-probing datasets, opening the door to world modeling.

Highlights vs. MAE

+3–5 ppImageNet-1K kNN
+45%instance retrieval
+22%animal re-ID
+43%species classification
+76%Omniglot character recognition
−64%state-prediction error

Citation

@article{fuller2026maskedswingers,
  title   = {Masked Swingers: Harnessing Data Augmentation to
             Advance Autoencoders for Self-Supervised Learning},
  author  = {Fuller, Anthony and Lowe, Scott C. and Kyrollos, Daniel G.
             and Taylor, Graham W. and Shelhamer, Evan and Green, James R.},
  journal = {arXiv preprint arXiv:2609.38278},
  year    = {2026}
}