A neat blog post by Mayank Pratap Singh with excellent visuals introducing ViTs from the ground up. The post covers:
- Patch embedding
- Positional encodings for Vision Transformers
- Encoder-only models ViTs for classification
- Benefits, drawbacks, & real-world applications for ViTs
- Fine-tuning a ViT for image classification.
Full blogpost here:
https://www.vizuaranewsletter.com/p/vision-transformers
Additional Resources:
- An Image is Worth 16x16 Words https://arxiv.org/abs/2010.11929
- Yannic Kilcher Discussion of the paper https://www.youtube.com/watch?v=TrdevFK_am4
- Generating Long Sequences with Sparse Transformers https://arxiv.org/abs/1904.10509
- Generative Pretraining from Pixels https://proceedings.mlr.press/v119/chen20s.html
I've included the last two papers because they showcase the contrast to ViTs with patching nicely. Instead of patching & incorporating knowledge of the 2D input structure (*) they "brute force" their way to strong internal image representations at GPT-2 scale. (*) Well it should be noted that https://arxiv.org/abs/1904.10509 does use custom, byte-level positional embeddings.
[link] [comments]
