Attention

Contents

• Vanilla Self Attention O(\(n^2\))
• Cross Attention
• Spatial Self Attention
• Window Attention
• Memory Efficient self Attention O(n)
• Gated Positional Self Attention
• Self Attention with Convolutional Projection