linear-attention

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

The semantic segmentation of remote sensing images

CUDA implementation of autoregressive linear attention, with all the latest research findings

The semantic segmentation of remote sensing images

LEAP: Linear Explainable Attention in Parallel for causal language modeling with O(1) path length, and O(1) inference

Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)

Taming Transformers for High-Resolution Image Synthesis