Online LaTeX compiler

Paste LaTeX or open a .tex file, press Compile, and download the PDF. pdfTeX runs in this tab via WebAssembly, so your document never leaves your device.

Open .tex file or drop a file here, or paste below

\documentclass{article}
\usepackage{amsmath}

\title{Sparse Attention for Long-Context Transformers}
\author{Your Name \and Co-Author Name}
\date{\today}

\begin{document}
\maketitle

\begin{abstract}
Full self-attention scales quadratically with sequence length, limiting transformer models on long-context tasks. We propose a block-sparse attention mechanism that restricts each query to a learned subset of keys, reducing the computational cost from $O(n^2)$ to $O(n \log n)$ while retaining accuracy within 0.3 points of dense baselines on language modeling benchmarks.
\end{abstract}

\section{Introduction}
Transformer models rely on self-attention to relate every token to every other token in a sequence. As sequence length $n$ grows, the $O(n^2)$ memory and compute cost of dense attention becomes the dominant bottleneck. Recent work on long-context applications --- document summarization, retrieval-augmented generation, and genomic sequence modeling --- has renewed interest in attention variants that scale sub-quadratically without sacrificing model quality.

\section{Method}
We partition the key sequence into fixed-size blocks and learn a routing function $r(\cdot)$ that selects the top-$k$ blocks each query attends to. Given query $q_i$ and key blocks $K_1, \dots, K_m$, the attention output is

\[
\text{Attn}(q_i) = \sum_{j \in r(q_i)} \text{softmax}\left(\frac{q_i K_j^\top}{\sqrt{d}}\right) V_j.
\]

The routing scores are computed from a lightweight projection $r(q_i) = \text{top-}k\left(q_i W_r K^\top\right)$, trained jointly with the rest of the network.

\subsection{Complexity}
With block size $b$ and $k$ selected blocks per query, the attention cost becomes $O(n \cdot k \cdot b)$ rather than $O(n^2)$. Setting $k \cdot b \propto \log n$ recovers the $O(n \log n)$ bound reported in the abstract.

\section{Results}
Table 1 compares our method against dense attention and a fixed local-window baseline on a held-out validation split.

\begin{center}
\begin{tabular}{lcc}
\hline
Model & Perplexity & Tokens/sec \\
\hline
Dense attention & 18.4 & 4{,}200 \\
Local window & 20.1 & 11{,}800 \\
Sparse (ours) & 18.7 & 10{,}600 \\
\hline
\end{tabular}
\end{center}

\section{Conclusion}
Learned block-sparse routing closes most of the gap to dense attention while delivering a 2.5x throughput improvement, making it a practical default for long-context transformer training.

\begin{thebibliography}{9}
\bibitem{child2019} Child, R. et al. Generating Long Sequences with Sparse Transformers. 2019.
\bibitem{beltagy2020} Beltagy, I. et al. Longformer: The Long-Document Transformer. 2020.
\bibitem{kitaev2020} Kitaev, N. et al. Reformer: The Efficient Transformer. 2020.
\end{thebibliography}

\end{document}

Ready. Press Compile (or Ctrl/Cmd + Enter).

Open in the full editor →

Want the full editor?

For live preview, autocomplete, error hints, figures, version history, and 20 templates, open TeXpad. It is just as private, and documents autosave in your browser.

Questions

Is this real LaTeX?

Yes. It runs pdfTeX (the SwiftLaTeX WebAssembly build) in your browser tab. Standard TeX Live packages load on demand and are cached.

Is my file uploaded?

No. The compile happens on your device. Only TeX packages are downloaded, from a public TeX Live mirror.

What about images and multiple files?

This page compiles a single .tex file. For a whole Overleaf project with figures and a .bib, use Import an Overleaf project. In the full editor you can also paste or drop images.

Popular templates

Free LaTeX tools

Import an Overleaf project: open a project zip, nothing uploaded
LaTeX symbols cheat sheet: every common command, with a printable PDF
LaTeX to PDF converter: drop a .tex, download a PDF
Word to LaTeX: convert a .docx, compile the result
Markdown to LaTeX: math and tables kept
LaTeX errors explained: fix the common compile errors, live