In functional analysis, a branch of mathematics, nest algebras are a class of operator algebras that generalise the upper-triangular matrix algebras to a Hilbert space context. They were introduced by Ringrose (1965) and have many interesting properties. They are non-selfadjoint algebras, are closed in the weak operator topology and are reflexive.

Nest algebras are among the simplest examples of commutative subspace lattice algebras. Indeed, they are formally defined as the algebra of bounded operators leaving invariant each subspace contained in a subspace nest, that is, a set of subspaces which is totally ordered by inclusion and is also a complete lattice. Since the orthogonal projections corresponding to the subspaces in a nest commute, nests are commutative subspace lattices.

By way of an example, let us apply this definition to recover the finite-dimensional upper-triangular matrices. Let us work in the ${\displaystyle n}$-dimensional complex vector space ${\displaystyle \mathbb {C} ^{n}}$, and let ${\displaystyle e_{1},e_{2},\dots ,e_{n}}$ be the standard basis. For ${\displaystyle j=0,1,2,\dots ,n}$, let ${\displaystyle S_{j}}$ be the ${\displaystyle j}$-dimensional subspace of ${\displaystyle \mathbb {C} ^{n}}$ spanned by the first ${\displaystyle j}$ basis vectors ${\displaystyle e_{1},\dots ,e_{j}}$. Let

${\displaystyle N=\{(0)=S_{0},S_{1},S_{2},\dots ,S_{n-1},S_{n}=\mathbb {C} ^{n}\};}$

then N is a subspace nest, and the corresponding nest algebra of n × n complex matrices M leaving each subspace in N invariant   that is, satisfying ${\displaystyle MS\subseteq S}$ for each S in N – is precisely the set of upper-triangular matrices.

If we omit one or more of the subspaces S_j from N then the corresponding nest algebra consists of block upper-triangular matrices.