In mathematics, especially in the field of group theory, a Carter subgroup of a finite group G is a self-normalizing subgroup of G that is nilpotent. These subgroups were introduced by Roger Carter, and marked the beginning of the post 1960 theory of solvable groups (Wehrfritz 1999).

Carter (1961) proved that any finite solvable group has a Carter subgroup, and all its Carter subgroups are conjugate subgroups (and therefore isomorphic). If a group is not solvable it need not have any Carter subgroups: for example, the alternating group A₅ of order 60 has no Carter subgroups. Vdovin (2006, 2007) showed that even if a finite group is not solvable then any two Carter subgroups are conjugate.

A Carter subgroup is a maximal nilpotent subgroup, because of the normalizer condition for nilpotent groups, but not all maximal nilpotent subgroups are Carter subgroups (Ballester-Bolinches & Ezquerro 2006, p. 100). For example, any non-identity proper subgroup of the nonabelian group of order six is a maximal nilpotent subgroup, but only those of order two are Carter subgroups. Every subgroup containing a Carter subgroup of a soluble group is also self-normalizing, and a soluble group is generated by any Carter subgroup and its nilpotent residual (Schenkman 1975, VII.4.a).

Gaschütz (1962) viewed the Carter subgroups as analogues of Sylow subgroups and Hall subgroups, and unified their treatment with the theory of formations. In the language of formations, a Sylow p-subgroup is a covering group for the formation of p-groups, a Hall π-subgroup is a covering group for the formation of π-groups, and a Carter subgroup is a covering group for the formation of nilpotent groups (Ballester-Bolinches & Ezquerro 2006, p. 100). Together with an important generalization, Schunck classes, and an important dualization, Fischer classes, formations formed the major research themes of the late 20th century in the theory of finite soluble groups.

A dual notion to Carter subgroups was introduced by Bernd Fischer in (Fischer 1966). A Fischer subgroup of a group is a nilpotent subgroup containing every other nilpotent subgroup it normalizes. A Fischer subgroup is a maximal nilpotent subgroup, but not every maximal nilpotent subgroup is a Fischer subgroup: again the nonabelian group of order six provides an example as every non-identity proper subgroup is a maximal nilpotent subgroup, but only the subgroup of order three is a Fischer subgroup (Wehrfritz 1999, p. 98).