In mathematics, an automorphism is an isomorphism from a mathematical object to itself. It is, in some sense, a symmetry of the object, and a way of mapping the object to itself while preserving all of its structure. The set of all automorphisms of an object forms a group, called the automorphism group. It is, loosely speaking, the symmetry group of the object.
In the context of abstract algebra, a mathematical object is an algebraic structure such as a group, ring, or vector space. An automorphism is simply a bijective homomorphism of an object with itself. (The definition of a homomorphism depends on the type of algebraic structure; see, for example, group homomorphism, ring homomorphism, and linear operator.)
The identity morphism (identity mapping) is called the trivial automorphism in some contexts. Respectively, other (non-identity) automorphisms are called nontrivial automorphisms.
The exact definition of an automorphism depends on the type of "mathematical object" in question and what, precisely, constitutes an "isomorphism" of that object. The most general setting in which these words have meaning is an abstract branch of mathematics called category theory. Category theory deals with abstract objects and morphisms between those objects.
In category theory, an automorphism is an endomorphism (i.e., a morphism from an object to itself) which is also an isomorphism (in the categorical sense of the word, meaning there exists a right and left inverse endomorphism).
This is a very abstract definition since, in category theory, morphisms are not necessarily functions and objects are not necessarily sets. In most concrete settings, however, the objects will be sets with some additional structure and the morphisms will be functions preserving that structure.
If the automorphisms of an object X form a set (instead of a proper class), then they form a group under composition of morphisms. This group is called the automorphism group of X.
The automorphism group of an object X in a category C is denoted AutC(X), or simply Aut(X) if the category is clear from context.
One of the earliest group automorphisms (automorphism of a group, not simply a group of automorphisms of points) was given by the Irish mathematician William Rowan Hamilton in 1856, in his icosian calculus, where he discovered an order two automorphism, writing:
so that is a new fifth root of unity, connected with the former fifth root by relations of perfect reciprocity.
In some categories—notably groups, rings, and Lie algebras—it is possible to separate automorphisms into two types, called "inner" and "outer" automorphisms.
In the case of groups, the inner automorphisms are the conjugations by the elements of the group itself. For each element a of a group G, conjugation by a is the operation φa : G → G given by φa(g) = aga−1 (or a−1ga; usage varies). One can easily check that conjugation by a is a group automorphism. The inner automorphisms form a normal subgroup of Aut(G), denoted by Inn(G); this is called Goursat's lemma.
The other automorphisms are called outer automorphisms. The quotient group Aut(G) / Inn(G) is usually denoted by Out(G); the non-trivial elements are the cosets that contain the outer automorphisms.
The same definition holds in any unital ring or algebra where a is any invertible element. For Lie algebras the definition is slightly different.