M₁₁ is one of the 26 sporadic groups and was introduced by Mathieu (1861, 1873). It is the smallest sporadic group and, along with the other four Mathieu groups, the first to be discovered. The Schur multiplier and the outer automorphism group are both trivial.

M₁₁ is a sharply 4-transitive permutation group on 11 objects. It admits many generating sets of permutations, such as the pair (1,2,3,4,5,6,7,8,9,10,11), (3,7,11,8)(4,10,5,6) of permutations used by the GAP computer algebra system.

M₁₁ has a sharply 4-transitive permutation representation on 11 points. The point stabilizer is sometimes denoted by M₁₀, and is a non-split extension of the form A₆.2 (an extension of the group of order 2 by the alternating group A₆). This action is the automorphism group of a Steiner system S(4,5,11). The induced action on unordered pairs of points gives a rank 3 action on 55 points.

M₁₁ has a 3-transitive permutation representation on 12 points with point stabilizer PSL₂(11). The permutation representations on 11 and 12 points can both be seen inside the Mathieu group M₁₂ as two different embeddings of M₁₁ in M₁₂, exchanged by an outer automorphism.

The permutation representation on 11 points gives a complex irreducible representation in 10 dimensions. This is the smallest possible dimension of a faithful complex representation, though there are also two other such representations in 10 dimensions forming a complex conjugate pair.

M₁₁ has two 5-dimensional irreducible representations over the field with 3 elements, related to the restrictions of 6-dimensional representations of the double cover of M₁₂. These have the smallest dimension of any faithful linear representations of M₁₁ over any field.

There are 5 conjugacy classes of maximal subgroups of M₁₁ as follows:

• M₁₀, order 720, one-point stabilizer in representation of degree 11
• PSL(2,11), order 660, one-point stabilizer in representation of degree 12
• M₉:2, order 144, stabilizer of a 9 and 2 partition.
• S₅, order 120, orbits of 5 and 6

Stabilizer of block in the S(4,5,11) Steiner system

• Q:S₃, order 48, orbits of 8 and 3

Centralizer of a quadruple transposition

Isomorphic to GL(2,3).

The maximum order of any element in M₁₁ is 11. Cycle structures are shown for the representations both of degree 11 and 12.

• Cameron, Peter J. (1999), Permutation Groups, London Mathematical Society Student Texts, vol. 45, Cambridge University Press, ISBN 978-0-521-65378-7
• Carmichael, Robert D. (1956) [1937], Introduction to the theory of groups of finite order, New York: Dover Publications, ISBN 978-0-486-60300-1, MR 0075938
• Conway, John Horton (1971), "Three lectures on exceptional groups", in Powell, M. B.; Higman, Graham (eds.), Finite simple groups, Proceedings of an Instructional Conference organized by the London Mathematical Society (a NATO Advanced Study Institute), Oxford, September 1969., Boston, MA: Academic Press, pp. 215–247, ISBN 978-0-12-563850-0, MR 0338152 Reprinted in Conway & Sloane (1999, 267–298)
• Conway, John Horton; Parker, Richard A.; Norton, Simon P.; Curtis, R. T.; Wilson, Robert A. (1985), Atlas of finite groups, Oxford University Press, ISBN 978-0-19-853199-9, MR 0827219
• Conway, John Horton; Sloane, Neil J. A. (1999), Sphere Packings, Lattices and Groups, Grundlehren der Mathematischen Wissenschaften, vol. 290 (3rd ed.), Berlin, New York: Springer-Verlag, doi:10.1007/978-1-4757-2016-7, ISBN 978-0-387-98585-5, MR 0920369
• Curtis, R. T. (1984), "The Steiner system S(5, 6, 12), the Mathieu group M₁₂ and the "kitten"", in Atkinson, Michael D. (ed.), Computational group theory. Proceedings of the London Mathematical Society symposium held in Durham, July 30–August 9, 1982., Boston, MA: Academic Press, pp. 353–358, ISBN 978-0-12-066270-8, MR 0760669
• Cuypers, Hans, The Mathieu groups and their geometries (PDF)
• Dixon, John D.; Mortimer, Brian (1996), Permutation groups, Graduate Texts in Mathematics, vol. 163, Berlin, New York: Springer-Verlag, doi:10.1007/978-1-4612-0731-3, ISBN 978-0-387-94599-6, MR 1409812
• Gill, Nick; Hughes, Sam (2019), "The character table of a sharply 5-transitive subgroup of the alternating group of degree 12", International Journal of Group Theory, doi:10.22108/IJGT.2019.115366.1531, S2CID 119151614
• Griess, Robert L. Jr. (1998), Twelve sporadic groups, Springer Monographs in Mathematics, Berlin, New York: Springer-Verlag, doi:10.1007/978-3-662-03516-0, ISBN 978-3-540-62778-4, MR 1707296
• Hughes, Sam (2018), Representation and Character Theory of the Small Mathieu Groups (PDF)
• Mathieu, Émile (1861), "Mémoire sur l'étude des fonctions de plusieurs quantités, sur la manière de les former et sur les substitutions qui les laissent invariables", Journal de Mathématiques Pures et Appliquées, 6: 241–323
• Mathieu, Émile (1873), "Sur la fonction cinq fois transitive de 24 quantités", Journal de Mathématiques Pures et Appliquées (in French), 18: 25–46, JFM 05.0088.01
• Thompson, Thomas M. (1983), From error-correcting codes through sphere packings to simple groups, Carus Mathematical Monographs, vol. 21, Mathematical Association of America, ISBN 978-0-88385-023-7, MR 0749038
• Witt, Ernst (1938a), "über Steinersche Systeme", Abhandlungen aus dem Mathematischen Seminar der Universität Hamburg, 12: 265–275, doi:10.1007/BF02948948, ISSN 0025-5858, S2CID 123106337
• Witt, Ernst (1938b), "Die 5-fach transitiven Gruppen von Mathieu", Abhandlungen aus dem Mathematischen Seminar der Universität Hamburg, 12: 256–264, doi:10.1007/BF02948947, S2CID 123658601

• MathWorld: Mathieu Groups
• Atlas of Finite Group Representations: M₁₁