The VEX coding scheme allows the definition of new instructions and the extension or modification of previously existing instruction codes. This serves the following purposes:

• The opcode map is extended to make space for future instructions.
• It allows instruction codes to have up to four operands (plus immediate), where the original scheme allows only two operands (plus immediate).
• It allows the size of SIMD vector registers to be extended from the 128-bit XMM registers to the 256-bit YMM registers. There is room for further extensions of the register size.
• It allows existing two-operand instructions to be modified into non-destructive three-operand forms where the destination register is different from both source registers. For example, c ←a + b instead of a ← a + b (where register a is changed by the instruction).

The VEX prefix replaces the most commonly used instruction prefix bytes and escape bytes. In many cases, the number of prefix bytes and escape bytes that are replaced is the same as the number of bytes in the VEX prefix, so that the total length of the VEX-encoded instruction is the same as the length of the legacy instruction code. In other cases, the VEX-encoded version is longer or shorter than the legacy code. In 32-bit mode VEX encoded instructions can only access the first 8 YMM/XMM registers; the encodings for the other registers would be interpreted as the legacy LDS and LES instructions that are not supported in 64-bit mode.

The VEX coding scheme uses a code prefix consisting of two or three bytes, which may be added to existing or new instruction codes.^[1]

In x86 architecture, instructions with a memory operand may use the ModR/M byte which specifies the addressing mode. This byte has three bit fields:

• mod, bits [7:6] - combined with the r/m field, encodes either 8 registers or 24 addressing modes. Also encodes opcode information for some instructions.
• reg/opcode, bits [5:3] - depending on primary opcode byte, specifies either a register or three more bits of opcode information.
• r/m, bits [2:0] - can specify a register as an operand, or combine with the mod field to encode an addressing mode.

The base-plus-index and scale-plus-index forms of 32-bit addressing (encoded with r/m = 100 and mod ≠ 11) require another addressing byte, the SIB byte. It has the following fields:

• scale factor, encoded with bits [7:6]
• index register, bits [5:3]
• base register, bits [2:0].

The REX prefix provides additional space for encoding 64-bit addressing modes and additional registers present in the x86-64 architecture. Bit-field W changes the operand size to 64 bits, R expands reg to 4 bits, B expands r/m (or opreg in the few opcodes that encode the register in the 3 lowest opcode bits, such as "POP reg"), and X and B expand index and base in the SIB byte.

The VEX3 prefix contains all bit-fields from the REX prefix as well as various other prefixes, expanding addressing mode, register enumeration, operand size and width:

• R̅, X̅ and B̅ bits are inversions of the REX prefix's R, X and B bits; these provide a fourth (high) bit for register index fields (ModRM reg, SIB index, and ModRM r/m; SIB base; or opcode reg fields, respectively) allowing access to 16 instead of 8 registers.
• One W bit, equivalent to the REX prefix's W bit, specifies a 64-bit operand; for non-integer instructions, it is a general opcode extension bit.
• Four v̅ bits are the inversion of an additional source register index.
• One L bit indicates the vector length; 0 for 128-bit SSE (XMM) registers, and 1 for 256-bit AVX (YMM) registers.
• Two p bits encode additional prefix bytes. The values 0, 1, 2, and 3 correspond to implied no, 0x66, 0xF3, and 0xF2 prefixes. These encode the operand type for SSE floating-point instructions: packed single, packed double, scalar single and scalar double, respectively.
• Five m bits are used to specify opcode map to use. Of the 32 possible opcode maps that can be encoded with m₄m₃m₂m₁m₀ , opcode maps 1, 2 and 3 are used to provide compact replacements for legacy 2-byte and 3-byte opcodes - these three opcode maps are equivalent to leading escape byte sequences 0x0F, 0x0F 0x38 and 0x0F 0x3A, respectively. (The other VEX opcode maps have seen little use - as of December 2023, the only known uses of other maps are map 0 for the Xeon Phi-specific JKZD/JKNZD instructions^[2] and map 7 for the planned URDMSR/UWRMSR instructions.^[3])

The VEX2 prefix is a 2-byte variant of the VEX3 prefix, that differs from the latter in the following points:

• X̅, B̅ and W bits are not present.
• m bits are not present in the VEX2 prefix; the 0x0F escape is implied.

Instructions that require any of these bit-fields need to be encoded with the VEX3 prefix.

The REX2 prefix is a 2-byte variant of the REX prefix, introduced with Intel APX extensions which add 16 Extended GPR registers.

• R₃, X₃, and B₃ bits are the same as R, X and B bits in the REX prefix.
• R₄, X₄, and B₄ bits are additional bits used to encode the 32 EGPR registers.
• W bit is the same as in the REX prefix.
• M₀ bit selects between legacy map 0 (1-byte opcodes, no escape) and legacy map 1 (2-byte opcodes, escape 0x0F).

Instructions coded with the VEX prefix can have up to four variable operands (in registers or memory) and one constant operand (immediate value). Instructions that need more than three variable operands use immediate operand bits to specify a 4th register operand (IS4 above). At most one of the operands can be a memory operand; and at most one of the operands can be an immediate constant of 4 or 8 bits. The remaining operands are registers.

The AVX instruction set is the first instruction set extension to use the VEX coding scheme. The AVX instruction set uses VEX prefix only for instructions using the SIMD XMM registers.

However, the VEX coding scheme has been used for other instruction types as well in subsequent expansions of the instruction set. For example:

• BMI introduced VEX-coded arithmetic and bit manipulation instructions that operate on general purpose registers.
• AVX-512 introduced 8 mask registers and added VEX-coded instructions to manipulate them. (VEX.B̅ is ignored when the field is used to encode a mask register, but VEX.R̅ and VEX.v̅₃ are not, and must be set to 1 in 64-bit mode.^[4])
• AMX introduced 8 tile registers and added VEX-coded instructions to manipulate them.

The VEX prefix's initial-byte values, 0xC4 and 0xC5, are the same as the opcodes of the LDS and LES instructions. Not supported in 64-bit mode, the ambiguity is resolved in 32-bit mode by exploiting the fact that a legal LDS or LES's ModR/M byte cannot specify a register source operand; i.e., be of the form 11xxxxxx. Various bit-fields in the VEX prefix's second byte are inverted to ensure that the byte is always of this form. Similarly, the REX prefix's one-byte form has the four high-order bits set to four, which replaces sixteen opcodes numbered 0x40–0x4F. Previously, those opcodes were individual INC and DEC instructions for the eight standard processor registers; x86-64 code must use ModR/M INC and DEC instructions.^[5]

Legacy SIMD instructions with a VEX prefix added are equivalent to the same instructions without VEX prefix with the following differences:

• The VEX-encoded instruction can have one more operand, making it non-destructive.
• A 128-bit XMM instruction without VEX prefix leaves the upper half of the full 256-bit YMM register unchanged, while the VEX-encoded version sets the upper half to zero.
• 128-bit XMM instructions without VEX prefix usually require any memory arguments to be 16-byte aligned - VEX-encoded versions allow misaligned memory operands.

Instructions that use the whole 256-bit YMM register should not be mixed with non-VEX instructions that leave the upper half of the register unchanged, for reasons of efficiency.^[6][7]

The VEX prefix is not supported in real mode and virtual-8086 mode (all instructions with the VEX prefix will cause #UD in these modes).

• In August 2007, AMD proposed the SSE5 instruction set extension which includes a new coding scheme for instructions with three operands, using an extra byte named DREX, and intended for the Bulldozer processor core in 2011.^[8][9] However, in 2009, SSE5 was canceled and never implemented.
• In March 2008, Intel proposed the AVX instruction set, using the new VEX coding scheme.^[10]
• In August 2008, commentators deplored the expected incompatibility between AMD and Intel instruction sets, and proposed that AMD revise their plans and replace the DREX scheme with the more flexible and extensible VEX scheme.^[11]
• In May 2009, AMD announced a revision of the proposed SSE5 instruction set to make it compatible with the AVX instruction set and the VEX coding scheme. The revised SSE5 is called XOP.^[12]
• January 2011. The AVX instruction set is supported in Intel's Sandy Bridge microprocessor architecture.
• 2011. The AVX, XOP and FMA4 instruction sets, are supported in the AMD Bulldozer processor.^[13]
• 2013. The FMA3 instruction set is supported in Intel Haswell processors.
• In July 2023, Intel announced Advanced Performance Extensions (APX) which use REX2 prefix and updated EVEX prefix.