Tessellation


OpenGL Rendering Pipeline

Tessellation is the Vertex Processing stage in the OpenGL rendering pipeline where patches of vertex data are subdivided into smaller Primitives. This process is governed by two shader stages and a fixedfunction stage.
The tessellation process is divided into three stages which form an optional part of Vertex Processing in the rendering pipeline. Two of the stages are programmable; between them is a fixed function stage. They are described below, in the order they are processed.
Generally, the process of tessellation involves subdividing a patch of some type, then computing new vertex values (position, color, texture coordinates, etc) for each of the vertices generated by this process. Each stage of the tessellation pipeline performs part of this process.
The Tessellation Control Shader (TCS) determines how much tessellation to do (it can also adjust the actual patch data, as well as feed additional patch data to later stages). Therefore, the TCS is primarily responsible for ensuring continuity across patches. So if you have two adjacent patches that need to have different levels of tessellation, the TCS invocations for the different patches need to use their tessellation controls to ensure that the shared edge(s) between the patches use the same level of tessellation. Without this protection, gaps and breaks in what are supposed to be contiguous patches can occur.
The TCS is optional; default tessellation values can be used if no TCS is provided.
The tessellation primitive generator takes the input patch and subdivides it based on values computed by the TCS or provided as defaults.
The Tessellation Evaluation Shader (TES) takes the tessellated patch and computes the vertex values for each generated vertex.
Patches
Tessellation stages operate on patches, a primitive type denoted by the constant GL_PATCHES. A patch primitive is a generalpurpose primitive, where every n vertices is a new patch primitive. The number of vertices per patch can be defined on the applicationlevel using:
void glPatchParameteri(GLenum pname, GLint value);
with GL_PATCH_VERTICES as target and a value which has is on the halfopen range [1, GL_MAX_PATCH_VERTICES). The maximum number of patch vertices is implementationdependent, but will never be less than 32.
Patch primitives are always a sequence of individual patches; there is no such thing as a "patch strip" or "patch loop" or such. So for a given vertex stream, every group of value number of vertices will be a separate patch. If you need to do something like triangle strips, you should use Indexed Rendering to get similar behavior, though it will not reduce the number of vertices in the index list. Fortunately, the PostTransform Cache should deal with any performance impact having more indices.
Tessellation control shader
The first step of tessellation is the optional invocation of a tessellation control shader (TCS). The TCS has two jobs:
 Determine the amount of tessellation that a primitive should have.
 Perform any special transformations on the input patch data.
The TCS can change the size of a patch, adding more vertices perpatch or providing fewer. However, a TCS cannot discard a patch (directly; it can do so indirectly), nor can it write multiple patches. Therefore, for each patch provided by the application, one patch will be provided to the next tessellation stage.
The TCS is optional. If no TCS is active in the current program or program pipeline, then the patch data is passed directly from the Vertex Shader invocations to the tessellation primitive generation step. The amount of tessellation done in this case is taken from default values set into the context. These are defined by the following function:
void glPatchParameterfv(GLenum pname, const GLfloat *values);
When pname is GL_PATCH_DEFAULT_OUTER_LEVEL, values is a 4element array of floats defining the four outer tessellation levels. When pname is GL_PATCH_DEFAULT_INNER_LEVEL, values is a 2element array of floats defining the two inner tessellation levels.
These default values correspond to the TCS perpatch output variables gl_TessLevelOuter[4] and gl_TessLevelInner[2].
Tessellation primitive generation
Primitive generation is a fixedfunction stage responsible for creating a set of new primitives from the input patch. This stage is only executed if a tessellation evaluation shader (TES) is active in the current program or program pipeline. Primitive generation is affected by the following factors:
 The tessellation levels, provided either by the TCS or the default values, as stated above.
 The spacing of the tessellated vertices, as defined by the subsequent TES stage. It may be equal_spacing, fractional_even_spacing, or fractional_odd_spacing.
 The input primitive type defined by the subsequent TES which may be one of triangles, quads or isolines. The TES can also force the generation of the tessellation as a series of points rather than triangles or lines by providing the point_mode primitive.
 The primitive generation order defined by the subsequent TES, which may be cw or ccw. This, coupled with the position of the generated primitives, is used to determine the primitive's Winding Order.
Abstract patch
Notice that the primitive generation is not affected by the userdefined outputs of the TCS (or vertex shader if no TCS is active), the TCS's output patch size, or any perpatch TCS outputs besides the tessellation levels. The primitive generation part of the tessellation stage is completely blind to the actual vertex coordinates and other patch data.
The purpose of the primitive generation system is to determine how many vertices to generate, in which order to generate them, and what kind of primitives to build out of them. The actual pervertex data for these vertices, such as position, color, etc, is to be generated by the TES, based on information provided by the primitive generator.
Because of this dichotomy, the primitive generator operates on what could be considered an "abstract patch". It doesn't look at the patch output from the TCS; it thinks only in terms of tessellating an abstract quad, triangle, or "isoline" block.
Depending on the abstract patch type, the primitive generator evaluates a different number of tessellation levels and applies different tessellation algorithms. Each generated vertex has a normalized position (i.e. the coordinates are on the range [0, 1]) within the abstract patch. This position has two or three components, depending on the type of the patch. The coordinates are provided to the TES via the builtin in vec3 gl_TessCoord input.
Tessellation levels
The amount of tessellation that is done over the abstract patch type is defined by inner and outer tessellation levels. These, as previously stated, are provided either by the TCS or by context parameters specified via glPatchParameter. They are a 4vector of floats defining the "outer tessellation levels" and a 2vector of floats defining the "inner tessellation levels."
The specific interpretation depends on the abstract patch type being used, but the general idea is this. In most cases, each tessellation level defines how many segments an edge is tessellated into; so a tessellation level of 4 means that an edge will become 4 edges (2 vertices become 5). The "outer" tessellation levels define the tessellation for the outer edges of the primitive. This makes it possible for two or more patches to properly connect, while still having different tessellation levels within the patch. The inner tessellation levels are for the number of tessellations within the abstract patch.
Not all abstract patches use the same number of values in the outer/inner tessellation levels data. For example, triangles only uses one inner level and 3 outer levels. The rest are ignored.
The patch can be discarded if any outer tessellation level is 0 or less, but only for tessellation levels that the abstract patch actually uses. The patch can also be discarded if one of these values is a floatingpoint NaN. A patch that is discarded does not get tessellated, and no TES is invoked for it. It is simply swallowed by the system as though it never were.
This allows a TCS to effectively cull patches by passing 0 for a relevant outer tessellation level.
The tessellation levels specified in this way are not directly used. They go through a clamping process to generate the effective tessellation levels that are used to tessellate the primitive. This process depends on the TES's spacing parameter.
In the below discussion, max is the maximum allowed tessellation level, as defined by the GL_MAX_TESS_GEN_LEVEL. It must be at least 64, so you have some room to play with.
The spacing affects the effective tessellation level as follows:
 equal_spacing
 Each tessellation level is individually clamped to the closed range [1, max]. Then it is rounded up to the nearest integer to give the effective tessellation level.
 fractional_even_spacing
 Each tessellation level is individually clamped to the closed range [2, max]. Then it is rounded up to the nearest even integer to give the effective tessellation level.
 fractional_odd_spacing
 Each tessellation level is individually clamped to the closed range [1, max  1]. Then it is rounded up to the nearest odd integer to give the effective tessellation level.
Edge tessellation spacing
At various points in the forthcoming discussion about tessellating the abstract patch, there will be statements that say to tessellate an edge of some primitive. This means to subdivide it into a series of segments according to some tessellation level. The total length of these segments is the length of the original segment. However, the length of individual segments depends on the spacing parameter specified in the TES.
Given an effective tessellation level, denoted by n, which applies to that edge, the vertices for an edge tessellated by n is defined as:
 equal_spacing
 The edge is divided into n segments. All segments will have equal length.
 fractional_even_spacing, fractional_odd_spacing
 If n is 1, then no subdivision occurs. Otherwise, the edge will be divided into n  2 segments of equal length, as well as 2 segments that have length equal to each other, but not necessarily to the first group. The length of these 2 segments, relative to the others, will be n  f, where f is the effective tessellation level value after clamping but before being rounded up.
 When n == f, the length of the 2 segments will be equal to the length of the other segments. As n  f approaches 2.0, the relative length of the 2 segments approaches 0.0.
 The exact location of the 2 shorter segments is not defined, but they should be placed symmetrically, on opposite sides of the subdivided edge. Also, the location must be invariant with the same f value (thus allowing shared tessellated edges to always generate the same vertices).
The purpose of the fractional spacing modes is to have smoother, more stable interpolation as tessellation levels change. This is best used if tessellation levels are based on the distance to the camera or something.
Primitive generation order
When rendering without tessellation, the order of primitives relative to one another is well defined. Rendering requires setting up a vertex stream. This defines an ordered sequence of vertices. The Primitive specified at rendering time determines exactly how the stream is broken down into base primitives. And thus, each primitive is ordered based on the vertices that generate it.
Tessellation makes things less well defined. While the order of vertices within a primitive is well defined (determined by the TES's input layout settings cw or ccw), the order of primitives generated relative to one another is not. Implementations define this ordering, so you cannot rely on any particular order.
Then again, most ways you could even tell rely on incoherent memory accesses...
Tessellating primitives
Triangles
The abstract patch of the triangle tessellation is a triangle, naturally. Only the first three outer tessellation levels are used, and only the first inner tessellation level is used.
Each vertex generated and sent to the TES will be provided Barycentric coordinates as the gl_TessCoord input. This defines where this vertex is located within the abstract triangle. With this coordinate, it is possible to multiply any vertex attribute from 3 vertices to compute the appropriate value from the tessellation unit.
Quads
The abstract patch of a quad is a rectangle, naturally. All 4 outer and 2 inner tessellation levels are used.
Each vertex generated and sent to the TES will be provided a normalized 2D coordinate as the gl_TessCoord input, representing the location of that vertex within the abstract patch.
Isolines
The abstract patch of isolines is a rectangle, oddly enough. Only the first two outer tessellation levels are used; none of the inner tessellation levels are used.
The rectangular abstract isolines patch represents a series of horizontal lines, as shown to the right. The first outer tessellation level defines how many lines are in the patch, and the second defines how many segments the lines are divided into. So if you want to tessellate a single line, you should pass 1 for gl_TessLevelOuter[0]. However, if you need more line tessellation than can be done in a single outer level value, the TES is perfectly capable of stitching multiple separate lines together. Remember: the abstract patch is abstract; the TES can position the vertices wherever it wants.
Each vertex generated and sent to the TES will be provided a normalized 2D coordinate as the gl_TessCoord input. The y component specifies which line (normalized to the halfopen range [0, 1) ) is being generated. The x component defines how far along that line this vertex should be generated for.
Examples
Isolines with tess levels (4,3)  Isolines with tess levels (6,2) 
Tessellation evaluation shader
The Tessellation Evaluation Shader (TES) is responsible for taking the abstract coordinates generated by the primitive generator, along with the outputs from the TCS (or vertex shader, if no TCS is used), and using them to compute the actual values for the vertices. This is where you code the algorithm that you actually use to compute the new positions/normal/texcoords/etc. The TES is a mandatory part of tessellation; if one is not present, then tessellation doesn't happen.
The TES is rather like a vertex shader, in that each invocation operates on a distinct vertex within the tessellated patch (though, as with a vertex shader, the system may call the TES more than once for the same vertex, so it should be deterministic). Also, the TES cannot cull vertices.
Patch interface and continuity
Tessellating patches so that there are no gaps between primitives generated by adjacent patches is very important. It also is not something that simply happens. Because OpenGL is blind to how the actual tessellation works (the primitive generator only deals with the abstract patch), there are specific things that you must do to ensure gapless tessellation between patches.
Rule 1: The TES implementation of interpolation for the vertices regarding patch edges must receive binaryidentical input values for those edges. The TES defines what the various patch data actually means, as it implements the interpolation. So your TCS, or pervertex patch data in general, must make sure that the data the TES uses to generate those edge positions is the same for the shared edge on the two (or more) patches.
Fortunately, OpenGL's invariance rules means that shader computations that perform the exact same match will produce the exact same results. So usually, this only means making sure that the inputs to the TCS which contribute to interpolation along the edges get the same values from the vertex arrays, as processed through the Vertex Shader. However, different methods of tessellation and interpolation, as implemented by your TES, may have specific needs.
Rule 2: The outer tessellation level for the shared edges must be binaryidentical. While you could simply ensure that the "effective tessellation level" of the edges is identical, the specification says that tessellation invariance is only guaranteed if the given levels are identical. So it's best to be safe by ensuring that the two edges that are supposed to join together get the exact same tessellation level.
The above two rules will ensure that the TES will receive binary identical values for the patch data used to compute shared edge vertices, as well as exactly matching gl_TessCoord values for those vertices along the shared edge. Therefore, the only other rule that must be followed is:
Rule 3: The TES must calculate edge vertices based only on the binary identical data using the exact same math between the shader invocations along the shared edges. OpenGL's standard shader invariance rules work so long as the same codepath is used for both invocations.
The complete invariance rules for tessellation, from OpenGL 4.5, Appendix A.4, page 632, explain how OpenGL provides those guarantees.
Despite the seeming complexity of these rules, they're really not hard to follow. Rule #1 can be hard to satisfy when using unusual tessellation algorithms, but for bezier or NURBS surfaces, so long as the control points are continuous in accord with those surface rules, these will be satisfied.
The most difficult one to satisfy generally would be rule #2, as it complicates the computations for outer tessellation levels. It can't simply be based on the distance to the overall patch; each patch has to know what the neighboring patches used so that they can all match. Essentially, you would have to compute the outer tessellation levels for a patch based on the distance for each edge to the camera. Or some similar algorithm that would be invariant across patches.
Rule 3 simply requires your shader to be reasonably sane, that it doesn't contain conditional logic that causes different patches to be generated using different math. Though if you need continuity between separate programs, that becomes a bit more challenging.
Examples
 OpenGL 4 Tessellation Shader Tutorial: A simple OpenGL tutorial with source code for using Tessellation Shaders. Based on VC++ 2010.
 Drawing curves using Tessellation Shaders: A tutorial for the rasterization of parametric curves using Tessellation Shaders with source code.