Given an observable random variable X over the probability space ${\displaystyle \scriptstyle ({\mathcal {X}},\Sigma ,P_{\theta })}$ , determined by a parameter θ ∈ Θ, and a set A of possible actions, a (deterministic) decision rule is a function δ : ${\displaystyle \scriptstyle {\mathcal {X}}}$ → A.

• An estimator is a decision rule used for estimating a parameter. In this case the set of actions is the parameter space, and a loss function details the cost of the discrepancy between the true value of the parameter and the estimated value. For example, in a linear model with a single scalar parameter ${\displaystyle \theta }$ , the domain of ${\displaystyle \theta }$  may extend over ${\displaystyle {\mathcal {R}}}$  (all real numbers). An associated decision rule for estimating ${\displaystyle \theta }$  from some observed data might be, "choose the value of the ${\displaystyle \theta }$ , say ${\displaystyle {\hat {\theta }}}$ , that minimizes the sum of squared error between some observed responses and responses predicted from the corresponding covariates given that you chose ${\displaystyle {\hat {\theta }}}$ ." Thus, the cost function is the sum of squared error, and one would aim to minimize this cost. Once the cost function is defined, ${\displaystyle {\hat {\theta }}}$  could be chosen, for instance, using some optimization algorithm.
• Out of sample prediction in regression and classification models.

• Admissible decision rule
• Bayes estimator
• Classification rule
• Scoring rule