Summary
A symlink race is a kind of software security vulnerability that results from a program creating files in an insecure manner.[1] A malicious user can create a symbolic link to a file not otherwise accessible to them. When the privileged program creates a file of the same name as the symbolic link, it actually creates the linked-to file instead, possibly inserting content desired by the malicious user (see example below), or even provided by the malicious user (as input to the program).
It is called a "race" because in its typical manifestation, the program checks to see if a file by that name already exists; if it does not exist, the program then creates the file. An attacker must create the link in the interval between the check and when the file is created.
A symlink race can happen with antivirus products that decide they will quarantine or delete a suspicious file, and then go ahead and do that. During the interval between decision and action, malicious software can replace the suspicious file with a system or antivirus file that the malicious software wants overwritten.[2]
Example edit
In this naive example, the Unix program foo is setuid. Its function is to retrieve information for the accounts specified by the user. For "efficiency", it sorts the requested accounts into a temporary file (/tmp/foo naturally) before making the queries.
The directory /tmp is world-writable. Malicious user Mallory creates a symbolic link to the file /root/.rhosts named /tmp/foo. Then, Mallory invokes foo with user as the requested account. The program creates the (temporary) file /tmp/foo (really creating /root/.rhosts) and puts information about the requested account (e.g. user password) in it. It removes the temporary file (merely removing the symbolic link).
Now the /root/.rhosts contains password information, which (if it even happens to be in the proper format) is the incantation necessary to allow anyone to use rlogin to log into the computer as the superuser.
Also in some Unix-systems there is a special flag O_NOFOLLOW for open(2) to prevent opening a file via a symbolic-link (dangling or otherwise). It's become standardized in POSIX.1-2008.
Workaround edit
The POSIX C standard library function mkstemp can be used to safely create temporary files. For shell scripts, the system utility does the same thing.