Worsley was born in Stonegrave, Yorkshire, the fifth of eight sons of Rev. George Worsley and Anne Cayley (daughter of Sir Thomas Cayley). He entered Trinity College, Cambridge as a pensioner in 1815. He matriculated in 1816, became a scholar in 1818, and was awarded the degree of Bachelor of Arts in 1820.

He became a Fellow of Downing College, Cambridge in 1824, and was elected the third Master of the college in 1836 in succession to William Frere. He served as Master for nearly 49 years up to his death in the Master's Lodge in 1885, thus being (in 2018) the third longest serving Head of House at Cambridge University, after Edward Atkinson, Master of Clare College from 1856 to 1915, and Francis Barnes, Master of Peterhouse from 1788 to 1838.^[2] He was Vice-Chancellor of Cambridge University from 1837 to 1838, and is buried in Hovingham, Yorkshire.^[1]

In 1842, he married Katharine Rawson of Wasdale Hall, Wasdale, Cumberland. They had no children.^[3] His widow post-deceased him on 6 March 1885.^[1] Thomas Worsley's brother, William, was the first Baronet of Hovingham Hall. William's great-great-granddaughter Katharine, Duchess of Kent, is the Patron of Downing College.^[4] Thomas's great-nephew Frank Worsley was a celebrated explorer. Sir George Cayley, aviator and engineer, was his uncle on his mother's side, and Arthur Cayley, mathematician, was a cousin of his mother.