Constructed in 1694, as Prospect Pen, by Sir William Taylor,^[2] it became the property of Scottish sugar planter Patrick Tailzour following his marriage to Martha Taylor, daughter of George Hanbury Taylor and Mary, of Caymanas, Jamaica. In turn he who would pass it in to his Jamaican born son Simon. Simon never married, instead electing to father children with his slaves, and Free Jamaicans of color; on his death the property would pass to his nephew, Sir Simon Richard Brissett Taylor, the son of his brother Sir John. On Sir Simon's death the property would pass to his sister Anna Susannah (1781–1853), and her planter husband George Watson-Taylor (1770–1841), the fourth son of George Watson of Saul's River, Jamaica.^[3] George later became the Liberal MP, for Devizes, England, and campaigner for the retention of slavery. Following abolition George Watson-Taylor's finances collapsed, with the property reverting to his wife, on his death, in Edinburgh, in June 1841. Later passing to their son Simon (1811 – 25 December 1902), in 1852.

In 1928 the property was acquired by the British government and was the official residence of the colonial secretary until independence in 1962. In 1980, Vale Royal took over from Jamaica House as the official residence of the prime minister.^[4] Latterly the prime minister has returned to Jamaica House and the Vale Royal building has fallen into disrepair.^[4][5]