Plumpton Place looks onto the nearby north-facing escarpment of the South Downs, with Plumpton College (formerly Plumpton Agricultural College) and the 11th-century church of St Michael's and All Angels immediately adjacent to the west and Plumpton village some 500m to the east. There is an entrance formed of two cottages designed by Sir Edwin Lutyens, with a Palladian porch and this leads to his modern bridge over the moat.

It was built in 1568 on the site of an earlier house which was mentioned in the Domesday Book. The North and South parts of the house date from the 1400s, some of which incorporates local flint.

Various building materials have been used in the construction of the house. It is believed that the north wing is the earliest, as there is a date-stone of 1568 with the initials I.M. The west wing seems to date from a later period, circa 1600. Over a hundred years later there was a period of rebuilding in brick and some additions by Lutyens. There are lakeside gardens by Edwin Lutyens and Gertrude Jekyll, within large grounds, that include both woodland and pasture.

In 1927 Plumpton Place with its mill house and 60 acre estate was purchased for £3,300^[3] from John Pelham, 8th Earl of Chichester^[4] by Edward Hudson (1854–1936), the founder of Country Life magazine, who initiated a major restoration of the property, which had fallen into a state of disrepair, by hiring Lutyens to revamp the main house and mill house, and Jekyll to oversee the 60 acres of land and lakes.^[5]

Plumpton Place was purchased in 1938 for £9,000^[6] by George Miles Watson, 2nd Baron Manton (1899–1968), who established a race-horse stud there, having sold his former residence Compton Verney in Warwickshire. He completed the work left unfinished by Hudson and built the surviving (2023^[7]) "charming, 19-box stable yard"^[8] beside the large Grade II-listed Elizabethan barn, and produced at Plumpton the sprinter Hard Sauce,^[9] who sired Hard Ridden, winner of the Derby in 1958.

In 1969, following Lord Manton's death in the previous year, a "local doctor" (possibly in reality a solicitor^[10]) bought the property after Lady Manton – a woman with a distaste for long-haired rock stars – refused to sell it to George Harrison of the Beatles and his wife Pattie Boyd, the latter having recounted in her autobiography:

We searched and searched and finally we found the perfect house. It was called Plumpton Place ... and had been designed by Edwin Lutyens with a garden by Gertrude Jekyll. You went through a big gate with little gatehouses on either side then crossed a bridge over a moat which split into a lake, which split into another. A woman showed us round and in every room the wallpaper was decorated with birds. In the garden she had an aviary with about 200 budgerigars, plus the odd robin and sparrow that had found its way in. She said she had started with just a few but they had bred over the years. We fell in love with the house and put in an offer - but she turned it down. She said she didn't want rock'n'roll musicians buying her lovely house and sold it to the local doctor instead. He realised what a treasure he had bought and sold it to Michael Caine ... who sold it on to Jimmy Page.^[11]

Lady Manton moved to Belton House, Grantham, Lincolnshire, having remarried to Peregrine Cust, 6th Baron Brownlow, and in 1970 Harrison and his wife purchased Friar Park in Surrey.

The "local doctor" sold Plumpton in 1972 for £200,000^[12] to Led Zeppelin guitarist Jimmy Page,^[5] who owned the property from 1972 to 1985.^[13] It was then purchased for £650,000 by property developer Philip Gorringe, who sold shortly after for "a reputed"^[14] £800,000 to the American financier Thomas Perkins^[15] who in 2010 offered it for sale for £8 million.^[16]

Plumpton Place was used as the main location for the 2019 film adaptation of Carmilla by Sheridan Le Fanu.^{[citation needed]}