The house was built around 1600 by Francis Buckland, opposite the Church of St Mary,^[5] and extended in 1650 by his son John Buckland MP, who married Elizabeth Phelips of Montacute House.^[6][1][7] It was inherited by John Buckland's daughter. She died without children and the house and surrounding manor was then owned by the Duchy of Cornwall from the end of the 17th century until the early 20th century.^[8] During this time, it was leased as a farmhouse.^[6]

The main house was restored and extended on the instructions of the future King George V, while he was its owner as Duke of Cornwall, with a view to becoming the home of his epileptic youngest son Prince John, but he died before moving in.^[7] The work was carried out under the supervision of the architect J. D. Coleridge at a cost of £2,000.^[6] New passages were constructed on the ground and first floors. The stables were converted into a servants' wing.^[6] In the First World War, it was used as a hospital, initially with fifteen beds, but there were 57 by the end of the war.^[9]

In 1928, the new Prince of Wales (the future Edward VIII) sold the house to Sir Edward Geoffrey Hippisley-Cox,^[7] who had occupied it as a tenant of the Duchy of Cornwall since 1913. Hippisley-Cox continued the restoration of the building and added a collection of antique and reproduction furniture which were contemporary with the house. In 1935, he had the small narrow passageway behind the front door made into part of the hall.^[6] He also laid out the grounds.^[10] The house is now the residence of Jacob Rees-Mogg, the Member of Parliament for North East Somerset, whose great-aunt served as a nurse at the house during the First World War.^[9] In 1964, his father William Rees-Mogg purchased the nearby Georgian mansion house Ston Easton Park, built by John Hippisley-Coxe (died 1769), a relative of his ancestress Dorothy Hippisley, the wife of John Mogg (died 1728) of nearby Farrington Gurney, whose son Richard Mogg (1690–1729) of nearby Chewton Mendip, purchased Cholwell, subsequently the principal seat of the Mogg family.^[11]

The house is built of red sandstone with Bath stone dressings and has a slate roof with Dutch gables. The two-storey north front has seven bays and a central porch with a balcony above it and Doric columns.^[1] An internal staircase rises from the east end of the inner hall to the first floor and has decorated covered urns as finials and pendants on the newel posts. The balusters form an arcade in Jacobean fashion.^[6] There are several decorated plaster ceilings and original fireplaces. The arms of the Buckland and Phelips families are carved on the outer porch and their crests on the inner porchway arch, along with the motto 'Altogether Vanity'.^[6]

The sandstone barn within the grounds was built in the late 17th century.^[2] The 1.5-metre-high (4.9 ft) gate piers were added in the 17th century.^[3] The entrance gates and railings were added between 1910 and 1913.^[4]