The site was previously occupied by "The Drong", a number of cottages operated by a church charity as almshouses. The town hall project was instigated by George Burt, as a means of instilling civic pride in the residents.^[1] Burt was a local building contractor who managed his uncle's construction firm Mowlem which carried out work on prominent buildings in London. Many of these were built with Purbeck stone shipped by barge from Swanage harbour. The barges required ballast to stabilise them for the return journey and Burt used material salvaged from buildings demolished in the capital for this. Many of these structures were re-erected in Swanage.^[2] The town hall reused the façade of the 1670 Mercers' Hall, designed by Edward Jerman, who was a pupil of Sir Christopher Wren. The town hall features original carvings of the Mercers' Maiden and two cherubs from the Mercers' Hall but other sculptures were destroyed in transit.^[1]

The architect of the town hall was George Rackstraw Crickmay.^[3] His plans were approved in July 1882 and the building was completed by 1883, at a cost of £4,500. The building was originally known as King Alfred Hall.^[1]

The building is two storeys tall and constructed in Purbeck stone. There is a balcony on the front, accessed by two doors set in a semi-circular arch. The balcony has a small hood, Tuscan pilasters either side and a pediment above (on which a clock is mounted). The façade extends above the roof of the main building where it features a decorative stone balustrade.^[3] The clock on the front of the building was made by Thwaites & Reed of Clerkenwell, London, in 1826. It was possibly salvaged from the church of St Mary Somerset in Upper Thames Street, London, which was demolished in 1872. Other features likely to have been brought from London by Burt are the lamp posts either side of the entrance, two sculpted heads on the north and east faces of the structure and a pediment in the council chamber with the inscription "Fear God, Honour the King". A set of iron columns in the basement are thought to have been salvaged from Billingsgate Market in 1874. A bust of Burt was placed in the council chamber.^[1]

The building did not receive universal approval. An architectural critic writing in the 1930s described the way that the façade had been incorporated into the building as "positively dreadful" and stated that "if ever a book comes to be written on 'How to murder Architecture', the Swanage Town Hall should find a place therein".^[1]

Burt leased the building to the Swanage Local Board of Health, which was the first civic government of the town.^[1] The basement was used to house the town's fire engine and a bell was installed on an external wall to be used to call the brigade into action. The association of the hall with this use was so strong that the road it stood on, Town Hall Lane, became commonly known as "Fire Bell Lane". One of the upstairs rooms was used as a magistrates' court, conveniently located for the police station opposite, and for this purpose the building had a mobile witness box. The building also housed meetings of the town's Pier Company and Cottage Hospital Committee as well as lectures, property auctions and dances.^[1]

The town hall and police station were both sold by the Burt family to the Swanage Urban District Council between 1919 and 1921. The council modernised the structure in the 1920s, installing central heating and electrical lighting. The clock on the structure has been kept wound by members of the same family since 1933. During the Second World War the hall was used by members of the Air Raid Precaution and Women's Royal Voluntary Service and an air raid siren was mounted to the building. The structure escaped damage, though a neighbouring cottage was destroyed by German bombing.^[1] The structure, as well as an adjoining building, were designated as listed Grade II on the National Heritage List for England on 26 June 1952.^[3][4] Immediately behind the town hall, but pre-dating it, is the Swanage lock-up. Dating from 1803, it is a scheduled monument.^[5]

In 1977 the Town Hall was a set for the BBC production of The Mayor of Casterbridge, which aired in 1978.^[1] In the late 20th century the building hosted the Purbeck Citizens Advice and until 2006 was used for election vote counting.^[6][1] The register office in the building was proposed for closure in 2017, though the structure would remain in use as a location for ceremonies such as weddings.^[7][8] The upstairs chamber remains in use for council sessions while the former magistrate's room serves as a committee and meeting room.^[1]

Swanage Town Hall holds a number of artworks, including 1901 busts of King Edward VII and Queen Alexandra by Sydney March,^[9][10] a bust of the local businessman, George Burt, by an unknown artist^[11] and a 1931 painting of a local scene by Henry Justice Ford.^[12]