Originally the location for fishermen to launch their boats, Castletown later developed during the 19th century with the construction of Portland Harbour's breakwaters, and the establishment of the naval base.^[1][3] During the 16th century, Henry VII had selected Castletown area for the building of Portland Castle.^[4] Castletown was an essential part of the Merchant's Railway, which was operational from 1826 to 1939. At a time when stone was transported by sea, the railway brought stone from the quarries at Tophill to Castletown's pier for shipping.^[5][6]

During the early 19th century, Castletown was still only a small settlement, with few residential properties.^[1] By the middle of the century, Castletown's character changed.^[1] A one-sided terrace was formed by 1864, and a number of businesses and residential properties established.^[1] Due to it being the gateway to the naval base, Castletown became a thriving commercial area.^[7] In 1944 Portland's harbour was commissioned as USNAAB Portland-Weymouth, becoming a major embarkation point for American troops during D-Day. Castletown saw many soldiers and vehicles pass through.^[8]

In the early 1990s, as part of defence spending cuts, it was announced that naval base was to close. It was forecast that the Royal Navy's departure would bring economical disaster to Castletown.^[3] The navy left in 1995, and Castletown lost the majority of its income, leading to the gradual loss of a number of businesses.^[9]

In October 2014 Nemesis Properties applied for planning permission to redevelop Castletown Pier,^[10] and work began during 2015 on transforming the pier into "Crabbers Wharf". In June 2015 further development plans were revealed for the regeneration of Castletown, including a "scuba diving tourist facility, an American themed D-Day attraction and museum." It was also suggested that talks with Portland Port may allow Portland's two Mulberry Harbour Phoenix Units to be opened to the public.^[11]

A small beach is found in the village, close to the entrance of Portland Port. The area still features several hotels and pubs, largely established for the navy personnel that once came from the port. The Osprey Leisure Centre is located in Castletown, which was formerly known as the Boscawen Centre, and was opened to the public in May 2007.^[12] Originally it was a private Royal Navy facility during the 1980s.^[13]

Castletown also holds the Portland Community Hospital. This was originally a Royal Naval Hospital which served Portland's naval base from the late 19th century until 1957, when the hospital was handed over to the NHS. Castletown's Stone Pier, where stone was once shipped, is now mainly used for boat maintenance.^[3]

To the southern side of the main road of Castletown two large multi-story accommodation buildings sit set into the steeply rising hillside. These blocks were known as the "Hardy Complex" with their construction allegedly costing between £25-30 million and were built during the late 1980s and early 1990s as state-of-the-art accommodation for Royal Navy personnel, but were abandoned less than 15 years later when the Navy left the island.^[14] They were later sold by the MoD to a private developer/construction company, and then subsequently sold on again. Eventually they were acquired by development company Comer Homes, who secured planning consent to convert and refurbish the two buildings to provide a total of 554 1, 2, and 3 bedroom apartments and penthouses. During 2004-05 both blocks were stripped back to just their re-enforced concrete frames and left as empty shells. Re-construction work began on the right hand block, (subsequently named Ocean Views) in 2007 and the external works were largely completed by June 2008.^[15] Many of the completed apartments were rented as accommodation for visiting foreign sailing teams for their training, preparation and living quarters for the sailing events of the 2012 Olympics, which took place in Portland Harbour. Today the left block still remains derelict awaiting construction works to restart.^[16]

Castletown has a wide array of architecture and buildings, a number of which are Grade Listed.

Portland Castle has been designated as a Grade I Listed building. It is one of three buildings on Portland to be Grade I Listed.^[17] Additionally, in October 1981, the castle had become scheduled under the Ancient Monuments and Archaeological Areas Act 1979.^[18] The Captain's House, a large detached house, adjoining Portland Castle, is designated a Grade II* Listed building.^[19] The gateway and curtain wall to the south east of the house became Grade II* Listed at the same time.^[20] Located around 23 metres (75 ft) south of the entrance to The Captain's House is a War Department/Admiralty boundary marker, which has been Grade II Listed since May 1993.^[21]

The Royal Breakwater Hotel was designated Grade II in May 1993.^[22][23] At the end of the village, at the entrance of Portland Port, is the former Dockyard Police Station. The MOD police station was initially a railway station, and is Grade II Listed.^[24][25]

• Archive photographs of Castletown, Portland