Brett was born in Windsor, Berkshire, the son of Oliver Sylvain Baliol Brett, 3rd Viscount Esher and Antoinette Heckscher (1888-1965). His paternal grandparents were Eleanor (née Van de Weyer) Brett and Reginald Brett, 2nd Viscount Esher, an MP and the Constable and Governor of Windsor Castle and a close friend and adviser of Edward VII and George V.^[4] His maternal grandparents were Anna (née Atkins) Heckscher and August Heckscher (1848–1941), a German-born American capitalist and philanthropist. His grandfather married Virginia Henry Curtiss after his grandmother's death in 1924.^[4]

He was educated at Eton and New College, Oxford, where he read history.^[3]

He proceeded to the Architectural Association, but left to learn from the traditionalist A. S. G. Butler and then, as a non-qualified partner of William and Aileen Tatton Brown, passed the RIBA external exams in the summer of 1939, winning the Ashpitel Prize.^[5]

He spent the Second World War mostly in Britain, training gunners in the Royal Artillery, until he went through France and Belgium to witness the surrender of Lübeck and Hamburg. In 1945, he stood as Liberal parliamentary candidate for Henley, coming third in the poll.^[3]

He formed a partnership with Kenneth Boyd to design new houses as Architect-Planner of Hatfield New Town and wrote the initial report of the Hatfield Development Corporation.^[6] In November 1957, some 50 of Hatfield's two-storey terraced houses lost their mono-pitched roofs in a storm, and the adverse publicity and financial liability ended his business. From this period, despite not wanting to be known as a country-house architect, he was most proud of small houses in Oxfordshire for Hans Juda and in Warwickshire for Lord Dormer. A design for the High Commissioner's residence in Lagos in 1958 was compromised by the taste of an incoming Commissioner's wife. A second practice terminated in 1971.^[3]

Esher's real interest was in planning, and he carried out a study of York for the government, after which he published York: a study in conservation (1968). After a period as Rector of the Royal College of Art (1971–1978) he turned again to writing. A Broken Wave: the rebuilding of England 1940-1980 (1981) was an attempt to chronicle and analyse the achievements of post-war architecture and planning, following on from Parameters and Images: architecture in a crowded world (1970).^[5][7] He served as president of the Royal Institute of British Architects from 1966 to 1967 and was appointed a Commander of the Order of the British Empire (CBE) in 1970.^[3][8]

National Life Stories conducted an oral history interview (C467/14) with Lionel Gordon Baliol Brett Esher in 1997 for its Architects Lives' collection held by the British Library.^[9]

In 1935, he married Helena Christian Pike, a painter. She was the daughter of Colonel Ebenezer John Lecky Pike and Olive Snell. Her sister, Katherine Mary Penelope Pike, was married to Lawrence Dundas, 3rd Marquess of Zetland. Together, Lionel and Helena were the parents of:^[10][3]

• Christopher Lionel Baliol Brett, 5th Viscount Esher (b. 1936)
• Hon. Michael Jeremy Baliol Brett (b. 1939)
• Hon. Guy Anthony Baliol Brett (1942–2021)
• Hon. Maurice Sebastion Baliol Brett (b. 1944)
• Hon. Olivia Clare Teresa Brett (b. 1947)
• Hon. Stephen Patrick Baliol Brett (b. 1952)

The Esher family lived in Watlington Park, a country house in the Chilterns, from where Lionel Brett also ran his architectural practice.^[11] He later gave the house to his eldest son, and built a house of his own design, named The Tower, on the grounds of the estate.^[12]

Esher's autobiography Ourselves Unknown records how he nursed his wife through a long mental breakdown in the 1960s, but notes that she gave him equal support and strength over nearly 70 years.^[4]

Viscount Esher died aged 90 in 2004.^[3][4]

• Photograph of Viscount Esher at the National Portrait Gallery, London
• Hansard 1803–2005: contributions in Parliament by the Viscount Esher