Phillimore was born at 21 Chester Square, in Belgravia, London, on 20 December 1856, the only son of John George Phillimore and Rosalind Margaret, née Knight-Bruce. Orphaned on the death of his mother in 1871, he was taken into the care of his uncle Sir Robert Joseph Phillimore. He went to Westminster Boys' School in London. He took his BA from Christ Church, Oxford in 1879 and an MA in 1883. He was admitted as a lawyer to the Middle Temple in 1877.

While at Oxford, Phillimore met John Rhys, Whitley Stokes and others who stimulated his interest in Welsh culture. He learned the language and collected Welsh books and manuscripts. Phillimore developed his interest in the Welsh language producing articles for Bye-gones, Archaeologia Cambrensis and Y Cymmrodor, widely referenced as providing guidance on the meanings of words and names.^[2] From 1903 he lived at Corris in Snowdonia. He wrote an article on bawdy Welsh words for a German periodical, Kryptadia, published in 1884.^[3][4] Phillimore's papers, including some poetry of his own composition as well as translations of works in Welsh, Latin, French, German and Greek, are kept at the National Library of Wales.

Phillimore married Susan Elsie Roscow in 1880 with whom he had one son and three daughters. Susan Roscow died in 1893, and Phillimore subsequently married Marion Owen in 1897. He kept his marriage to Owen a secret from many people, including his own children by Roscow. Marion Owen died in 1904.

• Baner ac Amserau Cymru, 1 January 1941
• Who's who in Wales, 1921 and 1933
• Burke's Landed Gentry, 1871, 1089–90
• The Montgomeryshire County Times, 5, June 1937
• Alumni Oxonienses