He was the son of Eliphalet Spencer (1760–1832) and Triphena (Austin) Spencer (1756–1825). He removed at a young age to Greenfield, Greene County, New York, and became a clothier, carpenter and joiner. In 1808, he removed to Lenox, Madison County, New York, studied law there in the office of his brother Gen. Ichabod S. Spencer (1780–1857), was admitted to the bar and practiced in partnership with his brother. In 1814, he married Clarissa Phelps (1788–1818), and they had two children. In 1819, he married Electa C. Dean (1796–1881), and they had eleven children.

In 1829, he removed to Utica, and practiced law in partnership with State Senator William H. Maynard who died in 1832.

Spencer was United States Attorney for the Northern District of New York from 1841 to 1845. In October 1841, he appeared for the defence in the trial of Alexander McLeod on a charge of murder in connection with the Caroline Affair, and secured McLeod's acquittal.

He was a member of the New York State Senate (5th D.) in 1846 and 1847. He was Mayor of Utica in 1848.

He was buried at the Forest Hill Cemetery, Utica.

• The New York Civil List compiled by Franklin Benjamin Hough (pages 135, 145 and 431; Weed, Parsons and Co., 1858)
• Annual Obituary Notices of Eminent Persons by Nathan Crosby (Boston, 1858; pg. 341ff)

• Joshua A. Spencer at Find a Grave