Elam was born in Nottoway County, Virginia on January 30, 1796, the son of Essex Elam and Lavinia Crowder. He enlisted in the army and served during the War of 1812, but saw no combat. He later served as Ensign aboard the U.S.S. Guerriere under the command of Captain Stephen Decatur, he was part of the crew that went on in 1815 to attack the Barbary Pirates off the coast of Algeria.

In 1820, Elam moved to Baton Rouge, Louisiana and became a lawyer. That year he married Rebecca Chambers. He engaged in a vigorous law practice and was politically active.

He ran unsuccessfully as a Whig for the United States House of Representatives in the Third District of Louisiana against John Bennett Dawson in 1843.^[1] Later, his son, James Essex Elam was elected Mayor of Baton Rouge, Louisiana.

James M. Elam died on November 7, 1856, and was buried in the family plot in the Magnolia Cemetery in Baton Rouge.^[2]