Morgan was born in South Amboy, New Jersey, in 1756, the son of Captain James Morgan Sr. and Margaret Evertson. The family's compound in Sayreville, New Jersey, known as Morgan Manor, was initially established by Margaret's father Nicholas Evertson.

He attended the public schools in New Jersey, and later served as an officer in the New Jersey Line during the Revolutionary War. He went on to serve as a representative in the general assembly in Philadelphia, 1794–1799, and, later, was elected as a Democratic-Republican to the Twelfth Congress (March 4, 1811 – March 3, 1813). He was also engaged in agricultural pursuits, and became a major general of militia. Middlesex County, New Jersey, historical records indicate that he was an enslaver and held at least three people in slavery in the 1810s.^[1] He died in Morgan Manor, New Jersey, on November 11, 1822 and was interred in the Morgan private cemetery.

• United States Congress. "James Morgan (id: M000951)". Biographical Directory of the United States Congress.