Wheeler was born in Humphreysville, Connecticut (now Seymour), Connecticut on February 11, 1823. He attended the common schools in Cheshire, Connecticut and moved to New York City in 1843. He was engaged in the hotel business with his father, and later became a dry-goods clerk.

A Democrat, he was elected to the Thirty-third and Thirty-fourth Congresses (March 4, 1853 – March 3, 1857) as the representative of New York's sixth district. Wheeler declined to be a candidate for renomination in 1856 to the Thirty-fifth Congress.

Wheeler served as commissioner and president of the New York City's Department of Taxes and Assessments from 1872 to 1880, and later served as a member of the Board of Estimates and Apportionments and Commissioner of Accounts.

In the 1870s he was a member of the Committee of Seventy, a group of anti-Tammany Hall Democrats who worked to overthrow William M. Tweed, and elected William Frederick Havemeyer as Mayor of New York City.

He died in New York City on April 1, 1906 and was buried at Woodlawn Cemetery.

• United States Congress. "John Wheeler (id: W000337)". Biographical Directory of the United States Congress.