Rowe was born in the environs of Richmond, Virginia, and went to work at a racetrack as a boy of 10 years old. He went from an apprentice rider to being considered the leading jockey in the U.S. at age 14, a position he kept from 1871 to 1873.

With age came weight, and in 1875, at the age of 18, Rowe retired from racing as a jockey. He began working for the Davis & Hall racing stable as an assistant to the trainer, David W. McConn. Rowe eventually became head trainer for the stable.

In 1879, Rowe joined the Dwyer Brothers Stable. On May 17, 1881, with the future Hall of Fame horse Hindoo, he became the youngest trainer to win the Kentucky Derby. Hindoo won eighteen straight races that year. The Dwyers pushed their horses hard, in a manner that would be unacceptable and illegal today. Rowe quit the Dwyers in 1885 in a dispute regarding over-racing their horses, especially the mare Miss Woodford.

Rowe trained for Alfred F. Walcott's Fairfax Stable, after which he was a public or freelance trainer until 1889. While he ran his own stable one of his clients was August Belmont. In 1891, Rowe was hired as the race starter at minor racetracks. In January 1892, he was hired by the Board of Control as starter for the Monmouth Park Association, Coney Island Jockey Club, Brooklyn Jockey Club, and the New York Jockey clubs.^[2]

In 1899, Rowe became head trainer for the racing operations of James R. Keene. In 1911, he took charge of the racing stable of Harry Payne Whitney, for whom he trained his second Kentucky Derby winner. Rowe was the leading money winner in horse racing in 1908, 1913, and 1915, the year the Whitney stables' Regret became the first filly to win the Kentucky Derby. Regret was named 1915's Horse of the Year. She was later elected to the Racing Hall of Fame. Rowe had four second-place finishes with horses in the Preakness Stakes and won it with Broomspun in 1921.

Despite his many victories in important races, it is for the performance of his horses in the Belmont Stakes that Rowe is most remembered. Between 1883 and 1913, he won the Belmont Stakes 8 times, more than any other trainer in history. One of those victories came in 1908 with a horse named Colin, who went unbeaten during his 15-race career. Rowe considered Colin to be the greatest horse he had ever trained. He once said that for his epitaph he wanted only these words: "He trained Colin."^[3]

Following the formation of the National Museum of Racing and Hall of Fame in 1955, James Rowe Sr. was inducted posthumously. His son, James Jr., also had a very successful career as a trainer but died at age 42 of a heart attack.^[4]

• Bowen, Edward L. (2007). Masters of the Turf: Ten Trainers Who Dominated Horse Racing's Golden Age. ISBN 978-1-58150-149-0.