Williams was born in Orangeburg, South Carolina. He began playing piano as a child and started playing saxophone at age twelve. He acquired a bachelor's degree in music education at South Carolina State College and then enrolled in a graduate program in music at Indiana University, but because the school did not have a saxophone performance concentration, he left after three semesters.^[1]

After leaving college, Williams relocated to Los Angeles and soon enlisted in the United States Army. He also studied at the United States Armed Forces School of Music. In the late-1960s, he did extensive work as a session musician in the horn sections of soul and blues groups, including Ike and Tina Turner and many Motown Records artists (The Four Tops, Marvin Gaye, Gladys Knight & the Pips, The Temptations, Stevie Wonder, Gerald Wilson, Louis Bellson, and The Frank Capp/Nat Pierce Juggernaut (Big Band). John also appears on several movie soundtracks with Quincy Jones including Cactus Flower. From 1970 to 1975 he was the baritone saxophonist in The Count Basie Orchestra, taking over for Cecil Payne, and then joined the group again in 1980 after the death of Charlie Fowlkes . Williams remained with The Count Basie Orchestra until his retirement in May 2013, continuing with the group after Basie died and leadership passed on to Thad Jones, Frank Foster, Grover Mitchell, Bill Hughes, and Dennis Mackrel, and Scotty Barnhart as of September 2013.

John resides in South Carolina and continues to be a mentor to members of the Count Basie Orchestra and other musicians who seek his advice and experiences counsel.