Williams began playing piano at age 13 and settled on trumpet soon afterwards; he may have played with blues pianist/singer Cow Cow Davenport while still a teenager. In the 1930s, he played in the territory bandsTrianon Crackerjacks and Brown Skin Models, and acted as musical director for the Dixie Rhythm Girls. Around 1940, he left Alabama for Philadelphia, where he played with Tiny Bradshaw's band before joining the U.S. Army during World War II.
After the war, Billy Eckstine hired Williams, and following this he worked with John Kirby and pianist Billy Kyle. In 1949, he began the first of several stints with Duke Ellington, who bestowed upon him the nickname "Cadillac". In 1951, he left Ellington's employ and moved to Paris, where he led his own bands and recorded for French labels. He returned to Ellington in 1956, and played with him again in 1969 on a tour of Europe.[1] He settled in the Netherlands, and died in 1973.