Dorsey met songwriter/producer Allen Toussaint at a party in the early 1960s, and was signed to the Fury record label. The song that launched his career was inspired by a group of children chanting nursery rhymes - "Ya Ya" went to number seven on the Billboard Hot 100 in 1961. He recorded other songs for Fury before the label folded, and Dorsey went back to his car repair business.
Toussaint later came back on the Amy label and began to work with Dorsey once again. From 1965 to 1969 Dorsey put seven songs in the Hot 100, the most successful of which was "Working In The Coal Mine" in 1966. It was to be his second and last Top Ten song. In 1970 Dorsey and Toussaint collaborated on an album entitled Yes We Can; the title song was Dorsey's last entry in the singles chart. It was later a hit for the Pointer Sisters under the title, "Yes We Can Can".