Khan joined the Indian National Congress and was elected to the Lok Sabha in 1980 and 1984. In 1986, he quit the Indian National Congress due to differences over the passage of Muslim Personal Law Bill which was piloted by Rajiv Gandhi in the Lok Sabha.
Khan joined the Janata Dal and was re-elected to the Lok Sabha in 1989. He left the Janata Dal to join the Bahujan Samaj Party and again entered the Lok Sabha in 1998. Khan held ministerial responsibilities from 1984 to 1990.