Image copyright PA
Conservative chairman Brandon Lewis has asked Boris Johnson to apologise for saying Muslim women wearing burkas "look like letter boxes".
Mr Johnson said full-face veils should not be banned but looked "ridiculous".
He has been criticised by Labour MPs, some Tories and Muslim groups, who said the party was not doing enough to tackle prejudice.
Earlier, Foreign Office Minister Alistair Burt said there was a "degree of offence" in Mr Johnson's comments.
On Twitter, Mr Lewis said he agreed with Mr Burt and called on the MP and former London mayor to apologise.
Speaking on BBC Radio 4's The World at One, Lord Sheikh, who founded the Conservative Muslim Forum, suggested Mr Johnson was "using Muslims as a springboard" for his ambition to lead the Tory Party.
"It is a joke but in very, very bad taste," he said, adding that the former foreign secretary had a "weird sense of humour".
Former Tory chairwoman Lady Warsi welcomed Mr Lewis's intervention and called for disciplinary action against Mr Johnson if he did not apologise.
Labour's equalities spokeswoman Naz Shah said Mr Johnson should attend "training and engagement with the Muslim community" and called on Prime Minister Theresa May to respond.
"Clearly the Tory party has an issue with Islamophobia, but over 24 hours later the prime minister is still yet to say a word," she added.
But Conservative backbench MP Andrew Bridgen said Mr Johnson had raised an important subject in a "light-hearted way".
The reaction "says a lot about internal Conservative Party politics" he told the BBC.
Another Conservative MP, Nadine Dorries, said the government should apologise and that "Boris didn't go far enough".
In his Daily Telegraph column, Mr Johnsonwho last month quit the government in protest at Prime Minister Theresa May's Brexit policywas commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgeryand schools and universities should be able to take the same approach if a student "turns up... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agreeand I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West and could lead to "a general crackdown on any public symbols of religious affiliation".