Female politicians have to work hard to be "noticed and seen" in the "bang-on-tables" world of Westminster, one of David Cameron's ministers has said.

Lady Verma of Leicester, an energy minister, said the way women have to "raise our tempo" to be heard is an unattractive aspect of politics today.

In an interview with the Guardian, the senior Tory said the presence of more female politicians has made debate "much more tempered" but it could still be improved by having more women right at the top of the coalition.

Asked about the biggest barrier to women in politics, Verma said it was "the way we conduct it", adding that there are "huge obstacles and challenges".

"I'm not a supporter of the bang-on-tables and be adversarial in politics and I still think that puts a lot of women off," she said. "Anywhere where we have to raise our tempo to be noticed and seen I just find unattractive and I think most people generally find that unattractive. We spend all our lives telling children not to do it and then we get older, go into politics and have to do it."

Her intervention comes at a critical time for female MPs and peers as Cameron and Ed Miliband, the Labour leader, are said to be mulling re-shuffles that are likely to see more women promoted. But in a sign that female elevation remains controversial, last week, Alan Duncan, a Tory development minister, warned Cameron against promoting "token women" at the expense of more talented men.

Verma, a former businesswoman who started her first company at 19, said female politicians should only be promoted on merit but pointed out there are "fabulous women" in parliament who deserve top jobs. She said Cameron "absolutely" needs more female ministers, while praising him for helping to make politics "a more diverse pot" than before.

The energy minister, speaking after an event on women in green industry, said there are similar challenges for female businesswomen and politicians. She added that she wanted to see more companies, especially big ones, offering flexible working now that technology allows people to work from home and different hours.

However, Verma said she would also like to see women put themselves forward for difficult jobs, be unafraid of failure and try not to be perfectionist. She said bringing up a family at the same time as running her company made her realise that she would have to compromise on keeping a "perfect house".

She added: "We've to got to look at how businesses, institutions organise themselves, but it's also about our self-attitude," she said. "Part of the problem is we over-analyse ourselves, we have an over-expectation of ourselves [as women]."

Verma was made a peer in 2006, and joined Cameron's government at the start of the coalition three years ago. Born in Punjab, India, she moved to Britain with her parents aged one. She then attended "a school that is self-deemed as one of the worst in Leicester" before her family stopped her from going to university and she decided to set up a fashion business."I have worked very hard to stand on my own two feet against a backdrop of a community where I come from that was incredibly male dominated, into a world of business that was incredibly male dominated into a world of politics, again very male-dominated," she said.

Despite shying away from calling herself a "feminist", she said her mission in life is to remove obstacles for ambitious women, especially those from ethnic minority backgrounds.

"Sexism you can deal with it by saying it exists, it shan't hold me back," she said. "When my father said I couldn't go to university, it was my mission to ensure anybody from an ethnic minority community who wanted to should never be stopped.

"Although my father passed away being incredibly proud of my achievements and recognising at the end perhaps he'd stopped somebody quite bright in her studies from going on, I've kept it as a mission that wherever I see obstacles for women I make sure that I help remove them."

Verma said she has "objected to quotas forever-more" because she would "hate" to be seen as "ticking two boxes, as a BME and a woman". However, the Tory minister said she did not retreat from pushing for female equality without being "biased towards women". She added: "Raising your head above the parapet isn't always a comfortable place".