"It is much more likely to happen after the vote [referendum]. If Cameron loses the referendum, he has to go anyway. If it is very very close, he might be brought down as well. But attempts to bring him down before the referendum face a few problems," Brooks said.
He noted that his party, from the outset, has been opposed to the Treaty of Rome and the common market.
British politics has been swept by clashes over whether the country should remain in or leave the European Union, and a public vote on the issue is due to be held on the 23rd of June.
"If they try to move against Cameron before the referendum, it would be used by the Remain supporters to say they are trying to stop the referendum because they know [they] would lose," Brooks said.
Overall, he noted that the Conservative Party is in trouble and faces a lot of problems, because there is substantial support for Brexit – "there are very dissatisfied with what they feel as misinformation being given out by the Remain campaign led by the prime minister himself."
UK citizens are set to vote on June 23 in a referendum on the country's EU membership, after British Prime Minister David Cameron and the leaders of the 27 other EU member states reached a deal in February to grant the United Kingdom a special status within the bloc.
Supporters of the United Kingdom exiting the bloc argue that EU membership has eroded Britain’s independence to legislate, direct its economy and control its borders. Opponents warn that leaving the European Union could deeply harm the country’s economy.