The double act have attracted widespread criticism for their performances but are currently in the latter stages of the ITV1 show.
The prime minister told Manchester radio station Key 103 today that he does not rate the duo, though added that some of the hopefuls are "really good".
During a Q&A session, he stated: "I don't think [John & Edward] are very good. [But] there are some really good people on X Factor. It's really good. And I know it's competing with Strictly Come Dancing."
Brown later asked: "Does anyone want John & Edward to win X Factor?" Following a brief pause, only two said yes.
Conservative Party leader David Cameron yesterday admitted that he is addicted to John & Edward.