The two non-celebrity editions of the reality show that have aired on Channel 5 since 2011 both pulled in a series average of 1.6m viewers.
"The reason why it was down - we had to deal with one of the biggest, nastiest summers for any channel that's not the BBC, in terms of Wimbledon, the Euros..." Ford said at the MediaGuardian Edinburgh Television Festival.
"It's the lowest ever [figures for Big Brother] but this is Channel 5, it's not Channel 4. If you'd stuck it on ITV, it might do 4m. Do I want more? Of course, I do."
Quizzed on housemates' behaviour, Ford said: "For the most part, it's about actually laughing at these people, that's part of Big Brother."
However, he quickly corrected himself, adding: "Not laughing at them, it's laughing with these people. Everybody knows what they're going in for, so it's not like we're laughing at people."
Ford also dismissed the suggestion that Channel 5 is paying too much money to attract stars to Celebrity Big Brother.
"We're paying the money you have to, to get these celebrities," he argued. "I'm sure if you look at I'm a Celebrity... Get Me Out of Here!, they pay a lot of money too.
"The Voice [on BBC One] probably paid a lot of money for Tom Jones. The talent comes at a price and you either pay it or you don't."