The show's hosts did not learn that Brook had been recruited for the talent programme's judging panel until the first day of auditions in Manchester.
Brook was axed from the ITV1 series just six days later after producers decided that the format did not work with four judges.
"We didn't ask for her to be sacked. But it's completely true we were annoyed that she'd been brought in because we hadn't been told about it," Ant told the Mail On Sunday. "We arrived for work, got to the hotel in Manchester and found out there was to be a fourth judge for the show.
"Simon hadn't told us. He hadn't told anyone. Kelly was just there. She's a great girl and she looked fabulous, but if anyone had asked us straight off we'd have said it's almost irrelevant. It could have been anybody and it wouldn't have worked. We wish somebody had asked us. But no one did and no one consulted us."
Dec added: "We were angry about Kelly. But it was because no one asked our opinion. What was upsetting was that it happened in the first place. It was wrong from the start."
Britain's Got Talent returns to ITV1 in May.