The Football Factory actor claimed that he is not surprised by criticism of Horne's recent TV and film work with James Corden.
The pair's BBC Three sketch show Horne and Corden and movie debut in Lesbian Vampire Killers have both suffered frosty responses in the press.
Reflecting on the duo's recent troubles, Dyer told the Daily Star Sunday: "Mathew Horne is one of the worst actors I've ever seen in my life. Corden's obviously OK but Mathew was terrible in The Catherine Tate Show.
"He's just got nothing about him, the geezer. No substance and no depth. It annoys me a bit that no one's clocked that.
"I'm not trying to do anyone out of a pound note, I just think they're s**t. Mat is absolute rubbish."
Horne recently collapsed during a performance of Entertaining Mr Sloane in the West End, but he returned to the production days later.
Meanwhile, Dyer has announced that he is in talks for a role in BBC soap EastEnders.