The music mogul's ITV1 variety show was branded cruel earlier this year after young children like singer Hollie Steel and breakdancer Aidan Davis broke down in tears live on air.
A poll on The X Factor's website has now found that 45% of viewers believe that Cowell is becoming too soft on this year's series, STV reports. Meanwhile, a further 35% reportedly claimed that he is losing his reputation as the harshest judge.
Responding to the findings, Cowell's representative told the Daily Star: "Has he turned into a softie? No. He will always tell it like it is.
"However, he may have taken on board things that happened earlier this year when some of the younger contestants in Britain's Got Talent got upset.
"I am sure if someone comes along who is truly awful he will be back to his old self. Watch this space."
The X Factor continues tomorrow at 7.35pm on ITV1.