The 31-year-old singer, who finished in second place on last year's series, claimed that bosses have underestimated Phillips's importance on the panel.
BBC One controller Jay Hunt recently announced that the 66-year-old choreographer had been dropped in favour of former Strictly winner Alesha Dixon, 30.
Speaking to the Daily Star Sunday, Stevens admitted: "I think it's really bad they have got rid of Arlene. She brought lots of experience to the show and knew what she was talking about. I think it's horrible if they've replaced her because of her age."
Former GMTV host Fiona Phillips, who took part in series three, has also criticised the revamp, claiming that Dixon is not qualified to be a judge.
Meanwhile, a recent report suggested that the changes have sparked a backlash from the programme's professional dancers.