The TV judge said that the executive producer's return to the competition will bring much higher ratings for the show.
The X Factor's ratings have suffered a dip in the last two years, following Cowell's exit in order to front the USA edition on Fox.
The show was beaten by BBC One's Strictly Come Dancing on Saturday (November 3), and was watched by an average of 7.6 million viewers, three million lower than the equivalent episode of 2011. However, it peaked at 10.6 million on Sunday's results show.
> X Factor vs Strictly: Which show is winning Saturday nights?
"If Simon did come back, he'd have the number one show," Walsh told The Sun.
"That's the thing about Simon — he always wants to be number one.
"We need him in the building, he's always fun, he keeps everybody on their toes. I wish he'd come back."
Walsh added that Cowell has asked staff members to work harder on the show, saying: "Simon doesn't want anybody to be complacent."
Watch Digital Spy chat to Kye below: