The music mogul will still manage competitors off-screen, but Kate Thornton, the show's host, is to be replaced all together.
In another change, a new category for performers will be introduced in this year's series and the age limit for entrants will be lowered.
Announcing the move, Walsh said he would "focus my energy on my day job as an artist manager". He added: "I am also looking forward to continuing my work with Simon and The X Factor team managing the stars from this year's series."
Among his plans is a relaunch of Boyzone, while Thornton is to present other unnamed shows on ITV.
Paul Jackson, director of entertainment at the broadcaster, said: "We are always looking at ways of keeping the format as fresh and exiting as ever. Having an extra judge and additional category will give the show a whole new dimension."