The 'Candy' singer is said to be ITV producers' number one choice for the 2013 lineup, alongside his Take That bandmate Gary Barlow, according to The Sun.
A source said: "This is the dream team. We have been looking at the show and thinking the unthinkable.
"Robbie is exactly the right person for the show - but we don't want to lose Gary to get him. We want them both. Robbie's people are saying he is too busy to do the show.
"But we worked around Gary when he first joined the panel and he was touring with Take That and it was fine. We know Robbie loves the show and he and Gary have a great friendship and rivalry - there will be fireworks for sure."
Williams's appointment may not mean that Louis Walsh is forced out, as he is said to be close to signing a £750,000 deal to return once again, and producers are reportedly happy to have three men on the panel.
The source added: "Louis's future is as up in the air as it has been in any other year - but if Robbie and Gary do sign it will not necessarily mean curtains for him.
"The producers are relaxed about the gender make-up of the panel. It is all about having big personalities."
The fourth judging spot is expected to be handed to Nicole Scherzinger, with Tulisa Contostavlos recently rumoured to be leaving the show.
Sharon Osbourne has also been linked with a return to the show in recent months.
The X Factor Tour 2013 - Photo gallery: