According to The Sun, the Syco boss believes that Gary Barlow, Kelly Rowland and Tulisa Contostavlos "should" return for a second year alongside veteran Louis Walsh.
When asked about the panel, Cowell is quoted as saying: "My gut feeling is all four should come back."
Irish mogul Walsh last week said that Rowland is the only panellist whose position he was "not sure about".
The 59-year-old seemed to suggest Nicole Scherzinger, who recently announced her exit from The X Factor USA, as a possible replacement.
Meanwhile, Barlow and Contostavlos have consistently outlined their desire to stay on the ITV talent show.
Watch Barlow and Contostavlos talking about The X Factor at the National Television Awards below: