The former Oasis guitarist admitted that he felt bad denying the X Factor winner of the chart milestone as it was his debut album.
The Sun quotes Gallagher as saying: "Matt Cardle seems like a nice bloke. I'd have preferred it if he got it - share the wealth and all that, and he's a young lad just starting out.
"I was glad I made the record and when I finally heard the finished thing it was good but anything else is out of your hands.
"When I heard Matt Cardle was out the same day there was a collective sigh because it's The X Factor and these crazy schoolgirls tend to buy a lot of records for some reason."
He added: "But [topping the chart] didn't make me feel any greater or more special than I did the minute before someone told me it was number one.
"I like the record and if people like it, that's it, other than that whatever happens, happens."
Noel Gallagher's High Flying Birds topped the UK album chart at the weekend, managing to shift 120,000 copies in its first week.
Cardle's Letters came in second with 70,000 copies sold, making it the second highest-selling debut from a male X Factor winner based on first-week sales.
> Matt Cardle confirms new single 'Starlight', thanks fans for support
Watch the music video for Noel Gallagher's new single 'AKA... What A Life!' below: