According to the latest data from the Radio Joint Audience Research (RAJAR) board, 5 Live had a reach of 6.65 million listeners in the three months to March 31, down 440,000 on the record 7.09m in the final quarter of 2010.
Combined with sister station 5 Live Sports Extra, 5 Live had a total reach of 6.75m listeners, down on the previous quarter's 7.21m, and a share of 4.7%, down from 5.5%.
In contrast, the UTV Media-owned TalkSport achieved the highest ever audience in its history, with a weekly reach of 3.25m, up 37% on the same period last year. The station now has an audience share of 2.1%.
TalkSport said that its new show featuring Richard Keys and Andy Gray, who left Sky Sports earlier in the year after a sexism storm, has delivered a 35% increase in listeners to the mid-morning slot since February.
Moz Dee, TalkSport's programme director, said that this is a "promising start" to 2011 for the station, and a sign of "consistent and significant growth" in new listeners.
He added: "I'm very proud of everyone in the programming team, as they have worked so hard to build credibility in sports journalism and made TalkSport must-listen radio for sports fans."
TalkSport chief executive Scott Taunton said that the station's investment in programming has "paid clear dividends", and expressed his confidence for the future with exclusive coverage of the Rugby World Cup coming in September.
TalkSport is currently waging a concerted campaign against Radio 5 Live, arguing that the BBC station is failing in its remit to provide viewers with serious news output.
The BBC Trust is running a service review of 5 Live and 5 Live Sports Extra, and is seeking views from the public on the "quality, distinctiveness and value for money" of the stations.