During his Late Show taping, Letterman wished Leno a happy birthday, but the chatshow host then took a swipe at Leno, who took back The Tonight Show hosting duties after Conan O'Brien left the show due to a scheduling dispute last year.
"Happy birthday to Jay Leno - 61 years old today," Letterman said.
The comedian continued: "Jay, of course, being one of the few tyrants still alive. All of his friends got together and had a cake. Jay cut the cake with the same knife he stuck in Conan's back."
O'Brien had served only six months at the helm of the NBC late night programme, when he was replaced by Leno.
Since leaving NBC, O'Brien has started hosting a new late night show on TBS called Conan.
Letterman was awarded the highest honour at this year's first annual US Comedy Awards in March in New York City. Bill Murray presented him with the 'Johnny Carson Award for Comedic Excellence'.
Watch David Letterman joke about Jay Leno below: