In an interview with The Independent, the 70-year-old former Beatles singer revealed that he thought there was "something suspect" about the late DJ, who has been the subject of a string of child abuse accusations in recent months.
McCartney told the newspaper about The Beatles' friendship with Savile, who would often travel with the band early on in their career.
McCartney said: "It's very difficult to talk about it. The thing is we knew Jimmy and we worked with him, he was a DJ, an MC on some of the shows.
"We were working in Yorkshire and we were still living in Liverpool. And we were coming back from a gig and he came in our van over the Pennines, we gave him a lift."
The singer-songwriter described Savile as "the older hustler guy", adding that he and bandmates John Lennon, George Harrison, and Ringo Starr were "amused" by the anecdotes that Savile regaled to them as "he was a great entertainer".
McCartney also revealed a story about not being let into Savile's home after giving him a lift.
He said: "We dropped him off at his place outside his house and we said, 'Can we come in for a coffee', and he said, 'Oh no, not tonight lads'. When he'd gone we thought, 'Why doesn't he let us in? What is it?' because most people would have let us in that we gave a lift to. So we always thought there was something a little bit suspect."
McCartney also spoke about "the scene" in the 1960s and '70s, describing it as "not so PC".
He said: "[In] the post war boom, girls and guys, it was a much more open scene, and you know free love, and the pill had just come in, so it was a completely different scene.
"The other aspect of course is that we, though not quite Jimmy, were of the age of the girls, we were all young. So if you're now talking about a 17, 18-year-old boy with a 15-year-old girl, we all knew that was illegal. We knew it and it was like, 'No!'
"But the closer we were in age of course the less it seemed to matter. We knew with under-16s it was illegal, so we didn't do it.
"We tried to make sure, we couldn't always be sure. But there was a definite no-no involved in underage kids. Hey listen, we didn't have to worry. There were plenty of over 16-year-olds."
The full interview with McCartney will be published in The Independent on Saturday (November 24).
Photo gallery: Jimmy Savile:
Copyright: flickr/BBC Pictures