The EastEnders actor has a son with former fiancée Emma Sophocleous, but the couple broke up shortly before his relationship with co-star Tointon.
Tointon is currently dating her Strictly Come Dancing partner Artem Chigvintsev.
"I like the company of ladies and I get on well with them,' Swash told Now. "But I'm definitely not a ladies' man.
"I've been single since me and Kara split last year. But the biggest love of my life is my son. Next time it definitely has to be with 'the one'.
"I have to be careful otherwise I'll end up on Jeremy Kyle with kids all over the place."
Swash has filmed two special episodes of EastEnders to coincide with his on-screen brother Darren Miller's wedding.
> Joe Swash 'hates seeing Kara with Artem'