They were reportedly banned from wearing modern underwear during a filmed cricket match in case they showed through on screen.
Stars including Dan Stevens and Hugh Bonneville struggled to remain focused during filming while wearing the baggy garments, leading to several rude jokes.
David Robb - who plays Dr Clarkson on the period drama - told The Sun: "The bosses reckoned viewers might be able to see our Calvin Kleins, so we had to wear these big pants.
"But they don't really contain a chap - you just flop and dangle around all over the place - and the boys' jokes were flying, as you can imagine.
"I don't think the viewers will really be able to see anything but I'm not sure."
The cricket match between the house and the village, will feature as part of the drama's third season finale.
Robb added: "The cricket match was hilarious. We didn't really know what we were doing, but had great fun."
Downton Abbey's third series will air in the autumn on ITV1 in the UK. It will air in the US on PBS on January 6, 2013.