Milne's character has become a central love interest in the show, attracting attention and a kiss from lady's maid Miss Reid (Lucille Sharp).
Sophie McShera's Daisy Robinson also has a crush on the new arrival, but has yet to reveal her feelings to the latest staff member downstairs.
"Alfred's got a tormented time ahead. It's going to be good fun," Milne told Digital Spy.
"Edward Speleers's new character Jimmy Kent comes in next week and he's going to add a whole new dynamic downstairs. So there's lots of fun in store."
Speaking about arriving on set for the first time, Milne said: "I think it says a lot about the rest of the cast, they've built up a real family over the last two years, but they've really accepted me and the other new cast members into their working group.
"It says a lot about them about how accepting they have been and I'm really grateful for that.
"There's no jumped-up people whatsoever. It's great to learn from that because you see that's how the big boys like Hugh Bonneville and Dame Maggie Smith do it and I've learned from that."
Commenting on the show's rise to become the UK's most-watched TV drama, he added: "Julian Fellowes has just got a knack for creating characters that people are interested in. That and some great performances - it's a perfect match.
"It's great to be involved in something so wonderful. It's wicked to have a small part in something like this."
> 'Downton Abbey' prequel planned by Julian Fellowes
Downton Abbey continues on Sunday nights in the UK at 9pm on ITV1.
Outside of Downton Abbey, Milne is starring in Tennessee Williams's The Hotel Plays. Running from October 4 to 27 at the Grange Holborn Hotel in London, the production features three different plays in three different hotel suites.
"There's only 30 people in the audience and you'll be able to see the whites of the eyes of the actors and reach out and touch us - but hopefully people won't", joked Milne.
"It's like being on the front row every night. There are no bad seats."