The actor, who plays Raylan Givens on the show, also praised the work of his co-stars, including Walton Goggins and Margo Martindale.
Olyphant told TV Line: "I can take [something] in a different direction, and I know they're going to pick it up and... throw it back at me in a way that I just haven't seen before.
"As an actor, you're doing everything to, for, because of, and off of the other person. That is super easy to do when that other person is Walt Goggins. Or that other person is Margo Martindale. Or Natalie Zea."
Olyphant added that he was extremely impressed with child actress Kaitlyn Dever, who plays Loretta McCready.
"She was fantastic," he said. "She was just so good. She wasn't just good in terms of being present and truthful and capable, but I was really able to throw curveballs at her - change stuff on the spot - and she wasn't fazed by it."
Justified recently received four nominations for the Television Critics' Association's annual awards, including one for Olyphant and Martindale in the 'Individual Achievement In Drama' category. In addition, Martindale won at the inaugural Critics' Choice Television Awards for 'Best Supporting Actress in a Drama Series'.