There’s a reason Ralph Fiennes was chosen to portray He-Who-Must-Not-Be-Named in the Harry Potter movies — he’s scary, even to his costars!

In the latest episode of The Jess Cagle Interview, Daniel Radcliffe reveals that the villains of the Harry Potter movies were well-cast.

While Alan Rickman’s portrayal of Professor Snape took some getting used to, Radcliffe calls Fiennes, who portrayed Lord Voldemort, “genuinely intimidating.”

“I remember when I was 15 doing scenes with Ralph for the first time. He was even scarier than Alan was at first,” the Swiss Army Man actor tells PEOPLE and Entertainment Weekly editorial director Jess Cagle. “Alan was super intimidating to start off with as well, but then you get into it, but Ralph genuinely scared me for a few years.”

Radcliffe, 26, was cast to play the boy wizard in Harry Potter and the Sorcerer’s Stone when he was just 11 years old, and continued his run for seven more films over the course of a decade — culminating in, yes, an epic showdown with Fiennes’ Voldermort.

Watch the full interview with Radcliffe above.