By Jodi Guglielmi
Updated June 29, 2016 at 12:00 PM EDT
Astrid Stawiarz/Getty Images

Daniel Radcliffe didn’t have a love charm when it came time for dating.

Because Radcliffe began starring in the Harry Potter films when he was just 11 years old, trying to keep his relationships on the down-low was tough, he reveals on the latest episode of The Jess Cagle Interview.

In fact, Radcliffe remembers taking a girl out to a restaurant on his first “proper” date, hoping his fame wouldn’t ruin the night out.

“I just went out with a cap on, and you have to hope that the person you’re dating is understanding and not made to feel awkward by someone coming up or anything like that, but you just do it,” Radcliffe told PEOPLE and Entertainment Weekly editorial director Jess Cagle. “That was always the weird thing.”

Despite having a face recognized around the world, the 26-year-old Swiss Army Man actor insists that he experienced the same things as every other teen.

“I think people always imagined that I didn’t have a childhood or I missed out on some stuff, and I’m sure I had a very different childhood, but all those first dates and first kisses, those rites of passage, I still had them, and they were actually generally really private,” he explains. “I don’t feel like I have to share them with the world, and I felt like they were really my own, which is nice.”

Check out Radcliffe’s full interview, above.