By Lindsay Kimble
April 01, 2016 at 12:00 PM EDT
Rabbani and Solimene Photography/WireImage

This article originally appeared on

Anderson Cooper didn’t publicly come out as gay until 2012, but it was decades earlier that he had the conversation with his mother Gloria Vanderbilt — a discussion he “regretted” for years.

Cooper, now 48, recounted opening up about his sexuality to Vanderbilt, 92, in a recent interview with PEOPLE and Entertainment Weekly editorial director Jess Cagle. The CNN anchor said that his grandmother Gloria Morgan Vanderbilt being “accused” of being a lesbian during the 1930s had a profound impact on Vanderbilt’s understanding of same-sex relationships.

“I had sort of vaguely heard that my grandmother was a lesbian and had been accused of it and it had been this thing,” explained Cooper. “But, when I grew up, my mom had so many gay friends in the house.”

In particular, it was a performance of Bent, a play about the persecution of gays in Nazi Germany, that opened Cooper’s eyes to his own orientation. “I remember going to see the play when I was 11 — the same year I went to Studio 54 – which was all about two gay men in a concentration camp and it was Richard Gere when he had first come to New York in it. And a friend of my mom’s, Paul Jasmine, a photographer who lives in Los Angeles now, took me to it.”

Despite learning more about himself, Cooper said he couldn’t shake the impact of his grandmother’s experience on Vanderbilt – and that feeling made him delay coming out.

“I came out to my friends in high school, but it wasn’t until I was 21, I think, that I came out to my mom,” he said. “And it was interesting because we had never discussed it after I’d come out. She accepted it and met boyfriends I had and life continued, but we never talked about the actual moment that I had come out to her because we both had different perceptions and understandings of what I had said.”

Cooper said he “immediately regretted” including the phrase “I think I am” in his admission, because despite being happy to be gay, the star felt his mother perceived him to be “unsure.”

Cooper and Vanderbilt, whose memoir, The Rainbow Comes and Goes: A Mother and Son Talk About Life, Love, and Loss will be available from Harper on April 5, and HBO documentary April 9, said after the 1934 accusation of her mother, she thought being gay was “something terrible.”

“I thought, ‘Well, maybe I have inherited this terrible thing that people are put in jail for, that’s bizarre and not normal and people are freaks if they are,’ ” Vanderbilt said. “I thought, ‘Have I inherited this?’ I worried about it for many many years until I knew I was interested in boys and then I resolved it.”

Now, the designer and author said, she realizes “there’s no difference. Love is love.”

Watch more clips from the interview below.