'I would say he exerted the same level of charm he does today.'

By Nick Romano
March 30, 2018 at 08:26 AM EDT

Sean Penn doesn’t have fond memories, per se, of working with Steve Bannon in the 1990s. President Donald Trump’s now-ousted White House advisor had previously produced 1991’s The Indian Runner, Penn’s directorial debut. “I would say he exerted the same level of charm he does today,” Penn, completely deadpan, told Conan O’Brien while making the late-night TV rounds to promote his new book, Bob Honey Who Just Do Stuff.

Starring Viggo Mortensen and David Morse, The Indian Runner told of two brothers: one a small-town deputy sheriff and the other a criminal. The film was directed and written by Penn, who remembers Bannon “working for big bank” at the time and flying in from New York to be on the Iowa set.

“I noticed that, without any need for them, we had two cheerleader-looking new makeup assistants on the call sheet that, as the director of that movie, I had not called for and yet it was on my very minimal budget to make this film,” he said, noting they too had “coincidentally” flown in from the Big Apple.

“That was my first sense of him,” Penn continued, “that he was other than the C word, charming, he was a crook, I think is what I thought.”

Though, he didn’t know until much later that this was the same Bannon who would come to work for Trump. “I didn’t even know it was the same Steve Bannon until I looked at him closely and imagined the Steve Bannon I knew but where the toxins of the soul so deteriorate and bloat and weld into this… I don’t think you can age like that without hating people,” he said.

Penn has since shied away from the movie business, as he told Stephen Colbert during an unforgettable interview on The Late Show earlier this week. Still “a little bit” on the Ambien he took the night before and enjoying a lit cigarette on the show’s main stage, Penn said, “The greatest thing that an actor can bring to the party is to play well with others — it’s the collaboration — and I increasingly don’t play well with others.”

Watch his latest interview in the clip above.