By Nick Romano
March 27, 2018 at 08:36 AM EDT

Stephen Colbert was fully prepared for a Sean Penn interview on The Late Show Monday night, but the rest of us sure weren’t.

“You’ve inherited a little bit of the Ambien I had to take to get to sleep after a red-eye last night,” the Oscar winner began the chat, prompting the question, “How do you tell Sean-Penn-on-Ambien from Sean-Penn-NOT-on-Ambien?” Turns out, Sean-Penn-on-Ambien is more inclined to light a cigarette and start smoking live on Colbert’s stage than Sean-Penn-NOT-on-Ambien.

The actor — er, should we now say actor-on-his-way-out? — discussed why he no longer finds joy in acting, and it has to do with that collaboration element of it all.

“The greatest thing that an actor can bring to the party is to play well with others — it’s the collaboration — and I increasingly don’t play well with others,” he explained with a smile between puffs. “And so it becomes less enjoyable, because I love that process when I love it, but I’m not loving it anymore and that’s really why I finally came around to writing a novel because I didn’t have collaborators. I was never disappointed with me.”

The pair also talked about said novel, Bob Honey Who Just Do Stuff: A Novel, why Netflix has also helped turn Penn off from acting, and a comment about Netflix settlements that they both decided to quickly move away from.

“I think part of it is the girl I fell in love with was going into a movie theater in the dark with strangers and seeing something that might last forever,” he said. “Now there’s so much content, I can’t keep track of it and nothing seems special.”

At one point, Colbert did reprimand Penn for smoking in the studio, and while his guest snuffed out the cigarette, he responded, “This is job security for oncologists.”

Watch above.