By Christian Holub
February 04, 2016 at 02:09 PM EST
Comedy Central

Hannibal Buress is already several years deep into a solid comedy career. He has a supporting role on the beloved sitcom Broad City alongside several critically acclaimed stand-up specials. But for now, he’s perhaps most known bringing Bill Cosby’s history of alleged sexual assault back into the spotlight. (Buress joked about Cosby during a 2014 show, setting off renewed interested in Cosby’s past.)

Buress has discussed the joke in the months since, recently telling reporters at the Television Critics Association winter press tour that the Cosby commentary didn’t impact his life or career. But that’s not what Trevor Noah thinks. During Wednesday’s episode of The Daily Show, Buress recounted a recent comedy set in South Africa that he said went “horribly,” leading Noah to joke about the Cosby aftermath.

“This was the strange thing. Some of the reporters in South Africa didn’t think you were a comedian. Not in a bad way,” Noah told Buress after hearing of the bad show. “They thought you were a man who speaks out against rape. Because of the Bill Cosby thing. So they were like, ‘Hannibal Buress?’ And the first thing that pops up is the thing. So they were like, ‘Oh, so now he’s into comedy?’ So they thought you were just doing a show. So some people who came to the show were like, ‘Oh we heard of this guy.’ They were there to, like, watch you give a speech about quaaludes and stuff.”

Buress and Noah moved on from there to discuss his other projects, but the comic did comment on the extended life of the Cosby joke last month. “It’s weird that a joke turned into that,” Buress said. “I was doing a joke in my show — so that shouldn’t really influence public opinion or influence. It’s not what a joke should … I don’t know if it should go that far. You know what I mean?”

Watch the clip below.