Jimmy Kimmel is giving credit where credit is due.

During a sit-down with PEOPLE and Entertainment Weekly editorial director Jess Cagle, the late-night host revealed that the “I’m F—ing Ben Affleck” bit on Jimmy Kimmel Live! was, in fact, Jennifer Garner’s idea.

Before the Ben Affleck video was born, it began “early on” with the actor’s best friend Matt Damon, according to Kimmel.

“We had a really bad show. To call them D-list celebrities would have been an upgrade,” Kimmel said about the birth of the Damon joke. “I don’t remember who was on, but I was depressed and I was feeling pretty bad about myself and I just at the end of the show was like, ‘I want to apologize to Matt Damon. We ran out of time.’ It was just the first A-list star I could think of.”

Kimmel said that although he wasn’t friends with Damon at the time, the actor’s publicist called to tell the comic: “Hey, that’s funny. Don’t stop doing that.”

Sarah Silverman, who was dating Kimmel at the time, then created the “I’m F—ing Matt Damon” video — much to the surprise of Kimmel.

“She was a guest on the show and she was like, ‘I have a surprise for you.’ It was one of our anniversary shows and I hadn’t seen one second of it, which is kind of funny,” Kimmel said of the video. “The probably single most successful thing we’ve ever done on the show I had absolutely nothing to do with.”

Watch full episodes of The Jess Cagle Interview now at People/Entertainment Weekly Network. It’s free, and it’s available on streaming devices, including Apple TV, Roku, etc. Just download the PEN app on your Smart TV, mobile and Web devices, or you can check it out at

When the idea came to mind for an Affleck video, the Argo star’s wife suggested the idea to Kimmel.

“Jennifer Garner came up with the idea. She kind of thought it would be funny and was talking to Harrison Ford about it at a dinner, and Harrison Ford was tickled by the idea and said, ‘Well, I’d like to be in that.’ And when Harrison Ford said he might want to be a part of it, we decided to make it more like we are the world and bring all these celebrities in,” Kimmel admitted.

“The rest is Internet history, I guess,” he said of the instant classic.

Kimmel is slated to host the 68th annual Primetime Emmy Awards this Sunday, which will air live from Los Angeles’ Microsoft Theater, starting with red carpet coverage at 7 p.m. ET on ABC.