By Teresa Jue
Updated December 10, 2014 at 11:14 PM EST

Columbus Short, who was let go from ABC’s Scandal after season three after allegations of spousal abuse arose, has admitted he was struggling with drug abuse at the time.

In an interview with Access Hollywood Live, Short admitted to substance abuse, calling this past year a “series of unfortunate events.”

“I’ll be candid. I was struggling with drugs. I had a lot on my plate, and you know, I was using unhealthy ways to kind of self-medicate and deal with a lot of heavy-duty stuff in my life,” Short tells hosts Kit Hoover and Billy Bush. Short also admitted that his vices were cocaine and drinking, in order to maintain the 16-hour work schedule, also noting that Shonda Rhimes and the cast of Scandal were aware of his substance abuse.

“They protected me and they held me down and that was one of the real reasons… they just wanted me to get my stuff together. Sometimes the bottom has to be dropped out for you to really get it,” Short added, also saying he moved to Atlanta to reboot and work on his album.

While his character, Harrison, was killed off at the end of season three, he says he hopes to return to Scandal, although he notes, “I just know I let them down, I disappointed them. They love me, I love them, let’s leave it at that.”