Credit: D Dipasupil/FilmMagic

Melissa Rivers watched the dynamic shift on E!’s Fashion Police after her mother, the show’s ringleader, died in September 2014. Kathy Griffin stepped in for Joan Rivers, while Giuliana Rancic landed in hot water after making racist comments on the air. “It was a very, very difficult time,” Rivers, Fashion Police‘s co-creator and executive producer, told Hoda Kotb at the 92nd Street Y in New York City on Wednesday.

Griffin and co-host Kelly Osbourne left Fashion Police in the wake of the controversy; the former last less than three months on the show before departing. “My biggest complaint was the feeling that she kind of shit all over my mother’s legacy in her statement on leaving,” Rivers said. “I know everything was so heightened and everybody was so crazy, but that was my takeaway. I know that was not an intentional reading of it, but that’s how I felt. By calling the comedy and the style of it old-fashioned—it was like, I understand what you were doing, you’re trying to save yourself, but don’t crap all over my mother to do it.”

But Rivers said the entire cast was “out of control” when Fashion Police returned in January. “We went back too fast,” Rivers admits. “Just like a family, when the matriarch dies, the sisters started fighting. Someone tried to marry in. Not a great match. Live and learn.”

Trying to rebuild the show while maintaining her mother’s legacy was frustrating for Rivers. I felt like Fashion Police was this little jewel, and it was the last piece I had of my mother and I working together,” Rivers said. “They took the last thing I had and smashed it.” She believes none of the chaos would’ve ensued had her mother been alive.

Still, Rivers thinks her mom may feel a slight sense of satisfaction over the whole mess: “There might be a little bit of a twinkle in her eye saying, ‘Ya’ll couldn’t do it without me, could you?'”