Why do so many directors leave major franchises after the first film?

Fifty Shades of Grey has grossed more than half a billion dollars worldwide, rocketed newcomers Dakota Johnson and Jamie Dornan into the stratosphere, and confirmed that sex really does sell on the big screen. But despite all this success, media reports suggest that the film’s director, Sam Taylor-Johnson, will likely not return to direct the sequel. Should she step away from the kinky trilogy—Universal Studios says no decision has been made and the producers are still weeks away from discussing Fifty Shades Darker—the British director would join a growing number of filmmakers who have launched major movie franchises based on best-selling books, only to hand over the reins to someone else for the follow-ups.

Catherine Hardwicke left the Twilight universe after the first film grossed $392.6 million worldwide. The Hunger Games director Gary Ross left after just one turn in the arena, despite the fact that he’d just unleashed a $2.3 billion global juggernaut. Even Divergent director Neil Burger stepped off the YA-fiction hit after his Shailene Woodley-starring film racked up some serious coin. And of course the mother of all book franchises, Harry Potter, saw Chris Columbus helm the first two before three other directors finished the magical series. The question is: Why?

Being hired to get one of these juggernauts off the ground may seem like a director’s dream, but it often comes with a specific set of nightmares. You have to bring a world into being that exists in the mind of not only the author but also millions of readers, and there are heavy constraints on your creative freedom. You have to be loyal to the source material but still create an exciting film that isn’t nine hours long (unless you’re Peter Jackson). You have to discover and cast the actors who are going to become Hermione Granger and Bella Swan and Katniss Everdeen and Christian Grey, and no matter what choice you make, at least half the fans will hate-tweet you for it. Plus you’re grappling with pressure from the studio, criticism from the press, and often contractual control by the author. It’s hard enough to make any movie well, but making these particular movies well can feel like their own special kind of hell. Even if the directors succeed, their creative struggles with the actors, the author, or the studio during the production of that first film can make them persona non grata for the next installments. Or the directors themselves may decide the payoff simply isn’t worth the cost.

“It’s not inevitable that the director shouldn’t, doesn’t, or won’t come back for the second one,” said Wyck Godfrey, producer of the Twilight franchise, which employed a total of four directors to helm the five films in the series. “There are just so many factors with a hit. You have to look at it from the director’s choice, from the studio’s choice, and perhaps in [the case of Fifty Shades of Grey] the author’s controls, which may also play a part in it.”

Taylor-Johnson and Fifty Shades author E L James have been vocal about their disagreements during production, including their argument over the film’s ending and Taylor-Johnson’s desire to move further away from the source material than James—who has reportedly said she wants to write the script for the sequel—would allow. Compromising artistic vision can surely be exhausting, but often directors leave lucrative franchises simply because of time. Most films based on a book series, especially YA novels, are shot essentially nonstop. So a director who sticks around for the sequels can find herself forced to edit the first film while preparing to shoot the second and working on the script for the third all at once.

Ross, an Academy Award-nominated screenwriter (Seabiscuit), co-wrote the initial Hunger Games film in addition to directing it, but if he wanted to continue, he was required to commit to a production schedule that was too fast for his liking. “With franchises, release dates dictate schedules,” he says. “I personally didn’t have time to write and prep the film the way I envisioned on that time schedule.” He handed over the Hunger Games reins to Francis Lawrence, who went on to direct the final three films (Mockingjay—Part 2 will debut in November). “I have no regrets,” says Ross, who’s currently in production on the Matthew McConaughey-led project The Free State of Jones. “I pursued a passion project and that’s what I’m doing now.”

Hardwicke, the production designer–turned–indie helmer behind Thirteen and Lords of Dogtown counts Twilight as her biggest box office hit. She hasn’t been able to duplicate it since. Her Twilight follow-up, Red Riding Hood (2011), was a commercial and critical disappointment for Warner Bros., and her 2013 release, Plush, was barely granted a theatrical release. Still, she says, she was content to leave after the first round of vampire love. “I felt more inspired by the first book—the way Stephenie [Meyer] captured intense feelings of yearning,” she says in an email exchange from the U.K., where she is working on the Drew Barrymore and Toni Collette starrer Miss You Already. “I didn’t feel excited about the second book.”

On the surface, it would seem logical for studios to keep the same director in place for all the films. It certainly worked for the Dark Knight films, directed by Christopher Nolan, and Jackson’s The Lord of the Rings trilogy, the last of which won the Oscar for Best Picture. But Twilight producer Godfrey sees an upside to switching helmers midstream. “What [audiences] are really in love with are the characters and their stories,” he says. With each film, “you have the consistency of the actors, you have the consistency of the locations. The only differing factor is the director … and it can be exciting to see a new director come in and give their twist on the vision.”

What’s up for debate is how much creative latitude they have in the first place. With its world already built, directing a sequel can feel more dutiful than daring. “Most of the cast, the tone, and the look are set, so it is a bit like stepping onto a TV show,” Hardwicke says. “But every film has huge challenges.”

With James still in charge on Fifty Shades and little about the vision for the next one likely to change, Taylor-Johnson may find that moving on is her best option. The film has earned $486 million worldwide and propelled her into Hollywood’s top-director tier. “It’s nice to be able to do something where tens of millions of people got to see what you’re capable of,” says a veteran agent who reps directors. Put another way, if you’ve already won, why stick around for the sequel?