By Steve Helling
Updated August 14, 2016 at 12:00 PM EDT
Credit: Matt Hazlett/Getty Images

Ryan Lochte was held up at gunpoint as he headed to a party in Rio on early Sunday morning, according to his mother.

The 32-year-old swimmer was unharmed in the robbery.

“I think they’re all shaken up. There were a few of them,” Ileana Lochte told USA Today. “No, they were just, they just took their wallets and basically that was it.”

But Lochte’s coach, David Marsh, told USA Today via text message that he “was not held up” and was trying to figure out exactly what happened.

Two sources with the US Team confirmed to PEOPLE that Lochte was robbed, but had no additional details.

“Everything’s up in the air now,” says one source. “But we know that something happened last night and that it really shook Ryan up.” The source adds that Lochte is now safely with his family and friends.

Lochte himself confirmed to NBC that he and “three other swimmers” were held up at gunpoint.

“We got pulled over, in the taxi, and these guys came out with a badge, a police badge, no lights, no nothing just a police badge and they pulled us over,” Lochte said. “They pulled out their guns, they told the other swimmers to get down on the ground — they got down on the ground. I refused, I was like we didn’t do anything wrong, so — I’m not getting down on the ground. And then the guy pulled out his gun, he cocked it, put it to my forehead and he said, ‘Get down,’ and I put my hands up, I was like ‘Whatever.’ He took our money, he took my wallet — he left my cell phone, he left my credentials.”

On Sunday afternoon, USOC confirmed the incident in a statement:

“According to four members of the U.S. Olympic Swimming Team (Gunnar Bentz, Jack Conger, Jimmy Feigen and Ryan Lochte), they left France House early Sunday morning in a taxi headed for the Olympic Village. Their taxi was stopped by individuals posting as armed police officers who demanded the athletes’ money and other personal belongings. All four athletes are safe and cooperating with authorities.”

Lochte pal Thiago Pereira had invited him out to Club France on Saturday night to celebrate a mutual friend’s birthday. Pereira’s rep, Flávio Perez, tells the Washington Post that Pereira and his wife left the club earlier.

“Thiago, on finding out what happened, called Ryan,” said Perez. “Ryan is well, and Thiago said that the robbery, according to Ryan, was in the taxi. The taxi was robbed.”

On Saturday night, Lochte posted several shots of himself partying out in Rio with what appeared to be another Team U.S.A. swimmer. He shared a video of graffiti as he sped by inside a car and later showed off a group of men posing with their sneakers, which included two sets of Yeezys.

Later, Lochte and an apparent Team U.S.A. swimmer can be seen dancing inside a club.

According to the source with the U.S. Team, Lochte has been answering calls and texts from his family and friends, telling them what happened. “He’ll be okay,” says the source. “The important thing is that he wasn’t hurt.”