Minutes after Kanye West stormed the stage as she accepted her Video Music Award, a teary-eyed Taylor Swift pulled herself together to perform her award-winning song, "You Belong With Me," atop a cab parked in front of Radio City Music Hall.

But West only continued the drama by angering the entire Swift family. After Swift's mother, Andrea, asked to speak to the rapper and, a source tells PEOPLE, "tore into him," the rapper continued to maintain that Beyoncé's "Single Ladies" should have won Best Female Video instead and made no formal apology.

Later on his Web site, West confirmed that Swift's mother "said the same thing my mother would have said," but didn't offer a full apology until Monday afternoon, when he said, "That was Taylor's moment and I had no right in any way to take it from her. I am truly sorry."

Among those who did offer support backstage was Beyoncé's dad, Matthew, who comforted Taylor and her family, and arranged the moment in which Beyoncé gave Swift a second chance at the spotlight.

A spokeswoman for Taylor says she's been overwhelmed by her fellow musicians' "kindness, support and generosity."