Immediately after his late-night fight with Rihanna over Grammy weekend, Chris Brown knew what he had to do: Tell his mother.

"I went to my mom and I was upset," says Brown, who was raised by his mother, Joyce Hawkins, in the small town of Tappahannock, Va., where they both still live a few doors down from each other. "I told my mom what happened and just broke down. I was disappointed in myself and I felt like I had let my mom down."

For Hawkins, herself a victim of domestic abuse, Brown's confessional still haunts her. "It was the most painful moment in my life to hear him tell me that," she tells PEOPLE. "I just sighed and was like, 'What in the world happened?' And you know he started talking, but I could see the pain in his face and how hurt he was while he was trying to tell me. It wasn't easy for him to do."

Hawkins, who avoided watching TV in the wake of the incident, had strong words for her son. "I explained to him that in no way [was] what he did right or acceptable," she says of Brown's assault of Rihanna in his rented Lamborghini on Feb. 8. "And I just explained to him that if something of this magnitude ever happened again, what he should do – walk away from any situation and never involve yourself like that ever again."

Even so, Brown's behavior was a shock. "I've never seen any violence in Chris, ever. There was never a history of violence." Hawkins says of her son, who has no prior criminal record, "He’s always been my little angel."

As Brown begins serving his sentence, Hawkins says the ordeal over the last seven months has taken a toll. "He's dealt with it, but it hasn't been easy," she says, before adding that her heart also goes out to Rihanna. "I love her. She's like a daughter. She became family to us. She's spent Christmas with us and for the last two years."