Brandi Glanville and Eddie Cibrian Photo by: John Shearer / WireImage

Long before Eddie Cibrian and LeAnn Rimes took their relationship public last week, Cibrian's wife Brandi Glanville had an inkling that her husband was attracted to the singer.

Recalling when she and Cibrian went on double dates with Rimes and husband Dean Sheremet in Canada during the filming of Lifetime's Northern Lights, Glanville says she noticed a little flirting between the costars.

"We went to play pool, went for sushi, dancing," Glanville tells PEOPLE, adding that although she's used to woman fawning over her husband, "it was different this time. It was a little more reciprocated."

At the same time, Glanville noticed how affectionate Rimes, 27, was with Sheremet, 28. "They were very touchy-feely," she says. "They seemed happy. [But] I felt like it was almost like 'I love my husband. I'm not after your husband.'"

After reports of an affair surfaced in March, Glanville, 36, says she and Cibrian, 36, sought counseling to try and salvage their marriage, and she publicly stood by in him in April.

"We were in therapy, but he never stopped talking to [LeAnn] so how could that have worked?" asks Glanville, who says she plans to file for divorce.

Cibrian and Rimes took their relationship public last week - one month after Rimes and Sheremet announced their separation - at a public golf date Thursday and a Kings of Leon concert Saturday.

Now, Glanville says she's been "trying to get a hold of" Sheremet. "I feel bad for Dean. It's a horrible situation for him. I don't think he's ready to have a conversation about not being with her."