I would generally say they are not aware of it during the spell if they failed the RR, but might realize later if their actions were uncharacteristic or atypical. In the case of Calming, if they did know during the spell, that might make them mad, which sort of defeats the purpose of the spell. If their behavior was natural, and things went ok after that, it's really human nature to believe it was their own behavior and not magic. We come up with rational but false explanations of our own behavior all the time, we actually aren't that good act figuring out our own emotional responses. Conversely, if they know a mentalist was present, and something went wrong, it would also be natural to blame the mentalist and not themself, whether that's correct or not.
If they make the RR, either they know if they make the RR by a certain amount (e.g. by at least 50), or you could alternatively give them a perception roll. The former is how the rules tend to handle it. The latter gives an advantage for someone who is highly alert (or even one who has a relevant situational awareness skill), which might be appropriate, although it's another roll.