AT 2, 5, and 6 are not "armor" in the normal sense anyway. They are not intended to protect against attacks.
Light graduation-robe style wizard's robes (like shown in many cartoons) should be treated as AT 1. AT 2 is cardinal/Pope style heavy ceremonial robes. I would not want to be wearing those in a fight. So who would be wearing them? Someone not planning to get in a fight. Such people do get attacked on occasion. Also, my house rules give a SCSM (RMSS/FRP equivalent of ESF rolls) bonus to appropriate spell-casters wearing the right robes.