For a surprise Subdual attack, you can modify the critical roll using your Ambush skill ranks. That should increase the chance of incapacitation, and decrease the chance of accidental death, since Ambush can modify the result in either direction as needed.
Without the element of surprise, I agree with Sable Wyvern that a knock-out without doing any real damage is going to be very difficult.
RMSS subdual is fatal on B 96-99 and E 96-99 (so 0.8% across all severities), two week coma on C 100 and one month on D 96-99. The RMU table is more lethal, with death on B 98-99, C 96-97, D 91-95, D 100, E 86-90, and E 98-99 (3.2% across all severities). Too lethal?