So, apparently this is all not true.
I have all the twos and threes (two of cups, two of batons, etc). But that does not stop them from attacking me.
first of all, if I attack a Cardian with the corresponding card in my inventory it fights back, so the card doesn't prevent them from fighting.
But also, the the Cardians will walk right up an attack me even if I'm carrying their card.
I don't get it.