NEW ANSWER
Found it. In the method setZIndex(), this line is the one that moves the children:
children.insert(index, this);
So the Z index is not a "true" z index which represents the depth, but the index of the children in the parent's children array. So if you try to put a child at position 3 in the array when there are only 2, it will be put at the end. Same when you try to put it at position 2.
If you want a correct ordering, you will then have to put them at positions 0 and 1, no more, no less, or else the results will be counter-intuitive.
EDIT: To apply this to your problem, then you put an actor at index 2, libGDX tries to put the actor at position 2 in the array. 2 being too big for the array, it puts it at the end. Same for 3. So that is why it is always the last one to call setZIndex that will be put last, and drawn in front. If you use 0 and 1, like I said, everything will work expectingly.
EDIT 2:
Summary: It is not a conventional Z-Index, it is the index in the ArrayList. If you try to place B or C in a cell that does not exist, like 2, 3, -1, etc., then the results will not be what you want.
To put it more simply, you have to understand how ArrayLists work. ArrayList are indexed just like arrays, starting at index 0 and incrementing by one each time a value is added. In your case, You have a group A that contains a ArrayList of children containing B and C. B is (probably) the first one in the list, placing it at index 0. C is at index 1.
If you call setZIndex(0) on C, then A will replace index 0 by C and push B at index 1. Simple as that. B will then be drawn on top of C.
But if you call setZIndex(3) on B, A cannot place B at index 3 because it is too far in the list, there is no cell 3. Instead, it places it at the end of the list, at index 1. Then you call setZIndex(2) on C and the same thing happens. A will, in the end, place C at index 1, pushing B at index 0. C will be drawn on top of B.
Source: https://github.com/libgdx/libgdx/blob/master/gdx/src/com/badlogic/gdx/scenes/scene2d/Actor.java
OLD ANSWER
Simply because Z index is depth. The more depth, the further it is. An instance with less depth will be drawn on top of one with more depth.
EDIT: Look at it like this. Your screen is index 0. Past your screen, behind it, is positive depth. In front of it, it is negative. If you put your hand in front of your screen, it has a smaller depth, but it renders in front.
B.setZIndex(3); C.setZIndex(2)
is not the same asC.setZIndex(2); B.setZIndex(3)
- even though the values are the same. – Saturn Jul 11 '14 at 12:49