Would this bother you or not?

I would be annoyed - he's told you this and then left you with the thought of him working with the person he chose! Of course it's going to play on your mind. It's a very immature game to be playing - what next, spin the bottle?!