Hey welcome to the forum. I also have an 03 custom and my left mirror has moved a couple of times from vibrations. The left side vibrates more and I have read that it is the left mirror that had a problem with breaking off on the bikes prior to the 03 models so HD changed the mirrors for the 03 models.
I don't know why the left side vibrates more but I can only guess. I believe it is because you are holding on to the right side with a firmer grip because you are always using the throttle and you relax your left hand after getting in fifth gear. That's my guess.