My car has got the same problem than yours, I asked to fix it under warranty as I was servicing the car. But they seem to have fix it just by moving the armrest to the passenger side. It works but without any washer or spacer, the gear can easily go out from the small white gear - which it did quickly !
I may try your trick, I didn't know the back of the console could easily be pulled out. I was more thinking of a thick washer / rubber gasket to prevent the armrest to slide towards the driver seat ?