It’s fine in that most modern watch movements can withstand any brutal short term damage to the watch. Is the vibration doing zero damage? No and eventually after years and years and years it will add up and it will need to be serviced sooner than if you had not done that.

Personally I would say wear your watch and if it needs servicing sooner then service it. Remember you are wearing your watch - it’s not wearing you. If your watch can’t adapt to the life you already live it should be sold or thrown in the trash.