Hi Liam,
I would remove the retaining clips. I have three amps where the valves hang upside down and do not have retainers on some or all of the valves and all have been fine.
As you said, the valve pins are usually tightly held by the valve bases. The only minor exception I have is a little Peavey Classic 20 where I can sometimes knock the valves a bit when lifting the mains lead out of the back. Even there, it is simply a matter of checking all is well before switch on.
To re-assure you big time, I can confirm that my (mighty) Fender 'Twin' has always had the four power output valves hanging upside down AND has been gigged a lot. They never fell out or got loose.
Regards,
Derek. :D