Get an attenuator. I think they make a hell of a difference. Also try and listen to your neighbors at home/out habits. Everyone has a pattern to some degree. When they're out turn it up a bit more if you feel you need to. Will's point about self conscious is very true too.
If you sit in the focus of a speaker cab it sounds a lot louder.
When we were at the jam, my amp was attenuated by 40% and it wasn't even close to full wack even with the loadbox on. Really cracked volumes with your amp without an attenuator will make you feel sick after a while. :D You can wake the dead with a 100w valve amp. Whats the output of your VHT?