Yes. In my opinion the mismatch doesn't help, and you will definitely lose power.
Now we are lucky are plenty of decent 4 ohm 8" speakers for guitar amps out there, however in the past is suspect that if you blew your Champ speaker you used whatever you could find, which would probably be 8 ohm, cheap and nasty.