The speaker wood affects the tone quite a lot. Even the Tolex and the handles do! Otherwise, every cab would sound the same with the same speaker.
As a general guideline, you'd want the heaviest and strongest wood for the speakerbaffle, so that it vibrates the least. The speaker back is a prime candidate for lighter wood, to let the cab more resonate and the sides are logically in between.
This is just a very rough guideline, different cab builders have different views on this. For example, Mesa/Boogie and especially ENGL build cabs like tanks. There's nothing loose or resonating there. Marshall cabs are a bit more loose and the Fender Bassman cab is famous for very light wood.
More vibration/resonating cab means: more wooden tone, less effect pf the EQ. Good for cleans and crunch, bad for death metal.
Heavy cab with not much vibration/resonance: bigger effect of the amp's EQ, less wooden tone. Not so good for clean, great for a lot of gain and bass (aka metal).