Yes. And with no licence to lose, you can get fined into oblivion.
You can get stopped for speeding on one, too, which was news to me (who wasn't actually breaking the limit, just getting close enough that a cop felt the need to warn me).
There is no law to stop you exceeding a speed limit on a bike, per se, as there is no requirement for you to have something that measures speed on it. There is, however, a law governing riding without due care and attention, and it's that which you'd be having to contest if you were stopped for 'speeding'.
I think the technicality about speeding is that it's not motorised, and therefore not subject to speed limits, only what can be demonstrated to be 'unsafe'.
That's obviously a pretty grey area: if you're going 40 downhill in a 30, I think many policemen would stop you arguing that it's too fast and you're likely to badly injure or kill yourself in the event of an accident. I don't know what the courts would say to that, though.
And yes, being drunk in charge of a bicycle is also an offense.
Roo