I can't claim to understand the workings of the rotary switch, but presumably all the pickup wires have to connect directly to it for the coil-splits etc to work. That being the case, I can't really see a way of running separate volume controls.
If it was just a normal 3-way switch and two volume pots, then you'd have:
White to "hot" lug of respective volume pot
Black to ground
Red taped off (as not being used to coil-split)
Output lug of pot to one side of the switch
Centre lug of switch to jack
You could play around by connecting the white wires to separate pots, then take the outputs from the pots to the points where the white wires currently connect on the switch. I'm 99% sure it won't work though, because I suspect with the 5-way switch some positions have white as hot, black as ground.... but in other positions it will be reversed to black hot, white ground. So you'd probably have silence in certain positions on the 5-way! :?