^ +1
If you open a dispute/resolution and say you've "mutually agreed" to cancel the sale, they'll contact the buyer, get him to confirm that's OK, and they'll refund the Final Value Fee. The whole thing should be sorted out pretty quickly.
Might be worth contacting him first to say that's what you're going to do, and since he's messed you about you'd appreciate his prompt co-operation.