Obviously I was as gentle as I could and was enlarging the hole by touching the sides being careful to keep it even. At the beginning I was worried about the finish if it`s gonna come off or something but no, everything was fine, and the job was done within minutes.
It might be little drastic method but thankfully worked no problem.