The input is the GT8 is designed to accept guitar signals, ie -10 dBu and 1 M input impedance. It is NOT designed to be used in an effects loop. The signal from an effects loop (which is assumed to be line line level, although in practice is often much more than this) will overdrive the input stage of a GT8. This will sound terrible, and might explain why the GT8 doesn't work very well in your effects loop. Incidentally, on the Cornford forum, many users were were finding problems with using a Boss digital delay pedals in series effects "loops".
You should use the GT8 before your amp.