Hmm probably repeating what you know, I don't know better methods :(
I sqrt(x^2 - 8x) dx
= I sqrt[ (x-4)^2 - 16 ] dx
since you don't want to do it by parts we have hyperbolic substitution,
definitions cosh(t)= [e^x + e^(-x)]/2 , sinh(t)= [e^x - e^(-x)]/2
let (x-4) = 4cosh(t), dx =...