don't worry bout him, he just wants to see more of your gorgeous face.
anyway, i'd say end it, as you said it seems to hard.
Isn't there a saying; "If you love somebody, let them go. If they return, they were always yours. If they don't, they never were."