Harry hasn't encountered someone who's so forceful about being concerned for him. For a long time, Kingsley and Ron and Hermione backed away and let him do his own thing. And even now that Kingsley's put him off the case, it's not like they're chasing him down and forcing him to go to St. Mungo's or anything like that.

Plus, he thinks Draco has a much better chance of fulfilling his fantasies, and that's tempting to him, whether he likes to think it is or not.