He's not the only one who hasn't thought this invitation thing all the way through. Because Draco lives at the Manor with his Mum, and he doesn't think of it as the place where Hermione was tortured, Harry was captured, and Dobby died. He thinks of it far more broadly and.... yeah. Harry doesn't want to casually visit right now. They need another place to hang out.