Harry assumed that Draco would have been a lot more panicked if his captor could return at any moment. Plus, there wasn't any obvious clue that anyone would (the floor was covered with dust, mentioned in the first chapter, which showed that no one had been there in a while).

More is coming!