So in the moral system you’re proposing, it would be an offense to use those buildings without their owner’s permission. The fact that people denied access to those buildings will die as a result, though, is just a part of nature
So, why would you expect a perspective which values property more than people to stir anyone’s moral feeling? If you don’t expect that, then why are you bothering to frame it in terms of right and wrong?
Do you understand the difference between an unused housing unit and an in-use one?