I think the point is more that you should take care to consider why it was put there because it might be something that is not immediately obvious.
You should identify how a structure interacts with it’s current environment.
OK, but what if it was put there to stop something that only happens once every 10 years? Without taking the time to learn this, you might tear it down and then after a few years you’re scrambling to solve a problem that was already solved.
To be fair, it’s both.