Space Engineers had a very good inpresion on me with this one.
Every thing you bulid whould require you to put materials in it and use according tools or not.
Let’s say you are building a simple log wall. You would need logs. You would start building the wall by making or placing like in SE a wall placeholder that would require only minimal amount of materials.
You have the materials? Put them into the placeholder and start hitting that thing with your hammer.
To remove all that wooden wall business you would need to make some saw acton.
Brick walls:
??? to build
Sledgehammer to remove
Metal walls:
Blowtorch to build
Grinder to remove
Repairig it by adding missing resources and start hitting it with your tools.
Only the owner can dismante the constracton.