I think this is a great idea, but I would like to add a suggestion. The quality of the structure (ie buliding/wall/tower hitpoints) could be based on the quality of the builder. Maybe a wall built by humans has 1000 hp, but the dwarf built walls has 1500. But inversly, the dwarf wall take 50% longer to build. You could even have the dwarves "upgrade/reinforce" the human walls after the fact, so you don't force player to tear down walls. This would just require time and resources.
You could have some of these special builders as quest rewards, which increases the incentive of doing the story line.