Yes some damage types don't effect other types, like normal arrows cant hurt a building, fire arrows can, start it on fire.
Walls cant even be attacked by almost everything except some siege equipment and or other units just like in real life, good luck slashing through that 5 foot thick stone wall with a sword like you see in other games. Archers still try to volley their arrows over walls though so they are still effective.
My advice that is historically true is not to destroy the walls but rather capture them with siege towers and ladders and use the walls to your advantage with archers on top of them shooting at everything below behind the wall. you get enough good bottlenecks at the staircases and he wont be able to do a thing about you taking his own walls from him, this is especially useful when fighting Orcs seen as they only have 1 layer and pretty much all of their buildings and army are most times inside( unless the players your fighting against likes building them outside there Orcs, they build anywhere.)