If you enemy does not have proper siege equipment he can't do anything to you inside your walls besides sieging it (preventing you from collecting resources) and shooting some arrows over the walls (you can send arrows right back though...)
Ranged siege equipment which are Human Trebuchet, Orc Catapult, and Elven Bolt Thrower are able to damage walls slowly, each of these siege engines drain stone resource to fire, this again means that if you forgot to bring stone in your siege, trying to use siege weapons are useless.
Gate Siege Equipment include the Human and Orc Battering Ram and the Elven Treent which kicks the door slowly. Gate Siege Equipment are the only units that can damage even gates, so you won't even have swordsman slashing gates to death.
Third type siege equipment is the Wall Scaling Siege equipment, these include siege ladders and siege towers, they move to the walls and attach gaining access to the wall, your units can move up and gain control of the wall.
But there are many counters to the enemy Scaling your walls, but then again if done right it is by far the fastest.
Just a note about gates, each gate has gate gears behind it, if attacked by melee units it will open the gate.
You can quickly rush up with siege towers and ladders, scale the walls and fight your way to the stairwell down the stairs and over to the gate gears to open the gate; very effective if you have an army of calvary waiting to rush through the gate.