I fully agree to this. I was disappointed when I fully upgraded everything to find out that my city would not turn in to one of those amazing castles like you can visit/siege. Still grass within the walls and straw roofs... Quite empty aside from your soldiers stationed there. I've kinda stopped playing the game for now. Only PVP left and some quests, but if I can't build on my city any longer, which was the best part for me, then I don't feel the urge to keep playing anymore.
I wanted the ability to make a sprawling living castle. Aw well.