It's strange for a single town to bug out like this, have you perhaps tried checking your game files? Both Steam and Desura offer this option, the standalone client should do it at startup.
- Desura: In your game list right-click on Dawn of Fantasy and select Verify Files.
- Steam: In your Library, right click on Dawn of Fantasy, select Properties -> Local Files and click Verify integrity of game cache....
My first thought would be that it's a bunch of corrupted files relating to orcs and the game goes haywire trying to load those files. let me know if that helps or not.
If it doesn't, can I ask you to post some game logs taken after trying to log into your orc town? simply leave game after it starts lagging and add the most recent log here. Info on where to find logs and how to add them can be found here
Thanks, and sorry for the problems