There are two possible explanations - either somehow your town got corrupted and causing these crashes - although unless you are transferring large amount of units between hometown and army, or fighting in a large quests, or siege battle - the game shouldn`t crash. But lets try the following based on our knowledge base:
1) Open your homeland and transfer all the Peasants/Laborers e.t.c worker units into an Army. Leave them there and try playing to see if it`s the workers. I think there is a possible crash related to worker unit corruption.
2) Create a new town - and see if the crash continue. If it is your internet and not something in your town this should also crash.