Well as I've experienced it, when I take my workers to go out and loot bodies they sometimes encounter a couple of stragglers that they end up fighting and killing. I think this results into the unit gaining experience and leveling up. However, workers have no level ups so the program doesn't know what to do with this. I say this because the first bugged worker unit that I've had when selected with other units show up in the lower unit icons that it has leveled up, but when I click on the worker unit there are no level up or skill points to be had. Then after the battle and when I transfer them back to the homeland and move them near a well the game just crashes. I hope that helps.
Here's a screenshot of my unit after a fight, notice how the skill points is -3