What affects building time?
 

Building times seem to be all over the place. As an example, just now I wanted to build a sylvan shrine. Before I started it said 4 hours construction time. As soon as I gave the order, it was down to just over one hour. Then, about 20 minutes later, it was completed! Is this working as intended, and if so, what's causing this behaviour?

The first several buildings build at a sped up rate, once the speed up for the first few buildings you build goes away the timers work right :) Also you can spend crowns or task orc labourers to help speed up buildings.

OK thanks. So eventually if it says 5 hours to construct, it will definitely take 5 hours?

Yup, I "think" it was the first 5 or 10 buildings that got the speed boost. I think it was 5. Then the timers work. Good question :)