The biggest effect from walls is when you're up against ranged attackers (archers and such, or any unit with a ranged magical attack): walls will protect the units inside, though there's quite a random factor included.
Also, if you're defending and one of your units is significantly stronger than the others, place that behind the gap in the wall, forcing all non-flying units to fight your strongest unit first.
Besides, enemy units running around in your town during combat, will increase the chances of buildings being destroyed. Walls help to keep them out.
But honestly, I never bother building any walls
