Hah, that same door holds you too
It's easily overlooked but there is a 4th cave, somewhere in the middle left. When you walk by it, there is the key inside. I walked by it half a dozen times without seeing the entrance.
Status report:
Two areas after Valley of Shadows I'm lurking around in dragons temple, first level cleared. All 3 fighters have made some decent exp in their dual-class and will regain their fighter-skills with two more levels to go.
It just gets always harder so my party needs "heal" spell urgently.