From an interface consistency standpoint, it would be nice if the Map and Quest Log closed with Esc as to (nearly) all other windows.
I find myself hitting Esc first out of habit, and then having to remember that for these windows I need to click, or to hit M or L again.
(I am extreeeeemely keyboard-focused in my approach to UI :)