One thing I've done in the past is add a cornered state to the enemy. Put yourself in the shoes of the fleeing enemy. It is scared, it is hurt, it is fleeing and trying to just get away! Perfect mindset to get yourself cornered, which can be really fun for the player.
Once cornered, the enemy can do different things, ideally depending on the personality of the enemy itself:
- They can cower and duck into the corner and wait for death. Only really works if you can convey those emotions. A single-character rogue-like enemy probably can't convey that without text or something. From here, the player could decide to leave the enemy alone, showing compassion. Could be really neat if the enemy then becomes passive. Or if healed, even become an ally of the player.
- They could go into a blind rage and launch a last-ditch attack against the player, or unleash some kind of escape spell or skill; such as a squid squirting ink, or a lizard playing dead.
These are really more interesting than avoiding the corner in the first place. And are probably faster to compute, which will let you spend those cycles elsewhere.
But if you MUST avoid the corners at least a little, the best way I've found is to discourage walking down tight lanes. Simply check the 8 spaces that are 2 spaces away from the enemy. Such that if the enemy is on space (9,9) check spaces: (7,7)(11,11)(7,11)(11,7)(7,9)(9,7)(9,11) and (11,9). If any of these are open, check the 4 spaces around that space too. Weight movement in those directions according to the number of spaces you see. In a rogue-like, the walls are rarely even that far away, and this will encourage the enemy to stay in the same room.
Nothing is more annoying than an enemy that runs off down a long corridor, regardless of that corridors termination, and leaves the player to have to chase it down.