KR: I remember I hated Raiders and Pillagers when I fought them. With their magic damage, they could easily rip apart any soldier, even paladins, in a matter of moments. Also, the fact that they had different types of resistance meant that a Mage or archer could pick the wrong target and the Raider and Pillagers could just continue walking to the exit without a scratch. Overall, they were a challenge I had not faced with the enemies of MCs. And they were just so annoying.
KRF: I found no enemy particularly annoying, but Blood Tricksters and Reapers/Parasytes were a nuisance. For me, Blood Tricksters were difficult because while they gave my towers and soldiers more enemies to fight, they stayed in the back lines. Reinforcements couldn't kill them, and it was difficult for me to position my hero so he could duel them. For me, it wasn't really the zombies who were annoying, but the Tricksters themselves.
When I played the Lost Jungle the first time, my strategy wasn't the best, so I would eventually get overrun by Gorillons and Reapers because of their damage (Reapers) and health (Gorillons). But then I changed my strategy and everything was better.
Note: I haven't played the MCs in Frontiers, so this might change.
I don't have Origins, so I can't say what my most hated enemy is in that game, for obvious reasons.