After a few more fights here and there, I've gotten him to stack 3 buffs for 100%+ reduction at once, forcing me to tactically retreat and try again. Wasn't sure what was doing it, but yeah the successful kills did go a little slower, must bug it if you go too fast 9(specifically if you take him over more than one barrier threshold at once)
If the mystic barriers overwrote one another instead of stacking, this encounter would be less invulnerable more often.