I spec'd out of the Agility AA a few LU's ago.. but when I had it, I was casting fast even with reequipping the symbol. Hell, it was talked about back when the AAs first came out and people were posting about the symbol thing.
If it doesn't anymore, that sucks.