Here's some feedback...

[otlg]

<p>Where was the QA/Testing on this morning's hot-'fix'??</p><p>Seriously.. where was the push to test on this to find the issue before you shoved it onto the live servers and ruined an entire day of play for some people?</p><p>I really don't care what the mods think of this post, someone (I'm looking at you on this one Scott, it's your ship) has to step-up and explain how, with two test servers, something like this can happen.  This is *not* the first time this has happened.  No patch should EVER bypass going to test, in exactly the same manner it would go to live.  Period.  End of story.</p><p>If patches are arbitrarily going to be allowed to bypass the test server, why even have it?  Seriously.  The entire point of a QA process is that it is followed... allways... EVERYTIME you guys deviate from the 'proper' way to do things, some little bug ends up causing non-stop grief.</p><p>And, when things go wrong (and they will and do, I'll accept that, even with proper policy things will break), the better part of a day to recover??  If Sony was smart there would be a nice SAN unit that the databases run off of, and that SAN would be snapshotted before hot fixes/big patches.  It can then be instantly be put back into production if thing go to crap.  This stuff is enterprise datamanagement 101 guys.  If you need to know about 24/7 or mission ciritical system design hire someone that has a clue.</p><p>Anyway, I know I won't get an official responses from someone like Scott due to policy reasons, but I'm hoping he will see this post and address these issues through another channel (perhaps posting in a separate forum).  SOE you guys are human and are allowed to make mistakes, *but* we, as your customers, deserve some sort of explanation.  </p><p>Oz</p>

[Josgar]

Generally changes to  a few particular items shouldn't be able to cause this many problems... Im guessing what happened was they pulled in something they shouldn't have...

[otlg]

<p>Josgar:</p><p>Agreed.  Which is why you TEST the deployment first, on <gasp> the test server </gasp>.  Again, it's inexcusable.  There is NO valid reason to not have every patch go through test first for a day.  To fix a <b>VERY MINOR, <i>BETA RELATED</i></b> PVP issue, Sony, by cutting corners, knocked the servers out for just about the entire day.</p><p>If it took an extra day for folks wearing <b>BETA RELEATED</b> LON cloaks on a PVP server to get a fix, would that have been so bad?  I mean they could have always.. oh... un-equipped the f'ing things.  It is after all a beta that those people are CHOOSING to participate in.  Just like having a character on test means you don't get to complain when they actually hose you for a day or two on test.... that's the whole point of test and testing and beta...</p><p>Sorry I'm not letting this one go that easily.  Laziness is the sole reason this happened today.</p>

[Josgar]

If I had to guess: They had this on the test server and when they pulled if off they pulled too many changes with it. A simple item change shouldn't do this. It seems more like they pulled extras down with it.

[otlg]

<p>A simple item change shouldn't have required any of the changes and should have at most required 1 file to be patched.. maybe 2.. why the light didn't go off at SOE that more than that was happening is beyond me.</p><p>Also, above and beyond that.. they exact same patch package should have been issued to test before hand.  Yes I know test currently has GU38 patches on it.. and that's a major problem too.  There is no way for Sony to REALLY test the deployment of their patches, because the test server isn't in-sync with live.</p><p>Whole situtation is horrible, and whoever is in charge of the QA/Testing plan there needs to be fired ro demoted and someone with a half-clue on those sorts of things hired to replace them.</p>

[Dasein]

The problem looks like an issue with DB communication, where transactions are not being committed or sometihng to that effect (I've been at work all day, and have not had a chance to actually play to see the problems myself, so I'm going off descriptions others have posted). Thus, what you are looking at is an aspect of the code that is likely specific to each server, as I would imagine each server has it's own database back-ends. Thus, testing this change on a QA environment might not catch the problem.

[Rijacki]

<cite>otlg wrote:</cite><blockquote><p>A simple item change shouldn't have required any of the changes and should have at most required 1 file to be patched.. maybe 2.. why the light didn't go off at SOE that more than that was happening is beyond me.</p><p>Also, above and beyond that.. they exact same patch package should have been issued to test before hand.  Yes I know test currently has GU38 patches on it.. and that's a major problem too.  There is no way for Sony to REALLY test the deployment of their patches, because the test server isn't in-sync with live.</p><p>Whole situtation is horrible, and whoever is in charge of the QA/Testing plan there needs to be fired ro demoted and someone with a half-clue on those sorts of things hired to replace them.</p></blockquote>Not all changes are noted in the patch notes.  If there was a change to correct an exploitable situation, they might not want to have the delay to  put it on Test first for a span of time and then move it on to Live.  With changes to exploitable situations, they also generally do not publish the changes made (especially if the exploit was caught early and only affecting a small area).The LON cloak, it has already been said by Red Names, had -zero- to do with the change that wrecked havoc.  It was a different, and unpublicised, fix.

[otlg]

<cite>Dasein wrote:</cite><blockquote>The problem looks like an issue with DB communication, where transactions are not being committed or sometihng to that effect (I've been at work all day, and have not had a chance to actually play to see the problems myself, so I'm going off descriptions others have posted). Thus, what you are looking at is an aspect of the code that is likely specific to each server, as I would imagine each server has it's own database back-ends. Thus, testing this change on a QA environment might not catch the problem.</blockquote><p>Well seeing as my server is back offline again because the broker, bank and mail were all broken when they brought it back up after 4 hours down time, I'm begining to wonder if anyone there even knows what they are doing. </p><p><i>Rijacki:</i></p><p>"Not all changes are noted in the patch notes.  If there was a change to correct an exploitable situation, they might not want to have the delay to  put it on Test first for a span of time and then move it on to Live.  With changes to exploitable situations, they also generally do not publish the changes made (especially if the exploit was caught early and only affecting a small area).The LON cloak, it has already been said by Red Names, had -zero- to do with the change that wrecked havoc.  It was a different, and unpublicised, fix."</p><p>I don't care if the patch was the cure for cancer, AIDS and just about every other disease on the face of the planet.  That's STILL no reason not to test it.  Sony also seems to believe in security through obscurity.  That has been proven to be one of the worst methods of security.   If Sony was smart, they would announce the exploit they fixed when issue the patch, then watch to see what happens when half the people who play the game try every possible variant thereof, and find out which corner cases they missed.</p>