Nagafen/Venekor Server Merge

Kiara · 10-21-2009, 07:31 PM

We're still planning on going forward with the PvP server merge Wednesday night at midnight PST. Initially only Venekor will be brought down to backup the database and export all of the data. When the export is complete, Nagafen will be brought down for us to merge the data. After the merge takes place, we have some other background tasks that could take a few hours. This includes cleaning up universal-chat, merging shared banks and updating player names and guild names that had conflicts. Be prepared for as much as a 24-hour downtime for Venekor players. Nagafen players will experience around 12 hours of downtime depending on how fast the processes go.

What this means:

The Venekor server will be brought down at midnight tonight, Wednesday October 21st. Nagafen will be brought down around 8am PDT on Thursday October 22nd.

Neither Butcherblock nor Lucan D'Lere should be affected by this downtime.

Please watch this thread for updates about the merge.

Discussion and questions can be directed to this thread.

Thank you all for your patience with this process and we apologize for the inconvenience.

Rothgar · 10-22-2009, 04:37 AM

Maintenance Plan:

Bring down Venekor (Done)

Characters level 10 or less, with less than a plat, that have not been logged in since Feb 2008 have been flagged inactive. (Done)

Remaining active characters ( over 100k ) are being exported to flat file. Est. Time of Completion for EXPORT ONLY is 12 hours. We will post a more accurate ETA after the export has been running for awhile. (In Progress)

Three hours before the export is due to be completed, Nagafen will be brought down for a final backup. Currently this is scheduled for 8:00 am PST but may happen sooner if the export is running fast.

After the Nagafen backup is complete and the Venekor export is complete, the import of the flat file into Nagafen will begin. ETA on this process is another 12 hours. We will also post another ETA several hours into this process with a more accurate time of completion.

When Nagafen import is completed, scripts will be run to handle name conflicts, exports shared bank and move in-game mail. ETA for this script is 3 hours.

After all scripts are complete, we will perform initial verification that shared banks and guild halls have moved successfully.

QA will start their smoke test. ETA is 1 hour

Nagafen will be unlocked

Rothgar · 10-22-2009, 06:16 AM

Status Update

The export process is just over 10% done. It seems to be going slower than our two initial test runs and we are trying to determine why.

I'll give another status update in the morning.

Rothgar · 10-22-2009, 01:20 PM

Status Update

At 7:00am this morning our export script experienced a crash. This is not something that has happened in any of our test runs or previous merge processes so we're looking into causes. We suspect that it could have been a hardware issue (faulty RAM, hard disk error, etc) or it could have been data related. Since our test runs used actual copies of Venekor data, a data error doesn't seem likely unless it developed in the last week.

We've restarted the export process as of 7:00am PST on different hardware and are monitoring it closely. The current impact is an extended downtime for Venekor players of roughly 12 more hours. For Nagafen players, this would mean a delayed maintenance schedule, but hopefully not an extended downtime.

We will keep you posted throughout this process.

Rothgar · 10-22-2009, 06:27 PM

Status Update

At 2:00pm PST we're just over 30% done with the export. Unfortunately the process is still going slower than our test cases. We are nearing the point in the export where the script failed this morning at 7:00am. I'll update again in an hour and let you know if we're still progressing without problems.

At this rate, it looks like we may need another 11 hours of export time before we are able to bring down Nagafen for a final backup.

Rothgar · 10-22-2009, 07:47 PM

Status Update

At last check, we were over 40% done with the export process. We've passed the point where the script had a failure this morning, so I'm still optimistic that it was just a hardware problem on the other server.

I'll update again in a couple of hours.

In the meantime feel free to join the party on Test-Copy. If you don't have a version of EQ2 patched to the Test Server, no worries, you can test out our new streaming client! Check out this thread for more information.

Calling all Testers! Streaming Client Now Available!

Rothgar · 10-22-2009, 10:14 PM

Status Update

The export process is over halfway complete. At the current speed we expect to take Nagafen down around midnight, give or take a couple of hours, for a final backup. After the backup is complete, we will begin to import the Venekor data into the Nagafen database.

We *hope* this process will go faster than our estimation of 12 hours due to Nagafen's new database hardware.

I'll post another update in a couple of hours.

Rothgar · 10-23-2009, 12:09 AM

Status Update

The process is over 70% complete. It looks like process is running a little faster now. We may be looking at bringing Nagafen down in an hour or two.

Rothgar · 10-23-2009, 01:10 AM

Status Update

As of 9:00pm PST the export process is over 80% complete. Nagafen should be coming down now for a final backup before we begin the import process.

ETA on import is still 12 hours until we see what kind of throughput we get.

I'll keep the updates coming.

Rothgar · 10-23-2009, 03:12 AM

Status Update

The export process is complete.

The Nagafen backup is 50% complete.

We will start the import of Venekor data as soon as the backup is done.

Rothgar · 10-23-2009, 05:10 AM

Status Update