EU Server Performance - Nov 2011

[TSR-JoshuaM]

We are experiencing ongoing connectivity and latency issues for a great deal of players connecting to Splitpaw and Valor.  The issue itself does exist outside of our control but SOE is doing everything we can to assist in resolving the source of the problem.  I'll do my best to provide updates as this progresses but I cannot discuss, nor do I have, specifics.

Please keep all reports on this issue right here.  The intent here is to track the issue and so that you, the player, have an avenue to report that you are also affected.  This is not a soap box.  Commentary should be left to a discussion forums outside of the support boards of which there are many to choose from.  I am simply creating a centralized location that I can provide more information as it comes to me.

[wullailhuit]

Many guilds had to cancel raids tonight , we were getting reports of anything from between 3-10+ seconds lag between spells being clicked and actually casting.

[EvilAstroboy]

Still an issue. Raids cancelled again, going on a week now. Everyone in the chat channels is pretty annoyed.

[TSR-JoshuaM]

It is understandable that many people are upset over this and SOE is doing everything it can to help resolve the issue.  Keywords being 'help resolve', the problem itself exists outside of our control.

[Cronqar]

just cancelled raid after experiencing spell-lags of 5-10secs up to 20-30secs. We run those em raidzones normally in about 45minutes, but it is impossible/incurable at the moment. Last Monday we tortured our way through, but most likely will have to cancel Sunday if their is no change. It is ust unplayable in raidzones.

Groupzones are barely affected, but even here there is a more than usual delay. And occasional the lag finds its way into the regular overland zones and definetly is noticable at the PQ's.

[Helmarf]

Impossible to raid on Splitpaw tonight again. But what i found abit strange is, when i moved around in HM Throne of storms i got packet loss in some areas and moving away its gets perfectly fine and moving back to the spot = packet loss agian?? Samething trouh out the zone.

And if it is now some sort of problems in the area of Amsterdam Holland with routers there most be hundreds or thousands of other internet based stuff in that area having same problems with traffic but still nothing happends to fix it close to 1 week now?

[emagine]

i really dont know what to say, just very disappointed, Ima US based player on Splitpaw, and the lag is horrible.. i go over to guk where all my other alts are and i dont have a problem at all... when last week I had no problems at all on splitpaw...

[Arynai]

Thanks for keeping us updated. While I can't speak for the rest, this is exactly what I was personally looking for. Am I right to assume you don't need more information from people who have already opened tickets (unless specifically asked for)?

Just one additional question: should this external issue be resolved,

• should it be reduce lag for most of the players on these servers at once, or
• is it possible/likely/certain that there are several different connectivity issues that would affect different, large shares of the population? (Random individual troubles aside.)

[Luhai]

Situation on Valor last night was the same as on Splitpaw: betwen 8pm CET and about 10pm it was almost impossible to raid because of the lag. Several raidforces cancelled their raids because of this.

Around 10 pm CET the lag suddenly was gone.

[DarkMirrax]

This is getting beyond a joke now , the server is unplayable and im not sure how it can be an issue outside of SOE control ? we pay YOU for the running/hosting of the game so to blame someone else is unnaceptable.

As a side note why does the lag only exist everywhere EXCEPT Drunder ?

[TSR-JoshuaM]

DarkMirrax wrote:

This is getting beyond a joke now , the server is unplayable and im not sure how it can be an issue outside of SOE control ? we pay YOU for the running/hosting of the game so to blame someone else is unnaceptable.

As a side note why does the lag only exist everywhere EXCEPT Drunder ?

Without holding an entire tech class to educate you on how networking works, or how routing plays a part in the issues you are experiencing, the best I can do to explain to you is that in order for your data to reach our servers it must pass through many 3rd party routers.  It is an actual physical link in the most literal sense.  If any of these 'hops' experience issues or latency, everything after it will be affected such as your connection with the servers.  

That is the issue we are experiencing.  Several 'hops' are experiencing their own, external, issues.  This is why SOE cannot affect it.  We do not own the affected hardware and the best we can do is put the pressure on to let them know exactly how many of our mutual customers are being affected.

[Banedon_Toran]

To put it another way, if the lag monster is hanging around Belgium/Holland then your stuffed, but U.S. servers will be unaffected.

Sony don't own the lag monster, it's not their fault that it keeps chasing your internet packets and chewing them up before they reach Sony's server.

The internet is like a big game of pass the parcel, it's the equivalent of passing a usb stick with your game data to the first person you see outside your house and asking them to pass it on to other people until it evenerally reaches Sony HQ.

[TSR-JoshuaM]

I like that analogy  Thanks!

[Helmarf]

I heard that the problems allready was fixed for a coupel of days ago in the holland/belgium area.

[Pimmy]

The Juniper BGP problem should have been fixed already.

Or did they just forget to update their Amsterdam network? I think not.

[DarkMirrax]

that doesnt explain why drunder zones are completely unaffected , any reasoning on this ?

[TSR-JoshuaM]

Pimmy@Splitpaw wrote:

The Juniper BGP problem should have been fixed already.

Or did they just forget to update their Amsterdam network? I think not.

Fixed is one thing.  Deployed is another.  Level 3 have their issues resolved, according to them, but the Juniper issues require a SW fix that needs to be downloaded by their customers manually.

[Arynai]

EDIT: NVM.

[Vitriol]

DarkMirrax wrote:

that doesnt explain why drunder zones are completely unaffected , any reasoning on this ?

Drunder has been terrible for us as well as any other raid zone.

[Luhai]

DarkMirrax wrote:

that doesnt explain why drunder zones are completely unaffected , any reasoning on this ?

Unfortunately Drunder is laggy, too.

[Zergosch]

Gimme a IP to tracert and i post a live result asap ^^

Short news:

From my start to paris my pings constant 30ms. From Paris to Washington the ping raise to 110ms.

From Washington to a subnet from SoE (64.*.*.*) the ping raise again up to 200ms.

Well, from my point, there is a routfault, not primary from soe, but on the other side of the sea

[TSR-JoshuaM]

Zergosch wrote:

Gimme a IP to tracert and i post a live result asap ^^

(your server name).everquest2.com

[Vitriol]

SOLO, IN GUILDHALL:

Tracing route to amseq2-03-01.everquest2.com [195.33.139.96]

over a maximum of 30 hops:

  1     9 ms    17 ms     7 ms  10.97.148.1

  2    47 ms    23 ms     7 ms  gate-geam-1b-ge110.network.virginmedia.net [80.0

.145.149]

  3    43 ms    12 ms    27 ms  gate-core-1b-ae0-0.network.virginmedia.net [80.0

.144.141]

  4    14 ms    28 ms    59 ms  manc-bb-1b-as2-0.network.virginmedia.net [213.10

5.175.169]

  5   170 ms   111 ms   149 ms  popl-bb-1a-as4-0.network.virginmedia.net [212.43

.162.85]

  6    41 ms    21 ms    21 ms  popl-tmr-1-ae4-0.network.virginmedia.net [213.10

5.159.2]

  7    29 ms    29 ms    29 ms  amst-ic-1-as0-0.network.virginmedia.net [213.105

.175.6]

  8    31 ms    59 ms    31 ms  nlamtr1104er1.am.nl.attglobal.net [195.69.144.37

]

  9    30 ms    31 ms    31 ms  32.112.232.10

 10    45 ms    33 ms    31 ms  mdf002c7613r0004-tge-10-1.ams1.attens.net [32.42

.0.178]

 11    43 ms    43 ms    57 ms  vl50.amsixmsfc-2.sonyonline.net [195.33.129.77]

 12    44 ms    57 ms    43 ms  195.33.139.96

Trace complete.

RAID INVITED, BUT IN GUILD HALL NOT DOING ANYTHING:

C:Documents and SettingsTEMP>tracert splitpaw.everquest2.com

Tracing route to amseq2-03-01.everquest2.com [195.33.139.96]

over a maximum of 30 hops:

  1   620 ms   562 ms   601 ms  10.97.148.1

  2   809 ms   993 ms   833 ms  gate-geam-1b-ge110.network.virginmedia.net [80.0

.145.149]

  3   385 ms   341 ms   380 ms  gate-core-1b-ae0-0.network.virginmedia.net [80.0

.144.141]

  4   460 ms   423 ms   505 ms  manc-bb-1b-as2-0.network.virginmedia.net [213.10

5.175.169]

  5   508 ms   511 ms   483 ms  popl-bb-1a-as4-0.network.virginmedia.net [212.43

.162.85]

  6   623 ms   633 ms   635 ms  popl-tmr-1-ae4-0.network.virginmedia.net [213.10

5.159.2]

  7   820 ms   637 ms   689 ms  amst-ic-1-as0-0.network.virginmedia.net [213.105

.175.6]

  8   621 ms   650 ms   644 ms  nlamtr1104er1.am.nl.attglobal.net [195.69.144.37

]

  9   802 ms   763 ms   920 ms  32.112.232.10

 10   720 ms   937 ms  1059 ms  mdf002c7613r0004-tge-10-1.ams1.attens.net [32.42

.0.178]

 11   356 ms    81 ms    73 ms  vl50.amsixmsfc-2.sonyonline.net [195.33.129.77]

 12    38 ms    37 ms    53 ms  195.33.139.96

[Zergosch]

TSR-JoshuaM wrote:

Zergosch wrote:

Gimme a IP to tracert and i post a live result asap ^^

(your server name).everquest2.com

if i tracert that way, my ping is even lower then the eq2 client tells me.

its from 29-42ms with a los at hop 15. eq2 client tells me 100-140ms.

[Pateras]

that tracert looks kinda interesting

if it was router problem wouldnt be the same results even if u in raid or solo in the same place ?

since he is in the guild hall and all

the results showing 10x latency just by joining the raid not moving to the actual zone server

btw lag is alive and kicking in splitpaw still if u were wondering

[TSR-JoshuaM]

Vitriol@Splitpaw wrote:

SOLO, IN GUILDHALL:

Tracing route to amseq2-03-01.everquest2.com [195.33.139.96]

over a maximum of 30 hops:

  1     9 ms    17 ms     7 ms  10.97.148.1

  2    47 ms    23 ms     7 ms  gate-geam-1b-ge110.network.virginmedia.net [80.0

.145.149]

  3    43 ms    12 ms    27 ms  gate-core-1b-ae0-0.network.virginmedia.net [80.0

.144.141]

  4    14 ms    28 ms    59 ms  manc-bb-1b-as2-0.network.virginmedia.net [213.10

5.175.169]

  5   170 ms   111 ms   149 ms  popl-bb-1a-as4-0.network.virginmedia.net [212.43

.162.85]

  6    41 ms    21 ms    21 ms  popl-tmr-1-ae4-0.network.virginmedia.net [213.10

5.159.2]

  7    29 ms    29 ms    29 ms  amst-ic-1-as0-0.network.virginmedia.net [213.105

.175.6]

  8    31 ms    59 ms    31 ms  nlamtr1104er1.am.nl.attglobal.net [195.69.144.37

]

  9    30 ms    31 ms    31 ms  32.112.232.10

 10    45 ms    33 ms    31 ms  mdf002c7613r0004-tge-10-1.ams1.attens.net [32.42

.0.178]

 11    43 ms    43 ms    57 ms  vl50.amsixmsfc-2.sonyonline.net [195.33.129.77]

 12    44 ms    57 ms    43 ms  195.33.139.96

Trace complete.

RAID INVITED, BUT IN GUILD HALL NOT DOING ANYTHING:

C:Documents and SettingsTEMP>tracert splitpaw.everquest2.com

Tracing route to amseq2-03-01.everquest2.com [195.33.139.96]

over a maximum of 30 hops:

  1   620 ms   562 ms   601 ms  10.97.148.1

  2   809 ms   993 ms   833 ms  gate-geam-1b-ge110.network.virginmedia.net [80.0

.145.149]

  3   385 ms   341 ms   380 ms  gate-core-1b-ae0-0.network.virginmedia.net [80.0

.144.141]

  4   460 ms   423 ms   505 ms  manc-bb-1b-as2-0.network.virginmedia.net [213.10

5.175.169]

  5   508 ms   511 ms   483 ms  popl-bb-1a-as4-0.network.virginmedia.net [212.43

.162.85]

  6   623 ms   633 ms   635 ms  popl-tmr-1-ae4-0.network.virginmedia.net [213.10

5.159.2]

  7   820 ms   637 ms   689 ms  amst-ic-1-as0-0.network.virginmedia.net [213.105

.175.6]

  8   621 ms   650 ms   644 ms  nlamtr1104er1.am.nl.attglobal.net [195.69.144.37

]

  9   802 ms   763 ms   920 ms  32.112.232.10

 10   720 ms   937 ms  1059 ms  mdf002c7613r0004-tge-10-1.ams1.attens.net [32.42

.0.178]

 11   356 ms    81 ms    73 ms  vl50.amsixmsfc-2.sonyonline.net [195.33.129.77]

 12    38 ms    37 ms    53 ms  195.33.139.96

Hop 1 is your router/modem, Hop 2 exists in your ISP.  If you notice the ping times, you are experiencing a great deal of lag just between your PC and service provider.  While you may also be experiencing additional lag due to the routing issues to our EU severs you definitely have an issue of your own within your network.

1   620 ms   562 ms   601 ms  10.97.148.1

2   809 ms   993 ms   833 ms  gate-geam-1b-ge110.network.virginmedia.net [80.0.145.149]

[Elfin Hoi Man]

EU Server Performance - Nov 2011  = [Removed for Content]

Why don't you just get rid of the EU servers, I don't mind playing with a slightly higher ping, at least I would be able to PLAY the game without lag.

What I don't understand, and I i'm not an unreasonable person is, is why this lag came into being, at exactly the same moment, for ALL raid guilds on Splitpaw, which conincided with an SOE hardware upgrade? Or am I mis-informed about something.

There will be no hardcore raid guilds left on Splitpaw in a few weeks time if this issue is not resolved.

[TSR-JoshuaM]

The bulk of the lag reports started on Nov 8 with an accompanying 3rd party outage, not SOE.  Hardware upgrades happened in July/August.

We don't need to get rid of the EU servers for you to play on another.  That's ... ya.  

[Helmarf]

Here is mine just before my raid inv in guild hall

 1    38 ms    99 ms    99 ms  dsldevice.lan [192.168.1.254]

  2    17 ms    17 ms    23 ms  gw3-no41.tbcn.telia.com [81.232.58.1]

15    56 ms    51 ms    52 ms  vl177.amsixsw-2.sonyonline.net [213.244.164.58]

And i still suffer from realy bad lag???? Ping is jumping up and down like a roller coaster in raid will fighting. So that realy dont explains it????

I contacted my ISP to hear what they have to say about it, but still waiting for an answear. Everything else works fine, other games/intertet stuff, but Everquest 2 do not So 9 days now for Splitpaw as the problems started atleast for our guild the 5 november.

So aparently SoE cant do nothing about this, but what do you recomend us to do more then just wait till it will dissapear?

[TSR-JoshuaM]

My role in this is to make sure the issue is reported and help anyone experiencing lag through the troubleshooting process. 

Just out of curiosity where is hop 3 - 14?