SiMPLE USA Lag WorkGroup?

lonewolf-mcquade · 2015-10-11 23:17:14

Hello Team SiMPLE,

SiMPLE USA has been struggling with lag issues for about 2 weeks. I've noticed over the last 5 days, much lower player counts and several players leaving in frustration after commenting in chat. Obviously this is not ideal! I'd like to offer to help in any way that I can, as I do this type of work for a living. I monitor and maintain a huge national network with resources just like your server that must be accessed consistently and with quality across multiple national broadband carriers 24/7/365. I don't know where you-folks are on this problem, but I have some suggestions if anyone is interested. I'd like to get a work-group together in TS, or would volunteer to join anyone at SiMPLE who is working on this to help.

I've done some preliminary testing with ping and tracert that seems to indicate an issue from hop 8 down on my path.

C:\WINDOWS>tracert 162.217.248.97

Tracing route to 162.217.248.97 over a maximum of 30 hops

1 1 ms 1 ms <1 ms [redacted]
2 9 ms 11 ms 8 ms [redacted]
3 9 ms 8 ms 9 ms [redacted]
4 13 ms 32 ms 12 ms ae-36-0-ar02.charlvilleco.va.richmond.comcast.net [68.86.125.145]
5 32 ms 23 ms 16 ms et-5-0-0-0-ar02.staplesmllrd.va.richmond.comcast.net [162.151.59.10]
6 28 ms 27 ms 27 ms he-1-1-0-1-10-cr02.56marietta.ga.ibone.comcast.net [68.86.91.149]
7 49 ms 53 ms 51 ms be-11314-cr01.dallas.tx.ibone.comcast.net [68.86.85.21]
8 80 ms 82 ms 78 ms be-11315-cr02.losangeles.ca.ibone.comcast.net [68.86.85.142]
9 78 ms 86 ms 82 ms he-0-1-0-1-pe01.losangeles.ca.ibone.comcast.net [68.86.85.98]
10 82 ms 81 ms 93 ms ae2.Comcast.lax.us.AS40676.net [23.238.223.49]
11 * * * Request timed out.
12 78 ms 77 ms 78 ms hosted-by.iniz.com [104.216.1.50]
13 78 ms 77 ms 81 ms 2.164.161.107.iniz.com [107.161.164.2]
14 79 ms 106 ms 77 ms 162.217.248.97

Trace complete.

Hop 8 is registered to Comcast and Tech Admin POC info is registered on their Network Whois result. If, with further exploration, Comcast is pinpointed, they absolutely should care about this problem. In that scenario, the issue likely isn't only affecting your server, but any customers that route through that node. That means any priority traffic like video conferencing and QOS (quality of service) voice traffic could be screwed up also.... for ALL customers. These types of issues often self-resolve. But if you want it corrected ASAP, it can require a little hand-holding. It's best to try to have your server host company contact Comcast (for example.) If they will not, there is nothing wrong with doing it yourself, and conferencing the two parties together once you have conjured up support/interest.

OrgTechHandle: IC161-ARIN
OrgTechName: Comcast Cable Communications Inc
OrgTechPhone: +1-856-317-7200

Remember, my comments are **preliminary.** It would be best to compare results with a few other users before proceeding with any course of action. Please let me know if I can be of assistance to Team SiMPLE. That's the least I could offer in return for the hours of fun I've had on the server.

Last edited by lonewolf-mcquade (2015-10-11 23:20:20)

ATF_SurrenderMonkey · 2015-10-12 01:04:43

Here is mine, from Sacramento, CA .Oct 10 400pm PDT ..I have no idea what it all means but if it helps, please use it.

Microsoft Windows [Version 10.0.10240]

C:\Windows\System32>tracert 162.217.248.97

Tracing route to 162.217.248.97 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.0.1
2 10 ms 10 ms 18 ms 96.120.14.61
3 8 ms 9 ms 9 ms xe-7-2-2-0-sur03.sacramento.ca.ccal.comcast.net [68.87.212.193]
4 9 ms 9 ms 8 ms ae-2-0-ar01.sacramento.ca.ccal.comcast.net [162.151.18.133]
5 11 ms 12 ms 11 ms 68.86.93.193
6 21 ms 33 ms 20 ms be-10915-cr02.losangeles.ca.ibone.comcast.net [68.86.86.98]
7 19 ms 19 ms 35 ms he-0-1-0-0-pe01.losangeles.ca.ibone.comcast.net [68.86.85.94]
8 19 ms 19 ms 19 ms ae2.Comcast.lax.us.AS40676.net [23.238.223.49]
9 19 ms 20 ms 19 ms 10.98.98.1
10 21 ms 20 ms 20 ms hosted-by.iniz.com [104.216.1.50]
11 19 ms 19 ms 20 ms 2.164.161.107.iniz.com [107.161.164.2]
12 19 ms 30 ms 20 ms 162.217.248.97

Trace complete.

Trench · 2015-10-12 18:42:34

Until it becomes established that "this particular route is slow /regardless/ of when we measure it", noting the time of day for a particular sample is probably important. So that as additional samples are collected, a picture emerges of whether the router itself gets slower at a certain time of day, or maybe that slow-performing router only /becomes/ part of the route at certain times as Comcast's network reacts and adjusts to loading, etc.

I come out through Dallas, which your Virginia route also crossed through right before the "bad hop". At least when I measured it just now (Monday 11am CDT), it wasn't routed or leaving for Los Angeles using exactly the same routers your trace had shown. But it did enter Los Angeles through the same router, and I saw the same ~30ms bump seen in your trace:

c:\>tracert 162.217.248.97

Tracing route to 162.217.248.97 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  Wireless_Broadband_Router.home [192.168.1.1]
  2     4 ms     4 ms     4 ms  xxxx.verizon-gni.net [xx.xx.xx.xx]
  3     7 ms     6 ms     9 ms  xxxx.verizon-gni.net [xx.xx.xx.xx]
  4     *        *        *     Request timed out.
  5     6 ms     6 ms     6 ms  0.ae1.XL3.DFW7.ALTER.NET [140.222.226.99]
  6     8 ms     9 ms     6 ms  TenGigE0-4-0-3.GW5.DFW13.ALTER.NET [152.63.98.205]
  7     9 ms     9 ms     9 ms  be-202-pe02.1950stemmons.tx.ibone.comcast.net [66.208.229.169]
  8    10 ms    11 ms     9 ms  be-11-cr01.dallas.tx.ibone.comcast.net [68.86.82.133]
  9    37 ms    39 ms    39 ms  be-11315-cr02.losangeles.ca.ibone.comcast.net [68.86.85.142]
 10    37 ms    37 ms    36 ms  he-0-0-0-1-pe01.losangeles.ca.ibone.comcast.net [68.86.85.30]
 11    44 ms    43 ms    44 ms  ae2.Comcast.lax.us.AS40676.net [23.238.223.49]
 12     *        *        *     Request timed out.
 13    43 ms    44 ms    44 ms  hosted-by.iniz.com [104.216.1.50]
 14    44 ms    44 ms    44 ms  2.164.161.107.iniz.com [107.161.164.2]
 15    45 ms    43 ms    44 ms  162.217.248.97

Trace complete.

Which, for getting from Dallas to Los Angeles, I'm not sure whether ~30ms is really to be considered abnormal. Guess we might also need to come up with the proof that it doesn't always take that long.

-Trench

Last edited by Trench (2015-10-12 18:44:59)

lonewolf-mcquade · 2015-10-13 00:09:30

Thanks trench. The anomalies I found were not present in the tracert I provided above. That was more just a mapping of how my data travels. When I ran an extended ping to each node, I had a reoccurring and discernable spike from the destination down to node 8 (working backwards.) From 7 to me, it was as "clean as a whistle" using the same standard. If we can at least pinpoint where the spikes are happening or what range, Comcast engineers should be somewhat obligated to investigate with their more thorough tools in their own NOC environment. It would be nice to get a couple folks on TS, and run the extended pings and see who gets what, and where.

-lwmq

Last edited by lonewolf-mcquade (2015-10-13 00:11:04)

PitViper · 2015-10-13 01:31:13

06 United States    24.30.174.65 tge0-9-0-20.lamrcadq01h.socal.rr.com
07 United States    72.129.9.214    agg15.lamrcadq01r.socal.rr.com
08 United States    72.129.9.0    agg28.lsancarc01r.socal.rr.com
09 United States    66.109.6.102    16.lsancarc0yw-bcr00.tbone.rr.com
10 United States    107.14.17.250    0.ae1.pr1.lax00.tbone.rr.com
11 United States    50.248.118.177 pe01.losangeles.ca.ibone.comcast.net
12 United States *Walnut 23.238.223.49    ae2.Comcast.lax.us.AS40676.net
13 United States *Walnut 104.216.1.50    hosted-by.iniz.com
14 United States *Walnut 104.216.1.50    hosted-by.iniz.com
15 United States *NY 107.161.164.2    2.164.161.107.iniz.com
16 United States *LA 162.217.248.97    SiMPLE ~

OC to LA, to Walnut ,to New York, to LA Pathetic.

It's comcast alright...Check out my hops....I live less than 50 miles away from the server. Just stupid.
*iniz.com seems to be a problem along with comcast.

Last edited by PitViper (2015-10-13 03:46:45)

Catpain_Blackadder · 2015-10-13 03:08:02

Here is my output using mtr, which basically does a series of traceroutes. Not sure what's going on with hop 2, which is my ISP (maybe they're throttling me, or maybe it has to do with my 1 Gb home LAN going through a 6-8 Mb DSL connection), but hop 10 on Comcast also is showing a high packet loss. Hop 7 looks like it is going through several alternate network paths, but does not seem to be affecting things.

-----------------------------------

My traceroute [v0.80]
charlie (0.0.0.0) Mon Oct 12 20:51:17 2015
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. x.x 0.0% 81 0.4 0.5 0.4 1.3 0.2
2. x.x.x.sbcglobal.net 80.0% 81 25.4 25.0 22.4 31.5 2.5
3. x.x.x.x 0.0% 81 22.2 24.2 22.0 33.0 1.9
4. 12.83.79.17 0.0% 81 24.3 40.6 22.2 78.8 18.2
5. cgr1.cgcil.ip.att.net 0.0% 81 31.8 43.9 27.6 87.9 16.2
6. 192.205.37.22 0.0% 81 27.6 46.2 27.4 97.0 19.9
7. he-2-5-0-0-cr01.350ecermak.il.ibone.comcast.net 1.2% 80 29.7 42.8 27.9 82.6 15.6
hu-3-12-0-0-cr01.350ecermak.il.ibone.comcast.net
hu-3-3-0-0-cr01.350ecermak.il.ibone.comcast.net
hu-3-0-0-0-cr01.350ecermak.il.ibone.comcast.net
hu-2-9-0-0-cr01.350ecermak.il.ibone.comcast.net
hu-3-1-0-0-cr01.350ecermak.il.ibone.comcast.net
hu-3-2-0-0-cr01.350ecermak.il.ibone.comcast.net
hu-0-5-0-5-cr02.350ecermak.il.ibone.comcast.net
8. be-10617-cr02.denver.co.ibone.comcast.net 0.0% 80 61.1 72.5 55.7 114.8 17.9
9. hu-0-3-0-0-cr01.denverqwest.co.ibone.comcast.net 0.0% 80 57.7 72.5 56.0 116.5 16.3
10. he-10-1-3-cr01.sunnyvale.ca.ibone.comcast.net 57.5% 80 33707 33794 32878 34487 450.7
11. be-10915-cr02.losangeles.ca.ibone.comcast.net 0.0% 80 82.7 96.8 79.6 155.1 18.0
12. he-0-0-0-0-pe01.losangeles.ca.ibone.comcast.net 0.0% 80 82.7 92.4 78.6 134.1 15.6
13. ae2.Comcast.lax.us.AS40676.net 0.0% 80 85.9 101.0 85.4 142.2 17.9
14. ???
15. hosted-by.iniz.com 0.0% 80 87.7 101.5 86.4 141.0 16.3
16. 2.164.161.107.iniz.com 1.2% 80 88.4 99.9 84.4 139.9 15.9
17. 162.217.248.97 0.0% 80 84.9 98.9 83.9 140.0 15.7

Russ · 2015-10-13 18:05:36

Lonewolf. I'm sorry that the laws of physics and the speed of light are inconveniencing you, but I don't think contacting comcast can have any effect on the constants of the universe.

The absolute straightline path for light from dallas to LA and back takes 13ms. I'd say after including the slower speed of light through a fiber, plus router delays 30ms is actually pretty good.

Anyway, as someone with a 4ms round trip delay to the server, I can assure you that the problem is not the network. Just watch the game timer during really bad lag. The core of the problem will become immediately apparent to you.

Archer! · 2015-10-14 03:16:28

This...

Russ wrote:

Anyway, as someone with a 4ms round trip delay to the server, I can assure you that the problem is not the network. Just watch the game timer during really bad lag. The core of the problem will become immediately apparent to you.

but try not to sound so smug about it you localhost haxx0r...

Russ · 2015-10-14 04:57:36

Archer! wrote:

This...
Russ wrote:
Anyway, as someone with a 4ms round trip delay to the server, I can assure you that the problem is not the network. Just watch the game timer during really bad lag. The core of the problem will become immediately apparent to you.
but try not to sound so smug about it you localhost haxx0r...

https://www.youtube.com/watch?v=qQEIRAnNY4o

lonewolf-mcquade · 2015-10-16 01:48:47

Russ - There has never been a complaint about the average response time at the nodes. So I'm not sure what 30ms responses or your astonishingly fast 4ms round-trip have to do with anything, or why the tone of your comment is even necessary. I'm trying to be helpful (and show gratitude) to SiMPLE. While this is what I do professionally, it doesn't make me a genius at anything. But, it also doesn't require a PHD in physics to simply read what was already posted...

lonewolf-mcquade wrote:

Thanks trench. The anomalies I found were not present in the tracert I provided above. That was more just a mapping of how my data travels. When I ran an extended ping to each node, I had a reoccurring and discernible spike from the destination down to node 8 (working backwards.)From 7 to me, it was as "clean as a whistle" using the same standard.

lonewolf-mcquade wrote:

If, with further exploration, Comcast is pinpointed

lonewolf-mcquade wrote:

Remember, my comments are **preliminary.** It would be best to compare results with a few other users before proceeding with any course of action.

Extended pings on every Comcast node listed from the destination through hop 8 revealed huge inconsistent lag bursts (600+ms in some cases) Those spikes could potentially be a cause for performance issues as they appeared to be more frequent than I'm used to seeing on commercial paths of similar complexity. It would be interesting to get several people with different routes together to run the same extended pings and see the results. Additionally several admin have stated in TS and server chat that the server had all but been eliminated as the source of the problem by the host. I'd just like to see the problem solved for the benefit of the server, and I'm really not interested in a pocket-protector pissing match about it. If you have something better to suggest, please?

Last edited by lonewolf-mcquade (2015-10-16 01:57:22)

Russ · 2015-10-16 03:22:39

lonewolf-mcquade wrote:

Extended pings on every Comcast node listed from the destination through hop 8 revealed huge inconsistent lag bursts (600+ms in some cases) Those spikes could potentially be a cause for performance issues as they appeared to be more frequent than I'm used to seeing on commercial paths of similar complexity. It would be interesting to get several people with different routes together to run the same extended pings and see the results. Additionally several admin have stated in TS and server chat that the server had all but been eliminated as the source of the problem by the host. I'd just like to see the problem solved for the benefit of the server, and I'm really not interested in a pocket-protector pissing match about it. If you have something better to suggest, please?

Pinging individual hops doesn't give you an accurate picture unfortunately. The way icmp packets are handled too and from that one hop can vary significantly with how other packets get handled and traversed along the way.

The 30 ms referred to the increase in delay between the two hops.

And as I stated, the server is screwed up and it has nothing to do with the connection. Today on Battle of Britain on three separate occasions, the server hung long enough for the timer to jump backwards a full 10 seconds.

lonewolf-mcquade · 2015-10-16 03:40:10

I brought up the server initially and that idea was vigorously shot down by several different admin. Network latency can potentially affect game clock representation. While ICMP can be handled differently, with some devices discarding it alltogether, this is one of the tools we have. It's why I suggested it would be best to get other sources to get a better idea if it's an indicator of an issue. If someone has server access and can run wireshark that would be ideal (to eliminate several possibilities,) but after several conversations in TS and gamechat, no one seems to be able to do this. Some folks claim that the host company has looked at the server, can this be verified??

Last edited by lonewolf-mcquade (2015-10-16 03:44:08)

Russ · 2015-10-16 05:34:20

Ya, it can effect stuff. But having a very good connecting to the server, you really start to notice the very odd behavior and how it differs from actual network issues. The game will repeat certain sequences of events, and it gets really clear when you're in a plane. You'll be headed towards some building, or the ground, expecting to pull up, but you'll get a connection issues popup. And you know, in the next few hundred milliseconds, you will crash. But the connection issues thing goes away, and instead of being a pile of aircraft, you're actually backed up away from the obstacle and have a chance to control the plane, just as if nothing had happened. Depending on the severity, this can actually repeat a few times before clearing.

As far as ICMP being handling differently, it isn't necessarily an issue of being discarded, it's an issue of being queued differently. Likely locally generated packets (ping replies) go into a different queue than forwarded packets. For instance, here's my ping statistics on the server:

100 packets transmitted, 100 received, 0% packet loss, time 99132ms
rtt min/avg/max/mdev = 2.951/3.461/5.539/0.553 ms

And some of the hops:

103 packets transmitted, 103 received, 0% packet loss, time 102157ms
rtt min/avg/max/mdev = 3.323/4.434/11.244/1.097 ms

101 packets transmitted, 101 received, 0% packet loss, time 100135ms
rtt min/avg/max/mdev = 4.744/9.815/86.941/11.787 ms

103 packets transmitted, 103 received, 0% packet loss, time 102153ms
rtt min/avg/max/mdev = 3.665/4.476/13.877/1.456 ms

106 packets transmitted, 106 received, 0% packet loss, time 105170ms
rtt min/avg/max/mdev = 2.417/4.458/6.608/1.152 ms

As you can see, all of them have worst average ping than the server, with the main difference being that the variance in RTT's is higher. One of them even had an 87ms RTT on one of the pings. But none of this is actually reflected in my connection with the server.

ATF_SurrenderMonkey · 2015-10-16 05:37:50

wtf...I thought English was mandatory in this blog.....

bud · 2015-10-16 19:01:23

Russ wrote:

As you can see, all of them have worst average ping than the server, with the main difference being that the variance in RTT's is higher. One of them even had an 87ms RTT on one of the pings. But none of this is actually reflected in my connection with the server.

I dont think the Echo Request is very high in priority, some routers doesnt even respond to it. Like hop 9 in the picture.

hop nine

-------

From my point of view there is no point letting the server getting clogged down by it either, you play with UDP packets after all and not ICMP:

-A INPUT -p icmp -m icmp --icmp-type 8 -m limit --limit 9/sec --limit-burst 1 -j ACCEPT

lonewolf-mcquade · 2015-10-16 19:39:02

Last night I didn't experience the lag problem and the spike that I noticed on those hops went away during my play time. I would agree the spikes I saw are likely not the cause, but they may be another symptom that can get us closer to a resolution.

Considering the strong comments I received from multiple people that it wasn't the server, Russ' comments, and the fact that the problem is still happening, we still can't rule out anything. Can anyone in SiMPLE verify personally that the host company was contacted and has looked at the host environment? Can anyone wireshark to the server? Those answers would seem to take a lot of the mystery from this situation.

Last edited by lonewolf-mcquade (2015-10-16 19:39:40)

lonewolf-mcquade · 2015-10-16 23:45:05

I'm stepping away from this unless I'm needed. The conversation is going and thanks to those that posted. Ultimately, I hope it's helpful to the server.

[ECGN]BladeRunner · 2015-10-16 23:49:50

i played for a while last night and I didn't notice any skips or jerks in the game.
Maybe it's getting better on its on. :beer:

ATF_SurrenderMonkey · 2015-10-17 01:13:43

Thanks LW

Winston Smith · 2015-10-17 08:10:40

[ECGN]BladeRunner wrote:

i played for a while last night and I didn't notice any skips or jerks in the game.
Maybe it's getting better on its on. :beer:

I tend to get better when I'm on beer, but after too many I start lagging and hopping too.

SiMPLE | Forum

Announcement

#1 2015-10-11 23:17:14

SiMPLE USA Lag WorkGroup?

#2 2015-10-12 01:04:43

Re: SiMPLE USA Lag WorkGroup?

#3 2015-10-12 18:42:34

Re: SiMPLE USA Lag WorkGroup?

#4 2015-10-13 00:09:30

Re: SiMPLE USA Lag WorkGroup?

#5 2015-10-13 01:31:13

Re: SiMPLE USA Lag WorkGroup?

#6 2015-10-13 03:08:02

Re: SiMPLE USA Lag WorkGroup?

#7 2015-10-13 18:05:36

Re: SiMPLE USA Lag WorkGroup?

#8 2015-10-14 03:16:28

Re: SiMPLE USA Lag WorkGroup?

#9 2015-10-14 04:57:36

Re: SiMPLE USA Lag WorkGroup?

#10 2015-10-16 01:48:47

Re: SiMPLE USA Lag WorkGroup?

#11 2015-10-16 03:22:39

Re: SiMPLE USA Lag WorkGroup?

#12 2015-10-16 03:40:10

Re: SiMPLE USA Lag WorkGroup?

#13 2015-10-16 05:34:20

Re: SiMPLE USA Lag WorkGroup?

#14 2015-10-16 05:37:50

Re: SiMPLE USA Lag WorkGroup?

#15 2015-10-16 19:01:23

Re: SiMPLE USA Lag WorkGroup?

#16 2015-10-16 19:39:02

Re: SiMPLE USA Lag WorkGroup?

#17 2015-10-16 23:45:05

Re: SiMPLE USA Lag WorkGroup?

#18 2015-10-16 23:49:50

Re: SiMPLE USA Lag WorkGroup?

#19 2015-10-17 01:13:43

Re: SiMPLE USA Lag WorkGroup?

#20 2015-10-17 08:10:40

Re: SiMPLE USA Lag WorkGroup?

Board footer