Server network connection upgrade info

[10001110101]

One of the best discussions about maxpackets I ever read was at the ax forums. The post is still there but some of the links might be dead... you might track down cached versions somewhere though if you are interested.

http://www.anarchic-x.com/forum/viewtop ... 16&t=34249

[Borch]

I have turned off antilag as an experiment in reducing lag (and turned it back on, did not seem to be noticed much except by Europeans, and negatively).

[Borch]

I have analyzed the server quite thoroughly and see no problems. I have run MTR and see no problems. I have verified that there is no swapping going on, I have verified that the processor is not overloaded.

I have captured internet traffic and see no problems.

If the server were malfunctioning, everyone would see the same error at the same time, this is not happening, or only very infrequently.

Perhaps 3.0 is an issue, but I had noticed that 2.9 had issues as well before, so maybe the mod is just not great or is overloaded. I do not have the tools, time, or inclination to try to analyze the code in the mod.

I have posted queries to patmansan.duckdns.org to try to get his input, he largely seems to ignore me, probably because he has no answers or ideas.

The server feels robust to me, but perhaps folks have trouble reaching Chicago. The NFO server group has more capacity than any others, I am told, but if there is overloading on the internet going on, it is possible that the path by which more people are having to contend is worse just because of the routing to that destination.

My suggestion is that anyone who feels that they are laggy, run MTR. In all instances so far, I've found MTR to be a good identifier of why people are having lag. I have run the app, as I can, from the server to the clients, but usually pings are not allowed on private residence modem/routers.

If you find servers that run better, identify them, it is possible we could "dumb down" our server to match some of those.

[AIAKnight]

Hi Borch - thanks for doing all those checks.

Regarding the "same error at the same time", I would say that a large number of people on the server are still feeling the same spikes at the same time. I know that multiple people are feeling them because when it happens someone will say "lag spike??", and multiple others will agree they just felt it. As Auburn mentioned to you, it seems to primarily happen during key events (flag pickup/drop, grenade explosion, airstrike, etc.). From a lagometer perspective (which I now have on when I play), it will be smooth consistently smooth and then the spike will be a big yellow bar for about half a second, then back to smooth. Last night of Tobruk we had roughly 6 vs 6 and we started counting in the chat the big spikes and got up to around 15 by the end of the game.

I see you've been playing around with the max packets, anti-lag, etc., but the server does not appear to have improved over the past week, so my suggestion is to now move it back to Atlanta. If you don't want to move the server, we could redirect everyone to my NFO server in Atlanta for a week or two to test it out, see if its any better. I recognize that NFO says Chicago is their best location, but maybe Atlanta has better routing for most of us.

I haven't felt these issues on any of the other servers I play on: DB Tactical, H4X, XI (Extreme Idiots), OUTLAW (Outlaw gets rare spikes, but also always has 40 people on)

[Borch]

I looked at the packets I captured over a 30 minute period on July 8.

These were the results:
Number of packets sent by the client, client IP, client name and finally, occurrences of a large gap between transmissions from the client (>100 ms)

83376 ..........71-219-62-236.chvl.qwest.net.28960 ..........eventhorizon.......... 29
91216 ..........pool-173-54-214-237.nwrknj.fios.verizon.net.28960 ..........knight ..........13
79147 ..........ip72-192-84-97.tu.ok.cox.net.28960 ..........tulsa ..........15
64945 .......... 75-44-127-128.lightspeed.mmphtn.sbcglobal.net.28960 ..........shaka ..........13
90948 .......... c-174-54-29-246.hsd1.pa.comcast.net.28960 jigsaw ..........8
58035 .......... 47-220-56-63.kngwcmta02.res.dyn.suddenlink.net.28960 .......... AmareFan1 ..........6
32446 .......... 76-250-48-194.lightspeed.lnngmi.sbcglobal.net.28960 .......... Borch ..........0
91061 .......... 99-89-115-163.lightspeed.frokca.sbcglobal.net.28960 .......... Nature ..........13
43361 .......... 136.35.188.118.28960 ..........Jagged ..........9
56694 .......... c-73-203-170-156.hsd1.la.comcast.net.28960.......... Mama ..........124
73205 .......... c-98-227-141-234.hsd1.il.comcast.net.28960.......... drillski ..........10
89766 .......... c-73-88-56-186.hsd1.mn.comcast.net.28960 .......... stepsoftly ..........9
13865 .......... 97-124-231-87.phnx.qwest.net.28960 ..........bobert ..........3
62261 .......... bras-base-lchtpq0202w-grc-03-70-49-80-68.dsl.bell.ca.28960 .......... gunner ..........161
72302.......... ool-45725e5b.dyn.optonline.net.28960.......... deadman ..........11
89697 .......... pool-108-30-185-124.nycmny.fios.verizon.net.28960 ..........glove .......... 13
91187 .......... d24-36-249-152.home1.cgocable.net.28960 ..........bananas ..........13

Unless there is something pretty obscure going on with the server it would seem to be network related. Nothing was changed in the move from Atlanta except the scripts where the IP was identified, along with system's IP (it the system IP was not changed it wouldn't do anything). Note that the time when we moved the server we did so to try to improve lagginess, so we were already feeling it at that time.

Note that during this period of time I have totally rebuilt the server, installed and run a server that was stock, analyzed the processor usage, verified that the programs were running properly (I did find a minor glitch that would have killed the server if it were to affect it, but didn't seem to), have run MTR on multiple clients from the server and had some submit their results (bananas in particular finally convinced his IP to fix his connection on that basis), have analyzed data packet transmittal and size, and after all that, the only thing I have found is that the internet has lots of bad connection issues.

I have no explanation about the glitch that occurs when a flag is grabbed, which would tend to point to an issue with the extreme mod, but being able to analyze game loading due to the extreme mod is out of my pay grade.

Considering that most of the people that complain about the lag don't seem to want to try to help (not running MTR) I have exhausted the avenues I can explore.

So we have two things that can be done:
1. Install the Extreme 2.9 mod, but considering we saw the problem with a stock installation of cod2, I don't believe that will fix much.
2. Go back to atlanta and hope that the pathway to the server is better for most. Note the variation in the occurrences of lost packets above from various clients--if it were the server, it would seem to be across the board.
3. Find a different mod, or go to stock with tweaked settings.

I have exhausted the amount of time that I wish to put into this. I have the ability to change the server IP and scripts, but this is getting in the way of my life and frankly, it is really f...ing annoying to hear everyone complaining about the lag all the time.

I have moved all the script alterations into an "ohmychanges" file, so perhaps I could try making that an empty file, making the game mod the base version, perhaps we are using up too much memory with all our modifications that we have installed. After I wrote that, I tried importing the ohmychanges a little at a time, and nowhere along that test path have I seen any glitches when taking the enemy flag. Of course, I was the only player. The possibility of having a bunch of players on who would stomach taking the time to test it over a 20 minute period is pretty close to nil.


The other option is we could shut down the server until covid is gone.

[Borch]

Yes, Keith, there are writeups talking about how to balance maxpackets with the server/client fps, but I was seeking to reduce the load on the server network, not to optimize the cl_maxpackets setting.

[Auburnman]

Borch even though we haven't found the root cause yet, all the work you've done is extremely helpful in that you've eliminated a lot of the usual suspects. My experience on the server is that the majority of lag spikes occur when an activity has occurred like a flag grab or a nade going off (or Knight entering the Matrix) and it's experienced by all the players at the same time. For this reason I don't think this is caused by good old fashioned internet latency where each player has their own unique path, hops, etc to the server and that's probably why the MTR data isn't showing anything. Obviously the internet isn't perfect and those connection issues still rear their ugly head but I don't think that's the source of our common, frequent and seemingly event-driven lag spikes. Any connectivity issues a player has will be in addition to whatever is going on with the server and will be seen only by that particular player. Again...the number and consistency of the event driven spikes feels to me to be sourced at the server either by the hardware or the program instance itself.

You've also eliminated or greatly reduced the likelihood of bumping up against resource limits (cpu, disk access....) at least to the extent it can be captured in the logs. I don't know how often the data is written to the log files but I guess it's possible a short duration "spike" could be missed. I think it was Spock that said instruments will record only what they're designed to record. Even though all the dials and gages show normal and the status lights are green, actual experience indicates something is amiss.

My own sense, for whatever it's worth, is that parts of the install aren't playing well together. Maybe there are settings fighting or in conflict with each other or maybe something capable of interrupting the game itself is competing for cpu cycles. Both the game settings and the mod settings would have to be considered. Is there anything equivalent to the Mac's "Activity Monitor" or a way to record high resolution resource usage for a few maps so we can see what's actually happening during a spike?

That short survey I did of other "smooth" servers shows a mixture of extreme mod versions, antilag states and server locations making me doubtful that simply rolling back a mod or movings the server as-is will fix things. There's something going on unique to us and I'd love to know what it is. Borch I know you've spent a lot of time on this and you're getting tired of messing with it but I'll do all I can to help. You giving me access to the file structure will help me understand things better and I started getting the lay of the land last night. I'm sure other will be willing to help as well. If you don't want to mess with it anymore at all, we can figure something out. Maybe giving me time to have a look and do some research will give you a break although I'm sure I'd need to consult with you occasionally.

Given all that I think our realistic choices are to either live with it as it is or do a redirect to Knight's server for the short term while we try to figure out what's going on with our current server. My preference is the latter. It may (or may not) mean foregoing a weekly map change or two but given Knight's server is windows based, there are others (Binary, Jagged et al?) that may be willing to help with some of the server care and feeding.

This is all just my opinion and I may be wrong about everything I said but I've been thinking about it a lot and this is where I am right now.

[Borch]

I turned off stuff to prevent the possibility of server loading due to the processor load testing.
Basically it was every 1 second when I run top. It would be virtually impossible for glitches to totally be ignored. I have observed rare high spikes for individual samples for years.

Look at /home/cod2/logs/process_cpu for a sample of some of the logging I have done.
first column is processor usage, second is map, third is time since the epoch, fourth is game time, act is the number of active players, inact the number of spectators,
ttl is total, and then the date/time.

I stopped it on July 3, just restarted it now.

The MTRs that I have gotten from people have shown significant issues, which is why there is so much variation in the messages I posted earlier.

I do believe that there is something amiss with the server. Part of me is beginning to wonder if it has any relation to the shared computer we have. There is software that manages multiple clients using the same computer system, (though we have a pretty good chunk at the moment, so dunno how that works).

[Borch]

The server is running 2.9.

[Borch]

Installed map peculiar settings file that were created by shart from the v3 server.
Installed the weapons file from the v3 server (has the knife fix).
Removed the ban.txt file, since goofy and perhaps glove were in it, but nothing was labeled, so just removed them all. Note that we have had two different systems for banning, b3 and the game. Probably better to ban in b3. Automatic banning in the game was not turned on, so all the bans there were done manually.