What went wrong during the FTX on 14FEB - Statement

[RET J.Drake]

Yesterday's FTX was a rough ride for those who run and oversee the missions, so we'd like to give a glimpse on the issues that lead to yesterday's events. Here's a timeline of what happened and the investigation we did afterwards.

Events

We started off our FTX on a new server, moments after loading in, we realized that Zeus was acting strange. We couldn't place any hostile units anymore.
Realizing it was a pretty severe issue, we decide to swap back to our old operation server on the windows dedicated box and run the mission on there.
Everything seemed fine until suddenly the Zeus issue came back, but we could still place down independent units, so we just stuck with that.
Then something in the mission caused the server to lock up, running at ~50% CPU (2/4 cores at 100%) and RAM usage spiked to 20+ GB.

While the server hung, we decided to swap back to the new server, which still had the mission file and we continued on there. Everyone used a teleporter to go to our last known location and continued on.
Zeus seemed to work again, however, later on, the same issue came back and we were once again unable to place down enemy units.
We decided to just roll with it, eventually even using 3CAB to remote control blufor units, since we could still place those down. Changing the side of blufor units also did not work.
The new server seemed to hold up fine but there was still a noticeable performance issue (server running only at 7-10 ServerFPS).

Investigation

So we encountered 2 massive issues during this mission.

1. Why could we no longer spawn hostile units?
2. Why did the server suddenly go into memory dump mode and completely lock up?

Turns out, both were related! And both were caused by these fuckers:

1. Why we couldn't spawn hostile units:

Turns out this is an Arma/Zeus bug. Whenever these SAM sites are present (our server/mod setup may or may not influence this) and you use the Zeus function to "change side relations" it then stops Zeus from placing down units from that nation. We had to change the relations to turn greenfor hostile to blufor and greenfor friendly with opfor. After checking the server logs, nothing seems to indicate any errors related to that event.

(credit to @1LT B.Lambert, @SGT D.Sklenar and @WO1 R.Paxton for finding this one)

2. The server lockup

When the SAM site engaged a friendly drone. About 5 missiles were launched. 1 of them was enough to take out the drone, however, it seems like once the drone was destroyed, all the other missiles just stopped and hung mid-air. We suspect that the missiles were no longer able to find its target which caused massive slowdowns. We have this event and the minutes that follow it on video too. Pay attention to the monitor debug messages on the bottom left too and see server FPS drop and Memory usage spike until no more monitor messages are coming through at all.

(at the moment of writing, the video is still processing, shouldn't take too long to be available though)

Conclusion

So, how do we go forward? Well, first and foremost, we will no longer be using this MIM-145 Defender SAM launcher. Additionally, we've identified a common pattern where mission files are submitted very last-minute, so starting next FTX, mission files will have to be submitted on Saturday 2359 EST. That allows us time on Sunday to start the server, test out some basic functions, and possibly catch these errors before they occur.

Our apologies if this ruined any gameplay for anyone, it was a very hectic event for all event hosts, COE and BMO. Despite these technical difficulties we hope everyone still had a great time and we'll be more rigorous in our mission testing for any future missions. With Zeusing and dynamic missions there's always cases that we can't cover. Arma is known to be quite buggy, so we simply couldn't have predicted this and I strongly believe nobody is to blame for this. We continue to do our best to host fun events and give everyone plenty of MilSim goodness!

Signed,
1SG J.Drake,
COE NCOIC.

[RET A.Pendergast]

Nice analysis ! And good job for finding the issue so quickly.

Personally, I still had a lot of fun during the FTX, the objective was good and the enemies challenging. And with all those technical issue, the staff and Zeus were able to deliver a very enjoyable experience overall, even more knowing everything behind the scene.

So good job overall, and thank you for keeping us updated !

[MAJ J.LaFlash]

Hooah, Thanks COE and BMO for all you!  Awesome job finding the issues and being transparent so that it can be explained to the masses of the unit!