User Tools

Site Tools


en:games:star_trek_armada_1:de-sync_and_crash_test

De-Sync and Crash Test

This page is dedicated to keep records of tests revolving around the IPX server use, de-syncs and crashes when playing Star Trek: Armada.

De-Sync

What is it?

This term stands for de-synchronization of game states. These are most likely either problems with communication (tracking game states across all players games via P2P connection) or random occurrences not taken care of properly by the game. The phenomenon shows it self quite plainly by the game stating having discovered discrepancies of the game states of the different peers/players. (And advising to stop the game, as in essence the different players are already playing different matches.) The game discovers de-syncs by having each player's game simulation generate a CRC of the game state and send it to the other players. Of the CRC is found to differ, then from the game's perspective the reality are not synced any more. This can be more or less accurate. The next section will describe this in more detail.

True De-Syncs, Pseudo-De-Syncs and Pseudo-Syncs

Pseudo-De-Sync

It could be observed, that there are cases where the game discovers a kind of temporary de-sync, that only technically looks like the game went de-sync but is recovered soon after. Unless actual de-syncs come into play (being a problem on their on already), these cases are pseudo-de-sync, because the game state is eventually the same again shortly after.

One way of provoking this is triggering of a movement or selection spamming. While the latter is difficult to provoke on purpose but happens, the former can be provoked by doing the following thing:

Send a unit via right-click and hold the right mouse button. It does not always happen, but often you can see the game start showing the green send-confirmation-cross under the mouse pointer in rapid succession. It then appears a bit thicker and brighter if your mouse pointer is stationary or drags a tail of click even confirmation animations after it:

 Click spam triggered. You can clearly see the tail of click event confirmations here.

The normal click event confirmation on the other hand looks like this:

 Normal click event. The green cross appears just once.

The spamming of such a command continues until you release the mouse button. Sometimes this starts right away when clicking (the most common form probably, because it can happen during any selection or movement command, where you click only shortly but the issue comes up anyway). Sometimes it does not start at all, even after holding the mouse button for a considerably duration.

In any case, these instances cause the internal game mechanism to queue up the given commands and CRCs in a buffer as the network communication frames have a limited space for sending. That means, not all these commands can be sent out in as rapid of a succession as they happen, basically piling up in the buffer, waiting to be sent to the other players. This includes CRCs basically being delays by that issue. During such a time, when this buffer is still filled, the game's internal CRC mechanism will come to the same conclusion for the player having triggered as the other players. But because the CRCs go out delayed, they may temporary mismatch until the queue has catched up with the other players as well. Once every player received the entire list (the buffer is now empty), the game CRCs are up to date for everyone and therefore the same again. Only the buffering of commands creates the short duration during which the game thinks it went de-sync. After that the messages disappear again and everything works as intended.

De-Syncs

Actual de-syncs, where the game states persistently differ, are what the game actually intends to warn about and what the CRC mechanism is meant to discover. This means, ships are in different positions, leading also to orders being given, that cannot be followed in one game state but in another, entity IDs are different, unit orientations are different, resources are different, etc…

These situations usually do not recover any more, because the deviations propagate throughout the different simulations. E.g. in one simulation the game allows for already firing weapons, because source and target are already close enough, in another it does not. Subsequently, a unit may be destroyed already in one game state, while it survives in another. Such a propagation can cause massively different game states in a rather short amount of time. This is also why the game advises players to leave the session. Because there is basically no point in continuing such a situation.

Identifying the Different Situations

Instances of pseudo-de-syncs (where everything converges shorty again) must be separated from actual de-syncs, where the game state does not recover any more. One easy distinction are the cases where the game becomes visibly different for some players, as described in the last subsection. As the game never states the fact that it recovered sync (it only ever gives out notifications when it considers game states to be de-sync and for as long as that persists), it is easy to be mistaken for a real de-syncs (especially if you terminate the session immediately, as the game advises you).

The main observable behavior for pseudo-de-syncs is basically, that the de-sync warning stops appearing shortly or immediately after the first occurrence. If the de-synced state persists, the warning will be repeated once every two minutes until the end of the match. IF the state has also nominally recovered, the message is not triggered again. So in short, if you do not get a de-sync message once every two minutes, even if you got one at some point, it means that the game is (most likely) in fact in sync.

Pseudo-Syncs

Most likely, because there were also cases observed, where the game does not discover the game state being de-sync already, because the CRCs of the different players don't differ, yet, while some of the game different states aspects are provable different already, e.g. by logging certain information.) Those are essentially pseudo-synced states. They are the exact opposite of the pseudo-de-syncs, as the game will also eventually discover that the game states are not synced any more and start triggering the de-sync message. In those instances it is only a matter of time when that starts (when the states are sufficiently different so that also the CRC is now affected). Those instances are difficult to spot at first but will be discovered eventually, when the de-sync-messages start happening.

Effects

Over time these discrepancies can amount to vastly different game states, up to the point where one player may already finish the match by destroying all opposing units and stations, while for another the game continues very much, as there are still units or stations left. This is also the reason why the game advices to stop the match. There is actually little point in continuing, as all players are potentially already in essentially their own single-player games.

Suspected Causes of De-Syncs

Over time people suspected a number of reasons why a game may go de-sync. Most of them are basically only anecdotal in nature:

  • Using Transwarp Drive: It was reported, that making heavy use of Transwarp Drive may also produce de-syncs. These reports could not be reproduced yet (see Test Results De-Sync). But they may very well be the root cause for de-syncs, if there are some irregularities how the game places transferred units. If there is enough discrepancy in that, it may very well make units go different paths on the map or come in contact with units in one game state, whereas this does not happen in another.
  • Using super weapons: This one could not be confirmed, yet. While there is strong evidence, that using the Transwarp Gate in games with AI players causes Crashes, de-syncs are not a problem of any of the available super weapons.
  • Different OS: This one has not been tested systematically, yet. The suspicion is, that when using Windows 10 and Windows 11 as operating system, matches based on both OS in the player base tend to go de-sync. Other combinations, where it seems like de-syncs become more likely, are Windows-Linux combinations (playing Armada under Wine).
  • Presence of file ATVI in the game's folder (either in general, or when placed there only for a fraction of players having it): This is a means for turning on debugging. However, there hasn't been any conclusive evidence yet, that this changes anything until de-syncs happen already (or at least the game things there are some). Once a de-sync happens (or the game things one has), the game pulls a new random value when creating the sync dump. If not all players do that (some have the file, some don't), it means their game is now permanently out of sync because their simulations will from now on use different random values, leading to different outcomes.

Confirmed Causes of De-Syncs

De-Construction of Ships

This particular cause could be confirmed, specifically for ships that follow the Borg movement physics. In the stock game those are:

This was observed wore than once, after de-constructing a class ship, Spheres and Cubes. In one game state the unit still sat at the last location prior to de-construction while in another it was properly deconstructed (at the site of the player controlling the unit it persisted).

One match was conducted with ATVI file in place, where the economy was basically based on assimilating opposing ships and building more Cubes from it. The match was carried out via IPX and went on for 1:17 h. 2 Humans were involved, as well as 2 medium AIs. Systematic reproduction of the problem seems difficult.

Another was basically attempting to provoke the problem with Spheres, where it could quickly be shown, that this does indeed cause de-syncs.

Further investigation code-wise revealed a systematic flaw in the game logic: For some reason does the game rotate the units in a random but unsynced fashion. Meaning in one reality it may turn further than in another: When sending a ship somewhere (and be that to deconstruction in a yard) it will turn by 0 to 270 degrees in 90 degree units. Problem with that is, that the exact number is unsynced in the unmodified game, which means, it can depend heavily in what orientation a ship arrives at the yard for deconstruction. That in turn influences how much time passes until the ship eventually is threaded into the yard queue. Especially when it is the only/next unit in line (so the variance in arrival time is not compensated by having to wait for another ship to be processed) this obviously influences the exact moment the deconstruction starts and finishes.

This brings four problem with it:

  • The mere issuing of the decommission command already turns the ship in a non-synced fashion. That does not necessarily cause huge problems, but can already. Depending on the following occurrences the ship for example might on one game state be able to fire (or be fired upon), while that is not the case in another.
  • The decommission itself can throw off entity ID numbers. In one world the ID is freed up a bit earlier than in another and possibly re-used earlier than in another.
  • The deconstruction may be prevented in one world (because there was still enough time left to do so) while it cannot be avoided any longer in the other (because it has already started). So if a player stops the deconstruction command form going through, the existence of the ship may depends entirely on the duration that was left for halting the deconstruction command.
  • And the last problem comes right after the finalization of the deconstruction: The resources freed up by it may be used for the next construction, making production of a new unit start possibly earlier in one reality than in another, or depending on whether the process was stopped, may prevent another ship from being build entirely.

A possible fix here would be to only use synced randoms or make the pattern of turning systematic.

One idea for the latter approach: Make the turning angle switch sign with each new command of movement given and pick a modulo 4 direction starting the initial direction by its entity ID. Unless the entity IDs are already not synced any more, the movement would appear to be random (unless you pay close attention to the pattern) but still be very much systematic. As each command is also synced (you know the number of move commands issued implicitly), this should never de-sync.

Test Results De-Sync

This one has so far only one systematic test to show for, regarding the Transwarp Drive.

De-Syncs by Transwarp Drive

Even when transferring 8 groups of 8 Interceptors one directly after another, the game went on stably. Also re-doing that over and over did not cause problems in a test match with player Defiant.

Also in other matches when making not as excessive use of them, the matches went on just fine.

Recommendation

This definitely needs further testing involving other players, that have reported the problem (notably Michael). Maybe the actual root cause is something differently, e.g. slightly altered games, or different OS being used.

Partial or General Presence of ATVI

The presence of the file is not systematically a problem. There are people, that claimed, that the mere presence of that file alone reliably leads to de-syncs. After having played consecutively 5 matches via IPX, 2 players, 2 medium AI every time, the results look like this:

Duration ATVI present Map Result
0:39 h no 4first Nothing out of the ordinary
0:50 h both 4first Nothing out of the ordinary
1:12 h both 4first Nothing out of the ordinary
1:17 h mixed (only one player) 4first Nothing out of the ordinary
0:32 h/1:12 h mixed 4mama De-sync after 0:32 h, continuation on the same map with the file worked out just fine for 30 minutes until the match concluded.
Recommendation Regarding ATVI

Apparently it does not reliably lead to de-syncs. The tests do not show any increased likelihood for de-syncs, but further testing is recommended, especially the mixed case.

Crashes

The game is known to crash for certain causes, that can occur random, but can also kind of be provoked. So there is a systematic component behind them. These crashes happen without an advanced warning or an error message. The game just suddenly shuts down, bringing the player back to their desktop.

Suspected Causes of Crashes

Here are some observations, of things probably or certainly causing crashes of the game:

  • ALT + TAB: One thing that very certainly causes crashes, but not in every instance, is switching to another application with this short cut. Sometimes it goes well, sometimes when trying to re-enter the game, it just crashes. Side-Note: While in the menus (e.g. setting up a match) using the short cut is very well known to also cause buttons to be blacked out. Meaning, the button exists very much, but is not shown any more, until the mouse pointer enters its surface. There is a way of making the game considerably more robust against that problem: Using the GOG Wrapper. The Presentation value set to Windowed.
  • Using Transwarp Gate: This one is a little bit peculiar, as it does not happen right away and not always. Once a player uses the TWG, the game still continues. But at some point it just crashes. This may also happen at the very moment a TWG is used, but it is not a necessity.
  • Using a lot of Ultritium Burst cast at the same time (e.g. an entire 8 Diamond squad) is reported to cause crashes, but not always.
  • Self-Destruct used after getting affected by Holo-Emitter is said to cause crashes (source is player Nic4747). At the moment the evidence does not support that, but there is a smaller chance that this is real.
  • Issuing alert status changes after getting affected by Holo-Emitter is said to cause crashes (source is player Nic4747). At the moment the evidence does not support that, but there is a smaller chance that this is real.
  • Changing the alert level while being affected by Holo-Emitter is also said to cause crashes (source is player Nic4747). This definitely requires some testing as changing alert level after being hit by Holo Emitter is actually a standard move, if the player still controls the affected units.

Test Results Crashes

ALT + TAB (Confirmed)

This one is very well confirmed.

Recommendation ALT + TAB

Do not use ALT + TAB during a running match, regardless whether it is an Instand Action Match or an actual multi-player match. And even when still in the menu, crashes can happen and it is a well known phenomenon that menu visibility gets messed up when tabbing back into the game.

Using Transwarp Gate (Semi-Confirmed)

Known Facts TWG

The Transwarp Gate is a bit of a mystery as of yet. When using it in multi-player games without an AI even after some 10 - 20 uses, the game went on stably until the end. A test match with player Defiant on map klnghell.bzn (Kling Hell by rih'goha) clearly showed that. After clearing out all team 0 units, TWG was used, many times, including other super weapons, such as the Phoenix.

But when using it in conjunction with the AI, regardless of being an Instant Action match, or an actual multi-player match, the game will crash, even after using the TWG only once. It may take a while, e.g. some 15 minutes, but eventually it will happen. This also means, if the match finishes very shortly after issuing TWG, you may not be hit by a crash. But when making heavy use of it or having a long match, probability is very high, the game will crash.

Hypothesis TWG

Obviously somehow the AI seems to be involved. Also it is very peculiar, that the AI never uses the TWG itself, even when having it built. Suspicion: The AI tries to use TWG points, that are no longer open. Technically the TWG points are very similar to worm holes, that are rather static in nature. In multi-player they do not suddenly occur or vanish. But a TWG point lasts not very long. If by any chance the AI does not remove TWG points from its known map objects, then it stands to reason it tries using them, long after they got removed, which may very well lead to crashes.

Recommendation TWG
  • Do not use it, at least not when having AI players participate in the match.
  • Further test matches on the map klnghell.bzn with AI players would be interesting. Maybe the problem is actually tied to maps, not whether the AI is present. Maybe it is the AI, so having it present on the map would be a benchmark.

Using Self-Destruct While Under the Influence of Holo-Emitter (Still Investigated)

Known Facts SD+HE

A test was conducted on map 2duel.bzn (On The Ready Line), eight Negh'Var ships were destroyed one by one, by using Self-Destruct. While the Self-Destruct was already under way, Holo-Emitter was cast by an opposing ship, leaving the Negh'Vars affected by and destroyed while so. The result was uneventful each time. The Self-Destruct took its normal course of action.

Another seemingly successful attempt was made by player nic4747, as shown in this video: https://www.twitch.tv/videos/2079696018 But this time it was Shadow vs. Cube and the Holo-Emitter was issued twice after issuing Self-Destruct.

Important notice: Self-Destruct could not be cast, while already under the influence of Holo-Emitter. So in a normal match, the order can only be like casting Self-Destruct and then being affected by Holo-Emitter. This leaves an interesting edge case that could be a race condition: If the Self-Destruct is issued at the same time while Holo-Emitter is cast, the player using Holo-Emitter might get the information a tad to late, so that from their perspective they issue Holo-Emitter and the opponent still casts Self-Destruct. Maybe this is the cause for the reports of that problem. But this would imply, that this is a rather rare event, as hitting this exact time frame, while casting Self-Desctruct may be a problem, is highly unlikely.

Recommendation Self-Destruct While Affected by Holo-Emitter

At this time, no recommendation can be given, except that maybe further testing is required. Test-Suggestion: A clean match, Shadow vs. Cube and the Holo-Emitter gets issued at least twice after issuing Self-Destruct on the Cubes.

Changing Alert Status While Under the Influence of Holo-Emitter (Still Investigated)

Known Facts AS+HE

A test was conducted on map 2duel.bzn (On The Ready Line), two different Negh'Var ships were used to change the alert status (into random directions, simply pressing the buttons in a random order) while Holo-Emitter was already affecting them. The result was uneventful each time. The alert status was simply changed and the match went on.

Recommendation Alert Status Change While Affected by Holo-Emitter

At this time, no recommendation can be give, except that maybe further testing is required.

Using Other Super Weapons (Rebutted)

Known Facts Super Weapons

While the Transwarp Gate most certainly causes crashes in certain circumstances, the others rather certainly do not. Test matches were conducted with player Capt.Cedaris(V), testing Temporal Stasis Field, Rift Creator and Shockwave. Neither of the three did cause a single crash when used or after, even after some 50 attempts in a single match and even when doing stress-tests by issuing many of them at the same time. The game just went on.

Recommendation Super Weapons

Just use them, except for the Transwarp Gate.

IPX Server

By default the game worked via three multi-player connections:

  1. The WON, which got shut down, so is not usable any longer.
  2. TCP, which only works on the same network, making VPN use mandatory for matches via the internet.
  3. IPX, similar to TCP but not available natively from Windows Vista on. See Using IPX.

There are currently efforts made to determine how reliable the IPX-via-UDP wrapping method in conjunction with Daniel Collins' wrapper and A Perl RFC 1234 IPX server is.

The desired long-term solution would be to convince GOG to run their own IPX server and deliver A1 with the latest version of the Wrapper and default-settings pointing to their server. That way no third-party tools and/or VPNs would be required any longer.

Current Test Results

  • Multiple 2-4 player matches were played, without any incidences with the unaltered variation of the Perl server.
  • At least two matches cross-Atlantic were conducted, with no obvious problems.
  • Something could also be observed via LAN already, is the game looking for open matches immediately when using IPX, while when using TCP the game seemingly skips the first attempt at looking for open matches, making them appear only after 40 seconds (the interval the game waits between attempts of finding open games). So essentially IPX allows for a faster discovery for open games.

Required Tests

  • Larger matches: At best with some 8 full players, to cause maximum load for the server and peers.
  • Multiple running matches in parallel.
  • Combination of the two: Having at least two full matches playing out.
  • Latency-Variation: Players of different origin, causing different latency times, e.g. a player in EU, one in US, and maybe even one in AU.
This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website.More information about cookies
en/games/star_trek_armada_1/de-sync_and_crash_test.txt · Last modified: by 7saturn

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki