[1.60] Frozen Galaxy - RAM hardware issue detected

[HBK]

All AI is frozen. My AI. Other AI too.

Nothing moves anymore except the ship I’m piloting. No build order is completed. Etc.

I think I pinpointed the save where it all started. When loading that save, I’m getting the starter tutorials (it’s a 20h save), but this may be unrelated.

I’ll upload the saves ASAP.

Edit: All AI have their orders, there just don’t move anymore. Also my pilot is stuck in its cabin and won’t take the helm anymore.

[HBK]

A save (autosave) which seems to be alright, minutes before the save which started it as far as I can tell :
https://www.dropbox.com/s/6y2ytmh7mg3ls ... ml.gz?dl=0

The save which when loaded shows some starting tutorials and where my pilot is stuck (and only a few AI ships are moving) :
https://www.dropbox.com/s/z79enysh1uj6g ... ml.gz?dl=0

A save a few hours later when I realized the issue (everything is utterly frozen) :
https://www.dropbox.com/s/osx01rxxtfbgq ... ml.gz?dl=0

--------------------------------

Game is 1.60 HF1 unmodified and pass steam integrity check. English language, "The Young Gun".

[Alan Phipps]

Tell us about your game please - source, version, mod state, etc. If from Steam, have you verified the gamefiles cache since 1.60 HF1?

[HBK]

Tell us about your game please - source, version, mod state, etc. If from Steam, have you verified the gamefiles cache since 1.60 HF1?

I edited the post above while you were answering

Steam 1.60 HF1, passes integrity check, English language, "The Young Gun" gamestart.

Game is unmodified, save shows unmodified.

[Xenon_Slayer]

Hi,

Well, I can confirm that the AI are no longer running any scripts and the mission director is also complaining, which usually points to a failure when saving the game. Do you happen to remember having a crash during saving at some point? If so, we may be able to get some clues from a crash dump.

There was a similar issue in the earlier versions but not only was the crash fixed, I believe we put some safeguards in place to prevent loading of incomplete saves. Seeing as this game was started in 1.60, it looks like we may need to look a bit closer into this.

For now, I can only suggest reverting to the earlier save. Sorry about that.

Owen

[HBK]

Hi,

Yeah I loaded up that "good" save and everything seems to be alright so far.

I wish I could tell you what happened but that’s the thing. Nothing happened. The save went okay and I exited the game without issue. No crash (so no crash dump), nothing.

Also aside from the gamebreaking issue the "broken" save loads fine. No error, no modified tag.

Sorry I can’t be of more help.

Cheers,

HBK

[Xenon_Slayer]

Nah, no need for you to apologise. We'll look for clues in the savegame to help detect these broken saves and stop them happening in the first place. Just glad you had something to fall back on.

[KlausM]

A save (autosave) which seems to be alright, minutes before the save which started it as far as I can tell :
https://www.dropbox.com/s/6y2ytmh7mg3ls ... ml.gz?dl=0

The save which when loaded shows some starting tutorials and where my pilot is stuck (and only a few AI ships are moving) :
https://www.dropbox.com/s/z79enysh1uj6g ... ml.gz?dl=0

A save a few hours later when I realized the issue (everything is utterly frozen) :
https://www.dropbox.com/s/osx01rxxtfbgq ... ml.gz?dl=0

Hi HBK,

apparently there's something weird going on on your machine, which affects the savegame content while it's constructed in RAM (before it's written to disk). Both the first and the second savegame have unexpected characters in the XML structure. The second savegame (and possibly others that you haven't uploaded) was damaged in a way that the XML parser cannot handle. As a result, the savegame gets loaded in a broken and unusable state, and there isn't any indication of a failure (except in the debug log). We're looking into improving that behaviour.

We don't know the cause of the corruption. It could be a RAM defect that causes random bit flips in memory - please try running the Windows Memory Diagnostic tool.

Unfortunately it's impossible to recover from the second savegame and all games that were saved afterwards. Although there's an error also in the first savegame, it appears to be harmless, so you could continue the game from there, once the underlying cause of the corruptions is identified and fixed.

[HBK]

Hi KlausM,

Thanks for the reply. I'll run diagnostics tonight though I doubt it'll detect anything (it's a rock stable not-so-new machine, but your never know ...)

Now that you mention it, I run a French Windows 10 OS, could those unexpected characters come from that? Like messy accentuated characters in the XML or something?

Also, do we have access to the debug log ourselves, or is it limited to the debug version of the game?

If possible I'll test some saving/loading and look into those debug logs, especially with my latest saves which seem to be alright so far (at least I see no gamebreaking issue when playing).

I'll come back to you with the diagnostics results (maybe there is an hardware fault after all).

Best regards,
HBK

[KlausM]

Now that you mention it, I run a French Windows 10 OS, could those unexpected characters come from that? Like messy accentuated characters in the XML or something?

No, there were expected characters in the savegame format that got replaced with invalid ones, which shouldn't originate from the game or the XML library.

Also, do we have access to the debug log ourselves, or is it limited to the debug version of the game?

You can use these command-line parameters to write error messages to a file in your personal folder:
-logfile debuglog.txt

The errors are also available in the in-game debug log, which is very similar to the one in XR (see instructions here).

[HBK]

I can't find the report but Windows Memory Diagnostics found no issue according to the notification tray (I was also watching it as it progressed and showed no error).

To be sure, I'm currently running a Prime95 stress test, but so far it's all OK.

I'll now try to see what the game debug logs can tell me (it seems -logfile mylog.txt does the trick).

Edit: We crossed posts it seems. Thanks, I'll look into that.

[HBK]

Ok so this is from an autosave from yesterday playtime :

Code: Select all

[General] 194620.37 ======================================
[=ERROR=] 194620.37 Failed to import formation leader node. Some formations will likely be broken.
[General] 194620.37 ======================================
[General] 194620.37 ======================================
[=ERROR=] 194620.37 Error on AI director import: Could not find entity [0xc8ae3b9]
[General] 194620.37 ======================================

And this is from a save I did about eight minutes later :

Code: Select all

[General] 195383.77 ======================================
[=ERROR=] 195383.77 Error on AI director import: Could not find entity [0xc8ae3b9]
[General] 195383.77 ======================================

Then ten minutes after the autosave (those are the last saves I made yesterday before quitting) :

Code: Select all

[General] 195445.15 ======================================
[=ERROR=] 195445.15 Failed to import formation leader node. Some formations will likely be broken.
[General] 195445.15 ======================================
[General] 195445.15 ======================================
[=ERROR=] 195445.15 Error on AI director import: Could not find entity [0xc8ae3b9]
[General] 195445.15 ======================================

Note: It seems for some reasons I don't always get the "Failed to import formation leader node." though I loaded those saves multiple times (due to a firefox script crash ...)

Then this is what happens when I load that save, save it, and load it again :

Code: Select all

[General] 195579.83 ======================================
[=ERROR=] 195579.83 Failed to import formation leader node. Some formations will likely be broken.
[General] 195579.83 ======================================

To be sure, after quitting the game and loading that save, I get the same thing :

Code: Select all

[General] 195579.83 ======================================
[=ERROR=] 195579.83 Failed to import formation leader node. Some formations will likely be broken.
[General] 195579.83 ======================================

So it seems loading the save somewhat removes (at least part of) the issue but as the issue was present in multiple consecutive saves, it seems unlikely to be purely a "memory glitch while saving" issue. I'd say it was likely an issue within the running game world yesterday.

To be sure I loaded an autosave from about an hour before (the first mentioned autosave) and I got no error in the debug log.

And an autosave from about half an hour before (the first mentioned autosave) and I got this :

Code: Select all

[General] 191906.74 ======================================
[=ERROR=] 191906.74 Failed to import formation leader node. Some formations will likely be broken.
[General] 191906.74 ======================================

So the issue seem to have appeared during gameplay.

Finally, from the day before, two saves separated by about an hour of playtime :

Code: Select all

[General] 165165.26 ======================================
[=ERROR=] 165165.26 Failed to import formation leader node. Some formations will likely be broken.
[General] 165165.26 ======================================
[General] 165165.26 ======================================
[=ERROR=] 165165.26 Failed to import formation leader node. Some formations will likely be broken.
[General] 165165.26 ======================================

Code: Select all

[General] 168543.03 ======================================
[=ERROR=] 168543.03 Failed to import formation leader node. Some formations will likely be broken.
[General] 168543.03 ======================================
[General] 168543.03 ======================================
[=ERROR=] 168543.03 Failed to import formation leader node. Some formations will likely be broken.
[General] 168543.03 ======================================

----------------------------------------------------------------

Also, for the record, 30 minutes of Prime95 found no error/warning, so the hardware fault lead seems rather inconclusive for now.

----------------------------------------------------------------

I'll now try to look deeper into the XML files to see where those problematic characters lie.

Edit: Ok I found the garbled ("backspace"?) character in the broken save doing a well-formedness check. I'm now scanning my current eleven saves but this may take some time ...

Edit 2: I'm not sure if that means anything but at least all of my current saves are well-formed.

[HBK]

Ok I did a final check by pausing the game (pause button) and doing two consecutive saves to see if I could find any weird character in the diff (thanks beyond compare ...) and all I'm getting are a bunch of additional :

<parameter value="assets\fx\gui\textures\factions\faction_holyorder" name="diffuse_map" type="texture"/>

And lots of ... small position/rotation changes. It may not mean anything but at least I didn't find any garbled character (which if random would have been different between the two saves, and could have helped me identify any potential culprit like some ant-virus software of something).

Oh well, time to do something else!

[HBK]

Or not.

I loaded that original "good" save (which still had some issues) and one from half an hour before (also an auto save).

This is the "30 mn before" autosave (the earliest I have) :

Code: Select all

[General] 70543.74 ======================================
[=ERROR=] 70543.74 Failed to import formation leader node. Some formations will likely be broken.
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x58bcec9]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x58a5ad1]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x58cc568]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x688a884]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x688c63e]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x6870943]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x787eccd]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x78cff02]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x78b6bae]
[General] 70543.74 ======================================
[General] 70543.74 ======================================
[=ERROR=] 70543.74 Error on AI director import: Could not find entity [0x9078a4e]
[General] 70543.74 ======================================

And this is the autosave made 30mn later (i.e. the "good" save linked in my second post) which had an xml attribute value typo (<item macro="countermeasure_flares_01_macro" ammunt="1"/>):

Code: Select all

[General] 75043.78 ======================================
[=ERROR=] 75043.78 LookupKeyName::LookupName(): The key name "ammunt" is not recognized in lookup 'SaveXML'. Originated from: ""
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Failed to import formation leader node. Some formations will likely be broken.
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 LookupKeyName::LookupName(): The key name "ammunt" is not recognized in lookup 'SaveXML'. Originated from: ""
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x58bcec9]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x58a5ad1]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x58cc568]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x688a884]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x688c63e]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x6870943]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x787eccd]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x78cff02]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x78b6bae]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x9078a4e]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb0b46d9]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb0834da]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb0a13f2]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc0a1ea8]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb8f1823]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc0b1ede]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc072b73]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc0922c1]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb0d696a]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb898ebd]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc0b69d1]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb8acc4c]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb8feca2]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc8b9554]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0x10c9bb0]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xc87a7fe]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xd0ad5c1]
[General] 75043.78 ======================================
[General] 75043.78 ======================================
[=ERROR=] 75043.78 Error on AI director import: Could not find entity [0xb8bd541]
[General] 75043.78 ======================================

What's interesting is that the errors in the earliest autosave are also in the next autosave (with even more errors), from [0x58bcec9] to [0x9078a4e], which seem to indicate a problem in the game world rather than a pure save issue.

It can still be an hardware fault mind you, but you gotta admit it also looks like a degenerating program error.

With that, I wish you good luck hunting those nasty bugs and a nice evening!

HBK out.

[CBJ]

Most of the error messages there are harmless and almost certainly either have nothing to do with the problem, or are a result of it rather than the cause. The two important ones are the backspace character and the "ammunt", both of which are "bit flip" errors that can only have happened between the game creating the savegame image in memory, and that image being written to the disk. It isn't possible for them to have be caused by errors in the "game world".

[HBK]

Most of the error messages there are harmless and almost certainly either have nothing to do with the problem, or are a result of it rather than the cause. The two important ones are the backspace character and the "ammunt", both of which are "bit flip" errors that can only have happened between the game creating the savegame image in memory, and that image being written to the disk. It isn't possible for them to have be caused by errors in the "game world".

Your’re right. Il corresponds both times to the second bit of an ascii character (I’m guessing the BS was supposed to be a LF). I’ll try to find a better memory checker and let it run overnight.

Thanks for your help!

[HBK]

Ok, Windows Memory Diagnostics is crap it seems. MemTest86 found an error. Now on to checking if its a chip defect or some bad setting (looks like a chip defect).

Oh well, let’s look at the bright side, today I learned that memory chips can deteriorate over time (I did an extensive check when setting up this PC some time ago ...).

Thanks for your help guys!

Edit: Actually it failed the hammer test so it may just be my motherboard aging and I may be able to fix this by bumping up DRAM voltage a little bit. Anyways it seems it was an hardware fault after all.

Edit 2: I misread the test and it’s just a plain chip error. What’s funny is that after numerous tests and tweaks it always fails at a very specific test (bit fading, always the same bit/adresss) which may explain why it only causes issues in very specific use cases (such as, you know, saving in X4).

So problem explained it would seem. Thanks again.

[RainerPrem]

Since the XML structure was mentioned here, there is perhaps a hint:

In my savegames (German game version) there is often an xml-element "ventures" close to the end of the file. It contains an escaped xml-string (&lt; instead of < etc.) . But this inner string is encoded in characterset windows-8859, and not utf-8 like the rest. The escaping does not escape the umlauts contained in the German text like "XEN Überfallkommando". So this is not a legal XML-File, as e.g. the DotNet-XMLDocument.Load() method complains.

[KlausM]

Since the XML structure was mentioned here, there is perhaps a hint:

In my savegames (German game version) there is often an xml-element "ventures" close to the end of the file. It contains an escaped xml-string (&lt; instead of < etc.) . But this inner string is encoded in characterset windows-8859, and not utf-8 like the rest. The escaping does not escape the umlauts contained in the German text like "XEN Überfallkommando". So this is not a legal XML-File, as e.g. the DotNet-XMLDocument.Load() method complains.

Indeed, there has been an issue with the ventures element, which could cause a minor issue when loading a savegame - but the cause was different, and it was already fixed in 1.60.