[Rejected] Idea for a faster game save

[Roeleveld]

A left-field method of “speeding up saves”, especially for people with many-threaded CPUs would be to duplicate all memory before saving... wait for it... while playing! I know, you’ve heard it before, but give me a chance to explain.

Memory blocks could be copied over time, then continuously updated either manually, or by replicating the game sim threads on the copied memory. Gradually getting bigger, more complete, and representative of the final save. Once the copy is complete and aligned with the main game data, it is frozen, processed and saved to disk in the background, but that stuff is boring.

You would need to be running duplicates of most of the game threads which would hammer the CPU and maybe frame rates, but it might work. Obviously there would be considerable delay between pressing save and the save actually happening. There might also be some ‘drift’ between what happens in the game and what actually gets saved, but I doubt it would be meaningful.

Would be great for autosaves, especially.

I doubt the memory bus can handle the additional required bandwidth.
And ensuring the copy is consistent will require even more CPU-resources, impacting the FPS even more.

[DavidGW]

I doubt the memory bus can handle the additional required bandwidth.
And ensuring the copy is consistent will require even more CPU-resources, impacting the FPS even more.

No idea, but depending on how much work it would be to implement (which likely only the devs would have an idea of) if it was me I’d give it a shot.

A bad frame rate is still better than zero, especially for the 30ish seconds this would probably take.

[BaronVerde]

A left-field method of “speeding up saves”, especially for people with many-threaded CPUs would be to duplicate all memory before saving... wait for it... while playing! I know, you’ve heard it before, but give me a chance to explain.

Memory blocks could be copied over time, then continuously updated either manually, or by replicating the game sim threads on the copied memory, mirroring what is happening in the real game. Gradually getting bigger, more complete, and representative of the final save. Once the copy is complete and aligned with the main game data, it is frozen, processed and saved to disk in the background, but that stuff is boring.

You would need to be running duplicates of most of the game threads which would hammer the CPU and maybe frame rates, but it might work. Obviously there would be considerable delay between pressing save and the save actually happening. There might also be some ‘drift’ between what happens in the game and what actually gets saved, but I doubt it would be meaningful.

Would be great for autosaves, especially.

This will not work if there's some sense to the simulation. To avoid inconsistencies a simulation snapshot must save all relevant state at a single point in time, which is between hard, impractical and impossible to parrallelize and synchronize. Also, rendered frames are only the front end we players see, they may be completely decoupled from the internals, that may well happen at their own heartbeat.

Many-, multi-, poly- or omni-threaded CPUs are a fine thing, but a problem must lend itself to parallelization. If it doesn't, then no programming gymnastics will save the day.

All in all, none of us is in a position to make proposals that the makers haven't thought of before ... waiting a few 10s for saving to occur really is a luxury problem, or not ? And the case is closed anyway.

@ImperialGood:
You may be right, I'was having fun speculating

[jlehtone]

Once the copy is complete and aligned with the main game data, it is frozen

(Disk) storage (file)systems have "snapshot". At least some of them are copy-on-write,
i.e. on every change after a snapshot the original block is moved to "snapshot volume"
and new content is in "active volume". That way the snapshot retains "intact copy" of
some timepoint. Alas, the data structure is not likely to be filesystem-like.

Your proposal: (1) have two mirrored copies, (2) stop mirroring and start writing a file.
What if I want to save a second time before the previous write is complete and mirror
has been resynced?

Would be great for autosaves, especially.

Isn't there the option to disable autosaves, if they bother you?

No idea, but ...

That summarizes all "faster save/load" threads. We do not know. Most of the thrown ideas
appear to be based on (false) assumptions.

Are there angles that have not occurred to the devs? Possibly. What is the chance that one
of us would find that needle from the haystack and a dev would actually spot it?

[Alan Phipps]

Anyway, now that we know that we can have pause active after completion of a load or save, just set it up and go do something before the next play session (make a drink or take a comfort break, etc).

[jlehtone]

Yes, that. If you hit 'Pause' while game loads, it indeed is in paused state afterwards, but all (NPC) objects are not visible before you unpause a bit. I had saved while chasing Oddy. After load (and "pre-paused") the 3D model was visible, but no icon on map or gravidar. Had to unpause (briefly) to be able to select/target it again.

[Roeleveld]

There is one thing I want to still try to speed it up more:
- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

This should also help with other games that have large savegames.

I have no clue if using a RAM-disk helps though, so will need to test that first.

[Panos]

There is one thing I want to still try to speed it up more:
- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

This should also help with other games that have large savegames.

I have no clue if using a RAM-disk helps though, so will need to test that first.

a) Use Linux. It has better save times because of how the OS and Filesystem (even the archaic ext4) are designed and ofc much better Scheduler. The game runs 20% faster on same rig under Linux, using my 12 core (24 thread) CPU too compared to windows.

b) On Windows do the following to speed the save times.

• remove save folder from indexing

• remove save folder from the Windows Defender through exception

• remove save folder from any other antivirus you have

Test if you can save time by not compressing the save file (it option at settings). If you have a quad core it will save time. If you have 8-12-16 core (16/24/32 thread) CPU not that much.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

[Roeleveld]

There is one thing I want to still try to speed it up more:
- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

This should also help with other games that have large savegames.

I have no clue if using a RAM-disk helps though, so will need to test that first.

a) Use Linux. It has better save times because of how the OS and Filesystem (even the archaic ext4) are designed and ofc much better Scheduler. The game runs 20% faster on same rig under Linux, using my 12 core (24 thread) CPU too compared to windows.

Already do

b) On Windows do the following to speed the save times.

• remove save folder from indexing

• remove save folder from the Windows Defender through exception

• remove save folder from any other antivirus you have

Test if you can save time by not compressing the save file (it option at settings). If you have a quad core it will save time. If you have 8-12-16 core (16/24/32 thread) CPU not that much.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

Got NVMe and 16 core CPU.
Still planning on testing using tmpfs for the savegames.

[abisha1980]

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

[Roeleveld]

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

What percentage of the save is the log?

[Imperial Good]

- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

If this is the case then it would mean sub optimal I/O calls. I doubt it is the case.

HDDs are not slow in this situation. The save file is a continuous file and as such most HDDs should be able to hits speeds of at least 60 MB/sec write and probably higher. In this case writing the entire save file to a mechanical drive should take roughly 2 seconds. Compared with the 30-40 seconds to save this should get masked entirely by asynchronous I/O. This is assuming the file cache does not swallow it up entirely in which case you are effectively writing to a RAM drive already which is faster than even a high end NVMe drive.

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

Destroyed stuff that has fully decayed should not get saved. Of course if it is all cleaned up properly is another question and it is possible that there could be bugs causing objects or data to effectively leak.

[Panos]

- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

If this is the case then it would mean sub optimal I/O calls. I doubt it is the case.

HDDs are not slow in this situation. The save file is a continuous file and as such most HDDs should be able to hits speeds of at least 60 MB/sec write and probably higher. In this case writing the entire save file to a mechanical drive should take roughly 2 seconds. Compared with the 30-40 seconds to save this should get masked entirely by asynchronous I/O. This is assuming the file cache does not swallow it up entirely in which case you are effectively writing to a RAM drive already which is faster than even a high end NVMe drive.

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

Destroyed stuff that has fully decayed should not get saved. Of course if it is all cleaned up properly is another question and it is possible that there could be bugs causing objects or data to effectively leak.

Look, as I wrote earlier,if you use Windows 10 deactivate the indexing and the antivirus check on the save folder
It will greatly speed your saves.
Blame sht MS products on that. Even the archaic ext4 is faster than Windows Filesystem let alone the more current new ones.

[Imperial Good]

Blame sht MS products on that. Even the archaic ext4 is faster than Windows Filesystem let alone the more current new ones.

The file systems used by Windows is more than fast enough for such effectively trivial workloads. My own personal testing with Java showed I could easily hit read speeds of over 2 GB/sec (including decompression, garbage collection stalls and other sub-optimal overhead) and write speeds which saturate an underlying mechanical drive. Maybe it is not the fastest, but if it goes effectively the speed limit does it matter for tasks like this?

[XGamer]

Blame sht MS products on that. Even the archaic ext4 is faster than Windows Filesystem let alone the more current new ones.

The file systems used by Windows is more than fast enough for such effectively trivial workloads. My own personal testing with Java showed I could easily hit read speeds of over 2 GB/sec (including decompression, garbage collection stalls and other sub-optimal overhead) and write speeds which saturate an underlying mechanical drive. Maybe it is not the fastest, but if it goes effectively the speed limit does it matter for tasks like this?

It may be fine with large continuous blocks of data. However IO can be pretty terrible compared to Linux when it comes to iterating or generally operations with more than a handful of files.
Not that that is necessarily of any concern here.

[abisha1980]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

[XGamer]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Majorly different games, you just cannot compare them.
Cyperbunk doesn't have to care about anything that is beyond your view distance. X4 on the other hand has to create at least some kind of representation in memory for every single ship and station in the entire universe, whether or not you even know about that station/ship does not matter. The entire economy is simulated at all times, as such all the ships and stations have to exist. It also needs to simulate them traveling, track ship/station inventory, subtract or add from/to it. To be able to do that the station has to have some form of existence in memory. It won't load any of the assets for it, it is not being rendered, there are no collision checks etc, but it still has to exist and be tracked in order to be able to simulate the economy properly.
They are already kind of cheating with OOS combat for example iirc, to reduce the amount of calculations necessary (e.g. no projectile simulation for fights, just look at weapons equipped compare strengths and be done with it [oversimplified]).

[abisha1980]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Majorly different games, you just cannot compare them.
Cyperbunk doesn't have to care about anything that is beyond your view distance. X4 on the other hand has to create at least some kind of representation in memory for every single ship and station in the entire universe, whether or not you even know about that station/ship does not matter. The entire economy is simulated at all times, as such all the ships and stations have to exist. It also needs to simulate them traveling, track ship/station inventory, subtract or add from/to it. To be able to do that the station has to have some form of existence in memory. It won't load any of the assets for it, it is not being rendered, there are no collision checks etc, but it still has to exist and be tracked in order to be able to simulate the economy properly.
They are already kind of cheating with OOS combat for example iirc, to reduce the amount of calculations necessary (e.g. no projectile simulation for fights, just look at weapons equipped compare strengths and be done with it [oversimplified]).

hmm got me thinking if they remove teleport or release it with a hard limit of 5 sectors they can basic have to simulate only 5 sectors around the player
objects outside this region can be put on low calculation if object 1 moves at speed X it will take object 1 X time to reach destiny (very basic calculation a modern processor can handle this task with each for millions of objects

then they also need to add medium level (objects by players that gives a range view) say ships they only need to show a proxy of what's going on don't need to be accurate time the player is on the destination whole data set should be calculated

[Raptor34]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Do you even know what you are talking about? Cyperpunk doesn't have 68.9GB sized saves neither does Foundations have 19.8GB saves.

[Old Drullo321]

I don't want to be rude but before throwing in suggestions for improvements, you should understand why there is a difference in loading times (and maybe also savegame filesize) between different games. We don't know the source code for the saving/loading algorithm but just based on pure theory, it should be clear, that it will take longer and why there are difference between your referenced Cyberpunk, or even X3 and X4. Maybe the save/load code is crappy, but even if it would be perfect, it would still be slower then what you have in many other games, like X3 or Cyberpunk. The game design philosophy of X4 is to be a much better simulation than every predecessor. And based on this you have certain advantages, but also disadvantages. Besides more CPU calculation, it results in longer load times. Your suggestions would result in a simulation that is more like X3.

And just comparing game folder size doesn't get you anywhere. By the way, in most cases, those difference are caused by simple more assets like mesh/textures, voice, etc.