[Rejected] Idea for a faster game save

[jlehtone]

Yes, that. If you hit 'Pause' while game loads, it indeed is in paused state afterwards, but all (NPC) objects are not visible before you unpause a bit. I had saved while chasing Oddy. After load (and "pre-paused") the 3D model was visible, but no icon on map or gravidar. Had to unpause (briefly) to be able to select/target it again.

[Roeleveld]

There is one thing I want to still try to speed it up more:
- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

This should also help with other games that have large savegames.

I have no clue if using a RAM-disk helps though, so will need to test that first.

[Panos]

There is one thing I want to still try to speed it up more:
- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

This should also help with other games that have large savegames.

I have no clue if using a RAM-disk helps though, so will need to test that first.

a) Use Linux. It has better save times because of how the OS and Filesystem (even the archaic ext4) are designed and ofc much better Scheduler. The game runs 20% faster on same rig under Linux, using my 12 core (24 thread) CPU too compared to windows.

b) On Windows do the following to speed the save times.

• remove save folder from indexing

• remove save folder from the Windows Defender through exception

• remove save folder from any other antivirus you have

Test if you can save time by not compressing the save file (it option at settings). If you have a quad core it will save time. If you have 8-12-16 core (16/24/32 thread) CPU not that much.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

[Roeleveld]

There is one thing I want to still try to speed it up more:
- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

This should also help with other games that have large savegames.

I have no clue if using a RAM-disk helps though, so will need to test that first.

a) Use Linux. It has better save times because of how the OS and Filesystem (even the archaic ext4) are designed and ofc much better Scheduler. The game runs 20% faster on same rig under Linux, using my 12 core (24 thread) CPU too compared to windows.

Already do

b) On Windows do the following to speed the save times.

• remove save folder from indexing

• remove save folder from the Windows Defender through exception

• remove save folder from any other antivirus you have

Test if you can save time by not compressing the save file (it option at settings). If you have a quad core it will save time. If you have 8-12-16 core (16/24/32 thread) CPU not that much.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

Got NVMe and 16 core CPU.
Still planning on testing using tmpfs for the savegames.

[abisha1980]

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

[Roeleveld]

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

What percentage of the save is the log?

[Imperial Good]

- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

If this is the case then it would mean sub optimal I/O calls. I doubt it is the case.

HDDs are not slow in this situation. The save file is a continuous file and as such most HDDs should be able to hits speeds of at least 60 MB/sec write and probably higher. In this case writing the entire save file to a mechanical drive should take roughly 2 seconds. Compared with the 30-40 seconds to save this should get masked entirely by asynchronous I/O. This is assuming the file cache does not swallow it up entirely in which case you are effectively writing to a RAM drive already which is faster than even a high end NVMe drive.

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

Destroyed stuff that has fully decayed should not get saved. Of course if it is all cleaned up properly is another question and it is possible that there could be bugs causing objects or data to effectively leak.

[Panos]

- Cache the save-game directory on a RAM-disk and synchronize back to permanent storage using a low priority process while the game is running.

Ofc saving on HDD or SSD it still much slower than on M.2 NVME.

If this is the case then it would mean sub optimal I/O calls. I doubt it is the case.

HDDs are not slow in this situation. The save file is a continuous file and as such most HDDs should be able to hits speeds of at least 60 MB/sec write and probably higher. In this case writing the entire save file to a mechanical drive should take roughly 2 seconds. Compared with the 30-40 seconds to save this should get masked entirely by asynchronous I/O. This is assuming the file cache does not swallow it up entirely in which case you are effectively writing to a RAM drive already which is faster than even a high end NVMe drive.

Do the game save Data of destroyed objects? seams unlogical to keep track of objects no longer present in the universe
i know some games saves the data where a object came from etc and what happen to it (it if for example station is destroyed it's objects no longer have a function so it should no longer be present within the savefile
also i would not mind a game setting's that reduce log files to a hour, if the savefile getting smaller by it. (smaller savefile faster loading)

Destroyed stuff that has fully decayed should not get saved. Of course if it is all cleaned up properly is another question and it is possible that there could be bugs causing objects or data to effectively leak.

Look, as I wrote earlier,if you use Windows 10 deactivate the indexing and the antivirus check on the save folder
It will greatly speed your saves.
Blame sht MS products on that. Even the archaic ext4 is faster than Windows Filesystem let alone the more current new ones.

[Imperial Good]

Blame sht MS products on that. Even the archaic ext4 is faster than Windows Filesystem let alone the more current new ones.

The file systems used by Windows is more than fast enough for such effectively trivial workloads. My own personal testing with Java showed I could easily hit read speeds of over 2 GB/sec (including decompression, garbage collection stalls and other sub-optimal overhead) and write speeds which saturate an underlying mechanical drive. Maybe it is not the fastest, but if it goes effectively the speed limit does it matter for tasks like this?

[XGamer]

Blame sht MS products on that. Even the archaic ext4 is faster than Windows Filesystem let alone the more current new ones.

The file systems used by Windows is more than fast enough for such effectively trivial workloads. My own personal testing with Java showed I could easily hit read speeds of over 2 GB/sec (including decompression, garbage collection stalls and other sub-optimal overhead) and write speeds which saturate an underlying mechanical drive. Maybe it is not the fastest, but if it goes effectively the speed limit does it matter for tasks like this?

It may be fine with large continuous blocks of data. However IO can be pretty terrible compared to Linux when it comes to iterating or generally operations with more than a handful of files.
Not that that is necessarily of any concern here.

[abisha1980]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

[XGamer]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Majorly different games, you just cannot compare them.
Cyperbunk doesn't have to care about anything that is beyond your view distance. X4 on the other hand has to create at least some kind of representation in memory for every single ship and station in the entire universe, whether or not you even know about that station/ship does not matter. The entire economy is simulated at all times, as such all the ships and stations have to exist. It also needs to simulate them traveling, track ship/station inventory, subtract or add from/to it. To be able to do that the station has to have some form of existence in memory. It won't load any of the assets for it, it is not being rendered, there are no collision checks etc, but it still has to exist and be tracked in order to be able to simulate the economy properly.
They are already kind of cheating with OOS combat for example iirc, to reduce the amount of calculations necessary (e.g. no projectile simulation for fights, just look at weapons equipped compare strengths and be done with it [oversimplified]).

[abisha1980]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Majorly different games, you just cannot compare them.
Cyperbunk doesn't have to care about anything that is beyond your view distance. X4 on the other hand has to create at least some kind of representation in memory for every single ship and station in the entire universe, whether or not you even know about that station/ship does not matter. The entire economy is simulated at all times, as such all the ships and stations have to exist. It also needs to simulate them traveling, track ship/station inventory, subtract or add from/to it. To be able to do that the station has to have some form of existence in memory. It won't load any of the assets for it, it is not being rendered, there are no collision checks etc, but it still has to exist and be tracked in order to be able to simulate the economy properly.
They are already kind of cheating with OOS combat for example iirc, to reduce the amount of calculations necessary (e.g. no projectile simulation for fights, just look at weapons equipped compare strengths and be done with it [oversimplified]).

hmm got me thinking if they remove teleport or release it with a hard limit of 5 sectors they can basic have to simulate only 5 sectors around the player
objects outside this region can be put on low calculation if object 1 moves at speed X it will take object 1 X time to reach destiny (very basic calculation a modern processor can handle this task with each for millions of objects

then they also need to add medium level (objects by players that gives a range view) say ships they only need to show a proxy of what's going on don't need to be accurate time the player is on the destination whole data set should be calculated

[Raptor34]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Do you even know what you are talking about? Cyperpunk doesn't have 68.9GB sized saves neither does Foundations have 19.8GB saves.

[Old Drullo321]

I don't want to be rude but before throwing in suggestions for improvements, you should understand why there is a difference in loading times (and maybe also savegame filesize) between different games. We don't know the source code for the saving/loading algorithm but just based on pure theory, it should be clear, that it will take longer and why there are difference between your referenced Cyberpunk, or even X3 and X4. Maybe the save/load code is crappy, but even if it would be perfect, it would still be slower then what you have in many other games, like X3 or Cyberpunk. The game design philosophy of X4 is to be a much better simulation than every predecessor. And based on this you have certain advantages, but also disadvantages. Besides more CPU calculation, it results in longer load times. Your suggestions would result in a simulation that is more like X3.

And just comparing game folder size doesn't get you anywhere. By the way, in most cases, those difference are caused by simple more assets like mesh/textures, voice, etc.

[abisha1980]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Do you even know what you are talking about? Cyperpunk doesn't have 68.9GB sized saves neither does Foundations have 19.8GB saves.

whole data of both games, a save of foundation is 88,4 MB compare to Cyberpunk save is 2.2MB while Stellaris have a average of 5.5MB save and BG3 have save files of 8,6 MB
it clearly shows something is wrong with foundations it store 10 to 44 times more data then other games and it shows when you load the game

[Raptor34]

it's obvious the game is to blame for the slow loading, i use NVMe on my C drive and have foundations stored on my other NVMe drive that have a speed of 3.2GB/s
i read somewhere on this topic or a other the problem is world building, so that needs fixing

so why not use the same tactics other games use loading low resources then add higher resources into the memory?
if i load for example cyberpunk 2077 it only takes 5 seconds to load a game no matter the size of the save (cyberpunk is 68,9GB that's almost 3.5 more data) (foundations is only 19,8GB)

The devs can also cheat by loading only the sector the player is in, i do not see why the game need to build the whole universe if you only can be present in a single location.
also i think this need to be addressed before the devs go making more content to the game because more data means even longer loading

Do you even know what you are talking about? Cyperpunk doesn't have 68.9GB sized saves neither does Foundations have 19.8GB saves.

whole data of both games, a save of foundation is 88,4 MB compare to Cyberpunk save is 2.2MB while Stellaris have a average of 5.5MB save and BG3 have save files of 8,6 MB
it clearly shows something is wrong with foundations it store 10 to 44 times more data then other games and it shows when you load the game

That's because it stores a lot more data in its saves.

[al_dude]

Unzip and take a look at the save.

You will see why it takes 15 seconds to save (on my PC at least) and about a minutes to load.

It keeps track of every single item (ships, stations, literal items, NPCs) in the universe.

[XGamer]

it clearly shows something is wrong with foundations it store 10 to 44 times more data then other games and it shows when you load the game

All that shows really is what I have said above already: It simulates and keeps track of every single ship and station in the entire universe (known to you or not), even if you have never even seen the sector and its not on your map, it is still being simulated and kept track of.
Reason is simply that most of the economy is actually happening.
Of course if you want to resume that you will have to be able to restore that state during load, which means all these objects need to be present in the save game, which is why it is big.
For all the stations and ships, it needs to save where they are, what their current orders, equipment, health, cargo, etc. is. If it didn't then you would load a game and information would be lost. Your HQ might be getting an NPC delivery of 500 Hull parts for example. If that information were not in the save, then upon loading that delivery wouldn't happen anymore after loading.
You might have saved the game after having destroyed all turrets on a capital ship or station. If it didn't write the status of every equipment (e.g. turrets) on ships and stations to the save file for example, you would load that save later only to be greeted by all the turrets being in working order.

[abisha1980]

Unzip and take a look at the save.

You will see why it takes 15 seconds to save (on my PC at least) and about a minutes to load.

It keeps track of every single item (ships, stations, literal items, NPCs) in the universe.

their is no need to keep track of all objects within the universe

few thing's to improve
1. only save how many ships (traders etc) are within the current universe and reseed the game with new ships do you really recall XSSF-DDSDD trader 2 days ago? why is this even need to be stored
2.it don't need to store data on what happen to destroyed objects, ships or what asteroid you destroyed 2 day's ago
3.it only need to store data on what a object for example khaak station on sector and seed the game again
4.it don't need to store all station data, that's already present within the game it only need to store what happen to given object if it's not present it's no longer needed. (so a read log not some complex calculation etc)
5.just reseed marines and mechanics nobody knows the face of Khaargakkssaaa (only the pilots and station holders faces need to be saved)