v4 save load times

[Clownmug]

Other massive scale simulation games also take this long - look at the Paradox games as an example.

Which Paradox games? Stellaris takes like 2 seconds to save/load games for me.

[Alan Phipps]

Let's stay on topic please. Other games are not really relevant to this thread.

[Panos]

Hello! Will something be done in the v4 for a faster save and load times? For me, on my SSD, it takes 18s to save(well ok), and 75s to load on a new game(not so ok). I tried to disable compression but it changes nothing(except taking more disk space).
To be honest, it is the main reason because I prefer playing others games right now (this and the no skills gain for pilots, and the not so much interesting lategame).

a) In Windows 10, remove indexing & antivirus scan for the save folder.

b) Use Linux

[dtpsprt]

a) In Windows 10, remove indexing & antivirus scan for the save folder.

b) Use Linux

Even though you are "technically" correct, it still takes a long time based not on the actual mechanics of the procedure but on the player's perspective that needs to go back to fighting after an autosave, or reloading because he was blasted out of nowhere (happens some times) or, more importantly and most often, when he/she has to both save and reload to make a trader appear, be able to get out of an elevator, be able to speak to the mission target or, simply, because the guidelines disappeared from the map and the destination is not shown...

[Slashman]

a) In Windows 10, remove indexing & antivirus scan for the save folder.

b) Use Linux

Even though you are "technically" correct, it still takes a long time based not on the actual mechanics of the procedure but on the player's perspective that needs to go back to fighting after an autosave, or reloading because he was blasted out of nowhere (happens some times) or, more importantly and most often, when he/she has to both save and reload to make a trader appear, be able to get out of an elevator, be able to speak to the mission target or, simply, because the guidelines disappeared from the map and the destination is not shown...

The bottom line here is that it is not going to change. Why are we still even discussing it? It is literally a hard coded by-product of the way the game works.

[chew-ie]

The bottom line here is that it is not going to change. Why are we still even discussing it? It is literally a hard coded by-product of the way the game works.

Because there is nothing better to do. Oh wait ... seems like public beta 4.0 is live

[Slashman]

The bottom line here is that it is not going to change. Why are we still even discussing it? It is literally a hard coded by-product of the way the game works.

Because there is nothing better to do. Oh wait ... seems like public beta 4.0 is live

Well played sir!

[Cdaragorn]

That's pretty much what CBJ said they do now in here from one of the links above, but it doesn't just take a second!

CBJ also mentions elsewhere ".. we can't just "copy" the gamestate as it's not just one monolithic "state". That copying is the save data being written to memory before being written to disk."

The reason he assumes it can be done in seconds is because he's just taking the theoretical maximum transfer speed of memory and assuming you can actually transfer data that fast.
In practice you never hit those speeds or anything even close to them. There are always other factors and parts of the complete pipeline that limit you to far slower speeds than what just the memory is technically capable of reaching.
While I do agree that there are almost certainly ways that this process could be sped up it's far from anything I get too concerned about in the game. I'm able to enjoy hours of playing after loading once and only need to save occasionally.

I do wish that people would stop using text formats like xml and json for storing large amounts of data like this. Something like Sqlite can read and write data far faster for a lot of reasons, not the least of which is that it doesn't have to write nearly as much meta data for each piece of actual data it needs to write.

[dtpsprt]

I do wish that people would stop using text formats like xml and json for storing large amounts of data like this. Something like Sqlite can read and write data far faster for a lot of reasons, not the least of which is that it doesn't have to write nearly as much meta data for each piece of actual data it needs to write.

Even though you are correct I'm afraid that xml is a must for running as smoothly as possible in windows... Like you said, you don't actually mind the load time as long as you enjoy the game...

[hxsgame]

What needs to be saved is the logical world, not the data stream. so CPU computing power is the key, not SSD.
The more complex the dynamic universe, the longer the time.

Two ways maybe :
1, Reduce things in your universe, especially stations with complex logic levels, They are cpu killers (Bad idea, i know……);

2, Faster CPU and smaller memory latency.

This "problem" has been discussed repeatedly, I "contributed" my insights, and wasted a lot of professional developers' time.

Now, I believe it has been sufficiently optimized, and The time required is necessary to enjoy the dynamic free X4universe

[Raevyan]

Maybe it’s time to port X4 to PS5

[zakaluka]

With some slightly clever memory layout of the internal structures, the game could save in the background so would effectively be zero seconds. Loading would not be affected, but at least you could play while you saved.

Asynchronous saving is a much harder problem for X4 than you probably believe. It comes down to, the universe is a complex data structure, not a flat data structure, and every object is simulated in real time as the game engine traverses that data structure constantly. It could be accomplished but it would dramatically increase the cost of saving, and make it a constant I/O debt - enough to harm game performance quite a lot.

I used to say a lot of the things you are saying here. I still think flattening to and reading from XML must add at least a significant overhead, and I wonder how things would change with a cheaper format.

[chew-ie]

With some slightly clever memory layout of the internal structures, the game could save in the background so would effectively be zero seconds. Loading would not be affected, but at least you could play while you saved.

Asynchronous saving is a much harder problem for X4 than you probably believe. It comes down to, because every object in the universe is simulated simultaneously. It could be accomplished but it would dramatically increase the cost of saving, and make it a constant I/O debt - enough to harm game performance quite a lot.

I'd love to see a contest with an open sourced X4* engine where everybody can prove his/her/its own theory about the optimization being simple/easy - within public repos which can show the needed time in a transparent way.

*probably won't happen in a very long time / ever, but I had to propose it

Given enough time one can solve every problem, the interesting result would indeed be how long it would take for the individual solutions to be stable.

[zakaluka]

I'd love to see a contest with an open sourced X4* engine where everybody can prove his/her/its own theory about the optimization being simple/easy - within public repos which can show the needed time in a transparent way.

Maybe, but it's a non-starter. I do want to point out that it's natural for this topic to come up every x months, because save and load is absolutely maddening in x4. I totally get that people are frustrated with it. And I wish there was an obvious solution.

For instance I know egosoft could make things at least *somewhat* faster by switching to a non-human-readable fileformat. And this is usually the kind of enhancement that could be made in an afternoon. But it's pretty clear they don't want to try, and they probably have valid reasons. Maybe they think it's not that much of a gain, for what they would lose in the process.

[Lord Crc]

Asynchronous saving is a much harder problem for X4 than you probably believe. It comes down to, the universe is a complex data structure, not a flat data structure, and every object is simulated in real time as the game engine traverses that data structure constantly.

The point I was trying to make is that you can store that complex data structure in memory in such a way that taking a snapshot of it is easy and fast.

Once you have a snapshot copy, serializing the copy to disk can be done in in a background thread without impacting the game significantly (assuming you got spare CPU cycles). I don't care about saving taking a minute if I can play normally while it saves.

edit: I'd just like to reiterate that I understand why they're sticking with XML, and I fully appreciate that the solution I've mentioned cannot simply be bolted on without much ado.

[dan1101]

Maybe, but it's a non-starter. I do want to point out that it's natural for this topic to come up every x months, because save and load is absolutely maddening in x4. I totally get that people are frustrated with it. And I wish there was an obvious solution.

For instance I know egosoft could make things at least *somewhat* faster by switching to a non-human-readable fileformat. And this is usually the kind of enhancement that could be made in an afternoon. But it's pretty clear they don't want to try, and they probably have valid reasons. Maybe they think it's not that much of a gain, for what they would lose in the process.

Yes, the save time is ok if I want to go get a drink or something, but if I'm in the middle of something important it really disrupts what I'm trying to do.

As far as the format I guess it's nice to be able to look at savegame files directly as XML files, but if they were encoded into a smaller/faster format it also wouldn't be hard to make a tool to unencode/encode savegame files. Especially if that meant every single player's savegames would be faster.

[Cdaragorn]

I do wish that people would stop using text formats like xml and json for storing large amounts of data like this. Something like Sqlite can read and write data far faster for a lot of reasons, not the least of which is that it doesn't have to write nearly as much meta data for each piece of actual data it needs to write.

Even though you are correct I'm afraid that xml is a must for running as smoothly as possible in windows... Like you said, you don't actually mind the load time as long as you enjoy the game...

Sorry but this has me completely confused. What does xml have to do with running smoothly in anything? Windows doesn't depend on xml. It doesn't even know anything about it. Other applications know how to read and write it.

To write to a format like xml I have to load the entire data into memory because I can't write to just one piece of it without writing everything around it. This is also true of json and pretty much any text format.
With sqlite I can target the specific data I need no matter where it is or what is around it. I can also write and read the exact same data several times faster. In my experience sqlite is always smoother than a raw text format.

[Cdaragorn]

Asynchronous saving is a much harder problem for X4 than you probably believe. It comes down to, the universe is a complex data structure, not a flat data structure, and every object is simulated in real time as the game engine traverses that data structure constantly.

The point I was trying to make is that you can store that complex data structure in memory in such a way that taking a snapshot of it is easy and fast.

Once you have a snapshot copy, serializing the copy to disk can be done in in a background thread without impacting the game significantly (assuming you got spare CPU cycles). I don't care about saving taking a minute if I can play normally while it saves.

edit: I'd just like to reiterate that I understand why they're sticking with XML, and I fully appreciate that the solution I've mentioned cannot simply be bolted on without much ado.

What you seem to keep ignoring is that people have pointed out that your proposed solution would not be either easy or fast. Ego even pointed out that they actually do what you are proposing already. Unless of course you're proposing keeping that structure in memory constantly which is another non starter. You don't have enough memory to do something like that. Working with memory is also never as fast as you are insisting it is.

[Lord Crc]

What you seem to keep ignoring is that people have pointed out that your proposed solution would not be either easy or fast.

I think you fundamentally misunderstand my proposal. The key point of my proposal is that it would reduce the time the game needs to be paused when saving, that is the perceived time. The actual process of generating the savegame file would not be any quicker, but it would happen in the background.

As I've said I understand that it's time consuming (and possibly difficult) to retrofit, but in a new code base it's quite easy to add support for this as the concept itself is easy.

Working with memory is also never as fast as you are insisting it is.

Sequential memory access is very fast on human timescales. We're talking several GByte/s, up to 15GB/s for higher-end memory these days. Given that the game state serialized to XML is roughly 500-1000MB and the overhead of XML, the in-memory representation is certainly an order of magnitude less, so 100MB tops. Copying a 100MB sequential block of memory takes less than 100 milliseconds on a modern computer, a blink of an eye.

[zakaluka]

everyone understands your proposal. What you fail to realize is that the x4 universe can't reasonably be represented as a flat block of structured data. It's a complex branching tree of pointers that sometimes points back to itself, for a number of compelling reasons (that you could get insight into by reading the xml of a savefile if you wanted to). Flattening it to a block of structured data would be infeasible for many different reasons that aren't worth re-hashing again (and again and again)

At this point you're ignoring what the devs have said directly on this topic on a number of occasions. It's a non-starter.