Witch the new API of "NVIDIA RTX IO", Can we expect a smoother Save/Load experience?

[Imperial Good]

No. That tech makes loading of data from disk to GPU memory faster. Textures, I presume. Your savegame is not such data.

The exact specifications or limits of the tech do not seem widely spread at the moment. Looking at some articles posted by Nvidia for their server sector which has been using this tech for years I am guessing that it can work both directions with the GPU also being able to send compressed data back to the storage directly. If this is the case then the GPU might just have dedicated compression/decompression hardware on it, similar to the new consoles, in which case it could be used as a more efficient compression offload engine for general data resulting in a reduction of CPU time and memory bandwidth.

Furthemore, it requires tech from Microsoft that will be available "only for Windows 10".

The technology is not tied to any specific platform. Just it requires a kernel and API that supports its use. I am fairly certain that at some stage Linux, or variations there of, will offer support for a same purpose API.

If we are going to see any speed on the writing/compression it will be with the AMD solution used on the new consoles as they build up the whole ecosystem in hardware level and going to be software agnostic.

AMD is not responsible for this on the consoles as far as I am aware. At least Sony sourced their compression/decompression engine from another company. Just working with AMD they put it all the designs together into a single functional chip that Sony can order from TSMC.

If you want to speed up the save you need NVME drive, good CPU, save un-compressed file and preferably use Linux.

The difference in saving time between compressed and un-compressed on a NVMe drive was recently measured as at most 10% in favour of uncompressed by someone on the discord. Given how much larger the saves become when uncompressed, the trade off is fair.

I am still not sure why this trade-off exists since technically a multi-threaded pipeline of sorts could be used that creates XML data, passing chunks of it to a separate thread to be compressed and then using asynchronous I/O to write out the resulting compressed data to storage efficiently in parallel and good scaling. In this case save time would be limited by whichever stage takes longest to complete, which would likely be XML generation due to that requiring iterating the game state graph. Since saving requires iterating a lot of data, it is possible that this 10% difference comes from CPU memory bandwidth limitations rather than CPU computational speed since that is a finite resource that does not scale with core count.

[Panos]

The difference in saving time between compressed and un-compressed on a NVMe drive was recently measured as at most 10% in favour of uncompressed by someone on the discord. Given how much larger the saves become when uncompressed, the trade off is fair.

I am still not sure why this trade-off exists since technically a multi-threaded pipeline of sorts could be used that creates XML data, passing chunks of it to a separate thread to be compressed and then using asynchronous I/O to write out the resulting compressed data to storage efficiently in parallel and good scaling. In this case save time would be limited by whichever stage takes longest to complete, which would likely be XML generation due to that requiring iterating the game state graph. Since saving requires iterating a lot of data, it is possible that this 10% difference comes from CPU memory bandwidth limitations rather than CPU computational speed since that is a finite resource that does not scale with core count.

On same system I use Linux and Windows 10.

On the Windows 10 the following steps reduced the time to save
a) Removed the save folder from Indexing
b) Removed the save folder from the antivirus checking

Those 2 reduced the time to save significantly making the saving compressed/uncompressed on top of them irrelevant on my system as the difference was 1-2 seconds (3900X well tuned for 4.65Ghz dual core boost)

However on Linux the saving if way faster than on Windows 10 without doing anything else.
Same system only using Manjaro and a Samsung 970 Evo instead of Windows 10 an MP600 on pcie 4.0 slot.

So the Operating System and File System is also a factor on how fast the game will be saved at the end of the day.

As for the whole concept of serializing perfectly understand. I am working as a software developer 23y of which a good part of my career on a shipping company having to serialize thousands of pieces of cargo every second as they were moving across the North Sea on various systems on various ports, countries and statuses each with different XML formats depending gazillion of factors over multiple servers, and I am impressed of the internal engine that serializes the whole game including the state of each object to keep the realistic simulator working on a single CPU

Wish I was able to work with the code of this thing as I have a lot of ideas to improve the game through modding but requiring secondary file to store the data

[Imperial Good]

However on Linux the saving if way faster than on Windows 10 without doing anything else.

This is really strange. I/O calls alone on Windows do not make much of a difference with 3.2 GB/sec of random access easily being possible from within something like Java which is not tightly optimized to Windows. Unless the way saving happens is not at all optimized for Windows, which would mean a lot of small or deprecated I/O calls or I/O that does not take advantage of the file cache. Otherwise this would point towards optimization differences with the output binaries from the compiler setup.

Do you have example save times? I could imagine the difference between Windows and Linux being 5% or so given the same hardware is used but anything larger would point towards some other cause of slowdown.

[Alm888]

This is really strange.

The thing is, NTFS is a very old file system from 1993 and Microsoft has never bothered to replace it with something modern.
It should be noted though, that finding a sane benchmark is not a trivial task. The filesystems' "ecosystems" do not overlap and most researchers focus on either Windows (NTFS/exFAT/FAT32) or Linux (ext4/reiserFS/BtrFS) clusters, and those who do compare NTFS vs. ext4 often do so incorrectly (by using NTFS via FUSE in Linux, which absolutely trashes performance).

[dtpsprt]

This is really strange. I/O calls alone on Windows do not make much of a difference with 3.2 GB/sec of random access easily being possible from within something like Java which is not tightly optimized to Windows. Unless the way saving happens is not at all optimized for Windows, which would mean a lot of small or deprecated I/O calls or I/O that does not take advantage of the file cache. Otherwise this would point towards optimization differences with the output binaries from the compiler setup.

Do you have example save times? I could imagine the difference between Windows and Linux being 5% or so given the same hardware is used but anything larger would point towards some other cause of slowdown.

What's strange about that Imperial? To add insult to injury Microsoft started in Vista the total virtual running of the computer. This means that it creates a "virtual machine" for the player to use with no direct access to the hardware. This means that there is a "delay" for data to be actually transferred to the Hard Disk, the process is done once in the "virtual computer" and once in the real one.

One may argue that the "virtual" is on RAM, so it is "instant".

I have (very old) news on that. Ever since Win 3.1 Windows retains control of the physical RAM to be used by Microsoft Applications (so they'll seem to be the fastest of all) pushing all other applications in the Hard Disk RAM created on startup (most usually fractured of course).

So we have one Hard Disk Operation to write the save on HD (virtual machine), second one to compress it (virtual machine again), write it down (finally) on the "actual" Hard Disk. All these operations are done inside the Hard Disk since there is no physical RAM available.

In contrast Linux uses some 0,5 GB of RAM (or less), leaving everything else to the applications. This means that the file gets compressed in RAM (actual) and written down on the hard disk in one go. Even if there is not enough RAM to do the compression in it (in whole or part of it), it is still one process less than Windows (at best), therefore the difference.

P.S. If you are ready to "mod" Windows to actually behave like a normal programme, I can give you some great pointers...

[Imperial Good]

The thing is, NTFS is a very old file system from 1993 and Microsoft has never bothered to replace it with something modern.
It should be noted though, that finding a sane benchmark is not a trivial task. The filesystems' "ecosystems" do not overlap and most researchers focus on either Windows (NTFS/exFAT/FAT32) or Linux (ext4/reiserFS/BtrFS) clusters, and those who do compare NTFS vs. ext4 often do so incorrectly (by using NTFS via FUSE in Linux, which absolutely trashes performance).

One thing I know is that NTFS does perform quite poorly when it comes to file lookup operations (not file I/O). Or at least the Window driver for it. This was one of the issues Microsoft had with their native Linux support since Linux handles file lookup operations an order of magnitude faster and depends on such speed to run efficiently.

That said this does not really affect save/load of X4 since that is only a single large file with huge processing time.

What's strange about that Imperial? To add insult to injury Microsoft started in Vista the total virtual running of the computer. This means that it creates a "virtual machine" for the player to use with no direct access to the hardware. This means that there is a "delay" for data to be actually transferred to the Hard Disk, the process is done once in the "virtual computer" and once in the real one.

The OSes support virtualization, but it is not enabled by default. Enabling it does come at a performance penalty since then a hypervisor is used. This mode is needed to run things like other virtual machines and is how native Linux compatibility was implemented on Windows 10.

I have (very old) news on that. Ever since Win 3.1 Windows retains control of the physical RAM to be used by Microsoft Applications (so they'll seem to be the fastest of all) pushing all other applications in the Hard Disk RAM created on startup (most usually fractured of course).

Windows does not work like that.

The OS manages all physical memory as all modern OSes do (such as Linux and OSX). It then assigns each application running on it a private virtual memory address space. Virtual memory is a long existing feature supported by high performance processors that allows the creation of a custom and reprogrammable address space that is not directly related to the physical memory address layout of the system. Virtual memory is managed in a unit called "memory page" which represents some uniform amount of memory. The OS is then responsible for managing the virtual memory of each application, keeping track of and setting up mappings from virtual memory to physical memory. Since pages do not benefit much from being organized in a continuous way in memory and are uniform this makes managing memory and giving it to applications a lot easier and is one of the main reasons this technique is used.

Since memory pages can be moved around in physical memory easily, it becomes possible to write out memory pages to some persistent or bulk storage medium to free up physical memory. Hence why all modern operating systems use a page file, which can be used to hold memory pages when available physical memory is depleted and as such avoid out of memory crashes. Logically bulk storage is a lot slower than memory, and as such a victim algorithm is used to select pages that are infrequently touched in preference to ones frequently used to minimize the performance impact. When an application touches an address that is currently not resident in physical memory due to being paged out to the page file, a page fault interrupt is generated that the OS processes and uses to move the corresponding memory page back into physical memory before resuming application execution as if nothing had happened. Since using the page file does have an overhead, the OS tries its best to avoid using it when possible by first mostly filling physical memory before starting to evict infrequently used memory pages to the page file.

This also allows a different kind of I/O, memory mapped I/O. An application can request that part of its virtual address space be mapped to parts of arbitrary files within the file system as long as alignment requirements are respected. Since file data becomes memory pages, it also means that a feature such as file cache becomes trivial to implement. This allows frequently accessed file data to be stored in memory pages so when an application requests it then either the data is copied directly in bulk to the destination address, or if memory mapped I/O is used then the memory page is trivially remapped to the application address space without any copying occurring.

Since the Kernel, and the drivers running in it, have to also use memory there is special handling of such memory separate from applications. For example pages might be locked down and forced to reside in physical memory for correct DMA functionality, such as used by underlying I/O. This applies to all Windows drivers, third party drivers and the kernel itself. It does not apply to Microsoft applications, which run like all normal applications using standard user space virtual memory. Some Microsoft applications which have associated drivers might use this functionality within the driver part, but the user space application part will be like any other application. The logical reason for this is that Microsoft designed the OS and features of it around supporting applications like their own, so they do not need special treatment as what is provided to every application is good. Some Microsoft applications interface with the OS using obscure or non-public APIs, but this is not related to memory and more to OS interaction.

So we have one Hard Disk Operation to write the save on HD (virtual machine), second one to compress it (virtual machine again), write it down (finally) on the "actual" Hard Disk. All these operations are done inside the Hard Disk since there is no physical RAM available.

All those operations should happen in memory assuming you installed enough physical memory in your system (8 GB minimum, 16 GB recommended) and are not running a lot of memory hungry background applications (32 GB or more needed for this). The only time I/O should occur is when the file has completely been written, in which case the OS will eventually flush the file cache and write the save file out to disk. Of course if write through mode is used for I/O then the file cache is ignored, but this should only be used when manipulating complex persistent data files that are manipulated in small parts and prone to corruption if changes are not written out consistently such as databases. Save files are generated fresh each time so this is not really a concern, with maximum data loss period likely being only a few seconds after the save completes before the OS efficiently writes it out to disk from the file cache.

The I/O will likely involve kernel level copying. Direct read and write through has very specific memory alignment requirements and given how small the resulting save files are, this is unlikely to make much of a performance difference. It also can lower performance since it does not allow the file cache to work efficiently and schedule the data to be written out asynchronously later (a few seconds at most after the file is written).

[dtpsprt]

The OSes support virtualization, but it is not enabled by default. Enabling it does come at a performance penalty since then a hypervisor is used. This mode is needed to run things like other virtual machines and is how native Linux compatibility was implemented on Windows 10.

But Windows does it automatically. It's the way the "fine" programmers of Microsoft found to avoid, or at least minimize, the "Blue Screens" and halting of the system (obviously urged to come up with something after Bill Gates' fiasco in the presentation of Win 3.0. It is the reason that when your Windows "hangs" and you press Alt+Ctrl+Del he system does not reboot but takes you to the logon screen instead.

Windows does not work like that.

The OS manages all physical memory as all modern OSes do (such as Linux and OSX). It then assigns each application running on it a private virtual memory address space. Virtual memory is a long existing feature supported by high performance processors that allows the creation of a custom and reprogrammable address space that is not directly related to the physical memory address layout of the system. Virtual memory is managed in a unit called "memory page" which represents some uniform amount of memory. The OS is then responsible for managing the virtual memory of each application, keeping track of and setting up mappings from virtual memory to physical memory. Since pages do not benefit much from being organized in a continuous way in memory and are uniform this makes managing memory and giving it to applications a lot easier and is one of the main reasons this technique is used.

Since memory pages can be moved around in physical memory easily, it becomes possible to write out memory pages to some persistent or bulk storage medium to free up physical memory. Hence why all modern operating systems use a page file, which can be used to hold memory pages when available physical memory is depleted and as such avoid out of memory crashes. Logically bulk storage is a lot slower than memory, and as such a victim algorithm is used to select pages that are infrequently touched in preference to ones frequently used to minimize the performance impact. When an application touches an address that is currently not resident in physical memory due to being paged out to the page file, a page fault interrupt is generated that the OS processes and uses to move the corresponding memory page back into physical memory before resuming application execution as if nothing had happened. Since using the page file does have an overhead, the OS tries its best to avoid using it when possible by first mostly filling physical memory before starting to evict infrequently used memory pages to the page file.

This also allows a different kind of I/O, memory mapped I/O. An application can request that part of its virtual address space be mapped to parts of arbitrary files within the file system as long as alignment requirements are respected. Since file data becomes memory pages, it also means that a feature such as file cache becomes trivial to implement. This allows frequently accessed file data to be stored in memory pages so when an application requests it then either the data is copied directly in bulk to the destination address, or if memory mapped I/O is used then the memory page is trivially remapped to the application address space without any copying occurring.

Since the Kernel, and the drivers running in it, have to also use memory there is special handling of such memory separate from applications. For example pages might be locked down and forced to reside in physical memory for correct DMA functionality, such as used by underlying I/O. This applies to all Windows drivers, third party drivers and the kernel itself. It does not apply to Microsoft applications, which run like all normal applications using standard user space virtual memory. Some Microsoft applications which have associated drivers might use this functionality within the driver part, but the user space application part will be like any other application. The logical reason for this is that Microsoft designed the OS and features of it around supporting applications like their own, so they do not need special treatment as what is provided to every application is good. Some Microsoft applications interface with the OS using obscure or non-public APIs, but this is not related to memory and more to OS interaction.

Anyone tried to ever discover what those obscure and non-public API's are? You'll find my words coming true in there. It preloads the full Explorer (Edge in Win 10) and the whole Microsoft Office, except Microsoft Access.

Which brings us to:

All those operations should happen in memory assuming you installed enough physical memory in your system (8 GB minimum, 16 GB recommended) and are not running a lot of memory hungry background applications (32 GB or more needed for this). The only time I/O should occur is when the file has completely been written, in which case the OS will eventually flush the file cache and write the save file out to disk. Of course if write through mode is used for I/O then the file cache is ignored, but this should only be used when manipulating complex persistent data files that are manipulated in small parts and prone to corruption if changes are not written out consistently such as databases. Save files are generated fresh each time so this is not really a concern, with maximum data loss period likely being only a few seconds after the save completes before the OS efficiently writes it out to disk from the file cache.

The I/O will likely involve kernel level copying. Direct read and write through has very specific memory alignment requirements and given how small the resulting save files are, this is unlikely to make much of a performance difference. It also can lower performance since it does not allow the file cache to work efficiently and schedule the data to be written out asynchronously later (a few seconds at most after the file is written).

This is how Linux operates and the reason it is measurably faster using a simple wristwatch.

BTW if Microsoft used MOFT (the name of their stock in the stockexchange) instead of Microsoft Corporation inside their files, Windows would have been 5% smaller (not a small feat) and run 2% faster, since all the DLLs in the Windows folder along with the ones in other Microsoft Programs folders the moment they are touched they ask each other (honestly all of them) if they have Microsoft Corporation written in them. If you remove the "unnecessary" checks (and only them, like the DLLs that do not belong in the list of the program) then you get a 15% speed up.

Been there in person when these "experiments" were made and saw the difference with my own eyes...

[CBJ]

Since this has basically nothing to do with the game any more, over to off-topic it goes.

[Imperial Good]

But Windows does it automatically. It's the way the "fine" programmers of Microsoft found to avoid, or at least minimize, the "Blue Screens" and halting of the system (obviously urged to come up with something after Bill Gates' fiasco in the presentation of Win 3.0. It is the reason that when your Windows "hangs" and you press Alt+Ctrl+Del he system does not reboot but takes you to the logon screen instead.

You are referring to the Windows kernel and its various security features, most of which are shared with all other user orientated operating systems. The kernel runs in a special mode separate from applications for security reasons, and has permission to use instructions that applications cannot. This is all required so that applications are self-contained for security reasons so that if the application crashes it will not crash other applications and the applications cannot look at what other applications are doing. If one looks into the Linux Kernel it does exactly this, making sure each application is decoupled from each other. A bid issue with Spectre/Meltdown was that it could be used to break out of application barriers and read content from memory the application does not have access to. Even modern video game consoles run like this to a large extent.

This added abstraction is required and always present irrespective of user OS used. To get around this Kernel level APIs are needed that allow more low level control over some peripherals while not exposing the kernel or other applications to additional risk. An example of this is seen in the Vulkan API where one can issue GPU commands via queues to minimize the overhead of the Kernel level graphic drivers.

Hitting Alt+Ctrl+Delete brings up the logon screen for convenience. A user may want to rapidly log out or lock the system, such as at a computer cluster at a university. This is mostly for convenience. Thanks to improvements with GPU scheduling made with Vista, it also bypasses applications that are in exclusive full screen mode making it useful to get to task manager (set to be always on top) to close such applications in the case they freeze. Many people used to have to restart the system due to applications freezing in exclusive full screen mode since the OS could not present them task manager due to the application having control over the display.

Anyone tried to ever discover what those obscure and non-public API's are?

Mostly related to internal Kernel metrics, often performance counters. For example memory management data, application performance details, application loaded dependencies, e.t.c. These have no use to most application developers as they are generally only useful for Windows monitoring, management and control features.

A good example is RAMMap from System Internals (owned by Microsoft) which shows the current allocation of your system memory from the kernel's perspective. This includes what files are in the file cache, how much memory is in what state and much more. Very useful if you are trying to understand why an application is using a lot of memory, or why the OS is reporting a lot of memory in use. It even can order the OS to flush all ready to free pages from memory which is useful for I/O performance testing from a not in file cache state for consistency.

Another example is Process Explorer which is basically a better task manager for inspecting applications. This reports details on a per application basis including details such as what DLLs the application is loading, what TCP/UDP ports it is using, how much CPU time the application has been scheduled in total, how many page faults the application has caused, and much more. Very useful to help resolve the type of I/O an application is using as memory mapped I/O usually causes a lot of page faults to be logged but very few actual bytes to be read using traditional I/O techniques.

It preloads the full Explorer (Edge in Win 10) and the whole Microsoft Office, except Microsoft Access.

Edge is not Explorer, they might look and act similar at times and work hand in hand but they are completely separate processes/applications. Explorer is much more light weight and cannot load web pages.

The pre-loading you are describing is also a general feature of modern Windows that applies to both Microsoft and third party applications. In fact it happens to every executable, even installers, that the user runs. The operating system tracks the most recently used 200 odd applications and their dependencies in a data base and using that database loads their program files into the file cache on start-up. This helps improve responsiveness as when a user runs their most commonly used applications a lot of their data and code is already in the file cache and so will load much faster with much less I/O. This also gives the user some benefit from having a lot of memory, since otherwise most of the memory would be unused most of the time. If an application needs a lot of memory, then the file cache will evict the memory pages of these currently not running applications in preference of memory pages that are owned by running applications. An issue with the pre-caching is that it can poison the file cache with useless files the user is unlikely to touch. Another issue is that if the program files are on a mechanical HDD they might require significant time to load at start-up during which time I/O performance with the drive will be reduced. I personally wished there were better user features to manager this recently used application cache behaviour, such as limiting the amount of memory it will use, removing specific applications from it automatically when not used for a month or so, or even turning it off altogether since it offers little benefit to applications installed on performance NVMe drives due to how fast these drives are.

BTW if Microsoft used MOFT (the name of their stock in the stockexchange) instead of Microsoft Corporation inside their files, Windows would have been 5% smaller (not a small feat) and run 2% faster, since all the DLLs in the Windows folder along with the ones in other Microsoft Programs folders the moment they are touched they ask each other (honestly all of them) if they have Microsoft Corporation written in them. If you remove the "unnecessary" checks (and only them, like the DLLs that do not belong in the list of the program) then you get a 15% speed up.

They are not checking for having "Microsoft" in them. They are checking if the DLLs have been modified by checking a cryptographic hash against a database for what is expected with that file. This is a security feature required to help detect and prevent hackers running malicious kernel code. If a file is found that fails these checks then the OS will refuse to load it, and potentially refuse to start. This is much better than letting the OS run up with an infected file and then exposing the user and all their data to a compromised kernel. This will have no affect on application speed outside of loading. Memory in general has no inherent speed overhead, although accessing a lot of it frequently can decrease CPU cache efficiency which does have a performance impact.

[dtpsprt]

Edge is not Explorer, they might look and act similar at times and work hand in hand but they are completely separate processes/applications. Explorer is much more light weight and cannot load web pages.

I was referring to Microsoft's Internet Explorer. The File Explorer is a slightly castrated version of it (it actually works better as a web browser than the full shebang), sorry I thought it was obvious since right after I put Edge.

...
They are not checking for having "Microsoft" in them. They are checking if the DLLs have been modified by checking a cryptographic hash against a database for what is expected with that file. This is a security feature required to help detect and prevent hackers running malicious kernel code. If a file is found that fails these checks then the OS will refuse to load it, and potentially refuse to start. This is much better than letting the OS run up with an infected file and then exposing the user and all their data to a compromised kernel. This will have no affect on application speed outside of loading. Memory in general has no inherent speed overhead, although accessing a lot of it frequently can decrease CPU cache efficiency which does have a performance impact.
...

Very nice verbalisations and eloquent (in computer terms) justifying "We'll chop your head off to keep your body healthy". Everything falls when "security" gets involved.... Reminds me of nullifying piloting skills because some people were ordering 100 scouts to get some 15 3* pilots and then just sold the rest back, which the game's mechanics consider as "destroyed" (not resold or recycled for it's raw materials etc) eventually halting the economy...

I could go on with some remarks on the rest of your post but it really takes everything out of the topic.

[Panos]

However on Linux the saving if way faster than on Windows 10 without doing anything else.

This is really strange. I/O calls alone on Windows do not make much of a difference with 3.2 GB/sec of random access easily being possible from within something like Java which is not tightly optimized to Windows. Unless the way saving happens is not at all optimized for Windows, which would mean a lot of small or deprecated I/O calls or I/O that does not take advantage of the file cache. Otherwise this would point towards optimization differences with the output binaries from the compiler setup.

Do you have example save times? I could imagine the difference between Windows and Linux being 5% or so given the same hardware is used but anything larger would point towards some other cause of slowdown.

When build my system again I will do so. Currently moved back to UK for work and the PC is just motherboard and graphic card PCBs in the suitcase

However the idea to speed up the saving of the game is common knowledge over a year now. Folder Indexing and Windows Defender Antivirus hit had the whole process. Removing save folder from the indexing and restricting the antivirus greatly cuts the time. Is very observable if you try it and only few clicks.

On my Manjaro disk I am using ext4, not one of the newer faster Filesystems. However was surprised how even more faster this is done than windows. Same save file. (I copy them across the different OS when needed but using predominately Linux for X4 now).