Can I pay for a FPS DLC?

[Lord Dakier]

A single-core running at 12ghz would perform far, far better than a quad-core each running at 3ghz even if the game was multithreaded in its current design as much as possible.

You might want to look in how particle simulation on GPU works with modern graphic engines. It comes very close to simulating thousands of small ships in a universe.
Also the concept of one core waiting on another is exactly what you would want to avoid when implementing a proper algorithm.

So my point stands: the algorithm they are using is simply bad.

Well I'm sure Egosoft will be snapping you up to work in their office then. You clearly know more than they do.

[TwoThe]

There is no need for people to starting flinging insults, either towards the developers or at one another.

Sorry for that, I edited my post.

makes it extremely difficult to do efficiently any other way

Difficult maybe, but certainly not impossible. A step based approach with task queues should do the trick rather efficiently with modern CompareAndSwap.

[CBJ]

You might want to look in how particle simulation on GPU works with modern graphic engines. It comes very close to simulating thousands of small ships in a universe.

No, it really isn't. Particles just move according to a simple formula. They don't have orders, interactions with other objects, interactions with the player, or any of dozens of other features that ships need to have. The closest thing to what you describe there is the game's "mass traffic" system, which does indeed work a bit like a particle system, at least until you get close enough to be able to interact with the individual ships in the traffic.

Also the concept of one core waiting on another is exactly what you would want to avoid when implementing a proper algorithm.

I don't know what point you are trying to make here. Cores occasionally do have to wait for one another; this is known as a sync point, and happens, for example, when you need to ensure that everything is in a consistent state at the start of rendering a frame. Other than that, what makes you think that threads are waiting for one another? They're not.

[TwoThe]

Particles just move according to a simple formula.

Which is what ships do most of the time: they move. If you have a ship in combat with a 1s time to fire, that means that 59 Frames the ship is moving according to a particle-like formula, and 1 frame of actual interaction. Most ships however are not in combat, they simply move around in formation doing 99.9% particle stuff.

I don't know what point you are trying to make here. Cores occasionally do have to wait for one another; this is known as a sync point, and happens, for example, when you need to ensure that everything is in a consistent state at the start of rendering a frame. Other than that, what makes you think that threads are waiting for one another? They're not.

That is the old thread based approach to multithreading, while the modern way would be a task-/queue-based approach.

To illustrate that with an example: let's say two ships want to trade.
Now as I said above all the movement can be handled by a particle-like system, so in full parallel. What's left is the actual trading:
1) Ships A injects a trade request into Ship B's task queue.
2) Ship B injects a "trade accepted" return into A's task queue (or otherwise "trade refused"), containing a rendezvous point, which is an arbitrary object handling the actual trade interaction.
3) Ship A and B do "particle stuff" till they are in place, once they arrive they send an "In position" note to the arbitrary trade handler.
4) Once both ships have sent that they are in place, the trade handler queues a "wait" task into them - which will simply make them wait until further notice - and also takes the cargo to trade for either transfer or dump.
5) Once a specific time has passed the trade handler sends the cargo to the receiving ship's queue and released them from their waiting.

Now the rendering only needs to know the position and heading of the ship, which is - thanks to the particle system - always known. The actual trade interaction requires no rendering and can therefore be done off the rendering thread. If the player is looking the trade handler could also have some fake "cargo drones" fly back and forth, which actually do nothing but look good and drop a random part of the cargo in transfer when destroyed.

This form of task/queue based interaction is used in all modern logistic systems around the world, which is by nature "multi-threaded," and has proven to be very reliably and fail-safe.

[CBJ]

Which is what ships do most of the time: they move. If you have a ship in combat with a 1s time to fire, that means that 59 Frames the ship is moving according to a particle-like formula, and 1 frame of actual interaction. Most ships however are not in combat, they simply move around in formation doing 99.9% particle stuff.

No, they don't.

That is the old thread based approach to multithreading, while the modern way would be a task-/queue-based approach.

The part of the game that I described in the text you quoted is not something that can be farmed out to a task-based approach. Sync points are still needed between, for example, the render thread and the simulation thread, which is the scenario I was talking about. What you describe in your example on the other hand could indeed be done with a task-based system in theory, but is also exactly what I was referring to when I talked about it being extremely difficult in a game with so many interactions. The overhead of a task-based system in a game like this, for the particular areas we are talking about, can actually end up exceeding the benefits.

Perhaps if you didn't enter into this thread assuming that the developers were idiots, that you knew more than they did about the systems that they have been working with for many years, and that they hadn't even considered fairly standard programming techniques, you would find people more receptive to discussing things with you. As it is, I'm not going to spend any more time on this.

[TwoThe]

Perhaps if you didn't enter into this thread assuming that the developers were idiots, that you knew more than they did about the systems that they have been working with for many years, and that they hadn't even considered fairly standard programming techniques, you would find people more receptive to discussing things with you. As it is, I'm not going to spend any more time on this.

I am sorry for offending you. I think you are right, my initial post was inappropriate.

Sync points are still needed between, for example, the render thread and the simulation thread

The render thread needs to know the position and orientation of the object to render. If that is written in one object and then set as an attribute on the ship itself, it is essentially a pointer swap. Any pointer swap always happens atomically on every system, so this does not need a sync point of any kind.

[Imperial Good]

Which is what ships do most of the time: they move. If you have a ship in combat with a 1s time to fire, that means that 59 Frames the ship is moving according to a particle-like formula, and 1 frame of actual interaction. Most ships however are not in combat, they simply move around in formation doing 99.9% particle stuff.

Particles have a regular shape, usually a sphere of sorts. Ships do not. This means that collision detection, which needs to be handled for all ships nearby the player, has to be run constantly and using more complex logic. GPUs execute conditional logic very poorly as they have to use the worst case time that occurs if the longest conditional branch is taken. Even if this approach can work, it then takes up GPU time potentially pushing bottlenecks to the GPU. As it stands my GTX 760 is bottlenecking X4 performance quite a lot.

The render thread needs to know the position and orientation of the object to render. If that is written in one object and then set as an attribute on the ship itself, it is essentially a pointer swap. Any pointer swap always happens atomically on every system, so this does not need a sync point of any kind.

That is not portable. Since unless the memory model of the programming language used explicitly states that "pointer swaps" are atomic then there might be an implementation that is not. This also does not address the issue that the actual position and orientation data still has to be synced between cores using a sync point to keep caches consistent. Even if the pointer to that data is synced atomically it does not mean the data itself is synced. As soon as you make general regions of memory consistent between threads you are effectively performing a sync point of some kind, and all the overhead that entitles.

Ideally something like each sector should be able to be simulated on a separate thread with inter-sector interaction being limited to between updates. This would allow for much closer to linear scaling with core count, assuming core compute is the bottleneck. However it also would require a radicle change in the way scripts and the game works, and some features taken for granted might be regressed or have to be removed due to such limitations. For example the script to look for trades or objects in nearby sectors might require an asynchronous native implementation that takes some number of update cycles latency to complete as during each sector update a request message would need to be resolved and the final results collected to be fed back to the script. This is something I would recommend considering for future products where core count scaling becomes increasingly important but would be total madness to try to fit into an existing one.

[TwoThe]

Since unless the memory model of the programming language used explicitly states that "pointer swaps" are atomic then there might be an implementation that is not.

It is by definition that any read/write operation of the native CPU's bit size is atomic. Pointers are always of a size of the native's CPU, therefore pointer operations are always atomic. This might not be true for some exotic CPUs, but unless you plan to run X4 on a typewriter, I don't think that's going to be an issue.

data still has to be synced between cores

True, but that's the CPUs problem and well handled by anything manufactured past 2006. You can enforce thread visibility by using std::atomic<T> (or equivalents), which will automatically solve that issue.

Ideally something like each sector should be able to be simulated on a separate thread

This concept has been implemented by many games (Eve Online comes to mind) with always those same issue: as soon as one sector is overloaded the game starts lagging. And that can happen often if massive incursions occur, or simply because players love to make huge trading stations.

[Imperial Good]

It is by definition that any read/write operation of the native CPU's bit size is atomic. Pointers are always of a size of the native's CPU, therefore pointer operations are always atomic. This might not be true for some exotic CPUs, but unless you plan to run X4 on a typewriter, I don't think that's going to be an issue.

Only if the native sized read or write operation is cache aligned. If it is not then it might not be written out consistently like that. This also does not apply to native vector read or writes which are often used in optimising loops, memory moves, and other operations. For write operations it is only atomic with regard to the current core, requiring a sync point of sorts to make the change visible to all other cores.

Some instruction sets have optimised atomic write instructions for primitives like integers which are more efficient than just a general flush but still slower than not having to flush at all.

True, but that's the CPUs problem and well handled by anything manufactured past 2006. You can enforce thread visibility by using std::atomic<T> (or equivalents), which will automatically solve that issue.

Atomic variables have a large write overhead compared with non atomic variables because they are effectively enforcing a kind of sync point on write. If you relax the memory model to remove this you then have core consistency issues and so race conditions, and might as well not use atomic variables to begin with. It is almost certainly more efficient to update multiple variables normally and then flush with a sync point after than to do so for each variable.

Your original suggestion still would require sync points between updates since the render threads would need to see all objects consistently for a single update cycle. Until it has finished sampling all the required ship attributes the update thread cannot update those attributes with new values, even if all it does is a simple pointer swap to do so, otherwise a race condition will occur where the render threads renders a frame using attributes from more than 1 update cycle.

Note that in context of this discussion I am using a sync point to refer to any kind of instruction or sequence of instructions that enforces memory consistency between cores. These can be implemented by many kinds of language feature or construct such as atonic variables, fences, barriers, mutexes, e.t.c.

[TwoThe]

Only if the native sized read or write operation is cache aligned.

Any reasonable CPU implementation would have the cache size to be a multiple of the native bit size.

This also does not apply to native vector read or writes

That's why I said pointers.

Atomic variables have a large write overhead compared with non atomic variables

True, but if you use it where it is needed, you still have the multi-core benefit, without any dramatic impact on performance. If that wouldn't be the case then all multi-threading would be in vain, and for that reason those parts have been massively optimized in modern CPUs. If you compare massive calculation with and without such sync points, the performance loss (single-threaded) is less that 5%, while you gain (multi-threaded) another 90%-100% more performance per core if done right.

And I am not saying that this is easy, I am just saying that it is possible, and many modern games have demonstrated that. Take Noita as one fine example: a game where literally every pixel of the (massive) game world is physics simulated, and can of course interact with the player and the environment. The calculation is offset to the graphics card, therefore massively multi-threaded. And it works just fine! There is no reason that cannot be done in X as well. Probably not X4 as that would mean a major rewrite of everything, but possibly in an X5.

If you relax the memory model to remove this you then have core consistency issues and so race conditions

Modern algorithms try to work lock-free if possible, that means without blocking sync points but e.g. pointer overwrites or compare-and-swap. And obviously also without race conditions. It does require a different approach to what has been done traditionally, however the benefit is tremendous.

Take 4 threads that work through an array of items like a bunch of ships maybe: they read the specific item, modify it, write it back into the array. Traditionally approaches would have to gate all access to that array so no thread writes into the result of another. However with a very simple change that gate can safely be removed: if every thread only picks every 4th item, they are guaranteed to never come in touch with each other, no sync point needed at all. Or use an atomic counter with a compare-and-swap implementation that guarantees unique access to every individual item without blocking, with the only sync point being that one single counter. The cost for that is negligible, the gain is massive.

Your original suggestion still would require sync points between updates since the render threads would need to see all objects consistently for a single update cycle.

That is the point: it does not. A frame is rendered with positions (if we stick to those for now for the sake of the argument) of an unknown, but recent moment in time. Conceptually that is like playing on a server with negligible lag (less than 16 ms), which shouldn't be an issue for anyone. The benefit though is a much smoother experience overall. All you need is a proper decoupling of the game logic from the rendering. Again much like playing on a server, which has been demonstrated to work just fine by countless games.

[Imperial Good]

Any reasonable CPU implementation would have the cache size to be a multiple of the native bit size.

Yes however both reads and writes do not need to be aligned to cache or memory. Compilers do try to keep them aligned for optimum performance but there are cases where unaligned reads and writes can happen and even benefit performance.

Take 4 threads that work through an array of items like a bunch of ships maybe: they read the specific item, modify it, write it back into the array. Traditionally approaches would have to gate all access to that array so no thread writes into the result of another. However with a very simple change that gate can safely be removed: if every thread only picks every 4th item, they are guaranteed to never come in touch with each other, no sync point needed at all. Or use an atomic counter with a compare-and-swap implementation that guarantees unique access to every individual item without blocking, with the only sync point being that one single counter. The cost for that is negligible, the gain is massive.

The problem starts when the items start interacting with each other. Since now item 1 might depend on item 2, 6 and 8 meaning that those items cannot be updated by another thread in parallel to item 1 being updated. X4, like a lot of games, is basically a huge tangled web of objects depending on each other. Hence why that approach is not already being used. It is used where possible, but not all places are possible. Where it is being used is likely so performant and efficient that you do not notice it being used.

That is the point: it does not. A frame is rendered with positions (if we stick to those for now for the sake of the argument) of an unknown, but recent moment in time. Conceptually that is like playing on a server with negligible lag (less than 16 ms), which shouldn't be an issue for anyone. The benefit though is a much smoother experience overall. All you need is a proper decoupling of the game logic from the rendering. Again much like playing on a server, which has been demonstrated to work just fine by countless games.

The positions have to be consistent with respect to each other which mean they have to be a snapshot of a specific point in time. If they are not the behaviour is entirely down to race conditions and can introduce a lot of very hard to diagnose bugs. People will report all kind of visual bugs such as inconsistent mesh placements, movements, particle effects, e.t.c. that cannot reliably be reproduced by the developers due to being the product of race conditions. This is why all sensible software developers avoid race conditions. Sure on consoles some developers did rely on race conditions and it did give them maybe a small performance edge, but that is also why their games emulate very badly and might even have weird unexplained bugs on different revisions of real hardware or firmware.

Multiplayer does not rely on race conditions. Updates issued in multiplayer occur in a well defined order as communicated and ordered by the server. The client then applies these in an engine which simulates them in a well defined order. The server issued updates might drastically change the simulation state in a time dependent way, but again there is a well defined order to the process. The client does not try to mix data that is older with data that is newer, instead using a snapshot of the current simulation state as corrected by the most recently sent updates.

[Blitz4]

The fastest cpu in the consumer world

5800X3D is only the fastest for some applications like X4 (possibly some situations only). Multi core performance wise it is much slower than the 5900X and 5950X. It also loses out massively in non-cache constrained single thread performance to most of the Intel 12th gen series i5 and above with the higher end models even beating it multi thread performance wise.

These benchmarks. They weren't even in combat, just flying around some static meshes that don't move.

The meshes all have collision. 3D collision is quite complicated. Each station module has dozens of collision surfaces to match up with their geometry. Repeat this for hundreds of modules and performance can quickly tank. Even optimising the collision testing to cull all inappropriate meshes becomes resource intensive when having to deal with thousands of such meshes nearby.

Thank you. This post is brilliant. This is what I was hinting in the OP, wanting to know the why. I could think about the why while playing, which would cause me to envision solutions or be more invested in doing so. Not saying anything will come out of it, but it's better to be in a mindset of knowing where the splinter is and being able to see it and pretend I could envision creative solutions to remove it vs being in pain with no clue why.

I've been trying to get into the game again. Over 2000 hours in X3, 200 hours in Rebirth and 150 hours in X4. In X4, there's something with the sub-60 frame rate that's making it hard for me to stare at the screen. It only happens when walking on a station, flying around it doesn't affect me as much. That's why I used the word pain above. There's gotta be a solution.

Are there any graphical settings to help with that? I guess it's motion sickness. I've lots of experience with fixing it in other games, I disabled chromatic abb, I don't see any motion blur, or head bob. That only leaves a sub-60fps and AA/ghosting.

Why is the frame rate so much lower when walking around on a station vs flying?

Since there's no immediate fix, Is there a way todo all of the things needed when walking on station, but from a ship? If I never walked around on a station again, what would I be missing? If I wanted a full experience, what would the bare minimum I would need todo on foot that isn't possible in space?

How can fps be increased using mods, settings or cl flags? One mod mentioned to remove fog. What other mods are there, wasn't there a traffic reduction mod or another to optimize your settings?


I took a screen after letting the sim run for a while, my settings:
https://i.imgur.com/5fPO1oV.jpeg

Now, I can tell, compared to X4 at launch, there's been so much done to improve the game. screen is of MSI Afterburner, polled at it's fastest 100ms that span roughly 12 seconds. There's one stat that I disagree with, the 1% Low, FPS, that should be 10% Low, FPS as we can't see 1% Low, but there's no way to add that in MSI. Only other program I know that does that is Special K, which polls much faster, but that junk only does Direct X. Remember, my goal is sub-20ms.https://i.imgur.com/rwgJJqW.png
https://i.imgur.com/YQVDl52.jpg

Interesting shot in the menu, with the game is paused, the sub-50fps and 8fps 1% low is weird. If you don't notice input lag sub 20ms, why isn't he menu hitting that with the sim paused?
https://i.imgur.com/74jbNnY.png
https://i.imgur.com/7DhPW7M.jpg

Yes the AA is likely killing it. The issue is if I turn AA off, I'm not always sub-20ms. And combine low framerate with the jagged edges, it's motion sickness. I'll have to play around a while to find a balance. But I just don't want to deal with it. I'd like a way to not have to walk on a station again, because the settings I'm playing with in game and in nvidia control panel are ONLY when walking on foot, in space I don't need all of these settings maxed out and there's no easy way to toggle them.

Post your DXDiag. At least that way we can identify whether the performance you are seeing is to be expected from your hardware or not. If it's not then we can try and identify what's wrong.

Ok
DxDiag
vulkaninfo
(rig is a sworkstation being used for gaming until next gen gets here)
3900x, 2080 MSI Gaming X Trio 8GB solid@2040/8000, 128GB ECC 3200, sn850, asus ace, Arctic 420, 3x 27gn750-b (2 DP landscape, 1 HDMI portrait)
Win10 Pro on metal, all settings in nvidia control panel -> max quality, ryzen balanced power profile, nvidia broadcast routed to TOSLINK routed to external DAC


EDIT: Sorry a paragraph didn't save. John Carmack stated that we notice input lag above 15-20ms and even then he said that the average person is around 20ms.

[Imperial Good]

Why is the frame rate so much lower when walking around on a station vs flying?

Complex collisions of the platform, as well as NPC path finding and the like. Platforms are also probably not the most optimised they could be given that they are proportionately little of the gameplay compared with the space part (X4 is a space sim not a FPS).

For people used to high refresh rate FPS games I could see it being a problem, however you are not required to react quickly so in worst case should be able to move the camera slower to compensate.

Since there's no immediate fix, Is there a way todo all of the things needed when walking on station, but from a ship? If I never walked around on a station again, what would I be missing? If I wanted a full experience, what would the bare minimum I would need todo on foot that isn't possible in space?

Some missions and storylines require talking to people on stations to progress them. Outside these extremely brief times you need to be on a station or another ship most things can be done from space. There is also your personal office in the HQ where you can see various trophies you earned during the playthrough but you are not required to ever visit it.

3900x, 2080 MSI Gaming X Trio 8GB solid@2040/8000, 128GB ECC 3200, sn850, asus ace, Arctic 420, 3x 27gn750-b (2 DP landscape, 1 HDMI portrait)

I am guessing the ECC memory might not have the tightest timings which could explain slightly worse performance than other people using a 3900x. Memory timings are usually not a high priority for workstation memory. There is a good chance that a 3D cache version of Zen4 processor with decent DDR5 memory could give you the frame rates you crave. Even a 5800X3D might get pretty close although the memory timings might still hold it back.

[Raptor34]

Why is the frame rate so much lower when walking around on a station vs flying?

Complex collisions of the platform, as well as NPC path finding and the like. Platforms are also probably not the most optimised they could be given that they are proportionately little of the gameplay compared with the space part (X4 is a space sim not a FPS).

For people used to high refresh rate FPS games I could see it being a problem, however you are not required to react quickly so in worst case should be able to move the camera slower to compensate.

Since there's no immediate fix, Is there a way todo all of the things needed when walking on station, but from a ship? If I never walked around on a station again, what would I be missing? If I wanted a full experience, what would the bare minimum I would need todo on foot that isn't possible in space?

Some missions and storylines require talking to people on stations to progress them. Outside these extremely brief times you need to be on a station or another ship most things can be done from space. There is also your personal office in the HQ where you can see various trophies you earned during the playthrough but you are not required to ever visit it.

3900x, 2080 MSI Gaming X Trio 8GB solid@2040/8000, 128GB ECC 3200, sn850, asus ace, Arctic 420, 3x 27gn750-b (2 DP landscape, 1 HDMI portrait)

I am guessing the ECC memory might not have the tightest timings which could explain slightly worse performance than other people using a 3900x. Memory timings are usually not a high priority for workstation memory. There is a good chance that a 3D cache version of Zen4 processor with decent DDR5 memory could give you the frame rates you crave. Even a 5800X3D might get pretty close although the memory timings might still hold it back.

So for best performance I should be sitting at the pilot's seat? Don't they render your crew wandering around anyway, even though they are behind you?

[CBJ]

Don't they render your crew wandering around anyway, even though they are behind you?

No; frustum culling ensures that that doesn't happen. There is still some calculation cost, because we have to first work out where they are and make sure it's not in view, and this can be further complicated in some cases by the need to calculate whether an object's shadow is in view, but objects behind you are not themselves generally rendered.

[af_2017]

Started new game recently, X4:foundation without dlcs.
I suspect that OOS activity is less in the case.
Game loads/saves faster, which is explainable.
In sector activity (ships, pew-pewing) still impacts performance, fps lowers.

So I suspect each next content dlc (with new sectors, factions etc) affects game performance and eventually will make the game not playable performance wise.

[-=SiR KiLLaLoT=-]

I took a screen after letting the sim run for a while, my settings:
https://i.imgur.com/5fPO1oV.jpeg

SSAA 4x is a really heavy setting.
I would try 4x MSAA with FRS on high quality, or the same setting but with 2x SSAA.
Also disable chromatic aberration because it is a feature that blurs the edges of the screen too much.
What can greatly affect performance around stations is definitely the space reflections.

Maybe you could improve something but it will probably still be far from the performance you would like to achieve

[-=SiR KiLLaLoT=-]

So I suspect each next content dlc (with new sectors, factions etc) affects game performance and eventually will make the game not playable performance wise.

Likewise it is likely that subsequent content has not yet been as optimized as the initial content, which in the future may improve the overall experience instead of making it worse...

[af_2017]

Likewise it is likely that subsequent content has not yet been as optimized as the initial content, which in the future may improve the overall experience instead of making it worse...

I assume that "new content" is pure data which existing engine processes.
In ToA we've got changes in economy with scrap thingies but I would not expect much performance loss because of this.
And since "new content" is pure additional data, like objects, stations, ships and faction this results in more calculations.
That makes me think that whole engine requires optimizations. From this point of view I would even say that it is impossible to optimize one dlcs separately.

[-=SiR KiLLaLoT=-]

I'm not sure if that's the case. For my little knowledge I have the impression that "engine optimization" and "content" are two distinct and separate things.