Hint about next patch from Egosoft? CBJ just posted this...

[clj]

We all have been biting our nails waiting for ANYTHING from Egosoft, seeing as it has now been a month since they have released a new patch, and the game is still badly broken as of 1.25. I thought I would let everyone know that CBJ just posted this, which appears to be the first hint about the next patch:

http://forum.egosoft.com/viewtopic.php?t=364649

The specific dump file you've provided there shows the game running out of memory (that's memory accessible to a 32-bit application, not your PC as a whole which of course has plenty).

We have done some work relevant to this for the next patch, both generally in the game and specifically at the point of saving, so your best bet will be to join the Patch Public Beta and have a go with the new version, which should be along fairly soon.

This would seem to suggest that the next patch may include a 64 bit .exe (which could potentially bring MAJOR performance improvements). I am very curious what others here think "fairly soon" means, though. Do you think we will be able to play with the new beta this weekend, or is that just wishful thinking on my part?

[Skism]

I am actually thinking of giving this game another go soon, so fairly soon could well mean a week or so...although I stress that is purely me stating opinion.

[Nikola515]

For me the main problem is station management. There is no point building station or even having them because we don't have any way of controlling them and managers AI is too stupid to do simple tasks. So fixing game and adding new things is all nice, but without fixing the main game features(like economy/trading or mining) there is no point to staring again(for me anyway). Economy and trading is why I like X games without that X games are pointless.....

[graphicboy]

Conspiracy theory? Perhaps turning on all of the AI at the same time required resources that were well beyond the 32-bit EXE's limitations, but it was too late to make the switch, so they "broke" the AI, flagged for LAA and released.

I like that theory. It's a lot more palatable than previous ones.

And since all we have are theories, this is as good as the next.

[clj]

@ graphicboy

I, too, would love it if that were the case, as it would mean that with the release of the 64bit exe, MANY things would finally work. I'm hoping we will have the chance to find out for sure by the end of the week.

Personally, I think it seems more likely that they started running low on money, and had to release SOMETHING in order to get an influx of cash, which they could then use to keep their doors open long enough to fix the mess they initially sold to us. Otherwise, I'm not sure why they wouldn't just postpone the release date a bit. If they had waited another couple of months, and released the game even as it stands now, at version 1.25, things would have been very different. While it still would have had many issues that people would complain about, and be missing a variety of extremely important features, like manual ship and fleet commands, it wouldn't have gotten 1 star reviews from every game reviewer on the web that ever touched it, and Egosoft's fan base would feel far less betrayed. Egosoft had to know that it would be this way if they released a half finished game, so I can only guess that they were forced to by budget issues, or something like that.

[werewolves?]

Conspiracy theory? Perhaps turning on all of the AI at the same time required resources that were well beyond the 32-bit EXE's limitations, but it was too late to make the switch, so they "broke" the AI, flagged for LAA and released.

I like that theory. It's a lot more palatable than previous ones.

And since all we have are theories, this is as good as the next.

This is what I thought. It would explain the lack of smaller ships available to the player and in general trade – the small ship would prop up gaps in the economy

[Graaf]

fairly soon.

soon.

So sometime between now and 7 years.

Considering their once-a-month scedule you probably have a new patch by this weekend. But you will have to wait and see if that includes the 64-bit.exe

[Roguey]

This would seem to suggest that the next patch may include a 64 bit .exe (which could potentially bring MAJOR performance improvements).

I think thats highly doubful, as something like a 64bit .exe would take them ages at the current progress. Making XR into a 64bit app wouldnt also make the game faster; there is no reason why a 32bit app couldnt run fast.

The problem is that .exe works on .XML files in runtime, which is a huge drain in resources (as found by people). X-Rebirth is just poorly coded. Even on my site I notice around x300 times increase in load-times when loading from cached pages, compared to building the page with XR in-game XML's game-files. If your CPU in-game is working on 1000's XML's per second, then it will be slow. XR needs to pre-load the XML, then convert them into an array. If the x300 rule works the same, then this would automatically make XR run around 300fps+. When looking at the game, I cant see why XR would actually be much stress for processor. When you look at XR, how can it use multiple cores (unlike X3AP/TC, for me 4 vs 1) but yet run 10th of the speed (and x1000 slower when SETA is at x10 in X3AP).... techially for me X3AP runs fine with 3.4Ghz worth CPU, yet XR with 13.6Ghz CPU (max) is slower. XR is wasting a lot of my CPU grunt.

Yes X3AP used XML, but only for MD, universe, scripting and a few odd things (not all the time). Nothing that demanding. The rest of it (like ships, wares, lasers, bullets etc.) were loaded in from a CSV file (/types/) which is easy to convert into a array. If this is the case, then doesnt fill me much confidence. Performance lost like this are easily to spot, and shouldnt been in release version of the game.

[Holzapfel]

friday

[BigBANGtheory]

The problem is that .exe works on .XML files in runtime, which is a huge drain in resources (as found by people). X-Rebirth is just poorly coded.

I don't see a huge drain on my system but I do see threads and behaviour suggesting they are in wait state longer than is healthy. That might very well be due to XML interpretation I hadn't considered that.

Would the move to 64bit not allow Egosoft the possibility to use more memory in the way your cache analogy showed improvements?

[wysiwyg]

Would the move to 64bit not allow Egosoft the possibility to use more memory in the way your cache analogy showed improvements?

I think this is the key reason to move to the 64-bit exe. It breaks the 32-bit windows .exe constraint of around 3.5GB of memory. Hopefully this means more game resources cached in memory and therefore less of the freezing and stuttering that is currently apparent in highways, cutscenes, busy zones etc.

[CBJ]

The work alluded to in that post is related to reducing some memory overheads, not to the 64-bit version. We are, of course, working on the 64-bit version, as Bernd stated in the most recent update, but that's not slated for release as part of the next patch.

Regarding XML files, Roguey, I'm afraid you are quite simply wrong. Of course XML files are loaded and converted into internal formats for use in-game. We are not continuously reading data direct from XML thousands of times per frame.

[Roguey]

Regarding XML files, Roguey, I'm afraid you are quite simply wrong. Of course XML files are loaded and converted into internal formats for use in-game. We are not continuously reading data direct from XML thousands of times per frame.

Well I premued based on some digging from some guys did awhile back. It showed that 'xmlCleanupThreads' was taking the most CPU power (way more than the D3D calls);

Disclaimer: Unmanaged code is not really my main area of expertise, so take the following with a grain of salt.

I got curious and decided to dig a little deeper. I think the mutex indicated by the wait call is probably a red herring most of the time since it seems like Rebirth has a number of worker threads that are probably just blocked and waiting for work to do. Most other applications running have a bunch of similar calls.

http://s24.postimg.org/9844yvhs5/Rebirth_1.png

The first time I captured data from every thread in the application, and it indicates they are spending a lot of time in the mutex. However as per above I have seen this before with worker threads which are waiting for work (kinda like a blocking queue kind of thing if you know what I mean).

http://s24.postimg.org/7uci3kiit/Rebirth_2.png

This second dump is just from the main thread (which was spinning at 100% on one of my cores I might add). The call for xmlCleanupThreads looks very suspicious to me, obviously I have no idea what it actually does, but...

I guess I'm left wondering what on earth it is doing with xml that means the most expensive 3 things on all the threads in their application are xml related calls?

Ok, more stuff here. The xmlCleanupThreads call is not their code, it looks like an open source library for xml parsing called libxml2.

http://www.xmlsoft.org/

The API documentation for the call in question is available here:
http://stuff.mit.edu/afs/sipb/project/p ... reads.html

It does look like it uses mutex internally...

Now I'm really curious what they are doing with XML that is taking 12 times as long as the D3D calls and requires multiple threads..

The second most expensive call is related to loading XML into a DOM and manipulating it. What on earth would be continuously creating and cleaning up XML parsers and DOM's?

http://www.xmlsoft.org/html/libxml-SAX2 ... etPublicId

full thread here

[CBJ]

The poster did say that that data needed to be taken with a pinch of salt, and he's right. I'm not going to go into detail but asynchronously loading files on asset management worker threads is pretty standard stuff.

Where you went wrong after that is when you made the leap from the game loading XML files to it continuously referencing data contained in those files in XML form every time it wants something from them, which quite simply isn't the case.

[Darrosquall]

more fleets command are in the next beta patch or not?

[DaddyMonster]

This would seem to suggest that the next patch may include a 64 bit .exe (which could potentially bring MAJOR performance improvements).

I think thats highly doubful, as something like a 64bit .exe would take them ages at the current progress. Making XR into a 64bit app wouldnt also make the game faster; there is no reason why a 32bit app couldnt run fast.

The problem is that .exe works on .XML files in runtime, which is a huge drain in resources (as found by people). X-Rebirth is just poorly coded. Even on my site I notice around x300 times increase in load-times when loading from cached pages, compared to building the page with XR in-game XML's game-files. If your CPU in-game is working on 1000's XML's per second, then it will be slow. XR needs to pre-load the XML, then convert them into an array. If the x300 rule works the same, then this would automatically make XR run around 300fps+. When looking at the game, I cant see why XR would actually be much stress for processor. When you look at XR, how can it use multiple cores (unlike X3AP/TC, for me 4 vs 1) but yet run 10th of the speed (and x1000 slower when SETA is at x10 in X3AP).... techially for me X3AP runs fine with 3.4Ghz worth CPU, yet XR with 13.6Ghz CPU (max) is slower. XR is wasting a lot of my CPU grunt.

Yes X3AP used XML, but only for MD, universe, scripting and a few odd things (not all the time). Nothing that demanding. The rest of it (like ships, wares, lasers, bullets etc.) were loaded in from a CSV file (/types/) which is easy to convert into a array. If this is the case, then doesnt fill me much confidence. Performance lost like this are easily to spot, and shouldnt been in release version of the game.

Sure, but there's a limit to how many assets you can preload - which is where the move to 64bit comes in as memory is limited to under four gig. Even if you're not maxing out memory, a sudden requirement to load assets will tax the processor. Handling asynchronous preloading can be a good solution depending on the context and implementation. Sometimes there are just too many xml docs to pre-parse them all and it's impossible to predict which you will need. I suspect the current memory just isn't cutting it so they likely want to go to more memory and preload more. The improvement could be dramatic but that's based on lots of guesswork.

ps. does it really take less resources to parse csv into an array than xml?? The point might be xml tends to contain everything and the csv is divided into discrete data sets so maybe that's it.

[BigBANGtheory]

more fleets command are in the next beta patch or not?

I'd settle for an open discussion on the topic at this point.

[Sam L.R. Griffiths]

ps. does it really take less resources to parse csv into an array than xml?? The point might be xml tends to contain everything and the csv is divided into discrete data sets so maybe that's it.

In short, yes - but XML is more flexible and powerful as a data format - and with the right editors more user friendly. XML need not be costly to process though, the key is to picking your XML parser appropriately and not un-necessarily processing the XML in text form but rather keeping it in the parser's DOM form where possible. In addition, modern CPUs have special commands to assist with the parsing of XML data (which includes HTML I should add).

[khartsh]

ps. does it really take less resources to parse csv into an array than xml?? The point might be xml tends to contain everything and the csv is divided into discrete data sets so maybe that's it.

In short, yes - but XML is more flexible and powerful as a data format - and with the right editors more user friendly. XML need not be costly to process though, the key is to picking your XML parser appropriately and not un-necessarily processing the XML in text form but rather keeping it in the parser's DOM form where possible. In addition, modern CPUs have special commands to assist with the parsing of XML data (which includes HTML I should add).

The csv files before were used almost as read only with no modification to assets... it read what the stats for a Buster was, it didn't save information about a Buster to them.

The XML files today are being used for dynamic data and random generation of assets... CBJ can say he is wrong all he wants but if you look at the actual work the CPU Threads are doing, most of it involves parsing, cleaning and reading XML files or waiting for assets to become available for the next thread to do the same.

[CBJ]

I see we are back to certain people thinking they know better and dismissing everything anyone from Egosoft says as lies or incompetence. Under the circumstances there is very little point in me engaging further in this thread.