[X3LU/MAYHEM][MOD] Assorted technical questions

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 10:47

Hello!

I'm exploring an idea for a mod and am in need of technical feedback from knowledgeable modders about file structures, engine limitations etc... More questions are likely to arise as the concept is explored, so expect to see me posting here for a bit; or not if it ends up being completely impossible.

Thanks in advance,
-lpa

-------

Questions, round 1:

1. How high a number can an arbitrary pageId (ie not accessed by existing code) reach in a Tfile? (I'm somewhat expecting to get 2^31 as a reply

) The answer to that ties in to #3 and #4.

2. Same as above wr/t stringIds per pageId.

3. What is the maximum file size accessible by the X3AP binary? (likely tied to 32-bit but idk if it's 2gb or 4gb)

4. Does the script engine have access to a binary file format for I/O? If not, what would the be best way to store normal ship traffic XYZ coords and associated short strings? (xml parse speed versus tab separated tokenization etc...) Disk through-output considerations can be ignored for now.

5. Can the script engine start an independent external process?

6. Is there a 'tick' speed driving OOS ship movement, and how is it accessed?

7. How fast is the script engine compared to binary? An approximate answer will do.

8. Are the x-y minimum and maximum sector coordinates hard-coded or is just a map object limitation?

9. Does the map coord system handle a Z axis? (just making sure

)

10. Can a sector host more than 4 gates, aside from the Hub?

11. What data is routinely purged vs. constantly loaded objects?
---

...That's all I can think of for now. Thanks again!

Post by **Cycrow** » Wed, 13. Sep 17, 11:04

Ill start by answering a few of these

1. a page id can be a maximum of 9999. The game will only use 4 digits for page ids, the extra digits are used to define the game

so, 0007, 300007, 350007 and 380007 are all the same page id of 7

2. Not sure of the limit on the t ids, most likely a 32bit int limit. There are some pretty big ids in the menus page (35)

8. This is limited to 32bit ints, but its done in game units not m, so in m its abit less, 500 units in a m

9. yes, same restrictions as above

10. the game can handle 6 gates, technically you can add more, but any more than 6 and you will run into issues.

11. Data, like the models, textures, etc are all cached until the game gets close to its memory limit, which is when it purges. It wont purge anything thats currently in the active sector

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 11:15

Thanks for 1, 2, 10 and 11!

8. This is limited to 32bit ints, but its done in game units not m, so in m its abit less, 500 units in a m

9. yes, same restrictions as above

I was actually referring to the galaxy map space (ie kingdom end x0,y0 etc)... Sorry :/

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 11:18

Damn, forgot about this:

12. Aside from script commands that need to reside in memory by nature, how are the rest accessed, purged etc...?

Joubarbe · Post by **Joubarbe** » Wed, 13. Sep 17, 11:35

lostProfitssssArrgh wrote:Thanks for 1, 2, 10 and 11!

8. This is limited to 32bit ints, but its done in game units not m, so in m its abit less, 500 units in a m

9. yes, same restrictions as above
I was actually referring to the galaxy map space (ie kingdom end x0,y0 etc)... Sorry :/

Maximum is 25*20 sectors. That's x:0,24 and y:0,19. Filling that space is really huge and will certainly lead to serious problems. Also, you need to have a 0,0 sector or you will experienced freezes over time. If I remember correctly, Jack once said to me that all your sectors need to be connected in LU if you want your AI ships to work properly.

And if you want to have a correct in-game map of your new galaxy, you need to use this.

Also note that if I never really achieved something with my universe generator tool, that's because the game acted really strange about my random universe: some commands were not working, sectors appeared with graphic glitches, AI ships were not doing what they're supposed to do... In short: expect bugs.

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 11:40

Joubarbe wrote: Maximum is 25*20...

Yes. But where is that limit defined? Galaxy map, core code, something else..?

Joubarbe wrote: And if you want to have a correct in-game map of your new galaxy, you need to use this.

Links are dead

Joubarbe · Post by **Joubarbe** » Wed, 13. Sep 17, 11:42

To me it's not: https://forum.egosoft.com/viewtopic.php ... &view=next.

Hardcoded.

EDIT: uh yeah sorry, the download link is dead indeed. Let me upload that somewhere.
EDIT2: Here you go.

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 12:07

@Joubarbe
Thanks for link.

13. A tough question to formulate : how 'far' in terms of jumps are the NPC ships 'looking'..?
Another way to look at it : are NPC ships only 'aware' of their current sector despite having a destination that can be arbitrarily distant?

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 12:21

Joubarbe wrote: ...that's because the game acted really strange about my random universe...

14. Is it possibly due to static path-finding markers..?

And, answering my own question:

lpa wrote: 3. What is the maximum file size accessible by the X3AP binary? (likely tied to 32-bit but idk if it's 2gb or 4gb)

Technical limits to file size appear to be determined more by the file system than the bit-flavor of the processor (which can work by only extracting chunks of a large file). After consideration, the issue in the context of this projects' needs is mooted since using a larger number of smaller files has its benefits.

Joubarbe · Post by **Joubarbe** » Wed, 13. Sep 17, 13:00

You should probably wait for an answer from Jack; he's the Epic Warlord of X3 (there should be a special badge for that).

Jack08 · Post by **Jack08** » Wed, 13. Sep 17, 14:41

2. Its a signed 32bit integer, which means you can actually use negative t'ids too, -2^31 -> 2^31.

3. While i would be inclined to say thats OS dependent, you still face memory issues with large files, especially if your loading the data into the MSCI as you have to load it all at once from a single file, so if your loading 5gb of data, your likely to instantly crash. Not that i can come up with any scenario where you would need to do such a thing?

4. Using tab separation and splitting the input based on \n, and \t would work ok, in LU, you can read this in raw from any file using "$Data = read file: <file>". In vanilla you would have to embed this data in a textdb file instead.

5. No

6. There is a tick, i dont know exactly what it is. It is "hardcoded", and generally, this concept should be entirely ignored when it comes to scripting as its a lower level thing.

7. The MSCI, is a virtual machine, Wrapped in another virtual machine called KC, as such, Each MSCI instruction, would equate to more then 20-30 (perhaps closer to 40) machine code instructions, so a rough estimation would be 50x slower then C on an instruction amount, however with taking the fact that VM's suffering branch-prediction-totally-useless syndrome this value could be higher (depending on implementation).

While this sounds pretty bad, keep in mind that you can always prolong extensive tasks over a period of time, by doing a single (or a couple) of iterations per frame, using waits - it makes tasks take time, rather then bog down the cpu. Improved Races 2.0 does this in the economy booster, it has a very large analysts phase that can take longer then 15 minutes in some cases, but if run without waits, would freeze the game for a few moments.

8. Depends what your asking about, i am going to assume you mean: Argon Prime has a universal grid coordinate of X,Y - then yes the hardcoded limit is 24,20 (iirc) and no, a Z coordinate cannot be added to this to create additional sector dimensions, however...

9. While you cant give sectors a Z coordinate when ti comes to addressing the actual sector, a Z coordinate can be added to the universe "bod" file, which is what visually represents the universe in the universe map, this is an entirely visual representation in the universe map, nothing more. The "universe bod" is what makes the universe map look non-square in the vanilla game, in LU we use a "square bod" which eliminates this; This was our personal preference.

---

13. In vanilla, they dont care at all, there scripting remains unaware of adjacent sectors. In LU they do look ahed to see if the "path is clear", i dont remember the actual range of looking head.

14. The game internally uses an A* pathing method, but these results get cached to prevent extensive lookup, this is why the AI can bug out if you create sectors/gates on the fly. I believe this cache is cleared every time you close the game, but i never did any testing to prove it.

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 19:13

@Jack08
Great stuff! Thanks!

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Wed, 13. Sep 17, 19:19

(My apologies for the bump...)

Jack08 wrote: 4. Using tab separation and splitting the input based on \n, and \t would work ok, in LU, you can read this in raw from any file using "$Data = read file: <file>". In vanilla you would have to embed this data in a textdb file instead.

What's your take on the performance end of those methods..?

Jack08 · Post by **Jack08** » Wed, 13. Sep 17, 23:21

that depends on the amount of data your trying to push, and if your trying to do it in realtime or not... If its not realtime, then it can be done in a setup script and the performance hit can be entirely ignored.

If your going to try and receive a real-time data stream, then, your going to face significant problems depending on the stream size.

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Fri, 15. Sep 17, 16:23

lostProfitssssArrgh wrote:(My apologies for the bump...)

Jack08 wrote: 4. Using tab separation and splitting the input based on \n, and \t would work ok, in LU, you can read this in raw from any file using "$Data = read file: <file>". In vanilla you would have to embed this data in a textdb file instead.
What's your take on the performance end of those methods..?

After looking into this a bit, my best guess is that xml is faster when it comes to reading 'sorted' data. The script engine being MUCH slower than an api plugin, I really doubt that reading plain text and sorting it in-game can be faster; even if the actual loadCommand-to-dataAvailable time is shorter and the files smaller (untested).

@Jack08
You guessed correctly that the files would be accessed more than once.

-----------

Another question:

15. I remember someone, somewhere in the forum, posting that 'Kingdom End' *HAD* to be at (0, 0). Is this correct? Do we know why? (@Joubarbe, could this be source of your problems with EUG?).

Thanks again,
-lpa

Joubarbe · Post by **Joubarbe** » Fri, 15. Sep 17, 18:20

Not Kingdom End in particular. A sector must be at 0, 0. And I said that just above.

lostProfitssssArrgh · Post by **lostProfitssssArrgh** » Fri, 15. Sep 17, 18:34

Joubarbe wrote: Not Kingdom End in particular. A sector must be at 0, 0. And I said that just above.

Ok, thanks.

I used EUG to generate a few universes, and none of them so far had a (0, 0). Then adding it and linking it by hand seems to be an obligation according to the aforementioned info.