[TOOL] X4 Customizer 1.23

[SirNukes]

Download source from github:
https://github.com/bvbohnen/X4_Customizer
Compiled release (64-bit Windows):
https://github.com/bvbohnen/X4_Customizer/releases
Full documentation (describing settings and transform parameters):
https://github.com/bvbohnen/X4_Customiz ... ntation.md


X4 Customizer 1.23


This tool offers a framework for modding the X4 and extension game files programmatically, guided by user selected plugins (analyses, transforms, utilities). Features include:

• GUI to improve accessibility, designed using Qt.
• Integrated catalog read/write support.
• XML diff patch read/write support.
• Automatic detection and loading of enabled extensions.
• Three layer design: framework, plugins, and control script.
• Framework handles the file system, plugin management, etc.
• Plugins include analysis, transforms, and utilities.
• Plugins operate on a user's unique mixture of mods, and can easily be rerun after game patches or mod updates.
• Transforms can dynamically read and alter game files, instead of being limited to static changes like standard extensions.
• Transforms are parameterized, to adjust their behavior.
• Analyses can generate customized documentation.
• Transformed files are written to a new or specified X4 extension.
• Utilities offer extension error checking, cat pack/unpack support, and automated diff generation from pairs of files.

This tool is available as runnable Python source code (tested on 3.7 with the lxml and PyQt5 packages) or as a compiled executable for 64-bit Windows.


Running the compiled release:

• "Start_Gui.bat"
  - This starts up the Customizer GUI.
  - Equivelent to running "bin/X4_Customizer.exe"
• "Run_Script.bat [script_name] [args]"
  - This runs a script directly without loading the GUI.
  - Call from the command line for full options (-h for help), or run directly to execute the default script at "Scripts/Default_Script.py".
  - Script name may be given without a .py extension, and without a path if it is in the Scripts folder.
• "Clean_Script.bat [script_name] [args]"
  - Removes files generated in a prior run of the given or default control script.
• "Check_Extensions.bat [args]"
  - Runs a command line script which will test extensions for errors, focusing on diff patching and dependency checks.
• "Cat_Unpack.bat [args]"
  - Runs a command line script which unpacks catalog files.
• "Cat_Pack.bat [args]"
  - Runs a command line script which packs catalog files.
• "Generate_Diffs.bat [args]"
  - Runs a command line script which compares pairs of xml files, and outputs suitable diff patches.

Running the Python source code:

• "python Framework\Main.py [script_name] [args]"
  - This is the primary entry function for the python source code, and equivalent to X4_Customizer.exe.
  - When no script is given, this launches the GUI.

GUI sections:

• "Script" displays the current control script, alongside documentation on available plugins. Drag plugins to the script window for a fast templated copy. Scripts can be opened or saved, and default to the Scripts folder. Syntax is highlighted as Python code. Press "Run Script" to run the current script; other tabs displaying game information will be updated automatically with the script changes.
• "Config" allows customization of settings. These are saved to a json file in the main tool directory when the window closes.
• The "Edit" menu opens up tabs with editable tables of game object information. Objects may be displayed individually or in a table. See further below for details. All edits made are saved in a json file in the main tool directory when a script is run or the window closes.
• The "Utilities" menu currently has one option, the Virtual File System. This shows which game files the Customizer has loaded, patched from other extensions (blue), or modified itself (red). Right click a file for the option to view its contents.
• The File Viewer tabs display individual file contents, in their pre-diff patch, post-diff patch, and post-customizer versions. Select two versions and press "Compare" to get a summary on lines changed. Press "Reload" to force the file to be reloaded from disk, including any diff patches; this may be used to test customize diff patch files in another extension.

The control script:

• This tool is primarily controlled by a user supplied python script which will specify the desired plugins to run. Generally this acts as a build script to create a custom extension.
• The key control script sections are:
  - "from Plugins import *" to make all major functions available.
  - Optionally call Settings() to change paths and set non-default options; this can also be done through a setttings.json file or through the GUI.
  - Call a series of plugins with desired input parameters; see plugin documentation for available options.
  - Call Write_Extension() to write out any modified files, formatted as diff patches.
• Scripts of varying complexity are available in the Scripts folder, and may act as examples.

GUI based object editing:

• In addition to script selected transforms, game information can be directly edited for select objects on the appropriate edit tabs. Tabs include "weapons", "wares", and others as time goes on.
• Press "Refresh" on the tab to load the game information.
• The "vanilla" column shows the object field values for the base version of the game files loaded.
• The "patched" column shows field values after other extensions have had their diff patches applied.
• The "edited" column is where you may change values manually.
• Running a script that includes the Apply_Live_Editor_Patches plugin will apply edits to the game files.
• The "current" column will show the post-script field values, and mainly exists for verification of hand edits as well as other transform changes.

Writing custom edit code:

• This framework may be used to write custom file editing routines.
• The general steps are: use Load_File() to obtain the patched file contents, use Get_Root() to obtain the current file root xml (which includes any prior transform changes), make any custom edits with the help of the lxml package, and to put the changes back using Update_Root().
• Existing plugins offer examples of this approach.
• Edits made using the framework will automatically support diff patch generation.
• Non-xml file support is more rudimentary, operating on file binary data pending further support for specific formats.
• Routines may be written as plugin functions and put up for inclusion in later Customizer releases to share with other users.

Full documentation found in Documentation.md, describing settings and transform parameters.

Example input file:

Code: Select all

    '''
    Example for using the Customizer, setting a path to
    the X4 directory and running some simple transforms.
    '''

    # Import all transform functions.
    from Plugins import *

    # This could also be done in settings.json or through the gui.
    Settings(
        # Set the path to the X4 installation folder.
        path_to_x4_folder   = r'C:\Steam\SteamApps\common\X4 Foundations',
        # Set the path to the user documents folder.
        #path_to_user_folder = r'C:\Users\charname\Documents\Egosoft\X4\12345678',
        # Switch output to be in the user documents folder if needed.
        output_to_user_extensions = False,
        )

    # Reduce mass traffic and increase military jobs.
    Adjust_Job_Count(
        ('id   masstraffic*', 0.5),
        ('tags military'   , 1.3)
        )

    # Make weapons in general, and turrets in particular, better.
    Adjust_Weapon_Damage(
        ('tags turret standard'   , 2),
        ('*'                      , 1.2),
        )
    Adjust_Weapon_Shot_Speed(
        ('tags turret standard'   , 2),
        ('*'                      , 1.2),
        )

    # Get csv and html documentation with weapon changes.
    Print_Weapon_Stats()

    # Write modified files.
    Write_To_Extension()

Analyses:

• Print_Object_Stats
  Print out statistics for objects of a given category. This output will be similar to that viewable in the gui live editor pages, except formed into one or more tables. Produces csv and html output. Will include changes from enabled extensions.
• Print_Ware_Stats
  Gather up all ware statistics, and print them out. This is a convenience wrapper around Print_Object_Stats, filling in the category and a default file name.
• Print_Weapon_Stats
  Gather up all weapon statistics, and print them out. This is a convenience wrapper around Print_Object_Stats, filling in the category and a default file name.

Director Transforms:

• Adjust_Mission_Reward_Mod_Chance
  Adjusts generic mission chance to reward a mod instead of credits. The vanilla chance is 2% for a mod, 98% for credits.
• Adjust_Mission_Rewards
  Adjusts generic mission credit and notoriety rewards by a flat multiplier.

Jobs Transforms:

• Adjust_Job_Count
  Adjusts job ship counts using a multiplier, affecting all quota fields. Input is a list of matching rules, determining which jobs get adjusted.

Map Transforms:

• Scale_Sector_Size
  Change the size of the maps by moving contents (zones, etc.) closer together or further apart. Note: this will require a new game to take effect, as positions become part of a save file.

Live_Editor Transforms:

• Apply_Live_Editor_Patches
  This will apply all patches created by hand through the live editor in the GUI. This should be called no more than once per script, and currently should be called before any other transforms which might read the edited values. Pending support for running some transforms prior to hand edits.

Ships Transforms:

• Adjust_Ship_Speed
  Adjusts the speed and acceleration of ships, in each direction.

Text Transforms:

• Color_Text
  Applies coloring to selected text nodes, for all versions of the text found in the current X4 files. Note: these colors will override any prior color in effect, and will return to standard text color at the end of the colored text node.

Wares Transforms:

• Adjust_Ware_Price_Spread
  Adjusts ware min to max price spreads. This primarily impacts trading profit. Spread will be limited to ensure 10 credits from min to max, to avoid impairing AI decision making.
• Adjust_Ware_Prices
  Adjusts ware prices. This should be used with care when selecting production chain related wares.

Weapons Transforms:

• Adjust_Weapon_Damage
  Adjusts damage done by weapons. If multiple weapons use the same bullet or missile, it will be modified for only the first weapon matched.
• Adjust_Weapon_Fire_Rate
  Adjusts weapon rate of fire. DPS and heat/sec remain constant.
• Adjust_Weapon_Range
  Adjusts weapon range. Shot speed is unchanged.
• Adjust_Weapon_Shot_Speed
  Adjusts weapon projectile speed. Range is unchanged.

Utilities:

• Cat_Pack
  Packs all files in subdirectories of the given directory into a new catalog file. Only subdirectories matching those used in the X4 file system are considered.
• Cat_Unpack
  Unpack a single catalog file, or a group if a folder given. When a file is in multiple catalogs, the latest one in the list will be used. If a file is already present at the destination, it is compared to the catalog version and skipped if the same.
• Check_All_Extensions
  Calls Check_Extension on all enabled extensions, looking for errors. Returns True if no errors found, else False.
• Check_Extension
  Checks an extension for xml diff patch errors and dependency errors. Problems are printed to the console. Returns True if no errors found, else False.
• Generate_Diff
  Generate a diff of changes between two xml files, creating a diff patch.
• Generate_Diffs
  Generate diffs for changes between two xml containing folders, creating diff patches.
• Write_To_Extension
  Write all currently modified game files to the extension folder. Existing files at the location written on a prior call will be cleared out. Content.xml will have dependencies added for files modified from existing extensions.

[SimB]

Looking forward to the "full" version!

[Diggz]

There doesn't seem to be a release available for download.





To any other noob looking to use this:
1. download the repository as a zip, then double-click the solution file to open visual studio.
2. Visual Studio will ask you to install the python environment if you don't already have it installed, and click yes and let it run.
3. Once python is installed, go to View > Other Windows > Python Environments, then set the drop-down on the right to 'Packages' and enter 'lxml'. You will see 'install lxml (4.2.5)' in blue, double-click that and you will see that it installs in the output window at the base of visual studio.
4. In the Solution Explorer on the right, you will see an Authors_Transforms, double-click that and make the job changes you want to see.
5. Click the green play button at the top labelled 'Attach...'. You will see windows cmd dialogue pop up informing you how it went, and asking you to press any key. If there are no errors go ahead and press your any key.
6. In your X4 directory, you will now have an extensions folder. Open it up, go inside libraries, and open the jobs.xml, and confirm your changes are what you want.




You should be able to launch X4 as-is, but I have had some issues launching with mods in my X4 directory. As such, I moved mine into my Documents/extensions folder.

[Misunderstood Wookie]

Promising indeed.

[SirNukes]

You should be able to launch X4 as-is, but I have had some issues launching with mods in my X4 directory. As such, I moved mine into my Documents/extensions folder.

Thanks for trying out the tool! I wasn't quite expecting anyone to use it just yet. Visual Studio itself is completely optional; I just like it for editing, debugging, and its github plugin (plus it works well for my mixed python/c++ applications from my research work).

Over the weekend I plan to polish the framework further, and get the release generator updated. For the output directory, I plan to add an option to set the path to the documents folder (needed anyway to peek at content.xml), pick if the output should go in the game folder or documents folder, and name the output extension.

[Diggz]

I helped someone on discord do some troubleshooting. Our findings is the X4 root folder only seems to cause problems for windows 7 PCs.


A workaround is to package the extension into cat/dat files. The loose files seem to be the cause of the issue.

[SirNukes]

I helped someone on discord do some troubleshooting. Our findings is the X4 root folder only seems to cause problems for windows 7 PCs.

Odd; I am on Windows 7 and haven't had trouble with the X4 folder extensions. Maybe it is related to file permissions.

But anyway, the customizer has gotten a lot of love over the last 30 hours or so, and the framework is mostly complete. The diff patch support needs more polish and stress testing, since I just ended up writing my own (RFC 5261 doesn't seem super popular). Customizer settings were overhauled, including the option to write to documents/egosoft.

[SirNukes]

Updated to 0.9.3, and getting closer to release. Another 5-6 hours was enough to fill out the diff patch generation algorithm, so that it can make isolated node edits instead of doing a full xml replacement. Automated testing looks good at the moment: make random edits to some x4 xml, make a patch, apply the patch to the original xml, and verify recreation of the edited xml.

With the framework about done, I now need a bunch of cool transform ideas.

[SpaceCadet11864]

Hey, this is wonderful, thanks for your hard work on this.

[SirNukes]

An idea struck me that I could split out transforms from the rest of the design to make them more modular and accessible. This is done in version 0.10.

There is now three tiers to the tool: framework (compiled), plugins (transforms, utilities, analyses), and control scripts (similar to before, picking plugins to chain together). The plugins are uncompiled python code, which can be edited or added to without python installed.

To stress test a little, I added two utilities, a catalog unpacker and packer (just because they seem popular). They have some wildcard support, and the unpacker will skip files that have an md5 match at the destination already.

For an initial analysis plugin, I added Print_Weapon_Stats. The output format is a bit rough, just csv dumping a bunch of values (and maybe some incomplete weapons), but it respects installed extensions when gathering the numbers.

Right now, I am wondering if this tool should have a different (better) name. Spellcheckers keep putting red squiggles under the current name. Recommendations are welcome.

[ShadowTek49]

if I have a mod in say D:\X4\Pack and want to unpack it to D:\X4\UnPack how do I do it? Can this mod do that?
What exactly would the User_Transforms.py file look like?
Do you need a coding degree to use this thing?

[SirNukes]

if I have a mod in say D:\X4\Pack and want to unpack it to D:\X4\UnPack how do I do it? Can this mod do that?
What exactly would the User_Transforms.py file look like?

Right now, to unpack one cat file, it would look like this:

Code: Select all

from Plugins import *
Cat_Unpack(
    source_cat_path = r'D:\X4\Pack\ext_01.cat',
    dest_dir_path   = r'D:\X4\UnPack'
    )

With a little touchup it will be able to take just the mod folder instead of the full cat name, but right now the folder name only works for the base X4 folder (since it looks for "01.cat" and similar).

[ShadowTek49]

if I have a mod in say D:\X4\Pack and want to unpack it to D:\X4\UnPack how do I do it? Can this mod do that?
What exactly would the User_Transforms.py file look like?

Right now, to unpack one cat file, it would look like this:

Code: Select all

from Plugins import *
Cat_Unpack(
    source_cat_path = r'D:\X4\Pack\ext_01.cat',
    dest_dir_path   = r'D:\X4\UnPack'
    )

With a little touchup it will be able to take just the mod folder instead of the full cat name, but right now the folder name only works for the base X4 folder (since it looks for "01.cat" and similar).

OK thanks for the response, I tried that and it made the folders butt with no content in them?

EDIT: OK so only from main game folder, as in the mod must be in the main game folder to unpack, is that what you mean?

[ShadowTek49]

Yeah that doesn't work, it just makes the folders from the cat/dat file, tried everything I could think of. Its trying so I guess its on the right track.

EDIT: Wait, what do you mean by "but right now the folder name only works for the base X4 folder" can you give an example?

[SirNukes]

OK thanks for the response, I tried that and it made the folders butt with no content in them?

EDIT: OK so only from main game folder, as in the mod must be in the main game folder to unpack, is that what you mean?

Regarding the main game folder stuff, ignore that for the moment; I was digressing.

The unpacker is somewhat verbose; what messages does it print for files written or skipped?

[ShadowTek49]

OK, let me ask this, what would everything under:

# Fill in transforms below.

look like using these paths
D:\X4\UnPack
D:\X4\Pack

for example this:
# Write modified files.
Write_To_Extension()

do I delete that? Change it?

I get:
skipped Cat_Unpack due to a Exception exception

Thats exactly what it says weather I delete that part or not

EDIT:

This is what I did

Code: Select all

# Fill in transforms below.

from Plugins import *
Cat_Unpack(
    source_cat_path = r'D:\X4\Pack\ext_01.cat',
    dest_dir_path   = r'D:\X4\UnPack'
    )
    

Thats everything below:
# Fill in transforms below.

should there be something else?

Edit 2:
Keeping:

# Write modified files.
Write_To_Extension()

gives same error msg.

[SirNukes]

For the input script, the code I posted is all that's needed for just unpacking a catalog; the rest can be commented out.

For the exception, it sounds like the md5 hash check is failing, given the place it happens. (Catalog files include hashes of each game file, and this tool computes and verifies the hash when pulling data out of the .dat file.) I'll take a look into it; in the worst case I can change it to allow md5 mismatches.

[ShadowTek49]

The file I'm trying to unpack is a mod not a game file if that matters.

[SirNukes]

The file I'm trying to unpack is a mod not a game file if that matters.

It might, if the catalog was created with a bad hash.

At any rate, 0.10.1 is up on github with three changes:
A) It turns out the egosoft catalogs have a small bug with empty files, giving an md5 hash of 0 (when the proper empty hash is 'd41d8....'), so I added a special check for that case.
B) There is a new setting called "allow_cat_md5_errors" which will suppress the exception on a failed md5 check.
C) Cat_Unpack has an extra parameter when unpacking extension directories, to help it find the right files automatically. You can still feed it the specific cat file as before, or use the directory style example below.

So the code would now look like:

Code: Select all

from Plugins import *
Settings.allow_cat_md5_errors = True
Cat_Unpack(
    source_cat_path = r'D:\X4\Pack',
    dest_dir_path   = r'D:\X4\UnPack',
    is_extension    = True
    )

[ShadowTek49]

The file I'm trying to unpack is a mod not a game file if that matters.

It might, if the catalog was created with a bad hash.

At any rate, 0.10.1 is up on github with three changes:
A) It turns out the egosoft catalogs have a small bug with empty files, giving an md5 hash of 0 (when the proper empty hash is 'd41d8....'), so I added a special check for that case.
B) There is a new setting called "allow_cat_md5_errors" which will suppress the exception on a failed md5 check.
C) Cat_Unpack has an extra parameter when unpacking extension directories, to help it find the right files automatically. You can still feed it the specific cat file as before, or use the directory style example below.

So the code would now look like:

Code: Select all

from Plugins import *
Settings.allow_cat_md5_errors = True
Cat_Unpack(
    source_cat_path = r'D:\X4\Pack',
    dest_dir_path   = r'D:\X4\UnPack',
    is_extension    = True
    )

OK Thanks man for your responses in helping me, gotta goto work in a few hours (damn this game lol) so I'll try it after work tomorrow, again thanks for the support