[IDEA] X-Online -Initial Project Failure -Project Re-Conceptualized

The place to discuss scripting and game modifications for X³: Terran Conflict and X³: Albion Prelude.

Moderators: Moderators for English X Forum, Scripting / Modding Moderators

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

[IDEA] X-Online -Initial Project Failure -Project Re-Conceptualized

Post by s9ilent » Thu, 12. Feb 09, 14:03

Basic Idea: Make X3-online able


Initial Project Failure
  • Ok the original Idea of making an "online" sector (or sectors) with full access to all of X3's normal features has *failed* (dismally)

    The bandwith requirements for absolute information updating, is proving to high. (i.e. CS uses like +attack, + forward, but in X3 I can't do that, I'd have to return the x y z, and rotation angles etc etc)





But like I'm going to let that stop me....
In Lieu of having an complete X3 universe, I'm thinking of having a partial universe. (Specifically, with two partial universes)

Idea 1.
A single communal sector. Ships are *NOT* updated in real time (nor are they continuously updated).
In this sector you can
  • Make use of the Trade dock (one large -note: VERY large, internally any way...) and Eq dock. To trade between all players in the communal sector.
  • No Combat is allowed, you will be instantly destroyed!
note to self, add product waren, to prevent production cycle
include no products to prevent production cycle


Idea 2. Duals
-This idea is still premature. As I'd still have to restrict what the player can do/rewrite lots of scripts.
-An idea for this, was for it to exclude the player, and just be run on the server side. (And with out randomization to prevent desync) But then that would be more like playing an RTS then a Flight Sim.

Idea 3 <--- Curent idea
A global stock exchange (ware exchange) sort of thing, for players to trade on. Like Idea 1, but instead of in a communal sector, its done through a station command (instead of a communal sector)

Idea 4
Extension of Idea 3. Randomly/deterministicly sends ships around to each of the universes, to buy/sell wares for the stock exchange.


Any way, Ideas comments suggestions etc.
Volunteers?
Last edited by s9ilent on Sun, 15. Feb 09, 09:18, edited 8 times in total.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Thu, 12. Feb 09, 14:03

Current Ideas:

Idea 1. Communal Sectors
(X-Online Marketplace XOM? Sounds crap... hmm... Universal Trade Dock, Galactic! Trade Dock, Galactic Stock Exchange!
Suggest a name.. please You can clearly see my ideas are limited to coding and not to name making)



In short, a multiplayer stock exchange.
Location: Freedoms Reach? (I am tempted to create a new sector, but that would require a mod, and to start a new game, in truth, you could smack down the trade hub ANY where)
Or or or or!!! Why add to the game, Use the game!!! The Xenon hub!! (the xenon hub was always for connecting people, and now I can *puts on his big deep evil voice* "Now I can connect the entire planet, Mwahahaha" */end rant*

Using the station commands, you can access the online market place, place an order. A little while later, woosh, a ship comes in drops it off and the hub and your off :). And of course, the wares price depends on the quantity at the server.




Idea 2 is in the 5th post
Last edited by s9ilent on Sat, 14. Feb 09, 03:12, edited 4 times in total.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Thu, 12. Feb 09, 14:04

Implementation

1. The server has to be online at all times now (unfortunately..)
2. The command menu, spits out to a *log file* that is periodically read, goes to the program, shoots the request to the server, the server bounces it back, the program reads it, and viola your done.
3. To prevent buy 99999999 sell 9999999999 (as the price would go up when you deplete the market), there will be a restriction on trading. Also there will be a Bid/Offer spread, proportional to the volatility of the market.

4. Because I'm still a bandwidth nut, I'm not going to let players see the entire markets price list in real time. Probably... I might revert this decision later... But rather, when you open the menu AND when it hasn't updated in the last minute)
4 1 39 13 16 36
Hm.. 436 bytes... per catalogue (not inc overheads)... 0.5kb... Maybe the whole catalogue isn't that big...
Grumble grumble...
Ok 15 second refresh timer, on entire catalogue
5 second refresh timer on single type catalogue (eg just on lasers, or just on missiles)







Possible Ideas
Tbh, I dont like the below ideas, as they either a) would very poorly do something that is already available (eg xfire). or b) are extremely complicated and would obsolete the above

4. Direct player Trading.
5. Chating with other players (across the galaxy, not just whilst in the hub)
-The above two are a little tricky for me to implement at the moment... as I'm still not actually sure how do disconnect from the server.. Apparently I need to send a disconnect signal.. but I still need to play around with it. (Closing the program works as a disconnect :P but thats fair archaic..)

6. Voice chat??? I'd have to.. get an audio file.. make it into bytes, and spit it across... I have to say this idea has a VERY low priority.. as there are already many wonderful voice chat progs. And given that servers for this X-Online idea requires players to meet up and exchange IP's/ports first, you could easily organize a voice chat server first. (eg xfire)
Last edited by s9ilent on Sat, 14. Feb 09, 02:48, edited 4 times in total.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Thu, 12. Feb 09, 14:04

Problems/Solutions and other limitations


1. I'm going to restrict how often the price list updates. (As of last update, the idea was 15 seconds on the entire price list, 5 seconds on single catalogues)
(When actually ordering thou, it uses the latest figures, so it will bounce back a message to you saying your order was bounced, if the price fell out of your price range)


2. To prevent buy, prices goes up, sell, price goes down repeat. There will be a bid/offer spread. (If the price is 100, and the B/O spread is 3, then you buy at 103, sell at 97) And the more volatile a good has become, the greater the bid/offer spread becomes.


3. The server will have to be kept running.
Last edited by s9ilent on Sat, 14. Feb 09, 03:03, edited 8 times in total.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Thu, 12. Feb 09, 14:05

Idea 2

Simulated Combat Environment
N.B The Player ship can not participate in the combat.
This idea works more like an RTS, just using the X3 enginge.

In short, two fleets get pitted against each other. (or more?)

Combat, ONLY takes place on the server. The two clients are just "re-enactments" of the battle.



Implementation:
As the server is simulating the entire battle, only limited data is required to be sent to the server. (Only new orders actually, ship, order, arguments)
And because the clients are just re-enactments, only partial data will be required.
(ship, HP, Shields, x, y, z) (possibly exclude the Xyz)
(ship, new target)


With this, the acted battles will carry on as normal (but their ships are made invincible). So the ships will keep blasting each other apart on the host (not actually killing each other), and every time a new data packet is sent, it just applies the new HP/Shields to all the ships.
If a ship becomes overly desync'ed (eg. its xyz is WAY off from its itended position), a small pop up comes in, says desync, and moves the ship back to its "actual" position)
Last edited by s9ilent on Sat, 14. Feb 09, 03:12, edited 3 times in total.

4dams
Posts: 14
Joined: Fri, 16. Feb 07, 12:08

Post by 4dams » Thu, 12. Feb 09, 14:39

Ok I see you are updating it right now and with the second post you answered most of the questions I wanted to ask. Will take a look when you finished adding all the data.
Try to think outside of the box.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Thu, 12. Feb 09, 14:44

Ok, I'm done for the night

Feel free to post your questions away

4dams
Posts: 14
Joined: Fri, 16. Feb 07, 12:08

Post by 4dams » Thu, 12. Feb 09, 15:13

1. this is not really clear for me from your post so I need to ask. (maybe because my bad english)
Are you going with the idea of client to client data transfer where one client is also the server or with the idea of a dedicated server for storage and data transmission?

Edit: Ok is have taken another look on it and I see you go with client to client which is very risky AND you would need each client to communicate with every single one.
You could have the universe created and maintained on a dedicated one checking for not allowed scripts and ships transmitting same data and IDs for all clients.
Last edited by 4dams on Thu, 12. Feb 09, 19:06, edited 2 times in total.
Try to think outside of the box.

aquemnun
Posts: 247
Joined: Sun, 26. Oct 08, 19:47

Post by aquemnun » Thu, 12. Feb 09, 18:06

dude if you could manage this it will be amazingly awesome and amazing :lol:

although it'll be expensive for me as my gaming computer is strictly offline for security reasons and simply because i have no reason to connect it as of now. so i may have to spend some serious £££ to upgrade my download computer enough to pay TC.

HOWEVER, if you pull this off you will have granted many peoples wishes and the upgrade would be worth every penny.
There once was a buggy AI
who decided her subject must die
when the plot was uncovered,
the subject discovered
that sadly the cake was a lie.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Thu, 12. Feb 09, 22:45

@ 4adams, Yeh.. I'm not really much of a computer techy guy...

And its designed for small closed servers, so client-client (But as always one client has to be the server, so everyclient -> "server client", and "server client" -> everyone.
(Everyone does not directly connect with everyone thou)

4dams
Posts: 14
Joined: Fri, 16. Feb 07, 12:08

Post by 4dams » Fri, 13. Feb 09, 00:40

s9ilent wrote:Current Ideas on features:

2. There are thoughts about ware control...
This calls for a server based database as the easiest solution. You were right about that you need to rewrite the scripts and you must be able to actually read from and write into this database.

As for the so called "server client" part I can see no problems there. It should work quite well for small sized servers ("size" is actually determinated by the amount of data you need to process & transmit).
Try to think outside of the box.

User avatar
Jack08
Posts: 2993
Joined: Sun, 25. Dec 05, 10:42
x3tc

Post by Jack08 » Fri, 13. Feb 09, 05:17

Ive worked on WoW Private servers for years coding for them in c++ and database manipulation, witch its probably best to follow the same system.

Create a Central server with a database (Eg: MySQL) and store all infomation about players in the database.

There is one major problem i can see, if you plan on having any NPC's in the universe your going to have alot of bandwidth requirements between the server and its clients, or if you keep the current npc system your going to have the same issue alot of RTS games have, Desync'ing game simulations.

And what happens to your ships/stations when your offline?... ive thaught about MMORTS's alot and i just don't see a solution to this other then them vanishing. if they don't vanish then something has to take over them, an ai, but that would have be server side, and if you consider the amount of ships the game would build up too that would get extremely taxing on the CPU of the server. And on the same note you have the scale issue, the game is going to get bigger, and bigger, and well, you know what happens to the X3 Game after a while it starts to slow down and crash.

Without a major change in game mechanics, i don't see this working very well. witch is a shame because its such a great idea, But i'd be willing to help.

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Fri, 13. Feb 09, 07:27

Yeh... I don't say why your all worrying so much about data bases so much. I'm just going to use the game engine it self. Whilst this system is inherently inefficient (as latency becomes 2x ping + U[0,1), rather then just 2x ping), It is by far the best alternative, as no one really knows the mechanics of how objects work in X3. (So simply, just let X3 handle the mechanics)
The app just links the game's together.
In fact I'm making the app very simply, it just records and feeds numbers from the server to the hosts.

One Idea I have been entertaining however, is rather then uploading the entire universe and everything in it. Just have a couple of "communal" sectors, and then everything else is offline.
And as its going to be primarily based around small private servers, when the games offline, it's just offline.


Also, the majority of NPC's are highly predictable, given the same environment, they will do the same thing. So short of a huge player intervention, it should be possibly to "slow update" the universe.
So instead of updating every ship, every "refresh" get all the ships, and cycle the list over the course of XXX seconds.

Of course, several actions would require objects to refresh in realtime. Namely, the playerships and activesector ships.
Similarly, some actions would require an immediate refresh. Eg, trading a ware should be done immediate, to prevent de-synchronisation.


I'm hoping to be able to achieve this by doing the following
1. Replace some stock script editor commands, with a lib script, designed for XO use.
Eg, buy ware, will be replaced with lib
-> lib.xo.buyware $ware $amt
On the client side, it wont actually buy anything, but send a signal to the server, and wait for 3 seconds (for a return value). If it doesn't come, don't buy, return 0 (bought amount). If it does come, then buy the amount returned from the server.
Similarly for sell, load, unload etc.
Similarly, player ships firing will have to be replaced. (unfortunately). As I need to send some kind of firing signal, I'll just replace firing with a script: *check player target, if in firing cone, fire every 25 ms for 250 ms and return to server, player ship fired on target. Other wise "miss" fire at nothing/dont fire/dont send.

Of course, this can still lead to desynchronization with the server. But hey, these things happen.

One of the issues I'm conceptuatlizing atm thou, is what to do in the event of desync.

One HUGELY tangental idea I've been having thou, is to make X3O more like sins of the solar empire thou. Where essentially, all graphics and hits are eye candy.


But in any case, all this is sort of a moot point, as I'm still wondering exactly how am I going to structure the data... Idealy, just integers, but now that I'm a bit more advance in my bit-array ness, I could restrict the first to 2 bytes + ve, and then construct some kind of variable return system, to allow strings, or integers. But then I'd have to consider.. what do I need strings for?
And yep... conceptualizing

So for now I'm just building the program in a modular design. The server/client program will just send/receive numbers, and the X3 scripts that I will write, will just be able to read/print numbers. Then Ill worry about what the actual numbers should be.


Oh one more thing I need to do. Replace all the randomly generated numbers in current scripts, to read off from an LCG that I create, where I can set the initial seed, for best sync across the clients.

(As a side note, I can see the technical merit in why eve online was designed for there to be only 1 player controllable ship :S {perplayer})

Expansion of the LCG idea's, The initial seed can be the objects id from the server. (For LOCAL random numbers, globals will just use the global random number)

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Sat, 14. Feb 09, 02:11

Gah :(

Jack is right :(




After doing a bit more work on the data structure, and thawing my idea out:

Given the X3 environment and setup, the bandwidth requirements to write what I was planning to do is extremely unrealistic.

Even just updating the bare basics: shipid, x, y, z, hull, shields, speed, rot a, rot b, rot g -requires close to 50 bytes of information (per ship) limiting it to an effective (an ineffective rather) 60 ships per second (for the target 3kb/s)






So, in short, doing a full X3 multiplayer (even in a limited space of a single sector) becomes impractical for anything except for challenges (i.e. Single ship duals, or very limited ship number duels)

However... there are still other non-combat orientated possibilities. (Eg the entire trading side, as that is event driven. i.e. I don't need to continuously update, but rather, if you buy something, it sends a buy signal)



Ok, the new Ideas are posted
What are all your thoughts about it?

adecoy
Posts: 208
Joined: Wed, 24. Jan 07, 06:26
x3tc

Post by adecoy » Sat, 14. Feb 09, 22:20

i would be happy if i could get a friend in the game to play and influence the same game world as me, even if everything we do together has to be out of sector.

like, is it possible to set it up so that the ships from your "friends" empire are controlled by the ai, but are updated only whenever the friend issues a new command, fixing desyncs every 5-15 min?

i dont know alot about servers and bandwith, but if i am playing over the internet with a friend, wouldent 3kb per 60 ships be doable with a 30-40kb upload?

are you talking about in sector or out of sector?

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Sat, 14. Feb 09, 23:00

There is only slightly more data in sector then out

(Approximately 4 more integers, although thats just for what I had planned, if you go nuts and do everything, you could probably go to about 10 (4 bytes each, so 50 bytes)


It would be possible to implement something like that, although there wouldn't be any real interaction between the two, it would be mostly fluff. The on new command idea also requires a LOT of reworking of the scripts. (Every command has to be re-written). So for now I'm just going to stick with the galactic stock exchange Idea.

The upload per ship I estimated to be only 0.05 kb per ship. My target was just a 3kb/s. As 30-40kb's is a huge amount of bandwidth (as far as games go). (3kb is relatively smallish, cs only uses.. I think upto 5-6ish? )
But even at 5kb/s, thats still only 100 ships per second.
(And that excluding all on-command-data, that would require significantly more, as it needs strings, (strings are mostly for the arguments, the command them selves are number-able)

And considering there are some .. 236? odd sectors, with lets say an average of 10 ships per sector. We are talking about a 0.5 minute "join" time (with a similar 0.5minute dysnc) and then when you include commands in there (a command is probably 2 ships), each command would set you back by about 1 ship.


Whilst you could then restrict it to player only properties, their commands change fairly often. And if you further restrict the upload frequency, (to your suggested 15-5) it is potentially do able, but for all intents and purposes, your just adding more laggy civilians into the game.
Another problem is that complex's are not re-creatable. (whilst you could not re-create the stations at all, stations are like half of most people's empires)



So yeh... complications :S
So I'm going to finish the GSE first (as that is do able with VERY minimal data transfer), then anythign else I can just build onto that, as the data transfer frame work will already exist

User avatar
Jack08
Posts: 2993
Joined: Sun, 25. Dec 05, 10:42
x3tc

Post by Jack08 » Sun, 15. Feb 09, 03:12

Not the xenon hub, avoid that at all costs... im sure im not the only one but that sector randomly crashes my game, and the xenon hub consumes resources that are inside of it over time

@adecoy: What you suggested with the ai wouldn't fix the desync, infact it would just make it worse

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Sun, 15. Feb 09, 03:28

Uhh... are you sure???

Only Np...
Spoiler
Show
Uh... are you sure?
Only NPC stations randomly consume resources, and the xenon hub is suppose to be player owned a short while after you find it. (And it will be implemented via the command menu, so you can only ever use it when its under your control.

As for randomly crashing... hmmm...
I was originally going to put it on the PHQ ... but you need the hub for the phq, and if the hub is crashing, the I'd assume you don't have the phq yet..

(The server side doesn't need to run the game, it just holds a database, so resources won't vanish on the server. I will probably make it mean revert to 35-75 percent.. although I might not.. I'm still trying to get data sending up. )






While I'm posting
Quick status report

I've finished planing the data structure
I've finished the incomming data processor for the client side
I'm doing the logfile-> outgoing data processor for the client side

The server side isn't done, but it will be copy/paste for the incoming/outgoing (but in reverse) + the database management. (I've never done D'bases before... so I don't know if I'm going to just use an array, or an actual database)[/spoiler]

User avatar
Jack08
Posts: 2993
Joined: Sun, 25. Dec 05, 10:42
x3tc

Post by Jack08 » Sun, 15. Feb 09, 03:54

Ok sorry, perhaps i should reword what i said
Ive had multiple occasions where stuff ive put in the xenon hub has just... vanished, i put the trade barrier on there and even sat and watched it, any
wares inside the thing just dissapere for me, i avoid the thing like the plague
Mabye this was a glitch that got fixed?. im not sure but i still avoid storing anything in it, even energy cells for later use. Cause its creadits stay at zero yet they still dissapere.


I only crash when leaving the sector, im not sure what it is but during sector changes it just... crashed. probably due to a script possibly conflicts.


And no i don't have the PHQ, mission glitched on me because i have the conflicts script installed and the guy i need to talk to is <invalid> no biggie ill just script it in for myself, iv had it in previous games

Use a full MySQL database, if you need help with it let me know

All the database is is a place to store information, when the sever loads, all information inside the database *should* be loaded into active memory, in c++ you would load it into whats called a container
and when changes are made a update is sent to the database, and also updated in active memory.

i recommend you use that system, the other way to do it is to update it in active memory and update the database when the server a) shuts down, or b)crashes
but there's a major flaw with that, if the server crashes without initiating its crash handler all information is lost. and if there is a BSoD then information will be lost too because it wont be able to save

User avatar
s9ilent
Posts: 2033
Joined: Wed, 29. Jun 05, 01:45
x4

Post by s9ilent » Sun, 15. Feb 09, 04:08

Hmm

Initially I was going to animate the process (a TS comes in, docks transfers the goods), but then i remembered...
It took the AI about 20 passes to dock before it would actually dock (instead of running anti-collisions with the station), so thats when I decided I was just going to "zap" the wares over there (60 seconds later or something)

There was also the problem of the transport ships dieing, and what should happen if they did.




Hm.. do you have a suggested replacement? Instead of using the Xenon Hub?
A Tl? any station (in the Hub sector?) requires command software..? Requires initial payment to "connect station" ?

Post Reply

Return to “X³: Terran Conflict / Albion Prelude - Scripts and Modding”