network engine

[greenman]

We got some good and some not really good news :

We successfully merged the parts dumeni and I made together today and sent our first chat messages over the net In the beginning nothing worked until we found out that using ports beyond maybe 10'000 is critical (we used 5555 which didn't work). We use 55556 at the moment and everything works well

However we're thinking about the synchronisation of the universe now and we're not sure how we should do it:
1) We could write some macros to register synchronised variables which would have to be used by all who want something to be synchronised This would be a good and simple solution, but we haven't figured out yet how we should handle newly created objects and the transfer of the initial state to a freshly connected client.

2) We could go for the solution with the linked list (see above posts) in every synchronisable object. This would result in the same question like 1):
- How to tell the factory on each client to create a new object?
- How to build up the whole universe on a newly connected client?

3) Something else. Any suggestions?


We don't know which solution would work best with ogre and the object factory. Using solution 1) would be very well and (probably) efficient for synchronising variables but I really don't see how we can manage the "initial load", unless x3n provides us with a function which converts the whole universe to a bitstream and one which decodes it (on the client side) and builds up the universe.
x3n: What do you think?

I hope you (all) got some ideas because we could really use good advice here

Thanks in advance and cheers

[x3n]

I didn't spend too much time thinking about those problems, but it was enough time to realize that there are 3 closely connected problems:
1. save/load
2. xml-import
3. network

The link between the 3 problems is: In every case the developer needs an easy way to tell the system which of the variables of a class he wants to get saved/imported/synchronized.

In Orxonox v1 we used set-functions for every variable we wanted to load from an xml-file. For this purpose we needed function-pointers. We could do the same in Orxonox v2, but not only for xml-loading but for all 3 problems.

There's an example of how it looked in Orxonox v1:

Code: Select all

    LoadParam(root, "bLoop", this, Mover, setLoop)
        .describe("After getting triggered, the mover loops forever.")
        .defaultValues(false);

now look only at the first line:
- root: A "TiXmlElement", containing the informations loaded from the XML-file
- "bLoop": The name of the variable in the xml-file
- this: ...
- Mover: The name of the class
- setLoop: A function-pointer to the set-function

The function wherein the LoadParam-call was located was used to load classes from xml-files. We could use something similar to put all variables we need into a list or something or we use a function, in every "synchronizeable" class, that calls something like "LoadParam" which adds its informations to a bitstream or whatever.
Maybe we need a "mode" parameter to distinguish between sending and recieving (or, in the case of saving/loading, between the respective modes, but then we need an xml-element instead of a bitstream).

I really don't know if this was helpful, but I hope so



Oh, and congratulations on your success with the chat.

[beni]

Hey, x3n suggestion sounds cool. We may be able to do something out of this.

[greenman]

Oh, and congratulations on your success with the chat.

Thanks a lot. It's only a small progress but still

Hm ok, I think the variables to be synced are the easiest part of it. The problem I see is how we should recreate already created objects on a client which enters the game after some time. I mean we can easily sync all the variables, but this wouldn't help, because they wouldn't generate any objects themselves and ogre wouldn't know about those...

What I think we unfortunately need, is a function which produces an XML-stream (as you already wrote) of the current gamestate. Then it should be possible for the XML-Loader to use this stream in order to build up the universe (client-side). Is this realistic ?

[x3n]

Ok, a few ideas:
#1: The server knows all of its objects (there are fancy lists) - ogre has nothing to do with that.
#2: A connecting player could send an "updateWorld" command, that forces the server to send all existing classes.
#3: I would suggest to synchronize the whole world (not only the variables in the known classes) every 5-10 seconds to avoid problems with packet losses (packet losses can affect everything, even the "create a new object" command).
#4: Transfering an xml-file of the entire world takes too much bandwidth, but the idea is good: Create an xml-file, encode it to a small bitstream (maybe it's even well enough to just gzip the xml-file), transfer it, load the xml-file.
But maybe there's a better solution, because the xml-syntax uses way to much chars for very little information, like:

Code: Select all

<AClassThatContainsOnlyOneVariable>
  <theOneAndOnlyVariableWithAVeryLongNameToAvoidMisunderstandings>
    1
  </theOneAndOnlyVariableWithAVeryLongNameToAvoidMisunderstandings>
</AClassThatContainsOnlyOneVariable>

Better would be something like this:

Code: Select all

01111110    (a char that denotes the beginning of a new class)
52          (an integer that identifies the class)
1           (the value of the first variable)

It would be perfect if there was a way to encode the desired information in the smallest possible way (note: the order of the bytes in a packet is an information, too - if you know the order of the variables, there is no need for <blablabla>...</blablabla>).

I hope this was human read- and understandable

[greenman]

I got another idea now. We could build up the universe like a tree:
- There is a root (master node?).
- Every knot has a pointer to one layer down.
- Every knot has a pointer to the next object in the same layer.
- Every knot has a linked list containing the variables to be synchronized (these should be the same the factory needs for creating the object).
- Every knot has a bool changed (indicating a change in the object).
- Every knot has a identifier which tells us what kind of object it is.
- Every object has a unique identifier (given by the server).

All this would be the base-class synchronizable and the factory would have to build up the tree by using this class.

If we would build up the universe like that, both creating new objects, the synchronization and the initial load would be quite easy to handle:
- We could implement a packet which tells the receiver(client) to create a new object of a specific type and with specific values (the ones from the linked list).
- For the initial load we could just recursively go through the tree and send a create packet for each object that exists (a lot traffic, but its only done once).
- For the synchronization we could send an update packet for each object (that changed) in the tree identifying the object by its id. (not really much traffic, because we can keep the size of these packets very low).

x3n: I don't know how much you already implemented of the factory and the world hierarchy, but it would be great, if we could do it with these three. What do you think ?

btw: To avoid misunderstandings: Enet does not only offer unreliable but also reliable transfer, so we don't have to worry about safe transfer for important things... We will use the safe/reliable flag for packets like create object or delete object (btw: the factory could provide a function which deletes an object by taking its identifier )

[x3n]

There is a tree of the whole class hierarchy.
There are some sort of nodes, containing several informations (about the tree, the lists, the factory).
There are lists of all objects of a class.
There are identifiers (for local use).
I will implement a network-identifier given by the server (a simple integer).

It's possible to expand the nodes, so they have informations about the variables that have to be synchronized.

I'm still working on this, because there are some basic things to think about, concerning the factory and the lists.

[chrigi]

In the old framework we sent with each packet ids. Unique id (assigned by server) and a classId which can be used to create a object of this class. If a packet arrived, for which we could not find the instance with this unique id, we created a new instance.

[greenman]

Good evening

Update:

I implemented a class ClientConnection which manages the connection part for a client (establish, receiver thread, packetbuffer, close, send and of course receive). This class is similar to the ConnectionManager class which is used for the server.

I wrote a little dummy client to test the new class and it works now.

next steps are:
- Base class for synchronizables
- Classes Server and Client (main classes of network module)
- More testing?
- Establish interface to other modules

cheers

[greenman]

Hi everyone

the current status:
- the class Synchronisable should be finished, but its not tested yet (quite difficult, since its only a base class)
- I'm on the GameStateManager now and in order to implement it, I need to know some things:
I'm not sure, whether this belongs in here, but I think other people (dumeny for example ) may also be interested in this.

x3n: Can you tell me how I can access the list, where each class which inherited from the class Synchronisable, is listed in? Is it implemented like an stl-list with iterator and that stuff or more c-style ?

To be done:
- gamestatemanager
- Finish client / server class
- Extend PacketGenerator/Decoder for GameStates

greets

[x3n]

Is it implemented like an stl-list with iterator

Yes. It's not an stl-iterator, but it works similar:

Code: Select all

for (Iterator<class> it = rhabarber(tocome); it != 0; ++it)
  it->funnyFunction();

[greenman]

great

if I understand you correctly, then there is no function yet, which could give us the first element of the list ?

edit: In order to implement (and mostly test) the GameStateManager, we would need a merge of the branches (network and objecthierarchy) as soon as you (x3n) can provide everything Do you have a clue, when this'll approximately be?

[x3n]

Stop asking that many questions and I'll have it finished soon

And yes, you're right with the first element. At the moment the iterator starts automatically with the first element, but that's stupid if you want to use the -- operator (which might be interesting in some weird situations). Oh, and it looks a bit more STL-like if there are start() and end() functions.


At the moment its just

Code: Select all

for (Iterator<class> it; it != 0; ++it)

or even shorter

Code: Select all

for (Iterator<class> it; it; ++it)

But this looks a bit like magic, so I decided to implement a start() function, but that's causing some problems because the objectlist is hidden.

[x3n]

Code: Select all

for (Iterator<Synchronisable> it = ObjectList<Synchronisable>::start(); it != 0; ++it)

edit: There's still a bug in it (a very strange one), but I hope you'll be able to use the code as mentioned in this post. The bug is somewhere intern.

[greenman]

Cool
Yes, I think that we can use it that way (no, we only need your files in our branch...).

How is the speed of this list? Would it be worth it, if we cached it in order to access the elements faster? I mean we could, for example, put the pointers into an array and map the object-ids to the array iterator, instead of going through the list everytime we have to save something into an element (reading would still be with the list directly).