Enhancement of the XML connection

[x3n]

The last few posts were a bit technical, speaking a lot about XML. However there's a a second technique involved in our level files, that should not be forgotten: Lua scripts. So far we used Lua to include other XML files or to place random asteroids in the level. Also the randomized space station is written entirely in Lua. Additionally we planned to do even more scripting in the future, for example to include some pirates or some police ships in the level, depending on the fraction of the player, to make it challenging either way.

Now while this was all nice and good so far, it turns out that this will maybe not be supported anymore after reworking the XML connection in Orxonox. Why? Because of the way we process Lua scripts in level files.

That's how it works so far:

1. We have a level file with some Lua script in it (simplified XML code):
   this code first includes another file, called "other_file.xml", then creates 5 randomly placed asteroids whose scale increases from 1 to 5.

   Code: Select all

   <?lua
     include("other_file.xml")
   ?>
   
   <?lua for i = 1, 5 do ?>
     <Asteroid position="<?lua print(math.randomVector()) ?>" scale="<?lua print(i) ?>" />
   <?lua end ?>

2. The level file is loaded, but before XML parsing starts, our preprocessor removes all <?lua ?> tags and encloses the XML code in print("...") functions:

   Code: Select all

     include("other_file.xml")
   print("
   ")
   for i = 1, 5 do
   print("  <Asteroid position=\"")
   print(math.randomVector())
   print("\" scale=\"")
   print(i)
   print("\" />")
   end

3. This Lua script is now executed by the Lua interpreter. The output (everything printed by the print() function plus everything included by the include() function) defines the final XML code:

   Code: Select all

   <some>
     <content />
     <of />
     <the>
       <included />
       <file />
     </the>
   </some>
   
   <Asteroid position="7,6,2" scale="1" />
   <Asteroid position="2,2,9" scale="2" />
   <Asteroid position="6,4,7" scale="3" />
   <Asteroid position="9,6,3" scale="4" />
   <Asteroid position="4,8,2" scale="5" />

4. This XML code is then finally parsed by the XML parser.

So where's the problem? The problem is that if you save the XML file (e.g. after editing it in an editor), you will lose all Lua code. Instead you will only save the XML code back to your level file, which was generated by the Lua script in step 3. so if you load a level file which includes some other files and contains some Lua code, you will not only lose the Lua code, but the included files will also be saved inside your level file. This makes inclusion more or less pointless.

At this point you probably understand why I'm so eager to get rid of the include() calls. And if you've read the previous posts, you know that there are different ways to replace them, so this is not a big deal.

However the future of the Lua code within level files is a bit unclear. If we ever create an editor, it will certainly replace a lot of functionality that was achieved through Lua code so far, for example to create lots of randomly placed asteroids - a simple script in the editor will do the job. Other scripting can be moved into script objects, i.e. BaseObjects that contain Lua code that will be executed if the object is triggered. This can be used for story scripting and to include stuff like the randomized space station.

Finally, there are even feasible solutions to write Lua scripts that can be saved back to XML, for example if we use a helper object that remembers the Lua code and also manages the objects that are created by this code.

Your thoughts about Lua code in XML files and possible solutions for the presented problems are of course very welcome

[Mozork]

That sounds awesome.

We might also want to have some randomization in a level, to have levels that offer a different experience each time they're played. (So beyond randomization on level creation time) Much like the random maps e.g. in Age of Empires II.

But I think we can still do (and should) without lua in the Level files, and allow for dynamic objects in the form of Snipplet(-Objects) that can be included in XML and either execute lua or C++ code to do some stuff based no their parameters. (e.g. randomly distribute some asteroids)
This would also make the task for the level designer simpler, because he just has a range of Snipplets to choose from.

[beni]

Okay, that last post from x3n clearly showed the real challenges of this project. Indeed the back and fourth with lua and XML could/should be omitted and/or be simplified. Of course we won't be able to get rid of all lua code ever, which is not the idea anyway, but putting it into neat and practical script objects is a long overdue task.

One thing on randomization though: We could create huge levels with very little space when we create certain parts of our levels procedurally. I haven't thought it through, but I think it should also be put into consideration. Usually you would use a certain seed for the randomization function and you'll always create the same randomness. Which could be very neat.

BTW: The minecraft world generator works like this. You can enter a single seed in the beginning which will create the exact same huge world every time.

[1337]

I'm guessing that we're not going to have a reliable editor very soon. So it might be imperative that we keep some part of the XML scriptability.

However, as Fabian suggested, I would omit the pre-processing. Instead we might do post-processing of the attribute strings for now. Additionally, the Lua code could be initiated in special lua tags as we have it now. But that code should not be able to manipulate the XML file, but instead just the Lua state.
That would already deal with most of the cases we use Lua in level except for the repetitions. But this could be handled by a special generic XML object (whose attributes can again be lua-scripted).

It sounds a little complicated, but without an editor I don't see a way around some scripting in XML. And currently this is what I can come with give the other precondition (no pre-processing). Unfortunately it will not give us the same flexibility and easy implementation, but at least, the behaviour seems much better defined.

Let me give a little example as to how it could look like:

Code: Select all

<?lua
i = 0
?>
<ForLoop repetitions=5>
  <?lua i = i + 1 ?>
  <Asteroid position="@L math.randomVector() L@" scale="@L i L@" />
<ForLoop/>

Everything between @L and L@ would be evaluated as ONE expression and converted to a string. Calling a function without return value would then result in "", multiple arguments might be separted with a comma (just an idea).

I've come up with and worked out the idea within ten minutes, so there is a lot of room for suggestions. I'm not even sure whether it's any good at all

[greenman]

wouldn't that be pre-processing too (e.g. the code for scale)?
or how do you differentiate post- and pre-processing? post and pre xmlport?

[1337]

I would define it in the very same way as with C++: If you modify the file before putting it through the XML parser it would be pre-processing.
And if you use the output of the XML parser for processing, that would be post-processing.

[x3n]

We might also want to have some randomization in a level, to have levels that offer a different experience each time they're played. (So beyond randomization on level creation time) Much like the random maps e.g. in Age of Empires II.

But I think we can still do (and should) without lua in the Level files, and allow for dynamic objects in the form of Snipplet(-Objects) that can be included in XML and either execute lua or C++ code to do some stuff based no their parameters. (e.g. randomly distribute some asteroids)
This would also make the task for the level designer simpler, because he just has a range of Snipplets to choose from.

I always advocate level generators for Orxonox, because we are a bunch of coders and simply don't have the resources to create lots of levels by hand. So the randomization of certain parts of a level is very interesting in this context and I want to discuss that a bit further.

There are some points about randomized levels which makes this a quite complicated topic. First of all, it's unclear where the randomization takes place. There are different possibilities:

1. We use lua code in our XML files which will be executed while loading the XML level to create certain parts of the level randomly (e.g. asteroids, pretty much like we do it now)
2. We use a script in an editor which creates a randomized level, which will then be enhanced with hand-placed objects like space stations and quest elements. Finally the level is saved as a static XML file.
3. The level is saved as a static XML file and loaded without preprocessing, but it contains script objects which will create some randomness after loading the level (i.e. at runtime)
4. We could implement a level generator which creates the whole level randomly. So the player has the choice to either play a handcraftet level (loaded from XML) or a random generated level (which doesn't need an XML file at all, or maybe just some templates).

I don't have a clear preference here... the 1st option makes only sense when we write the XML files by hand, without editor, while option 2 requires an editor. option 3 can be used either way and also contain scripts that are not used for randomization. option 4 finally is completely independent of XML files and editors.

So in my opinion we may very well implement all 4 options, with the 1st one being a temporary solution until we have an editor and the 2nd one being its replacement afterward.

After all we shouldn't forget that randomization bears also some difficulties. For example randomly placed asteroids could block a spawnpoint - so we either need very smart scripts to place the asteroids, or the position of the asteroids has to be the same each time we load the level, so that we can ensure all spawnpoints are free.

Additionally, in multiplayer matches we have to ensure that all players see the same version of the level. Right now this is no problem because clients don't load the level themselves, all objects are synchronized with the server. But in future we may have some "static" objects that can be loaded through XML, while only the "dynamic" objects are synchronized. In this case, randomization could be a problem.

One thing on randomization though: We could create huge levels with very little space when we create certain parts of our levels procedurally.

We don't necessarily need procedural code to use random seeds. There are many ways to create random levels and you can always use a seed. But you're right that this is a great way to create random levels and still be able to re-visit some levels that you liked (if you remember the seed).

What you probably refer to is that you don't actually have to "create" the whole level once you defined the seed - you can just define some functions that return whether or not there is some object at some position. Whenever the player goes to some position, we will call these functions and create the objects on the fly. If the player moves away and the objects get out of sight, we destroy them again. If the player later comes back to the same location, we will create the same objects again, because the functions still return the same value for this position.

This way we save a lot of memory and don't have to store the objects in XML. However it's also quite difficult to get good results with this approach, since you can not call each function with literally every possible position... additionally it's hard to connect two functions, e.g. if you don't want to put a spacestation into an asteroid field, the function which defines the presence of spacestations has to know the output of the function which is responsible for asteroids.

There is another space game which uses something like that and we also discussed it quite a bit in the past, but I can't seem to find the topic in the forum and I also don't remember the name...

[x3n]

Now let's go back to Lua code in XML files.

I'm guessing that we're not going to have a reliable editor very soon. So it might be imperative that we keep some part of the XML scriptability.

Yes indeed. We could just go on with preprocessing until we have an editor, then drop it. Or we use lua in a way which also works together with an editor. In either way this corresponds to what I described in the 1st option in the post above.

Your example shows quite well how this could be done. In your example the XML elements would store the lua code and re-create it when the level is saved. You can do this even without @L L@ tags, for example like this:

Code: Select all

<ForLoop repetitions=5>
  <?lua i = i + 1 ?>
  <Asteroid>
    <position>
      <?lua math.randomVector() ?>
    </position>
    <scale>
      <?lua i ?>
    </scale>
  </Asteroid>
<ForLoop/>

This works, because XML doesn't enforce us to define variables as attributes, we can just as well define them as sub-elements. The XML parser will then take care of the <?lua ?> elements and all we have to do is to save the lua code in a helper object.

I'm not so sure however if constructs like <ForLoop> are a good decision. Maybe something like this works better:

Code: Select all

<LuaScript>
  <?lua for i = 1, 5 do ?>
  <Asteroid ... />
  <?lua end ?>
</LuaScript>

or maybe even like this, since XML also allows text within XML elements:

Code: Select all

<LuaScript>
  for i = 1, 5 do
  <Asteroid ... />
  end
</LuaScript>

I would say that we try to continue support for lua code in XML files, at least until everything can be done with an editor. However we have to make the lua code safe in a way that it doesn't require preprocessing and can be saved back to XML again.

[1337]

I like your improvements to my suggestions. Though I'm not sure how well your for loop works because it still requires XML processing after executing Lua code, meaning that the XML parsing has two stages.
Do you have a good idea as to how to implement this?

[x3n]

Yes, that's true and also intended. If we have a very general load(xml) function which just returns the created objects, then this should work. The question is rather how well we can handle the different parts of lua code (since the for-loop requires the end statement, but this is on a separate line).

As a counter question, how would you implement the <ForLoop> object? If this is really an object, it will probably also require more than just one stage. If it's not an object, then this means our XML loader becomes a primitive script interpreter. Or maybe you had another idea in mind?

Anyway, all these solutions have the problem that the XML parser will of course not understand this kind of scripting. This means we may lose the ability to validate the XML or to refer to these objects with XPath. In the worst case we even get errors because for example <bla><?lua blub ?></bla> evaluates to <bla></bla> which may be invalid in some cases.

On a side node, in any case the XML code would only be parsed once. That's why it's also important to have valid XML code (we do no preprocessing). But after the XML parser is done and returns a tree of C/C++ objects that represent the XML document, we will load them (i.e. create the corresponding orxonox objects and assign values to variables). This can be done in different stages without problems or performance issues.

[1337]

As a counter question, how would you implement the <ForLoop> object? If this is really an object, it will probably also require more than just one stage. If it's not an object, then this means our XML loader becomes a primitive script interpreter. Or maybe you had another idea in mind?

No. I see your point.

Anyway, all these solutions have the problem that the XML parser will of course not understand this kind of scripting. This means we may lose the ability to validate the XML or to refer to these objects with XPath. In the worst case we even get errors because for example <bla><?lua blub ?></bla> evaluates to <bla></bla> which may be invalid in some cases.

Are you sure this is invalid?

On a side node, in any case the XML code would only be parsed once. That's why it's also important to have valid XML code (we do no preprocessing). But after the XML parser is done and returns a tree of C/C++ objects that represent the XML document, we will load them (i.e. create the corresponding orxonox objects and assign values to variables). This can be done in different stages without problems or performance issues.

I don't quite see this, you'll have to help me.
We parse the XML file once, then we get some lua code that evaluates to multiple XML nodes. We will have to parse these as well. But how do we integrate that into the parser's XML tree?

[x3n]

Are you sure this is invalid?

No, not in general. But let's say we use a constraint in an XML Schema which doesn't allow a nil value for an attribute, this will result in an error because of invalid XML code.

On a side node, in any case the XML code would only be parsed once. That's why it's also important to have valid XML code (we do no preprocessing). But after the XML parser is done and returns a tree of C/C++ objects that represent the XML document, we will load them (i.e. create the corresponding orxonox objects and assign values to variables). This can be done in different stages without problems or performance issues.

I don't quite see this, you'll have to help me.
We parse the XML file once, then we get some lua code that evaluates to multiple XML nodes. We will have to parse these as well. But how do we integrate that into the parser's XML tree?

We parse the XML file once, then we get some XML nodes: some that contain lua code and some that represent an Orxonox object. The lua code will not create new XML code, it will rather call the load() function on the object-node.


A side note about the advantage of a general <LuaScript> object compared to a <ForLoop> object: It makes it easy to use it also for if/else blocks, which would be quite complicated with pure XML elements (except maybe with <if><else/></if>, but that's rather ugly)

[1337]

We parse the XML file once, then we get some XML nodes: some that contain lua code and some that represent an Orxonox object. The lua code will not create new XML code, it will rather call the load() function on the object-node.

Don't we have to have these nodes in an XML representation in the first place? I mean, XMLPort() gets XML node, not a string when loading an object. Or is this a change in the new XML system?

EDIT:
We've worked it out via IRC The idea is to be able to parse everything beforehand and let the scripts only do operations on theses parsed nodes like multiplicate (loop) or connect to a condition (if).
END_EDIT

A side note about the advantage of a general <LuaScript> object compared to a <ForLoop> object: It makes it easy to use it also for if/else blocks, which would be quite complicated with pure XML elements (except maybe with <if><else/></if>, but that's rather ugly)

I fully agree: if we have the possibility to use Lua in a non-preprocessing way, then I would always prefer it.

[1337]

A few days ago I recalled something I've given some thoughts a very long time ago: loading objects.

Withe the redesign of the XML system, we should probably also discuss that beforehand.
Currently, loading an object consists of multiple steps:

1. Construction
2. XMLPort() call
3. tick() call

In almost all cases the third point isn't relevant. But I believe this currently is being use in some classes, at least in an implicit way (e.g. mesh LOD for the SpaceShip chunks only get generated upon first kill).

Then, as x3n already mentioned I believe, we have a problem with default values being mixed in c'tor and XMLPort. But I'm not going into that right now.
What I'm interested in is actually trying to find a well defined loading process.
I'll make a few suggestions as to what could be done without judging any of them (just some brainstorming):

1. XMLPort could be part of the c'tor like setConfigValues
2. Some object's initialisation might be based on it's surroundings (AI objects maybe). So we might have an additional loading call after ALL objects have been generated.
3. The above point could also be done iteratively
4. An object should never have config values and an XMLPort at the same time

A few comments by my humble self:
Point 1 seems unsuitable to me because it doesn't allow virtual calls.
Point 3 might lead to frequent infinite loops (much like the frequent isInitialized() problem).
A feel quite strongly about points 2 and 4.

Another problem that currently is not well resolved in my opinion is creating objects after the official first loading process. This concerns projectiles, spawning space ships, pickups and a lot of other objects. Sometimes we use existing XML nodes if I remember correctly for use with XMLPort(). And sometimes it's just a hack.

So much from my side, please feel free to issue comments.

[x3n]

Yes that's definitely an interesting topic.

I have some clear guidelines I want to adhere while impementing the new XML system. This adds some constraints to the possible solutions.

1. Avoid inconsistency. It shouldn't matter if you create an object in C++ (new MyClass()) or in XML.
   => If we set default values in XMLPort, they should also be set if you create an object in C++. Or we don't set any default values in XMLPort at all.
   => Everything you can do in XML should also be possible in C++. You may think this is obvious, but right now we break this rule whenever there's additional code in XMLPort or if the XMLPort macros use private functions or variables. Example: MobileEntity.
2. All XML attributes should be changeable after the object was created and also after the object was ticked. If we build an editor, the level creators will play with the values and they should see the result.
   => It shouldn't matter in which order XML attributes are loaded (currently the order is defined by their order in the XMLPort function and it's possible that some code depends on it).
   => Objects must initialize on the fly. If a model gets a new mesh source through XMLPort, it has to change its appearance
   => Loading objects shouldn't depend on tick() because I'm not sure if we'll have a tick in the editor and also because you have no guarantee that XMLPort will always be called before tick()
3. Loading should not depend on the order of the objects in the XML file. This is a tricky one, because usually order DOES matter in XML files and we may rely on this in hand-written XML files. But if you create objects in an editor, it will have no sense of order. So loading should be independent of it.
   => If object A refers to object B, object B may not yet exist while A is being created. Since the reference is defined in XML, the loader should still be able to resolve the problem. In combination with guideline (2) it's also fine to set the reference at any later point during the loading process.
   => This guideline is also important for network synchronization since we don't know exactly in which order objects are synchronized.
   => Note that you may still have a defined order of subobjects (e.g. waypoints). I'm just talking about the loading process and references (e.g. if two objects are connected with events)
4. No postCreate() and preDestroy() functions (*see below). At least if this is possible in any way. These types of functions are not in the spirit of object oriented programming. Unfortunately the way virtual functions work (or don't work) in C++'s constructors and destructors is also not really in this spirit, but in almost every case it's possible to work around this problem.
   => No complex stuff in constructors/destructors. In particular no virtual functions.
   => I know these functions are not exactly what you meant with "additional loading call", but it's related. I don't like it for the same reasons and also because sometimes it's hard to tell when you finished loading (think about network synchronization, background loading, paging).

(*) I'm talking about something like this:

Code: Select all

MyClass* object = new MyClass();
object->postCreate();

object->function();

object->preDestroy();
delete object;

So I think this gives an impression of my general mindset, but it's way too early to make any final conclusions. In my opinion, creating an object just by calling "new" should be enough. After that, the object is in a neutral state where it doesn't crash and can be modified at any later time.

Just one question:

An object should never have config values and an XMLPort at the same time

I don't undestand this one. Why is that a problem?


Edit: An addition to my comment about postCreate() functions: I'm only refering to mandatory functions that must be called after the constructor. Of course this doesn't apply to optional setXYZ() functions. You're always allowed to change the settings of your object after creating it.

Edit 2: Clarified the point about order of objects in the XML file