Page 1 of 1

XMLFile loading

Posted: Mon Jul 20, 2009 11:24 am
by 1337
While looking through the Ogre Resource Management I realised something quite interesting:
The actual resource you can create (for instance with MaterialManager::create(...)) is of course not always the file itself but rather part of a content of a file. One *.material file may of course hold several materials.
The scripts get compiled upon initialiseAllResourceGroups(). However the resources are not being loaded there.

All that made me think: We could do the same with our XML files by parsing them at startup (or per reload button so you don't have to restart orxonox when editing level files). I'm not sure whether this works with the XML Templates (core class) though. But we could of course simply store temporary XML code (without the lua code) if that's easier.
The point I'm trying to get to is simple: Instead of giving Loader::load() a file and receiving nothing, we could give the LevelManager the name of the level and receive a orxonox::Level pointer.

I would implement this by deriving the LevelManager from Ogre::ResourceManager and write some code that can identify Level-XML-Nodes without actually loading them.

That idea could be extended for OverlayGroups or whatever we need to load as a whole. So the whole thing is more of an advanced orxonox::Template class...

What do you think?

Re: XMLFile loading

Posted: Mon Jul 20, 2009 11:45 am
by beni
I think that is great. Finally we get some insight in the more sophisticated loading process of the Ogre engine. I had a quick look at the system some years ago, but Orxonox was nothing then at the moment and I was just happy to be able to load a some meshes.
Your approach sounds fine I guess. Since the lua code only generates correct XML we should store that generated code temporarily, but rerun the lua code when reloading.

However reloading the level isn't really something for the user, but it could be a first step into the direction of a level editor. So this reloading function would be useful in the future anyway, maybe in another context then.

Re: XMLFile loading

Posted: Mon Jul 20, 2009 11:53 am
by 1337
Reloading is probably best interfaced with a console command. It's just a detail but we should keep that in mind in case my suggestion were to be found valid.

Re: XMLFile loading

Posted: Mon Jul 20, 2009 4:33 pm
by x3n
I thought about that too when we had all the include problems for overlay and template classes in XML files (you have to include the corresponding overlay file if your levelfile gives the player spaceship with a HUD). Of course it's "correct" if you always have to include the needed stuff, but there are some cases where it's a problem: We have Gametype specific HUDs (for example the Pong-HUD or the Health-Bar of the destroyer in the UnderAttack gametype). All those HUDs are currently defined in default.oxo (which also includes the default HUD with the chat overlay, because default.oxo is the only file which is loaded "hardcoded" - yes it's a hack). And this sucks - we really need a dynamic load-by-name function for this.

But: I'm pretty sure parsing all XML files at startup (and remember - somewhen we'll have a fucking great universe with thousands of spacesations and planets and of course hundreds of user created levels) will be a serious performance issue. It just takes ages to run each file through the Lua parser and after this through the XML parser just to get the name of the root-node (or, if we're using "incorrect" XML-syntax for a collection of classes, the name of all root-nodes).

And you also noticed the need for a reload button - yet another no-go in game design (and usability in general).

That's why I think we need a different approch. One of the simplest of course is to use the filename itself as the name of the root node. We could then use some sort of paths (maybe XPath if we want to start high - http://en.wikipedia.org/wiki/XPath) to point to the classes. So if we want to load the Pong gametype HUD which is still in default.oxo, then we simply load "default.pongHUD". The same notation could be used within other XML files to load templates (templates of Spaceships are needed for spawnpoints), so for example <SpawnPoint ... spawnclass="pirates.HeavyBomber" /> which would then load the whole "pirates.*" file and search for a class named <Template name="HeavyBomber"> ... </Template>

And levels are of course still loaded by name. Returning a Level* pointer makes sense if we agree to downgrade the Loader (it was initially built to load any type of XML files and to create all classes inside. and not just whole levels, but also parts of levels like for example a group of pirates. but since we're using the more generic xml templates and dynamic lua scripts, there's not much need for this feature anymore).

Re: XMLFile loading

Posted: Mon Jul 20, 2009 10:52 pm
by 1337
Not much of a reply today (it's late and I should really go to bed after an interesting match of night speedminton ;)):

Performance might be a problem, Ogre::initialiseAllResourceGroups already takes quite a few seconds (it parses all the scripts). On the other hand when you compile a source file that includes boost/thread.hpp then it only takes maybe four seconds but pareses an enormous amount of code (in an optimised environment of course...).

We could of course bind the file extension to the type of root class encountered and force it with loading exceptions. A little enumeration (in whatever form) might help there.

That's it for tonight and yeah, I haven't really thought about your answer yet x3n :P

Re: XMLFile loading

Posted: Tue Jul 21, 2009 8:31 am
by beni
Concerning the performance: Their are other functions beside initializeAllResourceGroups. Loading resources in Ogre has many steps and you can initialize, load, unload and stuff for each resource group separately. So this is possible to achieve with a bit more effort in splitting up our media into sensible resource groups and handle them as we need to.

Re: XMLFile loading

Posted: Tue Jul 21, 2009 8:55 am
by 1337
Exactly. You can even use those resource groups dynamically. So the group "Level" might change when you load a level. This way you don't have to deal with hundreds of different resource groups.
My idea was somehow provide an easy way to determine the resources associated with a level, but I have no idea how yet.

Re: XMLFile loading

Posted: Wed Jul 22, 2009 10:12 am
by greenman
as ogre provides some form of multithreaded/background resources loading, would it be possible to do other tasks (like class hierarchy creation, input initialization, ...) while loading/parsing the files in background?

Re: XMLFile loading

Posted: Wed Jul 22, 2009 10:19 am
by 1337
Unfortunately Ogre doesn't really support MT, or would you like to create MT packages for Ubuntu, Debian, etc.?
The current packages haven't been compiled with MT support.

Re: XMLFile loading

Posted: Wed Jul 22, 2009 3:56 pm
by beni
Well the OGRE forum is spewing from stuff about multithreading. However mostly about how and if one does do multithreading during runtime inside the engine or the efficiency to do multithreading with parts like AI and physics. OgreBullet for instance is multithreaded.

About loading resources I didn't find too much, but I imagine that the OGRE forums are worth a loow.

Re: XMLFile loading

Posted: Wed Jul 22, 2009 4:37 pm
by 1337
You can load any kind of resource in the background if Ogre was compiled accordingly. No doubt about that (there is information about that in the API documentation as well as in the book).

Re: XMLFile loading

Posted: Thu Jul 23, 2009 3:17 pm
by greenman
i think we should just use this whether it is supported (today) in the binary packages or not. probably if this feature is more stable it will become default for the various distribution packages.

my point is: if we now rethink/restructure our loading stuff we should definately support this, because it will probably be much easier than change everything later on when we need it.

Re: XMLFile loading

Posted: Thu Jul 23, 2009 5:57 pm
by 1337
On that point I totally agree of course. I've already had a few thoughts about this, but just a few..