Directory structure in /src

A place to discuss everything about the Orxonox framework.

Moderator: PPS-Leaders

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Directory structure in /src

Post by x3n » Wed Aug 12, 2009 1:12 pm

Hi, as you may have noticed, I'm currently dividing orxonox into smaller libraries (the old orxonox executable was built out of ~180 source files). This should help to decrease the linking time (on tardis it takes up to 4 minutes) and of course forces us to write nice code. This change also brings a complete new class of libraries, called "plugins" - they're loaded dynamically at runtime.

Now let's talk about the topic of this thread: With those changes we'll end up with a whole bunch of libraries and we definitively need a better structure in the src directory.

Let me start with a first proposal:

Code: Select all

src
  extern
    bullet
    ceguilua
    cpptcl
    ogreceguirenderer
    ois
    tinyxml
    tolua
  orxonox
    libraries
      core
      network
      tools (new)
      util
    main
      main library (new)
      orxonox executable
    plugins
      overlays (new)
      questsystem (new)
      weaponsystem (new)
      ...
Short explanation: The old orxonox executable was split up into a small (1 file) executable and one large (~180 files) library. Then the main library was divided up into smaller parts, some of which are linked into the main library (like tools) and some of which are built upon the main classes (plugins). So we basically have this structure:

Code: Select all

libraries
|  |  |
 \ | /
   v
main lib --> executable
   |
 / | \
v  v  v
plugins
That's what I want to reflect with the directory structure. And to make it more obvious, I also propose to put the external libraries into a separate folder.

What do you think?
Fabian 'x3n' Landau, Orxonox developer

User avatar
beni
Baron Vladimir Harkonnen
Posts: 949
Joined: Tue Oct 03, 2006 9:15 am
Location: Zurich
Contact:

Re: Directory structure in /src

Post by beni » Thu Aug 13, 2009 12:46 pm

I don't really understand much to be honest. Generally I see what you mean and if there is such a large advantage with the linking time I'm all for it.

What do you mean with the nicer code? What would change and what advantage would that change include regarding coding?

The directory structure proposal looks good as far as I can judge.

Are there possible disadvantages and is the term "plugins" wisely used in this context?
"I'm Commander Shepard and this is my favorite forum on the internet."

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Directory structure in /src

Post by x3n » Thu Aug 13, 2009 4:41 pm

Don't get me wrong about the linking time: Overall compile and linking time (for 100% of the project) will probably be increased, particularly on single core systems. But on multi core systems you will probably spend less time with linking, because many libraries can be linked in parallel (this holds for example for the network & tools libraries, or for independent plugins) while this was impossible with the old orxonox executable.
But the most important effect is that you don't have to relink the whole game if you change one line in one file. You just have to relink the library or plugin to which your file belongs (and link this library into other depending libraries if any).
This is very important for PPS students because for some unknown reason the linking time on tardis exploded by about a factor of 10 in the last months. This is extremely annoying and ruins the workflow. But with the new system, most students will probably work on a plugin and not on the main library, which will safe a lot of time.

About the nicer code: It forces you to write modular code and encapsule features in one part of the code instead of spreading your code all over the whole repository. It also forces you to use clean interfaces to interact with other classes (if they're in another library/plugin/part of the game) instead of using a hardcoded list of featured classes. And last but not least you have to build some features upon other features instead of linking and mixing them inseparable. This increases the flexibility of your code and decreases the risk of producing bugs if you change something in the future.

The term "plugin" is used because they're loaded dynamically at runtime, which means you can't use features of those plugins in the main library, hence all features of the plugin must be self-contained. Of course this doesn't necessarily mean they're optional, because in fact all of our plugins will be needed by at least one part of the game, but this is not a hardcoded dependency but rather a consequence of our level and menu design. For example if you omit the "overlays" plugin the game still loads and runs, but without a HUD. So you're obligated to load the plugin if you want a reasonable game, but it's "plugged in" at runtime.
However there IS the possibility to produce real optional plugins later in the development process, for example for modifications/mutators or stuff like editors or generators. Because they use totally the same technique, I decided to name them all "plugins". But if you know a better term for "not-really-optional plugin" please make a proposal, because I'm also not 100% happy with this choice.
Fabian 'x3n' Landau, Orxonox developer

User avatar
greenman
Baron Vladimir Harkonnen
Posts: 360
Joined: Wed Oct 03, 2007 2:53 pm
Contact:

Re: Directory structure in /src

Post by greenman » Thu Aug 13, 2009 4:56 pm

sounds good, but do we really need the orxonox folder? i think src/{extern,libraries,plugins,main} would be enough ?!

as for a different solution for plugin: we could use modules
i think this also expresses that they're loaded at runtime and especially that they're optional
There are only 10 types of people in the world: Those who understand binary, and those who don't.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Directory structure in /src

Post by x3n » Thu Aug 13, 2009 7:40 pm

Yes! Good points! I like it :)
Fabian 'x3n' Landau, Orxonox developer

User avatar
beni
Baron Vladimir Harkonnen
Posts: 949
Joined: Tue Oct 03, 2006 9:15 am
Location: Zurich
Contact:

Re: Directory structure in /src

Post by beni » Sat Aug 15, 2009 9:13 am

I see. Sounds good. "module" might really be a better..
"I'm Commander Shepard and this is my favorite forum on the internet."

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Directory structure in /src

Post by 1337 » Sat Aug 15, 2009 10:25 am

idea.

I agree more or less. Though I object to the folder structure. I would remove the libraries folder and put them directly into the orxonox folder. I suppose there's not going to be that many libraries.
The same could be done with the main folder by finding an appropriate name for the main library (which includes stuff like Level, Scene, WorldEntity) and calling the executable simply main (with orxonox as output name of course).

Of course I can see that this imposes one orxonox folder with quite many direct subfolders, but I don't see the point in making the folder structure too deep. Consider the following:
game/code/branches/mybranch/src/orxonox/libraries/core
(quite a mouthful..)

The module folder is ok like that I think. I also suggest to remove all traces of the name "plugin" and rename everything to plugin. Or don't use module at all. Though I prefer the module version. That would also better fit when considering that modules could depend on other modules, whereas with plugins it might almost seem like they're all independent.


And there's something else I would like to suggest: Apart from the ScopedSingleton it might be useful to let every module have a static function that gets executed just before loading the GameStates so the module can do some sort of initialisation. I would not do this at module loading time since then there's not even configValues. We could introduce a second function that gets called upon graphics loading but I don not claim this is necessary, just an idea.
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Directory structure in /src

Post by x3n » Sat Aug 15, 2009 1:21 pm

Ok, seems like "module" is the name of choice. Fine. :)
I'll rename this in a few days, but for the moment I'm heading towards the next exam.
And there's something else I would like to suggest: Apart from the ScopedSingleton it might be useful to let every module have a static function that gets executed just before loading the GameStates so the module can do some sort of initialisation. I would not do this at module loading time since then there's not even configValues. We could introduce a second function that gets called upon graphics loading but I don not claim this is necessary, just an idea.
Hm, basically I see the point about loading- and initialization-functions in modules, but in all modules I did so far there was no need for this. Do you see an example where it could be handy? The main "problem" probably is that something like a module-main-function also request a module-main-file. This makes modules much more powerful since this allows them to execute more than just some static "hello here I am" initialization code (aka CreateFactory). While this may be nice from the modules point of view, it also spreads the initialization code over several files which makes it less predictable. So personally I don't dislike the suggestion, but we should have a real reason for doing so.

---

Now after this threads topic shifted from directory structure to modules in general, I also want to discuss another point:
We have some code-parts in orxonox which are basically built like a module (mostly independent and self-contained), but in one point linked with a main class. This holds for example for the weaponsystem or the pickupsystem which both are comparable large parts of the code which will also expand much more in the future (we'll have tons of weapons and pickups). Both subsystems are completely independent of other code, but they are linked with the Pawn class (each pawn has a WeaponSystem and a PickupCollection). And both WeaponSystem and PickupCollection can't be separated from the rest of the module, because they each need ~5 other classes.

Now there are two options:

1) Put all basic classes of those subsystems into the main library and create a module only out of the spezialized weapon and pickup classes.
While this may be a good idea general, it also shows that for each new subsystem in the game the main library keeps growing and therefore it's impossible to add a new feature (which is somehow linked with the Pawn) in a module only (or a "real" optional plugin).

2) Don't link those subsystems directly with Pawn. Instead use the fact that we're facing a "has a" relation and outsource the link to somewhere else, for example (A) a static map in the module (like static map<Pawn*, WeaponSystem*>) which stores an instance of the subsystem for each Pawn.
We could also use (B) anonymous classmembers in Pawn, for example Pawn::map<Identifier*, BaseObject*> which would be used like this: mypawn.map[Class(WeaponSystem)] = new WeaponSystem() and afterwards myweaponsystem = orxonox_cast<WeaponSystem*>(mypawn.map[Class(WeaponSystem)]).
Of course both solutions need a map which slows the code down, but by definition those subsystem aren't central (you can only shoot once per tick or pickup one item at a time) an because the modules are independent, no other class can use them in other situations.
So the main problem of both suggestions (A) and (B) boil down to the need for a construction- and destruction-listener for Pawn (and for any other class in general).

Personally I like option 2, but I'm not yet decided whether I prefer (A) or (B). However my intention goes towards (B) since it doesn't need a static map (and "anonymous classmembers" sounds so damn cool I just HAVE to implement it :D)
Fabian 'x3n' Landau, Orxonox developer

User avatar
greenman
Baron Vladimir Harkonnen
Posts: 360
Joined: Wed Oct 03, 2007 2:53 pm
Contact:

Re: Directory structure in /src

Post by greenman » Sun Aug 16, 2009 6:57 am

as for reto's suggestion to the folder structure: I prefer the version with a library, module and main subfolders. I don't think we have to use the full path /code/branches/xxx/src/libraries/core really often (at the moment I don't see any case where we use this ^^).

about the dependency problem: what is the problem if some modules are dependant on/link against other modules?
There are only 10 types of people in the world: Those who understand binary, and those who don't.

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Directory structure in /src

Post by 1337 » Sun Aug 16, 2009 8:43 am

Actually the point was not that we're going to use the whole path anywhere. Then never mind my argument. Though I do not change my opinion without a good argument ;)

About the dependencies: Linking modules against modules should not be a problem on Linux but it is a rather big one on Windows. For development runs we have run scripts, so adding another directory to the PATH variable is easy.
But I see a big problem for the installed version (and no, I would really like not to use a run script there). But maybe we can modify the PATH variable from inside the program (only possible when loading dynamically of course).
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Directory structure in /src

Post by 1337 » Sun Aug 16, 2009 9:19 am

Just successfully tested a solution to the PATH problem. Simply insert this code before plugin loading and you're fine (should work on Linux too if the rPaths weren't set correctly):

Code: Select all

std::string pathVariable = getenv("PATH");
putenv(("PATH=" + pathVariable + ";" + configuration_->pluginPath_.string()).c_str());
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Directory structure in /src

Post by x3n » Sun Aug 16, 2009 12:10 pm

Or we move the plugins from lib/plugins to bin
Fabian 'x3n' Landau, Orxonox developer

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Directory structure in /src

Post by 1337 » Sun Aug 16, 2009 12:16 pm

I liked the idea of not having everything in bin just because of windows. The solution I presented should work quite well I suppose.
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Directory structure in /src

Post by x3n » Sun Aug 16, 2009 12:25 pm

Yes indeed. But just in case it doesn't work, we'd have a simple fallback solution.
Fabian 'x3n' Landau, Orxonox developer

User avatar
greenman
Baron Vladimir Harkonnen
Posts: 360
Joined: Wed Oct 03, 2007 2:53 pm
Contact:

Re: Directory structure in /src

Post by greenman » Mon Aug 17, 2009 7:34 am

does this solve our linking modules against modules problem, or was this another issue?

if it doesn't is it a problem to link a module against another one in just one direction?
There are only 10 types of people in the world: Those who understand binary, and those who don't.

Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests