data repository

[1337]

Hi all,

Since I'm not actually productive anymore in terms of coding, I can start muttering

Lately I've had a look at the data repository. My findings in a short summary:
- There is the old trunk/branches structure from Orxonox V1 and the new media folder
- Contentcreattion folder where all the artists put their work
- old_media folder with outdated material from Orxonox V2

Furthermore there are three old trunks: branches/old.trunk/ dates Juli 06 and old_trunk/ dates January 07. And there is of course the newest of the old trunks in the trunk/ folder, dating June 07.
I think this is quite a mess, especially since we don't modify it anymore anyway.


But that's the smaller problem I see. Suppose you want to sort out my preliminary work on the GUI.
As long as you only make changes in the media repository there will be no issues and everyone immediately benefits from the work. But since I'm not godlike I surely didn't do things right in the GUIManager class just like that and you will want to make some changes there. This might encourage you to make changes in the media folder that will break the other code banches.

And I think this is not such a good thing... Unfortunately I have not yet come up with a good solution. One solution would of course always be to create separate branch for every code branch. Only creating one for some selected branches is not an option because if you merge one of these back to the trunk, the other branches get broken upon svn up ../media

The other solution is a written rule: No breaking changes allowed, only adding. That's basically what we do know, but it's not written and communicated. The problem with this strategy is that the repository checkout will inevitably grow larger and larger. To counteract that selective cleaning might be done. But who can keep track of all the files required for all the code branches still out?


Something else: Do we have a naming and structuring rule? Because if I look at the media folder, I don't think so..

[1337]

Btw: I just realised that the repository actually already is broken: presentation_dm.oxw is not working anymore because the sky box pictures have been removed recently.

[BadElvis]

I tried cleaning up the current media repo to cut down the immense file size recently. I tried moving the skybox from the textures folder into the skybox.zip archive but obviously, it doesn't work. That's why the files are back now. The sad thing is, that I encountered this problem before and actually did solve it, but I don't remember how.

And no, there isn't a consistent naming structure for our media files. I don't like that either, but I can live with it, as long as there are no files called myship.mesh or tex.tga.

I already thought about introducing an all-lower-case-underscore-structure, but the fact that I would have to search Orxonox for every filename that I adjust and change, discouraged me.

[1337]

Here we go again:

I hereby open a discussion about how to treat the code part in the data repository. The current problem is as follows:
Say somebody decides to do some more work on the GUI. You would inevitably face the problem of breaking the trunk if you were to change the lua code in the media repository. Instead you'll have to copy the file and link that one to the C++ code.
In the meantime we're already at the third new file (within a semester!). And that is by no means very accommodating.

Mo solution proposal is to move some part of the media repository to the source repository. The problem would be deciding which parts. I have found the following file types in the media repository:

To be moved:

• Cfg and ini files
  Ogre *.overlay scripts
  GUI layout, imageset, looknfeel, scheme files
  Lua code
  Orxonox *.oxt, *.oxw, *.oxi, *.oxo files

Not be be moved:

• Tcl folders
  cg, hlsl and glsl files (shader files I suppose)
  Ogre *.material, *.compositor, *.particle, *.program, *.fontdef scripts
  fonts
  pictures (jpeg, etc.)
  Meshes
  archives (fonts and ogre packs)
  audio files

That's about it, no guarantee for completeness. The question now is of course: What is bound to the code branches and what can be considered 'self-contained'? What I mean: Which part can simply be edited without breaking anything? I'll make a post about that from my point of view later on. Now it's really time to start studying

There is also the other problem of the license, just mentioning...

[beni]

Let me pickup your last point there: I started a list of the all the data files to add the author to each file.
It really is a mess as I don't know the source of about 80% of the files. This really puts us in a tight spot when we want to release the data repository under a certain license.

Now with the splitting. We really should do that with the lua-scripts and I also think that levels should also be split away.
The question would be: where do you put it in the code repos?

The problem with the lua-scripts for the GUI: They need the images for the GUI and everything with CEGUI works over relative paths. That means you have to redefine some paths. I cannot assess at the moment how much you would have to change, but I think it'll be within reasonable measures.
Oh, maybe I have to clarify: GUI layout, imageset and looknfeel are XML files defining the images and look (and layout) of the GUI, while the lua-code is used for interactivity. I think all of it should be moved.

On the things that should not be moved for sure: audio, archives, images, fonts. Those things hardy change and if they do, they usually don't brake other branches.

[1337]

I mostly agree on your points.

Now with the splitting. We really should do that with the lua-scripts and I also think that levels should also be split away.
The question would be: where do you put it in the code repos?

I would put the code directly into trunk/media (or whatever branch you are in). And I fully agree on the lua scripts and the levels.

The problem with the lua-scripts for the GUI: They need the images for the GUI and everything with CEGUI works over relative paths.

Why's that? AFAIK CEGUI uses the resource provider from Ogre so you can directly access all files by the name?

Oh, maybe I have to clarify: GUI layout, imageset and looknfeel are XML files defining the images and look (and layout) of the GUI, while the lua-code is used for interactivity. I think all of it should be moved.

Agreed mostly. Layout-changes of course come in hand in hand with lua script changes. However style changes might be more independent. On the other hand it would be very inconsistent not to move them. Therefore I agree.

On the things that should not be moved for sure: audio, archives, images, fonts. Those things hardy change and if they do, they usually don't brake other branches.

I would like to add meshes and maybe (what do you think) *.material, all shader and compositor scripts, *.particle, *.program and *.fontdef scripts. But I would move the *.overlay scripts.
And of course moving the Tcl stuff doesn't make sense ^^

[beni]

I would put the code directly into trunk/media (or whatever branch you are in). And I fully agree on the lua scripts and the levels.

I see, what do you do with the media path? there will have to be two of them right? Otherwise we developer would have to start making symbolic links everywhere.

Why's that? AFAIK CEGUI uses the resource provider from Ogre so you can directly access all files by the name?

Not inside the lua-scripts that is. You are right that it works with the layout and the imagesets etc. If we can be sure that the scripts will never have to access the other repository no changes will have to be made.

Agreed mostly. Layout-changes of course come in hand in hand with lua script changes. However style changes might be more independent. On the other hand it would be very inconsistent not to move them. Therefore I agree.

Sure there will be problems with this approach as well, as some changes will affect layout and looks of other branches which may very much lead to quite some problems. However I see a lot less problems as images won't be changing that drastically as opposed to layout or scripts.

I would like to add meshes and maybe (what do you think) *.material, all shader and compositor scripts, *.particle, *.program and *.fontdef scripts. But I would move the *.overlay scripts.
And of course moving the Tcl stuff doesn't make sense ^^

I was very much uncertain with quite a lot of those so I only mentioned the binary data types, but I generally agree. *.material maybe should be moved, due to compatibility with Ogre 1.6. *.particle very much depend on the data and not that much on the code, am I right? so maybe no move them? With shaders I don't really care, as they are quite independent (or depend on other stuff). I'm not sure with the overlay, as I have never worked with those and don't know the connection to the code. So in the end I think we agree more or less on what to move or not.
I have no idea about the Tcl stuff, but if you say so .

[1337]

The media path is indeed a problem. We could make use of the Ogre Resource Manager, but then every file has to have a unique name. But since the folders mostly just divide between files of different types, that might not even be a problem. And say you've got two folder with almost the same content (maybe for testing or who knows what for), you could simply switch between them in the resource.cfg file (of which we will have two, obviously).

If you want to resolve compatibility issues with Ogre 1.6, then you'll have move ALL Ogre scripts. Though I was actually pretty sure they kept the syntax. So I'm a bit confused that it doesn't work at all anymore. That may need further investigation.

Btw. with "I would like to add..." I was referring to your sentence which was talking about files NOT to be moved. We then probably agree even more

[beni]

We should make more use of the resource management of Ogre anyway. At the moment we haven't implemented any optimizations or resource groups or what else, but that's another issue.
Unique filenames should not be an obstacle in my opinion. Having various versions of files and having to maintain those is a lot more troublesome than just keeping sure that all filenames are unique.
Compatibility with Ogre v1.6 is of course a different subject and maybe there is even a way to create scripts which work for both versions? I just wanted to explain my reluctance to include those files as well.
I was a bit confused, but I think I understood correctly. In the long run we can still move something back, if we decide that we made a mistake.

[1337]

Indeed, moving files between the repository might not big an extremely big deal. Still, it has to be avoided as much as possible.
But it is quite relieving to know that we are allowed to make mistakes

[1337]

I'm just having a look again at the messy (yes, it really is messy) repository and I'd like to do the following:
All three old trunks (before the time with Ogre) shall be moved:
- old_data/trunk_06_july
- old_data/trunk_07_jan
- old_data/trunk_07_june
The old_data folder doesn't exist yet and I would also like to delete the branches folder (which gets emptied).

The old media files shall be left where they are (old_media) and the current media folder will stay in media.

Any objections?

[x3n]

1) Will you also archive different "trunks" in old_media? Or is this just one folder?
2) Why deleting the branches directory? As far as I remember, you once said we have to change many material files or scripts when changing to Ogre 1.6. Therefore we may need a branch (and maybe also a new entry in old_media for older branches).

[1337]

No problem, I can organise it like the old trunk/branches structure, it wouldn't hurt at all.

old_media is just a folder created by Felix to throw out some old files in the media folder. And yes, old_media and old_data aren't similar at all, I've noticed as well. But I'd be happy to hear better names, since naming is not exactly my best feat.
But since we're back at the trunk/branches structure we might simply call it "tags". Though I still wouldn't know what to do with "old_media". I feel a bit lost..