Advice on file structure based on virtual folders

[Fredrik Averpil]

Hi all,

We are revising our file structure on disk, and I figured I'd just ask for some advice/ideas here for anyone interested.

Currently, we use the server/project/sequence/shot/ approach with assets being stored per sequence, but as we tend to rarely utilize "sequences" or "shots" as we do a lot of archviz and shorter commercials, this file structure is more often cumbersome than helpful. We are also a generalist group of people, meaning one or two people may run a project and complete it. Sometimes a project needs to get divided up on a much larger group of people, but this is pretty rare.

Therefore I'm contemplating whether to flatten the file structure into just /server/project/ or to keep the current one. My goal is to provide the power of structuring up a project but also be able to work in a much more flattened manner for smaller, one or two man projects. This flattened approach would mean that you would find a default Maya project directly underneath server/projectX/ and in here all scene files for anything is stored.

The reasoning behind flattening the file structure would mean I would accompany the file structure on disk with a project management tool which is available inside of Maya and Nuke. This tool would read a database or JSON file, which would dictate virtual folders. So a user would still be able to create sequences and shots, but no folders would get created on disk. Instead, only one Maya project directory exists within a project and each scene file gets tagged with which virtual folder it belongs to (along with other attributes such as whether the scene is in fact an asset or e.g. a render scene). In the project management tool, you navigate through this virtual folder structure and find your scene files. Whenever you want to save a new scene, a dialog pops up asking you where you wish to save it in this virtual structure and which type of file it is.

The project management tool allows you to select files and reference/import and do other things such as export proxies and such. When generating new data on disk, for the most used features, a custom dialog would pop up to ask the artist whether he or she would generate e.g. particle cache "globally" on tied to the scene. Depending on user choice, the data will get stored in subfolders or in the default location.

So everything seems to happen in a folder structure but this really does not exist on disk. There would be a few scenarios where users must deal with the real file structure on disk though, and for this reason there will always be a right click menu in the project management tool's navigation pane, where you can "Browse folder on disk" to immediately get to the physical location on disk without having to figure your way there.

There are of course scenarios where this approach (virtual folders) doesn't work at all, for example in any software where the project management tool won't run (perhaps because the lack of PySide support). But here in our studio, 90% of what we do, we do it in either Maya or Nuke, which both would support this. And since the project management tool is designed to run as standalone, artists can use it to get to application folders without having to navigate manually.

Now, I just need to make sure I'm not doing something stupid, such as making it impossible for us to run certain software in the future which would require us to completely overhaul this system. And I was thinking I'd just post this over here to see if anyone 1) has read this enormous email at all 2) has any ideas or comments to this type of approach, or even experience in something similar which they would like to share. Any comments are greatly appreciated.

Fredrik

[Fredrik Averpil]

I might add I have a working prototype project management tool which runs in Maya, Nuke and standalone. It supports all of the above and Maya and Nuke scene versioning as well as assets. It reads all user-defined names for files and folders from a mySQL database currently.

So it works. It's just whether the whole approach is flawed in a major way that I haven't thought of.

Fredrik

[Justin Israel]

Hey Fredrik,

Is your logic from the Project Manager separated out into a library that can be reused easily by other tools? If so, then I think that is a good path, since if you are going to move towards an almost pure filesystem abstraction, it will be pretty good to have a way to support tooling needs outside of the main UI that you all use. 

I've seen and heard different levels of filesystem abstractions, all the way up to where everything is flat with UUIDs and you 100% rely on your abstraction to make any sense of it. But the more abstract it is, obviously the better and more versatile the tool support should be. So I don't think it is all that crazy of an idea, provided you prepare for those circumstances. And it also does depend on how big your company is and how different your project structures are, how much you expect them to conform, and how much tooling you have around concepts that expect consistency. 

But I assume the amount of flattening that someone would go with might also be a balance of still wanting to be able to have enough flexibility to have different mount points to control which data lands on which storage backends, and being able to transparently migrate data between available storage. That is probably the reason I have seen at least the project/scene/shot granularity, but your mileage may vary. Also there is probably the consideration of how many files will end up in a given directory, which could make listings slower. 

All in all, the file system abstraction can totally make it easier to do renaming of items without breaking anyone, as long as everyone is going through the abstraction layer (back to having good tools). So ya I support your ideas here. But it might be good for me to play devils advocate and not think in terms of "we really only use Nuke or Maya" but rather "what if we didn't use Nuke or Maya one day?". You said you have that partially covered by having the standalone manager. It just means if someone were in, say, Photoshop, they would need to have a workflow that supports opening and saving files without too much fuss, or without you scrambling to make another application integration. That can be the reason for at least having a minimal amount of hierarchy. 

Anyways, sounds like a cool goal!

 

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWPVFHJ-B1gGjv9-t2x0pc7OidZnESJKiMiTwn%2BZip%3Dimw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

Cool, yes, practical, maybe.

By the looks of it, it sounds like your setting yourself up with required maintenance, as opposed to optional maintenance. This could potentially be dangerous; the difference between the two is that, without you or someone else to maintain such a system, no one will be able to perform any work. Worse yet, if this system fails or lacks in any way, everyone will be affected until it is either fixed or replaced.

Difficult to say without specifics, both in terms of usage and implementation. Context the is most important I think. If you were at Weta, complexity at this level (i.e. the file-organisational level) would be more likely to fly as there would be more personnel to do maintenance. But if you're at a 5-10 man shop, I'd suggest that you evaluate where you see gain, and where you see issues and try and weight them against each other.

[Fredrik Averpil]

Thanks for your feedback, guys.

It's good to hear the mostly positive feedback towards this virtual folder structure idea.

The logic behind everything is indeed made into a library, or rather python modules. Nuke and Maya imports the shared module (which is heavily focused on navigating the virtual folders) and then they import their app-specific module (tools and scripts to prevent the need to dive onto the disk structure). Same goes for the "standalone mode" which runs directly in your OS. An admin module allows for automated archiving or reviving of archived projects.

Regarding optional maintenance vs required maintenance, we are already in the required maintenance mode. It's just that this is about developing the company to support certain projects along side archviz/commercials in the very same folder structure. I would like to avoid building two structures demanding two versions of the same tool.

So my main concern is really if we were to e.g. move away from Maya in favor for Modo ... or maybe wanting to utilize Nuke Studio ... or Katana ... what will happen then?

These apps I am guessing are heavily reliant on folder structure on disk. Perhaps it is possible to customize their behavior in regards to creating folders and files on disk into my "flat" disk structure and so this won't be an issue. But I think this is what scares me the most. But right now it looks like we're going to gain a lot from this. So I think it's worth a go.

Fredrik

[Marcus Ottosson]

If what you're looking for is customisability of folder structure on a per-project basis, then have you considered working with schemas? I.e. pre-defined levels of each hierarchy onto which your code relies.

There are hints towards how this could work here.

https://github.com/shotgunsoftware/tk-config-default

Although I would favour a slightly more data-driven approach, schemas are resistant to maintenance failures and are very simple to customise; usually in the form of a YAML file or template hierarchy. Ultimately however, anything pre-defined is bound to hit a level of scale where maintenance outweigh any savings made.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWOrcZW24osm2adfsjS%3D4aBnqWG0QSKQpcDBqRmusDQ1Rw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

If what you're looking for is customisability of folder structure on a per-project basis, then have you considered working with schemas?

Yes, that's what I'm planning on doing - but I have not considered working with schemas. I'll take a look, thanks!

I was planning on having a .json file in each project's root directory, which would define the structure within the project. This would simply get read/written by the production management tool (or rather, it's library/module) in a similar manner to how you would work with Qt's QTreeWidget.

Today I have a mySQL database, but I have come to realize a .json file would be much better for several reasons; limiting issues to individual projects, allowing for quick and easy manual editing if an issue arise.

Regards,

Fredrik

[Justin Israel]

On Fri, May 9, 2014 at 10:41 PM, Fredrik Averpil <fredrik...@gmail.com> wrote:

If what you're looking for is customisability of folder structure on a per-project basis, then have you considered working with schemas?

Yes, that's what I'm planning on doing - but I have not considered working with schemas. I'll take a look, thanks!

I was planning on having a .json file in each project's root directory, which would define the structure within the project. This would simply get read/written by the production management tool (or rather, it's library/module) in a similar manner to how you would work with Qt's QTreeWidget.

Sounds like a schema to me! It concretely describes the projects layout and what it must conform to. 

 

Today I have a mySQL database, but I have come to realize a .json file would be much better for several reasons; limiting issues to individual projects, allowing for quick and easy manual editing if an issue arise.

Regards,

Fredrik

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWPQB7dohoUYAJ7upy94OTWesb%3DY_2sjLati9tXsNBzCVg%40mail.gmail.com.

[Fredrik Averpil]

Sounds like a schema to me! It concretely describes the projects layout and what it must conform to. 

Hehe, ok I just never heard that term before :)

// Fredrik

[Erkan Özgür Yılmaz]

What I think about having a disparity with the data structure and the folder structure is that you can do it in to some level, but then you shouldn't do it as much as to make it completely irrelevant from one to the other.

Your starting point of having a flat structure is because of your old implementations heavily non-flexible design, and your direction of adding an abstraction layer is not correct. You are trying to solve a problem by creating a bigger one, trust me.

Let me explain what I think is a better approach.

3 years ago I was doing it with XML files. Now switched to a SQL database and written a library called Stalker which is basically a Production Asset Management Library written completely in Python and it is OpenSource with LGPLv2 license.

I need to explain it over Stalker cause it is what I came along with while thinking about the same problem.

At this point I suggest reading the tutorial page of Stalker which explains all the idea behind Stalker in a smooth way, so please visit: http://pythonhosted.org//stalker/tutorial.html#part-vi-asset-management

So, in Stalker the base data is "Task" (actually everything starts from SimpleEntity). And all the Assets, Shots or Sequences are derived data types. Tasks can be hierarchically grouped with a parent/child relation.

You are not forced to have Shots stored under Sequences nor Assets under anything. You can freely create your own structure. And also the folder structure is not forced to be forced to reflect the parent/child relation of the Tasks/Assets/Sequences/Shots. You can freely define them by using per project and per data type Jinja2 templates.

In Stalker you can have an Asset under a Shot or a Shot under an Asset or let say a Shot under a Sequence or you can have a Task which is not an Asset or Shot but something that needs to be done and it is related with one or more files on disk.

Every project has its own structure or you can share ``Structure`` of one project with another. Also every project can be stored in a different placement which is called ``Repository`` in Stalker.

Back to your question, I don't think it is a good idea to use a completely irrelevant folder<->data structure, just to avoid having a constant/inflexible structure on all of the projects where things get unnecessary, the solution I think is just to make it flexible enough so you can have different project structures per project.

I hope my answer was helpful.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWN_Ty06fzm_jqpMaRdmA2CkVL06%3Due9QmNkHTdVDTrTNw%40mail.gmail.com.

[Chad Dombrova]

If you're planning on doing all of the work required to implement a virtual-folder system, you might consider implementing it using fuse.  Fuse let's you create a new file system that is run as application run by the user.  For example, people have used fuse to create file systems that mount over FTP and SSH and even git. To the user the new file system looks like any other, so files and folders are accessible in any application, including finder/explorer. 

There are python bindings for fuse here: https://github.com/terencehonles/fusepy. 

The API will probably have a steep learning curve, but on the other hand, you won't have to write any UI code.

chad.

[Marcus Ottosson]

Even though I'd consider heading down that route it an incredible fascinating experience and one I'd follow closely (if you for example where to blog or otherwise take notes on your journey) I'd stress the potential failure- and maintenance-rate you would be up against compared to the very slim winnings you would probably end up with.

Honestly, who does that and gets away with it? :) What it sounds like you need is flexibility in directory structure on a per-project level, so that you can keep your code as-is inbetween your variations.

I'm all for thinking differently, challenging the status quo and all that, but don't hurt yourself doing it. :)

Chad, can you see somehing like fuse work for this purpose? I mean, even where there are plenty of resources in pipeline and IT?

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7G8H%2BbohSMwKDTX84wdMhQw5NsEEtg7tTbrqJMK%3DLTMYg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

I did a fuse filesystem when I was at my previous studio. It wasn't the primary system. Just a mount that offered specific functionality.
When you navigated it,  it was aware of your user name and could show you a directory of just your assigned shots and then department specific areas under that shot. Then it also included general role ups of locations at the top level. You could go into a location a see just the published data because it referenced the production tracking and asset system.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCZx%3DUok5sw09OBCpYuOFqmh%3Dyjbu5OLb4bSGXUHjDJGw%40mail.gmail.com.

[Marcus Ottosson]

Yeah, I'm not arguing it's usefulness, just that it might not be useful for Fredrik's purpose. User-dependent views onto data sounds cool, and highly optional, which is where I think complexity such as this belongs.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0j9AsgP3sgi-aaqc3jfzn_Mnpa37cw43D6_UCeF%3D-PEQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

I'm not sure if fuse is the best option for me, but many thanks for suggesting it. Haven't thought about it, I've only used it very briefly in the past.

I'm going to look into Stalker this week. It seems it offers everything I'm looking for (and more), and it allows me to not limiting myself to a single project structure on disk. On paper I think I may end up forking Stalker or borrow some concepts. Erkan's done an amazing job there.

UI wise it's going to have to be custom PySide code which can run on top of the libraries.

But for any apps which aren't python enabled, fuse is an interesting idea - especially if it could generate a file system based off the Stalker project structures.

Erkan, what do you think?

// Fredrik