No project data imported after reopening Opentoonz - However data file seems to be intact

[Jeffrey Cumber]

Hi!

After a one-week work, I closed a project scene.

When I try to reopen it Opentoonz informs me "There were problems loading the scene T:\Programs\[...]. Some files may be missing."
Indeed, the project is not loaded and I am presented an empty canvas and a completely empty x-sheet. No information is restored whatsoever.
When closing the canvas, Opentoonz asks a question, but there are so many files listed I cannot even see the end of the box (see attachment). The box is irremovable. The only way I can close the program is via the task manager.

However, the data seem to be intact in the scene file (see attachment). It is sized 2MB.
I tried to analyze the file with a text editor, but really, I cannot see anything wrong in there. I suspect that there might be wrong relative paths in Opentoonz. The paintings are stored in a separate project file. I have not changed them, and the references to "extras" and "drawings" are alright.

What happened just before closing the file was, I had problem defining a mesh in the plastic tool since the mesh was not applied to the painting. I hit the redo and undo buttons several times which made the PC run busy for awhile. Finally, I decided that it was better to go back to the last "saved" point and start again. So I closed Opentoonz by "discarting all the changes", for I had made a full "save all" in the morning. When I restarted Opentoonz, I found that the data could not be restored as described. I had saved an ealier status in another file version which still opens quite alright. Even the one which now is causing problems worked find the day before when I opened it. I assume that somehow the problem occured while defining the mesh.

I do think, though, the data still is in the file, but Opentoonz somehow does not access to it. Since I have not saved any changes since the last time, it is quite mysterious, though.

I hope you know something about it since a whole week of work ist on stake.

Thanks a lot.
Jeffrey

[Rodney]

I downloaded your file and did a quick check to see if I could find anything.

Nothing stood out to me except that (as you've indicated) the images aren't referenced in the file.

It seems likely to me that a folder has been moved that contains the files (listinged in your 'QuestionOpenToonz.JPG" image.

These are the files but there is no indication of where those files reside.

That is a hint I think of the underlying problem.

Initial questions:

Is there any other location where the files might be placed?  (Documents folder, USB stick, network drive)

Did you recently install OpenToonz or one of it's other brands (OTX or Morevna)?

Did you start a new Project (via the Startup Popup) for this project (if you can recall) or have you done so since?

[Jeffrey Cumber]

Hey Rodney,

thank you for analyzing the file.
I do not recall having moved any sub folders in the project main folder or any files.
The drawings are still in the place where OpenToonz copied them, which are the two folders drawings and extras.
What I did, though, is I removed a few files in the folders from which I had dragged the images to the x-sheet as they were not used in the animation anymore. However, to my understanding these are not the images used by OpenToonz as the program uses the copies stored in the project folder, right? By the way, I had created a special folder from the popup window at the launching.

I am also wondering why there are not any red marks in the x-sheet. I read a lot that this was the case with broken paths. My x-sheet is completely empty as if nothing ever had been in there.

Anyway, if you say OpenToonz is missing the references, can I insert them in the file manually with a command line? Or how else can I do it?

I thought the references were in there pointing at relative paths. If I knew what to type in the file telling OpenToonz where to search for each individual file this might solve the issue.

Greetings,
Jeffrey

[Jeffrey Cumber]

Hey Rodney,

sorry, I oversaw your detailed questions:

Is there any other location where the files might be placed?  (Documents folder, USB stick, network drive)

- no nothing external from the harddrive

Did you recently install OpenToonz or one of it's other brands (OTX or Morevna)?

- no, I have not installed any other versions

Did you start a new Project (via the Startup Popup) for this project (if you can recall) or have you done so since?

- no, I have only one project and I do not think that I restarted it

Greetings
Jeffrey

[Rodney]

You asked about whether the images brought into OT were copies or the original.

It could be either. If we import then a copy is made and OT refers to that.  If Loaded then OT refers to the original file (location and all).

The paths in your project file would indicate that your images were imported.

I suppose one question to ask might be where the Project file (Gen5.tnz) is located when opening in OT.

We might assume it is in the right place but that assumption might be wrong.

It doesn't help that that location could be almost anywhere based on what I see in the file as long as it is relative to the path of the images it references.

So, if the Gen5.tnz file is located at the root (in the sandbox) then the file should open.

But if the option to append all of those directory (extra, palettes, drawings, etc.) was toggled on then the root directory might be anywhere.

BUT since we know where the image files are we should be able to deduce by that were we should place the .tnz file.

As  for reinserting them, yes by all means we can add those resources again.

There are probably easier ways to restore your project but if it takes longer to find out how to do that it might be easier to start from scratch and reimport/reload all the files.

I don't know of any command line way to do it but a fairly quick approach would be to use the OpenToonz browser.

Starting with a new Project begin to rebuild the project by browsing to the appropriate directory and loading the file.

I say 'load' because we want to use the file in its current location ('import' might create another copy and reference that).

As you have a limited list of resources you can get that accomplished in no time.

[John Dancel]

Jeffrey,

You said you had a prior version of your scene saved and that still loads OK? Does it load stuff that should be loading in the broken scene?

Please provide and I can see what might have changed between them to break it.

[Jeffrey Cumber]

Hey John,

I am glad you're asking.
Yes, Gen3.tnz is a very early version poiting at the same paths as version 5 with a lot less columns and lines. Version 5 (the broken one) uses all the pictures and files used in version 3 and more.
Version 3 is read in by OT entirely with no errors.
I am attaching the file, and I am also attaching the screenshots of what you see when the versions are opened.
All the shared files being referenced at are in exactly the same folders in either version.
So, for me, it is quite clear that there is an error in the Gen5.tnz that causes OT to "give up".

My assuption is that one reference error causes a complete error for all of the other files, although their references are alright.
I am guessing that there are operations provoked in the file with at least one image that is not alright. I am guessing it has to to with the crash of the paraglider mesh definition that I described earlier, but I really do not know. This is quite contradictarory though because I cannot see that any changes were saved then. The file Gen5.tnz was saved in the morning and not in the afternoon when the crash occured.
So we might skip this thought.

I had another problem with a sound file, column 71 I think (eagle). This file had vanished and I reimported it. This is the last save I made in the Gen5.tnz. I don't know if this information is any help.


Thank you for taking a look.
Jeffrey

[Jeffrey Cumber]

Hey Rodney,

I read your email a couple of times to better understand the procedure your suggesting.
I think you assume that I may have shifted files/ folders to another place.

I tried this:
I shifted the Gen5.tnz to the scenes-folder in "sandbox" and all the drawings and extras and everything else. Then I tried to open the project. Still broken!
I thought about a way how to load the images again, so OT might take over the intact parameters (key frames, positions). Doesn't work!
I thought of how the references to the files might be corrupt. I cannot think of any error because the relative links are alright and untouched compared to when it worked.

Unfortunately, I am unsure if I really understood all of what you suggested, but I think you're very much assuming that I have moved folders to other places, and you're too kind to say so more openly.
I am trying to to better understand what else I can do. I would even try to reimport - or reload as you say - the images/drawings and start again, but if I also had to restart with all of the parameters I frankly do not see how I can recreate what took a week of work.
( I have to confess I should have thought more about backups, but I was not sure what to backup. Now, I would save the entire program folder OpenToonz down to the project folder, hoping there aren't any files outside from that. People, think about saving your work first before you start. Being creative is not anything that can be remastered easily.)


Thanks again.
Jeffrey

[Rodney]

I don't know what happened so I"m just going through the process of exploring the various options.

I'm not ruling out a corrupted file but not pursuing that angle yet because I've yet to see anything that suggests that.

I'm following the trail of potentially broken links because that is a likely scenario and perhaps more importantly is one that would result in restoring your file to good health.  :)

You rightly surmised that I was talking about the scenes folder in sandbox.

I hadn't taken the time to remind myself of where that location was exactly.

A telling point in this is the error message received upon opening the .tnz scene file.

It relates that "some files may be missing".

Now, this is to be expected on my end because I've never had those files in the first place.

But if everything is in the right place on your computer that message should not appear.

The important take away from that however is... OpenToonz knows it is missing something!

So in examining the Gen5.tnz file in a text editor we can see where the files should be relative to the .tnz file.

The sad thing is that 2MB isn't a lot of data so there may be many things missing (i.e. your suggestion of keyframes, etc.)

Yes, backups are very important.

I keep meaning to explore OpenToonz SVN feature because versioning of files is great stuff.

Imagine being able to dial back to any point in your creativity to recover that moment.

That is the benefit of That's not a fail safe for every calamity but it surely helps.

The other aspect is that of duplicating that data at some other location so that it can be referenced if/when the primary backup is lost.

.

.

[John Dancel]

Thanks for providing the files. Will take a closer look at them later this evening when I get to my computer.

One thing I noticed in your images is the special characters in some of your file names (letter 'o' with 2 dots over it), particularly the mesh files. I'm wondering if that is somehow causing a problem. There is apparently an issue when loading files with apostrophes in file names. I wonder if this is similar.

Can you construct a simple scene with files that have that character in file name and then save. Then reload and see if that might be the issue?

[John Dancel]

Jeffrey,

Some good news. I've verified it was the special characters in your filenames.

What I did was construct a simple scene with 3 new vector columns with names like:

AAAA

BBBB

CCCC

I saved it, switched to a new scene and tried to load it.  Loaded fine.

I then added a new column with a special character in the name:

DöD  

Saved, switch to new scene and then tried to load.  Failed to load.

I was able to correct it. You can fix it by doing this:

1) Close OpenToonz if you have it open.

2) Make a backup of Gen5.tnz just in case

3) Rename any files in the scene by changing ö to something else.  (Stick to letters a-z)

4) Manually edit your Gen5.tnz and replace any ö with whatever you change it to in #1

5) Start OpenToonz load your scene.

6) Don't use special characters in filenames...at least until that issue is fixed.

Hopefully it will open.  

NOTE: The 1st time I did this it didn't work, but I think it's because I had OT open while I did this and must have tried to open using cached information is my guess.

Let me know how it went.

[Jeffrey Cumber]

Hey Rodney,

your approach is scientific and alright for me. I cannot be sure to have done things without knowing. And it helps me to understand OT better.
One should never rule out possibilities. I did it with special characters. It also came in my mind what John suggests that special characters may lead to reference errors but I ruled it out right away.

Greetings
Jeffrey

[Jeffrey Cumber]

Hey John,

that is good news. I will check it out later. We're working in different time zones, so it will take a while until I can proceed.
If it really is about special characters it is gonna be fixed in almost no time.
We will see. I'll let you know.

Thanks.
Jeffrey

[Rodney]

Awesome news there!

I know there is an open report for the special characters issue.

Perhaps the developers can program a hammer to emerge out the screen and hit the user on the head when special characters are attempted?  ;)

I do prefer that a message be displayed to the user that an unacceptable character is being used rather than to allow special characters.

I know that may sound inappropriaate but special character are troublesome in most software so if it's fixed in OpenToonz it may still be a problem somewhere else.

If a special character become popular across the board then it could be added but even then I think OpenToonz should issue a warning such as "Your filename contains special characters which may cause problems.  Proceed anyway?"

Glad you got it figured out.

Thanks John for your insight and sleuthing. 

As a related question that  may help others... is there a specific language that is being used here that promotes the use of a special character?

[Jeffrey Cumber]

I really thought John had it, but I have spent a couple of hours testing things out with no final success.


0. Version 3 and 4 are loaded correctly.

1. I replaced ä by a; ö by o; ü by u in the "Gen5"-scence file, which are all the special characters I could find. I do not think that "-" and "_" are not accepted since OT also generates them. --> Rodney, the files in which the letters apear are named in German, but I guess this would occur in almost any other language than English, especially in Asian languages, Russian or Greak. I restarted OT and I had the same error message that files were missing, only this time no files were enlisted (refer to "QuestionOpenToonz.JPG" in my first posting)

I replaced the files again by the backuped version and restarted the Gen5-file. Again the error message came up with the enlisted files.
I repeated all the steps and the error message came back with no file listing. This might indicate a tiny progression in this quest.

2. In some files there were double endings, such as .wav.mp3. I replaced them in the scene file and in the file names as well by leaving only one ending. I doubt that more than one ending is good.
No result when relaunching OT.

3. I replaced all spaces in file names either by deleting them or inserting hivens both in the file names (in extras and drawings) and in the scene file. I remembered that DOS does not like blanks.
No result.

4. I checked all the files separately in the order of file endings (.png, wav., .tif), etc. to see whether there are references to files that do not exist.
I found none. All the files are placed in the folders. Well, to be exact, I only checked whether the files exist. It cannot be excluded there is one file in the folder extras though it is supposed to be in drawings. I may check this later, but I doubt it is highly probable.
I noticed there are files like A..tif, B..tif that do not exist. I mean, A.tif exists but not A..tif. But this is very systematic, maybe it's to indicate there are several files starting with the same letters, such as A.0001.tif, A.0002.tif.
The double dots also appear even though there is only one picture, such as A.0001.tif an no others. I do not think this is of any importance, though.

5. I emptied the cache folder and restarted OT
No result.




This is pretty much what I did. I repeated a few steps to reduce mistakes. The scene file is still not working. In the first walkthrough I also replaced the names that are filled in the x-sheet (they appear in the scene file with no file endings), which are merely informative, in the second walkthrough I only replaced the characters in the file name.

I have a question: Do you know if OT stops importing information as soon as one single file is missing? Again, the x-sheet remains completely empty, when trying to load the scene. Maybe I should try because as I know there are correct references in the scene file and only the corrupt ones are listed or excluded from the import I might be in serious trouble.

I have not given up yet, but for now, I do not see what else I can try to do. Maybe, I should check again, whether I was being accurate enough to replace everything correctly, but the scene should have opened after replacing the letters "ä, ö, ü" already, correct?

Greetings.
Jeffrey

[John Dancel]

Odd since I was able to duplicate and fix the issue. I also find it interesting that there was no file lists after the fix. Seems like 1 issue was fixed but may be another that got introduced because of the special characters.

Normally when OT cannot find the file, the xsheet still loads. The missing column is there but in the cell area, the text is red indicating it is missing. also thumbnail is missing.

Can you try creating a small sample scene as I had done to create the error? then fix it as suggested?

The double file extensions shouldn't be an issue and the ones like A..tif are normal and shouldn't be changed.

I will continue to experiment to see if we can fix it. This time I will play with some meshes.

[Rodney]

This is probably (and maybe very likely) unrelated but worth checking...

I subscribe to a service (CodeTriage**) that pops up a short list of open issues for OpenToonz and one of those issues is:relates to the FFMpeg path where a file/scene wouldn't open correctly due to incorrect FFMpeg path set in Preferences.

https://github.com/opentoonz/opentoonz/issues/1153

It is worth ruling that one out.

**CodeTriage does not yet support C++ but it still identifies and shares that list of open issues so they can be given attention and closed if possible.

[Jeffrey Cumber]

I reloaded the backup with the wrong file references and all images to make sure I did not insert any other errors when I replaced about 30 file names and their respective references.
I replaced the letters ä, ü and ö again in the references of the scene file and the file names.

When opening the scene, as expectet, the error message appeared with no file list.
By doing so,  I reestablished the old status of the scene file correction to number 1 (first step).

Then, I removed the folder with the ffmpeg files and restarted OT. I could not see any difference.

I reinserted the ffmpeg folder. I will reread the thread Rodney has sent in more detail later. I tend not to think this causes the error as you will see later in my test scenarios.


Test Scenarios

1. I opened a new x-sheet and imported a file with a special character (ö) named "Eichhörnchen.jpg".
I saved a new scene "Test". The error message appeared related to missing files.
I renamed the file into "Eichhornchen.jpg" to make sure there are no special characters in the file name as well as in the reference in "Test".
I reloaded the "Test" scene. The error was not fixed.
I retried several times with no different outcomes. I also deleted the contents of the cache folder.

2. I opened another empty x-sheet and imported the same image, now named "Eichhornchen.jpg" without special characters.
I saved this as "Test2" in the same project folder.
I closed OT and reloaded the scene. The project opened correctly.

3. John, I understood that it finally worked out for you to reload the corrected scene in your "AAA, DöD [scenario]", but it did not at your first attempt. Had you changed anything?



The interesting result from my scenarios is that it did not (yet) solve the error to replace "ö" by "o" since the scene was not opened correctly after I had replaced the special characters.

The theory that derives from that - unless I have done anything wrong or some other parameters are involved - the special character in the image name either causes that OT writes something in the .tnz-scene file or leaves something out when the information is saved.

This would mean that the error is not only based on the file reference but also on what is written - or not - to the scene file as a consequence of the reference error.

I will make another test, importing an image named with a special character, performing operations in OT and saving it.
Then I will import the same image named without special characters, repeating the exact operations in OT and save it. I am curious to find if there are differences in the scene files.

Greetings
Jeffrey

[Jeffrey Cumber]

I compared my two test files with each other. Aside from the names in the file references and the time stamps they are completely identical.
However, I noticed that there is hidden information. When I open the LoadScene window, not the entire information is displayed. For example, the size of the camera and the number of frames are missing. Is there any hidden information in the .tnz-file that cannot be opened with a text editor or excel? If it is not in the .tnz-file it must be stored somewhere else or it somehow becomes reactivated. This makes the two files different from each other. (See attachment.)

For the Gen5-file this information is also missing.


I made a final test. I corrected the information from file1 and copied it into file2, which was intact and I opened scene file2. Now it worked.

I then transfered the corrected data from the Gen5-file into an intact file with different file info since there was only one frame, etc. Now the file information is restored, I could see the computer running, but the "some files are missing" error occurred again.


Greetings,
Jeffrey

[John Dancel]

I'm curious..did you just change the specific letter/character when you made changes in the .tnz file?

Try deleting the entire filename and retyping it instead of just the letter. I wonder if there is still some unprintable character you cannot see if you just replaced the letter. By not deleting it, it is still throwing off the way the file is being read into OT. That could explain why some information is missing in OT load window.

[John Dancel]

Jeffrey,

I think I found the 2nd problem.

When you saved the file with the umlaut characters, it changed the entire file from UTF-8 encoding to UTF-8-BOM encoding.  Apparently OT doesn't like UTF-8-BOM.

Changing the umlaut characters to non-umlaut characters was not enough.  Also had to re-encode the file to go back to UTF-8.

So I took your original Gen5.tnz file and replaced the umlaut characters.  It still failed to open the file. Then I re-encoded the file.  I was able to successfully open it in OT, albeit the columns were showing as missing since I didn't have the real files. However, no more error.

I've attached the corrected file.  Please let me know if you can open this one.

[Jeffrey Cumber]

Hey John,

that is incredible. You have identified the cause and fixed it.
I loaded the corrected scene file in OT and it opened perfectly. Impeccable! That is awesome!
Thank you very much for your help. You've made my day (and saved my entire last week).


Greetings
Jeffrey

[Jeffrey Cumber]

Hey Rodney,

thank you for your help and the insights into OT. I am glad, now it is solved.
I would also prefer a "hammer" message by OT. A test routine of allowed characters should be easy to do, and I really wish I'd rather have been hit by a hammer message than a sledge hammer, when I found out that everything had gone. It is only thanks to the discussions with you and John that I happened to recover everything in the end. It's worth mentioning that even backing up - in this case - would not have brought me any further because I would only have found out that the file is corrupted by reloading the scene. That is a hint for the programmers - who are doing a great job. I personally think that OT is a great tool.

I guess, many people dropped OT because they were having similar problems and simply gave up because they found their work not being saved.


Greetings,
Jeffrey

[John Dancel]

Yah! Glad I could help.

What OS and version are you running? I will put all this info together into an issue in GitHub so hopefully someone can fix this.

[Rodney]

Jeffrey,

I've moved this topic to the "Troubleshooting (Solved)" category because it appears we are good to go and others will surely benefit from this discussion.  :)

Feel free to set it back to "Troubleshooting" if that isn't the case.

[Jeffrey Cumber]

Alright, John,

my computer is running on Windows 7 Home Premium, Service Pack 1, 64 bits.


Greetings,
Jeffrey

[Rodney]

I'm moving this back to an open Troubleshooting because your exploration appears to be heading for a larger fix and perhaps even a bug report.