Ghost Face Swap

0 views

Skip to first unread message

Rosita Westhouse

unread,

Aug 4, 2024, 6:42:57 PM8/4/24

to defchilicomp

Ithink the problem (which I've been hit or miss on solving) is that sometimes an alignment file gets re-generated in another folder. Then you have two alignment files and the instructions are a bit crap at telling us which to target in the data sections so we get weird errors. I ran into several times where I did the extract then sorted, then removed faces manually, ran the remove faces program, and did an alignment extraction and would get "0 faces have been extracted". The system needs to have a way to keep alignment files updated through out the process a bit better. Also it should have an option on where to archive the files. The archives seem to send the trainer for a loop when there is two alignment files or more in the folder with the images. Took me awhile to figure out I'm better off axing the alignment files that it archives and just keeping the single alignment file.

However, my new problem is, I've made a working model, I can even preview the outcome. However when I run the convert I get a good several thousand messages saying the alignment for an image wasn't found. And before anyone assumes I didn't clean my alignment file, I've done it maybe three times on the same set of images these last few days. So I'm left to assume since I have group of images from a folder (we will call this data set A), and a group of images from a folder originally extracted from a video for a model/trainer (We will call this data set B) and then the original video (Data set C) and the original video's extracted files (Data set D) I'm a bit confused much of the time on which data set's alignment file or folder the whole thing wants me to point to. And even going back and looking at the instructions it gets confusing because it will mention one of the above data sets as "Where you extracted after blah" but not which type of extraction, not which steps were done before, and nowhere does it mention if the file placement is actually a requirement or not. There is one step in extraction with a "IMPORTANT***" tag that mentions moving a data set's alignment file to within the file. But when I do that it seems to give errors where it can't find anything. I assume because you shouldn't actually be moving it after some update or something, but I don't really know how to check the alignment file to see if there is some dead link somewhere that the alignment file points to a folder that no longer has the images because they were moved out during a sort or some other step.

I've gone back through the same data set each about three times and the best I can figure out is, after you sort and clean the file, assume the alignments file is useless and just start at the extraction step again (the very first extraction not the alignment extract) then go directly to the alignment extraction, do not pass sort, do not clean the file, because there isn't anything left to sort or clean. Then use that to do your model/training/convert.

However, the work flow I seem to get dragged into through something going wrong is:

Extract> Sort> Clean >Remove faces>Stop here and start over with extract, skip sort and clean after> Alignment Ext. > Train> (Repeat all the prior over with a second set of data) then model

And after that for the video I plan to swap onto:

Extract > sort> clean> Remove faces> Stop here and go back to extract then skip sort and clean> alignment extraction > use to convert

But even after this workflow I get results where seemingly the first/last image the system looks at for the swap gets plastered on ever scene of the end video over top of whatever might have happened behind it. Or, I end up with a video that has had no swapping done at all. (And no, I checked, I did select the correct direction for my model to make the swap, and the correct direction for the convert. I didn't switch A & B around.)

Hopefully my struggle busing along will help you get a little further. I'm still lost on the last few steps though, but I"m thinking that is because I wasn't using the manual tools to fix masking and such but idk, we'll see...

However, when an alignments file is generated, it is always generated in the same place (next to the source video/folder of images). This never changes. The problem comes from the fact that an alignments file serves 2 purposes. 1) it corresponds to a video to say where the faces are in a frame. 2) It can correspond to a folder of faces, which could be from multiple sources, to tell Faceswap how to train on each individual face. Unfortunately both these tasks need to be performed, but they do not play particularly nicely with each other.

This is almost always because an alignments file hasn't been generated properly for the video you want to swap on to (quite often it's because it has been generated with an Extract-Every-N setting higher than 1). First port of call will be to regenerate your alignments file for the video you want to swap on to.

Extracted faces can be entirely ignored for convert. They are not required. You just need a Video (where you want to swap faces). The alignments file that corresponds to this video, and the model that swaps the face in the video to the face you want.

I did some messing around and the biggest problem I encountered (and partially overcome) is that when I get files extracted and have to clean the files in the clean or sort step, is to not make a new folder for the sorted faces to go (this was my first mistake). That seems to keep the sort and cleaning steps from causing errors abound.

The new issue I've found is that using the manual tool and finding a face which isn't helpful and just right-clicking on it's bounding box and hitting "delete" causes more harm than good. Because after finding around 20+ of these mostly useless faces in the manual tool, the file becomes unable to be trained. It will hit an error rather than skip X number of faces (my assumption is that it has some maximum number of faces it can ignore, I know this because it will skip at least 1 in my current set and not err, and that when I hit "delete" in the manual tool it isn't deleting the image from the folder, or isn't deleting a space for information in the alignments file, idk which) however, by random guessing at which tool i can use to clean the alignment's file I've fixed this at least once successfully... I basically ran it through just about everything (missing faces, missing frames, missing alignments, leftover faces, whatever seemed like it may fix it, though I was doing this rather blindly since there isn't a good explanation I've seen on what those tools actually do really. So I just went with blind faith in running everything and hoped it's be okay.)

Also I noticed a big place where what the guide says is actually more confusing than what the tooltip in the program gives for an explanation. (this was a separate problem) I think mostly it revolves around the terms: "Faces" "Frames" and "input" when using each tool (my use of the word tool here is meant to include: the initial extraction, the alignment extraction, each sort and clean step, train, and convert etc.) It seems confusing how in one step a given set of data seems to be called the "faces folder" and in another it will be called the "frames folder" and in another we even see things like "training images" when best I can tell, I want to have every image in my training set unless I have a set of images beyond say 10,000. So I just select extract every N = 1.

Example:

Data set A = Folder 1 or "original video 1"

Data set b = Folder 2 or "original video 2"

then other files that may be created:

"Sorted folder 1" or "sorted folder 2"

"Alignment file 1" (being from data set A) or ""Alignment file 2" (for B data set) or "Alignment file 3" (For the video to swap onto)

since for the case of swapping onto video we would always have a 3rd data set (we could call it data set C) which goes through a different process than the others we could avoid calling it anything other than the "Final swap video" or the "Final swap video's alignment file"

To me calling it a "face folder" or a "frames folder" gets needlessly confusing, and within the guide from step to step what was once faces in one step becomes frames for another so it gets very "wait whut?" to the newcomers.

Also, if there were a tool for looking not at the outcome of the alignments file (like the manual tool) but instead for seeing the actual important code within and being able to edit it. I may be able to decipher what exactly is going wrong when I use the delete function on a bounding box in the manual tool, or when I add a new bounding box on an image that didn't have one. (As that too was a problem)

I do have a guess at the solution though for how to take a lot of work out of the user's end when using the manual tool and fix a few potential issues it causes. When we hit the save key the manual tool should do a final check on all the images within the alignment file and be sure all the images are accounted for, if one isn't then it should remove it from the file. Also it should go through and add in any new alignments given to images which had none before. Then once it's gone through and fixed those issues, save all the data we made changes on that already existed.

For me if the images and the alignments have de-synced on a file (not a video) to where I have several hundred images with correct facial alignments in terms of the form of the face, however they are far off from where the actual face in the image lies, I can go through and correct them with the manual tool as needed, (however idk why they happen, I've had about 1100 out of a data set of 2300+ faces) but I have noticed while correcting them I tend to stay on the bounding box screen while flipping through the frames with z & x keys, that if I try to pull a bounding box from out of the overall workspace into the workspace, the computer crashes. I assume what has happened here is since it is outside the workspace and cannot see past the edges of the workspace the bounding box doesn't know where to try to lock on to a landmark so it just crashes. I've found a work around however. If I swap to the view where I see the yellow box instead of the blue bounding box (i forget the name of this screen and I'm in the middle of a training set now so I can't check,) and then I pull the box within the bounds of the screen, (while still seeing the yellow box, I don't just flip in and out of the screens) then it won't crash. I can then flip back to the bounding box screen and the bounding box will be within the bounds of the workspace so it doesn't crash as it had.