Re: Replace Objects In Photo Intelligently Using AI: GANpaint Free
Hilke Mcnally
This article talks about a free tool that can replace objects in photo intelligently using AI. If you are into machine learning and AI, you might know about GAN (Generative Adversarial Networks). It is a neural network designed to perform artistic tasks like drawing a portrait or composing a symphony. Unlike many other deep learning neural networks, GAN generates data from scratch, generally image and music. This has a wide range of potential applications in various sectors. One such application is GANpaint Studio.
GANpaint Studio is a paint tool that uses GAN to add, remove, and replace objects in photos. This is a mutual project by MIT and IBM that takes a natural image of a specific category and allow modification using a brush tool that is not possible by simple strokes. Currently, this tool works with specific types of images like churches, dome buildings, kitchen, etc, and draw semantically meaningful units such as clouds, domes, doors, brick-texture, trees, and grass.
GANpaint Studio lets you replace the objects in a photo with a simple brush tool. This tool has a brush and eraser tool which you can use to add and remover objects to a photo. There is an online demo version available of this tool which you can check out here. This demo gives you a set of 6 specific types of photos which you can manipulate.
Simply choose an object and use the mouse cursor to highlight the area on the photo where you want to add that object. If you remove any of the above-mentioned objects from the photo, you can do that with the eraser tool. You can also paint over an existing object to replace it with the selected one. Here is an example of that. As you can see in the attachment below, I pained this church with a brick brush which replaced the texture of the building. I also added clouds and trees.
Apart from the given set of photos, you can also upload your own church-like image to test the tool. It takes about a minute to process the image and then you can manipulate it with semantic objects and download the output from the right-click context menu.
Creating digital renderings and editing images can take hours, but researchers from MIT and IBM want to change that. They've trained AI to generate photographic images from scratch and to intelligently edit objects inside them. While this could be beneficial for artists and designers, it also offers insight into how neural networks learn context, and the team hopes to leverage the tool to spot fake or altered images.
The product team added buildings and areas of interest even for smallest cities using satellite and street views photos. Their 3D models are so detailed that you can sometimes see the blades inside the rooftop fans. Looks like Apple does it manually.
An algorithm by Tom White draws abstract illustrations of real world objects. It's trained on photos and the result is close to usable in real products. It's a part of Google Artists and Machine Learning initiative (read its blog).
Here is runningstats.py, a useful little module for computing efficient online GPU statistics in Pytorch.Pytorch is great for working with small batches of data: if you want to do some calculations over 100 small images, all the features fit into a single GPU and the pytorch functions are perfect.But what if your data doesn't fit in the GPU all at once? What if they don't even fit into CPU RAM? For example, how would you calculate the median values of a set of a few thousand language features over all of Wikipedia tokens? If the data is small, it's easy: just sort them all and take the middle. But if they don't fit - what to do?import datasets, runningstatsds = datasets.load_dataset('wikipedia', '20200501.en')['train']q = runningstats.Quantile()for batch in tally(q, ds, batch_size=100, cache='quantile.npz'): feats = compute_features_from_batch(batch) q.add(feats) # dim 0 is batch dim; dim 1 is feature dim.print('median for each feature', q.quantile(0.5))Here, online algorithms come to the rescue. These are economical algorithms that summarize an endless stream of data using only a small amount of memory. Online algorithms are particularly handy for digesting big data on a GPU where memory is precious. runningstats.py includes running Stat objects for Mean, Variance, Covariance, TopK, Quantile, Bincount, IoU, SecondMoment, CrossCovariance, CrossIoU, as well as an object to accumulate CombinedStats....
This raises a question of whether the StyleGAN architecture is necessary and whether many GANs might work, if only one had good style transfer for anime images and could, to defeat the texture bias, generate many versions of each anime image which kept the shape while changing the color palette? (Current style transfer methods like the AdaIN PyTorch implementation used by Geirhos et al 2018, do not work well on anime images, ironically enough, because they are trained on photographic images, typically using the old VGG model.)
AI art refers to artwork that has been created or enhanced using artificial intelligence algorithms. These algorithms analyze and manipulate data to generate or alter images, often in ways that mimic human artistic styles or techniques. AI art can range from simple photo editing tasks to complex generative art creations.
One of the common tasks in AI art is removing backgrounds from images. While there are various techniques to achieve this, an efficient method is using the Content Aware Fill feature in Photoshop. With the Selection tools like the Marquee or Lasso tool, select the area you wish to remove, then apply the Content Aware Fill. This feature analyzes the surrounding image pixels and intelligently fills in the selected area, often yielding impressive results.
aa06259810