[On Painting] On Painting and Algorithms

[Duane]

With an apple I will astonish Paris

Paul Cezanne

If you placed an apple on a table and wanted to paint it in the manner of Cezanne, you would need to have an extensive knowledge of his work and a strong sense of his aesthetic intentions. Every brushstroke would need to be accompanied by the question, “what would Cezanne do?” You might stop frequently to refer to his paintings to see how he handled certain visual situations. As your painting progressed you would gradually develop perhaps a dozen general stylistic guidelines for yourself. These guidelines would be instructions along the lines of “when you see this, do this.” Of course, much of the process would be based on wordless intuition; a vague sense of when a group of marks looked “Cezanne-esque.”

On a basic level, this is not unlike a how a computer algorithm works. An algorithm is "a finite sequence of instructions, logic, an explicit, step-by-step procedure for solving a problem, often used for calculation and data processing and many other fields." It acts as a kind of flow chart which guides a computer through a series of evaluations and decisions. When translating, say, a Shakespeare sonnet from one language to another, a computer will use an algorithm to evaluate and substitute words and phrases into the other language. This is called “gisting” because computers are still not capable of making a translation that is much more than 80% accurate—for all their processing power, computers have a difficult time processing the complexities and nuances of contextual meaning. In art of literary translation there are no clear-cut right or wrong rules for choosing a phrase that means the same thing in one language as it does in another and that keeps the same rhythmic or emotional characteristics. From Wikepedia:

Fidelity (or faithfulness) and transparency are two qualities that, for millennia, have been regarded as ideals to be striven for in translation, particularly literary translation. These two ideals are often at odds. Thus a 17th-century French critic coined the phrase les belles infidèles to suggest that translations, like women, could be either faithful or beautiful, but not both at the same time.

Here is a Shakespeare sonnet translated from German into English by a computer program:

I am to compare one summer day you, which you

lovelier and moderate are? Mays expensive buds

drehn in the impact of the storm, and is all too short

summer period.

You get a general idea of the what the words mean but the poetry is obviously missing. The best a computer can strive for is faithful-- beautiful, for the time being, is out of the question.

In painting a Cezannesque apple you would, in essence, be acting as a kind of translator. Specifically you would be trying to translate one visual language (Nature’s) into another’s (Cezanne’s.) Or, in Photoshop parlance, you would be acting as a Cezanne filter.

There are no Cezanne filters that I am aware of but there are filters known as “art filters” that use algorithms to evaluates edges, values and color in a photograph and manipulate them into the appearance of a painting, sketch or an old photograph (here's a research paper that gives you feel for how a computer "sees" a picture.) On my iphone, for instance, I have dozens of image manipulation apps. Some apps mimic the look of various film-types like polaroid instant film or make video look like it was shot on an old movie camera from the 1920s (interestingly, in the 1800s a number of photographers attempted to make their photos look more like paintings by manipulating the development process to achieve painterly effects— an analog version of an algorithmic photo filter.) Other apps turn photos into the gist of a basic pencil sketch or cartoon. I even have an app that can turn video into an animated cartonon. So as I hold my iphone up to record video it instantly translates what it sees into a 30-frame-per-second animated live cartoon of my surroundings (I can imagine one day being able to view Pixar-quality animations of friends and family.) Applications written for full-powered computers can be quite sophisticated in terms of there ability to mimic natural media like oil paint or watercolor-- both in terms their appearance and their working properties. There are also more and more programs now that attempt to mimic specific styles of painting like, say, Impressionism or Pointillism. You feed the program a photo and it spits out an impressionistic version of it. The results are almost always very bad. Like language translation, art filters are far too simplistic to handle the contextual/aesthetic complexities of painting. It is hard enough for a human to define what a good Impressionistic painting is, much less write an algorithm that can define it for an unthinking computer.

But computers are getting more powerful and algorithms more sophisticated. Massive databases of information can now be accessed so quickly, and patterns discerned so efficiently, that a computer can appear to be sentient. You may have read recently about IBMs latest computer that competed against two Jeopardy champions. The computer, named Watson, had access to 200 million pages of information that consisted of raw data like encyclopedias and dictionaries, books, news, movie scripts etc. It was not connected to the internet or guided by any human helpers. As with most computers, Watson’s weakness is the inability to understand the nuances of speech and language or to have any life experiences to draw upon to devine answers, but scientists alleviated these problems by loading the data onto the computer’s RAM rather than the hard drive which made searches much more quick and nimble. Algorithms were then designed to take advantage of this increased speed to find subtle patterns and probabilities inside the mountain of data. So Watson listened to Alex Trabeck, rang the buzzer, and answered in the form of a question— all without human intervention. Watson won.

Another interesting example is a program called “Emmy” designed by David Cope, a professor at UC Santa Cruz. Cope was having trouble finishing an opera commission so he designed a program that could emulate the work of several great composers to help spur his thinking. Emmy uses an algorithm to find patterns in a great composer’s music and then uses that information to piece together the composer’s style and create a new composition. When an audience was asked to listen to an Emmy-created Bach composition and a real Bach composition, they could not tell the difference. An argument could made that the “new” compositions are merely derivative and so not new at all, but couldn’t the same be said of human composers? As Picasso once said, “good artists copy, great artists steal.”

At the University of Georgia, Gil Weinberg designed a robot, named Shimon, that can interact with other musicians and also, supposedly, play and improvise like Thelonious Monk.

From NPR:

Weinberg programmed Shimon to play like Thelonious Monk. He says that, though he and his team were trying to teach the robot to play like a machine, they first had to teach it how a human plays. To do that, they used statistics and analysis of Monk's improvisation. Once they had a statistical model of the pianist, they could program the robot to improvise in that model.

Weinberg says the robot won't play everything exactly like the bebop pianist — or any other jazz master — would, though he says, "It probably will keep the nature and the character of [the musician's] style."

"It's difficult to predict exactly what they would do in every single moment in time," he says. "But our algorithm pretty much looks at the past several notes that it plays and, based on that, it sees what is the probability of the next note to be, based on all of this analysis of a large corpus of transcribed improvisation."

Here’s a video of Shimon playing.

I think it is likely that in the near future there will be a painterly version of Emmy, Watson or Shimon that can digitally paint in the manner of an artist by intelligently analyzing an enormous database of that artist’s work and perhaps even the work of those who influenced him. The output will vary in quality, of course, and depend a great deal on the appropriateness of the input, but I’d guess that at least some of the resulting images will be quite convincing. Also, it is not hard to imagine that in 5 or 10 years, display screens will be capable of displaying images that, from a few feet away, are virtually indistinguishable from real paintings. Perhaps they will be like the Kindle screen, except able to reproduce millions (billions?) of colors and a have a resolution that not only reproduces the details (think of the Google Art Project) of the depicted image but also accurately conveys the texture, sheen and depth of the brushstrokes. This screen would likely be very thin, light and easy to hang on a wall. It would also be fairly inexpensive and have a battery life of months rather than hours or days. And, as an aside, maybe there will be a company called “Artflix” rather than “Netflix” whereby one could download an ultra-high resolution image of a great painting (I suppose the company would have to work out some kind of revenue sharing system with the museums and galleries that owned the rights to those paintings— like iTunes did with the music labels.) You could rent a Vermeer for a week…

paint-by-numbers

We all act as filters to some extent. Our minds edits incoming signals (photons don't have color, for instance-- we assign them colors via rods and cones)  These are primal algorithms over which we have very little control (we do not have a choice to see in black and white.) The human algorithms I am referring to are those decision matrices we use by choice in course of a painting-- processes, techniques and methods that we learned in our training and practice. In one sense they help us build our paintings by freeing us up to focus on the larger idea of what we want to express. In a painter's formative years he borrows algorithms because emulation is one way a painter learns from other painters. Borrowed algorithms serve as temporary bridges that allow him to cross artistic waters he may not have the experience or knowledge to navigate by himself. As he practices he slowly develops his own. However, an algorithm can devolve into a habit of sorts if a habit is defined as an automatic reaction to a specific situation. Painters are tempted to rely on such habits because they allow them to avoid the risk inherent in painting and thus mitigate the struggle. The quest for a technique or method often turns into a quest for shortcuts— that is, the successful deployment of a technique or style becomes an end in itself. The larger thought or idea that style or method was supposed to serve gets lost in the pursuit of risk-avoidance and efficiency. A kind of analog version of an algorithmic filter is the paint-by numbers painting system. In paint-by-numbers, one is presented with an image divided up into numbered sections, with each number assigned a color. You paint each section with the corresponding color. If one designed a PBN system using thousands of colors and thousands of sections and perhaps added other parameters like degrees of softness to edges or types of brushstrokes (thick or then, fats or slow etc), the end result might appear to be quite sophisticated and intricate. Add a few more subtle variations and a painter could come to believe he was following his own muse rather than a set of instructions. The whole purpose of paint-by-numbers is to make painting a pleasurable, soothing experience like putting together a jigsaw puzzle. It takes patience and some skill but your path is made plain and you know what the results are going to be beforehand. The painting is a foregone conclusion, no matter it’s complexity, and the smell of paint belies the fact that the painter is simply being a computer running an algorithm.

An example of efficiency in painting taken to an extreme can be found in the art factories of China. Sixty percent of the world’s mass produced, cheap oil painting copies come from one small town (1.5 square miles) in China, called Dafen. A worker there can produce a couple of dozen copies a day by hand and it is estimated that 5 million paintings are produced in Dafen every year. There are assembly lines too, as described in The Economist:

Dafen—and other villages like it—are bringing the factory assembly-line into the artist's studio. In a dimly lit hall on the outskirts of Dafen, “painter workers” stand side by side dabbing colours onto canvas. Liu Chang Zhen, a 27-year-old, works eight hours a day to complete more than 200 canvases a month—painting several copies of a picture at a time, methodically filling in the same patch on each before moving to a new part. At other factories, painters work on the same product, but specialize in different parts—in ears or hands or trees. They work from art books, postcards and images from the internet. Sometimes they just paint inside an outline copied electronically from a photograph, enlarged and stamped on the blank canvas.

These workers are trained to be, first and foremost, efficient. They find the quickest, easiest way to complete a technique so that it can be repeated without much thought. Apparently there is little pretense among the workers that this is high art, but workers do take pride in the specific skills required. In Dafen, for instance, there are regular art competitions where several dozen workers compete to see who can complete a copy (or a “replica” as they are referred to) of a masterpiece the fastest and most accurately. It is art as sport.

I am sure one day robots, using algorithms and printers (or perhaps using real brushes and paints) will replace these assembly line workers just as robots replaced many workers in industrial factories here. Looking at this medical robot, called Da Vinci, it is not hard to imagine it manipulating a brush.  Low level, repetitive jobs are always the ones that technology targets and replaces first. "Low-level" is always being defined up:

Tuesday was a great day for W. Roberts, as the junior pitcher threw a perfect game to carry Virginia to a 2-0 victory over George Washington at Davenport Field. Twenty-seven Colonials came to the plate and the Virginia pitcher vanquished them all, pitching a perfect game. He struck out 10 batters while recording his momentous feat. Roberts got Ryan Thomas to ground out for the final out of the game. Tom Gately came up short on the rubber for the Colonials, recording a loss. He went three innings, walked two, struck out one, and allowed two runs. The Cavaliers went up for good in the fourth, scoring two runs on a fielder's choice and a balk.

The above excerpt was written by a computer program that writes local sports stories using just the statistics from the game as a source.

Every painter experiences moments when he feels as though he is just filling in the numbers. The mind drifts and the algorithms take over. For too many painters this is a desirable and sought-after state because it is taken as a sign of skillfulness— an ease that comes from many hours of practice. But we have all seen paintings that are skillfully, even beautifully done, yet something is missing. It is as though the painter knew what his painting was going to look like before he started and did not allow any room for variations or tangential discoveries. There is no risk, no probing, no investigation, no surprise. No curiosity.

Here is a quote from a lecture given by David Breswick from the Center for Applied Educational Research, University of Melbourne, where he discusses the nature of curiosity:

The highly curious person will have a high regard for the uniqueness of the signal and for the integrity of the cognitive map, and so will be loathe to either assimilate or accommodate. He or she will seek the best possible fit, and typically that will require seeking additional information to build a suitable new integration of the incoming information with what was known before. So questions will be asked, calculations might be made, things will be turned over and looked under, there may well be much wondering and doubting, but after the ball has been kept bouncing for a sufficient length of time some sort of resolution will be reached in which sufficient accommodation occurs for the conceptual conflict to be resolved. The result is that a new order of representation of the world is developed.

He goes on:

To continue with the characteristics of highly curious people, I like to think of curiosity as belonging at the border between chaos and cosmos, so highly curious people will remain longer than others in situations of uncertainty, as well as being more likely to be there, that they will have developed a range of investigative skills to help resolve conceptual conflicts by gathering additional information, that they will have a sufficient sense of security in their world to put their cognitive map in jeopardy without debilitating anxiety, to run the risk of creating a new and better order, and that they will have the capacity to carry out the integration required to create a sense of cosmos where there was the threat of chaos. That is, they will be able, typically, and more than most people, to create, maintain, and resolve conceptual conflicts.

Curiosity is a quality more often associated with scientists than artists, but all the good painters I know are exceedingly curious. They like to “peel the onion” in that they peel away one layer of understanding in a painting so as to reveal another and another and so on. In so doing they learn to become comfortable with being lost in a painting; of not knowing what to do. Scientists are quite comfortable in this state of unknowing because it is where they spend most of their time. They have a “notion about the cosmos” that they then must test with experiments. The results, invariable, will lead him, or someone else, to further questions, theories and experiments. And so the onion is peeled. Paintings should be experiments too. Not in the sense of self-consciously trying to create something new or cutting-edge, but rather in the sense of being open to new possibilities as each painting develops.

Filmmaker Werner Herzog on making movies:

Coincidences always happen if you keep your mind open, while storyboards remain the instruments of cowards who do not trust in their own imagination and who are slaves of a matrix... If you get used to planning your shots based solely on aesthetics, you are never that far from kitsch.

and:

Very often, footage that you have shot develops its own dynamic, it's own life, that is totally unexpected, and moves away from you're original intentions. And you have to acknowledge, yes, there is a child growing and developing and moving in a direction that isn't expected-accept it as it is and let it develop its own life.

Painterly, to me, does not mean a painting with thick paint or bravura brushwork. It means a painting that is cultivated and allowed to grow in it’s own way. This is difficult to do because it requires that we wander into unknown territory where we have no rules to guide us and thus we are forced to make our own. This is where it is tempting to unthinkingly use “off-the-shelf” solutions, a ready-made pieces of "code" that we can insert into our painting to help us deal with an edge or a shape or composition that doesn’t seem to work. We ask, “What would so and so do?” or we reach for some well-worn solution of our own instead of exploring other possibilities or refinements. We have instant access to more painters and paintings than anytime in history. This is, by and large, a good thing but it can also be quite inhibiting because that means, at any point in a painting, we can peruse and find a number of solutions to whatever painting problem we are working on. Malraux wrote,"The poet is haunted by a voice with words must be harmonized." Today it is a million voices.

My reason for discussing such technology is not to sound an alarm about computers replacing painters but rather to study and perhaps become more sensitive to those moments when we become computers. No, the computer will not make painting obsolete anymore than photography did, but I do believe it will be disruptive. Much of what we see now in terms of painting and computers is in it’s infancy and, like most technology, when it first starts out it can appear simplistic and even silly. However, it did not take long for photography to become the de facto way to record visual facts and as cameras grew smaller, cheaper and more efficient it became evident: if your job as a painter was to merely paint facts, your equal became a box with a pinhole in it. Similarly, now, if your mission as a painter is to merely follow a set of visual rules (“when you see this, paint this”) then your equal will soon be a piece of silicon. Photography started an ongoing conversation about what painting is and what it should be and I believe computers will soon rekindle this conversation.

Cezanne had a unique goal in mind for what his painting should be and so he had to find unique solutions. Frankly, I don’t understand how Cezanne did what he did. I don’t know how he made an apple seem so dense and heavy and tangible. Obviously it has something to do with his deliberate marks, his use of color and strong edges and the way he structured his compositions and his perspective, but the traditional building blocks of rendering volume and weight don’t quite explain it. I’m not sure even Cezanne could explain it. Cezanne’s technique and style came as a result of the pursuit of his goal, a goal that was maybe, to him, beyond his technical ability or perhaps even his full understanding (he felt that he failed to reach it.)

In the opening paragraph of Cezanne’s Doubt, Maurice Merleau-Ponty writes:

It took him one hundred working sessions for a still life, one hundred- fifty sittings for a portrait. What we call his work was, for him, an attempt, an approach to painting. In September of 1906, at the age sixty-seven—one month before his death—he wrote: "I was in such a state of mental agitation, in such great confusion that for a time I feared my weak reason would not survive.... Now it seems I am better that I see more clearly the direction my studies are taking. Will I arrive at the goal, so intensely sought and so long pursued? I am working from nature, and it seems to me I am making slow progress”. Painting was his world and his mode of existence. He worked alone without students, without admiration from his family, without encouragement from the critics. He painted on the afternoon of the day his mother died. In 1870 he was painting at l'Estaque while the police were after him for dodging the draft. And still he had moments of doubt about this vocation. As he grew old, he wondered whether the novelty of his painting might not come from trouble with his eyes, whether his whole life had not been based upon an accident of his body. The hesitation or muddle-headedness of his contemporaries equaled this strain and doubt. "The painting of a drunken privy cleaner," said a critic in 1905. Even today, C. Mauclair finds Cezanne's admissions of powerlessness an argument against him. Meanwhile, Cezanne's paintings have spread throughout the world. Why so much uncertainty, so much labor. so many failures, and, suddenly, the greatest success?

Several years ago I read a lovely metaphor for how an artist develops a way of painting. I cannot seem to find it online anymore and I don't recall who wrote it.  Anyway, it went something like this: The Nautilus is born in a small shell that has seven chambers. As the nautilus feeds and grows it adds a new chamber, slightly larger than the last, to accommodate its new size. This growth continues until death (on average twenty years later.) At the end of its life the nautilus leaves behind an extraordinarily precise architecture that is as beautiful as it is strong. The Nautilus did not set out to make a beautiful shell-- the shell formed as a result of the nautilus living its life.

--
Posted By Duane to On Painting at 4/24/2011 08:44:00 AM