The first expands the frame of an image in any desired direction, creating new elements from the ground up that appear to blend into the original work. The second is a brush that replaces elements in the image with what you type in the command prompt.
If you haven’t already received a DALL-E invitation, the best way to experience the power of these tools is to load them this AR Instagram filter and see what they can do.
It’s pretty easy. You load the filter, find a flat surface in your room, and instantly set up a gallery of three classics: René Magritte’s The son of manJohannes Vermeers The girl with the pearl earringand Leonardo da Vinci mona lisa.
There’s nothing special about it until you get closer. Then Mona Lisa suddenly retreats backwards and you discover a whole new world of depth and detail around her.
The process of coloring and tracing
The new artwork looks like it was painted by the original artists, but was created by modern artists using only DALL-E.
“Actually, I come from a traditional field of fine arts, like oil painting. So I really wanted to make sure I showed respect for the picture,” she says Josephine Mueller, an art director and AR/XR/3D artist who created the expanded universe of The son of man.
Miller used DALL-E imaging powers to give his protagonist a face that perfectly suited Magritte’s style. But it wasn’t that easy to summon a faceless man’s face with AI magic. In fact, it took a lot of trial and error to get the right face as Miller envisioned: “I inpainted about 200 faces for this.”
As with the other two paintings, the process required creating different versions of some parts to create all the different elements needed to create the sense of depth.
In case of The son of manthe face and all other elements in the expanded painting have been cropped out to create a two and a half dimensional environment in which each of the cropped layers is set up in 3D space, like the set layers on a theater stage or the multiple layers used in camera animation will.
“When you first see the filter, the face looks like it did in the original painting. There’s no change,” he says Manuel “manu.vision” charan artist and XR design manager at Unity who created the enhanced version of mona lisa.
Only when you get closer does the entire painting begin to reveal itself. And then if you move your phone in any direction, you’ll see other things, like the actual face of the person behind the apple.
“There are these thousands of men floating around. They all have individual faces and Josephine worked on each individual face. It’s crazy,” says Sainsily.
The set was then programmed into a 3D space using Spark AR Studio to create the Instagram effect. The end product feels neat and satisfying, as if you’re accessing a secret, larger-than-life version of a painting that never was.
Much more than just AI
Aside from the work of clipping things and making the AR filter – which is tedious, regular work – the process of developing the entire product is very hard work, not as easy as typing in a command prompt and the end result.
To show that, and to make it clear that this AI technology is extremely powerful but just another tool in an artist’s arsenal, Sainsily says they all filmed themselves at work, which also helped the creative process, because it informed the other team members responsible of cutting, assembling and programming the criteria for making the final piece.
In the end, this process – and the end result – is also a testament to the importance of the human being in the creation of these works. While DALL-E, Stable Diffusion, and the rest of the text-to-image tools are awesome, they’re not at the point where they can make creative decisions just yet. You only spend. It takes the work of people like Miller and Sainsily to create compelling works of art with meaning. At least for now.
