AI is going to be huge for artists, and the latest demonstration comes from Nvidia, which has built prototype software that turns doodles into realistic landscapes.
Using a type of AI model known as a generative adversarial network (GAN), the software gives users what Nvidia is calling a “smart paint brush.” This means someone can make a very basic outline of a scene (drawing, say, a tree on a hill) before filling in their rough sketch with natural textures like grass, clouds, forests, or rocks.
The results are not quite photorealistic, but they’re impressive all the same.
This software isn’t groundbreaking exactly — researchers have shown off similar tools in the past, including one from Google that turns doodles into clipart — but it is the most polished demonstration of this concept we’ve seen to date. The software generates AI landscapes instantly, and it’s surprisingly intuitive. For example, when a user draws a tree and then a pool of water underneath it, the model adds the tree’s reflection to the pool.
/cdn.vox-cdn.com/uploads/chorus_asset/file/15972196/nvidia_gaugan_gif.gif)
Image: Nvidia
Demos like this are very entertaining, but they don’t do a good job of highlighting the limitations of these systems. The underlying technology can’t just paint in any texture you can think of, and Nvidia has chosen to show off imagery it handles particularly well.
For example, generating fake grass and water is relatively easy for GANs because the visual patterns involved are unstructured. Generating pictures of buildings and furniture, by comparison, is much trickier, and the results are much less realistic. That’s because these objects have a logic and structure to them that humans are sensitive to. GANs can overcome this sort of challenge, as we’ve seen with AI-generated faces, but it takes a lot of extra effort.
Nvidia didn’t say if it has any plans to turn the software into an actual product, but it suggests that tools like this could help “everyone from architects and urban planners to landscape designers and game developers” in the future.
“It’s much easier to brainstorm designs with simple sketches, and this technology is able to convert sketches into highly realistic images,” said Nvidia’s Bryan Catanzaro in a blog post.
