As
this post points out, it only took
14 months for Midjourney to go from this:
To this:
Check out using A.I. to bring children's sketches to life:
So
ControlNet is an approach of using different types of inputs (text prompts, depth maps, etc.) to
shape the output of A.I. That's why you can do things like take a sketch like the one above & apply a stylistic generative A.I. output, except now you can tweak it more to your heart's desires! More info here:
ControlNet is a neural network that controls image generation in Stable Diffusion by adding extra conditions. Details can be found in the article Adding
stable-diffusion-art.com
And here:
ControlNet models give users more control while generating images by providing Canny edge, HED edge, segmentation maps, and even pose detection models.
learnopencv.com
This is the point where A.I. is starting to become more
controllable, like Photoshop, along with the earlier spot-generation tools to tweak A.I.-outputted images in custom ways, rather than merely relying on a whole-image generated output. While I'm really excited for A.I. in the world of film animation (due to the cost & time required for hand-labor & being able to create design language for future films), it's also going to have some REALLY neat effects on things like 2D side-scroller sprite-based video games.
Back when I was learning game design (like...20 years ago when I first joined ATOT in college), you had to do the programming manually (there were no true, powerful game engines like Unreal or Unity available yet!) & the art was mostly hand-drawn, with the exception of like some texture-mapping type of programs & whatnot. Now you can do stuff like this by creating an A.I.-driven workflow. Truly bonkers stuff:
It has all KINDS of applications! For example, speeding up the design & creation of a final vision, such as being able to turn a simple hand-sketch into whatever style of final artwork you desire:
Then you can use ControlNET to tweak the output to achieve the final effect you desire:
It's an absolutely AMAZING tool for architects, home designers, interior decorators, etc:
The thing is, you don't even need good art skills to being communicating ideas for what you want...you can literally just blob out some splotches & then start spot-tweaking designs & styles:
It starts to get even MORE fun when you use tools like OpenPose to do camera capture:
The OpenPose library uses neural networks to perform real-time human body pose estimation for single- and multi-person video analysis.
viso.ai
You can take a simple wireframe capture of a human body & then generate anything you want: gender, age, clothing style, background images, lighting, etc.:
As an artist, this hurts my head...it's both brilliantly exciting & HUGELY terrifying!