Merely take an image of the fashion you need to copy, enter the textual content you. generate, and this algorithm will generate a brand new image out of it. The outcomes are extraordinarily spectacular, particularly should you contemplate that they have been made out of a single line of textual content! If that sounds fascinating, watch the video and study extra! You may even obtain that from solely textual content and may strive it proper now with this new methodology and their Google Colab pocket book accessible for everybody (see references). Simply look again on the outcomes above, such a giant step ahead!
Louis Bouchard
I clarify Synthetic Intelligence phrases and information to non-experts.
Have you ever ever dreamed of taking the fashion of an image, like this cool TikTok drawing fashion on the left, and making use of it to a brand new image of your alternative? Properly, I did, and it has by no means been simpler to do. In truth, you’ll be able to even obtain that from solely textual content and may strive it proper now with this new methodology and their Google Colab pocket book accessible for everybody (see references).
Merely take an image of the fashion you need to copy, enter the textual content you need to generate, and this algorithm will generate a brand new image out of it! Simply look again on the outcomes above, such a giant step ahead! The outcomes are extraordinarily spectacular, particularly should you contemplate that they have been made out of a single line of textual content! If that sounds fascinating, watch the video and study extra!
Watch the video
References
►Learn the total article: https://www.louisbouchard.ai/clipdraw/
►CLIPDraw: Frans, Okay., Soros, L.B. and Witkowski, O., 2021. CLIPDraw:
exploring text-to-drawing synthesis by language-image encoders. https://arxiv.org/abs/2106.14843
►StyleCLIPDraw: Schaldenbrand, P., Liu, Z. and Oh, J., 2021.
StyleCLIPDraw: Coupling Content material and Type in Textual content-to-Drawing Synthesis. https://arxiv.org/abs/2111.03133
►CLIPDraw Colab pocket book: https://colab.analysis.google.com/github/kvfrans/clipdraw/blob/important/clipdraw.ipynb
►StyleCLIPDraw code: https://github.com/pschaldenbrand/StyleCLIPDraw
►StyleCLIPDraw Colab pocket book: https://colab.analysis.google.com/github/pschaldenbrand/StyleCLIPDraw/blob/grasp/Style_ClipDraw_1_0_Refactored.ipynb
►My E-newsletter (A brand new AI software defined weekly to your emails!): https://www.louisbouchard.ai/e-newsletter/
Video Transcript
00:00
have you ever ever dreamed of taking a
00:01
image like this cool tick tock drawing
00:03
fashion and making use of it to a brand new image
00:06
of your alternative effectively i did and it has
00:08
by no means been simpler to do in actual fact you’ll be able to
00:10
even obtain that from solely textual content and also you
00:13
can strive it proper now with this new
00:15
methodology and their google collab pocket book
00:17
accessible for everybody merely take a
00:19
image of the fashion you need to copy
00:21
enter the textual content you need to generate and
00:23
this algorithm will generate a brand new
00:25
image out of it have a look at that such a
00:28
massive step ahead the outcomes are
00:30
extraordinarily spectacular particularly should you
00:31
contemplate that they have been made out of a
00:33
single line of textual content right here i attempted
00:35
imitating the identical fashion with one other
00:37
textual content enter to be trustworthy generally it might
00:40
look a bit everywhere particularly
00:42
if you choose a extra sophisticated or
00:44
messy drawing fashion like this one
00:46
talking of one thing messy in case you are
00:47
like me and your mannequin versioning and
00:49
useful resource monitoring appears to be like like this you
00:51
will be the good candidate to strive the
00:53
sponsor of two days video which is none
00:55
aside from weights and biases i all the time
00:57
assumed i may stack folders like this
00:59
and easily add outdated v1 v2 v3 and so forth to
01:03
my file names with none downside till
01:06
i needed to work with somebody whereas it might
01:07
be simple for me to seek out my outdated exams it
01:10
was unattainable to elucidate my thought
01:12
course of behind this mess and was my
01:14
teammate’s nightmare should you care about
01:15
your teammates and reproducibility do not
01:18
do like i did and provides weights and
01:20
biases a shot no extra notebooks or
01:22
outcomes saved in all places because it creates a
01:24
tremendous pleasant consumer dashboard for you
01:26
and your crew to trace your experiments
01:28
and it is tremendous simple to arrange and use
01:30
it is the primary hyperlink within the description
01:32
and that i promise inside a month you can be
01:34
utterly dependent
01:37
as we mentioned this new mannequin by peter
01:39
schaldenbrunn ethel referred to as fashion clip
01:42
draw which is an enchancment upon clip
01:44
draw by kevin franz in any respect takes an
01:46
picture and takes as inputs and may
01:48
generate a brand new picture based mostly in your textual content
01:50
and following the fashion within the picture so
01:52
the mannequin has to each perceive what’s
01:54
within the textual content and the picture to appropriately
01:56
copy its fashion as chances are you’ll suspect this
01:59
is extremely difficult however we’re
02:01
lucky sufficient to have a variety of
02:02
researchers engaged on so many various
02:04
challenges like making an attempt to hyperlink textual content with
02:07
photographs which is what clip can do rapidly
02:10
clip is a mannequin developed by openai that
02:12
can mainly affiliate a line of textual content
02:14
with a picture each the textual content and pictures
02:17
shall be encoded equally in order that they
02:19
shall be very shut to one another within the
02:21
new house they’re encoded in in the event that they
02:23
each imply the identical factor utilizing clip the
02:25
researchers may perceive the textual content
02:27
from the consumer enter and generate an
02:29
picture out of it in case you are not acquainted
02:31
with clip but i’d advocate watching
02:33
a video i made about it along with
02:35
dolly earlier this yr however then how did
02:38
they apply a brand new fashion to it clip is
02:40
simply linking current photographs to texts it
02:43
can’t create a brand new picture certainly we additionally
02:46
want one thing else to seize the fashion
02:48
of the picture despatched in each the textures
02:50
and shapes effectively the picture technology
02:52
course of is kind of distinctive it will not merely
02:55
generate a picture immediately slightly it
02:57
will draw on a canvas and get higher and
02:59
higher over time it is going to simply draw
03:01
random traces at first and create an
03:03
preliminary picture this new picture is then
03:06
despatched again to the algorithm and in contrast
03:08
with each the fashion picture and the textual content
03:10
which is able to generate one other model this
03:12
is one iteration at every iteration we
03:15
draw random curves once more oriented by the
03:17
two losses we’ll see in a second this
03:19
random course of is kind of cool because it
03:22
will permit every new check to look
03:24
completely different so utilizing the identical picture and
03:26
identical textual content as inputs you’ll find yourself with
03:29
completely different outcomes which will look even
03:31
higher right here you’ll be able to see an important
03:33
step referred to as picture augmentation it is going to
03:35
mainly create a number of variations of
03:38
the picture and permit the mannequin to
03:39
converge on outcomes that look proper to
03:42
people and never merely on the precise
03:44
numerical values for the machine this
03:46
easy course of is repeated till we’re
03:49
happy with the outcomes so this complete
03:51
mannequin learns on the fly over many
03:54
iterations optimizing two losses we see
03:56
right here one for aligning the content material of the
03:59
picture with the textual content despatched and the opposite
04:01
additional fashion right here you’ll be able to see the primary
04:03
lust relies on how shut the clip
04:06
encodings are as we mentioned earlier the place
04:08
clip is mainly judging the outcomes
04:11
and its determination will orient the subsequent
04:12
technology the second can be very
04:15
easy we ship each photographs right into a
04:18
pre-trained convolutional neural community
04:20
like vgg which is able to encode the photographs
04:22
equally to clip we then examine these
04:24
encodings to measure how shut they’re
04:26
to one another this shall be our second
04:29
decide that may orient the subsequent
04:30
technology as effectively this fashion utilizing each
04:33
judges we are able to get nearer to the textual content and
04:35
the wished fashion on the identical time within the
04:37
subsequent technology in case you are not acquainted
04:39
with convolutional neural networks and
04:41
encodings i’ll strongly advocate
04:43
watching the video i made explaining
04:45
them in easy phrases this iterative
04:47
course of makes the mannequin a bit gradual to
04:49
generate a lovely picture however after a
04:51
few hundred iterations or in different phrases
04:53
after a couple of minutes you have got your new
04:55
picture and that i promise it is definitely worth the wait
04:58
it additionally implies that it does not require
05:00
some other coaching which is fairly cool
05:02
now the fascinating half you’ve got been
05:04
ready for certainly you should utilize it proper
05:06
now free of charge or a minimum of fairly cheaply
05:08
utilizing the collab pocket book linked within the
05:10
description beneath i had some issues
05:12
operating it and i’d advocate shopping for
05:14
the professional model of collab if you would like
05:16
to play with it with none points
05:19
in any other case be happy to ask me any
05:21
questions within the feedback should you
05:22
encounter any issues i just about
05:24
went by all of them myself to make use of
05:27
it you merely run all cells like that
05:29
and that is it now you can enter a brand new
05:31
textual content for the technology or ship a brand new
05:33
picture for the fashion from a hyperlink and
05:35
voila now tweak the parameters and see
05:38
what you are able to do should you play with it
05:40
please ship me the outcomes on twitter
05:42
and tag me i would like to see them as they
05:44
state within the paper the outcomes can have
05:46
the identical biases because the fashions they use
05:49
comparable to clip which you must contemplate
05:51
should you play with it after all this was a
05:53
easy overview of the paper and that i
05:55
strongly invite you to learn each clip
05:57
draw and magnificence clip draw for extra
05:58
technical particulars and take a look at their collab
06:01
pocket book each are linked within the
06:02
description beneath thanks as soon as once more
06:05
weights and biases for sponsoring this
06:07
video and large due to you for
06:09
watching till the top i hope you
06:11
loved this week’s video let me know
06:13
what you suppose and the way you’ll use this
06:15
new mannequin
06:17
[Music]
