This week, we have an exciting announcement and teaser before diving into our post!
We have been collaborating with Professor Maurizio Forte from Duke University (Professor of Classical Studies, Art, Art History, and Visual Studies; LinkedIn) on the fascinating topic of using generative AI tools for archaeological reconstruction and immersive experiences.
The first results of this collaboration will be showcased at the Field Museum of Chicago, and Professor Forte will be delivering a lecture on the subject on April 26th at 12pm local time. We'll share more materials with you soon, but if you happen to be in the area and are interested in attending the lecture, please reach out, and we'll provide more details.
Let’s get back to our topic!
Why you must try ComfyUI
In general, it is very easy to get used to certain workflows, tools, and approaches; the same is true for AI toolkits. However, we think that given the speed at which generative AI is evolving, we cannot get attached to certain tools. Flexibility is and will remain to be one of the major treatments that can unlock the full potential of space, and lack of it could be a straight road to extinction. That’s why we are building FollowFox on the philosophy of high flexibility.
With that in mind, we decided to take a pause from Automatic1111 for a bit and explore another tool.
ComfyUI Interface for Stable Diffusion has been on our radar for a while, and finally, we are giving it a try. It’s one of those tools that are easy to learn but has a lot of depth potential to develop complex or even custom workflows. But most of all, it’s a visual display of the modularity of the image generation process is a great way to enhance our intuition on how the stable diffusion works and what are all the components that we incorporate in our generation process.
So far, we feel that working with it a slightly more overhead than working in Auto1111, but we have a lot more experience with the latter. At the same time, we developed a few workflows that are just tailored to specific tasks (for example, testing different VAEs), and having the whole chain in front of us really helps us ensure that we are changing just one thing at a time for our experiments.
So if you are like us and have been hesitating to get out of your comfort zone of Auto111 and try other tools, just follow along with this post and see if this is for you! In this post, we will go over the installation process and the first run.
Installation
There are many ways to install ComfyUI, and you can read some tips on how to do so on their repo (link). However, we will continue our tradition and, in the best practice of ML practitioners, run it using Linux through Windows WSL2. In fact, our guide on how to do the same for AUTO1111 to this date is our most popular post (link).
Preparing WSL2 (only if you don’t have it setup)
This part is identical to the AUTO1111 post, and if you already have a WSL2 ready in the manner that we did for Automatic, then you can skip to the next section:
Setup WSL2
Run CMD as admin
Run the command: wsl --install
Restart computer
After restarting, we see Ubuntu is being installed
It prompted us to create a username and password
Once installed, I ran update commands for Ubuntu:
sudo apt update
sudo apt upgrade
7.Close the ubuntu terminal 8.Run CMD as admin 9.Run the command: wsl --update 10.Problem: it installed Ubuntu 22.04, which is a newer version and causes problems. So I’ll uninstall and install 20.04 instead 11.Uninstall by going to windows, add or remove programs, finding ubuntu, and uninstalling. Restart windows once done 12.To install a new version, I opened Microsoft store, searched for Ubuntu 20.04
Launch, wait for installation, and repeat from step 5
to confirm it got cloned, type explorer.exe . command in wsl and confirm that there is a ComfyUI folder there
5.Create a virtual environment with Conda
start wsl and run the command: conda create --name comfy python=3.10
Press y when prompted
confirm installation by running conda activate comfy
6.Paste at least one stable diffusion model for the start
Go to the checkpoints folder by cd ComfyUI/models/checkpoints/
Use your preferred way to paste the model there. In our case, we open the explorer window by explorer.exe . pasting sd_v1-5_vae.ckpt checkpoint there
7.Install required packages and dependencies
make sure you have the conda environment activated:
a.conda activate comfy
go to ComfyUI main folder
b.cd ~/ComfyUI
run
c.pip install -r requirements.txt
restart wsl
First Test Run
start wsl
activate environment
a.conda activate comfy
Go to ComfyUI main folder.
b.cd ~/ComfyUI
Launch UI by running
a.python main.py b.or xformers should work too: python main.py --xformers
You will be provided a local link; go there. In our case, it’s http://127.0.0.1:8188/
You should see the UI in your browser. Let’s test if it generates an image
First, in the leftmost window, select your checkpoint
Then on the right-hand side, press Queue Prompt
What to do next?
We plan to write more about the workflows and explorations we are discovering now, but we won’t go there in this post. Instead, here are a few suggestions for what you can try:
Try to understand what each module is for the default view
Try to match output with your AUTO1111 setup (so far, we are getting close but failing to get 1 to 1 identical images).
Check out, download, and try examples from their examples repo https://comfyanonymous.github.io/ComfyUI_examples/
Right-click on the UI, press Add Node, and try to explore what each Node types do. Try to add that to your workflows