I’m reaching ~100 Stable Diffusion fine-tune runs, and I decided to share my current thoughts about the process and a mental model of how to approach it. I’m sure this framework will change and evolve as I learn more.
Fine-tuning stable diffusion models can be a complex and time-consuming process, as there is a wide range of different methods and approaches available. For example, you might choose to use a tool like Dreambooth, which offers various implementation options, or you might prefer to use a tool like Everydream Trainer. And more and more of such tools will continue showing up. Additionally, each tool has many different settings and configurations that you can use to fine-tune your model, making it difficult to know which approach is best. This can lead to a situation where you are left trying to evaluate an almost overwhelming number of permutations to find the perfect approach, which can be a challenging and time-consuming task.
Start with setting some goals.
This might sound boring, but so far, we find this step critical to narrowing down and choosing the right approach for the task.
Before starting the process of fine-tuning a stable diffusion model, it is important to outline your goals for the model clearly. This means thinking about what you want the model to be good at and what you want it to be able to do after the fine-tuning process is complete. Write out a few vivid descriptors of the ideal scenario of what happens after you are done fine-tuning and what kind of images are generated by the model. Think about the prompts you will want to input and the dream scenario outputs you want to get.
Think about the costs
Fine-tuning AI models can be a time-consuming and costly process. In addition to the cost of GPU computing power and storage, there is also the time and effort you put into the process. Each model has the potential to become a rabbit hole, as you may find yourself constantly re-running, fine-tuning with different methods, and adding more data for training. It's important to be aware of these costs and to set realistic time limits that align with your goals. Don't overlook the time and effort required for fine-tuning, as it can quickly become your main bottleneck.
What about the lower body Midjourney?
Factors to Consider When Fine-Tuning a Stable Diffusion Model
This is not an exhaustive list, but so far, we have found these aspects to matter a lot. And in some ways, you might need to make tradeoffs between them, and the goals set above would be super helpful for making such choices.
Preserving the Functionality of the Original Model
The first one is the importance of preserving the functionality of the original model. How intact do you want to keep it? In the absolute majority of my cases, this is the aspect I care the least about. If I want to train a model to generate Damon Albarn in different situations, I don’t care if my model is no longer able to generate beautiful landscapes or if all persons generated by this model somehow resemble Damon Albarn, no matter the prompt. As long as it does what I want to do, I usually don’t care about the rest. Remember that this will not be the case for all the fine-tunes, especially when one tries to make the overall model look better by fine-tuning on high-quality datasets.
Right Level of Accuracy
Another factor to consider is the accuracy at which you want to replicate the subject (or subjects) that the model is being fine-tuned for. Depending on the specific goals of your project, you may want to focus on achieving a high level of accuracy in your fine-tuning process, or you may be willing to accept a lower level of accuracy in exchange for more flexibility in your outputs or other benefits. For example, if I want photorealistic images of myself, I’d aim for high accuracy, while if I want to fine-tune the model to some new cool style, I might not care that much about the high level of consistency as long as the output looks cool.
Flexibility of the Model
Finally, it is important to consider the flexibility of the model or its ability to produce images of newly learned subjects in various versions and situations. A highly flexible model can put subject(s) in all kinds of images that are quite far from what you had in the training dataset. and usually, you want a lot of it from your fine-tuned models; at the end of the day, this is what AI art is about! but there are degrees to this too. For example, when generating images of shields as game assets, I wanted a lot of flexibility regarding the shapes of the shields and their textures. But I didn’t care about the ability to put these shields in different places and scenarios since that was not the goal of this model.
Understanding the combination of these factors you are aiming for can give you a lot of helpful clues to narrow down the possible approaches for your fine-tuning. For example, Dreambooth shines when it comes to being minimally invasive to the original model. Thinking about consistency and flexibility can help you create an optimal dataset for your training in terms of image types and quantities.
I expect that, over time, gold standards will emerge for the different combinations of goals and the factors mentioned above. Until then, we will have to keep pushing the limits, seeing what is possible, and, through trials and errors, finding better or even optimal ways to do different kinds of fine-tuning scenarios.
If you find this and my other posts useful, please consider subscribing here and sharing some of my posts.
