After diving into the rabbit hole of merging models, and being lost on what IN00 or OUT08 or M00 was, or what it did, or how to use it.
I started playing with SuperMerger:
https://github.com/hako-mikan/sd-webui-supermerger
And between hours of sliding them randomly without being able to tell what did what, I figured out the best way is to use a VERY anime Pony, and a very realistic SDXL to merge, so I can see a big difference on sliding values on the blocks.
Along with pestering ChatGPT with my findings, assumptions, corrections, and keeping it organized for me, and giving me feedback, this is what I found:
Note:
I may be wrong in this, if so feel free to correct me. I am only armchairing speculation over what it took to convince a pure anime Pony mode to make realistic pics, while keeping it Pony poses, knowledge, and proportions for people.
It seems it processes blocks in a linear(ish) pattern of noise to the final image, each impacting the process along the way, and which model it leans towards for that step in the process of noise -> final image.
In the example below:
Model A is a highly anime based Pony model.
Model B is a highly realistic SDXL model.
Input and Early Processing (Base and IN00–IN04)
Step: These blocks process the initial noise input and create the foundational structure of the image.
Blocks: Base, IN00–IN04
Function:
Define the scene layout (composition, perspective, and object placement).
Handle poses and anatomy for characters or objects in the scene.
Impact of Weights:
High Model A weight: Keeps the structure anime-like (e.g., exaggerated poses, stylized proportions).
High Model B weight: Makes the structure more realistic (e.g., accurate human anatomy, balanced proportions).
2. Downsampling Path (IN05–IN11)
Step: These blocks extract increasingly detailed features by reducing the image resolution (downsampling).
Blocks: IN05–IN11
Function:
Refine mid-level details, like shapes, textures, and shading.
Begin introducing style features (e.g., anime or realism textures, basic shading patterns).
Impact of Weights:
High Model A weight: Keeps mid-level details stylized (anime-style texturing and shading).
High Model B weight: Introduces realistic textures and consistent shading.
3. Bottleneck (m00)
Step: This block, also called the middle block, connects the downsampling and upsampling paths.
Blocks: m00
Function:
Encodes the global style and coherence of the image.
Balances the influence of broad structure and fine details.
Impact of Weights:
High Model A weight: Preserves anime-like global style.
High Model B weight: Shifts the overall style toward realism.
4. Upsampling Path (OUT00–OUT03)
Step: These blocks begin refining the image by adding detail as the resolution increases (upsampling).
Blocks: OUT00–OUT03
Function:
Handle mid-level textures and shading transitions.
Ensure style consistency between the structure and details.
Impact of Weights:
High Model A weight: Retains stylized textures and shading patterns.
High Model B weight: Adds realistic texturing and tonal depth.
5. Fine Detail Refinement (OUT04–OUT07)
Step: These blocks refine the high-resolution details, like facial features, skin textures, and intricate elements.
Blocks: OUT04–OUT07
Function:
Enhance faces, fine textures, and other small details.
Add realism or stylization to critical focal points (e.g., eyes, skin, hair).
Impact of Weights:
High Model A weight: Faces and details remain anime-like (simplified or stylized).
High Model B weight: Faces and details become highly realistic.
6. Final Touches (OUT08–OUT11)
Step: These blocks handle the final stages of image generation, adding shadows, highlights, and other finishing touches.
Blocks: OUT08–OUT11
Function:
Enhance lighting effects, shadows, and overall rendering quality.
Ensure the final image looks cohesive and polished.
Impact of Weights:
High Model A weight: Keeps stylistic shadows and artistic flair.
High Model B weight: Adds realistic shadows, highlights, and tonal balance.
Summary of Block Roles in Image Generation
Stage
Blocks Involved
Role
Initial Structure
Base, IN00–IN04
Define composition, pose, and basic structure.
Feature Extraction
IN05–IN11
Add mid-level textures, shapes, and shading.
Global Style
m00
Blend and set the overall style of the image.
Detailing
OUT00–OUT03
Handle mid-level textures and shading transitions.
Fine Refinement
OUT04–OUT07
Add realistic or stylized high-resolution details.
Final Rendering
OUT08–OUT11
Apply shadows, highlights, and polish the image.
Impact of Weight Changes
Early Blocks (Base, IN00–IN04):
Control the scene, composition, and pose.
Change these to lean toward Model A for anime-style composition or Model B for realism.
Mid Blocks (IN05–OUT03):
Balance textures and shading.
Adjust these to emphasize stylistic or realistic textures.
Late Blocks (OUT04–OUT11):
Focus on faces, details, and lighting.
Favor Model B for realistic faces and polished rendering.