Hugging Face's logo

zer0int
/
CLIP-GmP-ViT-L-14

Zero-Shot Image Classification
Transformers
Safetensors
clip
Model card Files
xet
 Community
17

Is it okay to use for Flux Lora training?

#8
by WilsonModt - opened
Discussion
WilsonModt

Would it be okay to use this model as a replacement for the clip-l?
zer0int
 Owner

Of course! It has an MIT license, same as the original CLIP-L by OpenAI. :)
aydin99

@zer0int Do you think this would improved text even more if we trained an object with text using the CLIP for training AND Image generation?

Vs just using the OG clip for training and this for image gen?
zer0int
 Owner

Assuming you mean "training CLIP and the diffusion model together": Yes, technically, that is possible. However, it's not the best approach, considering CLIP does contrastive learning (the more negative examples there are in a batch => the bigger the batch_size, the better [up to a certain point]). In my experience, it is best to:

  1. Fine-tune the CLIP model in full, text and image, standalone
  2. Use the CLIP fine-tune with the diffusion model for training, but KEEP CLIP FROZEN and only train the diffusion model for aligning with the fine-tuned CLIP.

Hope that helps!

Sign up or log in to comment