As mentioned HERE using a different version of transformers==4.28.0 helped for Swin Transformers,
transformers==4.28.0