Some starting code here for starting SD3 implementation and SVD

I was going to use this (now deleted?) SD3 weight map and try to get this extension to work with SD3. Potentially bypassing any abliteration (censorship) by disabling blocks in this node and injecting negatives for better qualaity. It seems SD3 is based on MM-DiT so it may not work like Unet? Can anyone assist me with with this?
This info was deleted on their huggingface. It seems like it gave some insights into SD3 blocks?
https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers/commit/b1148b4028b9ec56ebd36444c193d56aeff7ab56
Attached is some base code for a class PromptInjectionSD3 for the prompt_injection.py with placeholders for transformers_blocks numbers to match the delete data. These are located in the optional input_types. I am not sure what's going on exactly in comfyui model_patcher.py for SD3 or if anything exists there yet for SD3. if it needs to be updated or anyone can give some info on how to update these inputs properly to affect the MM-DiT blocks.
I am testing this non-working starter code on the comfyui simple sd3 workflow and using the single stableDiffusion3SD3_sd3MediumInclT5XXL.safetensors file.
Again the goal here is to disable certain SD3 blocks in the node, do clip text encode to the individual SD3 blocks directly to test if it breaks any abliteration (basically reverse engineering methods of censorship) I want to test with a conditioningzeroout node on just the positive and negative going into the ksamper (and on both), and also if negatives can enhance the results while reducing censorship with certain blocks disabled, or prompting blocks a certain way.
I would immediately type "a woman lying in grass" and start disabling blocks and see which blocks cause the most terror. I also added a
Side note: I also added a class ProjectInjectionSVD for svd injection but I don't know if actually works yet. It doesn't give errors but might be wrong also. There was a recent June 6th paper on pruning SVD blocks to enhance results, I am just getting started on that and may need a separate SVD pruning node but not sure.
The Non-working starting code for SD3 injection and SVD (disclaimer: probably a lot not right, just a starting base)
import comfy.model_patcher
import comfy.samplers
import torch
import torch.nn.functional as F

# Added this for SD3 could be wrong, no errors, but doesn't change image, could be wrong or wrong def patch below?
def build_mmdit_patch(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None, None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        if sigma_start <= sigma <= sigma_end:
            if block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']:
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)

                # Interpolate to match the sizes
                if c.size() != b.size():
                    b = F.interpolate(b.unsqueeze(0), size=c.size()[1:], mode='nearest').squeeze(0)

                out = torch.cat((c, b), dim=-1).to(dtype=context_attn1.dtype) * weight
                return n, out  # Ensure exactly two values are returned
        return n, context_attn1, value_attn1  # Ensure exactly three values are returned

    return prompt_injection_patch

def build_patch(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None,None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        batch_prompt = n.shape[0] // len(extra_options["cond_or_uncond"])

        if sigma <= sigma_start and sigma >= sigma_end:
            if (block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']):
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)
                out = torch.stack((c, b)).to(dtype=context_attn1.dtype) * weight
                out = out.repeat(1, batch_prompt, 1, 1) * weight

                return n, out, out

        return n, context_attn1, value_attn1
    return prompt_injection_patch

# Added this for SVD could be wrong, already added node class mapping at the end
def build_svd_clip_patch(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None, None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        if sigma_start <= sigma <= sigma_end:
            if block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']:
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)

                # Interpolate to match the sizes
                if c.size() != b.size():
                    b = F.interpolate(b.unsqueeze(0), size=c.size()[1:], mode='nearest').squeeze(0)

                out = torch.cat((c, b), dim=-1).to(dtype=context_attn1.dtype) * weight
                return n, out  # Ensure exactly two values are returned for SVD
        return n, context_attn1, value_attn1  # Ensure exactly three values are returned

    return prompt_injection_patch

class PromptInjectionSD3: # SD3 injection node added , already added node class mapping at the end
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {"model": ("MODEL",)},
            "optional": {
                "all": ("CONDITIONING",),
                "context_block": ("CONDITIONING",),
                "norm_out_linear": ("CONDITIONING",),
                "pos_embed_proj": ("CONDITIONING",),
                "proj_out": ("CONDITIONING",),
                "time_stack": ("CONDITIONING",),
                "transformer_blocks_0": ("CONDITIONING",), # These seem like placeholders? Maybe not, I don't know.
                "transformer_blocks_1": ("CONDITIONING",),
                "transformer_blocks_2": ("CONDITIONING",),
                "transformer_blocks_3": ("CONDITIONING",),
                "transformer_blocks_4": ("CONDITIONING",),
                "transformer_blocks_5": ("CONDITIONING",),
                "transformer_blocks_6": ("CONDITIONING",),
                "transformer_blocks_7": ("CONDITIONING",),
                "transformer_blocks_8": ("CONDITIONING",),
                "transformer_blocks_9": ("CONDITIONING",),
                "transformer_blocks_10": ("CONDITIONING",),
                "transformer_blocks_11": ("CONDITIONING",),
                "transformer_blocks_12": ("CONDITIONING",),
                "transformer_blocks_13": ("CONDITIONING",),
                "transformer_blocks_14": ("CONDITIONING",),
                "transformer_blocks_15": ("CONDITIONING",),
                "transformer_blocks_16": ("CONDITIONING",),
                "transformer_blocks_17": ("CONDITIONING",),
                "transformer_blocks_18": ("CONDITIONING",),
                "transformer_blocks_19": ("CONDITIONING",),
                "transformer_blocks_20": ("CONDITIONING",),
                "transformer_blocks_21": ("CONDITIONING",),
                "transformer_blocks_22": ("CONDITIONING",),
                "transformer_blocks_23": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"
    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, all=None, context_block=None, norm_out_linear=None, pos_embed_proj=None, proj_out=None, time_stack=None,
              transformer_blocks_0=None, transformer_blocks_1=None, transformer_blocks_2=None, transformer_blocks_3=None, transformer_blocks_4=None, transformer_blocks_5=None,
              transformer_blocks_6=None, transformer_blocks_7=None, transformer_blocks_8=None, transformer_blocks_9=None, transformer_blocks_10=None, transformer_blocks_11=None,
              transformer_blocks_12=None, transformer_blocks_13=None, transformer_blocks_14=None, transformer_blocks_15=None, transformer_blocks_16=None, transformer_blocks_17=None,
              transformer_blocks_18=None, transformer_blocks_19=None, transformer_blocks_20=None, transformer_blocks_21=None, transformer_blocks_22=None, transformer_blocks_23=None,
              weight=1.0, start_at=0.0, end_at=1.0):

        if not any((all, context_block, norm_out_linear, pos_embed_proj, proj_out, time_stack, transformer_blocks_0, transformer_blocks_1, transformer_blocks_2, transformer_blocks_3,
                    transformer_blocks_4, transformer_blocks_5, transformer_blocks_6, transformer_blocks_7, transformer_blocks_8, transformer_blocks_9, transformer_blocks_10,
                    transformer_blocks_11, transformer_blocks_12, transformer_blocks_13, transformer_blocks_14, transformer_blocks_15, transformer_blocks_16, transformer_blocks_17,
                    transformer_blocks_18, transformer_blocks_19, transformer_blocks_20, transformer_blocks_21, transformer_blocks_22, transformer_blocks_23)):
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}
        blocks = {
            'context_block': [0],
            'norm_out_linear': [0],
            'pos_embed_proj': [0],
            'proj_out': [0],
            'time_stack': [0],
            'transformer_blocks': list(range(24))
        }

        for block in blocks:
            for index in blocks[block]:
                block_name = f"{block}_{index}"
                value = locals().get(block_name, None)
                if value is None:
                    value = all
                if value is not None:
                    patchedBlocks[f"{block}:{index}"] = value

        m.set_model_attn2_patch(build_mmdit_patch(patchedBlocks, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class SVDPromptInjection: #SVD injection node added
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {"model": ("MODEL",)},
            "optional": {
                "all": ("CONDITIONING",),
                "time_embed": ("CONDITIONING",),
                "label_emb": ("CONDITIONING",),
                "input_blocks_0": ("CONDITIONING",),
                "input_blocks_1": ("CONDITIONING",),
                "input_blocks_2": ("CONDITIONING",),
                "input_blocks_3": ("CONDITIONING",),
                "input_blocks_4": ("CONDITIONING",),
                "input_blocks_5": ("CONDITIONING",),
                "input_blocks_6": ("CONDITIONING",),
                "input_blocks_7": ("CONDITIONING",),
                "input_blocks_8": ("CONDITIONING",),
                "middle_block_0": ("CONDITIONING",),
                "middle_block_1": ("CONDITIONING",),
                "middle_block_2": ("CONDITIONING",),
                "output_blocks_0": ("CONDITIONING",),
                "output_blocks_1": ("CONDITIONING",),
                "output_blocks_2": ("CONDITIONING",),
                "output_blocks_3": ("CONDITIONING",),
                "output_blocks_4": ("CONDITIONING",),
                "output_blocks_5": ("CONDITIONING",),
                "output_blocks_6": ("CONDITIONING",),
                "output_blocks_7": ("CONDITIONING",),
                "output_blocks_8": ("CONDITIONING",),
                "clip_text": ("CONDITIONING",),
                "clip_vision": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"
    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, all=None, time_embed=None, label_emb=None, input_blocks_0=None, input_blocks_1=None, input_blocks_2=None, input_blocks_3=None, input_blocks_4=None, input_blocks_5=None, input_blocks_6=None, input_blocks_7=None, input_blocks_8=None, middle_block_0=None, middle_block_1=None, middle_block_2=None, output_blocks_0=None, output_blocks_1=None, output_blocks_2=None, output_blocks_3=None, output_blocks_4=None, output_blocks_5=None, output_blocks_6=None, output_blocks_7=None, output_blocks_8=None, clip_text=None, clip_vision=None, weight=1.0, start_at=0.0, end_at=1.0):
        if not any((all, time_embed, label_emb, input_blocks_0, input_blocks_1, input_blocks_2, input_blocks_3, input_blocks_4, input_blocks_5, input_blocks_6, input_blocks_7, input_blocks_8, middle_block_0, middle_block_1, middle_block_2, output_blocks_0, output_blocks_1, output_blocks_2, output_blocks_3, output_blocks_4, output_blocks_5, output_blocks_6, output_blocks_7, output_blocks_8, clip_text, clip_vision)):
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}
        blocks = {
            'time_embed': [0],
            'label_emb': [0],
            'input_blocks': list(range(9)),
            'middle_block': list(range(3)),
            'output_blocks': list(range(9)),
            'clip_text': [0],
            'clip_vision': [0]
        }

        for block in blocks:
            for index in blocks[block]:
                block_name = f"{block}_{index}"
                value = locals().get(block_name, None)
                if value is None:
                    value = all
                if value is not None:
                    patchedBlocks[f"{block}:{index}"] = value

        m.set_model_attn2_patch(build_svd_clip_patch(patchedBlocks, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class PromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "all":  ("CONDITIONING",),
                "input_4":  ("CONDITIONING",),
                "input_5":  ("CONDITIONING",),
                "input_7":  ("CONDITIONING",),
                "input_8":  ("CONDITIONING",),
                "middle_0": ("CONDITIONING",),
                "output_0": ("CONDITIONING",),
                "output_1": ("CONDITIONING",),
                "output_2": ("CONDITIONING",),
                "output_3": ("CONDITIONING",),
                "output_4": ("CONDITIONING",),
                "output_5": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, all=None, input_4=None, input_5=None, input_7=None, input_8=None, middle_0=None, output_0=None, output_1=None, output_2=None, output_3=None, output_4=None, output_5=None, weight=1.0, start_at=0.0, end_at=1.0):
        if not any((all, input_4, input_5, input_7, input_8, middle_0, output_0, output_1, output_2, output_3, output_4, output_5)):
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}
        blocks = {'input': [4, 5, 7, 8], 'middle': [0], 'output': [0, 1, 2, 3, 4, 5]}

        for block in blocks:
            for index in blocks[block]:
                value = locals()[f"{block}_{index}"] if locals()[f"{block}_{index}"] is not None else all
                if value is not None:
                    patchedBlocks[f"{block}:{index}"] = value

        m.set_model_attn2_patch(build_patch(patchedBlocks, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class SimplePromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "block": (["input:4", "input:5", "input:7", "input:8", "middle:0", "output:0", "output:1", "output:2", "output:3", "output:4", "output:5"],),
                "conditioning": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, block, conditioning=None, weight=1.0, start_at=0.0, end_at=1.0):
        if conditioning is None:
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        m.set_model_attn2_patch(build_patch({f"{block}": conditioning}, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class SimplePromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "block": (["input:4", "input:5", "input:7", "input:8", "middle:0", "output:0", "output:1", "output:2", "output:3", "output:4", "output:5"],),
                "conditioning": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, block, conditioning=None, weight=1.0, start_at=0.0, end_at=1.0):
        if conditioning is None:
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        m.set_model_attn2_patch(build_patch({f"{block}": conditioning}, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class AdvancedPromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "locations": ("STRING", {"multiline": True, "default": "output:0,1.0\noutput:1,1.0"}),
                "conditioning": ("CONDITIONING",),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, locations: str, conditioning=None, start_at=0.0, end_at=1.0):
        if not conditioning:
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        for line in locations.splitlines():
            line = line.strip().strip('\n')
            weight = 1.0
            if ',' in line:
                line, weight = line.split(',')
                line = line.strip()
                weight = float(weight)
            if line:
                m.set_model_attn2_patch(build_patch({f"{line}": conditioning}, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

NODE_CLASS_MAPPINGS = {
    "PromptInjection": PromptInjection,
    "PromptInjectionSD3": PromptInjectionSD3,
    "SimplePromptInjection": SimplePromptInjection,
    "AdvancedPromptInjection": AdvancedPromptInjection,
    "SVDPromptInjection": SVDPromptInjection
}

NODE_DISPLAY_NAME_MAPPINGS = {
    "PromptInjection": "Attn2 Prompt Injection",
    "PromptInjectionSD3": "Attn2 Prompt Injection SD3",
    "SimplePromptInjection": "Attn2 Prompt Injection (simple)",
    "AdvancedPromptInjection": "Attn2 Prompt Injection (advanced)",
    "SVDPromptInjection": "Attn2 SVD Prompt Injection"
}

Update: Adjusted with standard input_types names as used in the other classes and guessing the input, middle, and output blocks. Still no effect.
import comfy.model_patcher
import comfy.samplers
import torch
import torch.nn.functional as F

def build_mmdit_patch(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None, None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        if sigma_start <= sigma <= sigma_end:
            if block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']:
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)

                # Interpolate to match the sizes
                if c.size() != b.size():
                    b = F.interpolate(b.unsqueeze(0), size=c.size()[1:], mode='nearest').squeeze(0)

                out = torch.cat((c, b), dim=-1).to(dtype=context_attn1.dtype) * weight
                return n, out  # Ensure exactly two values are returned
        return n, context_attn1, value_attn1  # Ensure exactly three values are returned

    return prompt_injection_patch

def build_patch(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None,None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        batch_prompt = n.shape[0] // len(extra_options["cond_or_uncond"])

        if sigma <= sigma_start and sigma >= sigma_end:
            if (block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']):
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)
                out = torch.stack((c, b)).to(dtype=context_attn1.dtype) * weight
                out = out.repeat(1, batch_prompt, 1, 1) * weight

                return n, out, out

        return n, context_attn1, value_attn1
    return prompt_injection_patch

def build_svd_clip_patch(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None, None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        if sigma_start <= sigma <= sigma_end:
            if block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']:
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)

                # Interpolate to match the sizes
                if c.size() != b.size():
                    b = F.interpolate(b.unsqueeze(0), size=c.size()[1:], mode='nearest').squeeze(0)

                out = torch.cat((c, b), dim=-1).to(dtype=context_attn1.dtype) * weight
                return n, out  # Ensure exactly two values are returned for SVD
        return n, context_attn1, value_attn1  # Ensure exactly three values are returned

    return prompt_injection_patch

class PromptInjectionSD3:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {"model": ("MODEL",)},
            "optional": {
                "all": ("CONDITIONING",),
                "context_block": ("CONDITIONING",),
                "norm_out_linear": ("CONDITIONING",),
                "pos_embed_proj": ("CONDITIONING",),
                "proj_out": ("CONDITIONING",),
                "time_stack": ("CONDITIONING",),
                "input_blocks_0": ("CONDITIONING",),
                "input_blocks_1": ("CONDITIONING",),
                "input_blocks_2": ("CONDITIONING",),
                "input_blocks_3": ("CONDITIONING",),
                "input_blocks_4": ("CONDITIONING",),
                "input_blocks_5": ("CONDITIONING",),
                "input_blocks_6": ("CONDITIONING",),
                "input_blocks_7": ("CONDITIONING",),
                "input_blocks_8": ("CONDITIONING",),
                "middle_block_0": ("CONDITIONING",),
                "middle_block_1": ("CONDITIONING",),
                "middle_block_2": ("CONDITIONING",),
                "output_blocks_0": ("CONDITIONING",),
                "output_blocks_1": ("CONDITIONING",),
                "output_blocks_2": ("CONDITIONING",),
                "output_blocks_3": ("CONDITIONING",),
                "output_blocks_4": ("CONDITIONING",),
                "output_blocks_5": ("CONDITIONING",),
                "output_blocks_6": ("CONDITIONING",),
                "output_blocks_7": ("CONDITIONING",),
                "output_blocks_8": ("CONDITIONING",),
                "output_blocks_9": ("CONDITIONING",),
                "output_blocks_10": ("CONDITIONING",),
                "output_blocks_11": ("CONDITIONING",),
                "output_blocks_12": ("CONDITIONING",),
                "output_blocks_13": ("CONDITIONING",),
                "output_blocks_14": ("CONDITIONING",),
                "output_blocks_15": ("CONDITIONING",),
                "output_blocks_16": ("CONDITIONING",),
                "output_blocks_17": ("CONDITIONING",),
                "output_blocks_18": ("CONDITIONING",),
                "output_blocks_19": ("CONDITIONING",),
                "output_blocks_20": ("CONDITIONING",),
                "output_blocks_21": ("CONDITIONING",),
                "output_blocks_22": ("CONDITIONING",),
                "output_blocks_23": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"
    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, all=None, context_block=None, norm_out_linear=None, pos_embed_proj=None, proj_out=None, time_stack=None,
              input_blocks_0=None, input_blocks_1=None, input_blocks_2=None, input_blocks_3=None, input_blocks_4=None, input_blocks_5=None, input_blocks_6=None, input_blocks_7=None,
              input_blocks_8=None, middle_block_0=None, middle_block_1=None, middle_block_2=None, output_blocks_0=None, output_blocks_1=None, output_blocks_2=None, output_blocks_3=None,
              output_blocks_4=None, output_blocks_5=None, output_blocks_6=None, output_blocks_7=None, output_blocks_8=None, output_blocks_9=None, output_blocks_10=None, output_blocks_11=None,
              output_blocks_12=None, output_blocks_13=None, output_blocks_14=None, output_blocks_15=None, output_blocks_16=None, output_blocks_17=None, output_blocks_18=None, output_blocks_19=None,
              output_blocks_20=None, output_blocks_21=None, output_blocks_22=None, output_blocks_23=None, weight=1.0, start_at=0.0, end_at=1.0):

        if not any((all, context_block, norm_out_linear, pos_embed_proj, proj_out, time_stack, input_blocks_0, input_blocks_1, input_blocks_2, input_blocks_3,
                    input_blocks_4, input_blocks_5, input_blocks_6, input_blocks_7, input_blocks_8, middle_block_0, middle_block_1, middle_block_2, output_blocks_0,
                    output_blocks_1, output_blocks_2, output_blocks_3, output_blocks_4, output_blocks_5, output_blocks_6, output_blocks_7, output_blocks_8,
                    output_blocks_9, output_blocks_10, output_blocks_11, output_blocks_12, output_blocks_13, output_blocks_14, output_blocks_15, output_blocks_16,
                    output_blocks_17, output_blocks_18, output_blocks_19, output_blocks_20, output_blocks_21, output_blocks_22, output_blocks_23)):
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}
        blocks = {
            'context_block': [0],
            'norm_out_linear': [0],
            'pos_embed_proj': [0],
            'proj_out': [0],
            'time_stack': [0],
            'input_blocks': list(range(9)),
            'middle_block': list(range(3)),
            'output_blocks': list(range(24))
        }

        for block in blocks:
            for index in blocks[block]:
                block_name = f"{block}_{index}"
                value = locals().get(block_name, None)
                if value is None:
                    value = all
                if value is not None:
                    patchedBlocks[f"{block}:{index}"] = value

        m.set_model_attn2_patch(build_mmdit_patch(patchedBlocks, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class SVDPromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {"model": ("MODEL",)},
            "optional": {
                "all": ("CONDITIONING",),
                "time_embed": ("CONDITIONING",),
                "label_emb": ("CONDITIONING",),
                "input_blocks_0": ("CONDITIONING",),
                "input_blocks_1": ("CONDITIONING",),
                "input_blocks_2": ("CONDITIONING",),
                "input_blocks_3": ("CONDITIONING",),
                "input_blocks_4": ("CONDITIONING",),
                "input_blocks_5": ("CONDITIONING",),
                "input_blocks_6": ("CONDITIONING",),
                "input_blocks_7": ("CONDITIONING",),
                "input_blocks_8": ("CONDITIONING",),
                "middle_block_0": ("CONDITIONING",),
                "middle_block_1": ("CONDITIONING",),
                "middle_block_2": ("CONDITIONING",),
                "output_blocks_0": ("CONDITIONING",),
                "output_blocks_1": ("CONDITIONING",),
                "output_blocks_2": ("CONDITIONING",),
                "output_blocks_3": ("CONDITIONING",),
                "output_blocks_4": ("CONDITIONING",),
                "output_blocks_5": ("CONDITIONING",),
                "output_blocks_6": ("CONDITIONING",),
                "output_blocks_7": ("CONDITIONING",),
                "output_blocks_8": ("CONDITIONING",),
                "clip_text": ("CONDITIONING",),
                "clip_vision": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"
    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, all=None, time_embed=None, label_emb=None, input_blocks_0=None, input_blocks_1=None, input_blocks_2=None, input_blocks_3=None, input_blocks_4=None, input_blocks_5=None, input_blocks_6=None, input_blocks_7=None, input_blocks_8=None, middle_block_0=None, middle_block_1=None, middle_block_2=None, output_blocks_0=None, output_blocks_1=None, output_blocks_2=None, output_blocks_3=None, output_blocks_4=None, output_blocks_5=None, output_blocks_6=None, output_blocks_7=None, output_blocks_8=None, clip_text=None, clip_vision=None, weight=1.0, start_at=0.0, end_at=1.0):
        if not any((all, time_embed, label_emb, input_blocks_0, input_blocks_1, input_blocks_2, input_blocks_3, input_blocks_4, input_blocks_5, input_blocks_6, input_blocks_7, input_blocks_8, middle_block_0, middle_block_1, middle_block_2, output_blocks_0, output_blocks_1, output_blocks_2, output_blocks_3, output_blocks_4, output_blocks_5, output_blocks_6, output_blocks_7, output_blocks_8, clip_text, clip_vision)):
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}
        blocks = {
            'time_embed': [0],
            'label_emb': [0],
            'input_blocks': list(range(9)),
            'middle_block': list(range(3)),
            'output_blocks': list(range(9)),
            'clip_text': [0],
            'clip_vision': [0]
        }

        for block in blocks:
            for index in blocks[block]:
                block_name = f"{block}_{index}"
                value = locals().get(block_name, None)
                if value is None:
                    value = all
                if value is not None:
                    patchedBlocks[f"{block}:{index}"] = value

        m.set_model_attn2_patch(build_svd_clip_patch(patchedBlocks, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class PromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "all":  ("CONDITIONING",),
                "input_4":  ("CONDITIONING",),
                "input_5":  ("CONDITIONING",),
                "input_7":  ("CONDITIONING",),
                "input_8":  ("CONDITIONING",),
                "middle_0": ("CONDITIONING",),
                "output_0": ("CONDITIONING",),
                "output_1": ("CONDITIONING",),
                "output_2": ("CONDITIONING",),
                "output_3": ("CONDITIONING",),
                "output_4": ("CONDITIONING",),
                "output_5": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, all=None, input_4=None, input_5=None, input_7=None, input_8=None, middle_0=None, output_0=None, output_1=None, output_2=None, output_3=None, output_4=None, output_5=None, weight=1.0, start_at=0.0, end_at=1.0):
        if not any((all, input_4, input_5, input_7, input_8, middle_0, output_0, output_1, output_2, output_3, output_4, output_5)):
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}
        blocks = {'input': [4, 5, 7, 8], 'middle': [0], 'output': [0, 1, 2, 3, 4, 5]}

        for block in blocks:
            for index in blocks[block]:
                value = locals()[f"{block}_{index}"] if locals()[f"{block}_{index}"] is not None else all
                if value is not None:
                    patchedBlocks[f"{block}:{index}"] = value

        m.set_model_attn2_patch(build_patch(patchedBlocks, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class SimplePromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "block": (["input:4", "input:5", "input:7", "input:8", "middle:0", "output:0", "output:1", "output:2", "output:3", "output:4", "output:5"],),
                "conditioning": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, block, conditioning=None, weight=1.0, start_at=0.0, end_at=1.0):
        if conditioning is None:
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        m.set_model_attn2_patch(build_patch({f"{block}": conditioning}, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class SimplePromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "block": (["input:4", "input:5", "input:7", "input:8", "middle:0", "output:0", "output:1", "output:2", "output:3", "output:4", "output:5"],),
                "conditioning": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, block, conditioning=None, weight=1.0, start_at=0.0, end_at=1.0):
        if conditioning is None:
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        m.set_model_attn2_patch(build_patch({f"{block}": conditioning}, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

class AdvancedPromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
            },
            "optional": {
                "locations": ("STRING", {"multiline": True, "default": "output:0,1.0\noutput:1,1.0"}),
                "conditioning": ("CONDITIONING",),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, locations: str, conditioning=None, start_at=0.0, end_at=1.0):
        if not conditioning:
            return (model,)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        for line in locations.splitlines():
            line = line.strip().strip('\n')
            weight = 1.0
            if ',' in line:
                line, weight = line.split(',')
                line = line.strip()
                weight = float(weight)
            if line:
                m.set_model_attn2_patch(build_patch({f"{line}": conditioning}, weight=weight, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

NODE_CLASS_MAPPINGS = {
    "PromptInjection": PromptInjection,
    "PromptInjectionSD3": PromptInjectionSD3,
    "SimplePromptInjection": SimplePromptInjection,
    "AdvancedPromptInjection": AdvancedPromptInjection,
    "SVDPromptInjection": SVDPromptInjection
}

NODE_DISPLAY_NAME_MAPPINGS = {
    "PromptInjection": "Attn2 Prompt Injection",
    "PromptInjectionSD3": "Attn2 Prompt Injection SD3",
    "SimplePromptInjection": "Attn2 Prompt Injection (simple)",
    "AdvancedPromptInjection": "Attn2 Prompt Injection (advanced)",
    "SVDPromptInjection": "Attn2 SVD Prompt Injection"
}

hi @brentjohnston I was also working on something similar (not in comfyui) but through diffusers's repository. I've made some changes here which currently lets me inject prompt into n layers out of 24 DiT layers of SD3, check the testing.py script to see how to use.
I am still not sure if I am doing everything correctly i.e specially these lines where I am injecting the prompt timestep embeddings and pooled projection embeddings into each selected transformer block. The main part I am confused with is at the time of injection how the previous encoder_hidden_states value will be treated since we are completely discarding the value currently while making an injection with new encoder_hidden_states_single value and then again going back to encoder_hidden_states for rest/other layers. I think some kind of combination or dependency is required here.
Assuming the above is solved (or atleast working properly), I am currently running some tests where in src/testing.py you can see I am trying with
{"rest": "A photo of a bunny", "single": "A photo of a tiger"}

and I am trying all 1 layer combination, 2 layer combinations, 3 layer combinations and 4 layer combinations possible (exhaustively) from 0-23 (since 24 DiT layers) and trying to note down the combinations which are giving me a bit merged look between tiger and bunny (Or any combination which gives an interesting result).
The above idea is taken from B-LoRA to figure out the layers affecting content and style atleast the most.

I didn't realize this ran so deep.. this seems complex. I'm definitely open to collaborating on this if you are.
So, if I've got this right, you're adding the ability to process and inject prompt embeddings encoder_hidden_states_single and pooled_projections_single into specified transformer blocks during the forward pass. And you modified forward function to handle multiple embeddings and determine which layers to inject based on single_layer_idxs.
It looks like single_layer_idxs is used to specify individual layers to affect in the 24 DiT layers of the model.
The problem being that injecting encoder_hidden_states_single into each block and discarding the original encoder_hidden_states might lead to a loss of the contextual information, and makes the output not great because it disrupts the continuity of context through those layers.
Maybe to fix this (like in your tiger bunny example, you could integrate the new embeddings encoder_hidden_states_single with the existing embeddings encoder_hidden_states? (if you haven't tried this yet) This might need concatenation, addition, or other methods to preserve the context while incorporating new information. Because If the new embeddings overwrite the original ones without preserving context, it might look like a muted unrelated thing, or worse, a woman laying in grass.. leading to to night terrors lol.
In transformer_sd3.py file. Maybe this could preserve the original context while injecting new information? I tried to modify forward function that implements this (have not tested yet so apologies ahead of time if it's bad, just throwing out ideas, it seems like unless this gets resolved this node won't work):
def forward(
    self,
    hidden_states: torch.FloatTensor,
    encoder_hidden_states: List[torch.FloatTensor] = None,
    pooled_projections: List[torch.FloatTensor] = None,
    timestep: torch.LongTensor = None,
    joint_attention_kwargs: Optional[Dict[str, Any]] = None,
    return_dict: bool = True,
    single_layer_idxs: List[int] = None,
) -> Union[torch.FloatTensor, Transformer2DModelOutput]:
    """
    The [`SD3Transformer2DModel`] forward method.
    """
    height, width = hidden_states.shape[-2:]

    hidden_states = self.pos_embed(hidden_states)  # takes care of adding positional embeddings too.
    temb_single = self.time_text_embed(timestep, pooled_projections[0])
    temb = self.time_text_embed(timestep, pooled_projections[1])
    encoder_hidden_states_single = self.context_embedder(encoder_hidden_states[0])
    encoder_hidden_states = self.context_embedder(encoder_hidden_states[1])

    for idx, block in enumerate(self.transformer_blocks):
        if self.training and self.gradient_checkpointing:
            def create_custom_forward(module, return_dict=None):
                def custom_forward(*inputs):
                    return module(*inputs, return_dict=return_dict)
                return custom_forward

            encoder_hidden_states_combined = torch.cat((encoder_hidden_states, encoder_hidden_states_single), dim=-1)
            temb_combined = torch.cat((temb, temb_single), dim=-1)

            hidden_states = torch.utils.checkpoint.checkpoint(
                create_custom_forward(block),
                hidden_states,
                encoder_hidden_states_combined,
                temb_combined
            )
        else:
            if idx in single_layer_idxs:
                # combines the embeddings instead of replacing them, haven't tested yet
                encoder_hidden_states_combined = torch.cat((encoder_hidden_states, encoder_hidden_states_single), dim=-1)
                temb_combined = torch.cat((temb, temb_single), dim=-1)
                encoder_hidden_states, hidden_states = block(
                    hidden_states=hidden_states,
                    encoder_hidden_states=encoder_hidden_states_combined,
                    temb=temb_combined
                )
            else:
                encoder_hidden_states, hidden_states = block(
                    hidden_states=hidden_states,
                    encoder_hidden_states=encoder_hidden_states,
                    temb=temb
                )

    hidden_states = self.norm_out(hidden_states, temb)
    hidden_states = self.proj_out(hidden_states)

    # unpatchify
    patch_size = self.config.patch_size
    height = height // patch_size
    width = width // patch_size

    hidden_states = hidden_states.reshape(
        shape=(hidden_states.shape[0], height, width, patch_size, patch_size, self.out_channels)
    )
    hidden_states = torch.einsum("nhwpqc->nchpwq", hidden_states)
    output = hidden_states.reshape(
        shape=(hidden_states.shape[0], self.out_channels, height * patch_size, width * patch_size)
    )

    if not return_dict:
        return hidden_states

    return Transformer2DModelOutput(hidden_states=hidden_states)

sure! @brentjohnston also thanks for the response! and yes you've got it right.
surely I think some kind of combining or dependency logic has to be there (I'll try out the above concat example though I think concat won't work because it'll change the embedding size, but I'll still try and let you know), will experiment with add, subtract and avg too I think.

it might look like a muted unrelated thing, or worse, a woman laying in grass.. leading to to night terrors lol.

True xD I am seeing its effect already somewhat, but I am also getting some interesting layers (DiT index to note down) from my results which I'll have to validate more.

and I am trying all 1 layer combination, 2 layer combinations, 3 layer combinations and 4 layer combinations possible (exhaustively) from 0-23 (since 24 DiT layers) and trying to note down the combinations which are giving me a bit merged look between tiger and bunny (Or any combination which gives an interesting result).

Currently I am letting this happen, once done I'll experiment with the combination logics. Will update once I have something to share. In the meantime I also tried finding more details on these prompt injection techniques but there doesn't seem to be many.

Sounds good, I gave wrong code before and updated, I didn't realize It flows forward through all blocks sequentially. Someone pointed it out to me. I have to head out but will be on later tonight!

I'm looking into this, but I need to check better how SD3 works before pushing the code. thanks for all the suggestions

I was experimenting on this and just pushed an update https://github.com/huggingface/diffusers/commit/f64c90ce476ac33613fe0a468f4b2d169e1561fc

surely I think some kind of combining or dependency logic has to be there (I'll try out the above concat example though I think concat won't work because it'll change the embedding size, but I'll still try and let you know), will experiment with add, subtract and avg too I think.

And found avg to be a good function (giving far better results compared to add), some results below from my bunny tiger example:
Experiment Image Results

In my somewhat half-day brute force experiments most important layers for main/objective content creation I found to be happening under: 0, 1, 2, 3, 4, 6, 12, 22
Though will have to experiment more with various other prompts to be further sure and find even more finer parts for layers.

Sorry for the delay, that looks really good! I wish I could get to that point. Here is where I'm at.
I have been stuck with this error mostly the entire day:
no CLIP/text encoder weights in checkpoint, the text encoder model will not be loaded.
clip missing: ['text_projection.weight']
Requested to load SD3ClipModel

Here is long summary. I updated the transformer_sd3.py in comfyui venv with your code (not using comfy portable just regular at the moment)
To start I made joint_blocks_0_context required because it seems to need that at least that as a bare minimum to give any sort of image or it would give me required positional argument errors. I'm not sure if attn2 is supposed to be used or attn1 patch methods from comfyui's model_patcher.py I also added something like isinstance(joint_blocks_0_context, list): because it seemed to be a tensor list, and for a while there I was trying to add joint_blocks_0_context_shape as an required input_type because it kept saying it was missing. So I made it required like this:

The next image is what happens when I connect a single noodle, it works with that bare minimum input at least.

So I was just starting small then.. still no clip though., it didn't really listen to clip text encodes at all. I am not sure how to follow this down to the transformer_sd3.py through comfyui's files and imports, I really hope it is in fact using your updated code, because I found a file called dummy_pt_objects.py that has a class SD3Transformer2DModel and I have no idea what that file does.
Here it is again with nothing typed in the positive prompt, and the same image comes out:

Here's when I connect a bunch of them. I feel like it's maybe not using the entire tensor list with how I'm doing it per each joint block? I sort of just added the ones I thought would affect clip.

Some earlier code:
Can probably ignore some of these imports, I was just trying to test if it was somehow using the dummy_pt_objects file and probably importing all of this stuff wrong I'm sure. I may be missing important imports.
import comfy.model_patcher
import comfy.samplers
import torch
import torch.nn.functional as F
# checking to ensure it's not the dummy object
if SD3Transformer2DModel.__module__ == 'diffusers.utils.dummy_pt_objects':
    raise ImportError("SD3Transformer2DModel is a dummy object, please ensure the correct model is available.")

This is a def build_patch_sd3 I first made, I don't know if this was right but tried to make it as right as I could for comfyui's model_patcher.py and the sd3 paper. I tried the original build_patch from repo but it did the same thing with no change. This may be overly complex.
def build_patch_sd3(patchedBlocks, weight=1.0, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None, None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        batch_prompt = n.shape[0] // len(extra_options["cond_or_uncond"])

        if sigma_start <= sigma <= sigma_end:
            if block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']:
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = patchedBlocks[f'{block}:{block_index}'][0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)

                # interpolate to match size if necessary
                if c.size() != b.size():
                    b = F.interpolate(b.unsqueeze(0), size=c.size()[1:], mode='nearest').squeeze(0)

                out = torch.cat((c, b), dim=-1).to(dtype=context_attn1.dtype) * weight
                out = out.repeat(1, batch_prompt, 1, 1) * weight

                return n, out, out

        return n, context_attn1, value_attn1
    return prompt_injection_patch

The class promptinjectionsd3 I added earlier today, which I did a few test input_types might have been wrong. I got the names from clicking on the little layer icon next to the .safetensor on huggingface. I was just testing a few blocks.
class PromptInjectionSD3:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
                "joint_blocks_0_context": ("CONDITIONING",)
            },
            "optional": {
                "joint_blocks_0_context_block_attn_proj_weight": ("CONDITIONING",),
                "joint_blocks_0_context_block_attn_proj_bias": ("CONDITIONING",),
                "joint_blocks_0_context_block_attn_qkv_weight": ("CONDITIONING",),
                "joint_blocks_0_context_block_attn_qkv_bias": ("CONDITIONING",),
                "joint_blocks_1_context_block_attn_proj_weight": ("CONDITIONING",),
                "joint_blocks_1_context_block_attn_proj_bias": ("CONDITIONING",),
                "joint_blocks_1_context_block_attn_qkv_weight": ("CONDITIONING",),
                "joint_blocks_1_context_block_attn_qkv_bias": ("CONDITIONING",),
                "joint_blocks_2_context_block_attn_proj_weight": ("CONDITIONING",),
                "joint_blocks_2_context_block_attn_proj_bias": ("CONDITIONING",),
                "joint_blocks_2_context_block_attn_qkv_weight": ("CONDITIONING",),
                "joint_blocks_2_context_block_attn_qkv_bias": ("CONDITIONING",),
                "weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"
    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, joint_blocks_0_context,
              joint_blocks_0_context_block_attn_proj_weight=None,
              joint_blocks_0_context_block_attn_proj_bias=None,
              joint_blocks_0_context_block_attn_qkv_weight=None,
              joint_blocks_0_context_block_attn_qkv_bias=None,
              joint_blocks_1_context_block_attn_proj_weight=None,
              joint_blocks_1_context_block_attn_proj_bias=None,
              joint_blocks_1_context_block_attn_qkv_weight=None,
              joint_blocks_1_context_block_attn_qkv_bias=None,
              joint_blocks_2_context_block_attn_proj_weight=None,
              joint_blocks_2_context_block_attn_proj_bias=None,
              joint_blocks_2_context_block_attn_qkv_weight=None,
              joint_blocks_2_context_block_attn_qkv_bias=None,
              all=None, weight=1.0, start_at=0.0, end_at=1.0, **kwargs):
        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        # think i'm determining shape of required joint_blocks_0_context.shape to avoid positional argument error. I don't get this model.
        if isinstance(joint_blocks_0_context, list):
            shape = joint_blocks_0_context[0][0].shape
        else:
            shape = joint_blocks_0_context.shape

        # I think added this because I kept getting this error about missing text_projection.weight in comfyui when using the node
        if 'text_projection.weight' not in m.model.state_dict():
            m.model.state_dict()['text_projection.weight'] = torch.nn.Parameter(torch.randn(shape))

        patchedBlocks = {
            'model.diffusion_model.joint_blocks.0.context_block.attn.proj.weight': joint_blocks_0_context_block_attn_proj_weight,
            'model.diffusion_model.joint_blocks.0.context_block.attn.proj.bias': joint_blocks_0_context_block_attn_proj_bias,
            'model.diffusion_model.joint_blocks.0.context_block.attn.qkv.weight': joint_blocks_0_context_block_attn_qkv_weight,
            'model.diffusion_model.joint_blocks.0.context_block.attn.qkv.bias': joint_blocks_0_context_block_attn_qkv_bias,
            'model.diffusion_model.joint_blocks.1.context_block.attn.proj.weight': joint_blocks_1_context_block_attn_proj_weight,
            'model.diffusion_model.joint_blocks.1.context_block.attn.proj.bias': joint_blocks_1_context_block_attn_proj_bias,
            'model.diffusion_model.joint_blocks.1.context_block.attn.qkv.weight': joint_blocks_1_context_block_attn_qkv_weight,
            'model.diffusion_model.joint_blocks.1.context_block.attn.qkv.bias': joint_blocks_1_context_block_attn_qkv_bias,
            'model.diffusion_model.joint_blocks.2.context_block.attn.proj.weight': joint_blocks_2_context_block_attn_proj_weight,
            'model.diffusion_model.joint_blocks.2.context_block.attn.proj.bias': joint_blocks_2_context_block_attn_proj_bias,
            'model.diffusion_model.joint_blocks.2.context_block.attn.qkv.weight': joint_blocks_2_context_block_attn_qkv_weight,
            'model.diffusion_model.joint_blocks.2.context_block.attn.qkv.bias': joint_blocks_2_context_block_attn_qkv_bias,
        }

        for key, value in patchedBlocks.items():
            if value is None:
                patchedBlocks[key] = all
       # Not sure why I'm using patchedBlocks instead of comfyui model_patcher.py, I think I was trying everything.
        prompt_injection_patch = build_patch_sd3(patchedBlocks, weight, sigma_start, sigma_end)
        m.set_model_attn2_patch(prompt_injection_patch)
        return (m,)

My final ditch effort gave me this tall skyscraper and attempted to somehow autofetches all of the tensors and inputs using some google research paper, it still produced the same image of a mountain and river which I feel is a metaphor that I am never going to get across comfyui.

PS. Another random attempt just now:

Is there anyway I can do it like you biswaroop1547 and just bypass Comfyui and use transformers and all of this stuff directly? (If that's what you are doing) I feel I don't know enough here to get past this.

I had a quick look at this but set_model_attn2_patch doesn't simply look to be the right hook for this

Yeah I'm not sure either. Maybe a good enough patch method for SD3 doesn't exist there yet?
I think I'm making progress but not sure. I got the maybe the right state_dict (state dictionary) key names used for the joint blocks now at least.   the comfyui model_detection.py has a bunch of stuff for sd3 under def detect_unet_config like if '{}joint_blocks.0.context_block.attn.qkv.weight'.format(key_prefix) in state_dict_keys: #mmdit model
I still can't get a lot of change in general when adjusting the numbers in the node's box.
The idea of the node here was it's supposed to take the clip conditioning at send it to biswaroop1547's single_layer_indxas per his modified transformer_sd3.py. that seems to target SD3's 24 joint blocks based on whatever number you give it from 0 to 23..
Earlier he recommended 0, 1, 2, 3, 4, 6, 12, 22 which, as an example, for the number 3 would send the conditioning to target to joint block 3 I believe and the n layers there using his script (PS. I don't know the specifics yet if it targets all of those joint block layers at once, or individual ones or if this is how it actually works):
I was able to get the conditioning output to zero from the node which is what I was currently working on, but I think it broke things. It doesn't want to give data to the model here now (but maybe it wasn't doing it before though either) I will change it back tomorrow but here's the screenshot of the zeroing and the code for it where I am leaving off now and going to bed.
Since adding the conditioning zero out it has errors about the keys "not found in model state dict." but I think if you remove the zero out conditioning it should go back to how it was earlier like earlier with the pretty woman in the grass with no torso, but I did also modify the def detect_unet_config pretty heavily.:

PS. I think I finally get what biswaroop1547 meant by "n layers" I guess it means in neural networks the number of hidden layers between the input layer and the output layer. I found out each layer has a certain number of neurons (nodes) that process inputs and pass their output to the next layer. And I think he meant his modifications to transformer_sd3.py target the number of hidden layers between the input layer and the output layers.
My bad and very long broken code apologies:
import torch.nn.functional as F
import torch
import comfy.model_patcher
import comfy.samplers
import logging

class PromptInjectionSD3:
    @classmethod
    def INPUT_TYPES(cls):
        required_inputs = {"model": ("MODEL",)}
        optional_inputs = {
            "conditioning": ("CONDITIONING",),
            "all": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.1}),
            "weight": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 5.0, "step": 0.05}),
            "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            "single_layer_idxs": ("STRING", {"multiline": True}),
        }
        return {"required": required_inputs, "optional": optional_inputs}

    RETURN_TYPES = ("MODEL", "CONDITIONING")
    FUNCTION = "patch"
    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, conditioning=None, all=1.0, single_layer_idxs="", **kwargs):
        m = model.clone()

        try:
            model_obj = m.model  # not sure if right for comfyui
        except AttributeError:
            print("Error: 'model' attribute not found. Recommend go through comfyui files to find right way to do this for SD3")
            return (model, conditioning)

        # debug, print all state dictionary keys
        print("State dictionary keys:")
        for key in model_obj.state_dict().keys():
            print(key)

        single_layer_idxs = [int(idx) for idx in single_layer_idxs.split(",")] if single_layer_idxs else None

        def create_patch(layer_idxs):
            def patch_function(weight):
                for idx, layer in enumerate(weight):
                    if (layer_idxs and idx in layer_idxs) or not layer_idxs:
                        for block, scale in kwargs.items():
                            if block in weight:
                                weight[block] = weight[block] * scale
                return weight
            return patch_function

        m.set_model_patch(create_patch(single_layer_idxs), "block_patch")
        m.patch_model()

        if conditioning is not None:
            cond, pooled = conditioning[0]
            joint_blocks = [
                "diffusion_model.pos_embed",
                "diffusion_model.x_embedder.proj.weight",
                "diffusion_model.x_embedder.proj.bias",
                "diffusion_model.t_embedder.mlp.0.weight",
                "diffusion_model.t_embedder.mlp.0.bias",
                "diffusion_model.t_embedder.mlp.2.weight",
                "diffusion_model.t_embedder.mlp.2.bias",
                "diffusion_model.y_embedder.mlp.0.weight",
                "diffusion_model.y_embedder.mlp.0.bias",
                "diffusion_model.y_embedder.mlp.2.weight",
                "diffusion_model.y_embedder.mlp.2.bias",
                "diffusion_model.context_embedder.weight",
                "diffusion_model.context_embedder.bias",
                "diffusion_model.joint_blocks.0.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.0.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.0.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.0.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.0.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.0.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.0.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.0.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.0.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.0.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.0.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.0.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.0.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.0.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.0.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.0.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.0.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.0.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.0.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.0.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.1.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.1.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.1.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.1.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.1.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.1.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.1.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.1.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.1.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.1.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.1.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.1.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.1.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.1.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.1.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.1.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.1.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.1.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.1.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.1.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.2.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.2.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.2.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.2.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.2.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.2.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.2.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.2.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.2.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.2.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.2.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.2.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.2.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.2.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.2.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.2.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.2.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.2.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.2.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.2.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.3.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.3.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.3.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.3.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.3.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.3.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.3.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.3.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.3.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.3.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.3.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.3.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.3.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.3.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.3.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.3.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.3.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.3.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.3.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.3.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.4.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.4.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.4.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.4.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.4.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.4.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.4.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.4.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.4.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.4.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.4.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.4.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.4.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.4.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.4.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.4.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.4.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.4.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.4.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.4.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.5.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.5.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.5.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.5.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.5.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.5.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.5.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.5.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.5.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.5.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.5.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.5.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.5.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.5.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.5.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.5.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.5.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.5.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.5.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.5.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.6.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.6.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.6.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.6.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.6.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.6.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.6.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.6.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.6.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.6.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.6.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.6.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.6.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.6.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.6.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.6.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.6.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.6.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.6.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.6.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.7.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.7.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.7.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.7.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.7.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.7.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.7.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.7.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.7.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.7.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.7.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.7.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.7.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.7.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.7.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.7.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.7.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.7.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.7.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.7.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.8.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.8.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.8.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.8.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.8.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.8.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.8.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.8.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.8.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.8.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.8.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.8.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.8.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.8.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.8.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.8.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.8.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.8.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.8.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.8.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.9.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.9.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.9.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.9.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.9.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.9.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.9.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.9.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.9.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.9.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.9.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.9.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.9.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.9.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.9.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.9.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.9.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.9.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.9.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.9.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.10.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.10.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.10.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.10.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.10.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.10.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.10.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.10.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.10.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.10.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.10.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.10.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.10.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.10.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.10.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.10.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.10.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.10.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.10.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.10.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.11.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.11.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.11.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.11.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.11.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.11.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.11.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.11.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.11.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.11.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.11.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.11.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.11.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.11.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.11.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.11.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.11.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.11.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.11.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.11.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.12.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.12.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.12.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.12.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.12.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.12.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.12.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.12.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.12.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.12.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.12.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.12.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.12.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.12.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.12.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.12.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.12.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.12.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.12.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.12.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.13.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.13.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.13.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.13.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.13.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.13.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.13.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.13.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.13.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.13.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.13.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.13.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.13.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.13.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.13.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.13.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.13.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.13.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.13.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.13.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.14.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.14.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.14.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.14.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.14.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.14.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.14.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.14.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.14.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.14.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.14.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.14.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.14.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.14.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.14.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.14.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.14.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.14.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.14.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.14.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.15.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.15.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.15.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.15.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.15.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.15.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.15.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.15.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.15.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.15.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.15.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.15.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.15.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.15.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.15.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.15.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.15.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.15.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.15.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.15.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.16.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.16.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.16.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.16.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.16.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.16.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.16.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.16.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.16.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.16.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.16.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.16.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.16.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.16.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.16.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.16.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.16.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.16.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.16.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.16.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.17.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.17.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.17.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.17.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.17.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.17.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.17.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.17.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.17.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.17.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.17.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.17.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.17.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.17.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.17.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.17.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.17.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.17.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.17.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.17.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.18.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.18.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.18.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.18.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.18.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.18.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.18.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.18.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.18.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.18.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.18.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.18.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.18.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.18.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.18.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.18.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.18.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.18.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.18.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.18.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.19.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.19.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.19.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.19.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.19.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.19.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.19.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.19.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.19.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.19.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.19.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.19.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.19.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.19.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.19.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.19.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.19.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.19.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.19.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.19.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.20.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.20.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.20.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.20.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.20.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.20.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.20.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.20.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.20.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.20.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.20.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.20.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.20.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.20.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.20.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.20.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.20.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.20.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.20.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.20.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.21.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.21.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.21.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.21.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.21.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.21.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.21.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.21.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.21.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.21.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.21.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.21.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.21.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.21.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.21.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.21.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.21.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.21.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.21.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.21.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.22.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.22.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.22.context_block.attn.proj.weight",
                "diffusion_model.joint_blocks.22.context_block.attn.proj.bias",
                "diffusion_model.joint_blocks.22.context_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.22.context_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.22.context_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.22.context_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.22.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.22.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.22.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.22.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.22.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.22.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.22.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.22.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.22.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.22.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.22.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.22.x_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.23.context_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.23.context_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.23.context_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.23.context_block.adaLN_modulation.1.bias",
                "diffusion_model.joint_blocks.23.x_block.attn.qkv.weight",
                "diffusion_model.joint_blocks.23.x_block.attn.qkv.bias",
                "diffusion_model.joint_blocks.23.x_block.attn.proj.weight",
                "diffusion_model.joint_blocks.23.x_block.attn.proj.bias",
                "diffusion_model.joint_blocks.23.x_block.mlp.fc1.weight",
                "diffusion_model.joint_blocks.23.x_block.mlp.fc1.bias",
                "diffusion_model.joint_blocks.23.x_block.mlp.fc2.weight",
                "diffusion_model.joint_blocks.23.x_block.mlp.fc2.bias",
                "diffusion_model.joint_blocks.23.x_block.adaLN_modulation.1.weight",
                "diffusion_model.joint_blocks.23.x_block.adaLN_modulation.1.bias",
                "diffusion_model.final_layer.linear.weight",
                "diffusion_model.final_layer.linear.bias",
                "diffusion_model.final_layer.adaLN_modulation.1.weight",
                "diffusion_model.final_layer.adaLN_modulation.1.bias"
            ]

            if hasattr(model_obj, 'diffusion_model'):
                diffusion_model = getattr(model_obj, 'diffusion_model')
                for idx, block_name in enumerate(joint_blocks):
                    if not single_layer_idxs or idx in single_layer_idxs:
                        if block_name in diffusion_model.state_dict():
                            original_weight = diffusion_model.state_dict()[block_name]
                            new_weight = original_weight * kwargs.get('weight', 1.0)
                            diffusion_model.state_dict()[block_name].copy_(new_weight)
                        else:
                            logging.warning(f"Block {block_name} not found in model state dict.")
            else:
                print("Error: 'diffusion_model' attribute not found in the model.")
                return (model, conditioning)

        # zero out conditioning for conditioning output
        if conditioning is not None:
            conditioning = [(torch.zeros_like(cond), torch.zeros_like(pooled)) for cond, pooled in conditioning if isinstance(cond, torch.Tensor) and isinstance(pooled, torch.Tensor)]

        m.patch_model(device_to=None, patch_weights=True)

        return (m, conditioning)

NODE_CLASS_MAPPINGS = {
    "PromptInjectionSD3": PromptInjectionSD3,
}

NODE_DISPLAY_NAME_MAPPINGS = {
    "PromptInjectionSD3": "Attn2 Prompt Injection SD3",
}

Please feel free to use any of this if it might be of use, and keep me updated! GN

So I deleted the transfomer_sd3.py and sd3 still works in comfyui hahah. So it's not even used in venv unless I'm using diffusers loader probably. But I did figure this out actually. Will post as soon as I iron out the bugs here.

Hello again. Here is a node I'm working on for SD3. It doesn't work fully yet. I don't know if the images are even injecting right or the clip tbh, or if it's just corrupting layers.
No idea why she's in a bikini btw when I corrupt layers. I did not ask for that lol. I'm using a batch of 9 real woman in grass and injecting those images with torch.stack or torch.cat to weight and bias layers using squeeze and unsqueeze to match the size. I don't know what I'm doing. As you can see it's completely fried but I guess I would prefer this over the alternative.
Are you making any progress with SD3? The idea of this node was to be a swiss army knife of sorts. Will create repo for it when it's not terrible.

Also, not sure if this is useful to you, didn't feel like forking and submitting pull request. I modified the class projectinjection so you can do per layer weight adjustments with clip text encode built into the node for each layer. Similar to your advancedprompt injection class but a little easier to use, but still needs a some work.
Think I turned off too many layers and not greatest quality, but you get the idea of node modifications I'm proposing for maybe an additional class:

The only issues I'm having (hope you can help) is for some reason all_text box is broken and does not actually fill in all of the text boxes like when you do them all manually with the same prompt. And I would to have it where all_text it still respects the per layers weight adjustments as seen. I'm sort of stumped because any changes I make to all_text breaks everything.
Also if I do a batch of more than 13 images it doesn't work  and shows distortion for all the images for some reason, still can't figure that out.
I have been using this all week and it's been pretty convenient and fast just pasting into the boxes to find stuff! I really like the quick fine adjustments of the strength per layer also. Let me know what you think!
import comfy.model_patcher
import comfy.samplers
import torch
import torch.nn.functional as F
from nodes import CLIPTextEncode

class PromptInjection:
    @classmethod
    def INPUT_TYPES(s):
        return {
            "required": {
                "model": ("MODEL",),
                "clip": ("CLIP",)
            },
            "optional": {
                "all_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "input_4_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "input_4_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "input_5_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "input_5_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "input_7_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "input_7_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "input_8_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "input_8_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "middle_0_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "middle_0_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "output_0_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "output_0_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "output_1_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "output_1_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "output_2_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "output_2_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "output_3_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "output_3_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "output_4_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "output_4_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "output_5_text": ("STRING", {"multiline": True, "dynamicPrompts": True}),
                "output_5_weight": ("FLOAT", {"default": 1.0, "min": -2.0, "max": 5.0, "step": 0.05}),
                "start_at": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
                "end_at": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.001}),
            }
        }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "patch"

    CATEGORY = "advanced/model"

    def patch(self, model: comfy.model_patcher.ModelPatcher, clip, all_text=None, input_4_text=None, input_4_weight=1.0, input_5_text=None, input_5_weight=1.0, input_7_text=None, input_7_weight=1.0, input_8_text=None, input_8_weight=1.0, middle_0_text=None, middle_0_weight=1.0, output_0_text=None, output_0_weight=1.0, output_1_text=None, output_1_weight=1.0, output_2_text=None, output_2_weight=1.0, output_3_text=None, output_3_weight=1.0, output_4_text=None, output_4_weight=1.0, output_5_text=None, output_5_weight=1.0, weight=1.0, start_at=0.0, end_at=1.0):
        if not any((all_text, input_4_text, input_5_text, input_7_text, input_8_text, middle_0_text, output_0_text, output_1_text, output_2_text, output_3_text, output_4_text, output_5_text)):
            return (model,)

        def encode_text(clip, text):
            encoder = CLIPTextEncode()
            return encoder.encode(clip, text)

        m = model.clone()
        sigma_start = m.get_model_object("model_sampling").percent_to_sigma(start_at)
        sigma_end = m.get_model_object("model_sampling").percent_to_sigma(end_at)

        patchedBlocks = {}

        def add_patch(block, index, text, weight):
            if text is not None:
                conditioning = encode_text(clip, text)[0]
                patchedBlocks[f"{block}:{index}"] = (conditioning, weight)

        if all_text is not None:
            for block in ['input', 'middle', 'output']:
                for index in range(9):
                    add_patch(block, index, all_text, weight)

        add_patch('input', 4, input_4_text, input_4_weight)
        add_patch('input', 5, input_5_text, input_5_weight)
        add_patch('input', 7, input_7_text, input_7_weight)
        add_patch('input', 8, input_8_text, input_8_weight)
        add_patch('middle', 0, middle_0_text, middle_0_weight)
        add_patch('output', 0, output_0_text, output_0_weight)
        add_patch('output', 1, output_1_text, output_1_weight)
        add_patch('output', 2, output_2_text, output_2_weight)
        add_patch('output', 3, output_3_text, output_3_weight)
        add_patch('output', 4, output_4_text, output_4_weight)
        add_patch('output', 5, output_5_text, output_5_weight)

        m.set_model_attn2_patch(build_patch(patchedBlocks, sigma_start=sigma_start, sigma_end=sigma_end))

        return (m,)

def build_patch(patchedBlocks, sigma_start=0.0, sigma_end=1.0):
    def prompt_injection_patch(n, context_attn1: torch.Tensor, value_attn1, extra_options):
        (block, block_index) = extra_options.get('block', (None, None))
        sigma = extra_options["sigmas"].detach().cpu()[0].item() if 'sigmas' in extra_options else 999999999.9

        batch_prompt = n.shape[0] // len(extra_options["cond_or_uncond"])

        if sigma <= sigma_start and sigma >= sigma_end:
            if (block and f'{block}:{block_index}' in patchedBlocks and patchedBlocks[f'{block}:{block_index}']):
                conditioning, weight = patchedBlocks[f'{block}:{block_index}']
                if context_attn1.dim() == 3:
                    c = context_attn1[0].unsqueeze(0)
                else:
                    c = context_attn1[0][0].unsqueeze(0)
                b = conditioning[0][0].repeat(c.shape[0], 1, 1).to(context_attn1.device)
                out = torch.stack((c, b)).to(dtype=context_attn1.dtype) * weight
                out = out.repeat(1, batch_prompt, 1, 1) * weight

                return n, out, out

        return n, context_attn1, value_attn1
    return prompt_injection_patch

NODE_CLASS_MAPPINGS = {
    "PromptInjection": PromptInjection
}

NODE_DISPLAY_NAME_MAPPINGS = {
    "PromptInjection": "Attn2 Prompt Injection"
}