Stage2 RuntimeError: The size of tensor a (22) must match the size of tensor b (23) at non-singleton dimension 3

~/# accelerate launch train_stage_2.py --config configs/train/stage2.yaml
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 1
--num_machines was set to a value of 1
--mixed_precision was set to a value of 'no'
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
10/12/2024 09:16:18 - INFO - main - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda
Mixed precision type: fp16
{'scaling_factor', 'force_upcast'} was not found in config. Values will be initialized to default values.
{'addition_time_embed_dim', 'time_embedding_type', 'num_class_embeds', 'encoder_hid_dim', 'encoder_hid_dim_type', 'addition_embed_type_num_heads', 'addition_embed_type', 'dual_cross_attention', 'dropout', 'resnet_out_scale_factor', 'attention_type', 'reverse_transformer_layers_per_block', 'projection_class_embeddings_input_dim', 'mid_block_type', 'conv_out_kernel', 'resnet_skip_time_act', 'use_linear_projection', 'class_embeddings_concat', 'time_embedding_dim', 'timestep_post_act', 'resnet_time_scale_shift', 'only_cross_attention', 'transformer_layers_per_block', 'class_embed_type', 'conv_in_kernel', 'time_cond_proj_dim', 'time_embedding_act_fn', 'mid_block_only_cross_attention', 'num_attention_heads', 'upcast_attention', 'cross_attention_norm'} was not found in config. Values will be initialized to default values.
Some weights of the model checkpoint were not used when initializing UNet2DConditionModel:
['conv_norm_out.weight, conv_norm_out.bias, conv_out.weight, conv_out.bias']
10/12/2024 09:16:25 - INFO - src.models.unet_3d - loaded temporal unet's pretrained weights from pretrained_weights/stable-diffusion-v1-5/unet ...
{'dual_cross_attention', 'use_linear_projection', 'num_class_embeds', 'upcast_attention', 'mode', 'task_type', 'resnet_time_scale_shift', 'only_cross_attention', 'class_embed_type'} was not found in config. Values will be initialized to default values.
10/12/2024 09:16:38 - INFO - src.models.unet_3d - Load motion module params from pretrained_weights/mm_sd_v15_v2.ckpt
10/12/2024 09:16:39 - INFO - src.models.unet_3d - Loaded 453.20928M-parameter motion module
10/12/2024 09:16:44 - INFO - main - Total trainable params 546
10/12/2024 09:16:45 - INFO - main - ***** Running training *****
10/12/2024 09:16:45 - INFO - main -   Num examples = 7755
10/12/2024 09:16:45 - INFO - main -   Num Epochs = 2
10/12/2024 09:16:45 - INFO - main -   Instantaneous batch size per device = 1
10/12/2024 09:16:45 - INFO - main -   Total train batch size (w. parallel, distributed & accumulation) = 1
10/12/2024 09:16:45 - INFO - main -   Gradient Accumulation steps = 1
10/12/2024 09:16:45 - INFO - main -   Total optimization steps = 10000
Steps:   0%|                                                                                | 0/10000 [00:00<?, ?it/s]10/12/2024 09:16:50 - INFO - src.models.unet_3d - Forward upsample size to force interpolation output size.
Traceback (most recent call last):
File "/root/MusePose/train_stage_2.py", line 773, in
main(config)
File "/root/MusePose/train_stage_2.py", line 602, in main
model_pred = net(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/accelerate/utils/operations.py", line 825, in forward
return model_forward(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/accelerate/utils/operations.py", line 813, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/root/miniconda3/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/root/MusePose/train_stage_2.py", line 96, in forward
model_pred = self.denoising_unet(
File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(args, **kwargs)
File "/root//src/models/unet_3d.py", line 505, in forward
sample = sample + pose_cond_fea
RuntimeError: The size of tensor a (22) must match the size of tensor b (23) at non-singleton dimension 3
Steps:   0%|                                                                                | 0/10000 [00:06<?, ?it/s]
Traceback (most recent call last):
File "/root/miniconda3/bin/accelerate", line 8, in
sys.exit(main())
File "/root/miniconda3/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
args.func(args)
File "/root/miniconda3/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1075, in launch_command
simple_launcher(args)
File "/root/miniconda3/lib/python3.10/site-packages/accelerate/commands/launch.py", line 681, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/root/miniconda3/bin/python', 'train_stage_2.py', '--config', 'configs/train/stage2.yaml']' returned non-zero exit status 1.

After a few hours of hard work, if anyone has the same problem as me, just replace 'MusePose/src/models/unet_3d.py' with the following:

Adapted from https://github.com/guoyww/AnimateDiff/blob/main/animatediff/models/unet_blocks.py
from collections import OrderedDict
from dataclasses import dataclass
from os import PathLike
from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.checkpoint
from diffusers.configuration_utils import ConfigMixin, register_to_config
from diffusers.models.attention_processor import AttentionProcessor
from diffusers.models.embeddings import TimestepEmbedding, Timesteps
try:
from diffusers.modeling_utils import ModelMixin
except:
from diffusers.models.modeling_utils import ModelMixin
from diffusers.utils import SAFETENSORS_WEIGHTS_NAME, WEIGHTS_NAME, BaseOutput, logging
from safetensors.torch import load_file
from .resnet import InflatedConv3d, InflatedGroupNorm
from .unet_3d_blocks import UNetMidBlock3DCrossAttn, get_down_block, get_up_block
logger = logging.get_logger(name)  # pylint: disable=invalid-name
@dataclass
class UNet3DConditionOutput(BaseOutput):
sample: torch.FloatTensor
class UNet3DConditionModel(ModelMixin, ConfigMixin):
_supports_gradient_checkpointing = True
@register_to_config
def __init__(
    self,
    sample_size: Optional[int] = None,
    in_channels: int = 4,
    out_channels: int = 4,
    center_input_sample: bool = False,
    flip_sin_to_cos: bool = True,
    freq_shift: int = 0,
    down_block_types: Tuple[str] = (
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "DownBlock3D",
    ),
    mid_block_type: str = "UNetMidBlock3DCrossAttn",
    up_block_types: Tuple[str] = (
        "UpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
    ),
    only_cross_attention: Union[bool, Tuple[bool]] = False,
    block_out_channels: Tuple[int] = (320, 640, 1280, 1280),
    layers_per_block: int = 2,
    downsample_padding: int = 1,
    mid_block_scale_factor: float = 1,
    act_fn: str = "silu",
    norm_num_groups: int = 32,
    norm_eps: float = 1e-5,
    cross_attention_dim: int = 1280,
    attention_head_dim: Union[int, Tuple[int]] = 8,
    dual_cross_attention: bool = False,
    use_linear_projection: bool = False,
    class_embed_type: Optional[str] = None,
    num_class_embeds: Optional[int] = None,
    upcast_attention: bool = False,
    resnet_time_scale_shift: str = "default",
    use_inflated_groupnorm=False,
    # Additional
    use_motion_module=False,
    motion_module_resolutions=(1, 2, 4, 8),
    motion_module_mid_block=False,
    motion_module_decoder_only=False,
    motion_module_type=None,
    motion_module_kwargs={},
    unet_use_cross_frame_attention=None,
    unet_use_temporal_attention=None,
    mode=None,
    task_type="action",
):
    super().__init__()

    self.sample_size = sample_size
    time_embed_dim = block_out_channels[0] * 4

    # Add this line in the __init__ method
    self.pose_projector = nn.Conv3d(4, 320, kernel_size=1)

    # input
    self.conv_in = InflatedConv3d(
        in_channels, block_out_channels[0], kernel_size=3, padding=(1, 1)
    )

    # time
    self.time_proj = Timesteps(block_out_channels[0], flip_sin_to_cos, freq_shift)
    timestep_input_dim = block_out_channels[0]

    self.time_embedding = TimestepEmbedding(timestep_input_dim, time_embed_dim)

    # class embedding
    if class_embed_type is None and num_class_embeds is not None:
        self.class_embedding = nn.Embedding(num_class_embeds, time_embed_dim)
    elif class_embed_type == "timestep":
        self.class_embedding = TimestepEmbedding(timestep_input_dim, time_embed_dim)
    elif class_embed_type == "identity":
        self.class_embedding = nn.Identity(time_embed_dim, time_embed_dim)
    else:
        self.class_embedding = None

    self.down_blocks = nn.ModuleList([])
    self.mid_block = None
    self.up_blocks = nn.ModuleList([])

    if isinstance(only_cross_attention, bool):
        only_cross_attention = [only_cross_attention] * len(down_block_types)

    if isinstance(attention_head_dim, int):
        attention_head_dim = (attention_head_dim,) * len(down_block_types)

    # down
    output_channel = block_out_channels[0]
    for i, down_block_type in enumerate(down_block_types):
        if task_type == "action":
            name_index, mid_name = None, None
        else:
            name_index, mid_name = i, "MidBlock"
        res = 2**i
        input_channel = output_channel
        output_channel = block_out_channels[i]
        is_final_block = i == len(block_out_channels) - 1

        down_block = get_down_block(
            down_block_type,
            num_layers=layers_per_block,
            in_channels=input_channel,
            out_channels=output_channel,
            temb_channels=time_embed_dim,
            add_downsample=not is_final_block,
            resnet_eps=norm_eps,
            resnet_act_fn=act_fn,
            resnet_groups=norm_num_groups,
            cross_attention_dim=cross_attention_dim,
            attn_num_head_channels=attention_head_dim[i],
            downsample_padding=downsample_padding,
            dual_cross_attention=dual_cross_attention,
            use_linear_projection=use_linear_projection,
            only_cross_attention=only_cross_attention[i],
            upcast_attention=upcast_attention,
            resnet_time_scale_shift=resnet_time_scale_shift,
            unet_use_cross_frame_attention=unet_use_cross_frame_attention,
            unet_use_temporal_attention=unet_use_temporal_attention,
            use_inflated_groupnorm=use_inflated_groupnorm,
            use_motion_module=use_motion_module
            and (res in motion_module_resolutions)
            and (not motion_module_decoder_only),
            motion_module_type=motion_module_type,
            motion_module_kwargs=motion_module_kwargs,
            name_index=name_index,
        )
        self.down_blocks.append(down_block)

    # mid

    if mid_block_type == "UNetMidBlock3DCrossAttn":
        self.mid_block = UNetMidBlock3DCrossAttn(
            in_channels=block_out_channels[-1],
            temb_channels=time_embed_dim,
            resnet_eps=norm_eps,
            resnet_act_fn=act_fn,
            output_scale_factor=mid_block_scale_factor,
            resnet_time_scale_shift=resnet_time_scale_shift,
            cross_attention_dim=cross_attention_dim,
            attn_num_head_channels=attention_head_dim[-1],
            resnet_groups=norm_num_groups,
            dual_cross_attention=dual_cross_attention,
            use_linear_projection=use_linear_projection,
            upcast_attention=upcast_attention,
            unet_use_cross_frame_attention=unet_use_cross_frame_attention,
            unet_use_temporal_attention=unet_use_temporal_attention,
            use_inflated_groupnorm=use_inflated_groupnorm,
            use_motion_module=use_motion_module and motion_module_mid_block,
            motion_module_type=motion_module_type,
            motion_module_kwargs=motion_module_kwargs,
            name=mid_name,
        )
    else:
        raise ValueError(f"unknown mid_block_type : {mid_block_type}")

    # count how many layers upsample the videos
    self.num_upsamplers = 0

    # up
    reversed_block_out_channels = list(reversed(block_out_channels))
    reversed_attention_head_dim = list(reversed(attention_head_dim))
    only_cross_attention = list(reversed(only_cross_attention))
    output_channel = reversed_block_out_channels[0]
    for i, up_block_type in enumerate(up_block_types):
        res = 2 ** (3 - i)
        is_final_block = i == len(block_out_channels) - 1

        if task_type == "action":
            name_index = None
        else:
            name_index = i

        prev_output_channel = output_channel
        output_channel = reversed_block_out_channels[i]
        input_channel = reversed_block_out_channels[
            min(i + 1, len(block_out_channels) - 1)
        ]

        # add upsample block for all BUT final layer
        if not is_final_block:
            add_upsample = True
            self.num_upsamplers += 1
        else:
            add_upsample = False

        up_block = get_up_block(
            up_block_type,
            num_layers=layers_per_block + 1,
            in_channels=input_channel,
            out_channels=output_channel,
            prev_output_channel=prev_output_channel,
            temb_channels=time_embed_dim,
            add_upsample=add_upsample,
            resnet_eps=norm_eps,
            resnet_act_fn=act_fn,
            resnet_groups=norm_num_groups,
            cross_attention_dim=cross_attention_dim,
            attn_num_head_channels=reversed_attention_head_dim[i],
            dual_cross_attention=dual_cross_attention,
            use_linear_projection=use_linear_projection,
            only_cross_attention=only_cross_attention[i],
            upcast_attention=upcast_attention,
            resnet_time_scale_shift=resnet_time_scale_shift,
            unet_use_cross_frame_attention=unet_use_cross_frame_attention,
            unet_use_temporal_attention=unet_use_temporal_attention,
            use_inflated_groupnorm=use_inflated_groupnorm,
            use_motion_module=use_motion_module
            and (res in motion_module_resolutions),
            motion_module_type=motion_module_type,
            motion_module_kwargs=motion_module_kwargs,
            name_index=name_index,
        )
        self.up_blocks.append(up_block)
        prev_output_channel = output_channel

    # out
    if use_inflated_groupnorm:
        self.conv_norm_out = InflatedGroupNorm(
            num_channels=block_out_channels[0],
            num_groups=norm_num_groups,
            eps=norm_eps,
        )
    else:
        self.conv_norm_out = nn.GroupNorm(
            num_channels=block_out_channels[0],
            num_groups=norm_num_groups,
            eps=norm_eps,
        )
    self.conv_act = nn.SiLU()
    self.conv_out = InflatedConv3d(
        block_out_channels[0], out_channels, kernel_size=3, padding=1
    )

    self.mode = mode

@property
# Copied from diffusers.models.unet_2d_condition.UNet2DConditionModel.attn_processors
def attn_processors(self) -> Dict[str, AttentionProcessor]:
    r"""
    Returns:
        `dict` of attention processors: A dictionary containing all attention processors used in the model with
        indexed by its weight name.
    """
    # set recursively
    processors = {}

    def fn_recursive_add_processors(
        name: str,
        module: torch.nn.Module,
        processors: Dict[str, AttentionProcessor],
    ):
        if hasattr(module, "set_processor"):
            processors[f"{name}.processor"] = module.processor

        for sub_name, child in module.named_children():
            if "temporal_transformer" not in sub_name:
                fn_recursive_add_processors(f"{name}.{sub_name}", child, processors)

        return processors

    for name, module in self.named_children():
        if "temporal_transformer" not in name:
            fn_recursive_add_processors(name, module, processors)

    return processors

def set_attention_slice(self, slice_size):
    r"""
    Enable sliced attention computation.

    When this option is enabled, the attention module will split the input tensor in slices, to compute attention
    in several steps. This is useful to save some memory in exchange for a small speed decrease.

    Args:
        slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`):
            When `"auto"`, halves the input to the attention heads, so attention will be computed in two steps. If
            `"max"`, maxium amount of memory will be saved by running only one slice at a time. If a number is
            provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim`
            must be a multiple of `slice_size`.
    """
    sliceable_head_dims = []

    def fn_recursive_retrieve_slicable_dims(module: torch.nn.Module):
        if hasattr(module, "set_attention_slice"):
            sliceable_head_dims.append(module.sliceable_head_dim)

        for child in module.children():
            fn_recursive_retrieve_slicable_dims(child)

    # retrieve number of attention layers
    for module in self.children():
        fn_recursive_retrieve_slicable_dims(module)

    num_slicable_layers = len(sliceable_head_dims)

    if slice_size == "auto":
        # half the attention head size is usually a good trade-off between
        # speed and memory
        slice_size = [dim // 2 for dim in sliceable_head_dims]
    elif slice_size == "max":
        # make smallest slice possible
        slice_size = num_slicable_layers * [1]

    slice_size = (
        num_slicable_layers * [slice_size]
        if not isinstance(slice_size, list)
        else slice_size
    )

    if len(slice_size) != len(sliceable_head_dims):
        raise ValueError(
            f"You have provided {len(slice_size)}, but {self.config} has {len(sliceable_head_dims)} different"
            f" attention layers. Make sure to match `len(slice_size)` to be {len(sliceable_head_dims)}."
        )

    for i in range(len(slice_size)):
        size = slice_size[i]
        dim = sliceable_head_dims[i]
        if size is not None and size > dim:
            raise ValueError(f"size {size} has to be smaller or equal to {dim}.")

    # Recursively walk through all the children.
    # Any children which exposes the set_attention_slice method
    # gets the message
    def fn_recursive_set_attention_slice(
        module: torch.nn.Module, slice_size: List[int]
    ):
        if hasattr(module, "set_attention_slice"):
            module.set_attention_slice(slice_size.pop())

        for child in module.children():
            fn_recursive_set_attention_slice(child, slice_size)

    reversed_slice_size = list(reversed(slice_size))
    for module in self.children():
        fn_recursive_set_attention_slice(module, reversed_slice_size)

def _set_gradient_checkpointing(self, module, value=False):
    if hasattr(module, "gradient_checkpointing"):
        module.gradient_checkpointing = value

# Copied from diffusers.models.unet_2d_condition.UNet2DConditionModel.set_attn_processor
def set_attn_processor(
    self, processor: Union[AttentionProcessor, Dict[str, AttentionProcessor]]
):
    r"""
    Sets the attention processor to use to compute attention.

    Parameters:
        processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`):
            The instantiated processor class or a dictionary of processor classes that will be set as the processor
            for **all** `Attention` layers.

            If `processor` is a dict, the key needs to define the path to the corresponding cross attention
            processor. This is strongly recommended when setting trainable attention processors.

    """
    count = len(self.attn_processors.keys())

    if isinstance(processor, dict) and len(processor) != count:
        raise ValueError(
            f"A dict of processors was passed, but the number of processors {len(processor)} does not match the"
            f" number of attention layers: {count}. Please make sure to pass {count} processor classes."
        )

    def fn_recursive_attn_processor(name: str, module: torch.nn.Module, processor):
        if hasattr(module, "set_processor"):
            if not isinstance(processor, dict):
                module.set_processor(processor)
            else:
                module.set_processor(processor.pop(f"{name}.processor"))

        for sub_name, child in module.named_children():
            if "temporal_transformer" not in sub_name:
                fn_recursive_attn_processor(f"{name}.{sub_name}", child, processor)

    for name, module in self.named_children():
        if "temporal_transformer" not in name:
            fn_recursive_attn_processor(name, module, processor)

def forward(
    self,
    sample: torch.FloatTensor,
    timestep: Union[torch.Tensor, float, int],
    encoder_hidden_states: torch.Tensor,
    class_labels: Optional[torch.Tensor] = None,
    pose_cond_fea: Optional[torch.Tensor] = None,
    attention_mask: Optional[torch.Tensor] = None,
    down_block_additional_residuals: Optional[Tuple[torch.Tensor]] = None,
    mid_block_additional_residual: Optional[torch.Tensor] = None,
    return_dict: bool = True,
    self_attention_additional_feats = None,
) -> Union[UNet3DConditionOutput, Tuple]:
    r"""
    Args:
        sample (`torch.FloatTensor`): (batch, channel, height, width) noisy inputs tensor
        timestep (`torch.FloatTensor` or `float` or `int`): (batch) timesteps
        encoder_hidden_states (`torch.FloatTensor`): (batch, sequence_length, feature_dim) encoder hidden states
        return_dict (`bool`, *optional*, defaults to `True`):
            Whether or not to return a [`models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple.

    Returns:
        [`~models.unet_2d_condition.UNet2DConditionOutput`] or `tuple`:
        [`~models.unet_2d_condition.UNet2DConditionOutput`] if `return_dict` is True, otherwise a `tuple`. When
        returning a tuple, the first element is the sample tensor.
    """
    # By default samples have to be AT least a multiple of the overall upsampling factor.
    # The overall upsampling factor is equal to 2 ** (# num of upsampling layears).
    # However, the upsampling interpolation output size can be forced to fit any upsampling size
    # on the fly if necessary.
    default_overall_up_factor = 2**self.num_upsamplers

    # upsample size should be forwarded when sample is not a multiple of `default_overall_up_factor`
    forward_upsample_size = False
    upsample_size = None

    if any(s % default_overall_up_factor != 0 for s in sample.shape[-2:]):
        logger.info("Forward upsample size to force interpolation output size.")
        forward_upsample_size = True

    # prepare attention_mask
    if attention_mask is not None:
        attention_mask = (1 - attention_mask.to(sample.dtype)) * -10000.0
        attention_mask = attention_mask.unsqueeze(1)

    # center input if necessary
    if self.config.center_input_sample:
        sample = 2 * sample - 1.0

    # time
    timesteps = timestep
    if not torch.is_tensor(timesteps):
        # This would be a good case for the `match` statement (Python 3.10+)
        is_mps = sample.device.type == "mps"
        if isinstance(timestep, float):
            dtype = torch.float32 if is_mps else torch.float64
        else:
            dtype = torch.int32 if is_mps else torch.int64
        timesteps = torch.tensor([timesteps], dtype=dtype, device=sample.device)
    elif len(timesteps.shape) == 0:
        timesteps = timesteps[None].to(sample.device)

    # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
    timesteps = timesteps.expand(sample.shape[0])

    t_emb = self.time_proj(timesteps)

    # timesteps does not contain any weights and will always return f32 tensors
    # but time_embedding might actually be running in fp16. so we need to cast here.
    # there might be better ways to encapsulate this.
    t_emb = t_emb.to(dtype=self.dtype)
    emb = self.time_embedding(t_emb)

    if self.class_embedding is not None:
        if class_labels is None:
            raise ValueError(
                "class_labels should be provided when num_class_embeds > 0"
            )

        if self.config.class_embed_type == "timestep":
            class_labels = self.time_proj(class_labels)

        class_emb = self.class_embedding(class_labels).to(dtype=self.dtype)
        emb = emb + class_emb

    # 1. Check the shapes of both tensors
    print(f"sample shape: {sample.shape}")
    print(f"pose_cond_fea shape: {pose_cond_fea.shape}")

    # pre-process
    #sample = self.conv_in(sample)
    #if pose_cond_fea is not None:
    #    sample = sample + pose_cond_fea

    # pre-process
    if sample is not None:
        # Store original height of sample
        original_height = sample.shape[3]

        # Project sample to match pose_cond_fea's channel dimension
        sample = self.pose_projector(sample)

        # Adjust height of pose_cond_fea to match original sample height
        pose_cond_fea = F.interpolate(pose_cond_fea,
                                      size=(pose_cond_fea.shape[2], original_height, pose_cond_fea.shape[4]),
                                      mode='trilinear',
                                      align_corners=False)

        # Add debug prints
        print(f"Adjusted sample shape: {sample.shape}")
        print(f"Adjusted pose_cond_fea shape: {pose_cond_fea.shape}")

        # Ensure the shapes match before adding
        assert pose_cond_fea.shape == sample.shape, "Shapes still don't match after adjustment"

        sample = sample + pose_cond_fea

    # down
    down_block_res_samples = (sample,)
    for downsample_block in self.down_blocks:
        if (
            hasattr(downsample_block, "has_cross_attention")
            and downsample_block.has_cross_attention
        ):
            sample, res_samples = downsample_block(
                hidden_states=sample,
                temb=emb,
                encoder_hidden_states=encoder_hidden_states,
                attention_mask=attention_mask,
                self_attention_additional_feats=self_attention_additional_feats,
                mode=self.mode,
            )
        else:
            sample, res_samples = downsample_block(
                hidden_states=sample,
                temb=emb,
                encoder_hidden_states=encoder_hidden_states,
            )

        down_block_res_samples += res_samples

    if down_block_additional_residuals is not None:
        new_down_block_res_samples = ()

        for down_block_res_sample, down_block_additional_residual in zip(
            down_block_res_samples, down_block_additional_residuals
        ):
            down_block_res_sample = (
                down_block_res_sample + down_block_additional_residual
            )
            new_down_block_res_samples += (down_block_res_sample,)

        down_block_res_samples = new_down_block_res_samples

    # mid
    sample = self.mid_block(
        sample,
        emb,
        encoder_hidden_states=encoder_hidden_states,
        attention_mask=attention_mask,
        self_attention_additional_feats=self_attention_additional_feats,
        mode=self.mode,
    )

    if mid_block_additional_residual is not None:
        sample = sample + mid_block_additional_residual

    # up
    for i, upsample_block in enumerate(self.up_blocks):
        is_final_block = i == len(self.up_blocks) - 1

        res_samples = down_block_res_samples[-len(upsample_block.resnets) :]
        down_block_res_samples = down_block_res_samples[
            : -len(upsample_block.resnets)
        ]

        # if we have not reached the final block and need to forward the
        # upsample size, we do it here
        if not is_final_block and forward_upsample_size:
            upsample_size = down_block_res_samples[-1].shape[2:]

        if (
            hasattr(upsample_block, "has_cross_attention")
            and upsample_block.has_cross_attention
        ):
            sample = upsample_block(
                hidden_states=sample,
                temb=emb,
                res_hidden_states_tuple=res_samples,
                encoder_hidden_states=encoder_hidden_states,
                upsample_size=upsample_size,
                attention_mask=attention_mask,
                self_attention_additional_feats=self_attention_additional_feats,
                mode=self.mode,
            )
        else:
            sample = upsample_block(
                hidden_states=sample,
                temb=emb,
                res_hidden_states_tuple=res_samples,
                upsample_size=upsample_size,
                encoder_hidden_states=encoder_hidden_states,
            )

    # post-process
    sample = self.conv_norm_out(sample)
    sample = self.conv_act(sample)
    sample = self.conv_out(sample)

    if not return_dict:
        return (sample,)

    return UNet3DConditionOutput(sample=sample)

@classmethod
def from_pretrained_2d(
    cls,
    pretrained_model_path: PathLike,
    motion_module_path: PathLike,
    subfolder=None,
    unet_additional_kwargs=None,
    mm_zero_proj_out=False,
):
    pretrained_model_path = Path(pretrained_model_path)
    motion_module_path = Path(motion_module_path)
    if subfolder is not None:
        pretrained_model_path = pretrained_model_path.joinpath(subfolder)
    logger.info(
        f"loaded temporal unet's pretrained weights from {pretrained_model_path} ..."
    )

    config_file = pretrained_model_path / "config.json"
    if not (config_file.exists() and config_file.is_file()):
        raise RuntimeError(f"{config_file} does not exist or is not a file")

    unet_config = cls.load_config(config_file)
    unet_config["_class_name"] = cls.__name__
    unet_config["down_block_types"] = [
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "DownBlock3D",
    ]
    unet_config["up_block_types"] = [
        "UpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
    ]
    unet_config["mid_block_type"] = "UNetMidBlock3DCrossAttn"

    model = cls.from_config(unet_config, **unet_additional_kwargs)
    # load the vanilla weights
    if pretrained_model_path.joinpath(SAFETENSORS_WEIGHTS_NAME).exists():
        logger.debug(
            f"loading safeTensors weights from {pretrained_model_path} ..."
        )
        state_dict = load_file(
            pretrained_model_path.joinpath(SAFETENSORS_WEIGHTS_NAME), device="cpu"
        )

    elif pretrained_model_path.joinpath(WEIGHTS_NAME).exists():
        logger.debug(f"loading weights from {pretrained_model_path} ...")
        state_dict = torch.load(
            pretrained_model_path.joinpath(WEIGHTS_NAME),
            map_location="cpu",
            weights_only=True,
        )
    else:
        raise FileNotFoundError(f"no weights file found in {pretrained_model_path}")

    # load the motion module weights
    if motion_module_path.exists() and motion_module_path.is_file():
        if motion_module_path.suffix.lower() in [".pth", ".pt", ".ckpt"]:
            logger.info(f"Load motion module params from {motion_module_path}")
            motion_state_dict = torch.load(
                motion_module_path, map_location="cpu", weights_only=True
            )
        elif motion_module_path.suffix.lower() == ".safetensors":
            motion_state_dict = load_file(motion_module_path, device="cpu")
        else:
            raise RuntimeError(
                f"unknown file format for motion module weights: {motion_module_path.suffix}"
            )
        if mm_zero_proj_out:
            logger.info(f"Zero initialize proj_out layers in motion module...")
            new_motion_state_dict = OrderedDict()
            for k in motion_state_dict:
                if "proj_out" in k:
                    continue
                new_motion_state_dict[k] = motion_state_dict[k]
            motion_state_dict = new_motion_state_dict

        # merge the state dicts
        state_dict.update(motion_state_dict)

    # load the weights into the model
    m, u = model.load_state_dict(state_dict, strict=False)
    logger.debug(f"### missing keys: {len(m)}; \n### unexpected keys: {len(u)};")

    params = [
        p.numel() if "temporal" in n else 0 for n, p in model.named_parameters()
    ]
    logger.info(f"Loaded {sum(params) / 1e6}M-parameter motion module")

    return model

After a few hours of hard work, if anyone has the same problem as me, just replace 'MusePose/src/models/unet_3d.py' with the following:

Adapted from https://github.com/guoyww/AnimateDiff/blob/main/animatediff/models/unet_blocks.py
from collections import OrderedDict
from dataclasses import dataclass
from os import PathLike
from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.checkpoint
from diffusers.configuration_utils import ConfigMixin, register_to_config
from diffusers.models.attention_processor import AttentionProcessor
from diffusers.models.embeddings import TimestepEmbedding, Timesteps
try:
from diffusers.modeling_utils import ModelMixin
except:
from diffusers.models.modeling_utils import ModelMixin
from diffusers.utils import SAFETENSORS_WEIGHTS_NAME, WEIGHTS_NAME, BaseOutput, logging
from safetensors.torch import load_file
from .resnet import InflatedConv3d, InflatedGroupNorm
from .unet_3d_blocks import UNetMidBlock3DCrossAttn, get_down_block, get_up_block
logger = logging.get_logger(name)  # pylint: disable=invalid-name
@DataClass
class UNet3DConditionOutput(BaseOutput):
sample: torch.FloatTensor
class UNet3DConditionModel(ModelMixin, ConfigMixin):
_supports_gradient_checkpointing = True
@register_to_config
def __init__(
    self,
    sample_size: Optional[int] = None,
    in_channels: int = 4,
    out_channels: int = 4,
    center_input_sample: bool = False,
    flip_sin_to_cos: bool = True,
    freq_shift: int = 0,
    down_block_types: Tuple[str] = (
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "DownBlock3D",
    ),
    mid_block_type: str = "UNetMidBlock3DCrossAttn",
    up_block_types: Tuple[str] = (
        "UpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
    ),
    only_cross_attention: Union[bool, Tuple[bool]] = False,
    block_out_channels: Tuple[int] = (320, 640, 1280, 1280),
    layers_per_block: int = 2,
    downsample_padding: int = 1,
    mid_block_scale_factor: float = 1,
    act_fn: str = "silu",
    norm_num_groups: int = 32,
    norm_eps: float = 1e-5,
    cross_attention_dim: int = 1280,
    attention_head_dim: Union[int, Tuple[int]] = 8,
    dual_cross_attention: bool = False,
    use_linear_projection: bool = False,
    class_embed_type: Optional[str] = None,
    num_class_embeds: Optional[int] = None,
    upcast_attention: bool = False,
    resnet_time_scale_shift: str = "default",
    use_inflated_groupnorm=False,
    # Additional
    use_motion_module=False,
    motion_module_resolutions=(1, 2, 4, 8),
    motion_module_mid_block=False,
    motion_module_decoder_only=False,
    motion_module_type=None,
    motion_module_kwargs={},
    unet_use_cross_frame_attention=None,
    unet_use_temporal_attention=None,
    mode=None,
    task_type="action",
):
    super().__init__()

    self.sample_size = sample_size
    time_embed_dim = block_out_channels[0] * 4

    # Add this line in the __init__ method
    self.pose_projector = nn.Conv3d(4, 320, kernel_size=1)

    # input
    self.conv_in = InflatedConv3d(
        in_channels, block_out_channels[0], kernel_size=3, padding=(1, 1)
    )

    # time
    self.time_proj = Timesteps(block_out_channels[0], flip_sin_to_cos, freq_shift)
    timestep_input_dim = block_out_channels[0]

    self.time_embedding = TimestepEmbedding(timestep_input_dim, time_embed_dim)

    # class embedding
    if class_embed_type is None and num_class_embeds is not None:
        self.class_embedding = nn.Embedding(num_class_embeds, time_embed_dim)
    elif class_embed_type == "timestep":
        self.class_embedding = TimestepEmbedding(timestep_input_dim, time_embed_dim)
    elif class_embed_type == "identity":
        self.class_embedding = nn.Identity(time_embed_dim, time_embed_dim)
    else:
        self.class_embedding = None

    self.down_blocks = nn.ModuleList([])
    self.mid_block = None
    self.up_blocks = nn.ModuleList([])

    if isinstance(only_cross_attention, bool):
        only_cross_attention = [only_cross_attention] * len(down_block_types)

    if isinstance(attention_head_dim, int):
        attention_head_dim = (attention_head_dim,) * len(down_block_types)

    # down
    output_channel = block_out_channels[0]
    for i, down_block_type in enumerate(down_block_types):
        if task_type == "action":
            name_index, mid_name = None, None
        else:
            name_index, mid_name = i, "MidBlock"
        res = 2**i
        input_channel = output_channel
        output_channel = block_out_channels[i]
        is_final_block = i == len(block_out_channels) - 1

        down_block = get_down_block(
            down_block_type,
            num_layers=layers_per_block,
            in_channels=input_channel,
            out_channels=output_channel,
            temb_channels=time_embed_dim,
            add_downsample=not is_final_block,
            resnet_eps=norm_eps,
            resnet_act_fn=act_fn,
            resnet_groups=norm_num_groups,
            cross_attention_dim=cross_attention_dim,
            attn_num_head_channels=attention_head_dim[i],
            downsample_padding=downsample_padding,
            dual_cross_attention=dual_cross_attention,
            use_linear_projection=use_linear_projection,
            only_cross_attention=only_cross_attention[i],
            upcast_attention=upcast_attention,
            resnet_time_scale_shift=resnet_time_scale_shift,
            unet_use_cross_frame_attention=unet_use_cross_frame_attention,
            unet_use_temporal_attention=unet_use_temporal_attention,
            use_inflated_groupnorm=use_inflated_groupnorm,
            use_motion_module=use_motion_module
            and (res in motion_module_resolutions)
            and (not motion_module_decoder_only),
            motion_module_type=motion_module_type,
            motion_module_kwargs=motion_module_kwargs,
            name_index=name_index,
        )
        self.down_blocks.append(down_block)

    # mid

    if mid_block_type == "UNetMidBlock3DCrossAttn":
        self.mid_block = UNetMidBlock3DCrossAttn(
            in_channels=block_out_channels[-1],
            temb_channels=time_embed_dim,
            resnet_eps=norm_eps,
            resnet_act_fn=act_fn,
            output_scale_factor=mid_block_scale_factor,
            resnet_time_scale_shift=resnet_time_scale_shift,
            cross_attention_dim=cross_attention_dim,
            attn_num_head_channels=attention_head_dim[-1],
            resnet_groups=norm_num_groups,
            dual_cross_attention=dual_cross_attention,
            use_linear_projection=use_linear_projection,
            upcast_attention=upcast_attention,
            unet_use_cross_frame_attention=unet_use_cross_frame_attention,
            unet_use_temporal_attention=unet_use_temporal_attention,
            use_inflated_groupnorm=use_inflated_groupnorm,
            use_motion_module=use_motion_module and motion_module_mid_block,
            motion_module_type=motion_module_type,
            motion_module_kwargs=motion_module_kwargs,
            name=mid_name,
        )
    else:
        raise ValueError(f"unknown mid_block_type : {mid_block_type}")

    # count how many layers upsample the videos
    self.num_upsamplers = 0

    # up
    reversed_block_out_channels = list(reversed(block_out_channels))
    reversed_attention_head_dim = list(reversed(attention_head_dim))
    only_cross_attention = list(reversed(only_cross_attention))
    output_channel = reversed_block_out_channels[0]
    for i, up_block_type in enumerate(up_block_types):
        res = 2 ** (3 - i)
        is_final_block = i == len(block_out_channels) - 1

        if task_type == "action":
            name_index = None
        else:
            name_index = i

        prev_output_channel = output_channel
        output_channel = reversed_block_out_channels[i]
        input_channel = reversed_block_out_channels[
            min(i + 1, len(block_out_channels) - 1)
        ]

        # add upsample block for all BUT final layer
        if not is_final_block:
            add_upsample = True
            self.num_upsamplers += 1
        else:
            add_upsample = False

        up_block = get_up_block(
            up_block_type,
            num_layers=layers_per_block + 1,
            in_channels=input_channel,
            out_channels=output_channel,
            prev_output_channel=prev_output_channel,
            temb_channels=time_embed_dim,
            add_upsample=add_upsample,
            resnet_eps=norm_eps,
            resnet_act_fn=act_fn,
            resnet_groups=norm_num_groups,
            cross_attention_dim=cross_attention_dim,
            attn_num_head_channels=reversed_attention_head_dim[i],
            dual_cross_attention=dual_cross_attention,
            use_linear_projection=use_linear_projection,
            only_cross_attention=only_cross_attention[i],
            upcast_attention=upcast_attention,
            resnet_time_scale_shift=resnet_time_scale_shift,
            unet_use_cross_frame_attention=unet_use_cross_frame_attention,
            unet_use_temporal_attention=unet_use_temporal_attention,
            use_inflated_groupnorm=use_inflated_groupnorm,
            use_motion_module=use_motion_module
            and (res in motion_module_resolutions),
            motion_module_type=motion_module_type,
            motion_module_kwargs=motion_module_kwargs,
            name_index=name_index,
        )
        self.up_blocks.append(up_block)
        prev_output_channel = output_channel

    # out
    if use_inflated_groupnorm:
        self.conv_norm_out = InflatedGroupNorm(
            num_channels=block_out_channels[0],
            num_groups=norm_num_groups,
            eps=norm_eps,
        )
    else:
        self.conv_norm_out = nn.GroupNorm(
            num_channels=block_out_channels[0],
            num_groups=norm_num_groups,
            eps=norm_eps,
        )
    self.conv_act = nn.SiLU()
    self.conv_out = InflatedConv3d(
        block_out_channels[0], out_channels, kernel_size=3, padding=1
    )

    self.mode = mode

@property
# Copied from diffusers.models.unet_2d_condition.UNet2DConditionModel.attn_processors
def attn_processors(self) -> Dict[str, AttentionProcessor]:
    r"""
    Returns:
        `dict` of attention processors: A dictionary containing all attention processors used in the model with
        indexed by its weight name.
    """
    # set recursively
    processors = {}

    def fn_recursive_add_processors(
        name: str,
        module: torch.nn.Module,
        processors: Dict[str, AttentionProcessor],
    ):
        if hasattr(module, "set_processor"):
            processors[f"{name}.processor"] = module.processor

        for sub_name, child in module.named_children():
            if "temporal_transformer" not in sub_name:
                fn_recursive_add_processors(f"{name}.{sub_name}", child, processors)

        return processors

    for name, module in self.named_children():
        if "temporal_transformer" not in name:
            fn_recursive_add_processors(name, module, processors)

    return processors

def set_attention_slice(self, slice_size):
    r"""
    Enable sliced attention computation.

    When this option is enabled, the attention module will split the input tensor in slices, to compute attention
    in several steps. This is useful to save some memory in exchange for a small speed decrease.

    Args:
        slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`):
            When `"auto"`, halves the input to the attention heads, so attention will be computed in two steps. If
            `"max"`, maxium amount of memory will be saved by running only one slice at a time. If a number is
            provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim`
            must be a multiple of `slice_size`.
    """
    sliceable_head_dims = []

    def fn_recursive_retrieve_slicable_dims(module: torch.nn.Module):
        if hasattr(module, "set_attention_slice"):
            sliceable_head_dims.append(module.sliceable_head_dim)

        for child in module.children():
            fn_recursive_retrieve_slicable_dims(child)

    # retrieve number of attention layers
    for module in self.children():
        fn_recursive_retrieve_slicable_dims(module)

    num_slicable_layers = len(sliceable_head_dims)

    if slice_size == "auto":
        # half the attention head size is usually a good trade-off between
        # speed and memory
        slice_size = [dim // 2 for dim in sliceable_head_dims]
    elif slice_size == "max":
        # make smallest slice possible
        slice_size = num_slicable_layers * [1]

    slice_size = (
        num_slicable_layers * [slice_size]
        if not isinstance(slice_size, list)
        else slice_size
    )

    if len(slice_size) != len(sliceable_head_dims):
        raise ValueError(
            f"You have provided {len(slice_size)}, but {self.config} has {len(sliceable_head_dims)} different"
            f" attention layers. Make sure to match `len(slice_size)` to be {len(sliceable_head_dims)}."
        )

    for i in range(len(slice_size)):
        size = slice_size[i]
        dim = sliceable_head_dims[i]
        if size is not None and size > dim:
            raise ValueError(f"size {size} has to be smaller or equal to {dim}.")

    # Recursively walk through all the children.
    # Any children which exposes the set_attention_slice method
    # gets the message
    def fn_recursive_set_attention_slice(
        module: torch.nn.Module, slice_size: List[int]
    ):
        if hasattr(module, "set_attention_slice"):
            module.set_attention_slice(slice_size.pop())

        for child in module.children():
            fn_recursive_set_attention_slice(child, slice_size)

    reversed_slice_size = list(reversed(slice_size))
    for module in self.children():
        fn_recursive_set_attention_slice(module, reversed_slice_size)

def _set_gradient_checkpointing(self, module, value=False):
    if hasattr(module, "gradient_checkpointing"):
        module.gradient_checkpointing = value

# Copied from diffusers.models.unet_2d_condition.UNet2DConditionModel.set_attn_processor
def set_attn_processor(
    self, processor: Union[AttentionProcessor, Dict[str, AttentionProcessor]]
):
    r"""
    Sets the attention processor to use to compute attention.

    Parameters:
        processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`):
            The instantiated processor class or a dictionary of processor classes that will be set as the processor
            for **all** `Attention` layers.

            If `processor` is a dict, the key needs to define the path to the corresponding cross attention
            processor. This is strongly recommended when setting trainable attention processors.

    """
    count = len(self.attn_processors.keys())

    if isinstance(processor, dict) and len(processor) != count:
        raise ValueError(
            f"A dict of processors was passed, but the number of processors {len(processor)} does not match the"
            f" number of attention layers: {count}. Please make sure to pass {count} processor classes."
        )

    def fn_recursive_attn_processor(name: str, module: torch.nn.Module, processor):
        if hasattr(module, "set_processor"):
            if not isinstance(processor, dict):
                module.set_processor(processor)
            else:
                module.set_processor(processor.pop(f"{name}.processor"))

        for sub_name, child in module.named_children():
            if "temporal_transformer" not in sub_name:
                fn_recursive_attn_processor(f"{name}.{sub_name}", child, processor)

    for name, module in self.named_children():
        if "temporal_transformer" not in name:
            fn_recursive_attn_processor(name, module, processor)

def forward(
    self,
    sample: torch.FloatTensor,
    timestep: Union[torch.Tensor, float, int],
    encoder_hidden_states: torch.Tensor,
    class_labels: Optional[torch.Tensor] = None,
    pose_cond_fea: Optional[torch.Tensor] = None,
    attention_mask: Optional[torch.Tensor] = None,
    down_block_additional_residuals: Optional[Tuple[torch.Tensor]] = None,
    mid_block_additional_residual: Optional[torch.Tensor] = None,
    return_dict: bool = True,
    self_attention_additional_feats = None,
) -> Union[UNet3DConditionOutput, Tuple]:
    r"""
    Args:
        sample (`torch.FloatTensor`): (batch, channel, height, width) noisy inputs tensor
        timestep (`torch.FloatTensor` or `float` or `int`): (batch) timesteps
        encoder_hidden_states (`torch.FloatTensor`): (batch, sequence_length, feature_dim) encoder hidden states
        return_dict (`bool`, *optional*, defaults to `True`):
            Whether or not to return a [`models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple.

    Returns:
        [`~models.unet_2d_condition.UNet2DConditionOutput`] or `tuple`:
        [`~models.unet_2d_condition.UNet2DConditionOutput`] if `return_dict` is True, otherwise a `tuple`. When
        returning a tuple, the first element is the sample tensor.
    """
    # By default samples have to be AT least a multiple of the overall upsampling factor.
    # The overall upsampling factor is equal to 2 ** (# num of upsampling layears).
    # However, the upsampling interpolation output size can be forced to fit any upsampling size
    # on the fly if necessary.
    default_overall_up_factor = 2**self.num_upsamplers

    # upsample size should be forwarded when sample is not a multiple of `default_overall_up_factor`
    forward_upsample_size = False
    upsample_size = None

    if any(s % default_overall_up_factor != 0 for s in sample.shape[-2:]):
        logger.info("Forward upsample size to force interpolation output size.")
        forward_upsample_size = True

    # prepare attention_mask
    if attention_mask is not None:
        attention_mask = (1 - attention_mask.to(sample.dtype)) * -10000.0
        attention_mask = attention_mask.unsqueeze(1)

    # center input if necessary
    if self.config.center_input_sample:
        sample = 2 * sample - 1.0

    # time
    timesteps = timestep
    if not torch.is_tensor(timesteps):
        # This would be a good case for the `match` statement (Python 3.10+)
        is_mps = sample.device.type == "mps"
        if isinstance(timestep, float):
            dtype = torch.float32 if is_mps else torch.float64
        else:
            dtype = torch.int32 if is_mps else torch.int64
        timesteps = torch.tensor([timesteps], dtype=dtype, device=sample.device)
    elif len(timesteps.shape) == 0:
        timesteps = timesteps[None].to(sample.device)

    # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
    timesteps = timesteps.expand(sample.shape[0])

    t_emb = self.time_proj(timesteps)

    # timesteps does not contain any weights and will always return f32 tensors
    # but time_embedding might actually be running in fp16. so we need to cast here.
    # there might be better ways to encapsulate this.
    t_emb = t_emb.to(dtype=self.dtype)
    emb = self.time_embedding(t_emb)

    if self.class_embedding is not None:
        if class_labels is None:
            raise ValueError(
                "class_labels should be provided when num_class_embeds > 0"
            )

        if self.config.class_embed_type == "timestep":
            class_labels = self.time_proj(class_labels)

        class_emb = self.class_embedding(class_labels).to(dtype=self.dtype)
        emb = emb + class_emb

    # 1. Check the shapes of both tensors
    print(f"sample shape: {sample.shape}")
    print(f"pose_cond_fea shape: {pose_cond_fea.shape}")

    # pre-process
    #sample = self.conv_in(sample)
    #if pose_cond_fea is not None:
    #    sample = sample + pose_cond_fea

    # pre-process
    if sample is not None:
        # Store original height of sample
        original_height = sample.shape[3]

        # Project sample to match pose_cond_fea's channel dimension
        sample = self.pose_projector(sample)

        # Adjust height of pose_cond_fea to match original sample height
        pose_cond_fea = F.interpolate(pose_cond_fea,
                                      size=(pose_cond_fea.shape[2], original_height, pose_cond_fea.shape[4]),
                                      mode='trilinear',
                                      align_corners=False)

        # Add debug prints
        print(f"Adjusted sample shape: {sample.shape}")
        print(f"Adjusted pose_cond_fea shape: {pose_cond_fea.shape}")

        # Ensure the shapes match before adding
        assert pose_cond_fea.shape == sample.shape, "Shapes still don't match after adjustment"

        sample = sample + pose_cond_fea

    # down
    down_block_res_samples = (sample,)
    for downsample_block in self.down_blocks:
        if (
            hasattr(downsample_block, "has_cross_attention")
            and downsample_block.has_cross_attention
        ):
            sample, res_samples = downsample_block(
                hidden_states=sample,
                temb=emb,
                encoder_hidden_states=encoder_hidden_states,
                attention_mask=attention_mask,
                self_attention_additional_feats=self_attention_additional_feats,
                mode=self.mode,
            )
        else:
            sample, res_samples = downsample_block(
                hidden_states=sample,
                temb=emb,
                encoder_hidden_states=encoder_hidden_states,
            )

        down_block_res_samples += res_samples

    if down_block_additional_residuals is not None:
        new_down_block_res_samples = ()

        for down_block_res_sample, down_block_additional_residual in zip(
            down_block_res_samples, down_block_additional_residuals
        ):
            down_block_res_sample = (
                down_block_res_sample + down_block_additional_residual
            )
            new_down_block_res_samples += (down_block_res_sample,)

        down_block_res_samples = new_down_block_res_samples

    # mid
    sample = self.mid_block(
        sample,
        emb,
        encoder_hidden_states=encoder_hidden_states,
        attention_mask=attention_mask,
        self_attention_additional_feats=self_attention_additional_feats,
        mode=self.mode,
    )

    if mid_block_additional_residual is not None:
        sample = sample + mid_block_additional_residual

    # up
    for i, upsample_block in enumerate(self.up_blocks):
        is_final_block = i == len(self.up_blocks) - 1

        res_samples = down_block_res_samples[-len(upsample_block.resnets) :]
        down_block_res_samples = down_block_res_samples[
            : -len(upsample_block.resnets)
        ]

        # if we have not reached the final block and need to forward the
        # upsample size, we do it here
        if not is_final_block and forward_upsample_size:
            upsample_size = down_block_res_samples[-1].shape[2:]

        if (
            hasattr(upsample_block, "has_cross_attention")
            and upsample_block.has_cross_attention
        ):
            sample = upsample_block(
                hidden_states=sample,
                temb=emb,
                res_hidden_states_tuple=res_samples,
                encoder_hidden_states=encoder_hidden_states,
                upsample_size=upsample_size,
                attention_mask=attention_mask,
                self_attention_additional_feats=self_attention_additional_feats,
                mode=self.mode,
            )
        else:
            sample = upsample_block(
                hidden_states=sample,
                temb=emb,
                res_hidden_states_tuple=res_samples,
                upsample_size=upsample_size,
                encoder_hidden_states=encoder_hidden_states,
            )

    # post-process
    sample = self.conv_norm_out(sample)
    sample = self.conv_act(sample)
    sample = self.conv_out(sample)

    if not return_dict:
        return (sample,)

    return UNet3DConditionOutput(sample=sample)

@classmethod
def from_pretrained_2d(
    cls,
    pretrained_model_path: PathLike,
    motion_module_path: PathLike,
    subfolder=None,
    unet_additional_kwargs=None,
    mm_zero_proj_out=False,
):
    pretrained_model_path = Path(pretrained_model_path)
    motion_module_path = Path(motion_module_path)
    if subfolder is not None:
        pretrained_model_path = pretrained_model_path.joinpath(subfolder)
    logger.info(
        f"loaded temporal unet's pretrained weights from {pretrained_model_path} ..."
    )

    config_file = pretrained_model_path / "config.json"
    if not (config_file.exists() and config_file.is_file()):
        raise RuntimeError(f"{config_file} does not exist or is not a file")

    unet_config = cls.load_config(config_file)
    unet_config["_class_name"] = cls.__name__
    unet_config["down_block_types"] = [
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "CrossAttnDownBlock3D",
        "DownBlock3D",
    ]
    unet_config["up_block_types"] = [
        "UpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
        "CrossAttnUpBlock3D",
    ]
    unet_config["mid_block_type"] = "UNetMidBlock3DCrossAttn"

    model = cls.from_config(unet_config, **unet_additional_kwargs)
    # load the vanilla weights
    if pretrained_model_path.joinpath(SAFETENSORS_WEIGHTS_NAME).exists():
        logger.debug(
            f"loading safeTensors weights from {pretrained_model_path} ..."
        )
        state_dict = load_file(
            pretrained_model_path.joinpath(SAFETENSORS_WEIGHTS_NAME), device="cpu"
        )

    elif pretrained_model_path.joinpath(WEIGHTS_NAME).exists():
        logger.debug(f"loading weights from {pretrained_model_path} ...")
        state_dict = torch.load(
            pretrained_model_path.joinpath(WEIGHTS_NAME),
            map_location="cpu",
            weights_only=True,
        )
    else:
        raise FileNotFoundError(f"no weights file found in {pretrained_model_path}")

    # load the motion module weights
    if motion_module_path.exists() and motion_module_path.is_file():
        if motion_module_path.suffix.lower() in [".pth", ".pt", ".ckpt"]:
            logger.info(f"Load motion module params from {motion_module_path}")
            motion_state_dict = torch.load(
                motion_module_path, map_location="cpu", weights_only=True
            )
        elif motion_module_path.suffix.lower() == ".safetensors":
            motion_state_dict = load_file(motion_module_path, device="cpu")
        else:
            raise RuntimeError(
                f"unknown file format for motion module weights: {motion_module_path.suffix}"
            )
        if mm_zero_proj_out:
            logger.info(f"Zero initialize proj_out layers in motion module...")
            new_motion_state_dict = OrderedDict()
            for k in motion_state_dict:
                if "proj_out" in k:
                    continue
                new_motion_state_dict[k] = motion_state_dict[k]
            motion_state_dict = new_motion_state_dict

        # merge the state dicts
        state_dict.update(motion_state_dict)

    # load the weights into the model
    m, u = model.load_state_dict(state_dict, strict=False)
    logger.debug(f"### missing keys: {len(m)}; \n### unexpected keys: {len(u)};")

    params = [
        p.numel() if "temporal" in n else 0 for n, p in model.named_parameters()
    ]
    logger.info(f"Loaded {sum(params) / 1e6}M-parameter motion module")

    return model

Is this three-dimensional interpolation correct, because one dimension is time, and does denoising_unet need to be retrained after this change? Because the original denoising does not have interpolation, sample=sample+cond


Mother Cell's in Experimental are infinitely tiny.

I don't think this happened before but after a recent update I believe, the mother cells are bugged.

https://github.com/Barbosik/MultiOgar/blob/master/src/gamemodes/Experimental.js#L41
The "null" on the last argument may be the reason.  Try changing it to 149 for short-term solution.
    var mother = new Entity.MotherCell(gameServer, null, pos, 149);
Another problem I see is that the mother cells don't see enough food cell spawns, especially when playing alone or in small number.

Thanks @SimonOrJ, I wish to see what @Barbosik says about it first. It's quite confusing, it was working before a recent update.

you don't need to pass 149 size to MotherCell. Because it will use default size for null value. Default size for MotherCell is 149.
Actually this is not a bug. MotherCell stops to eject food when total food amount on the map reach foodMaxAmount value from gameserver.ini
It prevents lags for the server and clients. For example if MotherCell will eject 20000 foods, it will leads to extreme lags. You will gel very-very low fps on the client in such case. Also it will leads to server lags, because a lot of cells will be placed in the same location and it will need to process a lot of collisions.

I am making a pull request to fit the agar.io mothercell behavior. It allows for mother cells to get big and be fed a lot. I understand about the lag, should I include a maxSize value too?

I'll commit tomorrow, going to bed now.

@Barbosik I don't think you understand.
Can you play experimental on your version of MultiOgar, then you'll see what i'm talking about?
MotherCell's are extremely small and broken.

@darknight99x: ok, I will check

Yeah, there is a bug with MotherCell, will be fixed soon

Okay thanks @Barbosik
Glad there is active development here.

fixed


Reproducing 4k/8k segmentation results

Hi,
Can you share the config you used to train 4k and 8k iters? Most importantly, the learning rate schedule you were using?
Trying to reproduce the results.
Thank you.

I have the same question

Using the 80k schedule that you provided in the repo and just reducing the total number of epochs gives ~39 mIoU while you are reporting 43.9 in the paper for 4k steps. So I assume you are using a different config for 4k and 8k iters. Would be very thankful if you could indicate what is different there. Thanks in advance.

From the paper:
For the fast schedule (8K iterations), we linear scale
the learning rate schedule and set the warming-up iterations
to 150.
Is it the same for 4k? And what does linear scale exactly mean?


git submodule update --depth 1: reference is not a tree

This was first reported by someone else on https://svn.boost.org/trac10/ticket/13293 with:

Not sure if this is intended to be supported - but if so it's an issue with the git super-project - please file an issue on github against that.

Summary
Getting shallow local copy of boost submodules fails.
How to reproduce
git clone --depth 1 --branch boost-1.69.0 https://github.com/boostorg/boost.git boost
cd boost
git submodule init libs/config
# so far so good
git submodule update --depth 1
# boom

Expected
Clone okay
Observed
fatal: reference is not a tree: 67102d043643c51104663c5b00e375c2f476cdf5
Unable to checkout '67102d043643c51104663c5b00e375c2f476cdf5' in submodule path 'libs/config'

Additional information
Workaround: full-history submodule
Somehow wastes bandwidth, which will not suit everyone.
git clone --depth 1 --branch boost-1.69.0 https://github.com/boostorg/boost.git boost_works
cd boost_works
git submodule init libs/config
# so far so good
git submodule update

Git version
git --version

git version 2.7.4

Background information
Background information on How to make shallow git submodules? - Stack Overflow, your mileage may very depending on your version of git.
Any comment welcome.

There's nothing we can do about this issue. It's just how git works.

Agreed somehow. I'd rather say it's just how old versions of git work.
That said, newer versions of git might just compile as easily on old platforms as on new ones.
Back to the cause: this very issue is an implementation of that suggestion on svn.boost.org: #13293 (git submodule update --depth 1: reference is not a tree) – Boost C++ Libraries

Not sure if this is intended to be supported - but if so it's an issue with the git super-project - please file an issue on github against that.

I suggest we just close the issue and let by-passers comment. Silence here will be a hint that it's not a real issue.

--depth 1 doesn't work reliably on new versions of git too, because Github doesn't allow direct SHA fetch.

The exact same bug in question is also mentioned at: https://stackoverflow.com/questions/2155887/git-submodule-head-reference-is-not-a-tree-error/25875273#25875273
It works in Ubuntu 18.04 which has Git 2.17.
What I've been doing to work around the SHA fetch is to just keep a mirror on my GitHub account, and push a branch pointing to the commit that I want to fetch.


Fixes watch ignore blank line in pod logs

What type of PR is this?
/kind bug

What this PR does / why we need it:
In this PR, I fix watch ignore blank line in pod logs problem.
with enumerate yield first line even if line is blank. when lines split by '\n' char.
Which issue(s) this PR fixes:

Fixes #239
Does this PR introduce a user-facing change?

Fix watch ignore blank line in pod logs

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Codecov Report

Merging #240 (35fac6a) into master (d25434b) will not change coverage.
The diff coverage is 100.00%.

:exclamation: Current head 35fac6a differs from pull request most recent head 2f66636. Consider uploading reports for the commit 2f66636 to get more accurate results

@@           Coverage Diff           @@
##           master     #240   +/-   ##
=======================================
  Coverage   92.71%   92.71%
=======================================
  Files          13       13
  Lines        1703     1703
=======================================
  Hits         1579     1579
  Misses        124      124

Impacted Files
Coverage Δ

watch/watch.py
95.95% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d25434b...2f66636. Read the comment docs.

watch is used for receiving events, not for logs. please clarify how the fix in watch could help with pod logs blank lines issue.

/assign


Add a script for restarting the USB device

We are using our RealSense on a mobile robot platform, which means long cables and lots of motion. The cameras often crash, so we found it convenient to write a C program that finds and restarts all R200 cameras without power cycling them. (It might be more convenient if it were a Python script, but I was in a hurry to get this done and found a decent C template online.) Let me know if this is of interest and where to put it and I'll do a PR. Otherwise, hopefully it inspires something similar.
reset_r200.c.txt

To test it, compile:
$ cc reset_r200.c -o reset_r200
Make it executable:
$ chmod +x reset_r200

Adding a line to kill all zombie R200 processes, based on the process name "nodelet."
reset_r200.c.txt

Hi @AndyZe, it is not normal for R200 cameras to crash, but acknowledge that USB3 devices are more sensitive to long cables, etc.  I'd recommend including your reset utility as a .cpp file in the 'src' directory, and adding the required commands to the CMakeLists.txt file to build it as a standalone reset utility.


Support shebang to precise that a file is a Camel-K executable file

initial proposal is to use #!/usr/bin/env camel-k
This can be adjusted, just need to synchronize with Language Server implementation, see https://github.com/camel-tooling/camel-language-server/issues/163
(maybe not even use a shebang but rely on comments only or filename)

we can use modelines now


Question: can I use analog joystick input using ADC pins?

I have been looking into this recently, however I currently don’t have an RP2040 system to test this on as of now. I didn’t see any information in how (or if) you can control the sticks with analog inputs, as analog outputs. Just a quick question.
Thanks,
brickeduser

Hi @brickedware ,
At this time analog joysticks through ADC are not supported.  It is a heavily requested feature so there are a few people looking into it currently.
As a note - This project is no longer maintained.  Please check out the Community Edition project and post your feature request there.  The Community Edition issues section can be found here: https://github.com/OpenStickFoundation/GP2040-CE/issues


Order Export file Not found

Description
Using the order export with a csv template. I think the export is working correctly but if you want to download the file, you get an error message "Not found".
Expected Behavior
It should be possible to download the order export
Current Behavior
Error message not found is shown
Context
Support ticket
Possible Solution

Hello,
I am investigating this issue and so far I found these things:

Impex is using this command to export orders:

~$ /app/node_modules/.bin/order-export --projectKey 'coeur-staging' --accessToken '-----' --csvTemplate '/app/1501148497457_orders-tpl.csv' --csvFile '/app/tmp-18NYK8QtgJUevZ/exported-orders.csv' --exportCSVAsStream --logSilent --sphereHost 'api.sphere.io' --sphereProtocol 'https'

I looked to history and in the supported parameters there is missing --csvFile.


”Valea Viilor”, ”Motiș” are listed in the drop-down menu

Description:
When user writes ”Valea Viilor”, ”Motiș” the locations appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Valea Viilor"

Expected results
The location appears in the drop-down menu as "Sat Valea Viilor, SIBIU (Valea Viilor)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Motiș"

Expected results
The location appears in the drop-down menu as "Sat Motiș, SIBIU (Valea Viilor)".


requirements.txt - bokeh 2.0.1 only.

Forest falls over on bokeh 2.0.2 - it does not display anything within serve mode. The requirements file should be ==2.0.1 and not >=2.0.1 to reflect this until the origin of this is found.
It is likely a bokeh package error, as the issue persists across other applications.

@wolfiex I use 2.0.2 and it works fine, although I haven't installed the latest build of bokeh in a few weeks maybe there's a temporary bug


Fixed imports in TopAppBar demo

Copying this code directly didn't work since the imports were wrong.

Failed pictures (1): failed-pictures.zip

:postbox: Posted by artipost.io, through Travis build 410617060, for commit 02d1465.

@alexandrelaplante you need to download this image and replace it in tests/golden/wide


Fix isNumber check for empty strings

merged this, thanks!

Thank you for merging!
Could I ask for a new release with this fix?
It will allow to compare empty strings with STREQ from the example.

When #270 is working, I will make a release

As a sidenote, swapping the two checks would have had the same effect:
if (st.length() == 1 && st.charAt(0) == MINUS_SIGN) {

@ssheikin I just created version 2.7. Might take a while, until it pops up in Maven Central.


Fix in Typescript definitions

Some fix in Typescript definitions:

prop 'initial' in 'Scene' should be optional
all props of scenes that could be used in 'Router' should be optional there
optional wrapBy in 'Scene'
optional props in Action.pop

👍


macvim: Add sandboxProfile

This allows full filesystem access when the sandbox is in relaxed mode. We should lock this down a bit once we figure out what exactly Xcode wants access to.
Motivation for this change
The macvim package won't build in ofBorg due to sandboxing. I assume it still won't build with this change, but this should be friendlier to users who want sandboxing.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[X] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

I've pushed a new commit that uses sandboxProfile, but right now it's allowing complete filesystem access. I'll look into locking that down a bit later.

Updated. It still allows full filesystem access because we don't know where Xcode is located, but we're now blocking /usr/local. I went ahead and allowed fsctl and iokit-open because it seems harmless relative to full filesystem access.

Actually I take it back, I'm going to remove fsctl and iokit-open. Just because ibtool wants it doesn't mean we need to hand it over; the build seems to work perfectly fine without these.

It occurs to me I should add (with no-log) to the homebrew deny rule to get rid of a bunch of console spam

Updated accordingly. Here's the trivial diff:
diff --git a/pkgs/applications/editors/vim/macvim.nix b/pkgs/applications/editors/vim/macvim.nix
index 0951c226bb8..b639ab61784 100644
--- a/pkgs/applications/editors/vim/macvim.nix
+++ b/pkgs/applications/editors/vim/macvim.nix
@@ -139,7 +139,7 @@ stdenv.mkDerivation {
   sandboxProfile = ''
     (allow file-read* file-write* process-exec mach-lookup)
     ; block homebrew dependencies
-    (deny file-read* file-write* process-exec mach-lookup (subpath "/usr/local"))
+    (deny file-read* file-write* process-exec mach-lookup (subpath "/usr/local") (with no-log))
   '';

   meta = with stdenv.lib; {

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/60


Revise Public API Types

Related Issue
#30
Summary
This PR:

[x] Provides a new type CoinMap:
newtype CoinMap a = CoinMap { unCoinMap :: Map a Coin }

[x] Uses CoinMap to unify and replace the following types:

UTxO
[Input]
[Output]

[x] Uses CoinMap to adjust the definition of CoinSelection:
  data CoinSelection i o = CoinSelection
-    { inputs  :: [Input  i]
-    , outputs :: [Output o]
+    { inputs  :: CoinMap i
+    , outputs :: CoinMap o
     , change  :: [Coin]
     }

Benefits:

[x] the type of CoinSelection now prevents duplicate inputs.
[x] the type of CoinSelection now prevents duplicate outputs.
[x] we now no longer need tests to ensure that algorithms are agnostic to ordering.
[x] we now no longer need separate types for Input and Output.

[x] Uses CoinMap to adjust the definition of CoinSelectionAlgorithm:
- newtype CoinSelectionAlgorithm i o u m e = CoinSelectionAlgorithm
+ newtype CoinSelectionAlgorithm i o   m e = CoinSelectionAlgorithm
      { selectCoins
          :: CoinSelectionOptions i o e
-         -> UTxO u
-         -> NonEmpty (Output o)
-         -> ExceptT (CoinSelectionError e) m (CoinSelection i o, UTxO u)
+         -> CoinMap i
+         -> CoinMap o
+         -> ExceptT (CoinSelectionError e) m (CoinSelection i o, CoinMap i)
      }

Benefits:

[x] One fewer type parameter :u has been removed (it was always equivalent to i).
[x] The symmetry between inputs and outputs is clearer: they are both maps.

Remaining Work

[x] Make the test suite pass.

@paweljakubas wrote:

LGTM!. I am very fond of commit summary - it helped me to comprehend this no-so-small PR!

Thank-you for this feedback, I appreciate it. I tried very hard to write a summary that would make this PR easier to understand. I'm glad that you found it useful! :+1:


Ocelot does not build on windows

From<EMAIL_ADDRESS>on April 21, 2011 14:52:58
The title says it all, we need someone to actually step through the entire build process on windows.
Original issue: http://code.google.com/p/gpuocelot/issues/detail?id=51

From<EMAIL_ADDRESS>on May 11, 2011 14:35:14
Ocelot uses functions provided in cfenv/fenv.h. This header file is not part of MSVC standard library implementation.
Has anyone got around this isse? Perhaps by changing the source/ supplying own functions etc?
Thanks,
Ganesh

From<EMAIL_ADDRESS>on May 11, 2011 15:57:46
See https://code.google.com/p/gpuocelot/source/browse/trunk/branches/ocelot-windows/ocelot/executive/implementation/CooperativeThreadArray.cpp?r=1001 for what I did to get around this the last time I tried.  I couldn't immediately find an equivalent to fenv on windows so I just turned it off.  Ideally, we would want a way to set the rounding modes on windows...

From sschuberth on June 23, 2011 05:29:39
The FPU rounding mode can be set via _control87 on Windows, see http://msdn.microsoft.com/en-us/library/e9b52ceh(v=VS.100).aspx If there's currently no one working on Windows support, I'd be willing to give it a try.

From<EMAIL_ADDRESS>on June 23, 2011 10:28:12
Thanks for the link.  I think that we would have to do something to ensure that once we set the mode, the x87 FPU is used rather than the SSE unit (which I think most compilers end up choosing by default).  Also, we would probably want to wrap both fenv and _control87 behind a wrapper library.
I don't think that anyone is working on this now, although I am tracking it.
If you are willing, I would encourage you to give it a try.

From<EMAIL_ADDRESS>on June 23, 2011 15:05:53
Issue 32 has been merged into this issue.

From sschuberth on July 29, 2011 09:29:32
I'm currently working on compiling gpuocelot with VS2010 and will hopefully be able to send some patches to the mailing list soon.

From<EMAIL_ADDRESS>on September 27, 2011 10:52:47
There is currently support in the source code as well as the build system for windows.  I need someone to independently try this out and verify it.
Status: Started

From<EMAIL_ADDRESS>on September 27, 2011 10:58:39
I'm up. What is needed?

From<EMAIL_ADDRESS>on September 27, 2011 14:27:02
I added some experimental build instructions to the Installation wiki page.  You need to make sure that you have all of the prerequisites installed and then try out the commands listed.

From<EMAIL_ADDRESS>on February 01, 2012 15:37:35
any news about this subject? I would like to use it on my windows. Thanks.

From sschuberth on February 02, 2012 01:26:26
I believe this has been resolved quite some time ago already, starting with https://code.google.com/p/gpuocelot/source/detail?r=1405

From<EMAIL_ADDRESS>on February 02, 2012 08:56:19
All right, but I see no version, or public package available for windows... there is any possibility to see this?
Thanks.

From<EMAIL_ADDRESS>on February 02, 2012 09:08:48
I'm currently leaving this open until we have a library available on the website.

From<EMAIL_ADDRESS>on February 02, 2012 15:11:20
Thanks for the info. I will wait for news about it, because I plan to use Ocelot in many projects for windows I am developing, I have many cuda codes that needs to be executed on AMD GPU or CPU (My desktop currently have a low-end Nvidia card, one high-end AMD card, and an 2 x 8 core cpu, and I will greatly take benefit if I could use all of them at the same time). Best regards. Thanks.

From dmitrinesteruk on May 30, 2012 01:31:14
Any news on this?

From<EMAIL_ADDRESS>on May 30, 2012 08:48:32
There is preliminary support for building on windows.  Just run build.py from the command line.
The emulator is supported (although there are a few minor issues with rounding modifiers that makes floating point precision slightly different than on linux in some rare cares).
The multicore CPU backend has been tested a few times.
The NVIDIA and AMD backends build, but they have not been tested.
It would be useful to have someone donate a windows machine for running regression tests.

From<EMAIL_ADDRESS>on June 01, 2012 15:08:15
Very nice! I am going to try this. But for what I saw, in the code, I feel it is a very nice piece of tech, very well written. Thanks very much!
PS.: I dont found the Opencl backend, it is me, or it is really not present at the moment in trunk?
PS.: I saw here to find details about how to build for windows, https://code.google.com/p/gpuocelot/wiki/Installation , but, it seems something is omitted. A little more details would not hurt.

From<EMAIL_ADDRESS>on August 24, 2012 12:14:31
Hi, is it still possible to get the codes somewhere? I have VS 2010 Proffesional at my disposal and since my laptop doesnt have NVIDIA card, I would like to use Ocelot to emulate it in Windows environment. While I was able to get Ocelot working after about day and a half on Ubuntu 12.04, I wont be able to use it for active development since everybody around uses VS to develop CUDA apps. I can imagine it being pretty painful to keep linux and VS project synchronized. I am working on a naive "hybrid" solution involving virtual machine with linux and a shared folder containing project sources hosted on the host computer. Then, in theory, it could be possible to edit all files in VS, switch to virtual machine, compile the stuff in linux and see the result there too. Its just a thought, though, how efficient and comfortable that can be, no idea.
Still, if I could download the Windows sources for Ocelot from somewhere, I would certainly give it a try. Is there still such a place available, please?

From<EMAIL_ADDRESS>on October 18, 2012 18:39:04
I've actually managed to get Ocelot to compile under win7 with 2010 pro in both 32 and 64 bits after some work, but now when I try to link the ocelot.lib it complains about missing symbols.
for a very basic program that does nearly nothing:
C:\Users\micha-f\Documents\test>nvcc -m32 -c main.cu
main.cu
tmpxft_0000344c_00000000-5_main.cudafe1.gpu
tmpxft_0000344c_00000000-10_main.cudafe2.gpu
main.cu
tmpxft_0000344c_00000000-5_main.cudafe1.cpp
tmpxft_0000344c_00000000-15_main.ii
C:\Users\micha-f\Documents\test>cl main.obj C:\Programs\Ocelot\lib\ocelot.lib
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.
Microsoft (R) Incremental Linker Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.
/out:main.exe
main.obj
C:\Programs\Ocelot\lib\ocelot.lib
main.obj : error LNK2019: unresolved external symbol _cudaFree@4 referenced in function _main
main.obj : error LNK2019: unresolved external symbol _cudaConfigureCall@32 referenced in function _main
main.obj : error LNK2019: unresolved external symbol _cudaMemcpy@16 referenced in function _main
main.obj : error LNK2019: unresolved external symbol _cudaMalloc@8 referenced in function _main
main.obj : error LNK2019: unresolved external symbol _cudaSetupArgument@12 referenced in function "void __cdecl __device_stub__Z5helloPcPi(char *,int *)" (?__device_stub__Z5helloPcPi@@YAXPADPAH@Z)
main.obj : error LNK2019: unresolved external symbol ___cudaRegisterFatBinary@4 referenced in function "void __cdecl __sti____cudaRegisterAll_39_tmpxft_0000344c_00000000_8_main_cpp1_ii_794ab5f2(void)" (?__sti____cudaRegisterAll_39_tmpxft_0000344c_00000000_8_main_cpp1_ii_794ab5f2@@YAXXZ)
main.obj : error LNK2019: unresolved external symbol ___cudaUnregisterFatBinary@4 referenced in function "void __cdecl __cudaUnregisterBinaryUtil(void)" (?__cudaUnregisterBinaryUtil@@YAXXZ)
main.obj : error LNK2019: unresolved external symbol ___cudaRegisterFunction@40 referenced in function "void __cdecl __nv_cudaEntityRegisterCallback(void * *)" (?__nv_cudaEntityRegisterCallback@@YAXPAPAX@Z)
main.obj : error LNK2019: unresolved external symbol _cudaLaunch@4 referenced in function "enum cudaError __cdecl cudaLaunch(char *)" (??$cudaLaunch@D@@YA?AW4cudaError@@PAD@Z)
main.exe : fatal error LNK1120: 9 unresolved externals

From<EMAIL_ADDRESS>on August 18, 2014 13:54:43
I could definetly see about donating a windows machine to help further develop this project (if that's what's really needed, maybe it's actually more devs. that is needed?).
Meanwhile I'm also gonna give the windows build a go... How many have been successful so far?


Hvae you ever face this problem while training?

Global seed set to 23
Running on GPUs 2,3
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 241.49 M params.
Keeping EMAs of 278.
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
/root/miniforge3/envs/pid/lib/python3.8/site-packages/clip/clip.py:24: UserWarning: PyTorch version 1.7.1 or higher is recommended
  warnings.warn("PyTorch version 1.7.1 or higher is recommended")
Spatial Rescaler mapping from 3 to 3 channels after resizing.
Restored from ./pretrained/TeVNet_FLIR/epoch_1000.pth with 0 missing and 0 unexpected keys
Successfully load
Monitoring val/loss_simple_ema as checkpoint metric.
Merged modelckpt-cfg:
{'target': 'pytorch_lightning.callbacks.ModelCheckpoint', 'params': {'dirpath': 'logs/2024-11-11T23-02-18_cats512-vqf8/checkpoints', 'filename': '{epoch:06}', 'verbose': True, 'save_last': True, 'monitor': 'val/loss_simple_ema', 'save_top_k': 3}}
Caution: Saving checkpoints every n train steps without deleting. This might require some free space.
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Traceback (most recent call last):
  File "/home/LT/diffusion/PID/main.py", line 720, in <module>
    trainer.fit(model, data)
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 462, in fit
    self.train_loop.setup_fit(model, train_dataloader, val_dataloaders, datamodule)
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 117, in setup_fit
    self.trainer.data_connector.attach_data(model, train_dataloader, val_dataloaders, datamodule)
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 84, in attach_data
    self.attach_datamodule(model, datamodule, 'fit')
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 127, in attach_datamodule
    if is_overridden('val_dataloader', datamodule):
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/utilities/model_helpers.py", line 42, in is_overridden
    is_overridden = instance_attr.__code__ is not super_attr.__code__
AttributeError: 'functools.partial' object has no attribute '__code__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/LT/diffusion/PID/main.py", line 722, in <module>
    melk()
  File "/home/LT/diffusion/PID/main.py", line 703, in melk
    trainer.save_checkpoint(ckpt_path)
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/properties.py", line 327, in save_checkpoint
    self.checkpoint_connector.save_checkpoint(filepath, weights_only)
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 401, in save_checkpoint
    checkpoint = self.dump_checkpoint(weights_only)
  File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 312, in dump_checkpoint
    checkpoint['state_dict'] = model.state_dict()
AttributeError: 'NoneType' object has no attribute 'state_dict'
#### Data #####
train, FLIRTrain, 586
validation, FLIRVal, 100
accumulate_grad_batches = 1
Setting learning rate to 4.00e-06 = 1 (accumulate_grad_batches) * 2 (num_gpus) * 2 (batchsize) * 1.00e-06 (base_lr)
Summoning checkpoint.

Process finished with exit code 1

Sorry, I have not encounter such problems. From File "/root/miniforge3/envs/pid/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 312, in dump_checkpoint     checkpoint['state_dict'] = model.state_dict() AttributeError: 'NoneType' object has no attribute 'state_dict', it seems that the model is not well set.

Thanks for your suggestions. I tried to downgrade my transformers lib, and it works!


Widget close and minimize buttons are too close together

From "Tapella, Robert K (398H)"<EMAIL_ADDRESS>"In the right pane of the Search panel, each of the selected search groups have a close button next to a disclosure triangle to hide the section. The buttons are so close that I observed users closing the sections rather than hiding them."

Fixed.


BEM changes depending on calling path

Describe the bug
The BEM model is different depending on where the the code is called from.  From what I can tell it appears to be missing the CRAS offset.  This is  in the viz.plot_aligment, but I have also seen it in the source localization outputs.
Steps to reproduce
import mne
import os, os.path as op

subjects_dir=op.join(os.environ['HOME'],
                 'mne_data/MNE-somato-data/derivatives/freesurfer/subjects')

#In mne_data directory
os.chdir(subjects_dir)
mne.viz.plot_bem('01', subjects_dir=subjects_dir)

#In home directory
os.chdir(os.environ['HOME'])
mne.viz.plot_bem('01', subjects_dir=subjects_dir)

Expected results

Actual results

Additional information
Platform:      Linux-5.4.0-80-generic-x86_64-with-glibc2.10
Python:        3.8.8 (default, Feb 24 2021, 21:46:12)  [GCC 7.3.0]
Executable:    /home/jstout/miniconda3/envs/enigma/bin/python
CPU:           x86_64: 20 cores
Memory:        125.5 GB
mne:           0.23.0
numpy:         1.20.1 {blas=openblas, lapack=openblas}
scipy:         1.6.1
matplotlib:    3.3.4 {backend=module://ipykernel.pylab.backend_inline}
sklearn:       0.24.1
numba:         0.53.0
nibabel:       3.2.1
nilearn:       0.7.1
dipy:          1.4.0
cupy:          Not found
pandas:        1.2.3
mayavi:        Not found
pyvista:       0.29.0 {pyvistaqt=0.3.0, OpenGL 4.5.0 NVIDIA 460.91.03 via Quadro RTX 8000/PCIe/SSE2}
vtk:           9.0.1
PyQt5:         5.15.4

I am able to replicate this consistently on two machines (redhat and ubuntu), using different newly installed conda environments with mne 0.21-0.23 (0.20 seemed to work).  At the same time, I have had others in our group unable to replicate this issue on their machine.  I dont' know if this is a true issue or a configuration on my end.  I will keep looking into this.

I think that i have figured out the issue, but I am not sure what the preferred solution would be.  Basically during the plot_bem (and setup_volume_source_space), the command _check_mri will create a full path for the mri (from the subject + subj_dir ...) unless it finds the filename within the current folder.  The default mri name is T1.mgz for the function call.  I had a T1.mgz file in my home directory, so it was preferrentially using this file rather than the one in subjects_dir/subject/mri folder.
Moving a T1.mgz file to/from the path of the function call has been able to create/fix the problem on different machines (linux + mac).
def _check_mri(mri, subject, subjects_dir):
_validate_type(mri, 'path-like', 'mri')
if not op.isfile(mri):                    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
if subject is None:
raise FileNotFoundError(
'MRI file %r not found and no subject provided' % (mri,))
subjects_dir = get_subjects_dir(subjects_dir, raise_error=True)
mri = op.join(subjects_dir, subject, 'mri', mri)
if not op.isfile(mri):
raise FileNotFoundError('MRI file %r not found' % (mri,))
return mri

Can you open a PR?

Yeah I will submit a PR by the end of the day


ATS 8.0.x/8.1.x leaks HTTP/2 connections

we've been seeing this consistently across every ATS instance that handles HTTP/2 traffic:

vgutierrez@cp1085:~$ sudo -i traffic_ctl --run-root=/srv/trafficserver/tls metric match current_active_client_connections
proxy.process.http.current_active_client_connections 39
proxy.process.http.websocket.current_active_client_connections 0
proxy.process.http2.current_active_client_connections 1597
vgutierrez@cp1085:~$ sudo -i traffic_ctl --run-root=/srv/trafficserver/tls metric match transactions |egrep "current_(client|server)_transactions"
proxy.process.http.current_client_transactions 54
proxy.process.http.current_server_transactions 50

proxy.process.http2.current_active_client_connections only goes back to 0 if TS is restarted, and we don't believe that's only a metrics issue. Even on depooled servers we had a huge number of ESTABLISHED connections until we lowered proxy.config.net.default_inactivity_timeout from the default value of 86400 seconds to 300 seconds

is this still an issue with 8.1.x or 9.0.x ?

yes, I've tested ATS 8.1.x (commit bfd09aa) and this is still happening:

I found a case that proxy.process.http2.current_active_client_connections is not decremented. However, it seems like the connection is closed on the case.
It happens on master when ATS gracefully closes an H2 connection. There are two places that call clear_session_active, which decrements the metric, but the one in release_stream is skipped because of graceful shutdown, and the other in Http2ClientSession::do_io_close is also not called because do_io_close itself is not called.
Like I said, the case I found is only a metric issue, but something is obviously broken. I'll let you know if I find any.

we've tested #6729 and it doesn't fix the issue, stats look basically the same.

@jvgutierrez Do you still see the increase? This issue was automatically closed by GitHub when #6876 was merged, but I don't see any comments from you. Please reopen if this still happens.

we backported #6876 to our 8.0.8-1wm2 and it solved the issue for us:


update syntax in the "report" and "project" functions

the hashes in the "report" and "project" functions were choking in ruby 1.9.3 so i changed them and they seem to work now. maybe it's no good for more recent versions of ruby, but 1.9.3 is what comes default with ubuntu

Coverage decreased (-7.9%) to 91.399% when pulling c0ec2f391bc8b4a80a2caa9f8401e9a0cd7ad4b5 on ShaunGrey:patch-1 into 0efb1694d64c800f56fbee1e96261d413cb49943 on kanet77:master.


Beam Stream commands do not work

When using the ~beam command it does not notify when that streamer goes online.  I suspect it is related to the ~checkbeam command always returning "Streamer 0 is online." regardless of which streamer username I input.

Fixed in the latest push. Will be in the next release.

This is still not working and I have updated Nadeko.  It does now say "Streamer ibigfire is offline." when checking the Beam stream, but it will always say I am offline whether I am online or not.