DeepSpeed-Chat: prefetch of layers during reward model forward pass leads to error during sample generation

When running step 3 with ZERO stage 3 enabled for both the actor and critic models,
I get the following error (line numbers may be offset due to debug statements I've added):
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 529, in <module>
  main()
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 438, in main
  out = trainer.generate_experience(prompts)
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 103, in generate_experience
    seq = self._generate_sequence(prompts)
File "/path/site-packages/deepspeed/runtime/hybrid_engine.py", line 293, in generate
  seq = self.actor_model.module.generate(prompts,
File "/path/site-packages/deepspeed/runtime/hybrid_engine.py", line 293, in generate
  self.fuse_lora_weight()
File "/path/site-packages/deepspeed/runtime/hybrid_engine.py", line 128, in _fuse_lora
    weight.data += lora_scaling * torch.matmul(lora_left_weight.t(), lora_right_weight.t())
RuntimeError: The size of tensor a (8192) must match the size of tensor b (2048) at non-singleton dimension 1

This happens because the weight.data shape does not match the tensor shape resulting from the lora matmul operation.
I am using a system with 4x 16GB V100 GPUs per node with DeepSpeed 0.9.1.  I trained a 1.3b-param model in step 1 and 350m-param model in step 2.
  If I print the tensor shape for each parameter of each layer immediately after GatheredParameters like so:
https://github.com/microsoft/DeepSpeed/blob/050aee287d70157e29f242b3a629a4cb97b4e4e7/deepspeed/runtime/hybrid_engine.py#L238
                with GatheredParameters(non_active_layers):
                    if rank == 0:
                        for layer_id in range(len(self.layer_params)):
                            for p_id, p in enumerate(self.layer_params[layer_id]):
                                print("after gather layer_id", layer_id, p_id, p.shape, flush=True)
                    self._gather_latency = time.time() - self._t0

then I see the following output on the step just before the error:
nonactive all layers 931
after gather layer_id 0 0 torch.Size([0])
after gather layer_id 0 1 torch.Size([0])
after gather layer_id 0 2 torch.Size([0])
after gather layer_id 0 3 torch.Size([0])
after gather layer_id 0 4 torch.Size([0])
after gather layer_id 0 5 torch.Size([8192])
after gather layer_id 0 6 torch.Size([2048, 8192])
after gather layer_id 0 7 torch.Size([2048])
after gather layer_id 0 8 torch.Size([0])
after gather layer_id 0 9 torch.Size([0])
after gather layer_id 0 10 torch.Size([0])
after gather layer_id 0 11 torch.Size([0])
after gather layer_id 0 12 torch.Size([0])
after gather layer_id 0 13 torch.Size([0])
after gather layer_id 0 14 torch.Size([0])
after gather layer_id 0 15 torch.Size([0])
after gather layer_id 1 0 torch.Size([2048])
after gather layer_id 1 1 torch.Size([2048])
after gather layer_id 1 2 torch.Size([2048])
after gather layer_id 1 3 torch.Size([2048])
after gather layer_id 1 4 torch.Size([8192, 2048])
after gather layer_id 1 5 torch.Size([8192])
after gather layer_id 1 6 torch.Size([2048, 8192])
after gather layer_id 1 7 torch.Size([2048])
after gather layer_id 1 8 torch.Size([2048, 2048])
after gather layer_id 1 9 torch.Size([2048])
after gather layer_id 1 10 torch.Size([2048, 2048])
after gather layer_id 1 11 torch.Size([2048])
after gather layer_id 1 12 torch.Size([2048, 2048])
after gather layer_id 1 13 torch.Size([2048])
after gather layer_id 1 14 torch.Size([2048, 2048])
after gather layer_id 1 15 torch.Size([2048])

Note that dimensions of the parameters in layer_id=0 are mostly all zero.  On that steps that complete without an error, those parameters have non-zero shapes as shown below.  The count of non_active_layers in 962 below vs 931 above.
nonactive all layers 962
after gather layer_id 0 0 torch.Size([2048])
after gather layer_id 0 1 torch.Size([2048])
after gather layer_id 0 2 torch.Size([2048])
after gather layer_id 0 3 torch.Size([2048])
after gather layer_id 0 4 torch.Size([8192, 2048])
after gather layer_id 0 5 torch.Size([8192])
after gather layer_id 0 6 torch.Size([2048, 8192])
after gather layer_id 0 7 torch.Size([2048])
after gather layer_id 0 8 torch.Size([2048, 2048])
after gather layer_id 0 9 torch.Size([2048])
after gather layer_id 0 10 torch.Size([2048, 2048])
after gather layer_id 0 11 torch.Size([2048])
after gather layer_id 0 12 torch.Size([2048, 2048])
after gather layer_id 0 13 torch.Size([2048])
after gather layer_id 0 14 torch.Size([2048, 2048])
after gather layer_id 0 15 torch.Size([2048])

By adding the following lines for further details:
https://github.com/microsoft/DeepSpeed/blob/050aee287d70157e29f242b3a629a4cb97b4e4e7/deepspeed/runtime/hybrid_engine.py#L234-L238
             else:
                from deepspeed.runtime.zero.partition_parameters import ZeroParamStatus
                rank = dist.get_rank(group=self.mp_group)

                non_active_layers = get_inactive_params(self.all_layers_params)
                if rank == 0:
                    print("nonactive layers", len(non_active_layers))
                    for lay_id, lay in enumerate(self.all_layers_params):
                        print("all layers", lay_id, hasattr(lay, 'ds_id'), lay.ds_status == ZeroParamStatus.NOT_AVAILABLE, lay.ds_status)

               non_active_lora_params = get_inactive_params(self.all_lora_params)
                if rank == 0:
                    print("nonactive lora layers", len(non_active_lora_params))
                    for lay_id, lay in enumerate(self.all_lora_params):
                        print("lora layers", lay_id, hasattr(lay, 'ds_id'), lay.ds_status == ZeroParamStatus.NOT_AVAILABLE, lay.ds_status)

                non_active_layers.extend(non_active_lora_params)

It seems that the 0-shape parameters are marked as "ds_status == ZeroParamStatus.INFLIGHT" before calling "GatheredParameters":
[2023-04-17 15:33:56,759] [INFO] [loss_scaler.py:181:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 32768, reducing to 16384
epoch: 0|step: 2|ppo_ep: 1|act_loss: nan|cri_loss: nan|unsuper_loss: 0.0
average reward score: 3.267578125
-------------------------------------------------------------------------------------
|E2E latency=17.17s |Gather latency=0.46s (2.70%) |Generate time=7.04s (41.02%) |Training time=6.82s (39.71%) |Others=3.31 (19.27%)|CurSamplesPerSec=0.93 |AvgSamplesPerSec=0.60
nonactive layers 651
all layers 0 True False ZeroParamStatus.INFLIGHT
all layers 1 True False ZeroParamStatus.INFLIGHT
all layers 2 True False ZeroParamStatus.INFLIGHT
all layers 3 True False ZeroParamStatus.INFLIGHT
all layers 4 True False ZeroParamStatus.INFLIGHT
all layers 5 True False ZeroParamStatus.INFLIGHT
all layers 6 True False ZeroParamStatus.INFLIGHT
all layers 7 True False ZeroParamStatus.INFLIGHT
all layers 8 True False ZeroParamStatus.INFLIGHT
all layers 9 True False ZeroParamStatus.INFLIGHT
all layers 10 True False ZeroParamStatus.INFLIGHT
all layers 11 True False ZeroParamStatus.INFLIGHT
all layers 12 True False ZeroParamStatus.INFLIGHT
all layers 13 True False ZeroParamStatus.INFLIGHT
all layers 14 True False ZeroParamStatus.INFLIGHT
all layers 15 True False ZeroParamStatus.INFLIGHT
all layers 16 True False ZeroParamStatus.INFLIGHT
all layers 17 True False ZeroParamStatus.INFLIGHT
all layers 18 True False ZeroParamStatus.INFLIGHT
all layers 19 True False ZeroParamStatus.INFLIGHT
all layers 20 True False ZeroParamStatus.INFLIGHT
all layers 21 True True ZeroParamStatus.NOT_AVAILABLE
all layers 22 True True ZeroParamStatus.NOT_AVAILABLE
all layers 23 True True ZeroParamStatus.NOT_AVAILABLE
all layers 24 True True ZeroParamStatus.NOT_AVAILABLE
all layers 25 True True ZeroParamStatus.NOT_AVAILABLE
all layers 26 True True ZeroParamStatus.NOT_AVAILABLE
all layers 27 True True ZeroParamStatus.NOT_AVAILABLE
all layers 28 True False ZeroParamStatus.INFLIGHT
all layers 29 True False ZeroParamStatus.INFLIGHT
all layers 30 True True ZeroParamStatus.NOT_AVAILABLE
all layers 31 True True ZeroParamStatus.NOT_AVAILABLE

<snip>

nonactive lora layers 280
lora layers 0 True True ZeroParamStatus.NOT_AVAILABLE
lora layers 1 True True ZeroParamStatus.NOT_AVAILABLE
lora layers 2 True True ZeroParamStatus.NOT_AVAILABLE
lora layers 3 True True ZeroParamStatus.NOT_AVAILABLE
lora layers 4 True False ZeroParamStatus.INFLIGHT
lora layers 5 True False ZeroParamStatus.INFLIGHT
lora layers 6 True False ZeroParamStatus.INFLIGHT
lora layers 7 True False ZeroParamStatus.INFLIGHT
lora layers 8 True False ZeroParamStatus.INFLIGHT
lora layers 9 True False ZeroParamStatus.INFLIGHT
lora layers 10 True False ZeroParamStatus.INFLIGHT
lora layers 11 True False ZeroParamStatus.INFLIGHT
lora layers 12 True True ZeroParamStatus.NOT_AVAILABLE
lora layers 13 True True ZeroParamStatus.NOT_AVAILABLE

I think those parameters are marked as INFLIGHT because they have been prefetched.
Adding some more debugging lines to print the stack at the point where the status is set to INFLIGHT:
https://github.com/microsoft/DeepSpeed/blob/050aee287d70157e29f242b3a629a4cb97b4e4e7/deepspeed/runtime/zero/partition_parameters.py#L873-L885
        def all_gather_coalesced(params: Iterable[Parameter], safe_mode: bool = True) -> AllGatherCoalescedHandle:

            # fetches from nvme if the partition is not available and in nvme
            self._ensure_availability_of_partitioned_params(params)

            if self.world_size == 1:
                return _no_gather_coalesced(params)

            #for param in params:
            for p_id, param in enumerate(params):
                if param.ds_status != ZeroParamStatus.NOT_AVAILABLE:
                    raise RuntimeError(param.ds_summary())
                param.ds_status = ZeroParamStatus.INFLIGHT
                if dist.get_rank() == 0:
                    print(p_id, "INFLIGHT2")
                    if p_id > 20:
                        print(traceback.print_stack(file=sys.stdout))

I can see those layers are set to INFLIGHT here:
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 529, in <module>
  main()
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 452, in main
  actor_loss, critic_loss = trainer.train_rlhf(exp_data)
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 180, in train_rlhf
  value = self.critic_model.forward_value(**batch,
File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/utils/model/reward_model.py", line 125, in forward_value
  transformer_outputs = self.rwtranrsformer(
File "/path/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
  result = forward_call(*input, **kwargs)
File "/path/site-packages/transformers/models/opt/modeling_opt.py", line 759, in forward
  decoder_outputs = self.decoder(
File "/path/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
    result = forward_call(*input, **kwargs)
File "/path/site-packages/transformers/models/opt/modeling_opt.py", line 665, in forward
  layer_outputs = torch.utils.checkpoint.checkpoint(
File "/path/site-packages/torch/utils/checkpoint.py", line 235, in checkpoint
  return CheckpointFunction.apply(function, preserve, *args)
File "/path/site-packages/torch/utils/checkpoint.py", line 96, in forward
  outputs = run_function(*args)
File "/path/site-packages/transformers/models/opt/modeling_opt.py", line 661, in custom_forward
  return module(*inputs, output_attentions, None)
File "/path/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
  result = forward_call(*input, **kwargs)
File "/path/site-packages/transformers/models/opt/modeling_opt.py", line 337, in forward
  hidden_states = self.activation_fn(hidden_states)
File "/path/site-packages/torch/nn/modules/module.py", line 1137, in _call_impl
  result = hook(self, input)
File "/path/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  ret_val = func(*args, **kwargs)
File "/path/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 366, in _pre_forward_module_hook
  self.pre_sub_module_forward_function(module)
File "/path/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
  return func(*args, **kwargs)
File "/path/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 478, in pre_sub_module_forward_function
  param_coordinator.fetch_sub_module(sub_module)
File "/path/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  ret_val = func(*args, **kwargs)
File "/path/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
  return func(*args, **kwargs)
File "/path/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 333, in fetch_sub_module
  self.__all_gather_params(params_to_prefetch)
File "/path/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  ret_val = func(*args, **kwargs)
File "/path/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 381, in __all_gather_params
  handle = partitioned_params[0].all_gather_coalesced(partitioned_params)
File "/path/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  ret_val = func(*args, **kwargs)
File "/path/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 878, in all_gather_coalesced
  print(traceback.print_stack(file=sys.stdout))

It seems that the layers are being prefetched during the call to the critic model forward pass:
https://github.com/microsoft/DeepSpeedExamples/blob/2aa7a31b8fdcb34b8ccdc554021a1f5789752ab3/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py#L174
They are still in INFLIGHT status when trying to generate a sample.  The get_inactive_params function then only include params marked as NOT_AVAILABLE:
https://github.com/microsoft/DeepSpeed/blob/48297c4841374776a3c4d9319a9b57378f598d65/deepspeed/runtime/utils.py#L972-L975
Later, GatheredParameters may only consider params whose state is NOT_AVAILABLE:
https://github.com/microsoft/DeepSpeed/blob/48297c4841374776a3c4d9319a9b57378f598d65/deepspeed/runtime/zero/partition_parameters.py#L1058
Assuming that diagnosis is correct, I'm not sure what the recommended fix would be.  Should get_inactive_params include INFLIGHT params?

A second question that came up while looking at this... it seems like the if conditions here might always be true:
https://github.com/microsoft/DeepSpeed/blob/48297c4841374776a3c4d9319a9b57378f598d65/deepspeed/runtime/hybrid_engine.py#L123
https://github.com/microsoft/DeepSpeed/blob/48297c4841374776a3c4d9319a9b57378f598d65/deepspeed/runtime/hybrid_engine.py#L136
Should it be lora_param instead of lora_params?  Maybe change this to:
if len(lora_param) == 3:

Assuming that diagnosis is correct, I'm not sure what the recommended fix would be. Should get_inactive_params include INFLIGHT params?

@adammoody, thanks for the detailed analysis of this bug. To answer your question, no, INFLIGHT params should not be gathered again. Param gathering is asynchronous for performance reasons, and INFLIGHT params are part of ongoing gather operations.
This problem occurs because in RLHF we are context switching 5 models in a rank to share GPU memory. And so, the recommended solution here is to ensure that there are no INFLIGHT params when context-switching out a model. This will guarantee that the actor model params are all NOT_AVAILABLE before the next generate. The empty_partition_cache() can be used for this purpose.  I will share a PR asap.

Thanks for the explanation and quick reply, @tjruwase .  Unfortunately, I'm still hitting the same problem with this PR.
The problematic params seem to be from the first few layers of the actor_model, which have been prefetched due to a forward step of the critic_model.  I thought maybe we could move those empty calls to the end to try to clear any INFLIGHT actor params that the critic started to prefetch:
        self.actor_model.backward(actor_loss)
        self.actor_model.step()
        #self.actor_model.empty_partition_cache()

        value = self.critic_model.forward_value(**batch,
                                                return_value_only=True,
                                                use_cache=False)[:, :-1]
        critic_loss = self.critic_loss_fn(value[:, start:], old_values[:,
                                                                       start:],
                                          returns, action_mask[:, start:])
        self.critic_model.backward(critic_loss)
        self.critic_model.step()
        #self.critic_model.empty_partition_cache()

        self.actor_model.empty_partition_cache()
        self.critic_model.empty_partition_cache()

However, with that change I get the following error on the self.actor_model.empty_partition_cache() call:
  File "/path/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 358, in release_and_reset_all
    raise RuntimeError(f"param {param.ds_summary()} still in flight")
RuntimeError: param {'id': 0, 'status': 'INFLIGHT', 'numel': 102957056, 'ds_numel': 102957056, 'shape': (50272, 2048), 'ds_shape': (50272, 2048), 'requires_grad': True, 'grad_shape': None, 'persist': False, 'active_sub_modules': set()} still in flight

The problematic params seem to be from the first few layers of the actor_model, which have been prefetched due to a forward step of the critic_model. I thought maybe we could move those empty calls to the end to try to clear any INFLIGHT actor params that the critic started to prefetch:

This is confusing to me. Each model has independent prefetchers, so the critic_model should not affect the actor_model. Can you share a stack trace that you get with the PR?

@adammoody, by the way, I was not able to repro your error on my 4xV100-16GB setup. This makes it harder to resolve.

However, with that change I get the following error on the self.actor_model.empty_partition_cache() call:
  File "/path/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 358, in release_and_reset_all
    raise RuntimeError(f"param {param.ds_summary()} still in flight")
RuntimeError: param {'id': 0, 'status': 'INFLIGHT', 'numel': 102957056, 'ds_numel': 102957056, 'shape': (50272, 2048), 'ds

By the way, this is a separate bug that needs to be addressed.

Should it be lora_param instead of lora_params? Maybe change this to:
if len(lora_param) == 3:

I think you have found a bug here. Do you mind opening a separate ticket for this?

Should it be lora_param instead of lora_params? Maybe change this to:
if len(lora_param) == 3:

I think you have found a bug here. Do you mind opening a separate ticket for this?

Sure.  I'll post that one to the main DeepSpeed repo.

As another clue, it seems like the following changes work around the problem.  I defined a "wait on inflight" function in deepspeed/runtime/zero/partitioned_param_coordinator.py:
    @instrument_w_nvtx
    @torch.no_grad()
    def wait_on_inflight_params(self, current_submodule):
        params = frozenset(iter_params(current_submodule, recurse=True))
        for param in params:
            if param in self.__inflight_param_registry:
                print(param.ds_summary())
                with get_accelerator().stream(self.__allgather_stream):
                    while self.__ongoing_fetch_events and self.__ongoing_fetch_events[0].query():
                        self.__ongoing_fetch_events.popleft()
                    if len(self.__ongoing_fetch_events) > self.__max_ongoing_fetch_events:
                        self.__ongoing_fetch_events.popleft().synchronize()

                    self.__inflight_param_registry.pop(param).wait()

                    event = get_accelerator().Event()
                    event.record()
                    self.__ongoing_fetch_events.append(event)

                assert param.ds_status == ZeroParamStatus.AVAILABLE, param.ds_summary()
        get_accelerator().current_stream().wait_stream(self.__allgather_stream)

I then call that from partition_all_parameters in deepspeed/runtime/zero/parameter_offload.py, which is called from empty_partition_cache:
    def partition_all_parameters(self):
        """Partitioning Parameters that were not partitioned usually if parameters
        of modules whose input parameters do not require grad computation do not
        trigger post call and will therefore will remain unpartitioned"""
        self.get_param_coordinator(training=self.module.training).wait_on_inflight_params(self.module)
        self.get_param_coordinator(training=self.module.training).release_and_reset_all(self.module)

And then I call empty_partition_cache on both models after training on both:
        self.actor_model.step()

        value = self.critic_model.forward_value(**batch,
                                                return_value_only=True,
                                                use_cache=False)[:, :-1]
        critic_loss = self.critic_loss_fn(value[:, start:], old_values[:,
                                                                       start:],
                                          returns, action_mask[:, start:])
        self.critic_model.backward(critic_loss)
        self.critic_model.step()

        self.actor_model.empty_partition_cache()
        self.critic_model.empty_partition_cache()

If I drop the recurse=True option to change:
params = frozenset(iter_params(current_submodule, recurse=True))

to:
params = frozenset(iter_params(current_submodule))

then I still get this error:
  File "/path/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 400, in release_and_reset_all
    raise RuntimeError(f"param {param.ds_summary()} still in flight")
RuntimeError: param {'id': 0, 'status': 'INFLIGHT', 'numel': 102957056, 'ds_numel': 102957056, 'shape': (50272, 2048), 'ds_shape': (50272, 2048), 'requires_grad': True, 'grad_shape': None, 'persist': False, 'active_sub_modules': set()} still in flight

It seems that it missing the necessary parameters in that case.

And if I keep recurse=True, but call empty_partition_cache after each train step individually rather than at the end of both steps like so:
        ### process the new outputs
        batch = {'input_ids': seq, "attention_mask": attention_mask}
        actor_prob = self.actor_model(**batch, use_cache=False).logits
        actor_log_prob = gather_log_probs(actor_prob[:, :-1, :],
                                          inputs['input_ids'][:, 1:])
        actor_loss = self.actor_loss_fn(actor_log_prob[:, start:],
                                        log_probs[:, start:], advantages,
                                        action_mask[:, start:])
        self.actor_model.backward(actor_loss)
        self.actor_model.step()
        self.actor_model.empty_partition_cache()

        value = self.critic_model.forward_value(**batch,
                                                return_value_only=True,
                                                use_cache=False)[:, :-1]
        critic_loss = self.critic_loss_fn(value[:, start:], old_values[:,
                                                                       start:],
                                          returns, action_mask[:, start:])
        self.critic_model.backward(critic_loss)
        self.critic_model.step()
        self.critic_model.empty_partition_cache()

        #self.actor_model.empty_partition_cache()
        #self.critic_model.empty_partition_cache()

I still get the original error noted at the top of the issue:
    weight.data += lora_scaling * torch.matmul(lora_left_weight.t(), lora_right_weight.t())
RuntimeError: The size of tensor a (8192) must match the size of tensor b (2048) at non-singleton dimension 1

In summary, it seems that I need to do all three of these:

call empty_partition_cache after both steps
wait on any inflight params
call iter_params(..., recurse=True) when getting the parameter list

Thanks for sharing these details. I agree that empty_partition_cache needs a wait_on_inflight_params logic like you discovered. However, I would like to take a step back to understand a few things.
First, empty_partition_cache should guarantee that all params are NOT_AVAILABLE. But you mentioned that you were hitting the original problem with my PR which calls empty_partition_cache after actor_model.step(). I don't understand how that can possible. So, could you please share the stack trace of applying my PR?

The error message and stack trace when using the changes in the PR are the same as the original error report.  Right, I haven't given up on figuring out this problem either.  I have some more ideas to try to debug things.  I'll keep posting updates.

@tjruwase , I still haven't cracked it, but here are some more clues...
The first problematic layer corresponds to the vocab embedding layer of the actor model.  I did verify that layer actually belongs to the actor model, and that it is not shared with the critic model or any other model.
The stack trace for the prefetch of that layer is shown below.  The line numbers will vary because I've added lots of debug statements.
  File "/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 529, in <module>
    main()
  File "/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 452, in main
    actor_loss, critic_loss = trainer.train_rlhf(exp_data)
  File "/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 191, in train_rlhf
    value = self.critic_model.forward_value(**batch,
  File "/DeepSpeedExamples/applications/DeepSpeed-Chat/training/utils/model/reward_model.py", line 125, in forward_value
    transformer_outputs = self.rwtranrsformer(
  File "/path/python3.9/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/path/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 759, in forward
    decoder_outputs = self.decoder(
  File "/path/python3.9/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/path/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 665, in forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/path/python3.9/site-packages/torch/utils/checkpoint.py", line 235, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/path/python3.9/site-packages/torch/utils/checkpoint.py", line 96, in forward
    outputs = run_function(*args)
  File "/path/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 661, in custom_forward
    return module(*inputs, output_attentions, None)
  File "/path/python3.9/site-packages/torch/nn/modules/module.py", line 1148, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/path/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 337, in forward
    hidden_states = self.activation_fn(hidden_states)
  File "/path/python3.9/site-packages/torch/nn/modules/module.py", line 1137, in _call_impl
    result = hook(self, input)
  File "/path/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/path/python3.9/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 378, in _pre_forward_module_hook
    self.pre_sub_module_forward_function(module)
  File "/path/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/path/python3.9/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 492, in pre_sub_module_forward_function
    param_coordinator.fetch_sub_module(sub_module)
  File "/path/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/path/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/path/python3.9/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 368, in fetch_sub_module
    print(traceback.print_stack(file=sys.stdout))

Note that this happens from a param_coordinator.fetch_sub_module(sub_module) call.
That trace was kicked off by a call to:
value = self.critic_model.forward_value()

The ds_summary output for that the param being prefetched is:
-prefetch: {'id': 0, 'status': 'NOT_AVAILABLE', 'numel': 0, 'ds_numel': 102957056, 'shape': (0,), 'ds_shape': (50272, 2048), 'requires_grad': True, 'grad_shape': None, 'persist': False, 'active_sub_modules': set()}

I did verify that this layer belongs to the actor model by matching its id() value and similarly verified that it's not shared with the critic model.
I can see that it fails for me on what I think is the third training step.  This appears to be the first step where it has completed a trace and thus has enabled prefetching for the model.
Since there seems to be some "model mixing" in this case, one area that caught my eye is the global FWD_MODULE_STACK in deepspeed/runtime/zero/parameter_offload.py.
    @torch.no_grad()
    def pre_sub_module_forward_function(self, sub_module):
        see_memory_usage(f"Before sub module function {sub_module.__class__.__name__}", force=False)

        global FWD_MODULE_STACK
        FWD_MODULE_STACK.append(sub_module)
        if dist.get_rank() == 0:
            print("FWD_MODULE_STACK length", len(FWD_MODULE_STACK), "id(module)", id(sub_module))

        param_coordinator = self.get_param_coordinator(training=sub_module.training)
        param_coordinator.trace_prologue(sub_module)
        if param_coordinator.is_record_trace():
            param_coordinator.record_module(sub_module)
        param_coordinator.fetch_sub_module(sub_module)

You can see I've added a print above.  The vocab embedding layer is the 8th or 9th element at the time the problem occurs.  I'd have to double check which if you need an exact value.
I can see that the four base models correspond to the first four slots of that list.
    def setup_zero_stage3_hooks(self):
        self.hierarchy = 0

        #reset step if in inference mode
        @instrument_w_nvtx
        def _end_of_forward_hook(module, *args):

            if not torch._C.is_grad_enabled():
                self.get_param_coordinator(training=False).reset_step()

        #likely one of them should be enough but just to be safe
        self._register_hooks_recursively(self.module)
        self.module.register_forward_hook(_end_of_forward_hook)

        # Add top module to stack trace
        global FWD_MODULE_STACK
        FWD_MODULE_STACK.append(self.module)
        if dist.get_rank() == 0:
            print("FWD_MODULE_STACK length", len(FWD_MODULE_STACK), "id(module)", id(self.module))
            print(str(self.module))

I'm not sure what modules are in between off hand when I hit the problem.
I still haven't tracked down why code invoked on the critic model could end up fetching params for the actor model, but I wondered if there might be some linkage here.

@adammoody, kudos on the intensive debugging. I think I know what might be wrong, but I need your help to confirm. I have updated my PR with some asserts to verify that empty_partition_cache() is behaving as expected. Can you please try the PR again?

I added those new changes by hand, so my source file line numbers will be different.  I have been editing DeepSpeed files in place within my python environment, so it takes some effort to set up a clean environment at this point.  Anyway, if you trust that, I hit the tensor dimension mismatch here:
https://github.com/microsoft/DeepSpeedExamples/blob/ce049bee82bd4594209beb2bc0676a44af2b5758/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py#L77
6: 0: Traceback (most recent call last):
6: 0:   File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 529, in <module>
6: 0:     main()
6: 0:   File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 438, in main
6: 0:     out = trainer.generate_experience(prompts)
6: 0:   File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 106, in generate_experience
6: 0:     seq = self._generate_sequence(prompts)
6: 0:   File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 79, in _generate_sequence
6: 0:     seq = self.actor_model.module.generate(prompts,
6: 0:   File "/path/python3.9/site-packages/deepspeed/runtime/hybrid_engine.py", line 293, in generate
6: 0:     self.fuse_lora_weight()
6: 0:   File "/path/python3.9/site-packages/deepspeed/runtime/hybrid_engine.py", line 139, in fuse_lora_weight
6: 0:     self._fuse_lora(self.layer_params[layer_id], self.lora_params[layer_id])
6: 0:   File "/path/python3.9/site-packages/deepspeed/runtime/hybrid_engine.py", line 128, in _fuse_lora
6: 0:     weight.data += lora_scaling * torch.matmul(lora_left_weight.t(), lora_right_weight.t())
6: 0: RuntimeError: The size of tensor a (8192) must match the size of tensor b (2048) at non-singleton dimension 1

If I add a second call to assert_empty_partition_cache for the actor_model immediately after the check for the critic model:
        ### process the new outputs
        batch = {'input_ids': seq, "attention_mask": attention_mask}
        actor_prob = self.actor_model(**batch, use_cache=False).logits
        actor_log_prob = gather_log_probs(actor_prob[:, :-1, :],
                                          inputs['input_ids'][:, 1:])
        actor_loss = self.actor_loss_fn(actor_log_prob[:, start:],
                                        log_probs[:, start:], advantages,
                                        action_mask[:, start:])
        self.actor_model.backward(actor_loss)
        self.actor_model.step()
        self.actor_model.empty_partition_cache()
        assert_empty_partition_cache(self.actor_model, 'actor_model after rlhf step')

        value = self.critic_model.forward_value(**batch,
                                                return_value_only=True,
                                                use_cache=False)[:, :-1]
        critic_loss = self.critic_loss_fn(value[:, start:], old_values[:,
                                                                       start:],
                                          returns, action_mask[:, start:])
        self.critic_model.backward(critic_loss)
        self.critic_model.step()
        self.critic_model.empty_partition_cache()
        assert_empty_partition_cache(self.critic_model, 'critic_model after rlhf step')
-->     assert_empty_partition_cache(self.actor_model, 'actor_model after rlhf critic step')

then the assertion triggers:
  File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 204, in train_rlhf
    actor_loss, critic_loss = trainer.train_rlhf(exp_data)
  File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 204, in train_rlhf
    assert_empty_partition_cache(self.actor_model, 'actor_model after rlhf critic step')
  File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 290, in assert_empty_partition_cache
    assert_empty_partition_cache(self.actor_model, 'actor_model after rlhf critic step')
  File "/path/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 290, in assert_empty_partition_cache
    assert len(avail_or_inflight_params) == 0, \
AssertionError: actor_model after rlhf critic step empty_partition_cache failed to evict all params: remaining = [0, 1, 2, 3, 387, 388, 4, 5, 389, 390, 6, 7, 391, 392, 8, 9, 393, 394, 10, 11, 12, 16, 17]

Yes, I'm actually using a 350m model for the critic.  I had a cut-and-paste typo in the path name when I wrote out the checkpoint, so the path suggests it's a 1.3b param model, but it is really 350m.
I tried dropping the --enable_hybrid_engine option.  I do still hit the assertion.  I think it triggered one step earlier than before.
AssertionError: actor_model after rlhf critic step empty_partition_cache failed to evict all params: remaining = [0, 1, 2, 3, 387, 388, 4, 5, 389, 390, 6, 7, 391, 392, 8, 9, 393, 394, 10, 11, 12, 16, 17]

Here are some other work arounds that I found earlier but didn't list yet:

dropping from stage 3 to stage 2 works

--actor_zero_stage 2
--critic_zero_stage 2

hard-coding the prefetch buffer to be 0-size works (stage 3 w/ disable prefetch)
https://github.com/microsoft/DeepSpeed/blob/d92539509b1e9a6178cfdb921d5080e76f690bce/deepspeed/runtime/zero/parameter_offload.py#L242

#self._prefetch_bucket_sz = int(prefetch_bucket_size)
self._prefetch_bucket_sz = 0

Thanks for the update.

Hitting the assertion is good since it stops at the earliest violation of the invariant of empty_partition_cache().
It is good to know that --enable_hybrid_engine is not the cause.
Yes, stage 2 should not cause this since it is stage 3 specific.
Disabling prefetching (can be done through ds_config) should avoid this problem.

@adammoody, FYI I think this DeepSpeed PR from my colleague @HeyangQin might be relevant here. Please give him a bit more time to get it ready.

@tjruwase , I think I found the cause.
I believe the problem is that all four models share the same ReLU module object.  Each model registers a forward hook on that module.  When invoking the forward pass on the ReLU module from the critic model, the hook from the actor model is invoked, which leads to the prefetch of the actor layers.
I found this by adding the following code in deepspeed/runtime/zero/parameter_offload.py to print object addresses of all child modules of each model:
def print_children(module, indent):
  for name, m in module.named_children():
    spaces = " " * indent
    print(spaces, name, id(m))
    print_children(m, indent + 2)

<snip>

    def setup_zero_stage3_hooks(self):
        self.hierarchy = 0

        #reset step if in inference mode
        @instrument_w_nvtx
        def _end_of_forward_hook(module, *args):

            if not torch._C.is_grad_enabled():
                self.get_param_coordinator(training=False).reset_step()

        #likely one of them should be enough but just to be safe
        self._register_hooks_recursively(self.module)
        self.module.register_forward_hook(_end_of_forward_hook)

        # Add top module to stack trace
        global FWD_MODULE_STACK
        FWD_MODULE_STACK.append(self.module)
        if dist.get_rank() == 0:
            print("FWD_MODULE_STACK SETUP length", len(FWD_MODULE_STACK), "id(module)", id(self.module), type(self.module))
            print(str(self.module))
            for p_id, param in enumerate(iter_params(self.module, recurse=True)):
              key = id(param) if hasattr(param, 'ds_id') else id(param.ds_param_alias)
              print("  ", p_id, id(param), type(param), key)
            print_children(self.module, 2)

With that, I get the following example output for the actor and reward models.  You can see that the ReLU module has the same address for all layers in all models.
FWD_MODULE_STACK SETUP length 1 id(module)<PHONE_NUMBER>3104 <class 'transformers.models.opt.modeling_opt.OPTForCausalLM'>
3: 0:    model<PHONE_NUMBER>4992
3: 0:      decoder<PHONE_NUMBER>7776
3: 0:        embed_tokens<PHONE_NUMBER>7248
3: 0:        embed_positions<PHONE_NUMBER>7440
3: 0:        layers<PHONE_NUMBER>6144
3: 0:          0<PHONE_NUMBER>7392
3: 0:            self_attn<PHONE_NUMBER>4800
3: 0:              k_proj<PHONE_NUMBER>6480
3: 0:                lora_dropout<PHONE_NUMBER>6576
3: 0:              v_proj<PHONE_NUMBER>4320
3: 0:                lora_dropout<PHONE_NUMBER>8880
3: 0:              q_proj<PHONE_NUMBER>4848
3: 0:                lora_dropout<PHONE_NUMBER>8832
3: 0:              out_proj<PHONE_NUMBER>7296
3: 0:                lora_dropout<PHONE_NUMBER>6912
3: 0: -->        activation_fn<PHONE_NUMBER>7472  <--- same address for all layers, in each model
3: 0:            self_attn_layer_norm<PHONE_NUMBER>5040
3: 0:            fc1<PHONE_NUMBER>5616
3: 0:              lora_dropout<PHONE_NUMBER>8736
3: 0:            fc2<PHONE_NUMBER>4656
3: 0:              lora_dropout<PHONE_NUMBER>0560
3: 0:            final_layer_norm<PHONE_NUMBER>4336
3: 0:          1<PHONE_NUMBER>3472
3: 0:            self_attn<PHONE_NUMBER>3328
3: 0:              k_proj<PHONE_NUMBER>3984
3: 0:                lora_dropout<PHONE_NUMBER>6880
3: 0:              v_proj<PHONE_NUMBER>4432
3: 0:                lora_dropout<PHONE_NUMBER>6592
3: 0:              q_proj<PHONE_NUMBER>2800
3: 0:                lora_dropout<PHONE_NUMBER>7408
3: 0:              out_proj<PHONE_NUMBER>1264
3: 0:                lora_dropout<PHONE_NUMBER>7456
3: 0: -->        activation_fn<PHONE_NUMBER>7472  <--- same address for all layers, in each model
3: 0:            self_attn_layer_norm<PHONE_NUMBER>1456
3: 0:            fc1<PHONE_NUMBER>2512
3: 0:              lora_dropout<PHONE_NUMBER>7696
3: 0:            fc2<PHONE_NUMBER>1888
3: 0:              lora_dropout<PHONE_NUMBER>6160
3: 0:            final_layer_norm<PHONE_NUMBER>5504

<snip>

FWD_MODULE_STACK SETUP length 4 id(module)<PHONE_NUMBER>8272 <class 'utils.model.reward_model.RewardModel'>
3: 0:    v_head<PHONE_NUMBER>7440
3: 0:    rwtranrsformer<PHONE_NUMBER>8224
3: 0:      decoder<PHONE_NUMBER>8320
3: 0:        embed_tokens<PHONE_NUMBER>8416
3: 0:        embed_positions<PHONE_NUMBER>8368
3: 0:        project_out<PHONE_NUMBER>8512
3: 0:        project_in<PHONE_NUMBER>8560
3: 0:        layers<PHONE_NUMBER>8656
3: 0:          0<PHONE_NUMBER>8608
3: 0:            self_attn<PHONE_NUMBER>8704
3: 0:              k_proj<PHONE_NUMBER>8800
3: 0:              v_proj<PHONE_NUMBER>8848
3: 0:              q_proj<PHONE_NUMBER>8992
3: 0:              out_proj<PHONE_NUMBER>9040
3: 0: -->        activation_fn<PHONE_NUMBER>7472 <--- same address for all layers, in each model
3: 0:            self_attn_layer_norm<PHONE_NUMBER>8752
3: 0:            fc1<PHONE_NUMBER>9136
3: 0:            fc2<PHONE_NUMBER>9184
3: 0:            final_layer_norm<PHONE_NUMBER>9232
3: 0:          1<PHONE_NUMBER>9328
3: 0:            self_attn<PHONE_NUMBER>9376
3: 0:              k_proj<PHONE_NUMBER>9472
3: 0:              v_proj<PHONE_NUMBER>9520
3: 0:              q_proj<PHONE_NUMBER>4320
3: 0:              out_proj<PHONE_NUMBER>4368
3: 0: -->        activation_fn<PHONE_NUMBER>7472 <--- same address for all layers, in each model
3: 0:            self_attn_layer_norm<PHONE_NUMBER>9424
3: 0:            fc1<PHONE_NUMBER>7552
3: 0:            fc2<PHONE_NUMBER>7792
3: 0:            final_layer_norm3: 0:<PHONE_NUMBER>7840

As a test, I then found that I could work around the problem if I modify the OPT model to instantiate a unique ReLU object for each layer in transformers/models/opt/modeling_opt.py:
class OPTDecoderLayer(nn.Module):
    def __init__(self, config: OPTConfig):
        super().__init__()
        self.embed_dim = config.hidden_size
        self.self_attn = OPTAttention(
            embed_dim=self.embed_dim,
            num_heads=config.num_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=True,
        )
        self.do_layer_norm_before = config.do_layer_norm_before
        self.dropout = config.dropout
-->     #self.activation_fn = ACT2FN[config.activation_function]
-->     self.activation_fn = nn.ReLU()

@tjruwase , I think I found the cause.
I believe the problem is that all four models share the same ReLU module object. Each model registers a forward hook on that module in setup_zero_stage3_hooks(). When invoking the forward pass on the ReLU module from the critic model, the hook from the actor model is invoked, which leads to the prefetch of the actor layers.

Amazing debugging, @adammoody. Truly outstanding!
@stas00, FYI. It seems ReLU objects are shared across models in the same transformer process. Do you have context for this behavior?

yes, it's the same object:
https://github.com/huggingface/transformers/blob/b6865b9befad33f99adee0a6ef6361f72fcc8b42/src/transformers/activations.py#L206-L233
https://github.com/huggingface/transformers/blob/b6865b9befad33f99adee0a6ef6361f72fcc8b42/src/transformers/models/opt/modeling_opt.py#L288

raise RuntimeError(f"still have inflight params "f"{[p.ds_summary for p in self.__inflight_param_registry.keys()]}")
this error is still reported when I running step3 use bloomz + zero3.
I encountered this bug after the previous bug (https://github.com/microsoft/DeepSpeed/issues/3528) was solved. @HeyangQin

raise RuntimeError(f"still have inflight params "f"{[p.ds_summary for p in self.__inflight_param_registry.keys()]}") this error is still reported when I running step3 use bloomz + zero3.
I encountered this bug after the previous bug (microsoft/DeepSpeed#3528) was solved. @HeyangQin

I have the same issue. Have you resolved the problem?

raise RuntimeError(f"still have inflight params "f"{[p.ds_summary for p in self.__inflight_param_registry.keys()]}") this error is still reported when I running step3 use bloomz + zero3.
I encountered this bug after the previous bug (microsoft/DeepSpeed#3528) was solved. @HeyangQin

Encountering the same error here. The issue persists even after updating DeepSpeed and PyTorch Lightning to the latest versions.


Samples with Toolbox for v2.0.100

Provide better samples with some common workflows for publishes, managing and modifying hosted online content on either portal or AGOL.

closed, sample in samples folder


Makes Lizard animals REAL

Lizards  no longer look like ugly worms but more like komodos
About the PR
Screenshots

Changelog

:cl: Lizard Master

tweak: Lizards (animal) now look like Komodos instead of worms

but soul sprite

actually this is fine if you readd the old lizard sprites as a snake or something

Sorry but I don't add garbage :P

slug time


remove Manifest.toml (add to .gitignore)

Other packages (JuMP, MOI, Dualization etc) don't commit Manifest.toml, only Project.toml.
You can see the .gitignore of these packages.

For a set of tutorials I think it is very good to have a Manifest checked in so that you know that the users running the tutorial are running on the same version of the packages as the tutorial was created. This project also has zero [compat] entries so the tutorials here will definitely stop working as soon as any of the dependencies make a breaking change.

@chriscoey I added some compat bounds on Project toml on the "essential" dependencies of the project, so that we can keep manifest and up it without breaking (if all goes well)

I'm not able to run the notebooks with IJulia (I get a dead kernel) unless I remove the Manifest.toml file.  I've seen this with Julia 1.4.2 on both Windows 10 and Ubuntu 20.04.

the manifest is run with Julia 1.0 normally to ensure compatibility with the LTS

Closing as fixed.


[Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not 1 and max_tokens is large & Add tests for preemption

When refactoring the scheduler, there was the wrong assumption that num_new_tokens are always 1. It was detected and fixed in schedule_running, but it wasn't properly tested for swapping case, so it wasn't deleted.
This PR fixes the issue by removing the assertion. This PR also allows for scheduler to inject artificial preemption, so that we can do e2e testing for swapping and preemtpion. (I decided to add artificial injection code instead of calculating kv cache size to be intentionally small cuz I thought it is more prone to regression)
Fixes https://github.com/vllm-project/vllm/issues/4422

 PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.
What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

 After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
 After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
 After the review, the reviewer will put an  action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
 Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You
 Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

@cadedaniel for some reasons swapping test doesn't trigger preemption. I will fix it and ping you

@cadedaniel new approaach was a little difficult to make it work with swap test, so I just reverted to the original approach.

what's up with the swap test? happy to meet to get to the bottom of it

Discussed offline -- seems there's some more issues with beam search and swap. we can follow up later.


Add explicit JsonProperty annotation in ProxyConfiguration#type()

This change is

This allows ProxyConfiguration to be parsed when a KebabCaseStrategy is being used.


adding PR template and codeowners file

Rationale
PR Templates help contributors to communicate a set of information when submitting a new pull request. We've used this template in a few other repos, so let's add this here.
Changes

Add PR Template

Testing
PR template used in this PR description

Added CODEOWNERS as requested : )


[ESP32-C3] Cannot select UART1 for log output (IDFGH-13844)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.1.1 (e088c3766ba440e72268b458a68f27b6e7d63986)
Operating System used.
Linux
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
What is the expected behavior?
For ESP32-C3, I cannot select UART 1 as output for console in menuconfig. If I select Custom Uart in ESP System Settings, I can change the GPIO pins for RX and TX, but the option to change the UART device number is disabled, as shown in screenshot:

Option UART peripheral to use for console output should be enabled an I should be able to select 1.
What is the actual behavior?
Option UART peripheral to use for console output is disable and thus I cannot select UART 1.
Steps to reproduce.

Run idf.py menuconfig
Go to Component config -> ESP System Settings
In option Channel for console output select Custom UART.
Optionally change Channel for console secondary output, the result will be the same.
Option UART peripheral to use for console output is not available, cannot select UART1

Build or installation Logs.
No response
More Information.
No response

Support for this has already been added in 46d33e46efdf368f3f77d0879804a3f3283474bb, so it is available on newer release branches

Oh, thanks! This is not merged into 5.1.1, right? So I have to upgrade to 5.3?

Yup, that is correct.


Bug fix: unhandled exception during AjaxSelect load

There is an unhandled exception in AjaxSelect2Widget after any exception in model_view.insert_model.
The error occurs when you pick something in the Ajax select box and there is an exception in the overridden insert_model method. Here is a part of the traceback:
  File "/Users/diskream/Library/Caches/pypoetry/virtualenvs/aMY0E6K2-py3.12/lib/python3.12/site-packages/sqladmin/widgets.py", line 57, in __call__
    data = field.loader.format(field.data)
           │     │      │      │     └ <property object at 0x110181300>
           │     │      │      └ <sqladmin.fields.AjaxSelectField object at 0x110903b90>
           │     │      └ <function QueryAjaxModelLoader.format at 0x1101089a0>
           │     └ <sqladmin.ajax.QueryAjaxModelLoader object at 0x1101a0b30>
           └ <sqladmin.fields.AjaxSelectField object at 0x110903b90>
  File "/Users/diskream/Library/Caches/pypoetry/virtualenvs/aMY0E6K2-py3.12/lib/python3.12/site-packages/sqladmin/ajax.py", line 59, in format
    return {"id": str(get_object_identifier(model)), "text": str(model)}
                      │                     │                    └ '1'
                      │                     └ '1'
                      └ <function get_object_identifier at 0x110108360>
  File "/Users/diskream/Library/Caches/pypoetry/virtualenvs/aMY0E6K2-py3.12/lib/python3.12/site-packages/sqladmin/helpers.py", line 183, in get_object_identifier
    primary_keys = get_primary_keys(obj)
                   │                └ '1'
                   └ <function get_primary_keys at 0x1101082c0>
  File "/Users/diskream/Library/Caches/pypoetry/virtualenvs/aMY0E6K2-py3.12/lib/python3.12/site-packages/sqladmin/helpers.py", line 178, in get_primary_keys
    return tuple(inspect(model).mapper.primary_key)
                 │       └ '1'
                 └ <function inspect at 0x106b33f60>
  File "/Users/diskream/Library/Caches/pypoetry/virtualenvs/aMY0E6K2-py3.12/lib/python3.12/site-packages/sqlalchemy/inspection.py", line 147, in inspect
    raise exc.NoInspectionAvailable(
          │   └ <class 'sqlalchemy.exc.NoInspectionAvailable'>
          └ <module 'sqlalchemy.exc' from '/Users/diskream/Library/Caches/pypoetry/virtualenvs/aMY0E6K2-py3.12/lib/python3.1...

sqlalchemy.exc.NoInspectionAvailable: No inspection system is available for object of type <class 'str'>

As you can see, the widget receives a string from the form, not the model. A common approach is to handle exceptions at the time of parsing the form.
Here is snippet to reproduce error:
from typing import Any

from sqlalchemy import Column, Integer, ForeignKey, String
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
from sqlalchemy.orm import declarative_base, sessionmaker, relationship
from starlette.applications import Starlette
from starlette.requests import Request

from sqladmin import Admin, ModelView

async_engine = create_async_engine("sqlite+aiosqlite:///test.db")

Base = declarative_base()  # type: Any
session_maker = sessionmaker(bind=async_engine, class_=AsyncSession, expire_on_commit=False)

app = Starlette()

@app.on_event("startup")
async def startup() -> None:
    async with async_engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

admin = Admin(app=app, engine=async_engine)

class User(Base):
    __tablename__ = "users"

    id = Column(Integer, primary_key=True)
    name = Column(String(length=16))

    addresses = relationship("Address", back_populates="user")

    def __str__(self) -> str:
        return f"User {self.id}"

class Address(Base):
    __tablename__ = "addresses"

    id = Column(Integer, primary_key=True)
    user_id = Column(Integer, ForeignKey("users.id"))

    user = relationship("User", back_populates="addresses")

    def __str__(self) -> str:
        return f"Address {self.id}"

class UserAdmin(ModelView, model=User):
    form_columns = ("name",)
    form_ajax_refs = {
        "addresses": {
            "fields": ("id",),
        }
    }

class AddressAdmin(ModelView, model=Address):
    form_ajax_refs = {
        "user": {
            "fields": ("name",),
            "order_by": ("id"),
        }
    }

    async def insert_model(self, request: Request, data: dict) -> Any:
        1 / 0

admin.add_view(UserAdmin)
admin.add_view(AddressAdmin)

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="<IP_ADDRESS>", port=8000)

Thanks for the PR, I have a question since there's no issue related to this.

I decided to make a PR, not issue because I already have some workaround about it and provided snippet to reproduce. Should I have created an issue first?

Sorry I think my comment was discarded, my question was that how is this going to fix or workaround the issue?
This will just ignore the exception right?

Yes, my solution is to clear ajax field if there is any error on field loading. May be this is not the best solution, but due to there was no exception handling, I think there is no need to do something more serious here.

I agree. These changes will not have any negative side effects. It would be good to see this fix in the main branch.

@aminalaee
I have the same problem, how soon will it be fixed?
This my code:
class FileAdmin(ModelView, model=File):
    name = "Файл для игры"
    name_plural = "Файлы для игры"
    icon = "fa-solid fa-file"
    column_list = [
        File.file_id,
        File.path,
    ]
    form_columns = [
        File.game,
        File.path
    ]
    form_ajax_refs = {
        "game": {
            "fields": ("game_id", "order_id", "keyword"),
            "order_by": "game_id"
        }
    }

    async def on_model_change(self, data: dict, model: Mailing, is_created: bool, request: Request):
        if regex.search(pattern=r"\p{IsCyrillic}", string=data["path"].filename):
            raise ValueError(
                "Название файла не должно содержать кириллицу."
            )

When a user uses Cyrillic in the file names, a ValueError exception is thrown, but it cannot be handled properly:
sqlalchemy.exc.NoInspectionAvailable: No inspection system is available for object of type <class 'str'>

And the admin panel does not load:

@Mat0mba24 I don't think your issue is related to ajax, you are raising the error in on_model_change method.

@aminalaee it is.I debugged the admin on error raising in insert_model, update_model and delete_model when Ajax was turned on. It was in the Ajax field load that received a field value (in @Mat0mba24 case it was a Cyrillic string), instead of the Ajax field. I have provided a snippet to reproduce the issue.

@aminalaee
I'm removing ajax:
class FileAdmin(ModelView, model=File):
    name = "Файл для игры"
    name_plural = "Файлы для игры"
    icon = "fa-solid fa-file"
    column_list = [
        File.file_id,
        File.path,
    ]
    form_columns = [
        File.game,
        File.path
    ]
    # form_ajax_refs = {
    #     "game": {
    #         "fields": ("game_id", "order_id", "keyword"),
    #         "order_by": "game_id"
    #     }
    # }

    async def on_model_change(self, data: dict, model: File, is_created: bool, request: Request):
        if regex.search(pattern=r"\p{IsCyrillic}", string=data["path"].filename):
            raise ValueError(
                "Название файла не должно содержать кириллицу."
            )

And everything works as intended:

I'm returning it:
class FileAdmin(ModelView, model=File):
    name = "Файл для игры"
    name_plural = "Файлы для игры"
    icon = "fa-solid fa-file"
    column_list = [
        File.file_id,
        File.path,
    ]
    form_columns = [
        File.game,
        File.path
    ]
    form_ajax_refs = {
        "game": {
            "fields": ("game_id", "order_id", "keyword"),
            "order_by": "game_id"
        }
    }

    async def on_model_change(self, data: dict, model: File, is_created: bool, request: Request):
        if regex.search(pattern=r"\p{IsCyrillic}", string=data["path"].filename):
            raise ValueError(
                "Название файла не должно содержать кириллицу."
            )

And again the same mistake:

It was a bit confusing but at least I know what the issue is now.
Theform_ajax_refs creates the special HTML form and widget with Select2. When there's an error with this form, we only have the string PK of the model, and we don't have access to the User object instance, in this case.
The suggested solution works around this, but it causes a side-effect that when we are redirected to the form again, we lose the state of this select2 input because we set data to None.

Researching the same bug, I discover a possible source of the error
Shouldn't there be self.model instead of model? Its exactly represents the model we needed to get an identifier from
https://github.com/aminalaee/sqladmin/blob/3b1a355bd0f6bd0aee22222f37b4cb858f93c0c2/sqladmin/ajax.py#L61
return {"id": str(get_object_identifier(self.model)), "text": str(model)}


Issues with PowerMock

This toolkit uses PowerMock in order to test its methods.
There are some problems with PowerMock described in the documentation.
We should find a way to fix those problems. Right now it seems is not possible to execute test on Java8 (Java7 requires a extra command to disable the bytecode verification.

Are you open to removing PowerMock completely? I have found that it is not necessary to use byte code manipulation when the code being tested is designed with testing in mind.
For java-saml this might mean introducing some interfaces instead of using classes directly.

If you review the unit test you will see that I added PowerMock in order to be able to simulate HTTP Requests. How is your proposal to manage it?
Im open to suggestions, I dont consider me as a Java expert.
This java-saml toolkit is a port of the PHP/Python toolkits and I tried to implement as similar as possible in order to help me with the global maintenance so when I include a new feature in one I port the functionality on the rest.

I think we can close this :)


Dictionary Provider is broken.

https://api.dictionaryapi.dev times out. Looks like it is just a frontend for a Google API, which may either be rate limited, or removed.
curl -v https://api.dictionaryapi.dev
*   Trying <IP_ADDRESS>:443...
... hangs ...

Working for me.


Backport the zict-based dict-like for RE.md

This will allow configuration files to be compatible with 1.6.0 and 1.5.x.

Closed by #1311


React 16 + Ref Support

I've been spending a bit of time this week seeing if we can get SeatGeek working with Percy for a project I'm working on. I'm pretty close, but was running into a few issues which I've addressed in this PR:

@percy/react was restricting peer dependencies to only React v15, but React v16 has been out for a year. I've updated the peer dependencies allow both v15 and v16, and updated the unit tests here to run against v16.

I discovered that refs don't work properly because we were loading two copies of React on the page when snapshotting (it was included in both the snapshots and render bundles). I've updated the webpack config to extract react and react-dom into a vendor bundle to prevent it from loading multiple times, and added some integration tests here to verify that refs work. The tests failed before my changes, and now render properly with my updates.

@timhaines any ideas why the create-react-app snapshots are rendering blank? I ran react-percy --debug on that integration test project locally and loaded it up, and everything looks fine. Are some scripts not getting uploaded properly or something?


resolve deprecation warning in conformance_pack sub-module

what
When using the conformance pack module via examples/hipaa I get the following deprecation warning:
$ terraform plan -var-file=fixtures.us-east-2.tfvars
[snip]

Plan: 16 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + config_recorder_id = (known after apply)
  + storage_bucket_arn = (known after apply)
  + storage_bucket_id  = (known after apply)
╷
│ Warning: Deprecated attribute
│
│   on ../../modules/conformance-pack/main.tf line 12, in resource "aws_config_conformance_pack" "default":
│   12:   template_body = data.http.conformance_pack.body
│
│ The attribute "body" is deprecated. Refer to the provider documentation for details.
╵

why
The body parameter in the Terraform http data resource has been deprecated in favor of request_body
references
https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http

/terratest

/terratest

/terratest


add ECDH functions

:+1: would be very nice to have. Right now I have to use custom OpenSSL wrapper for that: https://github.com/bitchan/eccrypto/blob/master/ecdh.cc

@Kagami I add ECDH in v2.0.0, but secp256k1 implementation differs from yours, because apply sha256 to result..

Indeed.. It's a bit unfortunate, I guess I'm locked to OpenSSL for a while.
I'm wondering why would they do that, elliptic doesn't hash derived secret for example. Probably some additional security measure.

@Kagami I'm guessing since it multiplying two points together, a publicKey and privateKey, that the result isn't necessarily going to result in an uniform distribution.

@Kagami instead openssl you can try write your own implementation secp256k1_ecdh that use internal functions of secp256k1 (secp256k1_ecmult_const/secp256k1_ge_set_gej/secp256k1_fe_normalize ...). Probably it will be faster.

@fanatid thanks for the hint, I will look into it. They also have PR for x-only ECDH which is perfectly fine for my needs.


Low parallel requests on Arc with VLLM serving

Got only 10 parallel request on 2 Arc with Qwen1.5 model (1024 input/512 out), could you please to improve the performance?

We will verify this in 2 Arc GPUs environment.

We have verified in 2 Arc GPUs that it can handle more than 10 parallel requests.


Check the performance of putProducerState

val counter = new AtomicLong(0)
...

val c1 = System.nanoTime()
stream.client.putTransactionSync(transactionRecord, extTransportTimeOutMs.milliseconds)
val c2 = System.nanoTime()
counter.addAndGet(c2 - c1)

def putTransactionSync(transaction: ProducerTransaction, timeout: Duration = 1.minute) = {
    val f = client.putProducerState(transaction)
    Await.result(f, timeout)
}

mutual (10000 times) call shows me
Counter is: 0.70385027

per query (counter / 10000) which is very slow, because bare netty client-server round-trip time is about 0.14, so 0.56 is for processing of something. But actually it only takes to deserialize, send data to commit log, check auth and serialize the response back. It must perform much faster ~ about 0.2 (total) because there are no heavy or locked operations, etc. It must work very-very fast, near to 0.2 ms or even faster like 0.16. Check and find where it loses a lot of time there.

It might be the server works nice (measure server time too), but client works slow. Try to find where it loses such amount of time. Take a look at t-streams UdpClientServerTest, it's UDP + serialization + deserialization though, but I got same numbers with Netty and TCP before.


☣ ☠ Enable the AST verifier unless SWIFT_DISABLE_AST_VERIFIER is defined.

The verifier catches quite a few issues that otherwise end up
resulting in crashes later in compilation.
Let's leave it enabled, even when assertions are disabled.
This change is intentionally only modifying the bodies of
verify() and shouldVerify(), as opposed to also updating various call
sites that call into these that are currendly under #ifndef NDEBUG.
We can evaluate those particular callsites at a later time. For now,
it useful to run the verifier over the entire file after type
checking.

@swift-ci Please test compiler performance

@swift-ci please smoke test compiler performance

Full compiler performance suite is going to take over 24 hours.

Build comment file:
Summary for master smoketest
Regressions found (see below)

Debug

debug brief
debug detailed

Release

release brief
release detailed

Debug

debug brief

Regressed (1)

name
old
new
delta
delta_pct

time.swift-driver.wall
79.3s
81.3s
2.0s
2.53% :no_entry:

Improved (0)

name
old
new
delta
delta_pct

Unchanged (delta < 1.0% or delta < 100.0ms) (1)

name
old
new
delta
delta_pct

LLVM.NumLLVMBytesOutput
47,057,860
47,056,152
-1,708
-0.0%

debug detailed

Regressed (0)

name
old
new
delta
delta_pct

Improved (0)

name
old
new
delta
delta_pct

Unchanged (delta < 1.0% or delta < 100.0ms) (23)

name
old
new
delta
delta_pct

AST.NumImportedExternalDefinitions
113,826
113,826
0
0.0%

AST.NumLoadedModules
16,158
16,158
0
0.0%

AST.NumTotalClangImportedEntities
337,992
338,140
148
0.04%

AST.NumUsedConformances
9,443
9,443
0
0.0%

IRModule.NumIRBasicBlocks
135,979
135,979
0
0.0%

IRModule.NumIRFunctions
80,256
80,256
0
0.0%

IRModule.NumIRGlobals
74,295
74,295
0
0.0%

IRModule.NumIRInsts
1,501,778
1,501,778
0
0.0%

IRModule.NumIRValueSymbols
137,726
137,726
0
0.0%

LLVM.NumLLVMBytesOutput
47,057,860
47,056,152
-1,708
-0.0%

SILModule.NumSILGenFunctions
49,967
49,967
0
0.0%

SILModule.NumSILOptFunctions
90,409
90,409
0
0.0%

Sema.NumConformancesDeserialized
472,076
472,076
0
0.0%

Sema.NumConstraintScopes
980,790
980,790
0
0.0%

Sema.NumDeclsDeserialized
3,041,418
3,041,440
22
0.0%

Sema.NumDeclsValidated
162,677
162,677
0
0.0%

Sema.NumFunctionsTypechecked
64,396
64,396
0
0.0%

Sema.NumGenericSignatureBuilders
85,006
85,022
16
0.02%

Sema.NumLazyGenericEnvironments
574,090
574,095
5
0.0%

Sema.NumLazyGenericEnvironmentsLoaded
50,027
50,027
0
0.0%

Sema.NumLazyIterableDeclContexts
437,052
437,068
16
0.0%

Sema.NumTypesDeserialized
3,265,161
3,265,177
16
0.0%

Sema.NumTypesValidated
272,379
272,379
0
0.0%

Release

release brief

Regressed (0)

name
old
new
delta
delta_pct

Improved (1)

name
old
new
delta
delta_pct

time.swift-driver.wall
193.4s
191.4s
-2.0s
-1.03% :white_check_mark:

Unchanged (delta < 1.0% or delta < 100.0ms) (1)

name
old
new
delta
delta_pct

LLVM.NumLLVMBytesOutput
38,466,336
38,466,336
0
0.0%

release detailed

Regressed (0)

name
old
new
delta
delta_pct

Improved (0)

name
old
new
delta
delta_pct

Unchanged (delta < 1.0% or delta < 100.0ms) (23)

name
old
new
delta
delta_pct

AST.NumImportedExternalDefinitions
11,354
11,354
0
0.0%

AST.NumLoadedModules
462
462
0
0.0%

AST.NumTotalClangImportedEntities
32,031
32,031
0
0.0%

AST.NumUsedConformances
9,448
9,448
0
0.0%

IRModule.NumIRBasicBlocks
177,671
177,671
0
0.0%

IRModule.NumIRFunctions
61,695
61,695
0
0.0%

IRModule.NumIRGlobals
60,186
60,186
0
0.0%

IRModule.NumIRInsts
1,480,728
1,480,728
0
0.0%

IRModule.NumIRValueSymbols
110,185
110,185
0
0.0%

LLVM.NumLLVMBytesOutput
38,466,336
38,466,336
0
0.0%

SILModule.NumSILGenFunctions
25,070
25,070
0
0.0%

SILModule.NumSILOptFunctions
42,180
42,180
0
0.0%

Sema.NumConformancesDeserialized
152,043
152,043
0
0.0%

Sema.NumConstraintScopes
839,227
839,227
0
0.0%

Sema.NumDeclsDeserialized
255,520
255,544
24
0.01%

Sema.NumDeclsValidated
31,065
31,065
0
0.0%

Sema.NumFunctionsTypechecked
12,616
12,616
0
0.0%

Sema.NumGenericSignatureBuilders
7,610
7,610
0
0.0%

Sema.NumLazyGenericEnvironments
37,647
37,649
2
0.01%

Sema.NumLazyGenericEnvironmentsLoaded
5,033
5,035
2
0.04%

Sema.NumLazyIterableDeclContexts
24,467
24,467
0
0.0%

Sema.NumTypesDeserialized
312,243
312,249
6
0.0%

Sema.NumTypesValidated
43,872
43,872
0
0.0%

@swift-ci Please smoke test

More data. I attempted compiling the standard library three times with and without this change.
The average user+sys time without this change: 1m 54s 132ms
The average user+sys time with this change: 1m 49s 701ms
I wonder if we're seeing some strange cache affect here?

Build comment file:
Compilation-performance test failed

@graydon I don't seem to be getting a lot of feedback on why the full compiler perf test failed. Any insights on this?
Build step 'Execute shell' marked build as failure
Archiving artifacts
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 8s

Oh wait, wrong log.
Traceback (most recent call last):
  File "/Users/buildnode/jenkins/workspace/swift-PR-compiler-performance-macOS/swift-source-compat-suite/run_cperf", line 582, in <module>
    sys.exit(main())
  File "/Users/buildnode/jenkins/workspace/swift-PR-compiler-performance-macOS/swift-source-compat-suite/run_cperf", line 106, in main
    regressions = analyze_results(configs, args)
  File "/Users/buildnode/jenkins/workspace/swift-PR-compiler-performance-macOS/swift-source-compat-suite/run_cperf", line 386, in analyze_results
    (passed_modules, failed_modules) = find_module_statuses(configs, args)
  File "/Users/buildnode/jenkins/workspace/swift-PR-compiler-performance-macOS/swift-source-compat-suite/run_cperf", line 371, in find_module_statuses
    for (module, j) in all_driver_stats_files(configs, args):
  File "/Users/buildnode/jenkins/workspace/swift-PR-compiler-performance-macOS/swift-source-compat-suite/run_cperf", line 363, in all_driver_stats_files
    j = json.load(fp)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Closing. If we ever want to evaluate doing this in the future it's easy enough. For now there is concern about causing more deserialization and/or increasing declaration checking time.


FLEX-1901 Implements add and remove backend for device model firmware

FLEX-1901

All is well.

All is well.


We are unable to process your request. Please reach out to your support representative.

Amplify CLI Version
12.8.2
Question
Hi,
I am trying to integrate AWS Amplify with my application. However, I am getting the error "We are unable to process your request. Please reach out to your support representative". All of my IAM preferences are set as expected with the AWS Amplify Admin access, and I followed every step in the initialization process. This is taking place after running "amplify init" in my project root directory. Am I missing something? I've tried every troubleshooting step I came across but am still having the same error message. Any help would be greatly appreciated.

Hey @moldo1028, thank you for reaching out.
The issue appears to be similar to https://github.com/aws-amplify/amplify-cli/issues/13124 which occurs on a account level.
Similar to the GitHub issue are you unable to create a application on the console as well?
If you have a premium support plan, could you open a support case under the Amplify Console Support with a brief explanation with the error and being unable to create resources.
If you do not have a support plan and see a Basic Support Plan in the AWS support console,
create a new support case under Account and Billing using the following headings:
Service: Account
Category: Other Account Issues
General Question
add a subject line say Check Account Status - Amplify Console and describe the issue you are experiencing such as the error message and unable to create a app. Do link the GitHub issue to the support ticket to help track the issue.

Hey @ykethan thank you for your response. I tried to create an application on the console as well, and am getting the same error. I did open a support case (basic support) but have not heard back yet, how long is the typical turnaround time? Thanks

Hey @moldo1028, sorry to hear that. The support should provide a response soon. Could you provide us the case ID I can try to follow up with the team.

I am getting the same issue, when creating a new app and when creating with existing repo.
Any solutions? @ykethan

Im also facing the same issue with amplify. Any solutions for this issue ? @ykethan

Hey, as described here: https://github.com/aws-amplify/amplify-cli/issues/13470#issuecomment-1850691317. open a case on the AWS support console

I'm running into this issue as well. I'd be very surprised if it was an account quota issue in my case because this would be my first Amplify app.
My steps:
Begin from the Amplify console -> click "Get Started" under "Amplify Studio" -> Enter an app name and click "Confirm Deployment"
I get an error screen with this message:

Creating your AWS Amplify app failed.
We are unable to process your request. Please reach out to your support representative..

I have had this issue.
Could I open the case instead? @ykethan

Hey yes, do open a case with the support team here so that they can dive into your account.

I opened the case.
Case ID:<PHONE_NUMBER>01098

I had my issue resolved at AWS support center, but I didn't see what is the factor.
You should open your issue at AWS support center.

This issue accusers a lot during GitHub authorization process. Everything looks fine...But sometimes AWS takes you back to a different region


Add gradle build configuration

This PR adds a Gradle build configuration to aya

Run ./gradlew jar to build aya.jar and copy it into the root project directory
Run ./gradlew eclipse to generate an eclipse project

Can also import aya as a gradle project directly in eclipse or other IDE
Some of the eclipse files have been removed

You can also use the .bat gradle file to run the same commands on windows

Closing in favor of #106