[Question] Delegate deserialization to a generic Deserializeable in a visitor method.

I haven't figured out how I can achieve the following using serde, therefore I appreciate your help.
Introduction
The data structure I want to deserialize appears in two forms. It contains either a single string or an array of objects.
["This string maps to the error variant"]
[{keys: values}, {keys: values}, …]

The enum VariantData<T> should represent this data structure where T is a generic Deserializeable encapsulating the knowlegde how to deserialize itself.
enum VariantData<T> {
    Content(Vec<T>),
    Error(String),
}

In VecVisitor<T> calling SeqVisitor::visit() is enough, because every element is of type T and the method delegates deserialization to T.
I can use the same approach for VariantData<T> deserializing a single element to a helper enum FirstValue<T>. I guess an alternative would be to use a EnumVisitor directly but I'm not yet familiar enough with it.
enum FirstValue<T> {
    Content(T),
    Error(String),
}

The problem
In both cases I need to distinguish the type I want deserialize to based on the kind of structure in the raw data. If a map follows the target type is T (or Vec<T> if directly serializing the array instead of looping over it), if a plain string follows the target type is String.
My first idea was: (The code is incomplete since its enough to show my problem.)
impl<T> Deserialize for VariantContent<T> {
    fn deserialize<D: Deserializer>(deserializer: &mut D) -> Result<Variant<T>, D::Error> {
        struct VariantContentVisitor<T> {
            data: PhantomData<T>,
        }

        impl<T> de::Visitor for VariantContentVisitor<T> where T: Deserialize {
            type Value = VariantContent<T>;

            fn visit_seq<V>(&mut self, mut visitor: V) -> Result<Self, V::Error>
                where V: de::SeqVisitor
            {
                let first: FirstValue<T> = try!(visitor.visit());
                match first {
                    FirstValue::Content(x) => {
                        // Loop over the remaining Ts and add them to `Vec<T>`
                        let elem: T = try!(visitor.visit());
                        unimplemented!()
                    },
                    FirstValue::Error(x) => VariantContent::Error(x),
                }
            }
        }

        deserializer.visit(VariantContentVisitor { data: PhantomData })
    }
}

// …
impl<T> de::Visitor for FirstValueVisitor<T> where T: Deserialize {
    type Value = FirstValue<T>;

    fn visit_map<V>(&mut self, mut visitor: V) -> Result<Self, V::Error>
        where V: de::MapVisitor
    {
        // Deserialize a T. This is called for the first element.
        unimplemented!()
    }

    fn visit_string(&mut self, value: String) -> Result<Self, V::Error> {
        FirstValue::Error(value)
    }
}
// …

Now my question: How can I delegate the deserialization of a T to it's Deserialize implementation in visit_map()? It is a generic type, I can't know how to deserialize that type correctly. As mentioned above I also can't guide SeqVisitor::visit to the correct type directly, because I need to know if a map or a string follows next.
I also tried to use a EnumVisitor:
impl<O> Deserialize for VariantContent<O> {
    fn deserialize<D: Deserializer>(deserializer: &mut D) -> Result<Variant<O>, D::Error> {
        struct VariantContentVisitor<O> {
            data: PhantomData<O>,
        }

        impl<O> de::EnumVisitor for VariantContentVisitor<O> where O: Deserialize {
            type Value = VariantContent<O>;

            fn visit<V>(&mut self, visitor: V) -> Result<Self::Value, V::Error> {
                // How to determine which variant to decode?
            }
        }

        deserializer.visit_enum("VariantContent",
                                &["Content", "Error"],
                                VariantContentVisitor { data: PhantomData })
    }
}

It is shorter than my first attempt, but how can I determine which type (T or String) I need to deserialize to? It depends on the kind of the following data, map or primitive string.
Thanks for your help!

Have a look at https://github.com/serde-rs/serde/blob/master/serde_json/src/value.rs#L265 this shows how it's done for the json value enum

Ah, I need a visitor for the target type, the trait bound then would be T: de::Visitor. I can access the target type using the associated type de::Visitor::Value. I will try that.
Thanks

It seems a good idea, I introduced the type paramter TV: de::Visitor which can produce a value T. The biggest problem now seems to get a new TV. Maybe a constructor method should be added to the trait de::Visitor. I guess it's not as easy as I think, because stateful visitors would need additional parameters in their constructor method.
enum FirstValue<T> {
    Content(T),
    Error(String),
}

impl<T> Deserialize for FirstValue<T> {
    fn deserialize<D: Deserializer>(deserializer: &mut D) -> Result<Self, D::Error> {
        struct FirstValueVisitor<T> {
            data: PhantomData<T>,
        }

        impl<TV> de::Visitor for FirstValueVisitor<TV> where TV: de::Visitor {
            type Value = FirstValue<TV::Value>;

            fn visit_map<V>(&mut self, mut visitor: V) -> Result<Self::Value, V::Error>
                where V: de::MapVisitor
            {
                // Problem:
                let values = try!(TV::new().visit_map(visitor));
                unimplemented!()
            }

            fn visit_string<E>(&mut self, value: String) -> Result<Self::Value, E>
                where E: de::Error
            {
                FirstValue::Error(value)
            }
        }

        deserializer.visit(FirstValueVisitor { data: PhantomData })
    }
}

Whats wrong with type Value =

Sorry wrong button

Do you mean something like that:
struct FirstValueVisitor<'a, O, D: 'a> {
    data: PhantomData<O>,
    deserializer: &'a mut D,
}

If I put the deserializer into the visitor I can't call deserializer.visit(visitor) because the visitor borrows the deserializer.

No. I mean that you should call exactly like i wrote in the previous post. I don't have a computer here so i can't help you much. Sorry. Have a look at the code from serde json value, it's solving the same problem as you. Or look at the generated code from an enum deserialization

Now I found some time to work on this again.
My code now looks like:
enum FirstValue<O> {
    Content(O),
    Error(String),
}

impl<O> Deserialize for FirstValue<O> where O: Deserialize {
    fn deserialize<D: Deserializer>(deserializer: &mut D) -> Result<Self, D::Error> {
        struct FirstValueVisitor<O> {
            data: PhantomData<O>,
        }

        impl<O> de::Visitor for FirstValueVisitor<O> where O: Deserialize {
            type Value = FirstValue<O>;

            fn visit_map<V>(&mut self, mut visitor: V) -> Result<Self::Value, V::Error>
                where V: de::MapVisitor
            {
                unimplemented!();
                let values = Deserialize::deserialize(self.deserializer)
                    .or(Err(de::Error::syntax("FirstValue, visit_map")));
                values
            }

            fn visit_string<E>(&mut self, value: String) -> Result<Self::Value, E>
                where E: de::Error
            {
                Ok(FirstValue::Error(value))
            }
        }

        deserializer.visit(FirstValueVisitor { data: PhantomData })
    }
}

Of course I get an error:
src/protocol/response.rs:311:55: 311:72 error: attempted access of field `deserializer` on type `&mut protocol::response::FirstValue<O>.Deserialize::deserialize::FirstValueVisitor<O>`, but no field with that name was found
src/protocol/response.rs:311                 let values = Deserialize::deserialize(self.deserializer)

Because of that error I introduced a field deserializer in FirstValueVisitor<O> as I wrote in my last comment.
Maybe we talk past each other.

Another problem is, that at the time Deserializer::deserialize(self.deserializer) is called the opening brace was already consumed by visit_map. Therefore deserialize can't deserialize the object, because the next token it sees is probably a string (the key of the object). What I really want is to forward the visit_map call to the the visitor of the type O.
The reason I need this forwarding is, that visit_map is forwarded to a different type than visit_string.

Another problem is, that at the time Deserializer::deserialize(self.deserializer) is called the opening brace was already consumed by visit_map. Therefore deserialize can't deserialize the object, because the next token it sees is probably a string (the key of the object). What I really want is to forward the visit_map call to the the visitor of the type O

you could put an Option<u8> field into your deserializer where you can "put back" an item you read from the stream. when reading from the stream, you always then check the field first, and if it's None, only then you actually read from the stream.
anyway, I think it could be much simpler. Do you have your code somewhere so I can play with it?

The reduced code example is available in a gist. I extracted it from my actual code and commented some spots, just the tests are a bit contrived.

ok, I see now... this is indeed nontrivial... (also very odd with the single element vector...)

Yeah, the single element vector in case of an error was not my choice, but I want to deserialize that.

Finally this issue is solved!
Using a Deserializer to forward the call is possible. In the updated gist you can see how I did it. In short, there is a specialized Deserializer which knows how to deserialize from the supplied MapVisitor.
Thanks @oli-obk for your patience and your tips that brought me on track and @erickt whose de::value::*Deserializers gave me inspiration.

I've encountered this thread while trying to figure out how to do delegating deserialization myself; basically, I wanted to implement the logic of if string "None" then return Self::None, else parse as T and return Self::Some(T).
Unfortunately, gists by @tomprogrammer are now dead, so I was stuck on a problem: ok, I've tried to deserialize as "None", and failed, how do I now deserialize as T, given that the deserializer already has been consumed?
Well, I found the answer in the depths of serde's implementation of parsing untagged enums, which also need to parse the incoming stream several times. Yes, "deserializer" is indeed supposed to be one-shot. To work around that, serde just parses everything into a JSON-like tree ("Content" struct in serde's internals), and then just tries to convert that struct to various enum options.
So it is possible to emulate this behaviour with serde_json::Value, I guess. I opted for using an untagged enum + serialize_with / deserialize_with approach.
Hope this would be helpful to someone.


Error while starting container with unset DEBUG variable

I have an error while starting this container
//./docker-entrypoint.sh: line 104: [: : integer expression expected

I believe line right here:
https://github.com/mumble-voip/mumble-docker/blob/0dceaf54f5106602dfcdedb12e3a9869a624da02/docker-entrypoint.sh#L104
should be replaced by:
if [ -z "$DEBUG" ] || [ "$DEBUG" == 1 ]; then

or
if [ ! -z "$DEBUG" ] && [ "$DEBUG" == 1 ]; then

Did you have the last version ?
This issue should be fix with https://github.com/mumble-voip/mumble-docker/commit/6985fd658566b11a8e3c4d62ed06a76a79deeac7
Can you execute inside your container : cat /docker-entrypoint.sh | grep -iF debug to check if you have the good file ?

Yes I have last version.
 [ ! -z "$DEBUG" ] means not is empty string $DEBUG
therefore
[ ! -z "$DEBUG" ] || [ "$DEBUG" == 1 ] means not is empty string $DEBUG or $DEBUG as number equals 1
If we have $DEBUG empty, then
not is empty string "" or "" as number equals 1 ->
not true or "" as number equals 1 ->
false or "" as number equals 1 ->
"" as number equals 1 ->
error.

Indeed, I should replace this line by :
if [ ! -z "$DEBUG" ] && [ "$DEBUG" == true ]; then


Loading the model using ONNX

First, I tried without luck to download the model, it's really complicated to access the filesystem in the context of a browser extension. Then I went to the approach proposed here.
In this PR I am pushing the model and some supporting ONNX files to the repo so it can be bundled into the extension and loaded when the user presses the "Analyze" button.

thanks @lotif can we test it or you want just for us to see code?

@fcogidi yes, I forgot to add a close button :) easy to do, will work on it on follow up PRs. An inference session is made with the model every time the user clicks on the "Analyze" button.
@shainarazavi to test it, you can clone the repo and follow the instructions in the README. You will need to install npm beforehand.

@fcogidi yes, I forgot to add a close button :) easy to do, will work on it on follow up PRs. An inference session is made with the model every time the user clicks on the "Analyze" button.

Can there be a persistent inference session that gets started when the browser extension loads initially and waits to be called?

@fcogidi yes, we can do the persistent inference session, but I need to figure out how to do that properly first.
I don't think we should run this on the background as the extension code runs for every page that gets loaded in every browser tab. That's a lot of unnecessary computing we will be adding to anyone who has the extension. We can skip the button, though, and run it when the user brings up the extension. In short, I believe it should be an action initiated by the user.

@lotif Re: running in the background
I get the point of not wasting compute, but what I was thinking w.r.t to running in the background is more like running it once when the page loads (if it is possible, cache the results so that reloading the same page with the same content doesn't trigger an inference call). I was also thinking that we would have a list of websites that the extension would be active on (the whitelist). There are other checks that we probably need to do before inference too. For example, there needs to be at least 20 sentences for the predictions to be useful.
I'm looking at this from a theoretical standpoint of course. I don't know how much work it takes to make it work in practice.

@fcogidi I'm more comfortable with running it on load if we have a whitelist of sites. Not hard to do, we can literally just add the sites here:
https://github.com/VectorInstitute/nmb-browser-extension/blob/main/public/manifestv3.json#L14

@lotif Sounds good.


Remove handleChange prop

Remove handleChange prop and local 'value' state. Instead use value prop directly.
This caused ghost selections in posting grid.

it seems fine


how to put trainer.fit() in for loop?

Discussed in https://github.com/PyTorchLightning/pytorch-lightning/discussions/8380

Originally posted by anik123 July 12, 2021
I am trying to create multiple model using loop as below.
for client in clients:
    t.manual_seed(10)
    client['model'] = LinearNN(learning_rate = args.lr, i_s = args.input_size,  h1_s = args.hidden1, h2_s = args.hidden2, n_c = args.output, client=client)
    client['optim'] = optim.Adam(client['model'].parameters(), lr= args.lr)

However, trainer.fit() is an async method. To train multiple models, I need to put trainer.fit() in a loop as follows
for client in clients:
     trainer = pl.Trainer(
     max_epochs=args.epochs+1,
     progress_bar_refresh_rate=20,
     )
     trainer.fit(client['model'])

As this is an async method, it gives an error

AttributeError: can't set attribute

as it doesn't wait for finishing trainer.fit().
Is there any way to do that?
Thanks in advance.

Hi, where do you get AttributeError?
Could you please post the full error trace so we can see where it originates from?

@awaelchli Thanks for your reply. It actually giving because for loop doesn't wait to finish trainer.fit() to finish. It goes for another round while the previous trainer.fit() was still running.

Here is the error @awaelchli
GPU available: True, used: False
TPU available: False, using: 0 TPU cores
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:597: UserWarning: GPU available but not used. Set the gpus flag in your trainer `Trainer(gpus=1)` or script `--gpus=1`.
  "GPU available but not used. Set the gpus flag in your trainer"

  | Name   | Type   | Params
----------------------------------
0 | l1     | Linear | 64
1 | l2     | Linear | 1.1 K
2 | output | Linear | 33
----------------------------------
1.2 K     Trainable params
0         Non-trainable params
1.2 K     Total params
0.005     Total estimated model params size (MB)
Epoch 2: 100%
71/71 [00:00<00:00, 158.12it/s, loss=0.816, v_num=0, train_loss_step=0.722, train_loss_epoch=0.825]
GPU available: True, used: False
TPU available: False, using: 0 TPU cores

  | Name   | Type   | Params
----------------------------------
0 | l1     | Linear | 64
1 | l2     | Linear | 1.1 K
2 | output | Linear | 33
----------------------------------
1.2 K     Trainable params
0         Non-trainable params
1.2 K     Total params
0.005     Total estimated model params size (MB)
final

ses final
Epoch 2: 100%
71/71 [00:00<00:00, 164.66it/s, loss=0.837, v_num=1, train_loss_step=0.883, train_loss_epoch=0.830]
GPU available: True, used: False
TPU available: False, using: 0 TPU cores
final

ses final

Test set: Average loss for Global model: 0.02560, Accuracy: 0/500 (0%)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-15-3d42788289b5> in <module>()
     21          callbacks=[b]
     22          )
---> 23         trainer.fit(client['model'])
     24         '''
     25         while(b.wait):

10 frames
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloader, val_dataloaders, datamodule)
    458         )
    459
--> 460         self._run(model)
    461
    462         assert self.state.stopped

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run(self, model)
    715         self.call_setup_hook(model)  # allow user to setup lightning_module in accelerator environment
    716         self.call_configure_sharded_model(model)  # allow user to setup in model sharded environment
--> 717         self.accelerator.setup(self, model)  # note: this sets up self.lightning_module
    718
    719         # ----------------------------

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/cpu.py in setup(self, trainer, model)
     37             raise MisconfigurationException(f"Device should be CPU, got {self.root_device} instead.")
     38
---> 39         return super().setup(trainer, model)

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py in setup(self, trainer, model)
     88             model: the LightningModule
     89         """
---> 90         self.setup_training_type_plugin(self.training_type_plugin, model)
     91         if not self.training_type_plugin.setup_optimizers_in_pre_dispatch:
     92             self.setup_optimizers(trainer)

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py in setup_training_type_plugin(self, plugin, model)
    381     def setup_training_type_plugin(self, plugin: TrainingTypePlugin, model: 'pl.LightningModule') -> None:
    382         """Attaches the training type plugin to the accelerator."""
--> 383         plugin.setup(model)
    384
    385     def setup_precision_plugin(self, plugin: PrecisionPlugin) -> None:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/single_device.py in setup(self, model)
     67
     68     def setup(self, model: torch.nn.Module) -> torch.nn.Module:
---> 69         self.model_to_device()
     70         return self.model
     71

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/single_device.py in model_to_device(self)
     64             torch.cuda.set_device(self.root_device)
     65
---> 66         self._model.to(self.root_device)
     67
     68     def setup(self, model: torch.nn.Module) -> torch.nn.Module:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/device_dtype_mixin.py in to(self, *args, **kwargs)
    107         out = torch._C._nn._parse_to(*args, **kwargs)
    108         self.__update_properties(device=out[0], dtype=out[1])
--> 109         return super().to(*args, **kwargs)
    110
    111     def cuda(self, device: Optional[Union[torch.device, int]] = None) -> Module:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in to(self, *args, **kwargs)
    605             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
    606
--> 607         return self._apply(convert)
    608
    609     def register_backward_hook(

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    352     def _apply(self, fn):
    353         for module in self.children():
--> 354             module._apply(fn)
    355
    356         def compute_should_use_set_data(tensor, tensor_applied):

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    388                     should_use_set_data = compute_should_use_set_data(param.grad, grad_applied)
    389                     if should_use_set_data:
--> 390                         param.grad.data = grad_applied
    391                     else:
    392                         assert param.grad.is_leaf

I see this in your error:
<ipython-input-15-3d42788289b5> in <module>()
     21          callbacks=[b]
     22          )
---> 23         trainer.fit(client['model'])
     24         '''
     25         while(b.wait):

is this problem only happening in ipython?
I tried to reproduce in IPython with the bug report model like so:
for _ in range(3):
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        limit_train_batches=1,
        limit_val_batches=1,
        num_sanity_val_steps=0,
        max_epochs=1,
        weights_summary=None,
    )
    trainer.fit(model, train_dataloaders=train_data, val_dataloaders=val_data)

and it works fine, one fit runs after the other.
Does your client['model'] define an optimizer? Is it a valid LightningModule?

And what does this do?
for client in clients:
    t.manual_seed(10)
    client['model'] = LinearNN(learning_rate = args.lr, i_s = args.input_size,  h1_s = args.hidden1, h2_s = args.hidden2, n_c = args.output, client=client)
    client['optim'] = optim.Adam(client['model'].parameters(), lr= args.lr)

the optimizer here won't be used by Lightning, because Lightning takes what LinearNN.configure_optimizers returns.

Could you please take this template here and adapt it to reproduce the issue?
import os

import torch
from torch.optim import Adam
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import LightningModule, Trainer

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):

    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

def run():
    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)

    clients = [{"name": "bob"}, {"name": "alice"}]
    for client in clients:
        client["model"] = BoringModel()
        client["optim"] = Adam(client["model"].parameters(), lr=0.01)

    for client in clients:
        print(client["name"], "is training now")
        trainer = Trainer(
            default_root_dir=os.getcwd(),
            limit_train_batches=1,
            limit_val_batches=1,
            num_sanity_val_steps=0,
            max_epochs=1,
            weights_summary=None,
        )
        trainer.fit(client["model"], train_dataloaders=train_data)

if __name__ == '__main__':
    run()

Thanks for your help.
class LinearNN(pl.LightningModule):
  def __init__(self, learning_rate,  i_s, h1_s, h2_s, n_c, client):
    super(LinearNN, self).__init__()
    #print("init")
    #self.save_hyperparameters()
    self.l1 = nn.Linear(i_s, h1_s)
    self.l2 = nn.Linear(h1_s,h2_s)
    self.output = nn.Linear(h2_s, n_c)
    #self.drop = F.dropout(0.65, training=True)
    self.train_val = []
    self.validation_val = []
    self.learning_rate = learning_rate
    self.client = client
    self.wait = True
  def forward(self, x):
    out = t.tanh(self.l1(x))
    out = F.dropout(out,0.2, training=True)
    out = t.tanh(self.l2(out))
    out = F.dropout(out,0.2, training=True)
    out = self.output(out)
    return out
  def training_step(self, batch, batch_idx):
    self.wait = False
    data, target = batch
    data = data.send(self.client['hook'])
    target = target.send(self.client['hook'])
    pred = self.client['model'](data.get())
    l = MSELoss(pred,target.get())
    self.log('train_loss', l, on_step=True, on_epoch=True, prog_bar=True, logger=True)
    return {'loss': l}
  def configure_optimizers(self):
    return self.client['optim']
  def train_dataloader(self):
    return self.client['train_loader']

init
for client in clients:
    t.manual_seed(10)
    client['model'] = LinearNN(learning_rate = args.lr, i_s = args.input_size,  h1_s = args.hidden1, h2_s = args.hidden2, n_c = args.output, client=client)
    client['optim'] = optim.Adam(client['model'].parameters(), lr= args.lr)

calling
  for client in clients:
       trainer = pl.Trainer(
        max_epochs=args.epochs+1,
        progress_bar_refresh_rate=20,
        )
       trainer.fit(client['model'])

As requested, you will have to provide a runnable full implementation or colab link which shows the error in question.
Simple model
Colab link
Otherwise, we cannot easily help you find what is causing the error.

@carmocca Thanks for your suggestion
here is the code: colab
dataset: USA_Housing.csv

@awaelchli Any update?

@anik123 Hi, have you solved your problem? In my problem, I want to call trainer.fit multiple times but the pipeline seems to continue running even though the training has not even started yet and  produces an error.


Index rebuilds should refuse to run in parallel

As described here, the elastic search indexes do not properly reindex if the re indexing is run in parallel. This seems like it would be easily solved with a semaphore and some OSGI glue.

No, you actually need to rewrite the whole index code since the reindexing is not started sequentially and not in parallel. But re-indexing works by sending messages via network. A process that does not necessarily guarantee the order of the messages. You probably heared me complain about this in the past already ;-)

I'm not terribly familiar with the code in question, but if the issue is that the results of index B depend on index A we could at least solve some of the issue by enforcing that only a single index be rebuilt at a time, right?  It doesn't guarantee that index A hasn't been rebuilt with bad data, but at least it prevents the user from kicking off all of the indexing at once.

We are already doing that. In a way…
We are building index A and sending all the necessary messages. Only if that's done we start with index B. Unfortunately, sending all messages does not mean that all messages are already received and processed. That code is completely decoupled and isn't even directly related to index-rebuilds at all.
What we usually do for updates is that we set the log level for the message receiver to debug and wait until all messages are seemingly processed before initiating the rebuilding process for the next index. But that's a bit hacky.
We want to fix that for SWITCH and already started the development. But then CORVID19 happened and we need to scale up and do weird other things :D

Fair enough.  I'll assign this to you then, since you're already working on it, and SWITCH will make you finish it :)

Fair enough. I'll assign this to @KatrinIhler then, since she is already working on it, and SWITCH will make her finish it :)

@KatrinIhler , was this fixed in a pull?

@karendolan This is currently in progress! If you look at my open pull requests right now, starting at #2211, I'm working on rewriting how the index rebuild works. Part of this is ripping out the messaging stuff, so once this is done, this issue should be fixed. :D If you want too see the change for one service, check out #2351 for example - the other services should work similarly. 3 are already done, with 5 more to go! (Unfortunately, I started with the easy ones...)

This work was dome by @KatrinIhler and is in 10.1 as explained in #2893

Purely for completeness' sake: You can still do an index rebuild in parallel if you make multiple requests to the index rebuild service. There is currently no check forbidding that. (And maybe there should be, because this is a really bad idea in most - but not all - cases.) But if you start a complete index rebuild once, everything happening then is now guaranteed to be sequential.


Hitting backspace on multi-select converts element to text

Perhaps this is intended behavior but it seems odd to me.  If you use the backspace key to delete an element in a multi-select box, that element doesn't go away like it would if you clicked the 'x'.  Instead it is converted to text with an additional space at the end that you have to backspace through to get rid of.  Is this intended?  Would be nice to have an option to disable it if that is the case.  Fiddle providede below:
http://jsfiddle.net/y7fo61ms/

Is this intended?

Yup, totally intentional. https://github.com/select2/select2/commit/395e06aff52059f654aff47f03a26cad8955a092

Would be nice to have an option to disable it if that is the case.

I'm not sure how I feel about having another option for this, I'd like the get the current set documented first. But I can tell you that you can completely erase the search box in Search.searchRemoveChoice and it will do what you are looking for.

Hmm, ok.  I guess I'll have to see if I can wrap my head around this
behavior.  Thanks for the info.
On Wednesday, May 13, 2015, Kevin Brown<EMAIL_ADDRESS>wrote:

Is this intended?
Yup, totally intentional. 395e06a
https://github.com/select2/select2/commit/395e06aff52059f654aff47f03a26cad8955a092
Would be nice to have an option to disable it if that is the case.
I'm not sure how I feel about having another option for this, I'd like the
get the current set documented first. But I can tell you that you can
completely erase the search box in Search.searchRemoveChoice
https://github.com/select2/select2/blob/dd9af6893450bbb054ca79546a16bb18198f3e88/src/js/select2/selection/search.js#L127-L135
and it will do what you are looking for.
—
Reply to this email directly or view it on GitHub
https://github.com/select2/select2/issues/3354#issuecomment-101853485.

Wow, I was just surprised by this behavior too. Not what I was used to from 3.5.x. Well, it can be usefull, but there is also a nasty bug, which is already discussed here https://github.com/select2/select2/issues/3297

How to get the old behaviour? - the new behaviour is very undesirable and users are complaining why can they cannot backspace to remove an item and why does it turn into text. Is there any workaround in the meantime?

Can't agree with this behaviour. Just vote to get the old one.

Reopening this while we decide how best to handle this.

this is really odd,  my tag contains spaces, after I hit backspace, it breaks down to more small tags automatically, and it seems I can never delete all of them.

+1

I also vote for having an option to get back the old behaviour.
Thanks

+1
Thanks.

+1

Hey could someone by chance (@kevin-brown) explain if there's a way to override the Search.searchRemoveChoice without forking the repo? I'm trying to avoid this issue: https://github.com/select2/select2/issues/3297 by just removing the whole pill
Thanks!

:+1: I can't decide if Chosen is better despite not being able to keep the multi-select window open after selection, or if Select2 is better despite really buggy backspace behavior.

Hey could someone by chance (@kevin-brown) explain if there's a way to override the Search.searchRemoveChoice without forking the repo?

You can patch this method (not totally recommended by it gets the job done) or create a custom decorator (lacking docs to link to) that overrides this method.
Here's a jsfiddle that demonstrates the patching method
https://jsfiddle.net/zmh90ewu/

Greatly appreciated, @kevin-brown ! That works, thanks. :heart:

Thanks @kevin-brown :+1:

+1 To have this option, the full clear confused the hell out of our focus group when testing our UI which implements select2 4.0.0, causing multiple bouts of frustration.
Thanks for the patch, that'll suffice for now I guess.

This is very weird, in my case when you hit backspace for the first time the value disappears but is not immediately converted to text, you have to press backspace again to convert it to text.

I also have something similar on your example page but only on the second try :

@Intrepidd, I'd log this as a separate issue. I too experience this. (on FF and Chrome, Windows 7 64bit) on both the examples page for select2 and our own implementations.
(However for me the behaviour was undesired, so I've "resolved" it by changing the backspace behaviour to remove the entire pill, rather than converting it to text)

+1

+1 and thanks a lot @kevin-brown for the patch!

+1, would really like to be able to delete a whole tag on backspace.

I believe the use case of not removing the whole tag relies on the idea of

the case when you can add new tags on the fly (which is not enabled behaviour anyway) or

saving some keystrokes when trying (which IMHO does not justify the current behavior: clarity should win over brevity).

Besides, the current behavior is severaly broken. When you press backspace, the tag is removed and a cropped part of the tag text is shown. This happens at least on:
Latest Chrome / OSX:

Latest Firefox / OSX:

IE11 / Win7:

Thus, +1 for eg. the this.$search.val(''); fix, and maybe a +½ for an option to re-enable the current behavior (sans bugs).

:+1:. This behavior is really confusing. Would be nice to have an option to turn it off.

+1

+1

+1

+1

Just going to note that a patch is available for this in https://github.com/select2/select2/issues/3354#issuecomment-132389291.
The problem with the search box being cut off was fixed in 4.0.1.

Hmm - last time I tried the patch it didn't work, but this time it did. Whoops. PEBKAC.

@kevin-brown, patch in https://github.com/select2/select2/issues/3354#issuecomment-132389291 does not completely solve the problem when minimumInputLength = 0 and AJAX loading is used.
searchRemoveChoice() calls this.handleSearch() with search query != '' (empty string), so the correct solution (but not perfect, I think) is fully overriden searchRemoveChoice():
$.fn.select2.amd.require(['select2/selection/search'], function (Search) {
	Search.prototype.searchRemoveChoice = function (decorated, item) {
		this.trigger('unselect', {
			data: item
		});

		this.$search.val('');
		this.handleSearch();
	};
});

I have 2 use cases in my app where I use select2: a free-form tag list and a dropdown list with multiple choices. Let's have a look at the backspace behavior from the context of a user.
Tags: As a user, I'm typing a list of tags from the keyboard and select some that are already in the list. I expect to be able to quickly edit them and correct my mistakes with backspace. As a result, the current behavior of editing the selection when hitting backspace works well for tags because they are free-form.
Multiple choices: As a user, I am provided a list of options from which I can select one or many. I do not expect to be able to modify those choices (and if I try, I get a "No results found" message). As a result, the current behavior of allowing a user to edit the selection when hitting backspace is unexpected because it is a search helper.
I would use both behaviors in my app, depending on the context.
Hope this brings a useful point of view.

We patched this locally, so I have submitted a pull request.
Currently our items have a code as the ID and a description as the text, the text is just there to help, we  actually only care about the code. The current behaviour was no good to us as backspace would remove our code tag and replace it with a massive description instead of allowing editing of the code directly.
I created the option so that you could specify a string, if it matched a key on the item this is what is returned else if blank then the search will remain empty. Hopefully this helps most people.
@kevin-brown you mentioned documenting the current set of options before adding more, if you point me in the right direction I might be able to make a start on this. I am certainly not aware of all the options but could document as many as I have used / can find if that will help with this issue.
Cheers

Is there documentation for this option. Thanks

sory for spaming
Does this sill open or can be pached  whit version 4.0.1 ?
I am using it on MVC VS2015 and get Selcet2 via nuget.
I am facing this problem

@adopilot The patch mentioned in https://github.com/select2/select2/issues/3354#issuecomment-132389291 still works for 4.0.1 and will likely continue to work in the future until a more complete patch is brought in.

@kevin-brown The patch mentioned in #3354 (comment) does not fully work with 4.0.1 because in some cases, after removing the tag it converts the tag back to text which then needs to be deleted as well.

+1.  The new behaviour is a nice feature, but it is confusing in some contexts.

The behaviour is confusing, oh yeah.
It would be nice to have an option to bypass this.

+1. Think it is better to have an option to delete the tag completely when user type backspace. Otherwise users might have to type too much times or have to use mouse click when deleting unwanted tags.

When type backspace several times

+1 have an option to delete tag completely when user presses backspace.

+1 need delete whole tag on backspace press

+1

+1

my english is not good,there is my method:
var $eventSelect = $(".roles_selcet2");
$eventSelect.on("select2:unselect", () => {
$eventSelect.on("select2:open", () => {
$(".select2-search__field").val("");
})
    })

hope can be help

I ran into some issues with the callback not being called before other application code pulled in select2 via a require. To ensure this runs synchronously (before the original searchRemoveChoice is cached in a DecoratedMethod), use the following modification
$.fn.select2.amd.require(['select2/selection/search'], function (Search) {
    Search.prototype.searchRemoveChoice = function (decorated, item) {
        this.trigger('unselect', {
            data: item
        });

        this.$search.val('');
        this.handleSearch();
    };
}, null, true);

+1

+1

+1 to add the options proposed by @chrisrueger
https://github.com/DVD27/select2/commit/5800beea87dd9ab9dc294a4e0bcc7b5ab825c147
It make more sense
Best regards,

+1

+1

+1

+1

I even have a case where I would prefer the tags to be removed only by clicking the "x" icon, so: +1

Are updates on this? this is a severe usability issue, especially on mobile where ever tap counts
Why does this need to wait until for a new release and not fixed? this is a bug, not a feature

It's not a bug, as described in @kevin-brown's original response.  However it does seem like there are enough people asking for this, that it should be added as a configuration option.  Could also help to address the accessibility issue described in #4105.

+1

+1
Especially, when I'm using a templateSelection. I have to pass my text as some|text|part and in selection show only some. In this case, pressing Backspace, converts some to full editable some|text|part string.

@jmlacroix hit the nail on the head.
So a configuration option is very much needed.
And if I had to choose a default, I'd go with completely deleting the tag, because otherwise it doesn't make sense at all in the scenario where the user is choosing from premade options -- but does kinda work the other way around, just makes the user work a bit more, i.e. re-typing.

+1
Please! :)

Got around this by commenting out this line (2123 in 4.0.6-rc.0):
//this.$search.val(item.text);
Context:
Search.prototype.searchRemoveChoice = function (decorated, item) {
this.trigger('unselect', {
data: item
});
//this.$search.val(item.text);
this.handleSearch();

};

We don't have immediate plans to provide this. We are focused to fix some major UI bugs (that are majority of issues and PR's). But if you open a PR with unit tests, I will be glad to review and approve if everything is ok :+1:

Looks like randomly clicking a "Close" button :/

That worked for me :
// "this" is a DOM element binded with select2 :
if ($(this).attr('multiple')) {
    $(this).on('select2:unselect', function (event) {
        var $el = $(this)
        setTimeout(function () {
            $('.select2-search__field', $el.closest('.form-row')).focus();
            $el.select2('open');
            $('.select2-search__field', $el.closest('.form-row')).val('');
        }, 0)
    });
    $(this).on('select2:select', function (event) {
        var $el = $(this);
        $el.select2('open');
        setTimeout(function () {
            $('.select2-search__field', $el.closest('.form-row')).focus();
        });
    });
}

Another solution:
(function($select2Obj){
	$select2Obj
		.parent()
		.find('.select2-selection')
		.on('keydown', '.select2-search--inline', function(evt){
			var backspaceKey = 8;
			if(evt.which == backspaceKey){
				var currentSelection = $select2Obj.select2('data').map(function(s){return s.id});
				$select2Obj.val(currentSelection).trigger("change");
				$select2Obj.select2("close");
			}
		})
})($("#yourSelect2Id"));

Also for customization the selection and result, avoid encoding your values using separators like option text|another useful info that will go in a tooltip and include them in the JSON you provide as select2 data, easier and does not have issues with the backspace situation.

+1, Please provide a simple option if possible.. e.g. $("#foobar").select2({ clearTermOnBackspace: true })

The way I see things:

1 key benefits of select boxes is that users don't have to type it all
So if users don't have to type it all
Why make them untype it all?

Current behaviour means it can be considerably more work to UNDO than to DO which seems a UX anomaly (it's normally the other way around, you can simply delete, do Ctrl+Z, press cancel/previous etc...).
Example from the official demos https://select2.org/selections#limiting-the-number-of-selections:

Selecting California = 2 keystrokes (C+Enter)
Unselecting California = 11 keystores (1 initial backspace + 1 for each character)

Hence why IMO an option to allow removing the entire tag with 1 backspace would be good (clicking on the X icon forces the use of the mouse which reduces productivity as we need to get hands off the keyboard, click, and back on keyboard again).

+1, Please provide a simple option if possible.. e.g. $("#foobar").select2({ clearTermOnBackspace: true })
Thanks @kevin-brown for #3354 (comment)

Is there any chance of getting this option?

@kevin-brown : any timeline for merging https://github.com/select2/select2/pull/5752 ? Many thanks.

(function($select2Obj){
	$select2Obj
		.parent()
		.find('.select2-selection')
		.on('keydown', '.select2-search--inline', function(evt){
			var backspaceKey = 8;
			if(evt.which == backspaceKey){
				var currentSelection = $select2Obj.select2('data').map(function(s){return s.id});
				$select2Obj.val(currentSelection).trigger("change");
				$select2Obj.select2("close");
			}
		})
})($("#yourSelect2Id"));

Thanks for your answer: would you know how one would integrate the above into their existing select2 javascript code? Here is my existing javascript file:
import $ from 'jquery'
import 'select2/dist/css/select2.css'
import 'select2'

window.addEventListener('turbolinks:load', () => {
  var select2 = $('.select2').select2({
  	tags: true,
  	tokenSeparators: [',', ' ', ";"],
  	maximumInputLength: 5000
  });
})

At this point the behaviour is super broken depending on the options you're using. This will likely be fixed once we figure out the best way to handle deleting choices using the keyboard. That might mean changing the backspace key to instead focus the last option, with a second backspace meaning removing it entirely. Also the backspace key might be unbound for the "clear all" option, since apparently we don't consistently pick one of the other.

@kevin-brown chrs for the update.

One of the tracking WAI ARIA tickets that will allow us to figure out a good pattern for keyboard accessibility is https://github.com/w3c/aria/issues/689. Unfortunately it looks like that ticket has gone silent, so it may need to be kicked up again with some suggestions.
Until then I'm going to hold off on changing this behaviour around significantly. Right now the only way we can be somewhat accessible is by having it behave the way it currently does.

Not sure if a different issue, but when I press backspace I get a completely unrelated option removed from the select. See
https://jsfiddle.net/vsajip/pd642s5m/1/
for a demonstration. Notice how the "red" entry disappears after the first backspace, as well as losing the "black" pill:

so, all of these comments and still no solution!
OK, anything in the backlog of this library to get this issue fixed?

im using "Select2 4.1.0-beta.1"
Just replace the next line on script
}), this.$search.val(t.text), this.handleSearch()
for this one
}), this.$search.val(''), this.handleSearch()

im using "Select2 4.1.0-beta.1"
Just replace the next line on script
}), this.$search.val(t.text), this.handleSearch()
for this one
}), this.$search.val(''), this.handleSearch()

Hey could someone by chance (@kevin-brown) explain if there's a way to override the Search.searchRemoveChoice without forking the repo?

You can patch this method (not totally recommended but it gets the job done) or create a custom decorator (lacking docs to link to) that overrides this method.
Here's a jsfiddle that demonstrates the patching method
https://jsfiddle.net/zmh90ewu/
And here's the code for it
$.fn.select2.amd.require(['select2/selection/search'], function (Search) {
    var oldRemoveChoice = Search.prototype.searchRemoveChoice;

    Search.prototype.searchRemoveChoice = function () {
        oldRemoveChoice.apply(this, arguments);
        this.$search.val('');
    };

    $('#test').select2({
        width:'300px'
    });
});

This worked for me thanks

Hi, for me the above solution did not work, any other solution available?

Consider using key attribute as e.which is non-standard as stated here
MDN key reference

this is an old topic but still relevant as the default 'backspace to free text' behaviour is jarring with our users.  a select component has nothing to do with free-text (apart from the useful search feature) so it doesn't make sense to turn a select item into free text.  all this does is force the user delete the characters one by one.
i looked at the recent commit 77544c0 and it adds an option for multipleOnBackspaceChoice = unselect / none / text
however the unselect option clears the entire selection, not just the one the user hit backspace on...
would it work to have a fourth option to this setting multipleOnBackspaceChoice like 'single_unselect' or similar?  from the 81 comments on this thread over 9 years i think maybe it's a desirable behaviour :)  would be a great addition to select2 whether as default or as an option.


Is it impossible to apply an external css library like boostrap globally in version 2.0?

I am using polymer1.x. Polymer is the best !! I am working on a migration after version 2.0 is released this time. However, I found out that the css library (semantic ui) that is loaded as a link does not apply globally.
So I reviewed the 2.0 version of the document, tried to search it, and I tried everything and found that it was impossible to apply. Is it really impossible? If I do not apply the external css library like boostrap globally, I will not be able to go to 2.0.
I would like a detailed answer.

Global styles are controversal to the web components approach. You still can to create a shared style module, put your CSS there and reuse it via <style include="shared-styles">, but I would revommend to split those styles across components, e. g. button should keep its styles encapsulated, and then you will instanciate it using custom element, not <div class="btn">.

What you are probably seeing is Polymer 2.0 transitioning to use ShadowDOM by default.
Given that ShadowDOM provides an encapsulation mechanism for styling, your global stylesheet is no longer able to deeply apply styling.
If you absolutely need to use your global style, you can add a script to your application with window.ShadyDOM = {force: true} before webcomponents-lite.js
However, I would advise that you reconsider your use of global stylesheets, as forcing ShadyDOM incurs a performance and code size penalty for your users.

Also, as @web-padawan pointed out, if you can split your use of your styling library in such a way as it can be applied only to the elements that need it, that would be the best scenario.


QoL change. Takes into account default 'objective' param in LightGBM,

Hi, I just added a preliminary check to see if the params dictionary for a model in LightGBM has its 'objective' key populated. Since when not defining an objective its default value is 'regression' ( as can be seen here: https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective ). But this does not create an 'objective' entry in the params dictionary unless explicitly specified. Thus when trying to run TreeExplainer(...).shap_values(...) it throws a KeyError, as seen below:

This is just a QoL change to avoid that anyone else has to dig into why this happens, and allow a more fluid workflow.
P.S. This is my first PR and first time interacting in the open source environment. If there is something else i should have done, considered, please let me know.

@j-adamczyk No news from my side. I think i modified this on my local environment to avoid the bug, but shortly after i changed positions/projects so i completely lost track of this. As far as i can see here and the linked PR they solved it but didn't merge to master.


Project Goals

What does building this project produce? (I assume a GameCube iso)
What are the goals of this project?
How can I help?

Sorry, don't have notifications enabled here, so just saw your issue post.
Currently, the goal is getting the game working on a GameCube (support for Wii is also currently possible, the only technical limitation is the inability to use MEM2 without a custom allocator).
The biggest help right now would just be contributing, whether that's reverse engineering functions or even providing adequate names for functions. Likewise, I use the community documentation (and add to it myself), so any additions there for various structs are always appreciated.

I'm also confused about what building this project produces. It seems to be a dol executable.

How do you test this project?
Is there an option to build this as a hook that can be loaded into dolphin-emu?
Is there a patcher so the game functions are hooked (so it runs, even though the project is incomplete)?
Are there any plans for recompilation from PowerPC to other architectures (such as LLVM / object files) to allow cross compilation?
Are there any statistics as to how far along this project is?
Are there any plans for C++ (I prefer C; although there might be benefits in C++)?
What's the differentiation from https://github.com/filfat/openmelee ?

(Just for your information: I don't intend to work on this, I'm just curious)

@JayFoxRox I can answer the last question. openmelee was (is?) an attempt from me to re-implement melee’s game logic while doing everything else from the ground up. While this is (afaik) an attempt at a 1-1 recreation of melee based on RE.
You’ll probably see way more progress from FRAY since I am extremely busy with politics and work at the moment.
(And tbh I’ll probably archive openmelee and instead contribute to this in the future. 😄)

@JayFoxRox

I test via Dolphin's debugger currently. -O2 executables provide enough for me to trace, but O0 is preferred.

You can open the DOL with Dolphin's open option. Presently, there's some missing function prototypes that block -O0 compilation.

That'd require not targeting GX. I don't really know graphics APIs well enough to do that myself.

I haven't been tracking it in a while, but I know off-hand the title screen needed just a bit more work to load. HAL Sysdolphin was also about ~75% completel and was most of what was needed.

I don't really plan to presently. Smart pointers to cleanup the ref counting code and turning HSD into formal classes (versus their classes in C attempt) are really the only perceived benefits for me at present. I've got a private project where I've started that, but that relies mostly in re-architecting that currently takes too much time.

As filfat noted, this is mostly an attempt at a formal RE, so rewriting and properly naming everything versus recreating all the logic from scratch. It allows people to pull out functions for modding with the end-goal of being a compilable, trimmed down version of the game.


Error when AllowPrivilegeEscalation is nil

When AllowPrivilegeEscalation is nil privilege escalation is allowed, so we should error instead of warning.

Sorry for missing this :)


[release-0.58]Should remove and restore labeller only from workers

Currently in some tests we are waiting for
the node labeller to be resumed in any nodes
including unschedulable nodes and that will not work because master shouldn't be labeled by node
labeller.

What this PR does / why we need it:
manual back-port of:
https://github.com/kubevirt/kubevirt/pull/9215
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

/lgtm

/approve
/retest-required

@dhiller
Hey Daniel is there a problem with the ci in 0.58?

@Barakmor1 Hey, Daniel is OOO today, I will take a look at it

/hold
Please make sure you don't need rebase

/hold Please make sure you don't need rebase

/unhold
I just rebased 1 minute ago thanks :)

/lgtm


v3.2.5 with #6961 fix.

https://github.com/ReactTraining/react-router/pull/6961 fixed the bug in regards to how react versions are detected and was merged in 14 days ago but no new version has been cut with this fix. Can we get a 3.2.5 with that fix?
cc: @timdorr

Yes please, this would be great 👍

Sorry for the delay. Just pushed it now. npm install react-router@previous


can I import gl-matrix without webpack?

I want to use gl-matrix with import grammar, so I use it like this
<script src="./index.js" type="module"></script>
import mat4 from '../utils/src/mat4.js'
unfortunately, the chrome dev tool print

Uncaught SyntaxError: The requested module '../utils/src/mat4.js' does not provide an export named 'default'

if I use like this import {create} from '../utils/src/mat4.js', it would work. Howerver, I need to update it while I need use a new function

Yes, you very much can.
Your issue stems from the fact that mat4.js does not have a default export, but rather has a number of named imports for each function. If you want to import all of them at once, you can use
import * as mat4 from '../utils/src/mat4.js',

This is exactly what index.js does.
Now, if you want to import mat4 from index.js, you have to include curly brackets around it { }.
import { mat4 } from '../utils/src/index.js';

import { mat4, vec3, quat2} from '../utils/src/index.js'; // You can also import multiple things at once.

thank you very much, how foolish I am!

Glad I could help you.


youtube channel page downloads broken

Checklist

[X] I'm reporting a broken site support issue
[X] I've verified that I'm running youtube-dlc version 2020.10.31
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar bug reports including closed ones
[X] I've read bugs section in FAQ

Verbose log

youtube-dlc.exe --ignore-config -i -o "%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s" --prefer-ffmpeg --merge-output-format mkv --write-sub --all-subs --convert-subs srt --add-metadata --write-description --write-annotations --write-all-thumbnails --write-info-json --cookies youtube-cookies.txt -v --proxy "" https://www.youtube.com/channel/UC7qPftDWPw9XuExpSgfkmJQ
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ignore-config', '-i', '-o', '%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s', '--prefer-ffmpeg', '--merge-output-format', 'mkv', '--write-sub', '--all-subs', '--convert-subs', 'srt', '--add-metadata', '--write-description', '--write-annotations', '--write-all-thumbnails', '--write-info-json', '--cookies', 'youtube-cookies.txt', '-v', '--proxy', '', 'https://www.youtube.com/channel/UC7qPftDWPw9XuExpSgfkmJQ']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.07
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-04-24-a501947, ffprobe git-2020-04-24-a501947
[debug] Proxy map: {}
[youtube:channel] UC7qPftDWPw9XuExpSgfkmJQ: Downloading channel page
[youtube:playlist] UU7qPftDWPw9XuExpSgfkmJQ: Downloading webpage
[download] Downloading playlist: UU7qPftDWPw9XuExpSgfkmJQ
[youtube:playlist] playlist UU7qPftDWPw9XuExpSgfkmJQ: Downloading 0 videos
[download] Finished downloading playlist: UU7qPftDWPw9XuExpSgfkmJQ

Description

youtube playlist downloads seem to have broken sometime within the last few hours. it was working earlier this morning, now all playlists result in "0 videos" from youtube-dlc. presumably youtube made a schema change which broke youtube-dlc.
Individual video downloads still work, for example this works perfectly:
youtube-dlc.exe --ignore-config -i -o "%(uploader)s (%(uploader_id)s)/%(upload_date)s - %(title)s - (%(duration)ss) [%(resolution)s] [%(id)s].%(ext)s" --prefer-ffmpeg --merge-output-format mkv --write-sub --all-subs --convert-subs srt --add-metadata --write-description --write-annotations --write-all-thumbnails --write-info-json --cookies youtube-cookies.txt -v --proxy "" https://www.youtube.com/watch?v=SXGAiTjFiMA

@insaneracist @blackjack4494
I can confirm that this is broken for links like:
https://www.youtube.com/user/anjunadeep/videos
and "regular" playlists like this ID: PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV
Full link: https://www.youtube.com/watch?v=mhpfOQ6sf3s&list=PLOftnzGIKwJAfH33eTU2wDsBqwxOMRedV
Mix, search and user created playlists seem to be working just fine:
mix id to test: RDKR9wGi7gVLQ
search playlist to test: https://www.youtube.com/results?search_query=linkin+park+numb

seems widely confirmed: https://www.reddit.com/r/youtubedl/comments/jr56fs/yourubedl_sees_all_channel_links_as_empty/

I too am seeing this issue, looks to be a global thing, same problem wherever I initiate the download from

It would appear that youtube has disabled the "disable_polymer" feature that the YT extractor requires,
When visiting a playlist with the flag set, I now get unknown param

I can see the extractor is doing `
        query['disable_polymer'] = 'true'

so that is my hunch

@ainsey11 I can second what you've found there. Seems you're on the right track.

Sorry if its a dumb question: Why does the extractor need the "old theme / non-polymer"?
Is it not currently able to pull the video IDs from the new theme?

Sorry if its a dumb question: Why does the extractor need the "old theme / non-polymer"?
Is it not currently able to pull the video IDs from the new theme?

The extractor currently looks for HTML markup for each entry in a playlist. When the polymer frontend was introduced, YouTube changed from server-side rendering of HTML templates to client-side rendering. As a result, the HTML source received from the server no longer contains the markup the extractor looks for, and the only place that playlist items appear is in a JSON structure (and a fairly dense one at that). Now that the extractor can no longer opt out of polymer, it will need to be updated to pull data out of that structure.

It seems to be failing at youtube_dlc/extractor/youtube.py:_extract_playlist
https://github.com/blackjack4494/yt-dlc/blob/master/youtube_dlc/extractor/youtube.py#L2890-L2892
Every match after these lines isn't working.

Individual video downloads still work, for example this works perfectly

how to use playlist track number from Individual video downloads?

@candrapersada That's fine as a hack, but for all of us who do archival, it's a really bad hack to put in place. Using that method, there is no way to scan the archive for all of the playlist items and avoid downloading the page for every single one of those items. I archive over 150 channels regularly and dropping down to this method would be a little bit of a disaster.

If no one has taken up the task of getting this fixed yet, I might give it a try tomorrow.

Check network tab in browser. They likely completely changed over to api usage returning json instead of html events

what I mean is how to want youtube-dlc to name the files like 1-{name}, 2-{name}, ... n-{name} in order to be able to watch them in the same sequence as original Youtube playlist. In other words I need my downloaded videos to be prefixed with numbers. How can I do that?

@candrapersada, check out the readme section called output template
if playlist downloading isn't broken, you can do
youtube-dlc -o "%(playlist_index)03d - %(title)s.%(ext)s" "https://www.youtube.com/playlist?list=..."

Hm, the playlist's title is still missing from these fixes @ZenulAbidin @insaneracist
Similar to #109.
python3.9 ./__main__.py --flat-playlist --add-metadata --all-subs -J -F 'https://youtube.com/playlist?list=PLq-W-cmkohAPAcxZWxh_ysJWPhqDxkETs'

Doesn't return a valid title key.

@arter97 fixed

Expect a release with a patch for this coming today (anything between 1-10 hrs depending on when I got time). Python version will be fixed then as well as win xp support is back then.

I was booked today, now that I have some free time I’ll see where in the JSON our variables have moved to and hopefully find a regex to match them.

That's unfortunate
https://github.com/actions/upload-release-asset/issues/68
Release is delayed till this issue is fixed.

@blackjack4494 Seems that the linked issue is fixed, according to comments. Would you have a potential ETA on release? Thank you :pray:

why name folder broken %(playlist)s/?

Hi everyone. I am trying to use this command to download playlist but not able to do so since past 2-3 days. Till now I can't see any fix for this. If anyone can tell me how to go past this issue it would be great help.
youtube-dl -c -i -f 18  https://www.youtube.com/playlist?list=PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading webpage
[download] Downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] playlist PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading 0 videos
[download] Finished downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt

@harrypotter0 Which system are you using? For Manjaro Linux, a package update was available when I checked 9 hours ago, since then it works again. Some other distributions might take longer. Debian for example sometimes takes months, so you might want to download it from Github and install it that way instead.

Hi everyone. I am trying to use this command to download playlist but not able to do so since past 2-3 days. Till now I can't see any fix for this. If anyone can tell me how to go past this issue it would be great help.
youtube-dl -c -i -f 18  https://www.youtube.com/playlist?list=PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading webpage
[download] Downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt
[youtube:playlist] playlist PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt: Downloading 0 videos
[download] Finished downloading playlist: PLv_5Gzr5T6wz0G1Ff3SO97ElLq1G0POWt

This particular playlist is working on the latest release.
try also using --rm-cache-dir --no-cache-dir params in your command.

When I download my Youtube watch later playlist, it only downloads the first page. Any videos beyond the first 100 don't get downloaded. This happens using both youtube-dlc --cookies cookies.txt :ytwatchlater and youtube-dlc --cookies cookies.txt https://www.youtube.com/playlist?list=WL. I am using version 2020.11.11-3, and I've cleared my cache directory. I've tried messing with the --playlist-start and --playlist-end flags, but those don't seem to help either. Could this be because the watch later playlist has some unique polymer structure that got missed in the patch?

When I download my Youtube watch later playlist, it only downloads the first page. Any videos beyond the first 100 don't get downloaded. This happens using both youtube-dlc --cookies cookies.txt :ytwatchlater and youtube-dlc --cookies cookies.txt https://www.youtube.com/playlist?list=WL. I am using version 2020.11.11-3, and I've cleared my cache directory. I've tried messing with the --playlist-start and --playlist-end flags, but those don't seem to help either. Could this be because the watch later playlist has some unique polymer structure that got missed in the patch?

The same happen to me.  When the version 2020.11.11-3 came out, it fixes it. Now it broke again

This particular playlist is working on the latest release.
try also using --rm-cache-dir --no-cache-dir params in your command.

Can confirm that this works, nice

I updated to today's youtube-dl version through pip. Still, none of the auxiliary options like --rm-cache-dir --no-cache-dir seem to work. Tried to download several completely different channels' videos and yet, it always tells me, that there are only 0 videos to download, which is absolutely false.
What am I missing here? Is this truly fixed?

Try updating to master with git. Search for git in readme :)


[Bug][issue-2754] correct yarn url address

What changes were proposed in this pull request
Issue Number: close #2754
Verifying this change

Manually verified the change by testing locally.

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): (yes / no)

oh, I'm so sorry to provide a not correct url of sample for correct yarn application url. by test, the url is like follow:
http://host:port/cluster/app/application_xxxxxxxxxxxxxx_xxxx

oh, I'm so sorry to provide a not correct url of sample for correct yarn application url. by test, the url is like follow: http://host:port/cluster/app/application_xxxxxxxxxxxxxx_xxxx

It doesn't matter, the proxy url also works.


gcstar: 1.7.2 -> 1.7.3

Motivation for this change
Version bump. This is the first GTK3 version, so it is considered a major upgrade.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 122644 run on x86_64-linux 1

1 package built:

gcstar

It builds and it works :+1:


Added the setName function

We needed to change tag names during dom manipulations, so we added the setName function for this.

undefined method PHPHtmlParser\Dom\HtmlNode::setName()


Fixes #1434 Web-Admin: remove internal repositories

Fixes #1434
Remove internal repo from package-lock.json and use npm ci instead of npm install.

Ping @vsct-jburet @francoisno, this is a really small PR ;)

Thank you !


Dist version for es2015

I would like to use this library without babel for requirejs, is is posible? Thank you.

This works:
https://gist.github.com/aiphee/a3663228d87b7abfa1f6a641ca223d10


this function must be called on the uimanager queue

"react": "16.9.0",
"react-native": "0.61.5",
the android version not error but ios is a error

@luciamaj @minh-dai did anyone solve this? I would really appreciate any help. Thank you!

are you using the simulator or a real device?

@antoinerousseau  a real device

I'm having the same issue on a real iPhone, so I moved to react-native-rectangle-scanner.

Same here :( any news?


Feat(eos_cli_config_gen): Add support for "flow tracker hardware FOO" for interfaces that support it

          This PR does not allow to configure `flow tracker hardware FOO` under ethernet-interfaces. This will be tracked in another issue.

Originally posted by @gmuloc in https://github.com/aristanetworks/ansible-avd/issues/3199#issuecomment-1746994605

Actually added as part of https://github.com/aristanetworks/ansible-avd/pull/3199


Update to Sinatra 2.0 dependency for Rails 5

Updates for Rails 5.
Just testing at the moment.  No idea on backwards compatibility.

Actually.  Running away might be a good move until Sinatra officially releases 2.0:
Bundler could not find compatible versions for gem "rack":
  In Gemfile:
    heroku-bouncer was resolved to 0.7.1, which depends on
      rack (~> 1.0)

    heroku-bouncer was resolved to 0.7.1, which depends on
      omniauth-heroku (~> 0.1) was resolved to 0.3.0, which depends on
        omniauth-oauth2 (~> 1.2) was resolved to 1.4.0, which depends on
          oauth2 (~> 1.0) was resolved to 1.2.0, which depends on
            rack (< 3, >= 1.2)

    heroku-bouncer was resolved to 0.7.1, which depends on
      omniauth-heroku (~> 0.1) was resolved to 0.3.0, which depends on
        omniauth-oauth2 (~> 1.2) was resolved to 1.4.0, which depends on
          omniauth (~> 1.2) was resolved to 1.3.1, which depends on
            rack (< 3, >= 1.0)

    sinatra was resolved to 2.0.0.beta2, which depends on
      rack-protection (= 2.0.0.beta2) was resolved to 2.0.0.beta2, which depends on
        rack

    rack-test (~> 0.6) was resolved to 0.6.3, which depends on
      rack (>= 1.0)

    sinatra was resolved to 2.0.0.beta2, which depends on
      rack (~> 2.0)

Unlikely to change to ~> 2.0 as that will leave old users stranded without updates. >= 1.0, < 3 is fine if everything works with Sinatra 2.

Closing in favor of #52


fix new example location

ping @kartben, do you have time to fix the issues that the sanitychecker is complaining?

As this PR seems to be abandoned, I created a new one at #23522 so this one can be closed.


Heap buffer overflow in OP_SEND

The following input demonstrates a crash:
class << Kernel
  sym = '`'.to_sym

  alias_method :old_cmd, sym

  define_method(sym) { `test`; x }

  `test`
end

t = `#{sym}`

ASAN report:
==45104==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61d00001de88 at pc 0x000105b91386 bp 0x7fff5a1d4610 sp 0x7fff5a1d4608
WRITE of size 4 at 0x61d00001de88 thread T0
    #0 0x105b91385 in mrb_vm_exec (mruby:x86_64+0x10016f385)
    #1 0x105b88519 in mrb_vm_run (mruby:x86_64+0x100166519)
    #2 0x105bbaf89 in mrb_top_run (mruby:x86_64+0x100198f89)
    #3 0x105c8cca5 in mrb_load_exec (mruby:x86_64+0x10026aca5)
    #4 0x105c8d5f5 in mrb_load_file_cxt (mruby:x86_64+0x10026b5f5)
    #5 0x105a24906 in main mruby.c:227
    #6 0x7fffbbbba234 in start (libdyld.dylib:x86_64+0x5234)

0x61d00001de88 is located 8 bytes to the right of 2048-byte region [0x61d00001d680,0x61d00001de80)
allocated by thread T0 here:
    #0 0x105e38520 in wrap_realloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x56520)
    #1 0x105b1d235 in mrb_default_allocf (mruby:x86_64+0x1000fb235)
    #2 0x105a9e578 in mrb_realloc_simple gc.c:202
    #3 0x105a9ec5e in mrb_realloc gc.c:216
    #4 0x105a9f6e3 in mrb_malloc gc.c:237
    #5 0x105a9f77d in mrb_calloc gc.c:255
    #6 0x105b7f442 in stack_init (mruby:x86_64+0x10015d442)
    #7 0x105b7c600 in mrb_funcall_with_block (mruby:x86_64+0x10015a600)
    #8 0x105b7bf3a in mrb_funcall_with_block (mruby:x86_64+0x100159f3a)
    #9 0x105b7b727 in mrb_funcall_argv (mruby:x86_64+0x100159727)
    #10 0x105a629b5 in mrb_obj_new (mruby:x86_64+0x1000409b5)
    #11 0x105a8752d in mrb_exc_new_str (mruby:x86_64+0x10006552d)
    #12 0x105a912f7 in mrb_init_exception (mruby:x86_64+0x10006f2f7)
    #13 0x105ac2320 in mrb_init_core (mruby:x86_64+0x1000a0320)
    #14 0x105b1d1ce in mrb_open_core (mruby:x86_64+0x1000fb1ce)
    #15 0x105b1d39c in mrb_open_allocf (mruby:x86_64+0x1000fb39c)
    #16 0x105b1d367 in mrb_open (mruby:x86_64+0x1000fb367)
    #17 0x105a23818 in main mruby.c:171
    #18 0x7fffbbbba234 in start (libdyld.dylib:x86_64+0x5234)

SUMMARY: AddressSanitizer: heap-buffer-overflow (mruby:x86_64+0x10016f385) in mrb_vm_exec
Shadow bytes around the buggy address:
  0x1c3a00003b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a00003b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a00003ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a00003bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a00003bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1c3a00003bd0: fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c3a00003be0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c3a00003bf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c3a00003c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c3a00003c10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x1c3a00003c20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==45104==ABORTING
Abort trap: 6

Debug:
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = Heap buffer overflow
  * frame #0: 0x00000001004a3940 libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie()
    frame #1: 0x00000001004ba248 libclang_rt.asan_osx_dynamic.dylib`__sanitizer::Die() + 88
    frame #2: 0x00000001004a1207 libclang_rt.asan_osx_dynamic.dylib`__asan::ScopedInErrorReport::~ScopedInErrorReport() + 311
    frame #3: 0x00000001004a0c9a libclang_rt.asan_osx_dynamic.dylib`__asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) + 410
    frame #4: 0x00000001004a1d8e libclang_rt.asan_osx_dynamic.dylib`__asan_report_store4 + 46
    frame #5: 0x000000010016f386 mruby`mrb_vm_exec(mrb=0x000061400000a440, proc=0x000062f000001f90, pc=0x000060300000b0bc) at vm.c:1206
    frame #6: 0x000000010016651a mruby`mrb_vm_run(mrb=0x000061400000a440, proc=0x000062f000002080, self=mrb_value @ 0x00007fff5fbfe140, stack_keep=6) at vm.c:827
    frame #7: 0x0000000100198f8a mruby`mrb_top_run(mrb=0x000061400000a440, proc=0x000062f000002080, self=mrb_value @ 0x00007fff5fbfe340, stack_keep=0) at vm.c:2651
    frame #8: 0x000000010026aca6 mruby`mrb_load_exec(mrb=0x000061400000a440, p=0x000062800000c120, c=0x000060600000a040) at parse.y:5780
    frame #9: 0x000000010026b5f6 mruby`mrb_load_file_cxt(mrb=0x000061400000a440, f=0x00007fffc4ab40b0, c=0x000060600000a040) at parse.y:5789
    frame #10: 0x0000000100002907 mruby`main(argc=2, argv=0x00007fff5fbff4f0) at mruby.c:227
    frame #11: 0x00007fffbbbba235 libdyld.dylib`start + 1
(lldb) frame select 5
frame #5: 0x000000010016f386 mruby`mrb_vm_exec(mrb=0x000061400000a440, proc=0x000062f000001f90, pc=0x000060300000b0bc) at vm.c:1206
   1203	        bidx = a+n+1;
   1204	      }
   1205	      if (GET_OPCODE(i) != OP_SENDB) {
-> 1206	        SET_NIL_VALUE(regs[bidx]);
   1207	      }
   1208	      else {
   1209	        mrb_value blk = regs[bidx];

This issue was reported by https://hackerone.com/bku

https://hackerone.com/geeknik reported another input producing the same crash:
def`r
``end+``


Update task.json

Fixed small UI typo - changed "Test Errors as Warnings" to "Treat Errors as Warnings"

Thanks for the PR
Would you mind also editing the Extensions Readme to add a line along the lines of
2.2 Fixed Typo (PR#283)
I know I could do it, but if is in your PR you will appear as the contributor in the Squashed commit as opposed to me.

Sorry I did not explain myself, I meant add the readme.md edit to this  PR as a second commit.
I don't want to waste your time any more, so don't worry I will merge the two PRs and sort it all out, hopefully tomorrow

Sorry about that, I'm pretty new to using git. I've added the readme.md to this PR and closed off the other one.

The changes are live as 2.2.2, I only incremented the final . version as I was testing my release pipeline and needed to do two releases
Thanks for the contribution


Update ProprietaryHeadersRuleTest.kt

Fixes #1404
validateResponseHeaders should return no violation for a specified proprietary header

👍

Codecov Report
Base: 76.44% // Head: 88.09% // Increases project coverage by +11.64% :tada:

Coverage data is based on head (4dc9132) compared to base (2e5b8f7).
Patch has no changes to coverable lines.

Additional details and impacted files
@@              Coverage Diff              @@
##               main    #1405       +/-   ##
=============================================
+ Coverage     76.44%   88.09%   +11.64%
=============================================
  Files           170       32      -138
  Lines          2798      378     -2420
  Branches        489       55      -434
=============================================
- Hits           2139      333     -1806
+ Misses          413       32      -381
+ Partials        246       13      -233

Impacted Files
Coverage Δ

.../zally/ruleset/zally/AtMostOneBodyParameterRule.kt

...eset/zalando/QueryParameterCollectionFormatRule.kt

...ally/ruleset/zalando/PluralizeResourceNamesRule.kt

...ruleset/zalando/SuccessResponseAsJsonObjectRule.kt

...r/src/main/kotlin/org/zalando/zally/dto/RuleDTO.kt

...org/zalando/zally/apireview/ApiDefinitionReader.kt

...zalando/zally/ruleset/zalando/ApiIdentifierRule.kt

...tlin/org/zalando/zally/core/JsonSchemaValidator.kt

...in/org/zalando/zally/core/ContextRulesValidator.kt

...org/zalando/zally/apireview/ApiReviewRepository.kt

... and 128 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Reach final decision on resolve_code_hash_fn

So inside trace_decoder, we've long had a closure that the caller must provide to handle the case of encountering a code hash that is not present inside the actual trace. The closure is given the code hash that we are missing the byte code for and returns the corresponding byte code. Currently this is not use by zero-bin for Jerrigon's mpt compact.
Just as a recap, both of the two compact formats that we currently support (Jerrigon mpt & smt) embed all accessed contract code within the compact, but the intent of our decoder protocol is to remain flexible and not require this. As an example, we can make the compact even smaller if the byte code is never included in the compact and queried/cached by zero-bin every time we encounter one that we have not seen before.
Assuming that we still want to support multiple compact formats, I feel pretty strongly that we should support a code resolve closure being passed in (or some other similar mechanism). If however we want to drop support and keep the scope more narrow, then we probably should remove this.

I ran into this when working on https://github.com/0xPolygonZero/zero-bin/pull/51. My thinking is that it may be preferable to introduce a trait of the form:
/// A trait for resolving code hashes into bytes.
pub trait CodeHashResolver {
    /// Resolves the code hash into bytes.
    fn resolve(&self, c_hash: &CodeHash) -> Vec<u8>;

    /// Inserts the code hash and its bytes into the resolver.
    fn insert_code(&mut self, c_hash: H256, code: Vec<u8>);
}

Different objects could implement this trait depending on the mode of operation. It would then be the responsibility of the client to inject the appropriate object for code resolving.

@BGluth  Excellent - sounds good, thanks!

Just an FYI. With @0xaatif's refactor, this is still relevant. IIRC wasn't a PR created that made the changes that @frisitano mentioned above? Did it never get merged in?

Did you mean this PR @BGluth ?

It is my intention to remove all code paths that we don't exercise, and to have all features motivated by user need, so if we've been running without this, I have a preference for removing it is motivated

Alright. If the consensus is to remove this, then I guess we can probably close the issue.

Addressed in #514


Export a unity package breaks the actions in the animation steps

If I try to export a package with an Animation Sequencer Controller and open it in another unity project, I can see the steps, but all the actions are empty.
Is there anything I can do?
Thanks

I need to investigate this further, not sure if this is something easy to do to be honest.

Thank you. In the meantime do you have any suggestion to be able to copy an animation sequencer controller from a project to another?

Sadly I don't think there's an easy of doing this ATM, since most of the references are inside the package.
I'm have plans to implement a way of exporting the animation as a shareable file somehow, but its really far from production ready.


trying to access paris pronote but getting error

Hi,
Today I tried accessing my student pronote, but got following error:
%Run pronoteTest.py Traceback (most recent call last):   File "/home/pi/desksave/pronoteTest.py", line 8, in <module>     ent=ile_de_france)   File "/usr/local/lib/python3.7/dist-packages/pronotepy/pronoteAPI.py", line 71, in __init__     self.attributes, self.func_options = self.communication.initialise()   File "/usr/local/lib/python3.7/dist-packages/pronotepy/pronoteAPI.py", line 357, in initialise     self.attributes = self._parse_html(get_response.content)   File "/usr/local/lib/python3.7/dist-packages/pronotepy/pronoteAPI.py", line 494, in _parse_html     "Page html is different than expected. Be sure that pronote_url is the direct url to your pronote page.") pronotepy.pronoteAPI.PronoteAPIError: Page html is different than expected. Be sure that pronote_url is the direct url to your pronote page.
The code is following:
`import pronotepy
from pronotepy.ent import ile_de_france
creating the client and passing the occitanie_montpellier function to automatically get cookies from ENT
client = pronotepy.Client('https://0750611g.index-education.net/pronote/eleve.html',
username='secret.idunno',
password='DontPeek',
ent=ile_de_france)
check if sucessfully logged in
if client.logged_in:
print(len(client.messages())) # printing number of messages that the user has
else:
print('no login')`
If I use it normally i login using the paris ENT and then i click on the pronote app.
Help?

Hello, we don't support the paris ENT yet. Maybe @Xiloe could take a look?

I modified the URL from ile_de_france to https://ent.parisclassenumerique.fr/auth/login?callback=https%3A%2F%2Fent.iledefrance.fr%2F and it worked.
Waiting for the new copy-pasted and name-modified function to be implemented.

(Now added as paris_classe_numerique in the ENTs)


Issue

I'm new to this not sure what to do here

Uncompress it Sent from my iPhoneOn Jul 25, 2023, at 2:53 AM, Mesh3al1 @.***> wrote:
I'm new to this not sure what to do here
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>


Install utilities separately

I am having troubles trying to install just one utility (say, fbpython) with luarocks, automatically sourcing the code (i.e. without cloning the repo).
I've tried
luarocks install --server=https://raw.githubusercontent.com/facebook/fblualib/master/fblualib/python/rockspec/ fbpython

and, of course, the manifest is missing. Then, after building it (in my forked repo), I got the following error message
Error: Directory fblualib/python not found inside archive fblualib

Any idea of how I can install just the fbpython utility, avoiding cloning the whole repo?

Hi @Atcold , I was able to install fblualib on my mac using docker. Namely, I installed fblualib on top of this image and it worked:
docker pull kaixhin/cuda-torch

Sweet @vzhong, good to know!

@vzhong , I don't know much about docker.  Can you please share your installation steps? I need to install fblualib on my mac.  Thanks!

@jeiranj I haven't worked with Torch for a while now so some of this is a bit hazy. Basically you can

install docker on your mac (e.g. through homebrew brew cask install docker, then open Docker.app to finish the installation).
docker pull kaixhin/cuda-torch
boot up the instance: docker run -i -t kaixhin/cuda-torch /bin/bash
install fblualib

The last step:

git clone https://github.com/facebook/fblualib.git
cd fblualib.git
./install_all.sh

The catch, of course, is that now you must run th in your docker instance.

I see... thanks!


Problem with Custom PropertySortOrder

Consider this group of css properties:
.selector {
    position: absolute;
    top:    0;
    bottom: 0;
    padding: 20px;
    width: $drawer-narrow-width;
    background: $drawer-bg-color;
    transition: left 0.25s ease-out;
    overflow-y: auto;
    box-sizing: border-box;
}

in conjunction with the following PropertySortOrder settings:
PropertySortOrder:
  enabled: true
  ignore_unspecified: false
  separate_groups: false
  order:
    - display
    -
    - position
    - top
    - right
    - bottom
    - left
    - z-index
    -
    - margin
    - margin-top
    - margin-right
    - margin-bottom
    - margin-left
    -
    - margin-collapse
    - margin-top-collapse
    - margin-right-collapse
    - margin-bottom-collapse
    - margin-left-collapse
    -
    - padding
    - padding-top
    - padding-right
    - padding-bottom
    - padding-left
    -
    - width
    - height
    - max-width
    - max-height
    - min-width
    - min-height
    -
    - float
    - clear
    -
    - color
    -
    - font
    - font-size
    - font-style
    - font-family
    - font-weight
    - font-smoothing
    -
    - line-height
    - letter-spacing
    - word-spacing
    -
    - text-align
    - text-indent
    - text-shadow
    - text-overflow
    - text-rendering
    - text-transform
    - text-decoration
    - text-size-adjust
    -
    - word-break
    - word-wrap
    -
    - white-space
    -
    - background
    - background-size
    - background-color
    - background-image
    - background-repeat
    - background-position
    - background-attachment
    -
    - border
    - border-top
    - border-right
    - border-bottom
    - border-left
    -
    - border-image
    - border-spacing
    - border-collapse
    -
    - border-color
    - border-top-color
    - border-right-color
    - border-bottom-color
    - border-left-color
    -
    - border-style
    - border-top-style
    - border-right-style
    - border-bottom-style
    - border-left-style
    -
    - border-width
    - border-top-width
    - border-right-width
    - border-bottom-width
    - border-left-width
    -
    - border-radius
    - border-top-right-radius
    - border-bottom-right-radius
    - border-bottom-left-radius
    - border-top-left-radius
    - border-radius-topright
    - border-radius-bottomright
    - border-radius-bottomleft
    - border-radius-topleft
    -
    - box-shadow

The ignored_unspecified: false setting should allow any property, in any order, to be placed at the bottom of .selectors property block. If you try to lint the aforementioned code you'll notice that there is no way to get the last three properties to pass the linting.

Hey @meowsus,
You want ignore_unspecified: true, since you want to ignore transition/overflow-y/box-sizing etc. since they are not defined in your custom order.
Is there something in the documentation that would make this clearer?

ignore_unspecified: false seems to work almost perfectly for what I need, for the most part it allows all properties defined in the list to be forced in it's proper order, then, all other properties that aren't part of the defined order, should be put below the defined property order.
With ignore_unspecified: true the uncaught properties can be placed pretty much anywhere, because they are just ignored.
What I'm looking for is, first, the defined order is enforced then, second, the properties that are not defined are forced to the bottom of the block, in any order.
The documentation made it seem like that was the concept behind ignore_unspecified: false

Sorry @meowsus, I misunderstood what you were getting at.
The problem here is that Ruby's Array#sort method is not stable. As a result, the unspecified properties could have their relative order shuffled. When I run your example above, this is the order it expects:
.selector {
    position: absolute;
    top:    0;
    bottom: 0;
    padding: 20px;
    width: $drawer-narrow-width;
    background: $drawer-bg-color;
    overflow-y: auto;
    transition: left 0.25s ease-out; // <-- Swapped places with overflow-y
    box-sizing: border-box;
}

I'll implement a fix shortly.

Righteous! I'll snag the new version today and test.
Thanks for your help @sds!

Hey @sds, do you know when v0.39.0 might be released? I'm unable to pull it down from the github repo or from rubygems.org.

Hey @meowsus, 0.39.0 has already been released.
Your problem is likely due to the scss-lint gem being renamed to scss_lint. Please see the release notes for 0.38.0 for instructions.

Thanks for the link to the release notes! This will be very helpful.
I'm still seeing the issue in v0.39.0 however. Using the same property order as I posted originally, this block exhibits the same behavior. transform -> box-sizing can't be ordered without an error being thrown.
https://gist.github.com/89d80eb2436e27d63d91

Sorry @meowsus, I only read your comment about being unable to pull from Rubygems. I didn't look at the issue we were discussing. :)
This fix has not landed on master yet.

No problem. I could have done a better job making sense. Thanks for the response!


MaxQuant .Net Core issue

MaxQuant not working on hpZ8 computer. Error message keeps popping up each time a session is started making it unable to process.
netcore issue.pdf
.Net Core 3.1 is installed on my desktop and I have restarted my computer since installing the program but this error still pops up.
Any insights on what else I could do?

It must be because dotnet 3.1 is outdated update it to 6.0 or atleast 5.0

I met the same problem. And I have installed the latest version of .Net Core SDK, including ".Net Core SDK 3.1.421", ".NET SDK 6.0.302", ".NET Framework 4.8.1 SDK". With the older version of MaxQuant (<IP_ADDRESS>), the problem may be solved, but a new problem emerged: value is either too big or too small for a double.

CC @tannergooding

These all sound like issues with MaxQuant and are not actionable by the .NET team. You should file an issue with the authors of MaxQuant to get a fix.
The first issue might be related to whether the .NET Desktop Workload is installed. The latter issue does not have enough details for anything to be actionable. Something like a stack trace or minimal repro would be needed.