Multi-GPU when using Torchtext iterator for data loading

Hi there,
I have just discovered pytorch-lightning few days ago and it seems awesome (congratulations!)
I have a question I cannot solve by reading the docs and examples.
Is is fully compatible with Torchtext?
I am trying to use a Torchtext iterator to load the data in batches, and I have managed to make it work for a single GPU, but when I add additional GPUs to the trainer:
trainer = Trainer(experiment=exp, gpus=[0, 1])
it breaks saying:
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:397
I understand that the problem comes from the model and the data not being placed in the same GPU.
I am following the provided template, replacing the MNIST parts with my own data.
The way I load the training data is:
@pl.data_loader
    def tng_dataloader(self):
        print('tng data loader called')
        DEVICE = torch.device(next(self.parameters()).device if torch.cuda.is_available() else "cpu")
        print('Current device ', DEVICE)
        (train_iter,) = BucketIterator.splits(
            (self.train_data,),
            sort=False,
            batch_size=self.hparams.batch_size,
            shuffle=True,
            repeat=False,
            device=DEVICE  # what to do here? what will do Lightning w.r.t. device?
        )

        return train_iter

I use that little hack to get the current gpu device to parameterize the Torchtext BucketIterator, because if I leave the Torchtext iterator "device" field empty it defaults to cpu, and I get the corresponding complaint when the training starts with the model in the gpu:
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
But this hack does not work for more-than-one-gpu setting.
Am I missing something or am I doing something wrong?
I could also reimplement my data loading using regular Pytorch dataloaders as in the template, but I would like to know if I can stick to Torchtext and still get the multi-gpu goodies from Lightning :)
Thanks in advance!

hi! install from master and try again? i believe we pushed a fix for this on master. if not, i can look at it deeper

@aitor-garcia-p actually, just released a new version with these fixes. Try again? if not we'll take a deeper look at it

Hi again,
After digging a bit (with my limited understanding), I see that in this function,
https://github.com/williamFalcon/pytorch-lightning/blob/4c61d1f30a29db8404606c6c933a5a2f3c0ae1ae/pytorch_lightning/trainer/trainer.py#L1077
if the "batch" parameter is a torchtext.data.Batch object (as it happens when using a Torchtext Iterator) the Trainer function transfer_batch_to_gpu will miss it despite having several conditionals.
I have made a test adding this additional condition:
elif getattr(batch, 'fields', None):
            for f_name in batch.fields:
                setattr(batch, f_name, self.transfer_batch_to_gpu(getattr(batch, f_name), gpu_id))
            return batch

(Or any other condition that catches a torchtext.data.Batch instance)
And it has started working nicely in single GPU with any additional hack to set the device on the iterators.
But still I cannot make multi-gpu working when the batches come from a torchtext iterator.
I instantiate the Trainer like this:
trainer = Trainer(experiment=exp, gpus=[0, 1], distributed_backend='ddp')
And it complains about the following:
   mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "/DATA/agarciap_data/python_stuff/python_envs/venv_for_remotes/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 162, in spawn
    process.start()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/local/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/local/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/pickle.py", line 786, in save_tuple
    save(element)
  File "/usr/local/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/usr/local/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: 'generator' object is not callable

It seems that there is something with the torchtext iterator that prevents from a proper serialization for the distributed processes spawn.
Or there is something else missing. For example, I have noticed that for ddp the Pytorch Dataloaders needs a DistributedSampler, but I don't know how does the torchtext iterators deal with that.

yeah, looks like torchtext can't be pickled and thus not used with DDP. But you should verify that on the torchtext issues. If that's true, then i'd recommend DP or we can try to come up with a work around

Also, feel free to submit a PR with your changes so we can enable torchtext support

Hey @williamFalcon ,
Any progress on this, or a preferred workaround?
The ubiquity of torchtext together with this issue make if difficult to do NLP/seq2seq.

hey! sorry, been busy with deadlines but will look at it this week.

@ctlaltdefeat did you still want to submit this PR?
@jeffling @neggert anyone want to take a look at this?

I've been busy too, and I think it may be more of an issue between DistributedDataParallel and torchtext than anything that lightning adds per se.

That's correct, torchtext can't be pickled and you'll want to use DP.
Could you give a full stacktrace of the issue with DP? I'm not sure which step is emitting that error or if it's coming from dataloading or training.

The issue with DP (for me) is that the inability to use mixed-precision training offsets the benefit of multi-GPU training.
I think tomorrow I'll try to work around the issue by either converting the torchtext object to a standard DataLoader or by somehow separating the torchtext object from the model that needs to be pickled.

Any recent updates on this issue?

I am trying to run a torchtext dataset, it works fine with single GPU, but fails to work on dp, ddp (ddp2 out of bounds for me as no slurm). I think ddp may be an issue with another library (wandb.com).
But for dp I am getting the same error as OP.
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:397

@jeffling  this is the error trace for DP with torchtext:

16:17:51
Starting Model

16:17:51
Traceback (most recent call last):

16:17:51
File "/Siamese_BERT_blogpost/train.py", line 107, in <module>

16:17:51
trainer.fit(model)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 348, in fit

16:17:51
self.dp_train(model)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/dp_mixin.py", line 104, in dp_train

16:17:51
self.run_pretrain_routine(model)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 455, in run_pretrain_routine

16:17:51
self.evaluate(model, self.get_val_dataloaders(), self.nb_sanity_val_steps, self.testing)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop_mixin.py", line 50, in evaluate

16:17:51
test)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop_mixin.py", line 174, in evaluation_forward

16:17:51
output = model(*args)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*i

16:17:51
wandb: Waiting for W&B process to finish, PID 162

16:17:51
nput, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 65, in forward

16:17:51
outputs = self.parallel_apply(replicas, inputs, kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 69, in parallel_apply

16:17:51
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 199, in parallel_apply

16:17:51
raise output

16:17:51
File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/pt_overrides/override_data_parallel.py", line 165, in _worker

16:17:51
output = module.validation_step(*input, **kwargs)

16:17:51
File "/Siamese_BERT_blogpost/wrapper.py", line 42, in validation_step

16:17:51
out = self.forward(batch)

16:17:51
File "/Siamese_BERT_blogpost/wrapper.py", line 35, in forward

16:17:51
return self.siamese(batch)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py",wandb: Program failed with code 1. Press ctrl-c to abort syncing.

16:17:51
line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/Siamese_BERT_blogpost/models.py", line 46, in forward

16:17:51
premise = self.language_model(premise)[0]

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 735, in forward

16:17:51
embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 186, in forward

16:17:51
inputs_embeds = self.word_embeddings(input_ids)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__

16:17:51
result = self.forward(*input, **kwargs)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 114, in forward

16:17:51
self.norm_type, self.scale_grad_by_freq, self.sparse)

16:17:51
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1484, in embedding

16:17:51
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

16:17:51
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:400

I am running train.py from this repository: https://github.com/Genei-Ltd/Siamese_BERT_blogpost/blob/master/train.py

@aced125 it looks like the batches aren't put into the right GPU. Could you look at the example code the OP put out regarding the hack with torchtext to place things into the right GPU?
It also looks like @ctlaltdefeat had this working with DP, but couldn't use DP due to other reasons. Any tips?

@jeffling I've given up on torchtext datasets to be honest. It was easy enough to switch to a torch.utils.data.DataLoader instead.
I am going to try PL on graph convolutions soon (will be using the pytorch-geometric library which also uses a custom DataLoader (which inherits from the torch DataLoader) so will let you know if that works well.

Hello,
I am not able to get a simple toy example running using Torchtext iterators even on a single gpu. I am using a BucketIterator as follows:
train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), fields=(SRC, TRG))
SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)
train_iter, valid_iter, test_iter = BucketIterator.splits((train_data, valid_data, test_data), batch_size=batch_size)

My trainer code is:
model = SegmenterModule(80, 76)
exp = Experiment(save_dir=os.getcwd())
trainer = Trainer(gpus=1, max_nb_epochs=3, default_save_path='checkpoints')
trainer.fit(model)

But I get an error because the batch data is still on cpu and not moved to the gpu.
Stack trace:
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

As some people mentioned here, I cannot make it work even for a single GPU. I debugged the code and it seems like Batch object generated by torchtext.data.Iterator doesn't follow the rules described here
https://github.com/PyTorchLightning/pytorch-lightning/blob/45d671a4a81788b9d97fd6b47763816926e58e95/pytorch_lightning/trainer/distrib_parts.py#L420
As a result the data are not moved to GPU and the code gives my exception:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

@aitor-garcia-p @mateuszpieniak @jeffling let's close this one and continue discission how to improve the situation in #1245

Hello,
I am not able to get a simple toy example running using Torchtext iterators even on a single gpu. I am using a BucketIterator as follows:
train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), fields=(SRC, TRG))
SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)
train_iter, valid_iter, test_iter = BucketIterator.splits((train_data, valid_data, test_data), batch_size=batch_size)

My trainer code is:
model = SegmenterModule(80, 76)
trainer = Trainer(gpus=1, max_nb_epochs=3, default_save_path='checkpoints')
trainer.fit(model)

But I get an error because the batch data is still on cpu and not moved to the gpu.
Stack trace:
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
Can someone please help me figure out the problem or share a working example using torchtext iterators? Also, should I open a new issue with this problem or let this question be here?

Have you find out the solution yet?

Hello,
I am not able to get a simple toy example running using Torchtext iterators even on a single gpu. I am using a BucketIterator as follows:
train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), fields=(SRC, TRG))
SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)
train_iter, valid_iter, test_iter = BucketIterator.splits((train_data, valid_data, test_data), batch_size=batch_size)

My trainer code is:
model = SegmenterModule(80, 76)
trainer = Trainer(gpus=1, max_nb_epochs=3, default_save_path='checkpoints')
trainer.fit(model)

But I get an error because the batch data is still on cpu and not moved to the gpu.
Stack trace:
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
Can someone please help me figure out the problem or share a working example using torchtext iterators? Also, should I open a new issue with this problem or let this question be here?

Have you find out the solution yet?

I also have the same problem.

Currently, you need to transfer manually data to a GPU using torchtext. Take a look at my gist
https://gist.github.com/mateuszpieniak/f290b3a727db7e94b9da0bd3bd2e33c1 and the method transfer_batch_to_device.


add unn file

@Belta-shazzar I am sorry, the domain unn.edu.ng is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Program. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).


can not save screenshot on real device?

on simulator,I can save the current screen of the test app,but,on real device , I just save a black picture in UI Testing app,can not save screenshot on real device

The lib is using _UICreateScreenUIImage() private function to capture screenshots which doesn't work on devices. We need to find a different way of taking screenshots.

Thank you for a replay!
I will wait in hope~

Hi zmeyc,
Is there any luck on this issue ?
Hoping for the best from you


[msal-node-extensions]: Add electron webpack sample

Add the electron webpack sample for msal node extensions
Use a forked version of the bindings library

Codecov Report

Merging #4034 (f0fe422) into dev (7ca7297) will not change coverage.
The diff coverage is n/a.

Flag
Coverage Δ

*Carryforward flag

msal-angular
96.31% <ø> (ø)

Carriedforward from 7ca7297

msal-browser
87.74% <ø> (ø)

Carriedforward from 7ca7297

msal-common
85.08% <ø> (ø)

Carriedforward from 7ca7297

msal-core
76.66% <ø> (ø)

Carriedforward from 7ca7297

msal-node
81.45% <ø> (ø)

msal-react
94.17% <ø> (ø)

Carriedforward from 7ca7297

*This pull request uses carry forward flags. Click here to find out more.

Can you share the changes made in your fork of bindings?

Can you share the changes made in your fork of bindings?

Sure thing, I want to move the changes to a fork under AzureAD, so I can update the pull request.


Code mangles PATH variable that it's opened with

Using Insiders, just to be certain (impacts current release also)

VSCode Version: 1.26.0-insider
OS Version: macOS 10.13.6

Steps to Reproduce:
With a completely virgin install of macOS

At a system terminal, set $PATH to something interesting, then launch code:

extra-super-virgin-macos:~ alexw$ export PATH="/Users/alexw/Desktop:$PATH"
extra-super-virgin-macos:~ alexw$ echo $PATH
/Users/alexw/Desktop:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
extra-super-virgin-macos:~ alexw$ code-insiders --disable-gpu

In the vscode integrated terminal, show $PATH:

extra-super-virgin-macos:~ alexw$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/alexw/Desktop

That is to say, somehow, at some point, Code is taking the first item(s?) in my PATH and tacking them on the end instead, which, of course, alters the entire behavior of any shells with the altered version.
Does this issue occur when all extensions are disabled?: Yes (none installed at all)

Does not seem to be related to 30847.

What do you see when you type process.env['PATH'] in the console? Help -> Toggle Developer Tools -> Console

The same- whatever my base system path is, with whatever prepended customizations somehow appended instead.

In a system terminal (looks like VMWare did something since last I was testing this):
Alexs-MBP-2018:~
› echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public
Alexs-MBP-2018:~
› bash -l
Alexs-MBP-2018:~
› echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public

typically I'd workon my_env; code .
When I do that, I get this in the Code terminal:
Alexs-MBP-2018:~/Development/Python/Plotter
› echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Users/alexw/.virtualenvs/py3_plotter/bin
Alexs-MBP-2018:~/Development/Python/Plotter
› bash -l
Alexs-MBP-2018:~/Development/Python/Plotter
› echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Users/alexw/.virtualenvs/py3_plotter/bin

I'm not quite sure what bash -l is supposed to accomplish; it doesn't seem to be doing anything interesting.

@alexwhittemore can you also try this?
echo $PATH
export PATH="/Users/alexw/Desktop:$PATH"
bash -l
echo $PATH

I'm trying to identify whether code is running your login shell script again which is pushing /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin to the front of $PATH.
Relevant code pointer:
https://github.com/Microsoft/vscode/blob/5e5f6fc6ec785a602f1444500264f18d136b5e87/src/vs/code/node/shellEnv.ts#L78-L95

Ohhhhhhh interesting. That certainly looks to be what's happening:
Alexs-MBP-2018:~
› echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public
Alexs-MBP-2018:~
› export PATH="/Users/alexw/Desktop:$PATH"
Alexs-MBP-2018:~
› bash -l
Alexs-MBP-2018:~
› echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Users/alexw/Desktop

Related/duplicate of https://github.com/Microsoft/vscode/issues/55144 ?

@segevfiner I'm not sure about duplicate, but it sounds like it's related enough that perhaps your PR fixes it?
I've never compiled so much as an electron hello world, so I'm not sure how to go about testing your PR myself, but if I get a little more time than I have now, I'll look into it.
In the mean time, if you'd be willing to try it for me, the test case is pretty straightforward:
(system terminal:)
$ export PATH=~/Desktop:$PATH
$ echo $PATH
# should spit back "/Users/You/Desktop:/rest/of/your/PATH"
$ code .

(at code integrated terminal:)
$ echo $PATH
# if failed:
# /rest/of/your/path:/Users/You/Desktop
# If fixed:
# /Users/You/Desktop:/rest/of/your/path

Hmm @segevfiner I just noticed a behavior that does make this seem highly related - if I launch code . from a separate terminal NOT in a virtualenv, code adopts the environment of the first-opened window. In my case, that one was opened inside a virtualenv, and has a mangled path and the $VIRTUAL_ENV set.

@alexwhittemore I think given that bash -l repros this, you should be able to fix this by tweaking your launch scripts, you want to make them idempotent (don't break things if they're run multiple times).

I think I'm confused - what launch scripts? And what about them would take a perfectly fine path and do this nonsense to it?

Linux handles the profile and rc scripts differently and imo in a superior way, Windows is completely different so I do think this is a mac only problem. Here are the main differences:

Linux will run your profile script (ie. the login script) when you login to the system, when you launch a terminal Linux will run your .bashrc file
macOS will run your profile script when you launch a terminal

This caused a whole bunch of headaches a long time ago and I eventually caved and make macOS launch a login script within the terminal https://github.com/Microsoft/vscode/issues/20145#issuecomment-278172268, but that causes a whole bunch of other issues.
I don't normally do what you're trying to do, but I'm sure there's a proper way you're meant to do it. I also recommend reading up on the difference between .bashrc/.bash_profile and login shells/non-login shells.

Related: https://github.com/Microsoft/vscode/pull/55194

So I think the end result here is that two things are conspiring against my "virtualenv + code ." workflow, making it nowhere near as effective on Mac as on Win or Linux:

path_helper does awful things to a pre-existing $PATH variable, re-ordering PATH for any shell opened as login ("Why ever would you want your PATH to have the same order?! I'm a LOGIN shell! The original shell! The SUPREME shell!"). I can't simply disable path_helper, since other things expect it to be functional. And therefore it will be a pain in my neck forever, so long as Code needs to start in a login shell.

I can work around that at least a couple different ways - set a breadcrumb like OLD_PATH when I activate a virtualenv, and restore that to $PATH in bashrc. Or detect custom path items and blindly restore them to the front of the line. Or, application-specific, detect VIRTUAL_ENV and reactivate it blindly.
But then despite any of those, I still run into

Code windows don't support independent, differing environments. Subsequently-opened windows always inherit environment variables from the existing windows, and that will forever break, for example, working on two different virtualenvs in two different windows.

So I guess the Code workflow is just inherently worse on Mac than Windows, until there's first-class support for virtualenvs in the vscode Python plugin :(

I'd recommend joining the discussion over at https://github.com/Microsoft/vscode/issues/15452, also for Python specifically I'm not sure if the python extension has a nice workaround for handling virtualenv or not https://github.com/Microsoft/vscode-python

also for Python specifically I'm not sure if the python extension has a nice workaround for handling virtualenv or not

My understanding is, at this juncture, it's a discussion-in-progress.

I'd recommend joining the discussion over at #15452, also for Python specifically I'm not sure if the python extension has a nice workaround for handling virtualenv or not https://github.com/Microsoft/vscode-python

They have a separate "Create Terminal" command that creates a terminal with the active Python environment. But due to API limitations, this is a separate command and as such by default terminals are created using VS Code's environment via the default command.

@segevfiner sure, you could replace the keybinding though if it's all you want to use:
{ "key": "ctrl+`", "command": "thepythoncommand" }

You can even make this the default only when you have a Python file opened:
{ "key": "ctrl+`", "command": "thepythoncommand", "when": "editorLangId == 'markdown'" }


⚠ Fortnite has degraded performance

In 1278a7a, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1570 ms

Resolved: Fortnite performance has improved in 17ca7fd.


Added travis ci support for node 6

Hi @amilajack, thanks for opening this PR! Your support is very appreciated. 😺

No problem!


feat(routing): build the gRPC interface for communicating with the external service to perform elimination routing

Type of Change

[ ] Bugfix
[ ] New feature
[ ] Enhancement
[ ] Refactoring
[ ] Dependency updates
[ ] Documentation
[ ] CI/CD

Description
build the gRPC interface for communicating with the external service to perform elimination routing
Additional Changes

[ ] This PR modifies the API contract
[ ] This PR modifies the database schema
[ ] This PR modifies application configuration/environment variables

Motivation and Context

How did you test it?

Communication is being established with dynamo

Checklist

[ ] I formatted the code cargo +nightly fmt --all
[ ] I addressed lints thrown by cargo clippy
[ ] I reviewed the submitted code
[ ] I added unit tests for my changes where possible

please add test cases to all 3 endpoints and check the validations too

please add test cases to all 3 endpoints and check the validations too

+1 on that.
It would be better to have test cases. attached showing an example execution of these functionality.

please add test cases to all 3 endpoints and check the validations too

+1 on that.
It would be better to have test cases. attached showing an example execution of these functionality.

Okay sure


Add Release Preview column to RS4 / RS5

There may only be one build but if there was ever a release preview build in RS4 / RS5 could you add it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fa80910a-e655-34df-d26a-b094dd8702bb
Version Independent ID: 83b90d10-f455-fee5-cf94-2698f1ac33d6
Content: Flight Hub
Content Source: wip/flight-hub/index.md
Service: wip
GitHub Login: @lizap
Microsoft Alias: elizapo

Release Preview ring for RS5 has received 18 builds which are not documented by Microsoft anywhere:
https://changewindows.org/build/redstone5/17763/pc

@officedocsbot assign @mypil

@hoyty76  - Thank you for submitting feedback.
I will get this issue over to the Windows Insider writing team for investigation.
Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.

Thanks for the feedback. We'll take it under advisement and add a release preview column when and if it makes sense. Thanks!


Typed text is always read as en-US (mm-dd)

Picked date is sent correctly as per format. Typed date is always sent as mm-dd-yyyy.
So if you tab into the field, day and month will switch. With below setup, if you pass in prop 31.12.2020, it would also fail (no 31st month)
<script lang='coffee'>
  import vDatePick from 'vuejs-datepicker';
  import { hr } from 'vuejs-datepicker/dist/locale'
  import { format, parse } from 'date-fns'
  formatStr = 'dd.MM.yyyy.'
  r = required: true

  export default
    components: { vDatePick }
    props: file: r, type: r
    computed: value:
      get: -> if get = @file[@type] then @parse get
      set: (date) -> @file[@type] = @format date
    methods:
      format: (date) => format date, formatStr, new Date()
      parse:  (date) => parse  date, formatStr, new Date()
    data: -> hr: hr

</script>

<template lang='pug'>
  div
    p {{file[type]}}
    vDatePick(
      v-model='value'
      :format='format'
      :typeable='true'
      :language='hr' :monday-first='true')
</template>

Having the same issue here.


Problem with the OverlayModule after an update from Angular 4 to Angular 5

Bug, feature request, or proposal:
Before in Angular 4
In my constructor in app.component.ts i imported the OverlayContainer and then i was able to set the displayed theme.
constructor(public themingService: ThemingService, public overlayContainer: OverlayContainer) {     this.overlayContainer.themeClass = this.themingService.ActualTheme.className; }
Now after the update, I changed OverlayContainer to OverlayModule
constructor(overlayModule: OverlayModule, public themingService: ThemingService) {     overlayModule.getContainerElement().classList.add(themingService.ActualTheme.className);   }
What is the current behavior?
I can't start the app, because "Property 'getContainerElement' does not exist on type 'OverlayModule'"
What are the steps to reproduce?
Updating from Angular 4 to Angular 5
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
In Angular 5 my dependencies look like following
"dependencies": {
"@angular/animations": "^5.1.2",
"@angular/cdk": "^5.0.2",
"@angular/common": "^5.1.2",
"@angular/compiler": "^5.1.2",
"@angular/core": "^5.1.2",
"@angular/flex-layout": "^2.0.0-beta.12",
"@angular/forms": "^5.1.2",
"@angular/http": "^5.1.2",
"@angular/material": "^5.0.2",
"@angular/platform-browser": "^5.1.2",
"@angular/platform-browser-dynamic": "^5.1.2",
"@angular/platform-server": "^5.1.2",
"@angular/router": "^5.1.2",
"core-js": "^2.4.1",
"hammerjs": "^2.0.8",
"rxjs": "^5.5.2",
"zone.js": "^0.8.14"
},
"devDependencies": {
"@angular/cli": "1.6.2",
"@angular/compiler-cli": "^5.0.0",
"@angular/language-service": "^5.0.0",
"@ types/jasmine": "~2.5.53",
"@ types/jasminewd2": "~2.0.2",
"@ types/node": "~6.0.60",
"codelyzer": "^4.0.1",
"jasmine-core": "~2.6.2",
"jasmine-spec-reporter": "~4.1.0",
"karma": "~1.7.0",
"karma-chrome-launcher": "~2.1.1",
"karma-cli": "~1.0.1",
"karma-coverage-istanbul-reporter": "^1.2.1",
"karma-jasmine": "~1.1.0",
"karma-jasmine-html-reporter": "^0.2.2",
"protractor": "~5.1.2",
"ts-node": "~3.2.0",
"tslint": "~5.7.0",
"typescript": "~2.4.2"
}

Stack Traces:
ERROR in src/app/app.component.ts(15,22): error TS2339: Property 'getContainerElement' does not exist on type 'OverlayModule'.

OverlayModule is an NgModule and should be used in import in your applications NgModule.  OverlayContainer is an @Injectable service that is provided by OverlayModule.
You should be retrieving the container element from the OverlayContainer


How to load an sqlite file into absurd sql

I am working on a project, and I need to load sqlite database from server.
Although SQL.js provides a way to do this, I can't find any way to do this in absurd sql. It would be great if someone can help me with this.
In Sql.js
https://github.com/sql-js/sql.js/wiki/Load-a-database-from-the-server

I was searching for solutions for this myself and found one that works pretty well in my case. I didn't find any way to upload an existing database file but sqlite has the option to dump its schema and data into text files.
With the sqlite command line interface open - you can use the .output command to make it output to a file on disk instead of the terminal. Then you can use .schema to get the commands to create the tables or .dump to get all the info to recreate the database into your text file.
-> ~ sqlite3 signtest.sqlite3
SQLite version 3.39.4 2022-09-07 20:51:41
Enter ".help" for usage hints.
sqlite> .output ~/dbschema.txt
sqlite> .schema

Yields (this is just for the first table):
CREATE TABLE IF NOT EXISTS "sign_collection" (
	"sign_id"	INTEGER,
	"collection_id"	INTEGER,
	"date_added"	NUMERIC,
	PRIMARY KEY("sign_id","collection_id"),
	FOREIGN KEY("sign_id") REFERENCES "sign"("id"),
	FOREIGN KEY("collection_id") REFERENCES "collection"("id")
);

It includes the semicolons so if you split the file by semicolons you get each command separately. This javascript code loads a text file with fetch and iterates over it splitting the file by semicolons.
async function* splitTextFileBySemicolon(fileURL) {
  const utf8Decoder = new TextDecoder("utf-8");
  let response = await fetch(fileURL);
  let reader = response.body.getReader();
  let {value: chunk, done: readerDone} = await reader.read();
  chunk = chunk ? utf8Decoder.decode(chunk, {stream: true}) : "";

  let re = /;/gm;
  let startIndex = 0;

  for (;;) {
    let result = re.exec(chunk);
    if (!result) {
      if (readerDone) {
        break;
      }
      let remainder = chunk.substr(startIndex);
      ({value: chunk, done: readerDone} = await reader.read());
      chunk = remainder + (chunk ? utf8Decoder.decode(chunk, {stream: true}) : "");
      startIndex = re.lastIndex = 0;
      continue;
    }
    yield chunk.substring(startIndex, result.index);
    startIndex = re.lastIndex;
  }
  if (startIndex < chunk.length) {
    // last line didn't end in a newline char
    yield chunk.substr(startIndex);
  }
}

The last step is just to make the database like normally but then iterate over the .dump commands and execute them in the absurd-sql.js database.
for await (let line of splitTextFileBySemicolon('dbschema.txt')) {
    try{
        db.exec(line);
    } catch (error) {
        console.error(error)
    }
}

It might take a bit of messing with the dump files. The first time I tried it I just dumped the whole database and it worked fine but sometimes it's a bit more finicky and I have to arrange the commands a bit better:

create all the tables first.
then insert data
then create indexes

@odinndagur I really appreciate and thank you that you took time to write this solution.
I am doing the same thing for like past few months but inserting all data again is slow compared to loading sqlite file completely when you have a big file. Second thing, although it's quite unlike but there is a small chance of error while this while process is executed specially if a file is large, like 5 MB or 50 MB.
To handle the problem of foreign keys, and constraint checks I just disable the check itself which also makes insertions fast, yet it's not best solution in my opinion. It would be ideal if we can just load the sqlite file, just like sql.js which probably will take same or less time as inserting one by one from dump file. also, that there is a 100 line code sitting just for this thing and I am not sure if it will occasionally break during a large dump (I have taken enough majors for error handling and recovery).


Why does solang not recognize msg and block?

As shown in the picture。
The corresponding code is creator = msg.sender;``return block.number;``if (msg.sender == creator)

You are right, this is not implemented yet. However, I'm currently implementing these features the current milestone which is due the 25th of July: https://github.com/hyperledger-labs/solang/milestone/3

Work is underway here: https://github.com/hyperledger-labs/solang/pull/220

This should be fixed now.


Photo browser annotation filters show wrong value when no annotated obs

Describe the bug
The taxon photo browser annotation filters incorrectly display "Any" when set to value not represented in observations of the taxon in question.
To Reproduce
Steps to reproduce the behavior:

Go to the photo browser for a taxon with observations that have the Life Stage: Larva annotation, e.g. https://www.inaturalist.org/taxa/48663-Danaus/browse_photos, and set filters to the defaults, except set Life Stage to Larva. Here are the full filters:

Grouping: Non
Life Stage: Larva
Sex: Any
Alive or Dead: Any
Order By: Faves
Photo Licensning: Any
Quality Grade: Research

Go to the photo browser for a taxon with observations that do not have the Life Stage: Larva annotation, e.g. https://www.inaturalist.org/taxa/1080209-Cimbex-rubidus/browse_photos
The Life Stage filter is set to Any, even though it should be set to Larva, and no photos are shown

Expected behavior
When the Life Stage filter is set to Larva, the filter should still display as "Any" even when the taxon has no observations with that annotation.
Additional context
Reported by several people in the Forum and by Tony in Slack. Related to work on https://github.com/inaturalist/inaturalist/issues/2816

There are also issues when going between taxa that don't share the same annotation, let alone value. eg

Go to https://www.inaturalist.org/taxa/47125-Angiospermae/browse_photos and choose Flowering for Plant Phenology

Go to the photo browser for Kingdom Fungi and you'll see this:

There's no way to change the Plant Phenology filter here so you won't see any photos. I would expect that an irrelevant annotation would not affect the photo browser, or for there to be a "reset filters" option.

Will it take some time to roll this out on the site now, or is it not really fixed? I tried Bibio photo browser, set Larva filter, then went to Fungi photo browser and it says "No observations yet" still.

Will it take some time to roll this out on the site now, or is it not really fixed? I tried Bibio photo browser, set Larva filter, then went to Fungi photo browser and it says "No observations yet" still.


Add "and later" to clarify "Getting started" example

Thanks for the feedback and PR.
v0.2.0 now works with Node 12 as well, so I'm using >= 16 instead.


[Dev Site] Update all instrumentation to use tessen

This ticket is blocked by https://github.com/newrelic/gatsby-theme-newrelic/issues/559
Acceptance Criteria
note 1st, verify if the site has any browser agent page actions that were not covered by the theme changes.

[x] Bring in new version of the theme
[x] remove all browser agent page actions from the site
[x] Check all instances of the instrumentation prop and make sure it follows the correct tessen formatting (category is Title Case, event name is camelCase)
[x] Check for any uses of the useInstrumentedHandler hook and make sure it’s formatted correctly for tessen
[x] Check for any uses of the browser agent directly (window.newrelic.pageAction)
[x] Update any documentation/wiki references to the browser events
[ ] Update any dashboards that may be using the old browser events (events should be listed in the wiki but may not be up to date. You can also run this query to get a list of currently firing browser actions ((some may be coming from theme components))

Here are some important dashboards to look out for:

[DevEn] External I/O & Quickstarts Ecosystem
New Relic I/O Program Dashboard
[DevEn] Docs Ecosystem

docs ecosystem dashboard updates live in another dashboard -- docs ecosystem - tessen only. atm, many of the charts are empty due to docs website not being updated yet (i assume). i learned on friday that we could make all the changes we wanted w/o (potentially) impacting our charts because we still have page events going to our account when we use tessen. however, the charts in the ecosystem dashboard use the dev ex account rather than dev enablement, so we can't have data sources in two accounts.
tl;dr -- there are two dashboards atm. the old one which is mostly unmodified, and the new one which looks at data in the dev enablement and tessen accounts.


Any chance for ORM/pandas-like interface to augment the sql-like interface?

eg  supporting filter/groupby methods on the returned data object
should just be a matter of converting to AST under the hood?

Hmm, good question.  The way the SQL layer works is to:

Intercept a query request that is formatted as a string
Parse to AST and validate the SQL to ensure it's a query that can be safely done in a private way
Rewrite the query in the AST to ensure privacy.  For example, add reservoir sampling or bounds clipping
Convert the rewritten query to a string
Execute the rewritten query string against the database connection

Steps 2-4 need to be handled by SmartNoise to make sure we don't violate privacy.  But it should be straightforward to feed step 1 with the output from an ORM-style query.  As long as the ORM layer doesn't generate queries which are inconsistent with privacy, it will work.  I believe SQLAlchemy allows you to grab a SQL query string from the ORM query, so it should "just work".  I will try to whip up a sample later next week.  I don't know if it's possible to do that from Pandas, though.  If there is a way from Pandas to grab the SQL generated from the filter/groupby methods, it would presumably work.

Something like https://github.com/AmirPupko/pandas-to-sql looks helpful here!


added preserveWhitespace option

Added an option to preserve whitespace. Helps with Markdown....

Any opinions on this PR? If it's not possible to merge it, would an alternative approach be considered?

LGTM.


scala 3.0.0-RC1

Nice! I missed the release of Scala 3 RC1.

@hseeberger I just asked @SethTisue if the files will be available on the usual download location

see also https://users.scala-lang.org/t/downloading-scala-3/7136/3 so might try to get it from there

Looks like we will also need to wait for sbt 1.5.x
https://github.com/sbt/sbt/releases/tag/v1.5.0-M2

@hseeberger I just asked @SethTisue if the files will be available on the usual download location

I doubt it, I think you'll need to keep the "if Scala 2 this, if Scala 3 that" logic

Thanks for the update @SethTisue


Investigate warning about infinite loop in component render function

Description
In a number of tests, we can see the following type of warnings when it comes to VDataTables.
Investigate and verify if it is safe to ignore these warnings, or if there is some actual issue that might create performance issues.
  [Vue warn]: You may have an infinite update loop in a component render function.

      found in

      ---> <VDataTable>
             <VForm>
               <FindRoutes>
                 <VStepperContent>
                   <VStepper>
                     <TransferSteps>
                       <Root>

Acceptance criteria

Tasks

[ ]

While this warning is certainly annoying I am not sure we can do much about it. I spend quite some time investigating it but there is no solution I can see.
This warning appears to the only two components in which we use VDataTable. I tried a few different approaches trying to figure out if there is any part of our code causing, but without any success.
The only way to remove the warning was by removing the VDataTable completely. Also adding the data table component to any other of our own components will start showing the same warning. This means that the problem is possibly in the Vuetify component.
Unfortunately, there is no open issue about the warning, but there are a few issues about unnecessary re-rendering and performance that might be the cause of this warning.
I don't really see any realtime drawback at the moment other than an annoying warning during tests so I will close the issue since there is not much we can do about it.


Nightmare: Anemometer

Placement:

Can't place anemometer at top, because it would get in way of photonics.
Can't really place it at bottom, because it would be affected by mounting rod
Can't really place it in middle as it would get in the way of wires between modules, and weaken design, not to mention issues relating to Stevenson Screen.

Tech:

Ultrasonic is do-able, but not sure how to process the results
Microwave might be interesting to investigate - it's not sound, so unaffected by wind?
How does heat affect things ultrasonic/microwave?
Could use heated wire approach? - eats too much power
Normal spinning cup things will freeze in winter

Could potentially use [venturi style anemometers] (with barometers measuring differential between two points). But out in a field, the tubes will get clogged quickly?

5am brianfart...

Wind = pressure turbulence = sound turbulence

Why bother adding more sound to the mix? It's easier to filter to our added (ultrasound) because we know what it was like when we created it and we can determine how it's changed when we receive it. But... Surely we could just listen to the wind? Usually it's infrasound - we can't hear it, but we can feel it... as the wind picks up, we feel it more and eventually start to hear it "howling/whistling wind", if it got really turbulent it could even go in to ultrasonic? Anyway, if we know the sound frequency characteristics of an object for a given (eg. current) temperature, we know how it will affect sound (pressure) ...we can listen to that object - eg. with a gyroscope or wideband microphone - and work out what the wind is doing? Do that in 3 or more locations, and we can triangulate the wind direction, speed, etc? We're not interested in pressure as such (barometer), we're interested in changes in pressure = sound. Could the Stevenson Screen be turned in to an anemometer?


HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks.

What changes were proposed in this pull request?
During Ozone EC block recovery, we encountered a situation where some attempted recoveries of EC Stripe Blocks fail to reconstruct properly, resulting in the following error.
java.lang.IllegalArgumentException: The chunk list has 2 entries, but the checksum chunks has 3 entries.
They should be equal in size.

After carefully reviewing the EC code, We did not find any issues with the read and write operations. Upon discussion, we concluded that the closure of containers (e.g., when a container reaches its write capacity) may affect the writing of EC stripe data blocks. However, this is considered a normal occurrence, implying that occasional issues with some EC stripe data blocks are expected.

For EC type EC-3-2-1024K:
One of our EC stripes consists of 3 data blocks. Blocks 1 and 2 are correct, but Block 3 has an issue where its BlockGroupLength is shorter than that of Blocks 1 and 2.
If Block 2 is lost at this point, we would need to recover Block 2 using 2 data blocks (DataBlock1, DataBlock3) and 2 parity blocks (Parity1, Parity2).
We can read ECReconstructionCoordinator#calcBlockLocationInfoMap, which calls ECReconstructionCoordinator#calcEffectiveBlockGroupLen method.
https://github.com/apache/ozone/blob/9533066d2ca28706938c91cd9cf50a56ce16739d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ec/reconstruction/ECReconstructionCoordinator.java#L564-L581
This section of code selects the minimum BlockGroupLength, thus choosing the BlockGroupLength of DataBlock3, leading to the recovery of an incorrect data block.
ECBlockOutputStream#executePutBlock validates the generated data, so we will receive a notification.
java.lang.IllegalArgumentException: The chunk list has 2 entries, but the checksum chunks has 3 entries.
They should be equal in size.

I attempted to propose an improvement method:
if there are sufficient source data blocks available, we will exclude Block 3.

Solution description

Step1. Improve BlockGroupLength selection.
We will attempt to find the most suitable BlockGroupLength possible. We cannot avoid encountering some special situations during data writes, which may lead to inaccuracies in the length of some BlockGroupLength values.
However, in cases like the following, we can reasonably infer which BlockGroupLength is accurate:
when only one DN has a different BlockGroupLength compared to the others, we can assume that the BlockGroupLength of the other DNs is more reasonable.
Step2. Exclude DN whose BlockGroupLength is different from other DNs.

Remove the problematic DataNode from the data recovery pipeline.
Remove the problematic BlockData from the array of BlockGroups.

What is the link to the Apache JIRA
JIRA: HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks.
How was this patch tested?
Add Junit Test.

@slfan1989 If you intend to mark the PR as draft, please do so while creating the PR, not as a separate step afterwards.  This is to avoid unnecessary CI runs.
Similarly, please fill PR template while creating the PR, not afterwards.  (Tweaking the description is fine at any time, just avoid using the unedited template.)

@slfan1989 If you intend to mark the PR as draft, please do so while creating the PR, not as a separate step afterwards. This is to avoid unnecessary CI runs.
Similarly, please fill PR template while creating the PR, not afterwards. (Tweaking the description is fine at any time, just avoid using the unedited template.)

Thank you for the reminder. I will pay attention to these details when submitting future prs. HDDS-11243 (#7008) is already deployed in our production environment, and I will update its status to "ready for review" soon. As for HDDS-10985(#7009), after refining the relevant descriptions, I will also change its status to "ready for review".

@sodonnel @adoroszlai This is an improvement regarding EC block recovery. The primary focus of the PR is to eliminate inaccurate DNs caused by BlockGroupLength. If you have time, please assist in reviewing this PR. Thank you very much!

This section of code selects the minimum BlockGroupLength, thus choosing the BlockGroupLength of DataBlock3, leading to the recovery of an incorrect data block.

I am not sure this statement is correct.
When a stripe is written, the client must wait for all DNs to ack the write, indicating it was successfully saved. If any of the DNs do not return successfully, the stripe is abandoned by the client, and a new block is requested and the stripe is written to the new block again. This may duplicate some data.
To know for sure if this is happening, you need to look at the block length stored in OM for this block and see if it aligns with 2 strips (as stated by DN 3) or greater than 3 stripes, indicating that the "failed write handling" on the client did not do the correct thing. Unfortunately given the block ID, its not easy to find the key is it associated with. I believe it can be done via recon.

This section of code selects the minimum BlockGroupLength, thus choosing the BlockGroupLength of DataBlock3, leading to the recovery of an incorrect data block.

I am not sure this statement is correct.
When a stripe is written, the client must wait for all DNs to ack the write, indicating it was successfully saved. If any of the DNs do not return successfully, the stripe is abandoned by the client, and a new block is requested and the stripe is written to the new block again. This may duplicate some data.
To know for sure if this is happening, you need to look at the block length stored in OM for this block and see if it aligns with 2 strips (as stated by DN 3) or greater than 3 stripes, indicating that the "failed write handling" on the client did not do the correct thing. Unfortunately given the block ID, its not easy to find the key is it associated with. I believe it can be done via recon.

Thank you very much for your reply! Currently, there is a lot of data online in this situation. It seems that our client is not performing as expected. I added audit logs on a specific DN and found a large number of failed reconstructions.

cat dn-audit.log|grep "RECOVER_EC_BLOCK"|grep "FAIL"
353

example:

2024-07-25 07:25:15,830 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 951772 locID:<PHONE_NUMBER>32021583 bcsId: 0}, length=25165824, offset=0, token=null, pipeline=Pipeline[ Id: cc205dc9-49a1-4c07-92b9-0c27504268b6, Nodes: 62fa9baf-6002-4da9-bb06-eaf3e9549774(bigdata-ozone1431.online/<IP_ADDRESS>)d6b32839-fd22-4ee9-b1ef-0a2f7ae5e8d7(bigdata-ozone1302.online/<IP_ADDRESS>)8d7e54be-c94c-4bfd-ba11-4b618a1ca332(bigdata-ozone1718.online/<IP_ADDRESS>)e6599058-c6cf-4d3c-b669-4155bedc6631(bigdata-ozone1455.online/<IP_ADDRESS>)94bfe561-11b8-40b4-89e7-f3279f6b73e6(bigdata-ozone1474.online/<IP_ADDRESS>)f4bd3569-fbb4-40fa-a792-863a34df2cb4(bigdata-ozone1245.online/<IP_ADDRESS>)c1e06adb-46d9-4d08-ab65-bd9ef83607f8(bigdata-ozone1330.online/<IP_ADDRESS>)0e41e3ce-bc8d-4185-b956-d1d445b25cb9(bigdata-ozone1382.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T07:23:12.013617185+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 4 entries, but the checksum chunks has 5 entries. They should be equal in size.
2024-07-25 08:44:13,601 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 837314 locID:<PHONE_NUMBER>62943175 bcsId: 0}, length=100663296, offset=0, token=null, pipeline=Pipeline[ Id: 8aeaf9ec-7537-4ad0-82d3-be7b5e7702aa, Nodes: 6d856eab-b1ce-428e-b91c-b8eb950807ae(bigdata-ozone1354.online/<IP_ADDRESS>)1b29d271-631b-4703-9d63-a0b65bf30480(bigdata-ozone1258.online/<IP_ADDRESS>)447297bf-f236-42da-babd-6affdff5e845(bigdata-ozone1404.online/<IP_ADDRESS>)105617a7-5fa5-41fd-bd51-61b5200c3e1d(bigdata-ozone1695.online/<IP_ADDRESS>)dedf7b87-667b-4c84-b98b-94fb8bb8a2bc(bigdata-ozone1295.online/<IP_ADDRESS>)f76dda7e-d639-465c-a1ab-e9ef6ec4421c(bigdata-ozone1418.online/<IP_ADDRESS>)86e30199-e1b6-41ee-936a-c9c7638af580(bigdata-ozone1476.online/<IP_ADDRESS>)df941469-8358-402a-8600-0d3f508f9cda(bigdata-ozone1366.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T08:44:04.377586980+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 16 entries, but the checksum chunks has 17 entries. They should be equal in size.
2024-07-25 08:45:07,392 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 951070 locID:<PHONE_NUMBER>30944419 bcsId: 0}, length=56623104, offset=0, token=null, pipeline=Pipeline[ Id: 7d0a854b-dbeb-4958-91fb-16c2dcf2fdfa, Nodes: a2ddcc2d-8aa9-4030-9121-af6f9426ac04(bigdata-ozone1249.online/<IP_ADDRESS>)cac84fc4-835b-4c49-9566-10b3b252b44f(bigdata-ozone1408.online/<IP_ADDRESS>)e6599058-c6cf-4d3c-b669-4155bedc6631(bigdata-ozone1455.online/<IP_ADDRESS>)cffa6746-ae46-4e5f-8f54-c37bebdb36d6(bigdata-ozone1712.online/<IP_ADDRESS>)93435586-39df-4e4b-88e6-f25e3d926bba(bigdata-ozone1329.online/<IP_ADDRESS>)795e2ef6-4f22-44de-aad0-78bae5a153b3(bigdata-ozone1513.online/<IP_ADDRESS>)4f5b5892-63f4-4cf8-b617-2fb26e9e0ef5(bigdata-ozone1480.online/<IP_ADDRESS>)7c8f10a6-8027-488c-b187-8e4b3afadce3(bigdata-ozone1316.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T08:44:04.377586831+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 9 entries, but the checksum chunks has 10 entries. They should be equal in size.
2024-07-25 08:55:26,974 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 932402 locID:<PHONE_NUMBER>96121646 bcsId: 0}, length=18874368, offset=0, token=null, pipeline=Pipeline[ Id: 9a6bb8f1-8322-4846-990b-81d5b5de78e0, Nodes: 4dc2a747-2be3-4a58-ab7c-3fbe504a1189(bigdata-ozone1502.online/<IP_ADDRESS>)fcd1cc61-b679-4853-87f6-be6cbfd94b32(bigdata-ozone1352.online/<IP_ADDRESS>)6c2057a9-061a-44c2-bd5e-b7e430f5257d(bigdata-ozone1334.online/<IP_ADDRESS>)fe24377f-91a9-403a-b654-1f7c8cd184e4(bigdata-ozone1663.online/<IP_ADDRESS>)33c6fb6a-6ed6-4246-b8b9-814309b59750(bigdata-ozone1255.online/<IP_ADDRESS>)5b5316b6-0be1-4ff6-8c2b-52c3cb53a702(bigdata-ozone1414.online/<IP_ADDRESS>)6ea28f7b-462f-4b35-994a-33d1d7d288de(bigdata-ozone1315.online/<IP_ADDRESS>)91662b72-aa0d-4087-a1e9-f71b55c4a64f(bigdata-ozone1468.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T08:55:26.562523718+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 3 entries, but the checksum chunks has 4 entries. They should be equal in size.
2024-07-25 09:15:06,692 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 925553 locID:<PHONE_NUMBER>85174863 bcsId: 0}, length=56623104, offset=0, token=null, pipeline=Pipeline[ Id: 819c7cb8-a4f2-43a3-a2c1-c692c15f1c7f, Nodes: 77eaf094-d67b-40cc-a1c0-5eff52292a22(bigdata-ozone1351.online/<IP_ADDRESS>)14e3d6be-dd8f-476d-90b9-043d37e8d735(bigdata-ozone1472.online/<IP_ADDRESS>)cf38675f-987b-47c2-baa3-e6afdd3884fe(bigdata-ozone1451.online/<IP_ADDRESS>)003c3831-ffd7-4d6a-8b8c-f71a4d01bf94(bigdata-ozone1291.online/<IP_ADDRESS>)d8f3179c-7629-48f2-9030-45a89de389ab(bigdata-ozone1425.online/<IP_ADDRESS>)c31aca19-69cc-4cb6-9776-c995ee3e3132(bigdata-ozone1509.online/<IP_ADDRESS>)cde3f346-7a0d-42e1-9c53-2084e71febfd(bigdata-ozone1381.online/<IP_ADDRESS>)c27c68ff-966c-4d55-a92c-52ce9214ca40(bigdata-ozone1343.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T09:11:21.535920299+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 9 entries, but the checksum chunks has 10 entries. They should be equal in size.
2024-07-25 09:27:51,679 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 955780 locID:<PHONE_NUMBER>37274967 bcsId: 0}, length=12582912, offset=0, token=null, pipeline=Pipeline[ Id: 8431e3cf-7812-49bc-89f9-d43e1e3cdcf0, Nodes: b98a0679-cab6-4f1e-98ee-a04d27f0f1e4(bigdata-ozone1506.online/<IP_ADDRESS>)41ba9acc-b97c-4e7a-b0df-d1d05daebefb(bigdata-ozone1807.online/<IP_ADDRESS>)19c5d2a7-1fee-4e6c-8395-ee6d09cfd86a(bigdata-ozone1332.online/<IP_ADDRESS>)7509ee25-be69-49bb-8d03-38457712f2e9(bigdata-ozone1296.online/<IP_ADDRESS>)94bfe561-11b8-40b4-89e7-f3279f6b73e6(bigdata-ozone1474.online/<IP_ADDRESS>)60fe53ee-de63-41c3-b5f0-613a6b55444c(bigdata-ozone1423.online/<IP_ADDRESS>)fcd1cc61-b679-4853-87f6-be6cbfd94b32(bigdata-ozone1352.online/<IP_ADDRESS>)3ccd78c7-047f-4a45-ac88-891936f29dfe(bigdata-ozone1363.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T09:27:36.505353695+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 2 entries, but the checksum chunks has 3 entries. They should be equal in size.

To know for sure if this is happening, you need to look at the block length stored in OM for this block and see if it aligns with 2 strips (as stated by DN 3) or greater than 3 stripes, indicating that the "failed write handling" on the client did not do the correct thing. Unfortunately given the block ID, its not easy to find the key is it associated with. I believe it can be done via recon.

My plan for verification is as follows: when EC reconstruction fails, it prints conID, locID. I will offline analyze a copy of OM's Rockdb data, locate the file corresponding to these conID, locID pairs in the relevant data tables, and then follow the steps you mentioned to verify if this situation has occurred.

@sodonnel
I think I may not have expressed myself very clearly. I’d like to describe the functionality of this PR again and provide an example to illustrate the entire recovery process.

Background

We discovered some errors related to EC reconstruction online, with detailed descriptions in HDDS-10985.
java.lang.IllegalArgumentException: The chunk list has 2 entries, but the checksum chunks has 3 entries.
They should be equal in size.

After encountering these errors, we took the following steps:

We carefully reviewed the read and write code for EC and raised some questions about its implementation. After you provided answers, I understood your approach and concluded that there were no obvious issues with the EC write code.

We added extra auditLog entries for certain DataNodes (DNs) because finding EC reconstruction errors in DN logs is very challenging. I wrote the reconstruction error information into the audit.log. Wd can refer to the relevant pr for details(#6936).

After deploying this PR, I discovered many errors with the reconstruction blocks.
2024-07-25 07:25:15,830 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 951772 locID:<PHONE_NUMBER>32021583 bcsId: 0}, length=25165824, offset=0, token=null, pipeline=Pipeline[ Id: cc205dc9-49a1-4c07-92b9-0c27504268b6, Nodes: 62fa9baf-6002-4da9-bb06-eaf3e9549774(bigdata-ozone1431.online/<IP_ADDRESS>)d6b32839-fd22-4ee9-b1ef-0a2f7ae5e8d7(bigdata-ozone1302.online/<IP_ADDRESS>)8d7e54be-c94c-4bfd-ba11-4b618a1ca332(bigdata-ozone1718.online/<IP_ADDRESS>)e6599058-c6cf-4d3c-b669-4155bedc6631(bigdata-ozone1455.online/<IP_ADDRESS>)94bfe561-11b8-40b4-89e7-f3279f6b73e6(bigdata-ozone1474.online/<IP_ADDRESS>)f4bd3569-fbb4-40fa-a792-863a34df2cb4(bigdata-ozone1245.online/<IP_ADDRESS>)c1e06adb-46d9-4d08-ab65-bd9ef83607f8(bigdata-ozone1330.online/<IP_ADDRESS>)0e41e3ce-bc8d-4185-b956-d1d445b25cb9(bigdata-ozone1382.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T07:23:12.013617185+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 4 entries, but the checksum chunks has 5 entries. They should be equal in size.
2024-07-25 08:44:13,601 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 837314 locID:<PHONE_NUMBER>62943175 bcsId: 0}, length=100663296, offset=0, token=null, pipeline=Pipeline[ Id: 8aeaf9ec-7537-4ad0-82d3-be7b5e7702aa, Nodes: 6d856eab-b1ce-428e-b91c-b8eb950807ae(bigdata-ozone1354.online/<IP_ADDRESS>)1b29d271-631b-4703-9d63-a0b65bf30480(bigdata-ozone1258.online/<IP_ADDRESS>)447297bf-f236-42da-babd-6affdff5e845(bigdata-ozone1404.online/<IP_ADDRESS>)105617a7-5fa5-41fd-bd51-61b5200c3e1d(bigdata-ozone1695.online/<IP_ADDRESS>)dedf7b87-667b-4c84-b98b-94fb8bb8a2bc(bigdata-ozone1295.online/<IP_ADDRESS>)f76dda7e-d639-465c-a1ab-e9ef6ec4421c(bigdata-ozone1418.online/<IP_ADDRESS>)86e30199-e1b6-41ee-936a-c9c7638af580(bigdata-ozone1476.online/<IP_ADDRESS>)df941469-8358-402a-8600-0d3f508f9cda(bigdata-ozone1366.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T08:44:04.377586980+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 16 entries, but the checksum chunks has 17 entries. They should be equal in size.
2024-07-25 08:45:07,392 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 951070 locID:<PHONE_NUMBER>30944419 bcsId: 0}, length=56623104, offset=0, token=null, pipeline=Pipeline[ Id: 7d0a854b-dbeb-4958-91fb-16c2dcf2fdfa, Nodes: a2ddcc2d-8aa9-4030-9121-af6f9426ac04(bigdata-ozone1249.online/<IP_ADDRESS>)cac84fc4-835b-4c49-9566-10b3b252b44f(bigdata-ozone1408.online/<IP_ADDRESS>)e6599058-c6cf-4d3c-b669-4155bedc6631(bigdata-ozone1455.online/<IP_ADDRESS>)cffa6746-ae46-4e5f-8f54-c37bebdb36d6(bigdata-ozone1712.online/<IP_ADDRESS>)93435586-39df-4e4b-88e6-f25e3d926bba(bigdata-ozone1329.online/<IP_ADDRESS>)795e2ef6-4f22-44de-aad0-78bae5a153b3(bigdata-ozone1513.online/<IP_ADDRESS>)4f5b5892-63f4-4cf8-b617-2fb26e9e0ef5(bigdata-ozone1480.online/<IP_ADDRESS>)7c8f10a6-8027-488c-b187-8e4b3afadce3(bigdata-ozone1316.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T08:44:04.377586831+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 9 entries, but the checksum chunks has 10 entries. They should be equal in size.
2024-07-25 08:55:26,974 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 932402 locID:<PHONE_NUMBER>96121646 bcsId: 0}, length=18874368, offset=0, token=null, pipeline=Pipeline[ Id: 9a6bb8f1-8322-4846-990b-81d5b5de78e0, Nodes: 4dc2a747-2be3-4a58-ab7c-3fbe504a1189(bigdata-ozone1502.online/<IP_ADDRESS>)fcd1cc61-b679-4853-87f6-be6cbfd94b32(bigdata-ozone1352.online/<IP_ADDRESS>)6c2057a9-061a-44c2-bd5e-b7e430f5257d(bigdata-ozone1334.online/<IP_ADDRESS>)fe24377f-91a9-403a-b654-1f7c8cd184e4(bigdata-ozone1663.online/<IP_ADDRESS>)33c6fb6a-6ed6-4246-b8b9-814309b59750(bigdata-ozone1255.online/<IP_ADDRESS>)5b5316b6-0be1-4ff6-8c2b-52c3cb53a702(bigdata-ozone1414.online/<IP_ADDRESS>)6ea28f7b-462f-4b35-994a-33d1d7d288de(bigdata-ozone1315.online/<IP_ADDRESS>)91662b72-aa0d-4087-a1e9-f71b55c4a64f(bigdata-ozone1468.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T08:55:26.562523718+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 3 entries, but the checksum chunks has 4 entries. They should be equal in size.
2024-07-25 09:15:06,692 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 925553 locID:<PHONE_NUMBER>85174863 bcsId: 0}, length=56623104, offset=0, token=null, pipeline=Pipeline[ Id: 819c7cb8-a4f2-43a3-a2c1-c692c15f1c7f, Nodes: 77eaf094-d67b-40cc-a1c0-5eff52292a22(bigdata-ozone1351.online/<IP_ADDRESS>)14e3d6be-dd8f-476d-90b9-043d37e8d735(bigdata-ozone1472.online/<IP_ADDRESS>)cf38675f-987b-47c2-baa3-e6afdd3884fe(bigdata-ozone1451.online/<IP_ADDRESS>)003c3831-ffd7-4d6a-8b8c-f71a4d01bf94(bigdata-ozone1291.online/<IP_ADDRESS>)d8f3179c-7629-48f2-9030-45a89de389ab(bigdata-ozone1425.online/<IP_ADDRESS>)c31aca19-69cc-4cb6-9776-c995ee3e3132(bigdata-ozone1509.online/<IP_ADDRESS>)cde3f346-7a0d-42e1-9c53-2084e71febfd(bigdata-ozone1381.online/<IP_ADDRESS>)c27c68ff-966c-4d55-a92c-52ce9214ca40(bigdata-ozone1343.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T09:11:21.535920299+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 9 entries, but the checksum chunks has 10 entries. They should be equal in size.
2024-07-25 09:27:51,679 | ERROR | DNAudit | user=null | ip=null | op=RECOVER_EC_BLOCK {blockLocationInfo={blockID={conID: 955780 locID:<PHONE_NUMBER>37274967 bcsId: 0}, length=12582912, offset=0, token=null, pipeline=Pipeline[ Id: 8431e3cf-7812-49bc-89f9-d43e1e3cdcf0, Nodes: b98a0679-cab6-4f1e-98ee-a04d27f0f1e4(bigdata-ozone1506.online/<IP_ADDRESS>)41ba9acc-b97c-4e7a-b0df-d1d05daebefb(bigdata-ozone1807.online/<IP_ADDRESS>)19c5d2a7-1fee-4e6c-8395-ee6d09cfd86a(bigdata-ozone1332.online/<IP_ADDRESS>)7509ee25-be69-49bb-8d03-38457712f2e9(bigdata-ozone1296.online/<IP_ADDRESS>)94bfe561-11b8-40b4-89e7-f3279f6b73e6(bigdata-ozone1474.online/<IP_ADDRESS>)60fe53ee-de63-41c3-b5f0-613a6b55444c(bigdata-ozone1423.online/<IP_ADDRESS>)fcd1cc61-b679-4853-87f6-be6cbfd94b32(bigdata-ozone1352.online/<IP_ADDRESS>)3ccd78c7-047f-4a45-ac88-891936f29dfe(bigdata-ozone1363.online/<IP_ADDRESS>), excludedSet: , ReplicationConfig: EC{rs-6-3-1024k}, State:CLOSED, leaderId:, CreationTimestamp2024-07-25T09:27:36.505353695+08:00[Asia/Shanghai]], createVersion=0, partNumber=0}} | ret=FAILURE | java.lang.IllegalArgumentException: The chunk list has 2 entries, but the checksum chunks has 3 entries. They should be equal in size.

After reviewing the error logs, we found that most issues were caused by a single block missing the final chunk, leading to reconstructed blocks not meeting expectations. After some discussion, we proposed a recovery strategy: if among the participating blocks, there is exactly one block with a different blockgrouplength from the others (where all other blocks have the same blockgrouplength), we will treat this inconsistent block as a lost block and use the remaining blocks to complete its recovery.
Overall, this process is idempotent, because for EC-6-3-1024k, using 8 blocks to recover 1 lost block is equivalent to using 7 blocks to recover 1 lost block.

In your example, what do you do with EC-3-2, if 2 replicas are missing and one of the remaining 3 has a missing chunk? You cannot recover it.
There is certainly a problem here. If things are working as designed from the client size, and it is abandoning these blocks due to a failed ACK from one of the replicas, then the block length at OM should be the smaller size.
Then the problem is actually that we have truncated the block length to remove the duplicated chunks, but we have not truncated the checksums to match that, and that results in the failed recovery.
For some of the failing blocks, can you check the size of the block on OM to see if it matches the smaller or larger size of the block across the replicas?

In your example, what do you do with EC-3-2, if 2 replicas are missing and one of the remaining 3 has a missing chunk? You cannot recover it.

You are correct; in this scenario, even with code modifications, we can't recover the Block. However, the previous code also could not perform the recovery due to a java.lang.IllegalArgumentException.

There is certainly a problem here. If things are working as designed from the client size, and it is abandoning these blocks due to a failed ACK from one of the replicas, then the block length at OM should be the smaller size.

Thank you for your explanation! This information is crucial. I will recheck the lengths of the damaged blocks in OM to further confirm the issue. Once I complete the confirmation, I will get back to you with feedback.
Thanks again for your reply!

@sodonnel Thank you very much for your response！Your assessment is correct. there is no issue with the EC data writing in Ozone. The problem lies with the incorrect checksum selection during data recovery.
Here are some analytical approaches:

Analysis from the data writing

When writing data, we sometimes encounter situations like ContainerState is CLOSED, where the data may be written as follows:

Block-1

Data
Data
Data
Parity
Parity

EC-Stripe1
Chunk1
Chunk1
Chunk1
Chunk1
Chunk1

EC-Stripe2
Chunk2
Error
Chunk2
Chunk2
Chunk2

Block-2

Data
Data
Data
Parity
Parity

EC-Stripe2
Chunk2
Chunk2
Chunk2
Chunk2
Chunk2

When writing Block1's EC-Stripe2 on the client, an error was returned by the DN. As a result, the client retried the operation and requested a new data block to re-write EC-Stripe2. At this point, some "dirty data" have been written to Block-1.

Analysis from the EC data reconstruction

If DN3 data for Block-1 is lost at this point, we would encounter the following situation.

Data
Data
Data(Missing)
Parity
Parity

EC-Stripe1
Chunk1
Chunk1
Chunk1(Missing)
Chunk1
Chunk1

EC-Stripe2
Chunk2
Error
Chunk2(Missing)
Chunk2
Chunk2

We will use Data1, Data2, Parity1, and Parity2 to recover the Data3.
During the EC data recovery process, we will choose a smaller BlockGroupLength to restore the data.
However, during validation, we still chose the chunk size of the dirty data to verify the recovered data.
We should validate the data according to the chunk size of the data blocks corresponding to the smaller BlockGroupLength.

Following methods to analyze the data:

Parsed the OM metadata, which was written to a Hive table. This metadata includes keyPath, fileSize, ConId, and BlockId.
Found the EC recovery logs from a specific DN with errors to identify the ConId and BlockId.
Queried the corresponding files using ConId and BlockId from the table in step 1.
On OM, used ozone sh key info keyname to find the corresponding BlockGroupLength.
On SCM, used ozone admin container info <containerId> to find the corresponding container.

I’ve modified the code, and this PR truncates the checksums during EC data recovery to ensure that the restored data can pass validation.
cc: @errose28

@slfan1989 Thanks for digging into this. Your research confirms what I suspected. The client side is working as it was designed, but the checksum calculation is broken.
I had a look at the change and suggested a simplification which might be possible.
It would also be nice if we could get a test to reproduce this and validate the fix.

@slfan1989 Thanks for digging into this. Your research confirms what I suspected. The client side is working as it was designed, but the checksum calculation is broken.
I had a look at the change and suggested a simplification which might be possible.
It would also be nice if we could get a test to reproduce this and validate the fix.

Thank you very much for your guidance and suggestions! I will try to add unit tests to reproduce and test this functionality.

@sodonnel Can you help review this PR again? Thank you very much!

@sodonnel Thank you very much for reviewing the code! I appreciate your suggestions and insights during the process of solving this issue, as they played a crucial role in addressing the issue.


DO NOT MERGE

Add logging to help discover source of leaking FD_SOCK socket.

This has served it's purpose.  It did highlight the need for WFLY-17184, hence https://github.com/wildfly/wildfly/pull/16203


Update VaultChatHook.java

need update as soon as possible.

Thanks - my bad.

you're welcome

???
https://ci.lucko.me/job/LuckPerms/147/


Add step to publish to GitHub package registry

Summary
Add a step to the release-please.yml GitHub Actions workflow that publishes to the Fastly internal GItHub package registry as discussed in Slack.

Thanks @guybedford. Looks like I still have a failing test after a re-run. Do you have any insight on what's going on with that? I don't expect this PR to be related to the test failure.

@bekahs4 the CI issue is unrelated, so I've gone ahead and landed this in the mean time - will apply for the next release.

Thanks @guybedford!

@bekahs4 I see the first release just failed here - https://github.com/fastly/js-compute-runtime/actions/runs/7979097116/job/21785698469 due to credentials. Any ideas at what level we need to address that? Happy to take follow-ups as necessary here.

@guybedford I'll try to take a look at this later with @psbanka. On first glance I would be surprised if the release job failure is related to this change because this PR only made changes to the publish job.

Taking a look at that further, I believe this is also unrelated to this PR, and rather a separate issue with the release check. We can verify this properly when we tag the next release though, and if there are any issues with the GitHub package publish we can follow-up at that time.

Thanks for taking another look @guybedford. For planning purposes on our end, is there a timeline for the next tagged release?

@bekahs4 v3.8.3 just went out and it looks like the GitHub release worked correctly.


rewarded video ad not loading in release mode

logcatadmob.txt
I have tested the module in debug mode and the rewarded video ad works fine. But when i publish it in release mode and show the rewarded video Ad, the app crashes. What should i do? What could i be doing wrong?

Are you using which version of godot?

Version 2.1.3. its works perfectly in debug mode.🤔

I'm using this too and my application does not even open, I get "failed to start application", have you compiled it for 2.1.3? If so, could the APKs? So I can know if you are having any problems compiling the module or something else.

Well, I've released a game using godot 2.1.4 and everything is working fine.


Is this project dead?

it seems so...

Well, you could create a clone and keep on developing, if you do, please post it here!

Eric has stated that he's unfortunately unable to dedicate as much time to the project at this time, with it being a side project and all. However, we're currently on the verge of figuring out an approach for its succession, though!

Yep, what @aurae said. Development has definitely slowed down. Partially due to time and partially due to project maturity. You should see more soon, as we're trying to make the library more of a collaborative project instead of a one-man show.

Is  anyone going to take this up to maintain it? Are there any plans? Happy to help.


remove missing target in Makefile.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

When you already touch it, there are other phony tasks.

@tboerger I don't see any target about vendor、 docs or release.

I know, but there are other targets that should be added to phony.

@tboerger I think this is good to merge, no?

@bradrydzewski sure, it doesn't break anything. It does a minor cleanup but we can finish it in another PR.

Hi @tboerger @bradrydzewski
I merge this PR into #1750 ?

Please keep it like it is


SQL server connection issue

I have cloned and configured CSET v10.0.1 on my machine. The app was published properly.
But now I am facing the issue while running it on the browser(http://localhost:46003/).

The error is: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)

Here is the connection string added in the web.config file.

    <connectionStrings>
    <add name="CSET_DB" connectionString="data source=TEST\SQLEXPRESS;initial catalog=CSETWeb;persist security info=True;user id=test;password=test;MultipleActiveResultSets=True"/>
    <add name="ElmahConn" connectionString="data source=TEST\SQLEXPRESS;initial catalog=CSETWeb;persist security info=True;user id=test;password=test;MultipleActiveResultSets=True" providerName="System.Data.EntityClient" />
    <add name="HangfireConn" connectionString="data source=TEST\SQLEXPRESS;initial catalog=CSETWeb;persist security info=True;user id=test;password=test;MultipleActiveResultSets=True;App=HangFire" providerName="System.Data.EntityClient" />
  </connectionStrings>

TEST\SQLEXPRESS - server name where the DB is configured.

I checked the firewall and TCP port settings. Also, I have configured the SQL server to allow remote connections.
Can somebody suggest to me how can I resolve this problem?

For what ever reason your sql server is not there.   You need to check and see if you can hit the (localdb)\v11.0 connection.  Try changing "data source=TEST\SQLEXPRESS" to "data source=(localdb)\v11.0" and see if it works if it does then you just need to move the sql data file ".mdf and .ldf" files to your sqlexpress instance and attach them there.


[Bug] typing_extensions.Protocol unsupported under typing_extensions ≥ 4.6.0

This seems to be specific to typing_extensions.Protocol specifically for typing-extensions==4.6.1
from beartype import beartype
from typing import Callable, TypeVar
from typing_extensions import Protocol, runtime_checkable

T= TypeVar("T")

@runtime_checkable
class Partial(Protocol[T]):
    __call__: Callable[..., T]

def foo(x: Partial[float]):
    ...

# raises: beartype.roar.BeartypeDecorHintPepUnsupportedException:
#         function __main__.foo() parameter "x" type hint typing_extensions.Protocol
#         [~T] unsupported but erroneously detected as supported.
beartype(foo)

This error is emitted from:
                    # ............{ UNSUPPORTED                        }............
                    # Else, this hint is neither shallowly nor deeply supported and
                    # is thus unsupported. Since an exception should have already
                    # been raised above in this case, this conditional branch
                    # *NEVER* be triggered. Nonetheless, raise an exception.
                    else:
>                       raise BeartypeDecorHintPepUnsupportedException(
                            f'{_EXCEPTION_PREFIX_HINT}'
                            f'{repr(hint_curr)} unsupported but '
                            f'erroneously detected as supported.'
                        )

so I figured I should report this :)

Thanks so much for all the fine deets on how @beartype has let us all down. :sweat_smile:
More importantly, though... it's really fantastic to hear from you again, @rsokl! I love surreptitiously following the upward trajectory of my favourite users – and your trajectory is ever-upward. You've accumulated a career's worth of utterly fascinating Python-friendly projects in the scant time I've known you. Your legend is legendary.
@beartype: Back in Black (Fur)
Okay. Back to our shameful failures. typing_extensions is non-trivial (albeit critical) to support at runtime, because the type hints it exposes tend to differ from the runtime APIs of equivalent type hints in the standard typing module. But...
It kinda doesn't matter who's to blame in the end, right? It's still ultimately @beartype's fault. We support typing_extensions. So, we just gotta do this.
Thankfully, I have an awful useful interim solution until we do: please use typing.Protocol rather than typing_extensions.Protocol. I know; I know. Super-lame. But Python 3.7 is on the cusp of hitting its official End-Of-Life (EOL). Everything (including @beartype) will be dropping support for Python 3.7 shortly. Since typing.Protocol was first introduced by Python 3.8.0, typing.Protocol should now be safely importable everywhere.
There's basically no demonstrable benefit to importing typing_extensions.Protocol anymore. Of course, @beartype should still resolve this directly for the good of Python- and LLaMa-kind.

FYI, all the changes that have been made to typing_extensions.Protocol in v4.6.0 are also going to be included in typing.Protocol in Python 3.12. The changes (to both typing.Protocol and typing_extensions.Protocol) were mostly made in order to achieve a 20x speedup for isinstance() checks against runtime-checkable protocols.
typing_extensions also backports several significant bugfixes that have been made to typing.Protocol in recent Python versions, but haven't been backported as far as Python 3.8.

Here's a rundown of significant changes that have been made to typing.Protocol in Python 3.12. Without knowing which typing implementation details you're looking at in beartype, not sure which of these are likely to affect you or not:

https://github.com/python/cpython/pull/103034
https://github.com/python/cpython/pull/103141
https://github.com/python/cpython/pull/103152
https://github.com/python/cpython/pull/103159
https://github.com/python/cpython/pull/103160
https://github.com/python/cpython/pull/103280
https://github.com/python/cpython/pull/104622
https://github.com/python/cpython/pull/104556
https://github.com/python/cpython/pull/104559

There's also documentation on the upcoming changes in 3.12 here: https://docs.python.org/3.12/whatsnew/3.12.html#typing.

WOW. Thanks so much for that full-send dissection of typing{_extension,}.Protocol micro- and macro-optimizations, @AlexWaygood. Everything you listed is wonderful to see; this will give user-defined protocols a big boost with respect to runtime efficiency, community adoption, and feature parity with static type-checkers. I can only parrot @rsokl:

That rocks!!

Thankfully, this specific issue is almost certainly a trivial misstep on @beartype's end. Our test suite failed to exhaustively test all typing_extensions attributes. Now, we all pay the QA price. :face_exhaling:

Rejoice! I've escalated this to a priority issue in my head. Hilariously, I appear to have known about this several years ago and then promptly ignored it by explicitly excluding typing_extensions.Protocol from testing. ...actually, that's not funny.
In our voluminous test suite that exhausts my soul whenever I peek inside it, I just discovered this ominous phrasing:
# Note this iteration intentionally excludes the non-standard third-party
# "typing_extensions.Protocol" superclass, whose runtime behaviour has been
# shown to diverge significantly from that of standard implementations.

A pox on you, younger @leycec! May cats meow in the night unconsolably until I finally resolve this. :crying_cat_face:

Curioser and curioser. I've trivially re-enabled testing support for typing_extensions.Protocol, for which we can all be thankful.
The good news is that these tests now pass under typing_extensions < 4.6.0! So, @beartype now officially supports older implementations of typing_extensions.Protocol.
The bad news is that I don't know why these tests purportedly fail under typing_extensions ≥ 4.6.0. I smell a code smell... and it smells. :wilted_flower:

Ah-ha! After a week of relentless video gaming that has worn my thumbs to stubs, I finally investigated this tonight and can explain what is happening here. Pump those fists! Under:

Prior versions of typing and typing_extensions, @beartype detected protocols (i.e., typing{_extensions,}.Protocol subclasses) as generics (i.e., typing{_extensions,}.Generic subclasses). @beartype handled both using the exact same logic, because they were literally the exact same thing. Protocols and generics only appeared to be superficially different things from the end user perspective.
Newer versions of typing and typing_extensions, @beartype now detects protocols and generics as distinct kinds of things. This is a good thing. Although protocols are generics, they're a bit more than just generics. Right?

That said, the entirety of the @beartype codebase still crudely assumes that protocols are literally just generics in disguise. I'll need to tinker a bit in the steaming innards of @beartype to unwind this bad assumption. Let's see if I do something other than relentless video gaming for once. :sweat_smile:

Resolved by 8e318a8. Thanks for your effusive patience, wonderful userbase who are about to be rocked by the newly optimized typing_extensions.Protocol class. Likewise, thanks to @AlexWaygood for the deep dive into exactly what these optimizations are and why we all need them in our lives now.
Since typing_extensions support is critical, I'll be releasing this shortly with our upcoming beartype 0.14.2 patch release. Praise be to the Pythonistas. We drive humanity ever forward.


Transitions not working in Joy when using CssVarsProvider. Cannot read properties of undefined (reading 'create').

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version
@mui/joy 5.0.0-alpha.42
@mui/material 5.10.1

Current behavior 😯
When trying to use a Transition like Collapse within a project that extends the Joy theme and uses CssVarsProvider
You get error

Cannot read properties of undefined (reading 'create')

Expected behavior 🤔
Collapse should work within Joy's CssVarsProvider. The theme object in the Collapse component should have a transitions property.
Steps to reproduce 🕹
Steps:

import CssVarsProvider and extendTheme
extend the theme and give it to CssVarsProvider
Implement a Transition component like Collapse
Observe the error

Context 🔦
Trying to implement a collapsible list for a dashboard app.
Your environment 🌎
https://codesandbox.io/s/objective-bas-2p1zwv?file=/src/App.tsx

Running example:
https://codesandbox.io/s/objective-bas-2p1zwv?file=/src/App.tsx
Note:

Commenting out just the <Collapse> tags (lines 57 & 64) will sanity check that regular components work as expected
Commenting out just the <CssVarsProvider> tags (lines 55 & 67) will sanity check that the Transition works without CssVarsProvider

Potential workaround from the docs: https://github.com/mui/material-ui/issues/34075
Going to try to deepMerge an empty Mui theme to populate the missing transitions property. ⏳

Yeah actually that works. Adding the Mui themes to the Joy theme populates the missing transitions property.
An example Joy theme that works with Transitions and CssVarsProvider.
Pretend this is MyTheme.tsx
import { deepmerge } from "@mui/utils";
import { experimental_extendTheme as extendMuiTheme } from "@mui/material/styles";
import { extendTheme as extendJoyTheme } from "@mui/joy/styles";

declare module "@mui/joy/styles" {
  interface PaletteBackground {
    appBody: string;
    componentBg: string;
  }
}

const muiTheme = extendMuiTheme();

const JoyTheme = extendJoyTheme({
  colorSchemes: {
    light: {
      palette: {
        background: {
          appBody: "var(--joy-palette-neutral-50)",
          componentBg: "var(--joy-palette-common-white)",
        },
      },
    },
    dark: {
      palette: {
        background: {
          appBody: "var(--joy-palette-common-black)",
          componentBg: "var(--joy-palette-neutral-900)",
        },
      },
    },
  }
});

const MyTheme = deepmerge(muiTheme, JoyTheme);

export default MyTheme;

I had seen this in the docs before but just didn't look at it hard enough.
Might I suggest a Transitions page in the Joy docs that just has a short blurb like:

Material UI transitions work with Joy, but the Joy theme needs to be populated with properties from the Material UI Theme to work correctly. Implementation details can be seen at https://mui.com/joy-ui/guides/using-joy-ui-and-material-ui-together/#case-b-material-ui-in-a-joy-ui-project

or better yet- a solution that works without deepmerging themes. Perhaps copying over the transition components into Joy. Or giving the Joy default theme an empty Transitions property that will bolt up to Material UI components.
I assume populating the Joy theme by default with Material properties for better interop has already been ruled out?

I suggest that you visit the guide on "Using Joy UI and Material UI together".

My issue is fixed by following that guide, which is what my code sample above implements.
All good now. Thanks.


"There are no payments recorded for this order..." when there was no error in checkout

Using shoppe-stripe (perhaps I should post this there, let me know). Sometime I after checkout I check the admin site, and there's no payment for the order. Any idea why?
could it be stripe delay? and if that's the case, how do we guarantee that we always get a payment. Or prevent order completion until then?

looking at the stripe logs, there was never a request for a charge... yet it created the customer....

as you can see, the first request (9:35) everything worked fine, customer is created and is charged. But 9:41, no charge.

It looks like the first log is creating a customer so this wouldn't create a charge. Charges are created when the customer completes the order. The payment is then "captured" (taken from the card) when the order is accepted.

I understand that... but the order was completed (I was the customer and testing)... but it seems like stripe was never hit.

I'm running into this too, usually after things have been running in dev env for a while. e.g. last night I was working with everything running smooth, then this morning was getting the exact behavior outlined in this issue. Restarting the app fixes it, but it usually comes back sporadically after some time passes.
I also get the same error mentioned in #214 alongside this. Restarting the parent app fixes that too, but comes back up along with the other issue.

Got the same issue in development (concerned about production)
Tried with 8 orders, 6 went OK but 2 of them have "no payments" error.

Orders are placed as usual, then accepted
Stripe customers are being created correctly
Stripe doesn't report any payments (not even as uncaptured)

Just did a quick look into accept! action and Shoppe::Order.before_acceptance callback, nothing "weird" so far, let me know if you guys found something during your research, I'll try to dig more tomorrow.

sounds like some race condition...? any more discoveries?

Good info there @orendon. The good news is this seems to be isolated to development environments, and related to how those envs skip class caching. I've had an instance running in production mode with test stripe credentials for > 15 hours and it is not displaying this behavior.
It'd be nice to have a reliable test and dev setup though so I'm still interested in fixing the issue.
I think I've found the steps to reproduce this:

Start app which includes shoppe and shoppe-stripe gems in dev env.
Confirm purchases complete successfully with Stripe test credentials / card.
Edit a controller in the parent app (app including Shoppe gem) and save file to trigger code reload.
Complete another test purchase.
Check admin panel orders list to see completed order with no associated payment authorization etc.

My guess is the issue is with the setup method within shoppe-stripe and how it is triggered.
When the dev env reloads the application code it seems Shoppe::Stripe is either clobbered or at least the setup method is not ran. This causes Shoppe::Order.before_confirmation, Shoppe::Order.before_acceptance etc to never run on orders, which explains why it silently fails to authorize and charge appropriately.

Installed shoppe last night and ran into this issue within stripe also


Add stubs for pecl-stomp php extension

https://www.php.net/manual/en/book.stomp.php
issue - https://youtrack.jetbrains.com/issue/WI-46583

@shakura thank you very much!


Cannot build Android, asking for sdktool 33

Bug summary
FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':react-native-permissions:compileDebugAidl'.
> Installed Build Tools revision 33.0.0 is corrupted. Remove and install again using the SDK Manager.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1m 27s

error Failed to install the app. Make sure you have the Android development environment set up: https://reactnative.dev/docs/environment-setup. Run CLI with --verbose flag for more details.
Error: Command failed: ./gradlew app:installDebug -PreactNativeDevServerPort=8081

error Command failed with exit code 1.

Library version
3.6.1
Environment info
System:
    OS: macOS 12.1
    CPU: (8) x64 Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
    Memory: 36.39 MB / 8.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.16.0 - ~/.nvm/versions/node/v16.16.0/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 8.11.0 - ~/.nvm/versions/node/v16.16.0/bin/npm
    Watchman: 2<IP_ADDRESS> - /usr/local/bin/watchman
  Managers:
    CocoaPods: 1.11.3 - /Users/decky/.rvm/gems/ruby-3.0.2/bin/pod
  SDKs:
    iOS SDK:
      Platforms: DriverKit 21.4, iOS 15.5, macOS 12.3, tvOS 15.4, watchOS 8.5
    Android SDK:
      API Levels: 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33
      Build Tools: 26.0.2, 27.0.1, 27.0.3, 28.0.3, 29.0.2, 29.0.3, 30.0.0, 30.0.0, 30.0.2, 30.0.3, 31.0.0, 33.0.0
      System Images: android-25 | Google APIs Intel x86 Atom
      Android NDK: Not Found
  IDEs:
    Android Studio: 2021.2 AI-212.5712.43.2112.8512546
    Xcode: 13.4.1/13F100 - /usr/bin/xcodebuild
  Languages:
    Java: 13.0.2 - /usr/bin/javac
  npmPackages:
    @react-native-community/cli: Not Found
    react: ^16.13.1 => 16.14.0
    react-native: ^0.63.2 => 0.63.4
    react-native-macos: Not Found
  npmGlobalPackages:
    *react-native*: Not Found

Steps to reproduce
run react-native run-android
Reproducible sample code
Its error during build phase so no code to show

btw sdktool 33 is installed from SDKManager, i tried reinstall it many times

Hi @zoontek , facing the face issue, tested on different systems (windows, linux, mac) and still getting the above issue as raised.

I just did this
bash
git clone git@github.com:zoontek/react-native-permissions
cd react-native-permissions
yarn
cd example
yarn
yarn react-native run-android

It worked for me.0
Please note that JDK13 (listed in your environment output) is not viable. react-native specifically uses JDK11 right now.
Basically the links posted above are correct, please heed them
Specifically, your gradle is too old, most likely because your react-native is too old (0.63.x is way too old).
upgrade your react-native and you will get a version of gradle that works.
The example in this repo uses gradle 7.3.3 for instance, it works fine.

Same as @mikehardy here. I tried on two different computers without any issue.
I recommend using Temurin to install Java 11: https://adoptium.net/en-GB/installation
And same advice, try updating React Native (and Gradle) and Android Studio. Everything will work correctly then.
Sorry I can't help more here, but this is not related to react-native-permissions directly 😕

This seems to do the trick @deckyfx
change below to your Android SDK path
cd ~/Library/Android/sdk/build-tools/33.0.0
&& mv d8 dx
&& cd lib
&& mv d8.jar dx.jar

@luis-cicada that's a workaround, and it gets lots of praise on stackoverflow and issue trackers, but is just a workaround. It should not be necessary. There is something wrong in the local install of something, but renaming internal parts of the build-tools is not a great solution, I would dig deeper. Perhaps $ANDROID_HOME/cmdline-tools/latest/bin is not on the $PATH first, and an out of date gradle that cannot handle buildtools 33 yet