map/filter multiprocessing raises errors and corrupts datasets

After upgrading to the 1.0 started seeing errors in my data loading script after enabling multiprocessing.
    ...
    ner_ds_dict = ner_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
    ner_ds_dict["validation"] = ner_ds_dict["test"]
    rel_ds_dict = rel_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
    rel_ds_dict["validation"] = rel_ds_dict["test"]
    return ner_ds_dict, rel_ds_dict

The first train_test_split, ner_ds/ner_ds_dict, returns a train and test split that are iterable.
The second, rel_ds/rel_ds_dict in this case, returns a Dataset dict that has rows but if selected from or sliced into into returns an empty dictionary. eg rel_ds_dict['train'][0] == {} and rel_ds_dict['train'][0:100] == {}.
Ok I think I know the problem -- the rel_ds was mapped though a mapper with num_proc=12. If I remove num_proc. The dataset loads.
I also see errors with other map and filter functions when num_proc is set.
Done writing 67 indices in 536 bytes .
Done writing 67 indices in 536 bytes .
Fatal Python error: PyCOND_WAIT(gil_cond) failed

It seems that I ran into the same problem
def tokenize(cols, example):
  for in_col, out_col in cols.items():
    example[out_col] = hf_tokenizer.convert_tokens_to_ids(hf_tokenizer.tokenize(example[in_col]))
  return example
cola = datasets.load_dataset('glue', 'cola')
tokenized_cola = cola.map(partial(tokenize, {'sentence': 'text_idxs'}),
                             num_proc=2,)

and it outpus (exceprts)
Concatenating 2 shards from multiprocessing
Set __getitem__(key) output type to python objects for ['idx', 'label', 'sentence', 'text_idxs'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Testing the mapped function outputs
Testing finished, running the mapping function on the dataset
Done writing 532 indices in 4256 bytes .
Done writing 531 indices in 4248 bytes .
Process #0 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00000_of_00002.arrow
Process #1 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00001_of_00002.arrow
Spawning 2 processes

and then the program never stop.

same problem.
encoded_dataset = core_data.map(lambda examples: tokenizer(examples["query"], examples["document"], padding=True, truncation='longest_first', return_tensors="pt", max_length=384), num_proc=16, keep_in_memory=True)
it outputs:
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787500 indices in<PHONE_NUMBER>0 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Done writing 1787499 indices in<PHONE_NUMBER>6 bytes . Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns. Spawning 16 processes

Thanks for reporting.
Which tokenizers are you using ? What platform are you on ? Can you tell me which version of datasets and pyarrow you're using ? @timothyjlaurent @richarddwang @HuangLianzhe
Also if you're able to reproduce the issue on google colab that would be very helpful.
I tried to run your code @richarddwang with the bert tokenizer wasn't able to reproduce

Hi, Sorry that I forgot to see what my version is.
But after updating datasets to master (editable install), and latest pyarrow. It works now. Thanks.

Sorry,  I just noticed this.
I'm running this on MACOS the version of datasets I'm was 1.0.0 but I've also tried it on 1.0.2. pyarrow==1.0.1
Consider this code:

    loader_path = str(Path(__file__).parent / "prodigy_dataset_builder.py")
    ds = load_dataset(
        loader_path, name="prodigy-ds", data_files=list(file_paths), cache_dir=cache_dir
    )["train"]
    valid_relations = set(vocabulary.relation_types.keys())

    ds = ds.filter(filter_good_rows, fn_kwargs=dict(valid_rel_labels=valid_relations))

    ds = ds.map(map_bpe_encodings, batched=True, fn_kwargs=dict(tokenizer=vocabulary.tokenizer), num_proc=10)

    # add all feature data
    ner_ds: Dataset = ds.map(
        add_bio_tags,
        fn_kwargs=dict(ner_label_map=vocabulary.ner_labels, tokenizer=vocabulary.tokenizer),
    )
    rel_ds: Dataset = ner_ds.map(
        relation_ds_factory,
        batched=True,
        writer_batch_size=100,
        fn_kwargs=dict(tokenizer=vocabulary.tokenizer, vocabulary=vocabulary),
    )

The loader is essentially a jsonloader with some extra error handling. The data is a jsonlines format with text field and a list of span objects and relation objects.
In the ner_ds a field, ner_labels is added, this is used in the downstream relation_ds_factory. It all runs fine in a single process but I get a KeyError error:
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 348, in relation_ds_factory
    ner_labels = example["ner_labels"]
KeyError: 'ner_labels'

This is just one example of what goes wrong.  I've started just saving the dataset as arrow at the end because it takes a long time to map/filter/shuffle and the caching isn't working (tracked it down to byte differences in the picked functions).
^^ Interestingly if I heed the warning from Tokenizers and set the environment variable, TOKENIZERS_PARALLELISM=true the map just hangs:
[I 200921 21:43:18 filelock:318] Lock<PHONE_NUMBER> released on /Users/timothy.laurent/.cache/huggingface/datasets/_Users_timothy.laurent_.cache_huggingface_datasets_prodigy_dataset_builder_prodigy-ds-5f34378723c4e83f_0.0.0_e67d9b43d5cd82c50b1eae8f2097daf95b601a04dc03ddd504f2b234a5fa247a.lock
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.34ba/s]
#0:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#1:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#2:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#3:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#4:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#5:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#6:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#7:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
#8:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]

Thank you, I was able to reproduce :)
I'm on it

#659 should fix the KeyError issue. It was due to the formatting not getting updated the right way

Also maybe @n1t0 knows why setting TOKENIZERS_PARALLELISM=true creates deadlock issues when calling map with multiprocessing ?

@lhoestq
Thanks for taking a look. I pulled the master but I still see the key error.
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
#0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 21.56ba/s]
#1: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 17.71ba/s]
#2: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 20.45ba/s]
#3: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 26.05ba/s]
#4: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 26.83ba/s]
#5: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.00ba/s]
#6: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.40ba/s]
#7: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 25.91ba/s]
#8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22.46ba/s]
#9: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 20.15ba/s]
#10: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 26.81ba/s]
#11: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 27.45ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 322/322 [00:00<00:00, 1462.85ex/s]
Traceback (most recent call last):                                                                                | 0/1 [00:00<?, ?ba/s]
  File "text2struct/run_model.py", line 372, in <module>
    main()
  File "text2struct/run_model.py", line 368, in main                                                                    | 0/1 [00:00<?, ?ba/s]
    run_model(auto_envvar_prefix="GFB_CIES")  # pragma: no cover
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)                                                                                | 0/1 [00:00<?, ?ba/s]
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/core.py", line 1236, in invoke
    return Command.invoke(self, ctx)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "text2struct/run_model.py", line 136, in run_model
    ctx.invoke(ctx.command.commands[config_dict["mode"]])
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "text2struct/run_model.py", line 187, in train
    run_train_model(_parse_subcommand(ctx))
  File "text2struct/run_model.py", line 241, in run_train_model
    checkpoint_steps=config.train.checkpoint_steps,
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/train.py", line 153, in alternate_training
    max_len=config.model.dim.max_len,
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 466, in load_prodigy_tf_datasets
    folder, file_patterns, vocabulary, cache_dir=cache_dir, test_pct=test_pct
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 447, in load_prodigy_arrow_datasets
    fn_kwargs=dict(tokenizer=vocabulary.tokenizer, vocabulary=vocabulary),
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 348, in relation_ds_factory
    ner_labels = example["ner_labels"]
KeyError: 'ner_labels'

The parallelism is automatically disabled on tokenizers when the process gets forked, while we already used the parallelism capabilities of a tokenizer. We have to do it in order to avoid having the process hang, because we cannot safely fork a multithreaded process (cf https://github.com/huggingface/tokenizers/issues/187).
So if possible, the tokenizers shouldn't be used before the fork, so that each process can then make use of the parallelism. Otherwise using TOKENIZERS_PARALLELISM=false is the way to go.

Thanks for taking a look. I pulled the master but I still see the key error.

I am no longer able to get the error since #659 was merged. Not sure why you still have it @timothyjlaurent
Maybe it is a cache issue ? Could you try to use load_from_cache_file=False in your .map() calls ?

The parallelism is automatically disabled on tokenizers when the process gets forked, while we already used the parallelism capabilities of a tokenizer. We have to do it in order to avoid having the process hang, because we cannot safely fork a multithreaded process (cf huggingface/tokenizers#187).
So if possible, the tokenizers shouldn't be used before the fork, so that each process can then make use of the parallelism. Otherwise using TOKENIZERS_PARALLELISM=false is the way to go.

Ok thanks :)
Is there something we should do on the datasets side to avoid that that the program hangs ?
Also when doing .map with a tokenizer, the tokenizer is called once on the first examples of the dataset to check the function output before spawning the processes. Is that compatible with how tokenizers are supposed to be used with multiprocessing ?

#659 fixes the empty dict issue
#688 fixes the hand issue

Hmmm I pulled the latest commit, b93c5517f70a480533a44e0c42638392fd53d90, and I'm still seeing both the hanging and the key error.

Hi @timothyjlaurent
The hanging fix just got merged, that why you still had it.
For the key error it's possible that the code you ran reused cached datasets from where the KeyError bug was still there.
Could you try to clear your cache or make sure that it doesn't reuse cached data with .map(..., load_from_cache=False) ?
Let me know if it it helps

Hi @lhoestq ,
Thanks for letting me know about the update.
So I don't think it's the caching - because hashing mechanism isn't stable for me -- but that's a different issue. In any case I rm -rf ~/.cache/huggingface to make a clean slate.
I synced with master and I see the key error has gone away, I tried with and without the TOKENIZERS_PARALLELISM variable set and see the log line for setting the value false before the map.
Now I'm seeing an issue with .train_test_split() on datasets that are the product of a multiprocess map.
Here is the stack trace
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 451, in load_prodigy_arrow_datasets
    ner_ds_dict = ner_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/src/datasets/src/datasets/arrow_dataset.py", line 168, in wrapper
    dataset.set_format(**new_format)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/src/datasets/src/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/src/datasets/src/datasets/arrow_dataset.py", line 794, in set_format
    list(filter(lambda col: col not in self._data.column_names, columns)), self._data.column_names
ValueError: Columns ['train', 'test'] not in the dataset. Current columns in the dataset: ['_input_hash', '_task_hash', '_view_id', 'answer', 'encoding__ids', 'encoding__offsets', 'encoding__overflowing', 'encoding__tokens', 'encoding__words', 'ner_ids', 'ner_labels', 'relations', 'spans', 'text', 'tokens']

Thanks for reporting.
I'm going to fix that and add a test case so that it doesn't happen again :)
I'll let you know when it's done

Sure thing, @lhoestq.
https://colab.research.google.com/drive/1lg4fbyrUO6m8ssQ2dNdVFaUqMUfA2zZ3?usp=sharing

Thanks @timothyjlaurent ! I just merged a fix on master. I also checked your notebook and it looks like it's working now.
I added some tests to make sure it works as expected now :)

Great, @lhoestq . I'm trying to verify in the colab:
changed
!pip install datasets

to
!pip install git+https://github.com/huggingface/datasets@master

But I'm still seeing the error - I wonder why?

It works on my side @timothyjlaurent on google colab.
Did you try to uninstall datasets first, before updating it to master's version ?

I didn't -- it was a new sessions --- buuut - look like it's working today -- woot! I'll close this issue. Thanks @lhoestq


Map formatting

Why there is no space between : to ' in this test result?
https://github.com/opencypher/openCypher/blob/35f7e54042eadeac33c5986c91cf1e0b1a528a70/tck/features/expressions/literals/Literals6.feature#L217

While there is no semantic meaning to whitespace between symbols in Cypher (although at least one is often required), the style guide suggests that the map you reference should look like this:
{a: ' { b : ', c: {d: ' '}, d: ' } '}

which it doesn't, so thanks for raising our attention to it.
It isn't straight-forward to properly style this map manually since the literal string values look like map syntax themselves. However, all characters within the string delimiters (') should of course be retained verbatim from the query expression.


Sort entries in output.tf

Use Case
The output.tf file that is created during plural build is not sorted. This often causes the order of the outputs to be changed with subsequent plural build commands, causing the terraform to be deployed unnecessarily.
Ideas of Implementation
Sort the output entries in the output.tf file so that unnecessary terraform build and deploys don't occur.

@michaeljguarino

I thought we did sort these

I'll confirm again then, but I did see terraform deploy a couple times without actual changes because of something like this.


Static MJPEG example on M1S Dock

Hi, I'm trying to run the UVC static MJPEG example on my device. It compiles and flashes fine, and both Windows and macOS recognize the camera seemingly correctly. However, I never get an image transferred. Here is the log from the serial monitor:
  ____               __  __      _       _       _
 |  _ \             / _|/ _|    | |     | |     | |
 | |_) | ___  _   _| |_| |_ __ _| | ___ | | __ _| |__
 |  _ < / _ \| | | |  _|  _/ _` | |/ _ \| |/ _` | '_ \
 | |_) | (_) | |_| | | | || (_| | | (_) | | (_| | |_) |
 |____/ \___/ \__,_|_| |_| \__,_|_|\___/|_|\__,_|_.__/

Build:23:50:55,Aug 18 2023
Copyright (c) 2022 Bouffalolab team
======== flash cfg ========
flash size 0x01000000
jedec id     0xEF4018
mid              0xEF
iomode           0x04
clk delay        0x01
clk invert       0x01
read reg cmd0    0x05
read reg cmd1    0x35
write reg cmd0   0x01
write reg cmd1   0x31
qe write len     0x01
cread support    0x00
cread code       0xFF
burst wrap cmd   0x77
===========================
dynamic memory init success,heap size = 21 Kbyte
sig1:ffff32ff
sig2:0000ffff
video test
[I/USB] Close ep:0x81 type:1
CLOSE
CLOSE
[I/USB] Close ep:0x81 type:1
CLOSE
CLOSE
[E/USB] descriptor <type:3,index:4> not found!
[E/USB] standard request error
[I/USB] Setup: bmRequestType 0x80, bRequest 0x06, wValue 0x0304, wIndex 0x0409, wLength 0x00ff

Any suggestions?

I do not know. Sorry, It works in mine.

You"re using this one https://github.com/CherryUSB/cherryusb_bouffalolab/tree/master/examples/peripherals/usbdev/usbd_static_video?

Right

You"re using this one https://github.com/CherryUSB/cherryusb_bouffalolab/tree/master/examples/peripherals/usbdev/usbd_static_video?

This is the same with bouffalolab official.

I'll give it another try later! I'm compiling under darwin/arm64, do you think that could make a difference? (just spitballing here...)


Broken link on documents

🛑 IMPORTANT: You can get your feedback addressed faster if you use the comment section for the article in which you encountered a problem (if available).
Link to article:
There is a link in this article
https://docs.microsoft.com/en-us/samples/azure-samples/active-directory-aspnetcore-webapp-openidconnect-v2/enable-webapp-signin/
Under the heading Details of the chapters There are 3 URL links under point 5 all of the pages references are moved/missing (404)
Problem:
Under the heading Details of the chapters There are 3 URL links under point 5 all of the pages references are moved/missing (404)
The bad URL reference are: https://docs.microsoft.com/en-us/samples/azure-samples/active-directory-aspnetcore-webapp-openidconnect-v2/enable-webapp-signin/5-WebApp-AuthZ/5-2-Groups/README-incremental-instructions.md
https://docs.microsoft.com/en-us/samples/azure-samples/active-directory-aspnetcore-webapp-openidconnect-v2/enable-webapp-signin/5-WebApp-AuthZ

@tvanwijk Please move this issue to https://github.com/Azure-Samples/active-directory-aspnetcore-webapp-openidconnect-v2/issues

I'm not sure how to move an issue. This is more of a courtesy ticket because stuff isn't right, than something I'm invested in. Feel free to close or move or whatever.

@tvanwijk, you can just close this issue (copy and paste) and open one in https://github.com/Azure-Samples/active-directory-aspnetcore-webapp-openidconnect-v2/issues.  That's the repo where this issue's doc is maintained.  Thanks.

Looks like the links were already corrected.


write 104 unit test

Wrote three tests.

Whether the component and components passed to it as props are rendered correctly.
Whether the functions of opening and closing the component work.
Whether hiding styles are applied if isHiding is true.
Coverage 100%

Would you mind trying to change the path for the component?

Test looks very complicated and is hard to reading.
You can use this data-testId to finde open popover

I think you should ask mentor, how to make this test simplier!


Morning session P2P 30th March

Morning session P2P 30th March

Good work!

Very good work. No issues.

Why didn't you make --fix? :rofl:

Looks good to me. Great job @rammazzoti2000 ! 💯

Good job @rammazzoti2000, the project is clean. I'll just have a few remarks:

You linter asks me to put a space after the comma or closing parenthesis which ends a line and you agree with me that it's useless if there is nothing after them
The program breaks when there is no file passed in parameter to the function, Maybe you should manage the case when there is no file in order to check local files or just to throw a message and say there is no file
Again good job on this one, clean and simple


search on ListEntries fails with boolean/tinyInt field displayed as "check"

Bug report
What I did:
added in a XXCrudController.php/setup() on a tinyInt(1) mysql field
       $this->crud->addColumn([             'name' => 'active',             'type' => 'check',             'label' => 'Actif ?',         //    'visibleInTable' => false,         //    'searchLogic' => false         ]);
What I expected to happen:
search working
What happened:
error message with a huge not helpful stack trace in JSON as http response
What I've already tried to fix it:
I changed the type to "text", same error. Changed to "number" and it works. I guess this is highly related.
I thought that adding searchLogic => false would avoid it but it seems all fields are searched even with this. Also the error with disabled => disabled and also with visibleInTable => false, these parameters don't seem to affect the search behaviour.
I then remembered that I forgot to use     protected $casts = [         "active" => "boolean"     ]
in my Model but it is not better.
Backpack, Laravel, PHP, DB version:
BP 3.5, Laravel 5.7, MySQL 5.5.52
So, any idea ? Real bug ?
(( Thanks a lot by the way, I'm discovering Backpack and enjoying it. ))

Hi @flartet ,
I think the problem might be somewhere else. I couldn't get the same issue as you, no matter how much I tried doing the exact same thing:

tinyint(1) column

simple check column, with searchLogic or without

They don't mess up the search at all, it works fine for me:
https://cl.ly/1f65225e2406
Sorry I couldn't be of more help, but it doesn't look like the check column itself is the problem. Let me know if you figure this out. Might help others.
Cheers!

Thanks a lot for the research. At the moment, I figured it out by removing the column :)


Update build.gradle to use httpPort

gretty port has been deprecated and httpPort should be used instead.  Without this change, I see the error Could not set unknown property 'port' for object of type org.akhikhl.gretty.GrettyExtension. when I run ./gradlew jettyRun

This has been updated for all samples


GLFW alters values causing crash on win32 in Release with Dbg Info when alt-tabbing away

GLFW compile and runtime version: 3.3
I'm not really sure what's causing this, though I think its GLFW. I'm writing a game that uses GLFW as the keyboard/mouse/... input and rendering backend. I haven't tried this in the example programs but my game does this (using FMOD for sound backend). In my game I have the following game loop (sorry for no indentation, I haven't run a formatter over it yet):
void gameLoop(GLFWwindow* window) {
// Configure FPS rate
constexpr unsigned long long TICKS_PER_SECOND = 60;
constexpr unsigned long long SKIP_TICKS = 1000 / TICKS_PER_SECOND;
unsigned long long next_game_tick = glfwGetTimerValue();
long long sleep_time = 0;
bool done = false;
GLFWkeyfun keyfun = nullptr;
GLFWcharmodsfun charmodsfun = nullptr;
while (!done) {
// Moderate FPS
next_game_tick += SKIP_TICKS;
sleep_time = next_game_tick - glfwGetTimerValue();
if (sleep_time>=0)
std::this_thread::sleep_for(std::chrono::duration<long long, std::milli>(sleep_time));

// Update GLFW
glfwPollEvents();
glfwSwapBuffers(window);
// Update sound system
Audio::system->update();
if (glfwGetWindowAttrib(window, GLFW_FOCUSED)) {
// Are we in a menu, and if so, is the user data for this game mode set to something?
if (getGameMode()==gameMode::Menu&&getGameModeUserData().has_value()) {
// Try to cast the user data to a valid menu object
auto menu = std::any_cast<Menus::Menu*>(getGameModeUserData());
// Transfer keyboard input control to the newly cast menu object
keyfun = glfwSetKeyCallback(window, &menu->menuKeyCallback);
charmodsfun = nullptr;
} else if (getGameMode()==gameMode::TextInput) {
// We're getting text input, transfer all keyboard input control to this input object
auto input = std::any_cast<Game::UI::TextInputDialog*>(getGameModeUserData());
keyfun = glfwSetKeyCallback(window, &input->keyCallback);
charmodsfun = glfwSetCharModsCallback(window, &input->textInputCallback);
} else if (getGameMode()==gameMode::Exit) {
// Terminate
done = true;
} else if (getGameMode()==gameMode::Playing) {
// get the current layer object
player = std::any_cast<Player::Player*>(getGameModeUserData());
keyfun = glfwSetKeyCallback(window, &processKeysForPlayingGameMode);
// The below lines are what's causing the crash...
// *** CRASH START ***
if (player->getX()+1<dungeons[player->getFloor()].size() && player->getX()+1 >= 0)
northsouth_ch->setMute(false);
if (player->getX()-1>= 0 && player->getX()+1 < dungeons[player->getFloor()].size())
northsouth_ch->setMute(true);
if (player->getY()+1<dungeons[player->getFloor()][player->getX()].size() && player->getY()+1 >= 0)
eastwest_ch->setMute(false);
if (player->getY()-1>= 0 && player->getY()+1 < dungeons[player->getFloor()][player->getX()].size())
eastwest_ch->setMute(true);
} else {
// Clear keyboard control
keyfun = nullptr;
charmodsfun = nullptr;
}
// Check for Python errors
if (PyErr_Occurred() != nullptr) {
PyObject *exctype;
PyObject *value;
PyObject *traceback;
PyErr_Fetch(&exctype, &value, &traceback);
PyErr_NormalizeException(&exctype, &value, &traceback);
PyObject* str_exc_type = PyObject_Repr(exctype);
PyObject* pyStr = PyUnicode_AsEncodedString(str_exc_type, "utf-8", "Error ~");
std::string caption =  PyBytes_AsString(pyStr);
PyObject* str_exc_value = PyObject_Repr(value);
PyObject* pyExcValueStr = PyUnicode_AsEncodedString(str_exc_value, "utf-8", "Error ~");
std::string body =  PyBytes_AsString(pyExcValueStr);
boost::replace_all(caption, "\\'", "\'");
boost::replace_all(caption, "\\\"", "\"");
boost::replace_all(caption, "\\?", "\?");
boost::replace_all(caption, "\\a", "\a");
boost::replace_all(caption, "\\b", "");
boost::replace_all(caption, "\\f", "\f");
boost::replace_all(caption, "\\n", "\n");
boost::replace_all(caption, "\\r", "\r");
boost::replace_all(caption, "\\t", "\t");
boost::replace_all(caption, "\\v", "\v");
PyErr_Restore(exctype, value, traceback);
LOGE << caption << ": " << body;
auto selection = tinyfd_messageBox("Warning", "An error has occured in a running Python program. The program/and or game may be in an unstable state and may crash or exhibit strange behavior if you continue. However, this may not be an actual issue and you may be free to continue on. You may check the log at any time to determine the error. Would you like to continue running the game?", "yesno", "warning", 0);
if (!selection)
done = true;
}
// Check for OpenGL errors
auto glerr = glGetError();
while (glerr != GL_NO_ERROR) {
if (glerr==GL_INVALID_ENUM)
LOGW << "GL: An unacceptable value was specified for an enumerated argument. The offending command has been ignored and will have no other side effect than to set the error flag.";
else if (glerr == GL_INVALID_VALUE)
LOGW << "GL: A numeric argument is out of range. The offending command has been ignored and will have no other side effect than to set the error flag.";
else if (glerr == GL_INVALID_OPERATION)
LOGW << "GL: a specified operation is not allowed in the current state. The offending command has been ignored and will have no other side effect than to set the error flag.";
else if (glerr == GL_INVALID_FRAMEBUFFER_OPERATION)
LOGW << "GL: The framebuffer object is not complete. The offending command has been ignored and will have no other side effect than to set the error flag.";
else if (glerr == GL_OUT_OF_MEMORY) {
LOGF << "GL: There is not enough memory left to execute the command. The state of the GL is undefined, except for the state of the error flags, after this error is recorded.";
tinyfd_messageBox("Critical error", "There is not enough memory available to complete a requested operation in the graphics subsystem. The graphics subsystem has entered an unstable state. To avoid corruption or dangerous behavior that may be caused due to the instability of the graphics subsystem after this point, this program must terminate. Click OK, then click OK again, to terminate the application.", "ok", "error", 1);
std::terminate();
} else if (glerr == GL_STACK_UNDERFLOW)
LOGW << "GL: An attempt has been made to perform an operation that would cause an internal stack to underflow.";
else if (glerr == GL_STACK_OVERFLOW)
LOGW << "GL: An attempt has been made to perform an operation that would cause an internal stack to overflow.";
else
LOGW << "GL: error description not available. EC: " << glerr;
glerr = glGetError();
}
}
}
}

The issue is that when I alt tab out of the window and back in, the players coordinates get altered to a random value, and its usually outside of the range of valid coordinates, causing a crash. I've tried fixing this by checking if the game is focused but that doesn't seem to do anything. I don't know if its GLFW or what it is, but I thought I'd ask if that's a bug or not.

It's highly unlikely that glfw is modifying random parts of your program's memory, so I'd first encourage you to audit (sanity check as well as runtime check) any lines of code that change the player position to see if the bad values are being written by your code accidentally, or if the memory is being overwritten indirectly somehow.
A few other questions to guide your debugging:
Are the random values different every time, or always the same?
Does the issue occur only when alt-tabbing, or any time the window loses focus? Is it in fact the alt-tabbing, or does just pressing Alt or tab trigger it?
Your title mentions Release with debug info. Does this issue not occur in Debug builds? Or with debug info disabled?
I hope this helps; let us know if you either hit a dead end or manage to find the root cause.

The random values are different all the time.
It happens whenever the window loses focus for any reason, including minimizing. (Its maximized, if I'm not mistaken, but I don't think that's what's causing it.)
I can't build with debug, only release and release with debug info, since my program embeds Python and Numpy and Numpy fails to be imported when I build with debug.
So far as I can tell, nothing else is touching the value; the stack trace from the issue looks like:
0:000> kp
Child-SP          RetAddr           Call Site
(Inline Function) --------`-------- main!std::vector<int,std::allocator<int> >::size [c:\program files (x86)\microsoft visual studio\2017\enterprise\vc\tools\msvc\14.16.27023\include\vector @ 1704]
000000f0`c62fd640 00007ff7`760e8728 main!Game::gameLoop(struct GLFWwindow * window = 0x0000021f`24631e40)+0x436 [c:\users\ethin\source\game\client\src\game.cpp @ 69]
000000f0`c62fdaf0 00007ff7`761ad326 main!wWinMain(struct HINSTANCE__ * hInstance = 0x00000000`0000000a, struct HINSTANCE__ * hPrevInstance = 0x00000000`00000000, wchar_t * pCmdLine = 0x00000000`00000000 "", int nCmdShow = 0n0)+0x2328 [c:\users\ethin\source\game\client\src\client.cpp @ 155]
(Inline Function) --------`-------- main!invoke_main+0x21 [d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
000000f0`c62ffea0 00007ffd`db1c4034 main!__scrt_common_main_seh(void)+0x106 [d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
000000f0`c62ffee0 00007ffd`dd333691 KERNEL32!BaseThreadInitThunk+0x14
000000f0`c62fff10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

The offending lines are:
Line 69 of game.cpp:
if (player->getY()+1<dungeons[player->getFloor()][player->getX()].size() && player->getY()+1 >= 0)

Line 155 of client.cpp:
Py_FinalizeEx();

(That's really, really weird.)
I also think it meant line 154:
Game::gameLoop(window);

The players position is only modified in the processKeysForPlayingGameMode() function:
void processKeysForPlayingGameMode(GLFWwindow * window, int key, int scancode, int action, int mods) {
if (player->getX() >=0 && player->getX() < dungeons[player->getFloor()].size() && player->getY() >= 0 && player->getY() < dungeons[player->getFloor()][player->getX()].size() && player->getFloor()>= 0 && player->getFloor() <= maxDungeonDepth && player->getFloor() >=0 && player->getFloor() <= maxDungeonDepth) {
switch (key) {
case GLFW_KEY_DOWN: {
if (player->getY()-1>=0) {
if (dungeons[player->getFloor()][player->getX()][player->getY()-1]>0 && action==GLFW_PRESS) {
player->setY(player->getY()-1);
}
}
}
break;
case GLFW_KEY_UP: {
if (player->getY()+1<dungeons[player->getFloor()][player->getX()].size()) {
if (dungeons[player->getFloor()][player->getX()][player->getY()+1]>0 && action==GLFW_PRESS) {
player->setY(player->getY()+1);
}
}
}
break;
case GLFW_KEY_LEFT: {
if (player->getX()-1>=0) {
if (dungeons[player->getFloor()][player->getX()-1][player->getY()]>0 && action==GLFW_PRESS) {
player->setX(player->getX()-1);
}
}
}
break;
case GLFW_KEY_RIGHT: {
if (player->getX()+1<dungeons[player->getFloor()].size()) {
if (dungeons[player->getFloor()][player->getX()+1][player->getY()]>0 && action==GLFW_PRESS) {
player->setX(player->getX()+1);
}
}
}
break;
}
}
}

Not really sure how that function could be called when I'm outside the program. I've got the following window hints set:
glfwWindowHint(GLFW_CLIENT_API, GLFW_OPENGL_API);
glfwWindowHint(GLFW_CONTEXT_CREATION_API, GLFW_NATIVE_CONTEXT_API);
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 3);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 3);
glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);
glfwWindowHint(GLFW_RESIZABLE, GL_FALSE);
glfwWindowHint(GLFW_CONTEXT_RELEASE_BEHAVIOR, GLFW_RELEASE_BEHAVIOR_FLUSH);

If possible, I would ask you to try to create a minimal reproduction case. i.e. disable all FMOD code, disable Python integration, disable OpenGL rendering, etc, leaving just GLFW, game state, and the code that updates the player's position. That would really help to narrow down the issue.
It is quite possible that the wrong line of code is being reported as the site of the crash; I have seen this happen often when debugging Release builds in MSVC.
When in doubt, I resort to printf debugging. I don't know if you have done this already, but I would try putting a printf (or similar) of parameters inside setX and setY just to be absolutely sure the values are not being written by your code.
Also make sure you are compiling with as many warnings, bounds checks, etc. as possible.

I've moved Python integration to only be used when its needed. I can't
disable FMOD integration because I'm visually impaired and need some
kind of indicator of audio, and I can't remove the Python integration
because that's used to generate the map (I've tried porting the
library that I want to use to C++ but don't know the C++ equivalents
to the pythonic syntax used for most of the map generators). So a
minimalist example will be hard to reproduce.
I'll try the printf-style debugging though. I'm compiling with /W4 and
/wd4100 as my compiler flags, along with /openmp and /MP. (I have
/bigobj in there too, but that's because I have a huge file in there.)
Are there any other bounds-checking or other flags you think I should enable?
On 4/30/19, A. Tombs<EMAIL_ADDRESS>wrote:

If possible, I would ask you to try to create a minimal reproduction case.
i.e. disable all FMOD code, disable Python integration, disable OpenGL
rendering, etc, leaving just GLFW, game state, and the code that updates the
player's position. That would really help to narrow down the issue.
It is quite possible that the wrong line of code is being reported as the
site of the crash; I have seen this happen often when debugging Release
builds in MSVC.
When in doubt, I resort to printf debugging. I don't know if you have done
this already, but I would try putting a printf (or similar) of parameters
inside setX and setY just to be absolutely sure the values are not being
written by your code.
Also make sure you are compiling with as many warnings, bounds checks, etc.
as possible.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/glfw/glfw/issues/1473#issuecomment-488089180

--
Signed,
Ethin D. Probst

Update: so I added some OutputDebugString() calls and checked for debugger presence with IsDebuggerPresent(). All of my code is working properly... or so says my functions. OutputDebugString() reported no unusually high coordinates; all were within the limits of the map.
The failures seem to be caused by (supposedly if the debugger is to be believed that is) me retrieving the value. What's odd about that is the fact that no multithreading is going on. These functions aren't thread-safe anyway. GL calls only happen when:

The application starts -- this is used to clear the screen to black.
To initialize the
affine transformation with glViewport.
To draw text on the screen with a graphics function call. (This bug was happening before this function was even implemented, though.)

I can use true bounds checking on the map and suffer the overhead. I don't know how using that will tell me what's changing the value, though.

From the stack trace, the part that is crashing is this: dungeons[player->getFloor()][player->getX()].size(). Most likely dungeons[player->getFloor()][player->getX()] is returning an invalid reference due to one (or both) of the indices being out of range, and size() tries to operate on an invalid pointer.
Note that at() is safer than operator[] for std::vector, as it does bounds checking and will throw an exception rather than crashing when given an out of range index. There is a small performance cost associated with this.
But, as you say, this is just the symptom of whatever the true problem is.
As you have verified that the bad values are not being written by your setter functions, that points to memory corruption. I am not sure if this is stack or heap corruption.
Useful compiler flags for detecting stack corruption are are /GS and /RTC1.
Data breakpoints are probably your best bet for detecting heap corruption, see https://docs.microsoft.com/en-us/visualstudio/debugger/using-breakpoints?view=vs-2019#BKMK_set_a_data_breakpoint_native_cplusplus You will need to know the address of the coordinates within the player struct.

So, I'm doing something that may be a bit extreme. I know that its unlikely that GLFW is the problem, and I agree with that, but I still need to confirm it. So I'm swapping it out with SDL2. It won't be overly hard since GLFW didn't do very much, and this will allow me to make some improvements to the overall code while I'm at it. I'll switch it to /rtc1 (I just have /rtc). I'll update this thread when I'm done and report to you the results, though I don't expect any difference. Still, its good to do this kind of thing now rather than later. This is definitely the best phase to fix this kind of stuff (the game is only a week or two old).
Again, its probably very extreme, and I'd agree if GLFW actually did a lot. But it didn't, so this is a trivial process at this point. I'll try to find the issue while I'm doing the conversion.

OK, so its not GLFW or SDL2. Apparently I mistyped some compiler options. I'll keep working on this, but this has officially left GLFW territory, so I'll close this issue.


Update expo-router.md

"To use Solito in an existing Solito project" supposed to be "To use Expo Router in an existing Solito project", right?

Thanks! Closed with #170


kirkstone: ERROR: gstreamer1.0-plugins-bad-1.20.7-r0 do_configure: QA Issue: gstreamer1.0-plugins-bad: invalid PACKAGECONFIG: gpl [invalid-packageconfig]

Description
gstreamer1.0-plugins-bad fails to build due to removed gpl packageconfig option.
Steps to reproduce the issue:

Build with kirkstone branch

Describe the results you received:
ERROR: gstreamer1.0-plugins-bad-1.20.7-r0 do_configure: QA Issue: gstreamer1.0-plugins-bad: invalid PACKAGECONFIG: gpl [invalid-packageconfig]
Describe the results you expected:
Successful build.
Additional information you deem important (e.g. issue happens only occasionally):
Additional details (revisions used, host distro, etc.):

I met the same during the building, but in my case, bitbake classified it as a warning and finished normally the building process. Unfortunately gstreamer1.0 works only with specific settings and without hardware acceleration.

I still see information about it, but in my case, it warning and not an error (branch kirkstone uptodate):
WARNING: gstreamer1.0-plugins-bad-1.20.7-r0 do_configure: QA Issue: gstreamer1.0-plugins-bad: invalid PACKAGECONFIG: gpl [invalid-packageconfig]


PHP API: How to use go-ego riot in mysql datastore servers?

As the title suggests, how to use go-ego riot in servers w/ mysql as datastore?

It is recommended to use the binlog to synchronize.


Incorrect sizing of elements in a Voronoi Treemap

Hello,
I am trying to create a Voronoi Treemap of some population data in Italian cities but I believe Raw is not showing the data properly. Many elements are not proportional as they should, and as you can see in the example cities with values of 1-5 have roughly the same size as others sized ~50.
Here is a screenshot:

The issue arises with Raw version 2.0 in Firefox 102 and Chrome 103 on Windows 10, and from other people with different computers.
Attached is the Raw project file to reproduce the issue
centro-test.zip
If there's any additional info you might need please let me know.

Sizes are actually incorrect. That is due to the iterative approach of voronoi treemaps - it is computed several times to get as close as possible to the real values.
We should expose the convergenceRatio and maxIterationCount values: https://github.com/Kcnarf/d3-voronoi-treemap#voronoiTreemap_convergenceRatio

Thanks for the quick answer, mikima! If you think more iterations would get the shapes closer to the actual values perhaps it could be a good idea to add an option to increase the number of iterations

after some testing, it seems that the minWeightRatio is the best property to be used. will be added in next release.


Some callback errors aren't handled

https://github.com/crafatar/crafatar/search?utf8=✓&q=FIXME+not+handled

Using a flow control system like the async package or any promise library would let you elegantly handle errors at a single point - as well as clean up code quite a bit.


Making array.jl a bit simpler

This PR just makes the structure of array.jl simpler by reducing the amount of code that is needed for various operations.

Coverage increased (+0.3%) to 94.301% when pulling ddbb91182b783f2b1184a31387bf4ea0b6fef1bb on array into 6b072fec4a1cc53f1e30c32f3f443db0037583f1 on master.


BinaryReader in the Python API does not have a ReadCString method

We have it in the C++ API, but it is implemented in the C++ API. We will need to do the same in the Python API

Isn't that just:
https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.get_ascii_string_at
?

No. When you have a BinaryReader object you're usually parsing things out of a RawView to create a view. In C++ api we have a string BinaryReader::ReadCString(size_t maxSize) the python API doesn't have the equivalent.  Though looking at the C++ API side implementation its quite poor and we should really expose the one in the core instead. Since we moved our binaryviews to use the API I'm betting this is having some significant performance impacts and so we should prioritize this fixe.


The terminal IME composition view appears on top of the text when at the bottom of the screen

See it appears on top of the second character being composed.

Upstream issue: https://github.com/sourcelair/xterm.js/issues/208

@Tyriar not sure which language you were typing on what platform when you found the issue, but the behavior on Mac with Chinese IME is not that perfect, see below gif

While typing nihao, the text is sometimes covered by the IME hint popup and while typing constnatly, the popup keeps moving, which is a little bit annoying.

The bug was created in #11334 and technically this is done so I'll make this verified


Terraform apply issue

What did you expect to happen?
Terraform apply should run without errors
What actually happened?
Multiple errors:
Invalid template interpolation value
kubernetes_dashboard": [REMOVED]
Fix
Modify "null_resource" "apply" in k8s.tf and use "${google_container_cluster.vault.location}" instead of region since there isn't region in GKE resource
Modify gcp.tf and remove the addon for kubernetes_dashboard

Fixed in c98b3f4. Thanks!


Compiled DNN training with JAX causes segfaults.

Running dnn_training_repro.py causes segmentation faults at somewhat  random intervals. On my machine it does so every time, but doesn't cause segfaults if you move "GradientDescent" to the top of the OPTIMIZERS_TO_HPARAMS dictionary.
Aside from the optimizer order, both lines 90 and 95 are necessary to cause the error.

Ok, I've repro'd an issue like what you reported on discord:
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Compiling: Adam
  lowering through HLO
  compiling HLO --> LLVM-AOT
  loading compiled binary into runtime bindings
Created IREE driver dylib: <pyiree.rt.binding.HalDriver object at 0x7f3e501516b0>
SystemContext driver=<pyiree.rt.binding.HalDriver object at 0x7f3e501516b0>
Traceback (most recent call last):
  File "jax_mlp_crash.py", line 110, in <module>
    main()
  File "jax_mlp_crash.py", line 92, in main
    module = iree.rt.load_module(cpp_vm_module, config=rt_config)
  File "/usr/local/google/home/laurenzo/src/iree-build/bindings/python/pyiree/rt/system_api.py", line 325, in load_module
    return load_modules(module, config=config)[0]
  File "/usr/local/google/home/laurenzo/src/iree-build/bindings/python/pyiree/rt/system_api.py", line 318, in load_modules
    context = SystemContext(modules=modules, config=config)
  File "/usr/local/google/home/laurenzo/src/iree-build/bindings/python/pyiree/rt/system_api.py", line 269, in __init__
    self._vm_context = _binding.VmContext(instance=self._config.vm_instance,
RuntimeError: Error creating vm context with modules: INTERNAL

This happens on the very first module that we are attempting to load, and I was able to repro it by using compiler binaries from ~a week ago with a runtime from ~tonight. When I upgraded to yesterday's compiler binary snapshots, the error did not happen. I haven't done a full bisect, but I believe that this points to something changing in the binary ABI in the intervening time. I was then also able to get segfaults/stack traces as you reported that looked spookily like the gc bug I fixed last week. I checked, and I happened to have a branch checked out from right before that fix (I think).

I'm still having trouble activating ASAN in a session where jax is loaded, so I'm doing this the somewhat imprecise way (we will need to track that down but trying to get you to a good state first).

Also, please verify the versions that you have installed (something like pip freeze | grep iree). Mine are:
iree-compiler-snapshot==20210207.85
iree-runtime-snapshot==20210207.85
iree-tools-xla-snapshot==20210207.85

And your example has run multiple times without a crash on those versions. Given that I was able to repro both of your issues with combinations of older versions, I would like to double check...

(drive by: you can use iree_status_to_string in CheckApiStatus to get a source location and message for the status instead of just the code)

Also, please verify the versions that you have installed (something like pip freeze | grep iree). Mine are:
iree-compiler-snapshot==20210207.85
iree-runtime-snapshot==20210207.85
iree-tools-xla-snapshot==20210207.85

And your example has run multiple times without a crash on those versions. Given that I was able to repro both of your issues with combinations of older versions, I would like to double check...

I haven't taken the time to figure out how to mix the pip binaries with a local CMake build, and I haven't had the time to set up packaging for the JAX frontend, so I've just been building everything locally. I saw the segfaults at a commit from a few days ago (I think after your fixes), but updated to 707cf08c913f222ab73a641f5434b3ccc6f4341e to be sure before creating the repro.

(drive by: you can use iree_status_to_string in CheckApiStatus to get a source location and message for the status instead of just the code)

Thanks. Will send a PR shortly with the better message printing. Here is what it reports (indeed looks like a version skew problem): RuntimeError: Error creating vm context with modules: /usr/local/google/home/laurenzo/src/iree/iree/vm/context.c:146: INTERNAL; import function signature mismatch between module and source hal; expected 0r[i]i.r but got 0ri[i].

It is quite possible that there is something deeper going on, but from what I see, we've got quite a few bugs fixed or outstanding recently and the possibility of some stale state:

The "gc issue" #4738 (fixed 5 days ago)
A compiler<->runtime version issue that seems to have come about in the last 5 days
#4706 which generates bad code for (I think) any depthwise conv, resulting in weird segfaults (which was likely the initial trigger for mobilenetv1 issues)

I would not discount there be another undiscovered 4 on that list, but I haven't been able to repro it. The real answer is to find out why we can't enable ASAN and JAX at the same time... without that, we are flying blind a bit.

(drive by: you can use iree_status_to_string in CheckApiStatus to get a source location and message for the status instead of just the code)

Thanks. Will send a PR shortly with the better message printing. Here is what it reports (indeed looks like a version skew problem): RuntimeError: Error creating vm context with modules: /usr/local/google/home/laurenzo/src/iree/iree/vm/context.c:146: INTERNAL; import function signature mismatch between module and source hal; expected 0r[i]i.r but got 0ri[i].

hooray for good error messages :)
caused by HAL changes - there's only a bit of effort into making the compiler/runtime/executable ABI stable just yet so it's generally safe to assume that there will be issues if mixing versions until things settle down (I've got a few more PRs landing that have similar changes, but this was one of the first breakages in awhile) - I'd at least not build any non-debugging workflows around skewed versions :)

caused by HAL changes - there's only a bit of effort into making the compiler/runtime/executable ABI stable just yet so it's generally safe to assume that there will be issues if mixing versions until things settle down (I've got a few more PRs landing that have similar changes, but this was one of the first breakages in awhile) - I'd at least not build any non-debugging workflows around skewed versions :)

Yeah, it's fine for now. It just so happened that I was intentionally mixing with an old branch with some asan patches. Not something I endorse.

Ok, got asan working with python+iree+jax and it seems clean at head. I even scattered some random gc.collect() calls in (the usual way to smoke these things out).
Phoenix, if you need to do a local frankenbuild, here's what I've done:
# Build everything at head and install to a temp directory in release mode (good for getting all of the TF and compiler bits optimized while fiddling with the rest).
cmake -DIREE_BUILD_XLA_COMPILER=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/tmp/iree -DIREE_ENABLE_LLD=ON .

# This will create individual python packages under $HOME/tmp/iree/python_packages that you can manually add to your PYTHONPATH however you like.
ninja install

# Manually make a package for the non-installable jax frontend and symlink it from the source tree.
# We really should make this installable (I can help with that tomorrow once I find out whether Geoffrey broke my releases with the TF toolchain changes).
mkdir -p $HOME/tmp/iree/python_packages/jax/pyiree
ln -s $HOME/src/iree/bindings/python/pyiree/jax $HOME/tmp/iree/python_packages/jax/pyiree

Reconfigure to just build the runtime in Debug mode with ASAN.
cmake -DCMAKE_BUILD_TYPE=Debug -DIREE_BUILD_XLA_COMPILER=OFF -DIREE_BUILD_COMPILER=OFF -DIREE_BUILD_TESTS=OFF -DIREE_BUILD_SAMPLES=OFF -DIREE_BUILD_PYTHON_BINDINGS=ON
cmake -DIREE_ENABLE_ASAN=ON .
ninja
Create a franken PYTHONPATH with the bits you want from each location.
export PYTHONPATH=$HOME/tmp/iree/python_packages/iree_compiler:$HOME/tmp/iree/python_packages/iree_tools_xla:$HOME/tmp/iree/python_packages/jax:$(realpath $HOME/src/iree-build/bindings/python)
Run some jax+iree python program with ASAN injected.
You may need to root around on your file-system to find the asan runtime library that corresponds to your clang.
LD_PRELOAD=/usr/lib/llvm-11/lib/clang/11.0.1/lib/linux/libclang_rt.asan-x86_64.so python jax_mlp_crash.py

I don't necessarily expect you to be doing that, just documenting my process. A shortcut (albeit one that can cut you) is to just install the latest wheels with pip, then just configure IREE to build the runtime and only set your PYTHONPATH up to your local `iree-build/bindings/python` directory to get the parts that you want from a local build. As Ben says, though, there are still a couple more ABI breaking changes in queue, so there is some danger of version skew with this one.

Okay, I was able to rebuild and confirm that the issue seems resolved at head. Do you think it's reasonable to add the repro as a temporary test in bindings/python/pyiree/jax/e2e/?


Known Release Process Issues

I had to make the following changes manually when releasing v1.6:

Need to change name of initial release on vX.Y/releases/index.md
[Calico-CNI doesn't have latest builds, so build scripts don't correctly grep and replace them] #147
Other remaining steps that need to manually be done

At the very least, this needs to documented in README.md lest we forget


Increase Delay for BYFN CouchDB tests.

After Baseimage 0.4.11, the BYFN CouchDB tests are timing out with the default 3 second delay. This change is to increase the delay time to 15 seconds.

Thanks for the contribution! However, this is a read-only mirror. please read the contribution guide. Also, feel free to join us on chat!


Enhancing Blogs PAge UI

Is your feature request related to a problem? Please describe.
There is no hover effect on that page
Describe the solution you'd like
I will add hover effects and Framer Motion to it
Additional context
.
Kindly assign this issue to me under hactoberfest, gssoc-ext and relevant tags.

Something similar is already open on #58


HasuraRequestError: field 'types' not found in type: 'moment_bool_exp'

    Map map = await GqlPost.instance.hasuraConnect.query(""" query MyQuery {   moment(order_by: {created_at: asc}, limit: $_pageSize, offset: ${_dataList.length}, where: {types: { _eq: 2 }}) {     context     created_at     id     user     media   } } """);
The above query string is executed normally in the Hasura console
Using Hasura_ Connect error
HasuraRequestError: field 'types' not found in type: 'moment_bool_exp'

Forgot to set field permissions


#151948277 Accounts tests

What does this PR do?

Fixes existing tests in the accounts application and adds missing assertions in the tests.

Description of the task to be completed?

Go through each test file and follow the instructions to fix the tests. The instructions are distributed throughout each file and the instruction lines begin with ##

How should this manually be tested?

You can test this manually by issuing this command; python manage.py test hc.accounts.tests from the project root directory.

Pivotal Tracker story?

See [Chore #151948277]

Changes Unknown when pulling ac89737f156028891918b102b9b09dd2c4ae7691 on hc-accounts-tests into ** on master**.

You made pull request to wrong branch


GEODE-7875:  fix create index gfsh command on partitioned region

The new solution is added regarding comment from the dev list.
When creating an index on the partition region it should just send to one server as it will propagate the create index message to the rest of the members of that group(cluster or one the user specified in the command).
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

i like the idea of fixing this command for partition regions. The problem does exist that for partition region, the index creation is distributed to all the other servers, so subsequent CreateIndexFunction might get name conflict error. We can solve this by either of the two ways:

In the CreateIndexCommand, if user is creating the index on a partition region, only send the function to one member, not all members. Use GfshCommand.findAnyMembersForRegion method to get just one member to execute the function.
In the CreateIndexFunction,  when we get a NameConflictException or IndexExistsException, make the CliFunctionResult with a StatusState.IGNORABLE, this will make the command not consider this as a failure.

Either of these two approaches should work.

If we take the first approach, will the gfsh output only show that the index is created on one member or all members? If it only shows one member, it probably will give the user the wrong idea. Probably the 2nd approach is better.

If we take the first approach, will the gfsh output only show that the index is created on one member or all members? If it only shows one member, it probably will give the user the wrong idea. Probably the 2nd approach is better.

Yes, I agree with you that with first approach it will show one member, which will give the user the wrong idea.
I will go with the second option which will allow the command to pass and update the cluster config, and we will have an updated cluster config.

Hi @mhansonp, @kirklund,
can you please check the file you are codeowner?

@mkevo Hi Mario, Can you please take a look at GEODE-9089? It's been failing almost daily for over a year and it started failing right after merging PR #6008. Thanks!

Hi @kirklund,
On GEODE-9089 I saw that the latest failure was before 2 days on the support/1.14 branch. Also, the failure before was one year ago.
When I check the support/1.14 branch it doesn't have my fix as it will be delivered as a part of the 1.15.0 release.


getPage() methods for Parameters

I was wondering if there could be more page related methods in Parameters or at least making getPage() a public method. Here's something I'm looking for:
$page = $params->getPage('number');
$size = $params->getPage('size', $defaultSize);

would sound wrong to use different methods just for that, when everything else are methods on Parameters as well.

You can use the existing getOffset and getLimit methods to achieve what you want:
// input: ?page[number]=4&page[size]=10

$size = $params->getLimit() ?: $defaultSize; // 10
$offset = $params->getOffset($size); // 40
$page = $offset / $size; // 4

Nah. There are two concepts adressing pagination: offset/limit or page(number)/size. I'd go with the methods if they match my strategy but they don't. I subclassed your Parameters and made getPage public:
use Tobscure\JsonApi\Parameters as BaseParameters;
class Parameters extends BaseParameters {

	public function getPage($key, $default = null) {
		$page = $this->getInput('page', []);

		return isset($page[$key]) ? $page[$key] : $default;
	}
}

How does my code snippet not work for you though? limit is synonymous with size, and the correct page number can be derived from the calculated offset and size.

It's more of a mental thing. I'm looking for page number and size, instead I only have methods for limit and offset and need to do a calculation aftewards, while I now my parameters are page[number] and page[size] - that feels like doing it wrong. I want to use number for number and size for size and not limit for size or offset + formula for number.
I know these values are there and want to access them straight ahead, instead it says: "No please use this 'complex' formula instead".
Sorry, if this sounds harsh or something (it isn't). It is just for you to understand my position - hope you get it.


Limit max database connections to <= 20

After deploying a freshly generated bretels app, i got this error on Heroku:
PG::ConnectionBad: FATAL: remaining connection slots are reserved for non-replication superuser connections

Default puma settings in config/puma.rb allow for 3 workers (WEB_CONCURRENCY=3) with max 16 threads each (MAX_THREADS=16). In addition, MAX_THREADS is used to configure the connection pool size for the ActiveRecord database adapter (postgres, in our case):
on_worker_boot do
  # ...
  config['pool'] = ENV['MAX_THREADS'] || 16
 # ...
end

3 workers :heavy_multiplication_x: 16 connections = 48 connections.
Heroku's hobby-dev and hobby-basic postgres plans have a maximum of 20 connections.
I'm not 100% sure, but I think this maximum is causing the error message listed above.
I'm not sure how to fix this, maybe add a default .env file to bretels? Or include Heroku-specific connection limits in an app.json manifest and use this for configuring and deploying on Heroku?

Update: on Rails 4.2, setting the connection pool is best done in database.yml, even for Heroku.
See this dev center article for more information.
Probably best to just copy the default Puma config from there.

After searching some more: Heroku's suggested default puma config for Rails 4.2+


tagClassname, closeClassname

I tried to style tags with customTagRenderer but got errors when i tried to interact with react-hook-form using setValue as in documentation so i added tagClassname with closeClassname to props.

@virtueer  Thank you for your contribution! I appreciate your effort in trying to improve the project.
To better understand the issue you encountered with customTagRenderer and react-hook-form, could you please create an issue detailing the problem? It would be very helpful if you could include:

A detailed description of the issue.
Steps to reproduce the problem.
A video demonstrating the issue or a live reproduction of the problem.
Any relevant code snippets or examples that showcase the error.

This will help me replicate the issue on my end and understand the context of your PR better.

This issue has been resolved with the implementation in PR #59. The changes discussed have been incorporated


Do not use NeedMoreData to signal that a frame had been encoded

This avoids some confusion @dwbuiten reported some time ago. (yet another cherry-pick from the segment-threading set)
(crav1e will be updated accordingly)

Coverage remained the same at 83.226% when pulling 554cf402e41f4bc541306f4c9d7655070cf083b2 on rust-av:spurious-need-more-data into 0dc0c331f21288dfe586d957039b497b0f6d0fb4 on xiph:master.

Seems reasonable. The only other option I can think of (at 1am) is emitting a zero sized or null packet, but that would be uglier.


Does it work with Ubuntu?

yes, I have tried with Ubuntu default image in Azure DevOps.
But adding attachment does not work and you do not get emails at all via SendGrid. If you remove attachment , it start sending email!
Attachment Bug?


UrlHelper doesn't give access to a context

There is no way to get access to HttpContext in a UrlHelper extension, for instance to request some services. HtmlHelper however lets you access it.

Wait a sec - I see an IUrlHelper.ActionContext property: https://github.com/aspnet/Mvc/blob/dev/src/Microsoft.AspNetCore.Mvc.Core/IUrlHelper.cs#L13-L16
Isn't that enough? It's on the interface, so it's always "public", regardless of the concrete UrlHelper, though it also has a public property implementing it.

I see it has been added recently, this is why I couldn't use it (rc1).
Duplicate of
https://github.com/aspnet/Mvc/issues/3749

Ah, nice :smile:


FolderSink input property 'filter' seems to have a confusing description

File: themes/default/content/registry/packages/gcp/api-docs/logging/foldersink/_index.md
If I correctly understand the description of the string input property 'filter'

An advanced logs filter that matches the log entries to be excluded. By using the sample function, you can exclude less than 100% of the matching log entries. See Advanced Log Filters for information on how to write a filter.<

it would seem that any log entries NOT matched by this filter will be EXCLUDED from the destination. However, upon experimenting with the filter, it would seem that it is in fact an inclusion filter, meaning that any logs matched by the 'filter' input ARE then forwarded to the destination.
Please correct me if I misunderstood something, but if I have not, it would be worth changing the wording of the description.
Thanks!

Hi @chododom - thank you for filing this issue.
I'm going to move this to pulumi-gcp so we can take a look.
For any questions and help, please feel free to ask in our community slack.


never ever leave zombies lying around (using assassin.rb)

any feedback / thoughts on this @thibaudgg ?

Hey @ahoward, these zombies process defenitly need to be fixed but I'm wondering is doing it with an assassin script is the way to go. Is it really the standard way of killing zombies process?
@ttilley @e2 @andreyvit what do you think?
Having the specs broken on Ruby 2.3.1 is aslo annoying, sadly I don't have much time to fix them at the moment.

so so so hear you @thibaudgg but, here is the deal (i'd be over the moon if someone would correct me on being wrong)

you have 'library' code burried deep in 'rails server' - no one knows it is even running ;-/
the user, any gem, and app/ code, and process, can either invoke 'exit!' or send 'kill -9' to the front rails server process
if that happens, all the ruby code you can write via 'at_exit', etc.  cannot reap the child process spawned by rb-fsevent.  it.  will.  leak.

the really key part here is that any gem, and signal, can make you gem look bad ;-/  go unix.
my basic approach, as an idealist library writer, is to make the following assertion true:

i will create the resources i need

nothing you do will make me not clean up after them

i will say, in this specific situation that, clearly, some combination of pre-forking (spring loader) and client behaviour (redis calling exit!, etc.) triggers some really, really bizarre behavior where 100s, not just 1, process is leaked but, in defense of my paranoid library writer mentality i will say "who cares"....  the invariate outlined above, which is only possible in unix by stepping sideways outside the process-tree-signal-hierarchy, is a methodology that "works when people fuck up" and "does no harm if everyone plays fair".  in truth, i think every ruby libraray that spawns a process should have a similar strategy.
finally, i welcome being proved wrong on this because it feels ugly.  but.  libary code often is ;-/

I agree that the solution is conceptually inelegant (the code itself is fine).  The correct solution is, to fix the library, of course. However, this is a real issue, and failing that option, something is needed.
On my laptop (OS X 10.12) this morning:
% ps augxww | grep '[r]b-fsevent' | wc -l
      93

I suspect these were left behind by the sqlite3 gem core dumping (fixed in 1.3.12), another path to at_exit being missed.

the root cause lies somewhere here:
~ $ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]
~ $ rails -v
Rails <IP_ADDRESS>
~ $ cd /tmp
tmp $ rm -rf leak; rails new leak >/dev/null 2>&1
tmp $ cd leak/
leak $ ps augxww | grep '[r]b-fsevent' | wc -l
       0
leak $ ./bin/rails runner :leak
Running via Spring preloader in process 15275
leak $ ps augxww | grep '[r]b-fsevent' | wc -l
      20
leak $ ./bin/rails runner :leak
Running via Spring preloader in process 15820
leak $ ps augxww | grep '[r]b-fsevent' | wc -l
      20
leak $ ./bin/spring stop
Spring stopped.
leak $ ps augxww | grep '[r]b-fsevent' | wc -l
       0
leak $ DISABLE_SPRING=42 ./bin/rails runner :leak
leak $ ps augxww | grep '[r]b-fsevent' | wc -l
       0
leak $ exit

I believe if you added proper error handling on the fprintf(stdout invocations in fsevent_watch/main.c, you'd be able to catch the EPIPE that signals the parent process at the other end of the pipe has disappeared, and use that to exit(0).

agree.  the only thing that is troubling me is understanding why on earth spring causes 20 of those to fire up ;-/  surely that affects more than just rb-fsevent....

This is like mongrel all over again.... fork: Resource temporarily unavailable :(

@ahoward what do you think of the #62 approach? Seems nicer to me.

i do like this.   i'd merge it if i were you.  making the child never outlive the parent is needed.  having said that no one is understanding why spring makes 20 of the damn things start and that needs looked at too.  if i were you i'd merge #62 and then, if that still wasn't enough ,this.  it is really, really, really bad.  at work we have entirely stopped using puma (/cc @evanphx) not because of anything wrong with it - but because puma + rails5 hits rb-fsevent and, in the course of a day, causes multiple laptops to freeze ;-/

Version 0.9.8 released with @steakknife patch. Let me know if it fixes the issue for you guys. Thanks.

+1

Sorry, I'm crushed with work. Don't have OSX, either.
For OSX though, I'd suggest a client-server model. Like in Facebook's
Watchman listener. That way these processes would be managed by a
server that's auto-started and never dies (but would simply ignore
dead clients).
The burke/zeus project might have this solved already (though last
tiem I checked they use Go for both the server and the listeners).
Listen needs an overhaul to help with stuff like this (like
implementing the client-server model without much hassle), but again
... I'm crushed with work... sorry ;(
On Thu, Oct 20, 2016 at 12:12:55AM -0700, Thibaud Guillaume-Gentil wrote:

Hey @ahoward, these zombies process defenitly need to be fixed but I'm wondering is doing it with an assassin script is the way to go. Is it really the standard way of killing zombies process?
@ttilley @e2 @andreyvit what do you think?
Having the specs broken on Ruby 2.3.1 is aslo annoying, sadly I don't have much time to fix them at the moment.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/thibaudgg/rb-fsevent/pull/61#issuecomment-255027125

Yup. Spawning individual servers is nuts.
On Tue, Oct 25, 2016 at 11:54 AM Cezary Baginski<EMAIL_ADDRESS>wrote:

Sorry, I'm crushed with work. Don't have OSX, either.
For OSX though, I'd suggest a client-server model. Like in Facebook's
Watchman listener. That way these processes would be managed by a
server that's auto-started and never dies (but would simply ignore
dead clients).
The burke/zeus project might have this solved already (though last
tiem I checked they use Go for both the server and the listeners).
Listen needs an overhaul to help with stuff like this (like
implementing the client-server model without much hassle), but again
... I'm crushed with work... sorry ;(
On Thu, Oct 20, 2016 at 12:12:55AM -0700, Thibaud Guillaume-Gentil wrote:

Hey @ahoward, these zombies process defenitly need to be fixed but I'm
wondering is doing it with an assassin script is the way to go. Is it
really the standard way of killing zombies process?
@ttilley @e2 @andreyvit what do you think?
Having the specs broken on Ruby 2.3.1 is aslo annoying, sadly I don't
have much time to fix them at the moment.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/thibaudgg/rb-fsevent/pull/61#issuecomment-255027125

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thibaudgg/rb-fsevent/pull/61#issuecomment-256139247,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANsMAtvBEFltG4gYcRbBEyGE0ilWGCMks5q3lBSgaJpZM4KY4uu
.

It should be pretty straightforward to get rid of all these fragile, support-intensive Listen adapters and just use watchman.  No more mongrel fork: resource exhausted, less duplicated/non-value-add effort.
https://github.com/facebook/watchman/tree/master/ruby/ruby-watchman

@thibaudgg I am on 0.9.8 and still getting this on Mac

please try again with 0.10.1

bundle update rb-fsevent brought me up to rb-fsevent (0.10.2) and seemed to get the number of fsevent_watch processes back down to a reasonable level.


Sharing the weights for the corresponding layers of two models

Hello everyone,
I am trying to implement an architecture with two similar models for feature extraction. I need to extract parameters of the corresponding layers of the two models and minimize their norm difference between. Any Ideas how this could be done in Keras?

@AmirHussein96 You could use keras functional API. There are lot of resources on Stackoverflow and other places addressing similar architecture. Check Examples on Keras website.
Hi!
This issue isn't related to a bug/enhancement/feature request or other accepted types of issue.
To ask questions, please see the following resources :
StackOverflow
Keras Slack channel
Keras.io support
Gitter
Thanks!
If you think we made a mistake, please open another issue and fill all the information asked by the template. Thanks!

Closing this as it is better asked in other resources. Thanks!


add simclr config

add simclr clas config

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Figure out nativescript/pluginscript/singleton/what-have-you

See https://godotengine.org/article/look-gdnative-architecture
There's different type of things you can do with gdnative and each of them should be exposed in a simple and understandable way in these bindings.
Looks like we currently make it easy to write native scripts, but not necessarily the rest.

Native scripts are implemented.
Singletons are tracked in https://github.com/godot-rust/godot-rust/issues/338.
Plugin scripts are currently not supported; in case of real demand, we can reopen this or a new issue.


Groups object are encoded with numbers - how do I display the group names

Hello,
When I ran this command, I get great result however the "groups" column is showing only numbers.  it is kind of difficult to replace those value because I need to lookup the group names and sometimes groups value has 2 or more groups on it.
how do i replace that value/s with valid group name/s?
$site = siteurl   #fixed variable
cmdlet:
get-pnpuser -WithRightsAssigned | select title, groups, email, $site
sample output:
Title                            Groups             Email                                https://domain/sites/siteurl

IO.Messaging.APA     {610, 27}<EMAIL_ADDRESS>    ---
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 57ec7ac7-3858-172e-ec9c-ab7c948ae2cf
Version Independent ID: 3a171d0d-f00d-36fd-44c3-c6bb08222b14
Content: Get-PnPUser (sharepoint-pnp)
Content Source: sharepoint/sharepoint-ps/sharepoint-pnp/Get-PnPUser.md
GitHub Login: @techwriter40
Microsoft Alias: kirks

@officedocsbot assign @yogkumgit

Hello @plagooms
Thanks for your feedback.
Actually this resource is intended to address issues in documentation, besides that, I recommend that you save the result to a variable and iterate the Groups member that the User class has, I encourage you to read the Sharepoint CSOM referenced documentation to better understand how to handle the results.
Hope this is useful to you.
Thank you

@get-itips  Thank you very much for the contribution and sharing this explanation. @plagooms
Hope this comment is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.

Thank you Andres.
For some reason, when I store the output to a variable and try  a loop, the value for column "Groups" is like the attached.
Any ideas?
get-pnpuser -WithRightsAssigned | select Title, Email, Groups, $siteurl | Export-Csv -path c:\temp\test.csv -Append -NoTypeInformation
Sent from Outlookhttp://aka.ms/weboutlook

From: Andres Mariano Gorzelany<EMAIL_ADDRESS>Sent: Thursday, 20 June 2019 2:01 AM
To: MicrosoftDocs/office-docs-powershell
Cc: plagooms; Mention
Subject: Re: [MicrosoftDocs/office-docs-powershell] Groups object are encoded with numbers - how do I display the group names (#3551)
Hello @plagoomshttps://github.com/plagooms
Thanks for your feedback.
Actually this resource is intended to address issues in documentation, besides that, I recommend that you save the result to a variable and iterate the Groups member that the Userhttps://docs.microsoft.com/en-us/previous-versions/office/sharepoint-csom/ee542337(v%3Doffice.15) class has, I encourage you to read the Sharepoint CSOM referenced documentation to better understand how to handle the results.
Hope this is useful to you.
Thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/office-docs-powershell/issues/3551?email_source=notifications&email_token=AKVXMR3QMUFOY5WCPYODQHTP3JYBPA5CNFSM4HXX3Y22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYCVWCA#issuecomment-503667464, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKVXMR5LQZ62F6PF3UMN4QTP3JYBPANCNFSM4HXX3Y2Q.

This is how the output looks like in console...the "Groups" column has values
get-pnpuser -WithRightsAssigned | select Title, Email, Groups, $siteurl
Title                            Email                                Groups             https://hostname.domain/sites/sitename

user1<EMAIL_ADDRESS>     {}
user2<EMAIL_ADDRESS>       {36}
Sent from Outlookhttp://aka.ms/weboutlook

From: Dan Y<EMAIL_ADDRESS>Sent: Saturday, 22 June 2019 6:53 AM
To: MicrosoftDocs/office-docs-powershell; MicrosoftDocs/office-docs-powershell
Cc: Mention
Subject: Re: [MicrosoftDocs/office-docs-powershell] Groups object are encoded with numbers - how do I display the group names (#3551)
Thank you Andres.
For some reason, when I store the output to a variable and try  a loop, the value for column "Groups" is like the attached.
Any ideas?
get-pnpuser -WithRightsAssigned | select Title, Email, Groups, $siteurl | Export-Csv -path c:\temp\test.csv -Append -NoTypeInformation
Sent from Outlookhttp://aka.ms/weboutlook

From: Andres Mariano Gorzelany<EMAIL_ADDRESS>Sent: Thursday, 20 June 2019 2:01 AM
To: MicrosoftDocs/office-docs-powershell
Cc: plagooms; Mention
Subject: Re: [MicrosoftDocs/office-docs-powershell] Groups object are encoded with numbers - how do I display the group names (#3551)
Hello @plagoomshttps://github.com/plagooms
Thanks for your feedback.
Actually this resource is intended to address issues in documentation, besides that, I recommend that you save the result to a variable and iterate the Groups member that the Userhttps://docs.microsoft.com/en-us/previous-versions/office/sharepoint-csom/ee542337(v%3Doffice.15) class has, I encourage you to read the Sharepoint CSOM referenced documentation to better understand how to handle the results.
Hope this is useful to you.
Thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/office-docs-powershell/issues/3551?email_source=notifications&email_token=AKVXMR3QMUFOY5WCPYODQHTP3JYBPA5CNFSM4HXX3Y22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYCVWCA#issuecomment-503667464, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKVXMR5LQZ62F6PF3UMN4QTP3JYBPANCNFSM4HXX3Y2Q.

Hello @plagooms
I kindly suggest you to visit this resource to ask those questions, as you might know, PnP cmdlets are community-driven, so there is a big community that will help you in your scenario.
Other than that, the article does not need an update so we must keep this closed.

@officedocsbot close


Replace embObjInertial w/ embObjIMU – Stint 2

This is the follow-up of #420
Since now inertials3 service and embObjIMU covers all the functionalities of inertials2 and embObjInertials respectively we can:

[x] Remove all inertials2 references in icub-firmware
[x] Remove all inertials2 references in icub-firmware-shared
[x] Remove embObjInertial from icub-main
[x] Test inertial3 service
[ ] Test FT and STRAIN services

PRs

https://github.com/robotology/icub-main/pull/916
https://github.com/robotology/robotology-superbuild/pull/1545
https://github.com/robotology/icub-firmware/pull/442
https://github.com/robotology/icub-firmware-shared/pull/90

cc @marcoaccame @pattacini

I started from the easiest part, i.e. the icub-main part:

https://github.com/robotology/icub-main/pull/916
https://github.com/robotology/robotology-superbuild/pull/1545

I tried to test all the above changes running yarprobotinterface on my setup w/ an ems, an rfe and mtb boards, the yarprobotinterface starts fine but either from the can boards and mems I get this data published:
((0.0 0.0 0.0) -9.2559631349317831e+61)

It means that something broke removing the inertials2 structures
cc @marcoaccame

After a f2f alignment w/ @marcoaccame, it came out that instead of simply removing the enums not used anymore, we need to remove them and then shift the others in order to maintain them ordered.
Moreover, I had to change the CanMapping accordingly, and after these two commits:

https://github.com/robotology/icub-firmware-shared/pull/90/commits/fd7ef41d9c5be8b8f76b7d8a4a00b5367cdccce7
https://github.com/robotology/icub-firmware/pull/442/commits/25080d3e4b17b79188daf037ca6c8ffc1d689c19

Now I can instantiate correctly the inertials3 service and read both from legacy and "new" imus.
(((-0.125 -0.1875 -0.0625)<PHONE_NUMBER>.18483) ((-63.0 -3.0 6.0)<PHONE_NUMBER>.17504)) (((0.040000000000000001 1.3300000000000001 9.3900000000000006)<PHONE_NUMBER>.1848295) ((0.0 0.0 0.0)<PHONE_NUMBER>.2099471)) (((2.3875000000000001e-05 2.1875e-05 1.5562499999999999e-05)<PHONE_NUMBER>.1848297)) (((8.1875 0.0 -5.375)<PHONE_NUMBER>.1848302)) () () () () () ()

In order to be sure that these changes does not break other services, I will try to run:

The THERMO service on mtb4
The FT service on strain2

I am still not 100% convinced about this implementation:
extern uint8_t eocanmap_posOfEPEN(eOprotEndpoint_t ep, eOprotEntity_t en)
{
    enum { // keep in magic numbers. it is a check if any of the eoprot_xxx value changes
        s0 = 4, // eoprot_endpoints_numberof
        s1 = 8  // eoprot_entities_maxnumberofsupported
    };

    static const uint8_t pos[s0][s1] = // pos[eoprot_endpoints_numberof][eoprot_entities_maxnumberofsupported] =
    {
        {0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf , 0xf},     // ep->management [none, ...]
        {0,     1, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf , 0xf},     // ep->mc [joi, mot, none, ...]
        {2,     3,   4,   5,   6,   7,   8,   9 , 0xf},     // ep->as [str, mai, tem, ine3, psc, pos, ft, bs]
        {10,  0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf , 0xf}      // ep->sk [sk, none]
    }; EO_VERIFYsizeof(pos, sizeof(const uint8_t)*(eoprot_endpoints_numberof)*(eoprot_entities_maxnumberofsupported))

    // the order is joint-motor-strain-mais-temperature-inertial-inertial3-psc-pos-ft-bs-skin
    if((ep >= s0) || (en >= (s1)))
    {
        return 0x0f;
    }
    return pos[ep][en];
}

if fact I am getting these warning from the compiler
compiling EoCANprotSKperiodic.c...
../../../../../embobj/plus/can/EOtheCANmapping.c(774): warning: excess elements in array initializer [-Wexcess-initializers]
        {0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf , 0xf},     // ep->management [none, ...]
                                                  ^~~
../../../../../embobj/plus/can/EOtheCANmapping.c(775): warning: excess elements in array initializer [-Wexcess-initializers]
        {0,     1, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf , 0xf},     // ep->mc [joi, mot, none, ...]
                                                  ^~~
../../../../../embobj/plus/can/EOtheCANmapping.c(776): warning: excess elements in array initializer [-Wexcess-initializers]
        {2,     3,   4,   5,   6,   7,   8,   9 , 0xf},     // ep->as [str, mai, tem, ine3, psc, pos, ft, bs]
                                                  ^~~
../../../../../embobj/plus/can/EOtheCANmapping.c(777): warning: excess elements in array initializer [-Wexcess-initializers]
        {10,  0xf, 0xf, 0xf, 0xf, 0xf, 0xf, 0xf , 0xf}      // ep->sk [sk, none]

I am still not 100% convinced about this implementation:

In fact since now the map is a 4x8 not a 4x9 I removed the last column and compiles without warnings and works fine:

https://github.com/robotology/icub-firmware/pull/442/commits/fb3134a47f00b3766380deb424cddd16e3b86c10

I tried to validate my changes w/ the following setup:

ems
strain2
rfe

I ran the inertials3 service for getting the imu data from these 3 boards and alternatively the strain(legacy) and ft service for getting the ft data.
Here the list of ports created:
registration name /icub/left_hand/FT/measures:o ip <IP_ADDRESS> port 10009 type tcp
registration name /icub/left_hand/FT/rpc:o ip <IP_ADDRESS> port 10010 type tcp
registration name /icub/left_hand/inertialFT/measures:o ip <IP_ADDRESS> port 10007 type tcp
registration name /icub/left_hand/inertialFT/rpc:o ip <IP_ADDRESS> port 10008 type tcp
registration name /icub/left_hand/inertialMTB/measures:o ip <IP_ADDRESS> port 10003 type tcp
registration name /icub/left_hand/inertialMTB/rpc:o ip <IP_ADDRESS> port 10004 type tcp

And here is what I am reading:
✅ /icub/left_hand/FT -> FT + temperature (service strain(legacy))
() () () () (((31.199999999999999)<PHONE_NUMBER>.5865707)) (((-1.75634765625 -1.54022216796875 -5.8397216796875 -0.059844970703125 -0.015380859375 0.005218505859375)<PHONE_NUMBER>.9764657)) () () () ()

❌ /icub/left_hand/FT -> FT + temperature (service ft)
[ERROR] ServiceParser::check_analog() has found wrong SERVICE.type =  16 it is  eomn_serv_AS_ft it must be   TODO : tostring() function

We are configuring eomn_serv_AS_ft that is 16 but this check is failing:

https://github.com/robotology/icub-main/blob/23f59fe284d3cbdf526cf2ca43d5dd6948c06547/src/libraries/icubmod/embObjLib/serviceParser.cpp#L591-L595
Because type is strangely eomn_serv_AS_strain

❌ /icub/left_hand/inertialFT -> imu of the strain2
(((0.0 0.0 0.0) -9.2559631349317831e+61)) (((0.0 0.0 0.0) -9.2559631349317831e+61)) (((0.0 0.0 0.0) -9.2559631349317831e+61)) (((0.0 0.0 0.0) -9.2559631349317831e+61)) () () () () () ()

❌ /icub/left_hand/inertialMTB -> imu of rfe + gyro of the ems.
(((0.0 0.0 0.0) -9.2559631349317831e+61) ((0.0 0.0 0.0) -9.2559631349317831e+61)) (((0.0 0.0 0.0) -9.2559631349317831e+61)) (((0.0 0.0 0.0) -9.2559631349317831e+61)) (((0.0 0.0 0.0) -9.2559631349317831e+61)) () () () () () ()

The strange thing is that both the inertials3 launched on strain2 and ems/rfe standalone works, but when it is launched with another inertials3 service prints all zeros.
So the open problems are:

Make run two inertials3 services togheter
make run the eomn_serv_AS_ft service.

I am using the xml files in this branch

https://github.com/Nicogene/robots-configuration/tree/testemsmtb/experimentalSetups/ems_mtb

cc @marcoaccame @pattacini

❌ /icub/left_hand/FT -> FT + temperature (service ft)

I managed to make ti work replacing embObjFTsensor w/ embObjMultipleFTsensors, in fact now I am getting data from the strain2:
() () () () (((31.800000000000001)<PHONE_NUMBER>.6826351)) (((-2.41497802734375 0.176025390625 5.8397216796875 -0.0194091796875 0.028839111328125 0.007537841796875)<PHONE_NUMBER>.0)) () () () ()

This means that embObjFTsensor device works only with  eomn_serv_AS_strain service @marcoaccame ?

So the open problems are:

Make run two inertials3 services togheter

After a f2f chat w/ @marcoaccame it came out that both are not actually problems. For each eth board can run only one service per type, then for running the imu of the ft sensor I have to add colums to the same xml, not spawn another embObjIMU

This means that embObjFTsensor device works only with eomn_serv_AS_strain service @marcoaccame ?

Looking here:

https://github.com/robotology/icub-main/blob/23f59fe284d3cbdf526cf2ca43d5dd6948c06547/src/libraries/icubmod/embObjLib/serviceParser.cpp#L558-L563

It seems that embObjFTsensor  supports only eomn_serv_AS_strain, on the other hand `` is used only by embObjMultipleFTsensors, see:

https://github.com/search?q=repo%3Arobotology%2Ficub-main eomn_serv_AS_ft &type=code

Tomorrow I will do a double check and then I would say we are good to go.
cc @pattacini

The last test passed without problems 🎉
I run the Inertials3 service asking all this IMU data:
 on_eb2_gyros      l_leg_ft_acc      l_leg_ft_gyro   l_leg_ft_eul      l_leg_ft_mag    l_leg_ft_stat   rfeimu_acc    rfeimu_gyro     rfeimu_mag      rfeimu_eul      rfeimu_status

It is coming from ems, strain2, rfe and I am gettig this data:
(((-0.125 0.0 0.0)<PHONE_NUMBER>.7739019) ((0.25 0.125 -0.125)<PHONE_NUMBER>.7890337) ((-37.0 21.0 -140.0)<PHONE_NUMBER>.7790971)) (((0.51000000000000001 0.080000000000000002 -9.4399999999999995)<PHONE_NUMBER>.7739012) ((-1.3300000000000001 -0.60999999999999999 -9.6899999999999995)<PHONE_NUMBER>.7890329)) (((-2.4999999999999999e-07 -3.0750000000000002e-05 0.0001206875)<PHONE_NUMBER>.7739017) ((-6.7999999999999999e-05 -4.0624999999999996e-06 7.875e-06)<PHONE_NUMBER>.7890334)) (((178.9375 2.5625 -212.8125)<PHONE_NUMBER>.7739024) ((-176.4375 7.875 -280.5625)<PHONE_NUMBER>.7890339)) () () () () () ()

Then I tried both the eomn_serv_AS_strain and eomn_serv_AS_strain and they works just fine:
() () () () (((30.300000000000001)<PHONE_NUMBER>.2537851)) (((0.109771728515625 -0.9681396484375 4.910675048828125 -0.02587890625 -0.036529541015625 0.04522705078125)<PHONE_NUMBER>.59881)) () () () ()

I will proceed then opening the PR containing the new FW binaries
cc @marcoaccame

The three PRs on firmware can be merged, I will then make as ready for review the one in icub-main
cc @pattacini @marcoaccame


generateSummaryForHTML missing trailing dots when link tag is truncated

Describe the bug
The generateSummaryForHTML helper function is used to generate a summary for conversations on the collective page. When generateSummaryForHTML is used on the content, it adds a three trailing dots to the end of the summary indicating if part of the original content was left out of the summary. It looks like there's a bug when the truncated text happens to be a link tag (this probably happens for other tags accepted by sanitizeHTML) .
To Reproduce
Steps to reproduce the behavior:

Go to any collective page
Click on Conversations
Click on Create Conversation
Add a title and paste the text below, you will have to replace the links in the text.

This week we had a choice of all sorts of news, but as we cut the show together as a group [Danny](https://twitter.com/DannyCrichton) pushed all the funding rounds up. So, when [Alex](https://twitter.com/Alex) and [Natasha](https://twitter.com/nmasc_) jumped into the show we had a bunch of good news to cover. We’re avoiding COVID-19 news,

You can view the example at https://staging.opencollective.com/engineering/conversations
Expected behavior
I expected the summary to end with trailing dots to indicate more text to read.
Screenshots

Actual

Expected to be similar to this with trailing dots at the end

Good catch @bolariin, thanks for the detailed issue

I'll give it try at this one if it's not being worked on. :smile:

Done. This seems to happen because we do the second sanitization pass as the last step, overriding the truncation step and removing the .... I've corrected it. :smile:


Possible way to replicate like Fundraisup Donate Widget?

Hey there?
Love your Wizard thing. Just amazing! Is there any possibility to mold your wizard like this listed below:
https://fundraiseup.com/#demo-donate
One step at a time? Or may be any quick fix? Otherwise, It will take time to fix it for me
Thanks

Hi, @xcelpixel! Thank you for using our products. This I think it is possible but it takes some time to implement this because it's not quite simple. Regards, Dragos


Module cannot be downloaded (ez fix)

Hi again,
A couple of users reported that they couldn't download the module, with the error
An unexpected error occurred when downloading file https://github.com/bmarian/sadness-chan/releases/download/1.1.9/sadness-chan-v1.1.9.zip: Not Found
Upon further inspection I've seen I've used a different versioning tag (1.1.9 instead of v1.1.9)
So to fix this issue all that's needed is changing the download url in the module.json to https://github.com/bmarian/sadness-chan/releases/download/v1.1.9/sadness-chan-v1.1.9.zip
(notice the v1.1.9)
Sorry for the inconvenience and thanks again for your work!

My bad 😄

Thanks for the quick fix 🙏


can't get property bind in ui-radio

use the code in readme,   the fruit property is always undefined

      {{ui-radio name="fruit" label="Once a week" checked=fruit}}

      {{ui-radio name="fruit" label="2-3 times a week" checked=fruit}}

      {{ui-radio name="fruit" label="Once a day" checked=fruit}}

It appears the readme was wrong. I have updated the example on it. The checkbox does use checked, but the radio needs two fields to compare against (value and current). Clicking a radio sets current to the value. There is also an action you can subscribe too if you need one on check.
Here is the new example:

      {{ui-radio name="fruit" label="Once a week" value="apple" current=fruit}}

      {{ui-radio name="fruit" label="2-3 times a week" value="orange" current=fruit}}

      {{ui-radio name="fruit" label="Once a day" value="grape" current=fruit}}

it works , thank you


Small fix for exc. 5.4

In exc. 5.4 we should have m=-154, not -153.
Thanks @alptoksoz for notifying.

Fixed in #17