Pretrain & Finetune Dataset like Webvid, COCO, GQA, etc.

Hello, thanks for your great work. I would like to inquire about the datasets used during the finetuning phase of your model. Specifically, I'm interested in knowing if datasets like COCO, GQA, OCR_VQA, TextVQA, and VG were utilized. My confusion arises from the fact that these datasets are not mentioned in the Preparation Dataset section, yet they appear in the data/LLaMA-VID-Finetune/ directory. Could you please provide some clarification on this?
Additionally, if these datasets were indeed used for finetuning, could you offer some guidance on how to download and format them? 🥹

Hi, thanks for your interest in our work. For the instruction fine-tuning, we adopt the image-based dataset in LLaVA, you can download and organize them according to the Visual Instruction Tuning section in README.

Thank you for your response. 👍
I have another question and would appreciate your further explanation. Could you please clarify the structure of the data/LLaMA-VID-Pretrain/images and data/LLaMA-VID-Pretrain/videos directories? The Preparation section seems to lack this information, and specifically, I have 2 points of confusion:

Regarding the data/LLaMA-VID-Pretrain/images directory, I downloaded images from LLaVA-pretrain-images.zip, but I noticed that in llava_558k_with_webvid.json, one of the data entries lists the image path as 00037/000378145.jpg. How is this path structured?
Regarding the data/LLaMA-VID-Pretrain/videos directory, I used the script from webvid_download_script to download the 2.5M subset of videos from webvid. However, I noticed that in llava_558k_with_webvid.json, one of the data entries lists the video path as 000351_000400/3827210.mp4. How is this path determined?

Thank you for your response. 👍 I have another question and would appreciate your further explanation. Could you please clarify the structure of the data/LLaMA-VID-Pretrain/images and data/LLaMA-VID-Pretrain/videos directories? The Preparation section seems to lack this information, and specifically, I have 2 points of confusion:

Regarding the data/LLaMA-VID-Pretrain/images directory, I downloaded images from LLaVA-CC3M-Pretrain-images.zip, but I noticed that in llava_558k_with_webvid.json, one of the data entries lists the image path as 00037/000378145.jpg. How is this path structured?
Regarding the data/LLaMA-VID-Pretrain/videos directory, I used the script from webvid_download_script to download the 2.5M subset of videos from webvid. However, I noticed that in llava_558k_with_webvid.json, one of the data entries lists the video path as 000351_000400/3827210.mp4. How is this path determined?

I guess the images under data/LLaMA-VID-Pretrain/images should be downloaded from LLaVA-Pretrain instead of LLaVA-CC3M-Pretrian? 🤥

Yes, for images used in LLaMA-VID pretrain, you need to download images from LLaVA-Pretrain and put all the images under LLaMA-VID-Pretrain/images. For the video, download the webvid and put all the subsets in downloaded WebVid/data_2m/videos to LLaMA-VID-Pretrain/videos. The path in the json like 000351_000400/3827210.mp4 denotes the relative path to data/LLaMA-VID-Pretrain/videos.

Thank you for your response.
I used this script to download videos from webvid, and all the videos were directly placed in the 'LLaMA-VID-Pretrain/videos' directory without parent directories like 000351_000400/, so I modified the script a bit:
csvcut -c videoid,contentUrl data/pretrain/LLaMA-VID/results_2M_train.csv \
  | sed 1d \
  | parallel --resume -v --joblog val.log --bar -j 126 'video_id=$(echo {} | cut -d , -f 1) && url=$(echo {} | cut -d , -f 2) && extension=${url##*.} && folder=$((video_id / 10000 / 50)) && start=$((folder * 50 + 1)) && end=$(( (folder + 1) * 50 )) && folder_name=$(printf "%06d_%06d" $start $end) && mkdir -p "data/pretrain/LLaMA-VID/videos/$folder_name" && wget --no-clobber -qO "data/pretrain/LLaMA-VID/videos/$folder_name/$video_id.$extension" $url'

I'm sharing the modified script here, so if anyone needs it, they can use it directly to download video from webvid.

is there any way to finetune only activityNet(video)?
I want to jump to 2stage and FineTune on activitiyNet.
bash scripts/video/train/stage_2_full_v7b_224_fps_1.sh
I followed the scripts but I got this messages
Error in loading 12359, retrying...
Error in loading 30526, retrying...
Error in loading 86049, retrying...
Error in loading 38349, retrying...
Error in loading 47934, retrying...
Error in loading 14090, retrying...
and also confused about this preprocess function
preprocess_imgsp_v1 .
what should i change to finetune on activitiynet?

Hi, if you want to finetune the video only, you can directly delete other data that not contain the key video list in this file and only keep the video QA (actually in loaded_json[-98138:]).
But it should be noted that only fine-tune video will cause performance in image and multi-turn conversation.
And the preprocess_imgsp_v1  aims to pad image token <image> to each question in multi-turn conversation. Because we generate different context tokens for each question, it is essential to append different visual tokens before answering questions.

Hi, if you want to finetune the video only, you can directly delete other data that not contain the key video list in this file and only keep the video QA (actually in loaded_json[-98138:]). But it should be noted that only fine-tune video will cause performance in image and multi-turn conversation. And the preprocess_imgsp_v1  aims to pad image token <image> to each question in multi-turn conversation. Because we generate different context tokens for each question, it is essential to append different visual tokens before answering questions.

hi, @yanwei-li , thanks. I also only want to finetunning on subset of webvid.

So my question is, how to finetunning on videos only dataset? is anything goes wrong? thank you.

Hi, it seems the error is caused from the format of your produced data. Please make sure the format of video data aligns with that in llava_558k_with_webvid.json.

is there any way to finetune only activityNet(video)? I want to jump to 2stage and FineTune on activitiyNet.
bash scripts/video/train/stage_2_full_v7b_224_fps_1.sh
I followed the scripts but I got this messages
Error in loading 14090, retrying... Error in loading 31069, retrying...train() Error in loading 76494, retrying... Error in loading 30647, retrying...
File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 1204, in train trainer.train() File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train return inner_training_loop( File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1787, in _inner_training_loop for step, inputs in enumerate(epoch_iterator): File "/opt/conda/lib/python3.10/site-packages/accelerate/data_loader.py", line 384, in iter current_batch = next(dataloader_iter) File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in next data = self._next_data() File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data return self._process_data(data) File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data data.reraise() File "/opt/conda/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise raise exception KeyError: Caught KeyError in DataLoader worker process 0. Original Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop data = fetcher.fetch(index) File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 933, in getitem data_dict = preprocess( File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 772, in preprocess return preprocess_imgsp_v1(sources, tokenizer, has_image=has_image, refine_prompt=refine_prompt) File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 542, in preprocess_imgsp_v1 if roles[source[0]["from"]] != conv.roles[0]: KeyError: 0
and also confused about this preprocess function preprocess_imgsp_v1 . what should i change to finetune on activitiynet?

Hello, have you solved this problem?
I also encountered this error when trying to fine-tune on video and image data：
Original Traceback (most recent call last):
File "/home/kas/.conda/envs/llamavid/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/kas/.conda/envs/llamavid/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/kas/.conda/envs/llamavid/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/kas/kas_workspace/shuozhang/LLaMA-VID/llamavid/train/train.py", line 925, in getitem
data_dict = preprocess(
File "/home/kas/kas_workspace/shuozhang/LLaMA-VID/llamavid/train/train.py", line 774, in preprocess
return preprocess_imgsp_v1(sources, tokenizer, has_image=has_image, refine_prompt=refine_prompt)
File "/home/kas/kas_workspace/shuozhang/LLaMA-VID/llamavid/train/train.py", line 544, in preprocess_imgsp_v1
if roles[source[0]["from"]] != conv.roles[0]:
KeyError: 0

is there any way to finetune only activityNet(video)? I want to jump to 2stage and FineTune on activitiyNet.
bash scripts/video/train/stage_2_full_v7b_224_fps_1.sh
I followed the scripts but I got this messages
Error in loading 14090, retrying... Error in loading 31069, retrying...train() Error in loading 76494, retrying... Error in loading 30647, retrying...
File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 1204, in train trainer.train() File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train return inner_training_loop( File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1787, in _inner_training_loop for step, inputs in enumerate(epoch_iterator): File "/opt/conda/lib/python3.10/site-packages/accelerate/data_loader.py", line 384, in iter current_batch = next(dataloader_iter) File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in next data = self._next_data() File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data return self._process_data(data) File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data data.reraise() File "/opt/conda/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise raise exception KeyError: Caught KeyError in DataLoader worker process 0. Original Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop data = fetcher.fetch(index) File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 933, in getitem data_dict = preprocess( File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 772, in preprocess return preprocess_imgsp_v1(sources, tokenizer, has_image=has_image, refine_prompt=refine_prompt) File "/workspace/mount/SSD_11T/minwoo/projects/LLM/LLaMA-VID/llamavid/train/train.py", line 542, in preprocess_imgsp_v1 if roles[source[0]["from"]] != conv.roles[0]: KeyError: 0
and also confused about this preprocess function preprocess_imgsp_v1 . what should i change to finetune on activitiynet?

Hello, have you solved this problem? I also encountered this error when trying to fine-tune on video and image data：
Original Traceback (most recent call last): File "/home/kas/.conda/envs/llamavid/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop data = fetcher.fetch(index) File "/home/kas/.conda/envs/llamavid/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/kas/.conda/envs/llamavid/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/kas/kas_workspace/shuozhang/LLaMA-VID/llamavid/train/train.py", line 925, in getitem data_dict = preprocess( File "/home/kas/kas_workspace/shuozhang/LLaMA-VID/llamavid/train/train.py", line 774, in preprocess return preprocess_imgsp_v1(sources, tokenizer, has_image=has_image, refine_prompt=refine_prompt) File "/home/kas/kas_workspace/shuozhang/LLaMA-VID/llamavid/train/train.py", line 544, in preprocess_imgsp_v1 if roles[source[0]["from"]] != conv.roles[0]: KeyError: 0
====================================================================================
same error too, thanks

/

I have encountered the same problem. Have you solved it?


Poor formatting in sequential tuple struct invocations

Input
fn test() {
    self.1 .0;
}

Output
fn test() {
    self.1 .0;
}

Note the space after the 1
Expected output
fn test() {
    self.1.0;
}

Meta

rustfmt version: rustfmt 1.4.24-stable (eb894d53 2020-11-05)
From where did you install rustfmt?: rustup

Closing as duplicate, more context in https://github.com/rust-lang/rustfmt/issues/4355#issuecomment-723764276


Create release 0.5.0

This is a meta-issue to track the work needed before releasing v0.5.0. Closing this issue will trigger the release of v0.5.0.

[x] #7
[x] #79
[x] #84
[x] #86
[x] #87
[x] #88

I think that the cmake stuff has had enough bake time now and it might be wise to get that release out the door before we start doctoring with the runtime detection.

I just checked off the last two remaining issues from the list. As far as I'm concerned, it's time for a release.
With my unfamiliarity with CMake, I'm unsure of exactly how solid the CMake stuff is now. We recently had some pull requests to push some minor issues that were found in production. Should we wait a week or two to see if something else shakes out?
I guess we could just release 0.5.0, and release a 0.5.1 if any bugs are found.

I think that the major things have been ironed out since we have been able to integrate it with three package managers and htot even cross compiled with the build system.

I guess we could just release 0.5.0, and release a 0.5.1 if any bugs are found.

Sounds reasonable :shipit:

Hi, I tried the CMakeLists.txt cross compiling for a i.MX6ULL which is a Armv7-A processor.
According to this document,  "Armv7-A and AArch32 have the same general purpose Arm registers – 16 x 32-bit general purpose Arm registers (R0-R15)."
But at the same time, it seems to support NEON64:
"Armv7-A and AArch32 have 32 x 64-bit NEON registers (D0-D31). These registers can also be viewed as 16x128-bit registers (Q0-Q15). Each of the Q0-Q15 registers maps to a pair of D registers, as shown in the following figure."
As you can see from the output below, somehow the CMakeLists.txt does not detect that I am cross compiling for ARM:
-- The following features have been enabled:

 * CLI, enables the CLI executable for encoding and decoding

-- The following features have been disabled:

 * OpenMP codec, spreads codec work accross multiple threads
 * SSSE3, add SSSE 3 codepath
 * SSE4.1, add SSE 4.1 codepath
 * SSE4.2, add SSE 4.2 codepath
 * AVX, add AVX codepath
 * AVX2, add AVX2 codepath
 * NEON32, add NEON32 codepath
 * NEON64, add NEON64 codepath

If I read the documentation correctly, my target ARCH should be ARM32 but with NEON64 enabled, right?

ah I think this line is the problem:
cmake_dependent_option(BASE64_WITH_NEON32 "add NEON32 codepath" OFF _TARGET_ARCH_arm OFF)
add_feature_info(NEON32 BASE64_WITH_NEON32 "add NEON32 codepath")

IIRC we decided to disable NEON32 support by default even if the compiler supports it, because it cannot be detected at runtime whether or not the CPU supports the NEON extensions. Therefore you need to explicitly enable NEON32 support via the GUI or with the CLI like so:
cmake -G [...] -DBASE64_WITH_NEON32=ON ..

HTH

@aklomp I think we can conclude the stabilization phase now as there haven't been any new issues for a month now. So I don't think there is any point in delaying the release any further.

I agree, let's get it done. It's on my to-do list for tomorrow.


Use stabilized Duration::MAX for timer

It looks like the Duration::MAX is stabilized in Rust version 1.53.0. Is it possible to change the duration_max to Duration::MAX now?

We would like to avoid bumping the MSRV past 1.48, since that's the current version used by Debian Stable.

Thanks, I see. I'll close this now for that consideration.


Develop list of use cases for parks data

For the parks data alone -- not just for the GoRI site -- develop a diverse list of possible use cases that can be explained in sponsor meetings. Store list in the GitHub repo wiki.

Updated here.


luac checker fails on long paths

When editing a file at a long path name, syntastic fails to report errors.
The cause is probably that the luac checker trims long path names which are not matched fully by vim.
Steps to reproduce it:
$ mkdir -p test/with/not/a/very/long/path && cd $_ && echo . > test.lua
$ mkdir ../other && cd $_
$ vim ${PWD%/*}/path/test.lua +SyntasticCheck  # note full path

And an example of the luac: output:
$ luac ${PWD%/*}/path/test.lua
luac: ...ua/src/lua-5.3.1/test/with/not/a/very/long/path/test.lua:1: unexpected symbol near '.'

And how would syntastic convince luac not to shorten the path in its output?

It does not seem possible from a quick look in the LUA source. The file name is printed as part of a debugging facility and it is always trimmed when the length is too long. Is there a way to convince %f to compare the tail of the filenames?

That's handled by Vim, syntastic has no conmtrol over it.  So it's basically garbage in, garbage out. shrug

An idea to workaround this issue is to change the directory to the containing directory. luac is a simple syntax checker anyway (it tries to compile into byte code). Is that possible?

That isn't a simple change, and syntastic is not prepared to deal with it.  ghc-mod is no longer supported for a similar reason (cf. #1576).

Ok, directory changing is ruled out for now.
I looked in the gdb and found that the file name length is restricted in luaG_addinfo by LUA_IDSIZE which is defined as:
/*
@@ LUA_IDSIZE gives the maximum size for the description of the source
@@ of a function in debug information.
** CHANGE it if you want a different size.
*/
#define LUA_IDSIZE	60

No way to change it at runtime. File name aliasing in syntastic is apparently also hard and stdin cannot be used either?
Any way to catch this and report an error? Or maybe just make a note on the wiki and close this issue.

The checker receives the result of %f, not the raw output from luac.  This can't be changed by adding an explicit parser, but that would be yet another workaround for crap in a third party program's output, that happens once in a blue moon.  Adding a note to the wiki seems to be the easier option.

Reading the input file from stdin is possible, and it would solve your problem, but it would break any require.  Please report all this upstream.  Syntastic is not the right place to fix meaningless output of luac.  I added a note to the wiki.

Thanks, this note will do. I've updated the wiki with instructions for a workaround (I tried 512 which works). I doubt upstream will change this limit, lua is intended for embedding so the users should modify this configuration as needed/

I tried 512 which works

Define "works".  Will it work if the path length is 1021?  How about 1533?  Please refrain from offering incomplete solutions in the syntastic's wiki to problems that should be fixed upstream.  If they care to fix the problem, they will.  If they don't care, syntastic won't care to fix it for them either.

I doubt upstream will change this limit, lua is intended for embedding so the users should modify this configuration as needed/

The proper fix would be to add some runtime option to disable filename shortening.  When / if that happens, I'll consider adding support for it to syntastic.  Until then, as I said, synastic won't care about this issue more than luac developers.

"Works" as in less likely to trim the filename. Syntastic is not perfect and neither is lua, both have to make choices. The filename is trimmed due to how error reporting works, it does not store paths of arbitrary length to save memory and reduce complexity (remember its embedding purpose).
For that reason, I don't foresee that luac will gain the ability to store paths of arbitrary length.

Not "arbitrary length", pathconf(3) has been part of POSIX for a while.  Equally irrelevant to syntastic either way.


Search on dev gives index_not_found_exception

http://<IP_ADDRESS>/search/?q=finance gives:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/pyramid_debugtoolbar/panels/performance.py", line 57, in resource_timer_handler
    result = handler(request)
  File "/app/warehouse/raven.py", line 41, in raven_tween
    return handler(request)
  File "/usr/local/lib/python3.5/site-packages/pyramid/tweens.py", line 62, in excview_tween
    reraise(*attrs['exc_info'])
  File "/usr/local/lib/python3.5/site-packages/pyramid/compat.py", line 148, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/pyramid/tweens.py", line 22, in excview_tween
    response = handler(request)
  File "/usr/local/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 101, in tm_tween
    reraise(*exc_info)
  File "/usr/local/lib/python3.5/site-packages/pyramid_tm/compat.py", line 15, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 83, in tm_tween
    response = handler(request)
  File "/app/warehouse/cache/http.py", line 69, in conditional_http_tween
    response = handler(request)
  File "/usr/local/lib/python3.5/site-packages/pyramid/router.py", line 158, in handle_request
    view_name
  File "/usr/local/lib/python3.5/site-packages/pyramid/view.py", line 547, in _call_view
    response = view_callable(context, request)
  File "/app/warehouse/sessions.py", line 291, in wrapped
    return view(context, request)
  File "/app/warehouse/csrf.py", line 38, in wrapped
    return view(context, request)
  File "/usr/local/lib/python3.5/site-packages/pyramid/viewderivers.py", line 512, in csrf_view
    return view(context, request)
  File "/app/warehouse/cache/origin/__init__.py", line 81, in wrapped
    return view(context, request)
  File "/usr/local/lib/python3.5/site-packages/pyramid/viewderivers.py", line 442, in rendered_view
    result = view(context, request)
  File "/usr/local/lib/python3.5/site-packages/pyramid/viewderivers.py", line 147, in _requestonly_view
    response = view(request)
  File "/app/warehouse/views.py", line 199, in search
    url_maker=paginate_url_factory(request),
  File "/app/warehouse/utils/paginate.py", line 62, in ElasticsearchPage
    return Page(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/paginate/__init__.py", line 251, in __init__
    self.items = list(self.collection[first:last])
  File "/app/warehouse/utils/paginate.py", line 44, in __getitem__
    self.results = self.query[range].execute()
  File "/usr/local/lib/python3.5/site-packages/elasticsearch_dsl/search.py", line 579, in execute
    **self._params
  File "/usr/local/lib/python3.5/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/elasticsearch/client/__init__.py", line 548, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python3.5/site-packages/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.5/site-packages/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')

I've done make build and make serve. Maybe make initdb file should index the db as well?

Hmm, make initdb is supposed to index it, it's strange that it didn't for you.

Rerunning make initdb worked! Closing.


What happens after the OSDSM?

I get emails from people thanking me for the OSDSM. Many also ask what to do next, or what career they can choose after studying pieces of the curriculum.
Let's open the conversation:

What do you want to work on? (not job title, but the work itself)
What projects helped you learn?
What does the OSDSM lack?

Here's a fun list of all the job titles you could possible conceive of:
The brief multi-tweet enumeration of job titles encompassed by Data Science

Hi Clare!
How do you think someone who currently works full-time can do this type of massive studying part-time in evenings and weekends?
Also, what has been the reception of other employers towards aspiring data scientists who have taken your approach?

I'm a recent CS Engineering Graduate and while apply for my Master's to International Institutions I mentioned OSDSM, fingers crossed as of now!
Awaiting results.

Hi Clare - I'm not sure if it's the right place to do it - but I thought that I would raise an issue for facilitating team approaches to capstone projects.
I think that the OSDSM is great - if you were to compare it against the other 'bootcamps' out there, I think that it's just that team element around the capstone project that is missing.

Please, this is a great place to raise the issue! I'd love to hear more
about what you're thinking. There have been a few reddit threads where
learners have sought study groups, perhaps there's opportunity there as
well.
Clare Corthell
Co-Founder, summer.ai http://summer.ai* Machine Intelligence Strategy*
Author, The Open Source Data Science Masters
http://datasciencemasters.org/
email<EMAIL_ADDRESS><EMAIL_ADDRESS>twitter @clarecorthell http://www.twitter.com/clarecorthell | linkedin
http://linkedin.com/in/clarecorthell
☏<PHONE_NUMBER>
On Tue, Aug 18, 2015 at 9:31 PM, Nick Byrne<EMAIL_ADDRESS>wrote:

Hi Clare - I'm not sure if it's the right place to do it - but I thought
that I would raise an issue for facilitating team approaches to capstone
projects.
I think that the OSDSM is great - if you were to compare it against the
other 'bootcamps' out there, I think that it's just that team element
around the capstone project that is missing.
—
Reply to this email directly or view it on GitHub
https://github.com/datasciencemasters/go/issues/70#issuecomment-132441682
.

Yeah local study groups could work well, but I think that the strength of OSDSM is that you have a published list of courses that you should aim to complete, but it's not prescriptive in what order you complete it.
What if we created another folder /capstone projects, much like /transcripts. People embarking on a capstone project would then do the following:

Create you're git repository containing a readme.md outlining your proposed capstone project
Raise an issue linking back to your capstone project github repo calling for project partners?

OR if you don't have a project in mind we could raise an issue every week / fortnight / month calling for people to comment who are looking for a group to collaborate on capstone project with.
Side note:
Encouraging local meetups themed around the OSDSM would allow local groups to cycle speakers in and out relevant to the curriculum + facilitate introductions for capstone projects too.

I changed my mind. Here it is:
https://github.com/datasciencemasters/capstone/
This way, requests can be filed (people who aren't actually going to do the
work, business people, curious passerbys) and people can pick them up if
they want to take on a project.
I think the recipe is:

Dataset (must be available somehow, of course)
Question (that is perceived to be possible to answer with the data)

Then people can raise issues on particular capstone listings, which are
each an .md file in the /capstone repo. I like the idea of routinely making
a call for collaborators.
Thanks for raising this issue! I needed a little push to actually invest
ten minutes in a new part of the OSDSM for this. I really appreciate it.
Do you have further thoughts? Looking for a project team yourself?
Clare
Clare Corthell
Co-Founder, summer.ai http://summer.ai* Machine Intelligence Strategy*
Author, The Open Source Data Science Masters
http://datasciencemasters.org/
email<EMAIL_ADDRESS><EMAIL_ADDRESS>twitter @clarecorthell http://www.twitter.com/clarecorthell | linkedin
http://linkedin.com/in/clarecorthell
☏<PHONE_NUMBER>
On Tue, Aug 18, 2015 at 10:27 PM, Nick Byrne<EMAIL_ADDRESS>wrote:

Yeah local study groups could work well, but I think that the strength of
OSDSM is that you have a published list of courses that you should aim
to complete, but it's not prescriptive in what order you complete it.
What if we created another folder /capstone projects, much like
/transcripts. People embarking on a capstone project would then do the
following:

Create you're git repository containing a readme.md outlining your
proposed capstone project
Raise an issue linking back to your capstone project github repo
calling for project partners?

OR if you don't have a project in mind we could raise an issue every week
/ fortnight / month calling for people to comment who are looking for a
group to collaborate on capstone project with.
Side note:
Encouraging local meetups themed around the OSDSM would allow local groups
to cycle speakers in and out relevant to the curriculum + facilitate
introductions for capstone projects too.
—
Reply to this email directly or view it on GitHub
https://github.com/datasciencemasters/go/issues/70#issuecomment-132450900
.

This is great. It also allows the OSDSM project to continue to vet peoples work through pull requests etc too. Perhaps suggesting people (as a minimum) document their project, but ideally submit their code too.
I am / will be looking for a team. I do have some background in data sciences, so perhaps not as green as some, which is helpful. When I consider a capstone project, I'm look at it from two angles:

Personal: Is there a world view or hypothesis that you would like to test, and see an opportunity to do so using a data driven approach?
External: another individual or organisation has a hypothesis they need tested

If we just facilitate #2 then that's not all that difficult other then vetting good projects. But may be there are some materials that we could curate around #1 to help people along their capstone journey. I'm thinking just basic reminders of the scientific method to help stay focused.
With respect to #2, I came across Coursolve today, and have just finished signing up. That seems to be a really interesting approach to introducing real world problems for 'digital interns' to work on. Once I know more I'll share here. I suppose there'd be more resources like this that we could share as part of OSDSM.

I just looked at Coursolve - the idea is great, but it hasn't been taken up.
But it's worth checking out, you might have ideas against that that I haven't thought of yet.
I'll give some more rigorous thought to the capstone project idea that I had, and then use your system to raise a new project, and see how much attention we can attract from others in the community to join in.

I guess the next thing that came to mind is what an 'open source data science PhD' would look like...

I've concluded that's equivalent to working for a company for 2+ years. I
mulled it over many a time, but applications are the best place to hone
your deep knowledge.
On Wed, Aug 19, 2015 at 3:00 AM Nick Byrne<EMAIL_ADDRESS>wrote:

I guess the next thing that came to mind is what an 'open source data
science PhD' would look like...
—
Reply to this email directly or view it on GitHub
https://github.com/datasciencemasters/go/issues/70#issuecomment-132522904
.

The OSDSM has been significantly restructured, and took this feedback to heart. If you'd like, take a look and give some feedback! This issue can be reopened with that commentary. Cheers!


Building errors (Redefinition, Duplicate, Property has a previous declaration)

I'm trying to use this module but I'm getting those errors on build.

I tried to install Google-Mobile-Ads-SDK with pod and that created a separate project. I don't know if that is the issue.

@gsdias i've same issue too. Do you solve this?

@ayberkanilatsiz still waiting on help from anyone. My guess is I'm doing the import of frameworks in a wrong way.

I am also facing this issue.

@ayberkanilatsiz @pankaj-ag if you use RN v39 it will work. Its not compatible with most recent RN versions

Published https://github.com/sbugert/react-native-admob/releases/tag/v1.2.0
Please feel free to ping me if anything is wrong.

@alvaromb after upgrading, clean and build again I got this issue

This means you don't have Google's AdMob SDK configured properly. Check out the SDK installation instruction, specially the rewarded ad section.

ok I will but if I comment that to build I got in console when open app the following:
Feb 24 20:28:19 DUB0024-2 BabyTrack[38736]: Warning: Native component for "RNAdMob" does not exist
Feb 24 20:28:19 DUB0024-2 BabyTrack[38736]: Warning: Native component for "RNAdMobDFP" does not exist

Have you linked the lib? I'm testing it with a current project and it's working fine. Which RN version are you running?

I linked with react-native link react-native-admob. the first error is gone after importing again Google framework but still the second warning and a red dot. let me unlink and link back again

"react-native": "0.41.2",
"react-native-admob": "^1.2.0",

after a restart in the simulator got working, tks


[bug][k8s-keystone-auth] webhook image in the release-1.25 branch is not pinned to a specific version

/kind bug
What happened:
Using the release-1.25 branch example deployment of k8s-keystone-auth on arm64 kubernetes workers fails because the image listed is tagged at latest which currently only supports amd64.
What you expected to happen:
In previous release branches the container images are pinned to the release version

1.23
1.24

How to reproduce it:
Deploy the example k8s deployment into an arm64 kubernetes cluster
Anything else we need to know?:
None
Environment:
N/A

Here's a similar PR where the examples image was updated

@jichenjc beautiful.  That's what i was looking for!


Temp reprting - Incorrect

Any device that reports Degrees Fahrenheit shows incorrect reading in HomeKit. HomeKit assumes all devices translate Degrees Fahrenheit to Degrees Celsius, so when the device is imported there is no conversion causing a really high temperature reading.

Still an issue

This issue is present, should be reopened

This is indeed still an issue.


vocoder stdin/stdout support

Goal: Run the FARGAN feature extractor and vocoder as streaming applications, as a stepping stone to a real time implementation.
Write our own versions of the command line lpcnet_enc/lpcnet_dec utilities that support streaming I/O from stdin/stdout
Place source in radae/src
Link with OPUS library (cmake variable for path)
Build using radae cmake build system (which currently doesn't do any C compiling)
Skills: C, basic cmake

Sortred in #6


Fixes #2943 XAML has no Intellisense

Fixes #2943 XAML has no Intellisense
Supports assembly references without correct hint paths
Avoids creating too many analyzers at project load
Ensures references are transferred between analyzers

This is not quite finished because, despite the title, there's still no XAML Intellisense. But it seems to be for some weird editor reason, rather than because we are doing something wrong.
The situation is now better enough than our current release that I'd take this before the next build, though it would be nice to also figure out what magical incantation is needed to satisfy the WPF tools.

👍

Turns out the WPF team can't properly reproduce the XAML editor issue without this fix, so we'll take it now and come back to the main one.


ProvisionedThroughputExceeded does not trigger exponential backoff

I'm getting lots of ProvisionThroughputExceeded errors and they're surfacing immediately instead of after retries with exponential backoff. It looks like it's because of this check: response codes < 500 aren't retried. According to the AWS docs, this error is in the 400 range and should be retried.
I'm happy to supply a PR if you agree this is in fact a bug. Otherwise, it seems client code has to catch this exception after doing a batch.save() and retry with batch.commit()?

Actually it looks like there's no decent way to recover from this because batch.commit() clears out its list of pending requests.


Added wrappers and modified install script for MinGW bash

Added:

nodist.sh: creates nodist bash function
nodist: reverted to old code

Edited NSIS script to edit .bash_profile

@marcelklehr This is just a start. I'm new to NSIS, so I'm not sure I didn't screw something up. Tested and it appears to be working. Hope this helps

Hah, I guess I didn't pay much attention to the bash script and screwed it up single-handedly :) -- thanks for bringing this up again!
I'm wondering, why did you create two scripts?

I'll start by answering your second question: Yes, nodist will be available out of the box. However (and here's where I answer your first question), each bash script runs in its own separate environment, which is an exact copy of the caller's environment. Running in its own environment means that any changes to the env variables is reflected only within that same environment. By leaving just one script, the NODIST_VERSION env variable would disappear as soon as that one script finished executing.
When you want to run a script in the same context as the caller, you use the source command (See http://ss64.com/bash/source.html). That's why you need two scripts. One of them (the wrapper) is sourced, which ensures that the NODIST_VERSION variable will be set in the current shell's context. If the command is neither use nor env, it will call the second script (nodist) as would normally happen without a wrapper. I believe we can't ask the user to do source PATH_TO_NODIST/nodist (source requires a full path) every time they want to run use or env. That's where the third script comes in (the alias).
The third script is the one sourced by .bash_profile (See the install script here. source can also be called as a single dot .) to create an alias for source PATH_TO_NODIST/nodist called just nodist.  However, I felt it was safer and more appropriate to use the NODIST_PREFIX env variable and append /bin to it, instead of depending on the PATH entry for the nodist bin directory (feel free to change that). ${NODIST_PREFIX//\\/\/} means take the value of NODIST_PREFIX and replace \ with /.
The end result is that any time the user calls nodist in bash, they're calling the alias, which sets NODIST_VERSION when needed, or transparently calls the normal nodist command with whatever arguments were passed in.
Please, feel free to ask me anything if I wasn't clear enough.

@marcelklehr I went crazy and decided to redo the entire thing.
This is what will basically happen with these changes:

The installer will add the contents of bin/bash_profile_content.sh to ~/.bash_profile, if it exists. This will ensure that every time bash is launched, the contents of that file will be sourced (See source docs), creating a bash function called nodist which will be called INSTEAD of the nodist command in PATH
The nodist bash function will handle the use and env commands and forward all others to the nodist script in $NODIST_PREFIX/bin. Calling a function from bash ensures that it shares the exact same environment as the calling bash, and it can, therefore, modify the environment to add NODIST_VERSION.

I'm not 100% sure about having the installer check if ~/.bash_profile exists. Assuming a user were to install nodist BEFORE Git Bash, I'm not sure whether installing Git Bash after that would overwrite or just edit ~/.bash_profile. Feel free to change that to always write the commands to ~/.bash_profile. Either way, having the regular nodist script separate and still in the path will ensure that, even if ~/.bash_profile gets overwritten or deleted, nodist will still work (except for env and use).
You can look at this gist where I've pasted my Git Bash output after manually adding (that's why the version list is still using v0.7.2 formats) all the files the installer would add

Awesome work! Thank you for your in-depth explanation and :fireworks: for the new solution, I like it! I will  be compiling a pull request to the release/0.8.0 branch with your changes adapted in respect to the changes in that branch. I'll reference it here, so you can have a look at it, before I pull it.

Perfect! Let me know if there's any other way I can help. Thank you for all the work you've already put into nodist.

@marcelklehr rewriting comment here for visibility, since it's hidden because it's in old code:
We can apparently work around it by doing:
#!/bin/sh

DIR=$(echo "$NODIST_PREFIX" | sed -e 's,\\,/,g')
"$DIR/bin/node.exe" "$DIR/cli" "$@"

This does not mean env and use will work in Cygwin. Cygwin uses another set of .bash* files located at <cygwin_install_dir>\home\<user>\
My suggestion would be to provide detailed instructions, somehwere easily found, for Cygwin users to add the nodist() function manually

Ok, that makes sense. See https://github.com/marcelklehr/nodist/pull/130/commits/bae9b0a015e1053ba1108dc7123337f53fbe3c63 for the merge commit of this pr.

[ ] Add docs for cygwin users

@gmahomarf  from the initial script I would think that the node.exe file being called is in the same folder than cli:
#!/bin/sh
DIR="`dirname \"$0\"`"
"$DIR/../node.exe" "$DIR/../cli" $*

Is your solution calling that node.exe or the one that is one level deeper in the bin folder?
#!/bin/sh
DIR=$(echo "$NODIST_PREFIX" | sed -e 's,\\,/,g')
"$DIR/bin/node.exe" "$DIR/cli" "$@"

@gugol2 Ah, you're right! It should call $NODIST_PREFIX/node.exe. Fixed now. Thanks for reviewing :)


Signatures failing to validate on Google App Engine

I'm using SNS service running on GAE, and signatures fail to verify. I traced it down to GAE rewriting the request headers:

User-Agent. This header can be modified but App Engine will append an identifier string to allow servers to identify App Engine requests. The appended string has the format "AppEngine-Google; (+http://code.google.com/appengine; appid: APPID)", where APPID is your app's identifier.

I was able to work around this by removing this line in https://github.com/awslabs/aws-sdk-go/blob/master/aws/query.go#L37
	httpReq.Header.Set("User-Agent", "aws-go")

Is the User-Agent header required?

The user agent is required, but that header should not be signed, so GAE should be able to append values to it without affecting the request. Note that all development is currently happening in the develop branch, where the signer has changed, so this might not work in master but it will once develop is merged back in.

Develop was merged into master today, so I am marking this as closed since we should now be signing these requests correctly. Thanks for reporting! Feel free to re-open if you still run into this issue.

I am having this same issue with latest SDK and latest GAE SDK.
Stripped prohibited headers from URLFetch request: ['Content-Length']
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

It says check AWS Secret Access Key and signing method.
I'm just using this format:
`sess := session.New(&aws.Config{
Region:      aws.String("us-west-1"),
Credentials: credentials.NewStaticCredentials(XXX", "YYY", ""),
HTTPClient:  client,
})

I left token blank because I'm using root credentials.


Add command history functionality

Closes #135
You can now press up to scroll to a previously inputted command.
Only valid commands are saved in the history.
To test:

Input a valid command (i.e. one that doesn't throw ParseException or CommandException)
Input a type a few words
Press up, it should change the input to the previous command
Press up again, it shouldn't change the input since it's already at the oldest command
Press down, it should return to what you were trying to type
Press down again, it shouldn't change the input since it's already at the newest command

Codecov Report

Merging #143 (dc9f377) into master (7676f5e) will decrease coverage by 0.66%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #143      +/-   ##
============================================
- Coverage     78.51%   77.84%   -0.67%
  Complexity      670      670
============================================
  Files            90       90
  Lines          2001     2018      +17
  Branches        266      270       +4
============================================
  Hits           1571     1571
- Misses          351      368      +17
  Partials         79       79

Impacted Files
Coverage Δ

src/main/java/seedu/address/ui/CommandBox.java
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7676f5e...dc9f377. Read the comment docs.


Unable to build from source with GPU support

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
TensorFlow version and how it was installed (source or binary): source
TensorFlow-Addons version and how it was installed (source or binary): source
Python version: 3.8.10
Is GPU used? (yes/no): yes

Describe the bug
I've downloaded and built from source TF 2.9.1 with GPU support. No errors.
But an error occurred during tensorflow_addons (0.17.0) building from source
DEBUG: /home/alex/.cache/bazel/_bazel_alex/f67f41e413892adc9e99d88ee1f21ae3/external/org_tensorflow/third_party/repo.bzl:124:14:
Warning: skipping import of repository 'cub_archive' because it already exists.
DEBUG: /home/alex/.cache/bazel/_bazel_alex/f67f41e413892adc9e99d88ee1f21ae3/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:118:10:
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1596824487 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /home/alex/jupyter/build/addons/WORKSPACE:45:14: in <toplevel>
  /home/alex/.cache/bazel/_bazel_alex/f67f41e413892adc9e99d88ee1f21ae3/external/org_tensorflow/tensorflow/workspace0.bzl:107:34: in workspace
  /home/alex/.cache/bazel/_bazel_alex/f67f41e413892adc9e99d88ee1f21ae3/external/bazel_toolchains/repositories/repositories.bzl:35:23: in repositories
Repository rule git_repository defined at:
  /home/alex/.cache/bazel/_bazel_alex/f67f41e413892adc9e99d88ee1f21ae3/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
WARNING: /home/alex/.cache/bazel/_bazel_alex/f67f41e413892adc9e99d88ee1f21ae3/external/local_config_tf/BUILD:13345:8: target 'libtensorflow_framework.so.2' is both a rule and a file; please choose another name for the rule
INFO: Analyzed target //:build_pip_pkg (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/alex/jupyter/build/addons/tensorflow_addons/custom_ops/image/BUILD:7:18: Compiling tensorflow_addons/custom_ops/image/cc/kernels/adjust_hsv_in_yiq_op_gpu.cu.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/ubuntu20.04-gcc9_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF ... (remaining 60 arguments skipped)
/dt9/usr/bin/gcc: No such file or directory
nvcc fatal   : Failed to preprocess host compiler properties.
Target //:build_pip_pkg failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.429s, Critical Path: 0.12s
INFO: 25 processes: 25 internal.
FAILED: Build did NOT complete successfully

Code to reproduce the issue
git clone https://github.com/tensorflow/addons.git
cd addons

export TF_NEED_CUDA="1"

python3 ./configure.py

bazel clean --expunge
bazel build build_pip_pkg

But if I manually replace crosstool_top value in .bazelrc with "@local_config_cuda//crosstool:toolchain" - build successfull.

But if I manually replace crosstool_top value in .bazelrc with "@local_config_cuda//crosstool:toolchain" - build continues...
And another error occured:
ERROR: /home/alex/jupyter/build/addons/tensorflow_addons/custom_ops/seq2seq/BUILD:7:18: Compiling tensorflow_addons/custom_ops/seq2seq/cc/kernels/beam_search_ops_gpu.cu.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF ... (remaining 61 arguments skipped)
Traceback (most recent call last):
  File "external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc", line 269, in <module>
    sys.exit(main())
  File "external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc", line 256, in main
    return InvokeNvcc(leftover, log=args.cuda_log)
  File "external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc", line 207, in InvokeNvcc
    nvccopts += r'-gencode=arch=compute_%s,\"code=sm_%s\" ' % (
TypeError: not all arguments converted during string formatting

It can be fixed by removing last ", capability" here https://github.com/tensorflow/addons/blob/master/build_deps/toolchains/gpu/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl#L208
And after that build will be successful.

Can you try to submit a PR?

@seanpmorgan Do we still needs these build_deps with the new 2.9 toolchain?

Ugly fix on ubuntu 20.04
sudo mkdir -p /dt9/usr
sudo ln -s /usr/bin /dt9/usr/bin

Is @ubuntu20.04-gcc9_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda only intended to build from a docker image ?

Is @ubuntu20.04-gcc9_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda only intended to build from a docker image ?

It is mainly for producing manylinux2014 compatible wheels. But as we don't wan to maintain to much build configs we rely on this.

@bhack , this issue is still not resolved.
It still required to manually replace crosstool_top value in .bazelrc with "@local_config_cuda//crosstool:toolchain"
I think it should be either set automatically when building outside docker or specified via args in "configure" command and documented in readme.

@shkarupa-alex It was closed automatically as connect to your PR by Github "magic" keywords..


subtitles-rs: init at 20180611

Tooling to deal with video subtitles.
Motivation for this change
I have tested only the subtitles2srt binary.
If my memory is correct, I stole the derivation from @symphorien 's overlay.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Fits CONTRIBUTING.md.

Success on aarch64-linux (full log)
Attempted: subtitles-rs
Partial log (click to expand)
shrinking /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs/bin/subtitles2srt
shrinking /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs/bin/vobsub2png
shrinking /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs/bin/opusraw2txt
shrinking /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs/bin/substudy-backend
shrinking /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs/bin/substudy
strip is /nix/store/428gs2z4b8f9byvghzlpbjwjb3a7jwxx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs/bin
patching script interpreter paths in /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs
checking for references to /build in /nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs...
/nix/store/qydihppjfr1qqls1mxq19xhshf4bbnfg-subtitles-rs

Success on x86_64-linux (full log)
Attempted: subtitles-rs
Partial log (click to expand)
shrinking /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs/bin/vobsub2png
shrinking /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs/bin/subtitles2srt
shrinking /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs/bin/substudy
shrinking /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs/bin/submodel
shrinking /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs/bin/substudy-backend
strip is /nix/store/dxf1m7dhc4qb655bdljc1fsd74v1nag3-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs/bin
patching script interpreter paths in /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs
checking for references to /build in /nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs...
/nix/store/vcpa6pshk2rgnb0qah91756x2dn5nn2j-subtitles-rs

I tried to package the release given at http://github.com/emk/subtitles-rs/issues/46
  version = "substudy_v0.4.5";

  src = fetchFromGitHub {
    owner = "emk";
    repo = "subtitles-rs";
    rev = version;
    sha256 = "0q79q7irn7ac206s5ysla65g4dq0na0arj82jrip9qb155mww7vz";
  };

which ended up with

unpacking source archive /nix/store/jf562xpnwxsx3bm4hbdavllid2v0n97p-subtitles-rs-vendor
patching sources
configuring
building
Running cargo build --release
error: no matching version `= 0.3.8` found for package `log`
location searched: registry `https://github.com/rust-lang/crates.io-index`
versions found: 0.4.0, 0.3.9
required by package `substudy-backend v0.1.0 (file:///build/source/backend)`
note: keeping build directory '/tmp/nix-build-subtitles-rs.drv-1'
builder for '/nix/store/f7s47pa6nmmcldzb1z6cfp7h97ziig56-subtitles-rs.drv' failed with exit code 101
error: build of '/nix/store/f7s47pa6nmmcldzb1z6cfp7h97ziig56-subtitles-rs.drv' failed

Then I enabled the tests
running 5 tests
test file_path_checks_validity ... ok
test html::display_produces_canonical_string ... ok
test html::parse_valid_html_fragments ... ok
test html::error_on_invalid_html_fragments ... ok
test parse_metadata ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/cli_test_dir-d4c5577aca0a75b3

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/common_failures-b08bd1639fd8bda5

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/opusraw2txt-bcb7ef4b434e6d08

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/opusraw2txt-72556968f130f9f8

running 2 tests
test parse_large_file ... ignored
test parse_file ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/submodel-ca26d8664a0841c1

running 4 tests
test pair_frequency ... ok
test grapheme_frequency ... ok
test word_frequency ... ok
test write_model ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/submodel-a329f3a14d407533

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/substudy-b4a2e54f0381c7cb

running 16 tests
test decode::test_smart_decode ... ok
test align::test_alignment ... ok
test merge::test::merge_long ... ok
test export::csv::test_episode_prefix ... ok
test merge::test::merge_one ... ok
test merge::test::merge_zero ... ok
test merge::test::merge_short ... ok
test srt::test::subtitle_file_from_path ... ok
test srt::test::subtitle_file_to_string ... ok
test srt::test::subtitle_to_string ... ok
test srt::test::zero_duration_subtitle ... ok
test time::test_timestamp ... ok
test video::test_stream_decode ... ok
test clean::test_clean_subtitle_file ... ok
test srt::test::detect_language ... ok
test align::test_combine_files ... ok

test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/tests-06c00a0c7482236a

running 8 tests
test cmd_export_review ... FAILED
test cmd_export_csv ... FAILED
test cmd_list_tracks ... FAILED
test cmd_export_tracks ... FAILED
test show_help ... ok
test show_version ... ok
test cmd_clean ... ok
test cmd_combine ... ok

failures:

---- cmd_export_review stdout ----
thread 'cmd_export_review' panicked at 'assertion failed: output.status.success()', substudy/tests/tests.rs:99:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- cmd_export_csv stdout ----
thread 'cmd_export_csv' panicked at 'assertion failed: output.status.success()', substudy/tests/tests.rs:82:5

---- cmd_list_tracks stdout ----
thread 'cmd_list_tracks' panicked at 'assertion failed: output.status.success()', substudy/tests/tests.rs:129:5

---- cmd_export_tracks stdout ----
thread 'cmd_export_tracks' panicked at 'assertion failed: output.status.success()', substudy/tests/tests.rs:115:5

failures:
    cmd_export_csv
    cmd_export_review
    cmd_export_tracks
    cmd_list_tracks

test result: FAILED. 4 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out

As I am not familiar with rust, I don't intend to fix these. If that's not acceptable, I can maintain this package locally.

The failure does not seem spurious: substudy is run with these arguments and fails.
https://github.com/emk/subtitles-rs/blob/23c8a245290ba696e8f3eacf013eafacbd979070/tests/tests.rs#L74-L82

Success on x86_64-linux (full log)
Attempted: subtitles-rs
Partial log (click to expand)
shrinking /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs/bin/submodel
shrinking /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs/bin/substudy-backend
shrinking /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs/bin/substudy
shrinking /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs/bin/vobsub2png
shrinking /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs/bin/opusraw2txt
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs/bin
patching script interpreter paths in /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs
checking for references to /build in /nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs...
/nix/store/rnankrd0dnllnclakzv6vvjqfl12h2bd-subtitles-rs

Success on aarch64-linux (full log)
Attempted: subtitles-rs
Partial log (click to expand)
shrinking /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs/bin/subtitles2srt
shrinking /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs/bin/vobsub2png
shrinking /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs/bin/opusraw2txt
shrinking /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs/bin/substudy-backend
shrinking /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs/bin/substudy
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs/bin
patching script interpreter paths in /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs
checking for references to /build in /nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs...
/nix/store/wp6jybnjll3msai2spmwsharhn80f2sy-subtitles-rs

Are there any updates on this pull request, please?

I'll maintain it in my overlay for now.


Command given in tutorial "bpftrace -v -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'" fails

Environment:
OS: Debian 9
Kernel: 4.9.110-3+deb9u6
Since debian 9 did not have bpfcc-tools package, I downloaded the following packages from debian 10 and installed them on debian 9:

bpfcc-tools_0.8.0-4_all.deb
bpftrace_0.8+git60-gccac69c2239b-2_amd64.deb
libbpfcc_0.8.0-4_amd64.deb
libclang1-7_1%3a7.0.1-8_amd64.deb
libllvm7_1%3a7.0.1-8_amd64.deb
libtinfo6_6.1+20181013-2_amd64.deb
python-bpfcc_0.8.0-4_all.deb

I was following the lesson 3 from [bpftrace oneliners tutorial|https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md]. Getting the following error
root@fk-cloud-g1a-ms-1312745:/home/livingstone.se# bpftrace -ve 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
Attaching 1 probe...

Error log:
0: (bf) r6 = r1
1: (b7) r7 = 0
2: (7b) *(u64 *)(r10 -96) = r7
3: (7b) *(u64 *)(r10 -104) = r7
4: (7b) *(u64 *)(r10 -112) = r7
5: (7b) *(u64 *)(r10 -120) = r7
6: (7b) *(u64 *)(r10 -128) = r7
7: (7b) *(u64 *)(r10 -136) = r7
8: (7b) *(u64 *)(r10 -144) = r7
9: (7b) *(u64 *)(r10 -152) = r7
10: (7b) *(u64 *)(r10 -160) = r7
11: (7b) *(u64 *)(r10 -168) = r7
12: (7b) *(u64 *)(r10 -176) = r7
13: (7b) *(u64 *)(r10 -80) = r7
14: (7b) *(u64 *)(r10 -88) = r7
15: (bf) r1 = r10
16: (07) r1 += -88
17: (b7) r2 = 16
18: (85) call 16
19: (79) r1 = *(u64 *)(r10 -88)
20: (7b) *(u64 *)(r10 -168) = r1
21: (79) r1 = *(u64 *)(r10 -80)
22: (7b) *(u64 *)(r10 -160) = r1
23: (7b) *(u64 *)(r10 -16) = r7
24: (7b) *(u64 *)(r10 -24) = r7
25: (7b) *(u64 *)(r10 -32) = r7
26: (7b) *(u64 *)(r10 -40) = r7
27: (7b) *(u64 *)(r10 -48) = r7
28: (7b) *(u64 *)(r10 -56) = r7
29: (7b) *(u64 *)(r10 -64) = r7
30: (7b) *(u64 *)(r10 -72) = r7
31: (bf) r3 = r6
32: (07) r3 += 24
33: (bf) r1 = r10
34: (07) r1 += -8
35: (b7) r2 = 8
36: (85) call 4
37: (79) r3 = *(u64 *)(r10 -8)
38: (bf) r1 = r10
39: (07) r1 += -72
40: (b7) r2 = 64
41: (85) call 45
invalid func 45

Error loading program: tracepoint:syscalls:sys_enter_openat

Got the following error when I ran the command from lesson 5
 bpftrace -e 'tracepoint:syscalls:sys_exit_read /pid == 18644/ { @bytes = hist(args->ret); }'
Attaching 1 probe...
ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
Error attaching probe: tracepoint:syscalls:sys_exit_read

I referred a couple of other tickets similar to this like #378 and #207
Here are the relevant details requested in those tickets

debugfs

# mount | grep sys/kernel/debug
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
# cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_openat/format
name: sys_enter_openat
ID: 540
format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:int __syscall_nr;	offset:8;	size:4;	signed:1;
	field:int dfd;	offset:16;	size:8;	signed:0;
	field:const char * filename;	offset:24;	size:8;	signed:0;
	field:int flags;	offset:32;	size:8;	signed:0;
	field:umode_t mode;	offset:40;	size:8;	signed:0;

print fmt: "dfd: 0x%08lx, filename: 0x%08lx, flags: 0x%08lx, mode: 0x%08lx", ((unsigned long)(REC->dfd)), ((unsigned long)(REC->filename)), ((unsigned long)(REC->flags)), ((unsigned long)(REC->mode))

Kernel Parameters

root@fk-cloud-g1a-ms-1312745:/home/livingstone.se# grep BPF /boot/config-4.9.0-8-amd64
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m

perf output

# perf stat -e syscalls:sys_exit_open* -a -- sleep 10

 Performance counter stats for 'system wide':

                 0      syscalls:sys_exit_open_by_handle_at
             28674      syscalls:sys_exit_open
                 0      syscalls:sys_exit_openat

      10.014810684 seconds time elapsed

strace output
I'm not pasting the entire output. Just pasting lines that I though might be relevant

# strace bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffc3807ae7c, value=0x7ffc3807ae80, flags=BPF_ANY}, 112) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 52, {EPOLLIN, {u32=1646178688, u64=94314833031552}}) = 0
dup(2)                                  = 53
open("/dev/null", O_WRONLY)             = 54
dup2(54, 2)                             = 2
close(54)                               = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=264558}, 112) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=264558}, 112) = -1 EINVAL (Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=1, log_size=102400, log_buf=0x7fff75c6d4f0, kern_version=264558}, 112) = -1 EINVAL (Invalid argument)
write(2, "bpf: Failed to load program: Inv"..., 46) = 46
write(2, "0: (bf) r6 = r1\n1: (b7) r7 = 0\n2"..., 1128) = 1128
write(2, "HINT: bpf_probe_read_str missing"..., 57) = 57
uname({sysname="Linux", nodename="fk-cloud-g1a-ms-1312745", ...}) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=264448}, 112) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=264448}, 112) = -1 EINVAL (Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=1, log_size=102400, log_buf=0x7fff75c6d4f0, kern_version=264448}, 112) = -1 EINVAL (Invalid argument)
write(2, "bpf: Failed to load program: Inv"..., 46) = 46
write(2, "0: (bf) r6 = r1\n1: (b7) r7 = 0\n2"..., 1128) = 1128
write(2, "HINT: bpf_probe_read_str missing"..., 57) = 57
open("/usr/include/linux/version.h", O_RDONLY) = 54
read(54, "#define LINUX_VERSION_CODE 26459"..., 8191) = 97
read(54, "", 8191)                      = 0
close(54)                               = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=264592}, 112) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=264592}, 112) = -1 EINVAL (Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=70, insns=0x7f1438e1e000, license="GPL", log_level=1, log_size=102400, log_buf=0x7fff75c6d4f0, kern_version=264592}, 112) = -1 EINVAL (Invalid argument)
write(2, "bpf: Failed to load program: Inv"..., 46) = 46
write(2, "0: (bf) r6 = r1\n1: (b7) r7 = 0\n2"..., 1128) = 1128
write(2, "HINT: bpf_probe_read_str missing"..., 57) = 57
dup2(53, 2)                             = 2
close(53)                               = 0
futex(0x7f142fdd41a0, FUTEX_WAKE_PRIVATE,<PHONE_NUMBER>) = 0
write(2, "Error loading program: tracepoin"..., 68) = 68
write(2, "\n", 1)                       = 1
munmap(0x7f1438e1e000, 4096)            = 0
munmap(0x7f1438e1d000, 4096)            = 0
brk(0x5585e6a4d000)                     = 0x5585e6a4d000
close(3)                                = 0
exit_group(-1)                          = ?
+++ exited with 255 +++

At first glance it looks like a kernel headers mismatch (the "call 45" being invalid). The strace output has "HINT: bpf_probe_read_str missing" (why wasn't that in the normal output?) so it sounds like bpf_probe_read_str() is call 45, and the kernel doesn't have it, but the header files bpftrace has been built for has. Like everything in BPF, these were added in parts. bpf_probe_read_str() was added in 4.11 (https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md ).
AFAIK, the main development here is on Ubuntu, Fedora, RHEL, and ArchLinux, so you might be the only person with a handy environment to debug this further.

@brendangregg I've updated the strace output to include a couple of more relevant lines. Here is what I could figure out from the strace output

dup(2) returned fd 53
open("/dev/null") returned fd 54
dup2(54, 2) returned fd 2 effectively redirecting stderr to /dev/null
close(54)
So now fd 2, i.e., stderr points to /dev/null and fd 54 points to actual stderr
So when the HINT was written, it was written to fd 2 which was effectively pointing to /dev/null
Just after that
dup2(53, 2) returns fd 2. This restored the actual stderr to fd 2.
Whatever we see in the output is printed after this. This is why I think the HINT is missing in the output. Though I have no idea why bpftrace is written like that.

Regarding the actual problem I think what you suggested is correct. As I have mentioned in the environment section, I had downloaded the bpftrace package that was built for debian 10, i.e., 4.19 kernel and installed it on debian 9, i.e., 4.9 kernel with the corresponding header files installed since debian 9 did not have the bpf packages. I think that is the reason it is not working. I'm planning to upgrade the kernel to the one from stretch-backports which is 4.19+105~bpo9+1 and install the corresponding headers. Maybe that will get it working.
Thanks for the help. Will check and get back.
But I think it would be better off to have this mentioned(BPF Features by Linux Kernel Version) in the one liners tutorial so that new bpf newbies don't get confused.

After upgrading the kernel version to 4.19, I tested out the the commands in the first 11 lessons. All of them worked. Thanks.


"IP Address" section zeroed after upgrade from 2.1.5

Issue type
[ ] Feature request
[X] Bug report
[ ] Documentation
Environment

Python version:  3.4.5
NetBox version: 2.2.1

Description
After upgrade from 2.1.5 "IP Address" section is empty but on homepage of netbox counter of IP Addresses show some digits.
https://yadi.sk/i/s4LzW8px3NihJk
https://yadi.sk/i/q1bUGRwL3NihPt

This works fine for me. I also upgraded from 2.1.5 to 2.2.1 today.
Where are you clicking to see the IP Addresses? For me it works with both, clicking "IP Adresses" in the main page or clicking "IPAM->IP Adresses" in the menu.

Check that your installed version of django-tables2 matches what's listed in requirements.txt. If you need further assistance, please post to the mailing list.


Preparing CTE for the GA

Description
Preparing CTE for the GA
Implementation TODOs

[x] Update Identity SDK to point to new CTE path

[x] regenerate implementation from a new swagger spec
[x] update recordings

[x] Remove all occurrences of api_version parameter in other SDKs within the initialization of CommunicationIdentityClient
[ ] changes regarding to ACS verification of App ID & UserId

[ ] regenerate implementation from a new swagger spec
[ ] update client implementation
[ ] update & add tests
[ ] update recordings
[ ] update README & CHANGELOG
[ ] update tag and require path in swagger/README.md
-[ ] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.

All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API changes have been detected in azure-communication-identity. You can review API changes here
API changes
+     EMAIL = "email"

API changes have been detected in azure-communication-identity. You can review API changes here
API changes
+     EMAIL = "email"

API change check for azure-communication-identity
API changes have been detected in azure-communication-identity. You can review API changes here
API changes
+     EMAIL = "email"

API change check for azure-communication-networktraversal
API changes are not detected in this pull request for azure-communication-networktraversal

API change check for azure-communication-chat
API changes are not detected in this pull request for azure-communication-chat

@AikoBB if you by any chance read this, could you please give me contributor rights to your branch?

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-communication-identity

Do not merge until we get the ARB signoff

/azp run python - azure-communication-identity - tests

/azp run python - azure-communication-identity


No auto suggestions on stylesheet properties

Observation
When creating styles using the native Stylesheet, I get hints as I type for properties e.g fontFamily, color, etc! As seen in the image below

How ever, when creating styles using createStyles, I either only get suggestions for the first property only or none at all(see image below)

Environment
Node version: v18.15.0
Operating system: MacOS Sonoma
Editor: VSCode
Unistyles version: ^1.0.0-beta.1
Framework: Expo + typescript (expo router)
File structure

//styles.ts

import { breakpoints } from "constants/breakpoints";
import { theme } from "constants/theme";
import { createUnistyles } from "react-native-unistyles";

export const { useStyles, createStyles } = createUnistyles<
  typeof breakpoints,
  typeof theme
>(breakpoints);

export { theme };

// example usage

import { Text, View } from "components/base";
import { createStyles, useStyles } from "libs/styles";

export default function TabOneScreen() {
  const { styles } = useStyles(styleSheet);

  return (
    <View style={styles.container}>
      <Text style={styles.title}>Tab One</Text>
    </View>
  );
}

const styleSheet = createStyles((theme) => ({
  container: {
    flex: 1,
    alignItems: "center",
    justifyContent: "center",
    backgroundColor: theme.colors.light,
  },
  title: {
    fontSize: 42,
    color: "black",
  }
}));

Fixed with #12 !


[FEATURE]

Was this feature present in original?

[ ] This feature was present in original
[ ] This feature is new

Target platform
Online Version
Describe the request
I am having trouble understanding what some of the power ups do.
Additional information
if you could fix this it would really be helpful.

if there is a downloadable cheat sheet that would be amazing!

Hi! Thanks for reporting. The original had a website explaining everything in detail (and some additional assets). This list is by far the most requested addition. I can't make the buttons in-game work at the moment (I hope that with Ruffle's progression this will become possible), but I'll add the link on the webpage to the manual for now, it should be available soon.

I've added the list, it's linked on the main page and you can see it here

Thank you! I love this game!


Reimplemented all client services to work the same as in Java

Reimplemented all client services (ClientConnectionManager, ClientPartitionService, ClientInvocationService, ClientClusterService) to work the same way as Java client services. Implemented new utility classes and new features such as repetable execution of tasks or delayed execution. Implements the client connection strategy. Changes the client work the same way as Java client with a single connection. More async behaviour as in Java so that IO threads are not blocked.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

From Executor.cpp
int32_t SimpleExecutorService::DEFAULT_EXECUTOR_QUEUE_CAPACITY = 100 * 1000;
In java default value of queue capacity when user did not pass any value is INT_MAX

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

verify-linux

Test FAILed.

Test FAILed.

Test PASSed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test FAILed.

Test PASSed.

Test FAILed.

Test PASSed.

Test FAILed.

Test PASSed.

Test FAILed.

Test PASSed.

Test PASSed.

Test PASSed.

Test PASSed.

Test PASSed.

Test FAILed.

Test FAILed.

Test FAILed.

Test PASSed.

Test FAILed.

Test FAILed.

Test PASSed.

Test PASSed.

Test PASSed.

Test PASSed.

Test PASSed.

Test PASSed.

Test PASSed.

Test FAILed.

verify-windows

Test PASSed.

Test FAILed.

Test PASSed.

Linux test FAILed.

Windows test FAILed.

Windows test FAILed.

Linux test FAILed.

Linux test FAILed.

Windows test FAILed.

Windows test PASSed.

Linux test FAILed.

verify-linux

Linux test PASSed.


Storage domain type awareness for disk creation

Adopt logic that is used in webadmin to correctly handle disk attributes format and sparse based on the target storage domain.
This applies for when the disks are created new and when template disks need to be updated when creating a VM from a template.
Fixes: #1416
Fixes: BZ1957830
The rules for what disk format/sparse values are to be used for a selected allocation type (thin/pre) is based on the target storage domain and varies between storage domain types.  This information is determined and stored in each storage domain in the toInternal() transform.
Summary of specific changes:

Move the determination of all of the non-user input attributes for Create VM new disks to the composeAndCreateVm and related sagas.  This makes it much easier to verify the values and potentially change them as needed based on the VM attributes.

Due to how the REST API works, we can't create a VM from a template and only change 1 disk out of multiple disks.  If one disk needs to be changed, then all of the disks need to be specified in the create request.  We still only send the attributes that need to be changed per disk.

Add 'storageType' and other derived settings to storage domains when they are loaded.  This allows us to minimize lookup logic at runtime.

DiskImageEditor function composeDisk() refactored to have distinct logic between disk creates and disk edits.

@rszwajko - I tweaked the unit tests for determineTemplateDiskFormatAndSparse.  Hopefully they are grouped better logically.

I tried to test as much as possible in the webui side by side with webadmin and the only issue I found was that the default disk type offered when the storage domain is chosen/changed is always thin provision in webui whereas in webadmin it's preallocated for iscsi storage domains and thin provision for nfs domains regardless of desktop/server.
webui part

Create disk in VM wizard or in VM detail
Change/choose storage domain:

iscsi domain is selected, the disk type is thin provision
nfs domain is selected, the disk type is thin provision

webadmin part

Create disk in VM create or in VM detail disk tab
Change/choose storage domain:

iscsi domain is selected, the disk type is preallocated
nfs domain is selected, the disk type is thin provision

Not sure if this is intended or not, please let me know.

After some more discussion with @sjd78, I tested the scenario with read-only permissions on domains (which makes the user switch to another domain) and no domains available. Everything works as expected. Additionally, in webui, the default disk type will always be thin provision so that clears up the issue I found in the comment above.
LGTM

@sjd78 @isaranova I found one problem of a behavior different than for webadmin and which was also working correctly on web-ui prior to this fix: when creating a disk on nfs storage domain based on a server optimized template with type=thin,format=raw for non admin user that requires cloning to a different storage domain:
Actual result: the create vm wizard dialog does show the disk type as "thin provision" but the actual disk is created on cloned storage domain with type=preallocated, format=raw.
Expected result: the disk should be created on cloned storage domain with type=thin,format=raw

I just tested against engine version <IP_ADDRESS>-1.el8
Template
Here is my template with 2 disks, one preallocated, one thin provisioned, both on the same nfs storage domain:
$ curl -k -u"admin@internal:admin" -H"Accept: application/json" -H"Filter:false" \
  "https://chopper.local.dickerson-web.com/ovirt-engine/api/templates/?follow=disk_attachments.disk&search=name=*BaseServer*%20sortby%20name%20asc" \
  | jq '
    .template[] |
    {
      name: .name,
      desc: .description,
      disks: [.disk_attachments.disk_attachment[].disk | { alias:.alias, format:.format, sparse:.sparse, desc:.description, sd:.storage_domains.storage_domain[0].id }] | sort_by(.alias),
    }
  '
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11309    0 11309    0     0  46925      0 --:--:-- --:--:-- --:--:-- 46925
{
  "name": "BaseServer_2disk",
  "desc": "server, 1 thin and 1 pre both in data1/nfs",
  "disks": [
    {
      "alias": "BaseServer_2disk_Disk1_PRE",
      "format": "raw",
      "sparse": "false",
      "desc": "nfs, pre, data1",
      "sd": "5013d627-69ac-4e8a-9b03-b8ac4fb313c3"
    },
    {
      "alias": "BaseServer_2disk_Disk1_THIN",
      "format": "raw",
      "sparse": "true",
      "desc": "nfs, thin, data1",
      "sd": "5013d627-69ac-4e8a-9b03-b8ac4fb313c3"
    }
  ]
}

webadmin created VM
I created a VM in webadmin from that template where I manually changed the storage domains from the original nfs to a second nfs SD.  The format and sparse values are as expected.
$ curl -k -u"admin@internal:admin" -H"Accept: application/json" -H"Filter:false" \
  "https://chopper.local.dickerson-web.com/ovirt-engine/api/vms/?follow=disk_attachments.disk&search=name=BaseServer_webadmin_2%20sortby%20name%20asc" \
  | jq '
    .vm[] |
    {
      name: .name,
      desc: .description,
      disks: [.disk_attachments.disk_attachment[].disk | { alias:.alias, format:.format, sparse:.sparse, desc:.description, sd:.storage_domains.storage_domain[0].id }] | sort_by(.alias),
    }
  '
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15072    0 15072    0     0   114k      0 --:--:-- --:--:-- --:--:--  114k
{
  "name": "BaseServer_webadmin_2",
  "desc": "data1 to data2",
  "disks": [
    {
      "alias": "BaseServer_2disk_Disk1_PRE",
      "format": "raw",
      "sparse": "false",
      "desc": "nfs, pre, data1",
      "sd": "e86d0d23-8d10-440c-94a6-e6ea7beaed36"
    },
    {
      "alias": "BaseServer_2disk_Disk1_THIN",
      "format": "raw",
      "sparse": "true",
      "desc": "nfs, thin, data1",
      "sd": "e86d0d23-8d10-440c-94a6-e6ea7beaed36"
    }
  ]
}

web-ui create VM from non-admin user
Finally, I created a VM in web-ui with a non admin user who cannot create disk in the original nfs storage domain from that template.  The disks' storage domains changed from the original nfs to a second nfs SD.  This matches the change I did in webadmin.  The format and sparse values are as expected.
Storage step on create VM wizard:

Results of the create:
$ curl -k -u"admin@internal:admin" -H"Accept: application/json" -H"Filter:false" \
  "https://chopper.local.dickerson-web.com/ovirt-engine/api/vms/?follow=disk_attachments.disk&search=name=BaseServer_web-ui_userA%20sortby%20name%20asc" \
  | jq '
    .vm[] |
    {
      name: .name,
      desc: .description,
      disks: [.disk_attachments.disk_attachment[].disk | { alias:.alias, format:.format, sparse:.sparse, desc:.description, sd:.storage_domains.storage_domain[0].id }] | sort_by(.alias),
    }
  '
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15089    0 15089    0     0   110k      0 --:--:-- --:--:-- --:--:--  111k
{
  "name": "BaseServer_web-ui_userA",
  "desc": "userA created, data1 to data2",
  "disks": [
    {
      "alias": "BaseServer_2disk_Disk1_PRE",
      "format": "raw",
      "sparse": "false",
      "desc": "nfs, pre, data1",
      "sd": "e86d0d23-8d10-440c-94a6-e6ea7beaed36"
    },
    {
      "alias": "BaseServer_2disk_Disk1_THIN",
      "format": "raw",
      "sparse": "true",
      "desc": "nfs, thin, data1",
      "sd": "e86d0d23-8d10-440c-94a6-e6ea7beaed36"
    }
  ]
}

Summary
In summary:

Disk
Template
webadmin_2
web-ui_userA

BaseServer_2disk_Disk1_PRE
raw / false
raw / false
raw / false

BaseServer_2disk_Disk1_THIN
raw / true
raw / true
raw / true

Everything looks to be in order.  I can re-test against a master engine, probably would take another 2ish hours.  I don't really see the need since I didn't see any of the relevant code change recently.

Following up...
Looking at the tempalteDiskFormatToSparse() function that controls the mapping of a disk format to the disk sparse for template disks, the only path I see from a format of raw to sparse=false (i.e. preallocated) is if the original disk was actually sparse=false or if the new storage domain is not a file type storage domain.
The isCopyPreallocatedFileBasedDiskSupported is derived from the config key CopyPreallocatedFileBasedDiskSupported mapped through either the template's custom compatibility version, or the cluster's version.  Any version >4.2 should be true and therefore in file type storage domains, the sparsity of a raw disk is determined by the sparsity of the template's disk.
Here is the most relevant code:
https://github.com/oVirt/ovirt-web-ui/blob/3498aec9a08e83725732bd803858ae58ca22d19a/src/ovirtapi/transform.js#L703-L730


Is there a variable like {folder} for multi-folder workspace?

I need to add some mappings. Currently I have:
	"path-autocomplete.pathMappings": {
		"core": "${workspace}/core",
	},

So I have at workspace level a /core folder. But I also need some mappings that are folder specific. Do I need something like "${folder}/imports" ?

I also notice that if I have mapping set-up, I cannot navigate to that certain file, like I can with relative import paths. Is this something this package should do ?

After even more testing I realized that these pathMappings work only in the Workspace settings. If I remove it from there and have them in Folder Settings.
Tried this:
    "path-autocomplete.pathMappings": {
        "core": "${workspace}/core",
        "/": "${workspace}/microservices/app/",
    },

And this:
    "path-autocomplete.pathMappings": {
        "core": "${workspace}/imports/core",
    },

^ Thinking that now ${workspace} is actually the folder root, and made a symlink to core.

The ${folder} variable was added in 1.8.0
Please try it out and let me know if this works as expected for you.


A way to see what users are searching for

To implement in the application may be rather time-consuming.
But logging via matomo should allow poor man's view on this.

Need to talk about this in one of the next meetings to discuss the details.

Blocked until a meeting in June where Iulliana is present.

Information in Matomo is sufficient at the moment.


Update and rename lib/domains/al/edu to lib/domains/al/edu/fshn/fshn.txt

The email domain for staff working at the Faculty of Natural Sciences at University of Tirana is @fshn.edu.al

Adding the email domain for the staff at the Faculty of Natural Sciences at University of Tirana

If the email address is @fshn.edu.al, the file structure should be lib/domain/al/edu/fshn.txt.

@elephantsy Flagging as inactive.


with 4 gpu server, gpu:0 is frequently down

With the long training, gpu:0 is down about 2 hours working.
Two guesses:

H/W failures (replacement may be required)
Tensorflow bug?

I will emit the gpu:0 and try to train them again.

It may occur due to the H/W failure: the temperature of that gpu board is high than others


Debugging the Custom Plugins

Thank you so much for this wonderful repository. This helped to create custom plugin library in the most efficient way. Is there a way to test the enqueue function because there seems to be some bugs associated with the kernel calculations?
Thanks in advance
Jeethesh

Do you mean you met error during enqueue? should be caused by the kernel implementation?

I would suggest to debug it using cuda-gdb

Thank you @zerollzeng,
It is due to kernel implementation. I was able to debug using cuda-gdb but since Enqueue function inputs are pointers void const *const *inputs, void *const *outputs. I am not able to figure out how the array is passed to enqeue function. What the size of array is?

I am not able to figure out how the array is passed to enqeue function. What the size of array is?

It's the input of you plugin, so it depends on how do you configure it.

Hi @zerollzeng
if my input of my Plugin is something like this
MSDeformAttentionPlugin:
  attributes:
    im2col_step:
      datatype: int32
  inputs:
    tpg_input_0: # value
      shape: -1x18259x8x32
    tpg_input_1: # value_spatial_shapes
      shape: 4x2
    tpg_input_2: # value_level_start_index
      shape: 4
    tpg_input_3: # sampling_locations
      shape: -1x-1x8x4x4x2
    tpg_input_4: # attention_weights
      shape: -1x-1x8x4x4
  outputs:
    tpg_output_0:
      shape: -1x-1x256
  plugin_type: IPluginV2DynamicExt
  support_format_combination:
    - "float32+int32+int32+float32+float32+float32"

will the inputs be passed with the same style as passed here or will they be flattened?
e.g. should I access the tpg_input_1 like
const float* var1 = reinterpret_cast<const float *>(inputs[0]);
and then access individual elements like var1[0][18256][31] ?

Thanks

e.g. should I access the tpg_input_1 like
const float* var1 = reinterpret_cast<const float *>(inputs[0]);
and then access individual elements like var1[0][18256][31] ?

Yes.

I just cloned the new version, can you tell me how you debug this library file. Right now I am using
cuda-gdb trtexec_debug --loadEngine "someEnginefile.engine" --workspace 16000 --plugins <path to libPlugin.so file>

But I get the value of var1[0][18256][31] or var1[2] or anything as 0. Is this because trtexec application inputs some random zero values during the inference time or am I not accessing the plugin variable correctly. Is there a way to debug this shared library without attaching to trtexec application ? can I supply a random input which will make it more deterministc to debug for me.
Thanks a lot for your help
Jeethesh

trtexec use random inputs. There is also a --loadInputs option that let you load a real input in binary.

I tried using the method mentioned here. But still the variable inputs says 0 everywhere.

I tried MultiScaleDeformableAttn plugin of TensoRT8 on TensorRT7, I can convert onnx to trt file via trtexec, buts  output of MultiScaleDeformableAttn plugin is all zero, just same as here

我在TensorRT7上尝试了TensoRT8的MultiScaleDeformableAttn插件，我可以通过trtexec将onnx转换为trt文件，但是MultiScaleDeformableAttn插件的输出全为零，就像这里一样

请问你解决了这个问题了吗？我也遇到了相同的问题