ERROR: An extractor error has occurred. (caused by KeyError('asset',)) // Sportdeutschland.tv

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

PASTE VERBOSE LOG HERE
PS C:\Windows\system32> youtube-dl "https://sportdeutschland.tv/handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1" --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://sportdeutschland.tv/handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.4.1-essentials_build-www.gyan.dev, ffprobe 4.4.1-essentials_build-www.gyan.dev
[debug] Proxy map: {}
[SportDeutschland] handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError('asset',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\sportdeutschland.py", line 53, in _real_extract
KeyError: 'asset'
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\sportdeutschland.py", line 53, in _real_extract
KeyError: 'asset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 547, in extract
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('asset',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

URL
https://sportdeutschland.tv/handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1

Installed youtube-dl via chocolately today
Windows

The fix for #30348 finds the video:
$ python -m youtube_dl -v -F 'https://sportdeutschland.tv/handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://sportdeutschland.tv/handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 675e4ca6d
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[SportDeutschland] handball-bundesliga-frauen/hbf-sg-bbm-bietigheim-vs-buxtehuder-sv-1: Downloading JSON metadata
[SportDeutschland] 9454a4b2-1f95-49fd-8152-cf9f6eb72478: Downloading SMIL file
[SportDeutschland] 9454a4b2-1f95-49fd-8152-cf9f6eb72478: Downloading m3u8 information
[info] Available formats for 9454a4b2-1f95-49fd-8152-cf9f6eb72478:
format code  extension  resolution note
hls-1        mp4        unknown    avc1.4d401f, 50.0fps, mp4a.40.2

The modified extractor is available from this user's fork: https://github.com/FriedrichRehren/youtube-dl/raw/aa0b30825bdba36b3337255b534bfbe6efea9092/youtube_dl/extractor/sportdeutschland.py.
If Chocolatey installed yt-dl as a Python module, you can find the youtube_dl/extractor/sportdeutschland.py file and replace it. Otherwise (if yt-dl is a single ~2MB executable) uninstall yt-dl, install Python and use pip to install yt-dl as a module.


Add support for English versions of text items

[ ] Dataset title
[ ] Dataset description
[ ] Distribution title
[ ] Data service title
[ ] Keywords

Everything except keywords can be done by adding a second field (on the same row).
Keywords will be tricky. We could add a second row of keywords in English, however, we would not know which Czech keyword is which English keyword. Instead, we will need to be able to add, edit and remove "multilingual keyword" in a dialogue, which will record that the keyword object has a Czech and an English label. See this form and its source.
Also see https://github.com/opendata-mvcr/nkod/issues/2

 Druhá várka vylepšení formulářů směrem k DCAT-AP 2.0.0

I would place the button on the same line as the text inputs and choose an appropriate icon for the button (also for the spatial coverage). This way it is unintuitive.


Help With Enabling Non-Operators to Use Signs

When testing the commandsigns I found that when I deopped myself (because the server isn't open to the public yet) I couldn't use a sign which teleports me to a certain part of my server. If you could help me fix this issue it would be appreciated. Thanks.

https://github.com/Nokorbis/ar-command-signs/issues/54


Material Bug

How to fix ?

that looks like inverted normals, select all of the mesh and press alt-n and try flipping it to see if that works

Flip normals is not work :(

Sorry to revive dead thread, but this is caused by blending alpha or material settings to do with the alpha. Blender does terribly with blend alpha and is yet to fix it


maps/lbmap: fix maglev test suite build

Building the tests of this package currently fails with:
# github.com/cilium/cilium/pkg/maps/lbmap [github.com/cilium/cilium/pkg/maps/lbmap.test]
pkg/maps/lbmap/maglev_test.go:55:3: unknown field 'MaxEntries' in struct literal of type InitParams

Fix this by adjusting to the new InitParams member name for the maglev
map.
Fixes: 3798f32d8b2f ("daemon: Split --bpf-lb-map-max into multiple options")

/test-runtime

/test-runtime

Runtime tests passed, merging.


Containers Chargeback crashing on missing entity_id

Trying to create a chargeback report for container projects I get:
[----] E, [2016-04-20T00:15:07.111687 #19250:47b968] ERROR -- : [RuntimeError]: must provide option :entity_id  Method:[rescue in _async_generate_table]
[----] E, [2016-04-20T00:15:07.111881 #19250:47b968] ERROR -- : /home/simon/src/manageiq/app/models/chargeback_container_project.rb:42:in `build_results_for_report_ChargebackContainerProject'
/home/simon/src/manageiq/app/models/miq_report/generator.rb:201:in `_generate_table'
/home/simon/src/manageiq/app/models/miq_report/generator.rb:176:in `block in generate_table'
/home/simon/src/manageiq/app/models/user.rb:260:in `with_user'
/home/simon/src/manageiq/app/models/miq_report/generator.rb:176:in `generate_table'
/home/simon/src/manageiq/app/models/miq_report/generator/async.rb:96:in `_async_generate_table'
/home/simon/src/manageiq/app/models/miq_queue.rb:347:in `block in deliver'
/home/simon/src/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
/home/simon/src/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
/home/simon/src/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `catch'
/home/simon/src/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:32:in `catch'
/home/simon/src/rvm/rubies/ruby-2.2.4/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
/home/simon/src/manageiq/app/models/miq_queue.rb:343:in `deliver'
/home/simon/src/manageiq/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
/home/simon/src/manageiq/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/home/simon/src/manageiq/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'
/home/simon/src/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
/home/simon/src/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
/home/simon/src/manageiq/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/home/simon/src/manageiq/app/models/miq_worker/runner.rb:331:in `loop'
/home/simon/src/manageiq/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/home/simon/src/manageiq/app/models/miq_worker/runner.rb:153:in `run'
/home/simon/src/manageiq/app/models/miq_worker/runner.rb:128:in `start'
/home/simon/src/manageiq/app/models/miq_worker/runner.rb:21:in `start_worker'
/home/simon/src/manageiq/app/models/miq_worker.rb:346:in `block in start'
/home/simon/src/rvm/gems/ruby-2.2.4/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/home/simon/src/rvm/gems/ruby-2.2.4/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/home/simon/src/manageiq/app/models/miq_worker.rb:344:in `start'
/home/simon/src/manageiq/app/models/miq_worker.rb:274:in `start_worker'
/home/simon/src/manageiq/app/models/miq_worker.rb:154:in `block in sync_workers'
/home/simon/src/manageiq/app/models/miq_worker.rb:154:in `times'
/home/simon/src/manageiq/app/models/miq_worker.rb:154:in `sync_workers'
/home/simon/src/manageiq/app/models/miq_server/worker_management/monitor.rb:49:in `block in sync_workers'
/home/simon/src/manageiq/app/models/miq_server/worker_management/monitor.rb:47:in `each'
/home/simon/src/manageiq/app/models/miq_server/worker_management/monitor.rb:47:in `sync_workers'
/home/simon/src/manageiq/app/models/miq_server.rb:173:in `start'
/home/simon/src/manageiq/app/models/miq_server.rb:264:in `start'
/home/simon/src/manageiq/lib/workers/evm_server.rb:65:in `start'
/home/simon/src/manageiq/lib/workers/evm_server.rb:92:in `start'
/home/simon/src/manageiq/lib/workers/bin/evm_server.rb:4:in `<main>'

cc @zeari
Let me know what other information you need.

@miq-bot add_label providers/containers, chargeback, bug

@zeari is this fixed? Can you mention the PR (if any) here. Thanks.

@zeari the fix should be #8154 correct?

Yes, we can close this issue.
This was due to malformed reports since the UI didnt enforce having an entity_id.

Fixed in #8154


angular 2+ version?

it is planned?

also looking for this. could i update this myself and do a PR?


How use it?

$reader = new Xliff\Reader(); $reader->fromUri('MyDic.en-US.xlf'); $data = $reader->parse(); print_r($data); //null
$data null what wrong?

I think your code incorrectly parses files from microsoft multilanguage editor (https://developer.microsoft.com/en-us/windows/develop/multilingual-app-toolkit).

Or maybe you are using an other version of Xliff which currently not support by mime tools. Currently 1.2 and 2.0 are supported. However, please check your file's content following format accordingly.


Primary link can be automatically pulled from Wikidata

...specifically, from P856. That would avoid redundancy and have us rely on Wikidata, which is more likely to be updated more frequently.

A bit tricky but should be doable. Great suggestion, thanks!


Update TotalSpaces to 2.3.6

Thank you for the contribution. It was merged directly as f5c480c698035a23666a89afe9168cae910724aa to keep commit history cleaner. Your contribution is still credited to you.


Unnecessary import in int_enum.rs

warning: unused import: `flagset::FlagSet`
 --> src/int_enum.rs:2:5
  |
2 | use flagset::FlagSet;
  |     ^^^^^^^^^^^^^^^^

Which also means that the flagset part of the transpiler isnt being tested.

Creating a Cargo.toml with:
flagset = "0.4.0"

resolves some of the issues. Whether use flagset::FlagSet; is needed or not depends on whether the python code is doing set operations with flags. Yes, the transpiler could be smarter and add this to usings only as necessary.

I already have a Cargo.toml with that in it, hence only the warning.
I am hoping that when https://github.com/stevedonovan/runner/issues/8 and https://github.com/stevedonovan/runner/issues/7 are fixed we can use it to compile the generated code.


Mocks documentation

Is there documentation anywhere for how to get ng-cordova to use the mocks instead of standard plugins for testing? I've added mocks of the ble.js plugin but I'd like to test before I submit a PR.

Figured it out. See #702


File search function error in some datasets

Search function error in some datasets such as IDEA_TEST or Hackathon_test.
For instance, when typing the letter "i" in these datasets, the front-end shows nothing.
This error might be because of some invalid fields in some files in the datasets.

I think this is because the value of fields in some files are INVALID, the easiest solution is to delete such files directly as they don't have meaningful contents.
I guess this is because at the moment of Hackathon, the validation of file name only happened on the frontend, so when the user tried to update via APIs, they could upload some illegal files. After the Hackathon, the validation is also performed on the backend so there would be no such errors again.


Change: Use simdjson's ondemand API

In 0fe58d757627d0c3cac9ad36a5e60b29281483d9 we switched from on-demand -> DOM API as it was slightly faster. This switches back to the on-demand API while using a different approach to parsing to hopefully increase speeds over the current code using the DOM API.
Design
The first iteration of the new parsing interface also helped with the DOM API, which is why that was merged separately already. However, this patch also tries to port that idea of iterating over all fields linearly in every function. Because currently, we only iterate over fields for the root of the JSON document and use hashed strings to speed up the switch. For every other function the order of fields may still be in the exact opposite (worse case) as how we read from them, which is why I want to use this idea everywhere in fastgltf.
Performance
Performance has gotten worse which is why this is a draft PR and I will look into this drawback later on. From a quick profiling run it seems like the ondemand::parser::iterate function is taking a considerable amount of time while the rest executes within at most 500microseconds.

I'm on my Windows machine right now, so I can't give much info.

If you are compiling under Windows, I recommend using Clang if possible. See
https://lemire.me/blog/2023/03/03/float-parsing-benchmark-regular-visual-studio-clangcl-and-linux-gcc/

If you are compiling under Windows, I recommend using Clang if possible. See
https://lemire.me/blog/2023/03/03/float-parsing-benchmark-regular-visual-studio-clangcl-and-linux-gcc/

Thanks for the read. I've gone ahead and I've just run my benchmarks using Clang-CL 16 (I used clang-cl from the LLVM downloads). It has actually increased the runtime speed of the fastgltf cases by often fourfold with both master and this branch. Though this PR's branch is still lacking behind consistently.

It has actually increased the runtime speed of the fastgltf cases by often fourfold with both master and this branch.

I see something like a factor of two... but yeah, Visual Studio has disappointing performance in some cases I care about.

@lemire I've revisited this PR because I want to add support for #41, which I think I want to do using raw JSON tokens, as provided by the on demand API.
Using verbose logging I already managed to restructure my code to have no skips at all during the entire parsing process. I've looked through some things using Xcode Instruments, and have found some issues. I fixed some of the issues with allocations and vector resizing where it wasn't needed, but this one stumps me (there's more than just the preview shown, and ignore the return at the top of the function I accidentally committed that):
https://github.com/spnda/fastgltf/blob/fc5b3c2e9a105bed89332f6187b3d3d41a4a69eb/src/fastgltf.cpp#L1330-L1400
This lambda function takes up 36% of the entire average runtime of the parseAccessors function. It essentially just reads a number from an array, and then checks if it's a double, float, or integer and then converts them appropriately. Is there some better way to do this with simdjson, so that its runtime is not so excessive? Do you have any suggestions on improving that function?

@spnda Having a look.

This lambda function takes up 36% of the entire average runtime of the parseAccessors function. It essentially just reads a number from an array, and then checks if it's a double, float, or integer and then converts them appropriately. Is there some better way to do this with simdjson, so that its runtime is not so excessive? Do you have any suggestions on improving that function?

The whole function does a lot of complicated work.
If you turn it into something like ...
  auto v = std::get<double_vec>(variant);
  for (auto element : array) {
    // error handing as needed
    v.push_back(element.get_double());
  }

You should be pretty close to be limited by the number parsing speed. If not, then you have other overhead.
My recommendation is to try to break down your code into simpler functions. There is a lot going on.
It may help also to include a standard benchmark as part of the project. If I could run a benchmark easily, I might be able to give you more insights. Just looking at high level code buys only so much. I need to know what the code is doing in practice.

This part is important: I recommend including a standard benchmark as part of your project. If I could just do...
cmake --build build
./build/benchmark

That would tremendously useful. Not just to me but to anyone who cares about performance.

Feel free to steal some of my benchmarking code:
https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/tree/master/2023/11/28/benchmarks

My recommendation is to rely more on experiments and repeated high-quality benchmarks, than on profiling. Profilers mislead us all the time. They may point at a potential direction, but that's all. There are exceptions. If you do pure numerical analysis, you may find 3 lines of code where your code is spending 99% of its time, and then you rewrite them in assembly and you are done... but for a problem such as this one, experiments are very likely to be the way forward. One needs to gain insights into what is costing what.
For example, how much does the get_number() function costs as opposed to get_double()? This takes just one experiment to find it out.

This part is important: I recommend including a standard benchmark as part of your project. If I could just do...
cmake --build build
./build/benchmark

That would tremendously useful. Not just to me but to anyone who cares about performance.

I already include benchmarks (which I am also using right now locally) which use the Catch2 testing framework. I should probably add something about how to build tests and run them in the README...
Though this is how you'd build and run the benchmarks. Note that these do require external downloads of sample assets because I currently just bench an entire run over a JSON asset. I mostly just wrote them to work well for me locally and for the CI.
python fetch_test_deps.py
cmake -B build -DFASTGLTF_ENABLE_TESTS=ON
cmake --build build --target fastgltf_tests
build/tests/fastgltf_tests "[gltf-benchmark]"

(note that the quotation marks in the last line might just be a requirement of zsh which I'm using locally, so you might have to remove those).

I already include benchmarks

The parseAccessors is tiny component of this benchmark.

@spnda Here is my recommandation.
Take your current code, do all the parsing and error handling, but do not touch your data structures. Do not call reserve, emplace_back, push_back at least as far as the numbers are concerned.
Next do the reverse. Do not call get_number... iterate through the array, but do not parse the values. Just add random numbers in your data structures (say random numbers between 0 and 1).
And then compare with the current code.
Lay out the three results. This should tell  you a lot about where the time goes.
My generally point is that you need to run experiments, and collect data.

@spnda Here is my recommandation.
Take your current code, do all the parsing and error handling, but do not touch your data structures. Do not call reserve, emplace_back, push_back at least as far as the numbers are concerned.
Next do the reverse. Do not call get_number... iterate through the array, but do not parse the values. Just add random numbers in your data structures (say random numbers between 0 and 1).
And then compare with the current code.
Lay out the three results. This should tell you a lot about where the time goes.
My generally point is that you need to run experiments, and collect data.

Well, big thanks for taking the time to look over my project. I'll do some testing along the lines of what you suggested. I'm not knowledgeable about these low-level optimizations, so much of this is still new to me. Using profiling I managed to find a few issues in my code apart from the parseMinMax lambda, which has already had a big impact on the ultimate speed. The original question, of why DOM was faster, is mostly solved actually, or at least on my arm64 M3. I still expect the ondemand code to be quicker than it is currently, but any improvement is welcome right now. And on my 5800X I still have a measurable performance regression in all cases. I guess I'll just have to do more investigation on both chips at the same time to see what differences there are to cover both. Another user had also suggested on Discord that the slowdown might be cache/memory related, which I have yet to test/confirm.

@spnda The reason I raise these questions is that software performance is complex and requires hard numbers and experimentation.
For example, just by tracing the code without too much attention, I noticed that you are often enough storing a single integer in a std::vector which is itself a std::variant. This is fine if that's what you need, but constructing these data structures is expensive. And parsing routine themselves may not carry much weight.
But it is impossible to tell without experimentations and profiling won't tell you.


Chromium download always stuck at 99%

.../node_modules/puppeteer ❯❯❯ npm install
><EMAIL_ADDRESS>install .../node_modules/puppeteer
> node install.js

Downloading Chromium r686378 - 110.2 Mb [====================] 99% 0.0s

Always stuck. On latest yarn or npm. Removed cache and all lock files, still no difference.

@binarykitchen are you on windows by any chance? I've seen this happen because of eager firewalls in the past

No, on MacOS here. Firewall issue? Then it shouldn't block at 99%, but rather at the beginning of the download.
I'd double check code in install.js for robustness, error handling, timeouts and so on.

Still having the same issue :(

Try installing it as tarball file

Left a yarn install running only to find that the Puppeteer install script had been running for 12 hrs without gracefully failing. Have tried yarn cache clean to no avail.


Changing the npub on opreturn to event id

The event id is the true comitment to the project information and rules set by the founder so that is what will be published on the blockchain

This is goin to break testnet so I am gonna reset testnet and push the indexer changes as well


Prime

What the program do ?
Finds prime factors
In what programming language it is written?
C

This pull request contains conflicts. Please fix it and reopen


Using open-props fails when switching to the experimental esBuild

Description
When I have an app that uses open-props it fails to build when I switch to the esBuild builder.
I create a small reproduction repo
Minimal Reproduction
pull this small reproduction repo  and run ng build
it will fail like this:
While `ng serve works and build with the webpack builder also works. (and both do pull in the asked-for CSS!)
Exception or Error

Your Environment
Angular CLI: 15.0.0-next.4
Node: 18.4.0 (Unsupported)
Package Manager: npm 8.18.0
OS: linux x64

Angular: 15.0.0-next.5
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1500.0-next.4
@angular-devkit/build-angular   15.0.0-next.4
@angular-devkit/core            15.0.0-next.4
@angular-devkit/schematics      15.0.0-next.4
@angular/cli                    15.0.0-next.4
@schematics/angular             15.0.0-next.4
rxjs                            7.5.7
typescript                      4.7.4

From examining the package, it appears that the error is legitimate.  The open-props package contains an exports field which prevents deep imports and only allows imports from listed paths.  In this case, the path suggested (open-props/style) should be used and will cause the build to succeed.
There, however, is a bug on the Webpack side where the suggested path does not work but should.  Further investigation is in progress to resolve this for the Webpack build.

@clydin Right!
Because it worked with webpack I assumed an esbuild problem, instead of looking into it. (to be fair, the imports for open-props did change too, mine where the "old" way of importing)
Thaks for the pointer!


updates related to FRE notice

ME-180463890

Updated aca_entities gem version
Updated preview payload to include health and dental enrollments
Fixed bug related to missing contact_method information in some cases
Updated verabiage for Cub Care in application settings

@markgoho like ?

@Battula this is up to you as the developer to describe your changes in a way that conveys meaning and summarizes what's changing about the code
what you've written amounts to "made changes" -- yeah, okay, that's what a PR does, but what changes are you making?
If you're finding it difficult to summarize the changes, it's possible you've put too much in the PR and it needs to be broken out  in to separate changes, each able to be independently reviewed

@markgoho I have already summarized the list of changes in the commit message.
If you're finding it difficult to see the list of changes that I made, please click here.
The changes aren't too big to break them down in separate PRs. Thanks

@Battula I'm not finding it difficult to understand the changes, I'm finding it difficult to see how "made changes" properly summarizes a number of distinct changes
options:

find a better title that better summarizes the changes
break the PR up into multiple PRs

@markgoho I can't summarize all the changes in the PR title, so I came up with a generic title and summarized the changes in the comment section.

@Battula please submit PRs for each of these:

Updated aca_entities gem version
Updated preview payload to include health and dental enrollments
Fixed bug related to missing contact_method information in some cases
Updated verabiage for Cub Care in application settings

@markgoho What's wrong with this pull request ?

@Battula this PR clearly combines 4 different changes to the codebase that each need their own PR
here are the PRs I expect to see:

Updated aca_entities gem version
Updated preview payload to include health and dental enrollments
Fixed bug related to missing contact_method information in some cases
Updated verabiage for Cub Care in application settings

All the changes are related to FRE notice, and I don't see why they can't be on a single pull request. @markgoho
They aren't too big to split them into separate PRs.

@markgoho Please don't close this PR, we need this to be merged to trunk. Thank you!


Create gnzh theme and vundle plugin

Theme based on bira, display ruby version, git branch, root or normal user, ssh connection or local, exit codes. Preview http://dl.dropbox.com/u/4109351/pics/gnzh-zsh-theme.png

Awake at 4am by Proem
Merging in...3...2...


Parse delayed name in death_handler

This allows users who are still using the delayed extension to better monitor the dead queue by exporting DelayedAction#foo and DelayedModel#action instead of just Sidekiq::Extensions::DelayedClass.

Thanks seems safe.


Unable to call the API locally

Describe the bug
I can now use docker-compose to launch langflow locally.
Currently, I can configure the following flow to do the build.

Also, we were able to absorb conversations with AI.

We are currently considering using this from an API rather than a GUI.
Referring to the commands shown in the documentation and GUI, I executed the following command.
curl -X POST \
  http://localhost:3000/api/v1/process/32f1077c-70dc-49c0-8006-0c64ef8b02e6 \
  -H 'Content-Type: application/json'\
  -H 'x-api-key: sk-elo0IAfBfiMQkdpOZrw_6Hgxd1Pdo2zg22mD4BhCqIk'\
  -d '{"inputs": {"input":""}, "tweaks": {
  "AmazonBedrock-1rYBo": {},
  "ConversationChain-FFwTG": {}
}}'

The response is as follows.
{"detail":"Error: One input key expected got ['repr', 'input']"}

Browser and Version

Browser chrome
Version v0.5.5

To Reproduce
Steps to reproduce the behavior:

Build a flow
Generate an API key
Invoke curl command
See error

Hey @yosse95ai
Thanks for reporting this. Could I just ask you to update to the latest version?
I'll look into it and report soon.

Hi @ogabrielluiz
Thank you for your answers and development.
This issue was addressed by updating to the latest version.

Awesome!


CNI setup is only run in weave setup not weave launch

This is weird; we don't advertise weave setup as something you have to run.

I suggest we add this to launch_router.

we don't advertise

It is clearly stated on the CNI docs page https://www.weave.works/docs/net/latest/cni-plugin/

Yes, I know that. It still breaks the model.

We should ask the inverse question: is there any downside to running setup_cni in launch_router?


screenshot upscale

Checklist

[x] I've searched the issues and pull requests for similar looking suggestions.
[X] I've checked the Unreleased section of the changelog for newly added features that sound like my suggestion.

Describe your Suggestion
I feel like the send screenshot in discord is way too small. I have tried upscaling and going above 100% but it wont increase in size. Would be really cool for old people if the upload was just a little bit bigger. So i don't have to use magnifying glass to see the screenshot : D
Reasoning
No response

Different elements in the screenshot would benefit from different types of upscaling, us doing a full upscale would most like not look very good. Not completely against this, it's just something I'm not sure will give the results you want

Do you have any example of discord embeds larger than what we currently use so the image could appear larger?
The only one I've interacted with does not have any sized images above emoji sized, the increased width just comes from extra info being included.

e.g.

this has a slightly larger width than we do, but I don't believe even that would give you the outcome you want.

we can't really control how large the screenshot within the embed is

if you disable rich embeds, it does render a bit larger, but yeah you'll still want to click on the image, press Open in Browser to get a larger view (and sometimes discord appends query parameters to the image url that make it smaller, yuo can remove those)

@Erutpur could you clarify whether you mean the size of the screenshot within the rendered embed, or the size of the screenshot after you click Open in Browser?

If the former, we can't do anything about it
If the latter, feel free to re-open for consideration


Feature: Lambda Layers

Leaving this for consideration and see if we have customer traction as we go GA - Not necessarily a blocker for GA but could be helpful to customers.
If you'd like this to be available for all Python runtimes supported in addition to PyPi, please leave a 👍

Having an AWS managed & hosted lambda layer for this would be greatly appreciated!

In the meantime, this is available in the super helpful https://github.com/keithrozario/Klayers repo.

@mwarkentin True that. Unfortunately not usable in an enterprise context since this source is not safe to work with. An official layer from AWS would really help without the heavy lifting from customers site.

@am29d is currently working on it :) We're currently working on automating the Lambda Layers publishing pipeline to match our release process.

Awesome!

Can't wait!! :)

hey everyone - This is now AVAILABLE on SAR as of 1.4.0: https://github.com/awslabs/aws-lambda-powertools-python/releases/tag/v1.4.0
We've also taken the time to update the release process to ensure both PyPi and SAR semantic versioning match, so you can trust the CHANGELOG as you would normally :)
Docs: https://awslabs.github.io/aws-lambda-powertools-python/#lambda-layer
cc @keithrozario @markusbecker  @mwarkentin @djk

Is anyone aware of similar functionality for Serverless Framework that would allow us to embed the SAR app like this?

Hi @mwarkentin , the Serverless Framework can handle the Transform directive since v1.4.1:
https://github.com/serverless/serverless/pull/5997
This should like this:
resources:
  Transform: AWS::Serverless-2016-10-31
  Resources:
    AwsLambdaPowertoolsPythonLayer:
      Type: AWS::Serverless::Application
      Properties:
        Location:
          ApplicationId: arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer
          SemanticVersion: 1.3.1 # change to latest semantic version available in SAR

@am29d would it make sense to add a section about this to the docs? I'm sure there are lots of Serverless Framework users who would be interested in using this installation method.


pragma version error

please change var and other method in code i use 0.5.16 truffle version and it give error when i compile simplestorage.sol contract.
function createNewMultiSigbyUser(address instituteaddress) public {
   // var wa = wallets[msg.sender];
   // wa.inst=instituteaddress;
   // wa.stud=msg.sender;

}
function uploadAadhar(string memory a) public{
// var wa=wallets[msg.sender];
change in both function

@ashhadali10 Did you find any alternative?

@nikhildsahu please help in above problem, for the var wa = wallets[msg.sender];, it says, expected primary expression. i am not able to resolve it, please help

@nikhildsahu please help in above problem, for the var wa = wallets[msg.sender];, it says, expected primary expression. i am not able to resolve it, please help

instead of var wa = wallets[msg.sender]
use Multisig storage wa = wallets[msg.sender]

function createNewMultiSigbyUser(address instituteaddress) public {
    Multisig storage wa = wallets[msg.sender];
    wa.inst=instituteaddress;
    wa.stud=msg.sender;

}
function uploadAadhar(string a) {
 var wa=wallets[msg.sender]; //im getting error right here someone help me

}
}

Hey @ashhadali10 , @Tabinda788 and @Stranger-18, refer this code it's working:
pragma solidity >=0.4.24;

contract Ballot {
   struct Documents {
     string aadhar;
}

struct MultiSig
{
    address inst;
    address stud;
    Documents documents;
}
mapping(address=>MultiSig) public wallets ;

function createNewMultiSigbyUser(address instituteaddress) public {

        MultiSig storage wa = wallets[msg.sender];
        wa.inst=instituteaddress;
        wa.stud=msg.sender;
}

function uploadAadhar(string memory a) public{
    MultiSig storage wa=wallets[msg.sender];
}
}

@nikhildsahu now  you can close this issue.

Hey @ashhadali10 , @Tabinda788 and @Stranger-18, refer this code it's working:
pragma solidity >=0.4.24;

contract Ballot {
   struct Documents {
     string aadhar;
}

struct MultiSig
{
    address inst;
    address stud;
    Documents documents;
}
mapping(address=>MultiSig) public wallets ;

function createNewMultiSigbyUser(address instituteaddress) public {

        MultiSig storage wa = wallets[msg.sender];
        wa.inst=instituteaddress;
        wa.stud=msg.sender;
}

function uploadAadhar(string memory a) public{
    MultiSig storage wa=wallets[msg.sender];
}
}

@nikhildsahu now you can close this issue.

there is still issue in the migration.sol file. the same version problem. do you have a fix for that as well?


github-cli binary is conflicting with aliases

Expected Behavior
I should be able to run gh command for github-cli
Current Behavior
I am not able to run the command

Possible Solution

Context
My whole development workflow is disrupted. Need immediate action on this plz
Steps to Reproduce

follow the steps shown in screenshot
Your Environment

Bash-it version used:
List of enabled plugins, themes and aliases (use bash-it show (plugins/themes/aliases)):
bash-it doctor output:
Bash version:
Operating System and version:

Your Bash Config File

  # Your bash config file should be here

Is this happening when you open up a new shell? We have a check in place if gh already exists before adding the alias

Is this happening when you open up a new shell? We have a check in place if gh already exists before adding the alias

Is this happening when you open up a new shell? We have a check in place if gh already exists before adding the alias

Yes

Is this happening when you open up a new shell? We have a check in place if gh already exists before adding the alias

Yes

Which version of Bash-it are you using? Are you sure you did not modify it? This seems strange as we have an exact check for it.
On another thought, maybe the gh binary is not on your path?

Which version of Bash-it are you using? Are you sure you did not modify it? This seems strange as we have an exact check for it.
On another thought, maybe the gh binary is not on your path?

@NoahGorny I

Which version of Bash-it are you using? Are you sure you did not modify it?

Yes, and yes
I have clone repo to another directory and working from there

@NoahGorny I

Which version of Bash-it are you using? Are you sure you did not modify it?

Yes, and yes
I have clone repo to another directory and working from there

@tbhaxor please pull from the latest master, set it up and open up a terminal. Lemme know if the problem persists

@tbhaxor please pull from the latest master, set it up and open up a terminal. Lemme know if the problem persists

ping @tbhaxor

ping @tbhaxor

It's fixed. I guess you are using _is_binary

It's fixed. I guess you are using _is_binary


How to upgrade the HLS.js version in Clappr 0.13.x?

I need to upgrade the hls.js version used in latest clappr 0.13.x version. How can I do that?

@joaopaulovieira @leandromoreira @jhonatangomes If you guys could clarify how can I do this (the steps, I mean), it will be awesome 🎉 🤝

I think the steps is these below. Is that correct?
1 - Fork the repo
2 - Edit the file https://github.com/clappr/clappr/blob/0.3.13/package.json#L52, changing version of hls.js
3 - Edit the file https://github.com/clappr/clappr/blob/0.3.13/package.json#L3, changing version of clappr
4 - Run npm run release
5 - Use dist/clappr.js inside project
Right?

To rebuild Clappr with latest hls.js version, clone Clappr repository and create a local branch using tag version (for example 0.3.13) :
git clone https://github.com/clappr/clappr.git
cd clappr
git checkout tags/0.3.13 -b 0.3.13

Then you can use npm or yarn to change hls.js dependency version. (and install dev. dependencies)
Finally you can use npm run build to compile unminified version, or npm run release to compile minified version.

@kslimani Thanks for feedback! I will test here these steps that you described 🎉


Fix indexing

Had not specified the right parent (Credentials) for it to show up on Pages. Fixed now.

The page wasn't showing up on GitHub pages, hoping this fixes it.


asan: container-overflow in ParagraphTest.CenterAlignParagraph

ASAN_OPTIONS=detect_leaks=0 ../out/host_debug_unopt/txt_unittests --gtest_shuffle --gtest_repeat=2 --gtest_filter="ParagraphTest.CenterAlignParagraph"

Repeating all tests (iteration 1) . . .

Note: Google Test filter = ParagraphTest.CenterAlignParagraph
Note: Randomizing tests' orders with a seed of 42294 .
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ParagraphTest
[ RUN      ] ParagraphTest.CenterAlignParagraph
=================================================================
==84933==ERROR: AddressSanitizer: container-overflow on address 0x60c000022300 at pc 0x000106aba5cd bp 0x7ffee921c190 sp 0x7ffee921c188
READ of size 4 at 0x60c000022300 thread T0
    #0 0x106aba5cc in txt::ParagraphTest_CenterAlignParagraph_Test::TestBody() paragraph_unittests.cc:2610
    #1 0x10c3bb5ec in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) gtest.cc:2607
    #2 0x10c35d2af in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) gtest.cc:2662
    #3 0x10c35ce07 in testing::Test::Run() gtest.cc:2682
    #4 0x10c35ee1c in testing::TestInfo::Run() gtest.cc:2861
    #5 0x10c361cc9 in testing::TestSuite::Run() gtest.cc:3015
    #6 0x10c37e0b5 in testing::internal::UnitTestImpl::RunAllTests() gtest.cc:5855
    #7 0x10c3c7527 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) gtest.cc:2607
    #8 0x10c37d516 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) gtest.cc:2662
    #9 0x10c37d06a in testing::UnitTest::Run() gtest.cc:5438
    #10 0x106bb1c52 in RUN_ALL_TESTS() gtest.h:2490
    #11 0x106bb1a59 in main txt_run_all_unittests.cc:46
    #12 0x7fff203fff3c in start+0x0 (libdyld.dylib:x86_64+0x15f3c)

0x60c000022300 is located 0 bytes inside of 128-byte region [0x60c000022300,0x60c000022380)
allocated by thread T0 here:
    #0 0x1110d14bd in wrap__Znwm+0x7d (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x584bd)
    #1 0x106a01016 in std::__1::__libcpp_allocate(unsigned long, unsigned long) new:253
    #2 0x106c4b105 in std::__1::allocator<float>::allocate(unsigned long, void const*) memory:1813
    #3 0x106c4ae3e in std::__1::allocator_traits<std::__1::allocator<float> >::allocate(std::__1::allocator<float>&, unsigned long) memory:1546
    #4 0x106c4abc6 in std::__1::__split_buffer<float, std::__1::allocator<float>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<float>&) __split_buffer:311
    #5 0x106c49aaa in std::__1::__split_buffer<float, std::__1::allocator<float>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<float>&) __split_buffer:310
    #6 0x106c66b88 in void std::__1::vector<float, std::__1::allocator<float> >::__push_back_slow_path<float const&>(float const&) vector:1622
    #7 0x106c5b722 in std::__1::vector<float, std::__1::allocator<float> >::push_back(float const&) vector:1643
    #8 0x106c5af22 in minikin::LineBreaker::pushBreak(int, float, unsigned char) LineBreaker.cpp:358
    #9 0x106c5c250 in minikin::LineBreaker::computeBreaksGreedy() LineBreaker.cpp:399
    #10 0x106c5edb1 in minikin::LineBreaker::computeBreaks() LineBreaker.cpp:524
    #11 0x106ce2f42 in txt::ParagraphTxt::ComputeLineBreaks() paragraph_txt.cc:358
    #12 0x106ceeace in txt::ParagraphTxt::Layout(double) paragraph_txt.cc:686
    #13 0x106ab8fa0 in txt::ParagraphTest_CenterAlignParagraph_Test::TestBody() paragraph_unittests.cc:2591
    #14 0x10c3bb5ec in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) gtest.cc:2607
    #15 0x10c35d2af in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) gtest.cc:2662
    #16 0x10c35ce07 in testing::Test::Run() gtest.cc:2682
    #17 0x10c35ee1c in testing::TestInfo::Run() gtest.cc:2861
    #18 0x10c361cc9 in testing::TestSuite::Run() gtest.cc:3015
    #19 0x10c37e0b5 in testing::internal::UnitTestImpl::RunAllTests() gtest.cc:5855
    #20 0x10c3c7527 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) gtest.cc:2607
    #21 0x10c37d516 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) gtest.cc:2662
    #22 0x10c37d06a in testing::UnitTest::Run() gtest.cc:5438
    #23 0x106bb1c52 in RUN_ALL_TESTS() gtest.h:2490
    #24 0x106bb1a59 in main txt_run_all_unittests.cc:46
    #25 0x7fff203fff3c in start+0x0 (libdyld.dylib:x86_64+0x15f3c)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.

Frame 0: 0x7fff20339406 abort
Frame 1: 0x1110e5366 __sanitizer::Abort()
Frame 2: 0x1110e4a74 __sanitizer::Die()
Frame 3: 0x1110c9c91 __asan::ScopedInErrorReport::~ScopedInErrorReport()
Frame 4: 0x1110c9519 __asan::ReportGenericError()
Frame 5: 0x1110ca1a8 __asan_report_load4
Frame 6: 0x106aba5cd txt::ParagraphTest_CenterAlignParagraph_Test::TestBody()
Frame 7: 0x10c3bb5ed testing::internal::HandleSehExceptionsInMethodIfSupported<>()
Frame 8: 0x10c35d2b0 testing::internal::HandleExceptionsInMethodIfSupported<>()
Frame 9: 0x10c35ce08 testing::Test::Run()
Frame 10: 0x10c35ee1d testing::TestInfo::Run()
Frame 11: 0x10c361cca testing::TestSuite::Run()
Frame 12: 0x10c37e0b6 testing::internal::UnitTestImpl::RunAllTests()
Frame 13: 0x10c3c7528 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
Frame 14: 0x10c37d517 testing::internal::HandleExceptionsInMethodIfSupported<>()
Frame 15: 0x10c37d06b testing::UnitTest::Run()
Frame 16: 0x106bb1c53 RUN_ALL_TESTS()
Frame 17: 0x106bb1a5a main
Frame 18: 0x7fff203fff3d start
Frame 19: 0x4 0x0

We should fix issues found by these new variants. But if you want to enable sanitizers on CI, we can add these to the suppressions lists in //flutter/testing/<foo>_suppressions.txt. That way, we can test for new failures while we fix the current ones.


branch: pixi.js

Please describe. Possibly related to https://github.com/phetsims/scenery/issues/419.

I'm not familiar with it.

This branch demonstrates an exciting but possibly impractical way to convert shapes to pixi graphics.  If/when we continue with pixi we may wish to review this again.

I described it above.  Leaving issue open since the branch still exists.  Unassigning.


Unable to send email notification without SMTP authentication

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):

Deploy Rancher v2.2.0
Add an email notifier with the following settings
Port: 25
Use TLS: unchecked
Username: << blank >>
Password: << blank >>

Result:
level=error ts=2019-05-06T19:59:34.526761127Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="missing password for PLAIN auth mechanism"
level=error ts=2019-05-06T19:59:34.526896715Z caller=dispatch.go:177 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="missing password for PLAIN auth mechanism; missing password for PLAIN auth mechanism"
level=error ts=2019-05-06T19:59:44.527164139Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="missing password for PLAIN auth mechanism"
level=error ts=2019-05-06T19:59:44.527238936Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="missing password for PLAIN auth mechanism"
level=error ts=2019-05-06T19:59:44.527322616Z caller=dispatch.go:177 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="missing password for PLAIN auth mechanism; missing password for PLAIN auth mechanism"
Other details that may be helpful:
These settings work in Rancher v2.1.6 using rancher/prom-alertmanager:v0.15.2
Also, this look to be related to https://github.com/prometheus/alertmanager/issues/1721
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.2.0
Installation option (single install/HA): HA

We also have the same bug, I think this is a bug in alertmanager and has been fixed in 0.16.2, the prom-alertmanager in Rancher should be updated to 0.16.2 also.

Can be tested with v2.2.4-rc7

Tested with rancher/rancher:v2.2.4-rc7 (Environment: single install and HA ) from May 22
Verified fixed
Test1:

Freshly deployed rancher:v2.2.4-rc7 and add cluster
Add an email notifier without SMTP authentication
Add an alert
Check the alertmanager version
Check the alertmanager log
Validate cluster and project level alerting

Result1:

Alertmanager version upgrade to rancher/prom-alertmanager:v0.17.0.
Alertmanager does not have error log when SMTP authentication is not set.
Ability to send mail, use this notifier at the cluster and project level.

Test2:

Deploy Rancher:v2.2.0
Add an email notifier without SMTP authentication
Add an alert
Upgrade from v2.2.0 to v2.2.4-rc7, then validate the alert is still working.

Result2:

Manually update the cluster-alerting app and then be able to send alert emails.


add tezos_hash to all main chain operations

Problem
Currently main chain operations have no identifier shared with tezos, so we cannot ensure that an operation already happened.
This PR
It is a temporary fix, it adds the tezos_hash to allow main chain operations to be properly indexed.

It is a temporary fix

What do you think a better fix would be?

Except for that more descriptive issue, everything else looks fine to me.


Completed the paranthesies checker issue assigned to me

Have you read the Contributing Guidelines on Pull Requests?
(Write your answer here.)
Description
(Write your answer here.)
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[ ] I've edited the README.md and link to my code.

Related Issues or Pull Requests
(Write your answer here.)

@Kajol-Kumari @VijayaGB98  could u pls check and review the code

@bagladivyang03 Please make separate PR for different issue.
No two solutions can't be send through single PR


New form page and app structure

Hi guys,
I revamped and refactored the project structure quite a big to be more maintainable and extensible. Pages now live in their own files. I also added a prototype of the new submission form. Not ready by a long shot but I've added some fields to show how it will work and outlined todos.
Let me know:

If you have remarks. Things you like, don't like
If this is working on your PC as well - I've made all paths relative and tried to remove Apple specific things. If that broke something let me know why.
If you want to be involved in implementing some of the features

I will continue working on this the next couple weeks till we have a working form that incorporates everything from here: https://docs.google.com/document/d/1664Ho_kVuOHX-hu8vjXiyp0wjjuX1YNi7q6UnL-9TsM/edit.
Cheers
Robert

@juancastilla @samzipper @dzamrsky @kbefus If you are not familiar with the review process I can also got ahead an first merge this into dev instead of main. I'd just did not want to make major changes to the app without approval.
@juancastilla maybe you could give a quick go-ahead and tell me if I should merge into dev or main.
As an additional update: Kevin will contact CUAHSI if they are able to host the app early or at least provide us with information on what kind of database facilities they have. Because with the new data form we need a proper database to store the data. Storing it in a google table is way too hacky. I'm happy to take charge in doing these things.
I would need support in making the form more beautiful and usable though but maybe @dzamrsky can support me on this. @dzamrsky you could then use it as a blueprint for your efforts as well. Happy to support you there if you like.
Cheers
Robert

Ok I'm assuming that there are no objections to me merging and taking the lead on the next version so I'm going ahead will the merge and I will change some other things in the coming weeks as well without review.


Update README.md

Updated paper and references only. Did not specify the pages of interest.

Looks great Alex! Thank you  : )


[FAQ] How can I display the complete information of the legend when hovering?

As shown in the figure below, the legend text has been omitted. How can I configure it to display the complete text when hovering over it?

Solution
Different chart libraries have different solutions. VChart enables displaying complete text with omitted text by default. Just hover over it to display it.
Results
Online demo：https://visactor.io/vchart/demo/legend/custom-data

Related Documentation
Legend demo: https://visactor.io/vchart/demo/legend/custom-data
Legend tutorial: https://visactor.io/vchart/guide/tutorial_docs/Chart_Concepts/Legend
Legend configuration: https://visactor.io/vchart/option/barChart#legends-discrete.type
github: https://github.com/VisActor/VChart


suggestions for cool name for the project

the current name of the project is not too good. the current name just explaining the project specification. We need a new name for the project.

spotpl is the new name


Fixed Upload File Name Field

What kind of change does this PR introduce? (Bug fix)

Why was this change needed?

#809

Other information:

The Name Field in the Upload Documents was Editable causing wrong file upload name. As a result incorrect document name is stored in the backend.

By Putting "readonly" attribute in the Input Field  solved the problem

@dartpain pls review my PR and Happy New Year :)

Issue is, the filename should be editable

We have to make sure we use edited name in the vectors table


[Question] Current line vs selected item

Hello,
First and foremost, thank you for this amazing tool and all the answers that can be found in the wiki or the existing issues you take time to answer :pray:
I would like to keep a state of selected items (marked ones due to a fzf toggle action) across distinct searches or providers but I can't figure how to differentiate them from the current line depending on the context.
When I select multiple items I can get the selected ones as the first argument of the action and get the current line with require("fzf-lua").get_info()["selected"] (which may or not be part of the selection).
The same logic works when only one item is selected and the cursor is on another line, so far so good.
But when there is only one item involved and the cursor is on the same line I can't figure out how to know if the item is selected or not as fzf puts the current line into selected if no selection is made.
fzf has a focus event but no selected one on which I could rely to maintain a selected items list.
Is it possible at all or should I implement a custom selection independent from fzf one?
Thank you.

As you correctly identified, actions use the default fzf mechanism which prints only the selected items and not the selected item (unless --no-multi is used), run fzf in the shell and select multiple items and you'll see what I mean, only the selected items are printed regardless of the selected line.

fzf has a focus event but no selected one on which I could rely to maintain a selected items list.
Is it possible at all or should I implement a custom selection independent from fzf one?

You can use focus with a custom action but the default fzf "field index expression" is set as {+} which prints the selected items and not the selected line, to achieve what you want you need to set it to {} which translates to  the selected item only, you can do so in two different ways:

(1) Using the actions table:

require("fzf-lua").files({
  actions = {
    ["focus"] = {
      fn = function(sel) print("selected line:", sel[1]) end,
      exec_silent = true,
      field_index = "{}" -- change this to "{+}" to see the difference
    }
  }
})

(2) Using raw args and a custom action

local shell = require "fzf-lua.shell"
local act = shell.raw_action(function(items)
  print("selected", items[1])
end, "{}") -- changes this to "{+}" to see the difference
require("fzf-lua").files({
  fzf_cli_args = "--bind=" .. vim.fn.shellescape(string.format("focus:execute-silent(%s)", act))
})

I may end up pursuing this way if you don't think there is a simpler solution.

Are the purposed solutions I posted not working for your need?

Also, is it possible to trigger fzf native actions as part of the actions options?

Unfortunately the actions are triggered by the output of fzf after the process was already terminated,  however if you use exec_silent or reload (which don’t wait for termination) you might be able to hack you way with prefix, see #807.
Another possible way is via the keymap.fzf table which translates into --bind=.

I don't see how the focus event is enough to know that a single item is selected or not when applying the default action.
Did I miss something?

If you store the currently selected line (perhaps in the opts) and then compare the sited item to the selected array of the default action, if both are the same you can conclude a single item was selected no?

The prefix option perfectly does the job 👍
The previous binding was a suffix but the logic is the same

If a postfix makes more sense I can add an argument for it, lmk.

Thanks @dmy-gh, just in case you (or anybody) needs this in the future https://github.com/ibhagwan/fzf-lua/commit/6589bdd872385d120a1eb3cb85e5d15d37621c1d adds a postfix option to exec_silent or reload type actions.


[ISSUE #5780]Replace DLedgerServer's deprecated getdLedgerLeaderElector method with getDLedgerLeaderElector

Make sure set the target branch to develop
What is the purpose of the change
close #5780
Brief changelog

Replace DLedgerServer's deprecated getdLedgerLeaderElector method with getDLedgerLeaderElector

Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #5781 (887a18e) into develop (646d04f) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             develop    #5781      +/-   ##
=============================================
- Coverage      42.45%   42.42%   -0.03%
+ Complexity      8215     8206       -9
=============================================
  Files           1060     1060
  Lines          73193    73193
  Branches        9598     9598
=============================================
- Hits           31071    31050      -21
- Misses         38221    38233      +12
- Partials        3901     3910       +9

Impacted Files
Coverage Δ

...a/org/apache/rocketmq/broker/BrokerController.java
43.29% <0.00%> (ø)

...tmq/remoting/protocol/body/ConsumerConnection.java
95.83% <0.00%> (-4.17%)
:arrow_down:

...controller/impl/DefaultBrokerHeartbeatManager.java
82.27% <0.00%> (-3.80%)
:arrow_down:

...e/rocketmq/store/ha/autoswitch/EpochFileCache.java
78.47% <0.00%> (-2.78%)
:arrow_down:

...va/org/apache/rocketmq/store/util/PerfCounter.java
64.39% <0.00%> (-2.10%)
:arrow_down:

...ache/rocketmq/proxy/common/ReceiptHandleGroup.java
65.30% <0.00%> (-2.05%)
:arrow_down:

...org/apache/rocketmq/common/stats/StatsItemSet.java
41.79% <0.00%> (-1.50%)
:arrow_down:

...cketmq/broker/schedule/ScheduleMessageService.java
56.91% <0.00%> (-1.14%)
:arrow_down:

...mq/store/ha/autoswitch/AutoSwitchHAConnection.java
71.08% <0.00%> (-1.09%)
:arrow_down:

.../org/apache/rocketmq/store/index/IndexService.java
59.00% <0.00%> (-1.00%)
:arrow_down:

... and 12 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[feat]: Allow the user to disable health checks globally

Prerequisites

[X] I have searched for existing issues that already suggest this feature, without success.

Describe the Feature Request
Either in the UI or CLI, add a setting that allows the user to disable health checks globally
Describe the Use Case
Health checks use a lot of resources and I suspect that they slow things down a lot, particularly on boxes running lots of services. This will both help determine what impact they are having, and also mitigate their effects if the user has something time-sensitive to get done.
Describe Preferred Solution
A setting in the database, or in a file, that disables all health checks.
Describe Alternatives
None
Anything else?
No response

I can add a compiler flag for this, but it is not worth building a ui flow.

Disabling it should not "hold" healthcheck state though, maybe add a new status "disabled" to the state set? or hijack an existing one if there's a sensible option.

if this was ever going out to users, I'd agree. This is really just to prove out whether or not health checks are "causing problems"

Yes, but the last thing we want is our own devs drawing the wrong conclusions because they forgot they left this setting set one way or another and proceeded to understand health check state as normal.

It will also allow us to gather detailed performance data on health checks.

In favor of https://github.com/Start9Labs/embassy-os/issues/2055


Publish v2.2.1

@twodayslate lets publish a new release, since a few fixed have been merged recently.
Since I didn't setup the package manager stuff, I'm not sure how to do it.
Can you maybe tell me how it is done?
Is it as simple as making a new Github release, or is it more involved, like uploading to a package registry?

For CocoaPods you'll have to have to update the version number and push to trunk. https://guides.cocoapods.org/making/making-a-cocoapod.html
For SPM you just create a new release on GitHub.

Hi!
Please push the latest version to the CocoaPods trunk.
You can set up GitHub Actions to publish it automatically on the GitHub Release event. Here is the config for one of our repositories https://github.com/tesseract-one/Tuples.swift/blob/main/.github/workflows/deploy.yml
In this article you can find how to obtain token for CocoaPods trunk: https://fuller.li/posts/automated-cocoapods-releases-with-ci/

I get the error [!] {"name"=>["is not present"]} when trying to obtain the token. To be honest, it's been so long that I don't remember if I even published to cocoapods before, or maybe it was @twodayslate. I would need to have some kind of cocoapods account to be able to get that token, right?

try to register from scratch pod trunk register your@email 'Your Name' --description='CI session'

I think someone used the name BigNumber on CocoaPods already: https://cocoapods.org/pods/BigNumber
In my PR I will update the name of the CocoaPod (framework name in Swift will be left to BigNumber) but for the CocoaPod name I think Swift-BigInt will be ok (repo name)


Windows 7 colors not displaying.

What are the broken graphics on the left.

It was the same bug using "Material Theme" (https://github.com/equinusocio/material-theme).
My fix:

switch to the default theme
restart ST3
switch to the "Material Theme"

Profit ;)

I have the same issue and switching themes didn't help. Command-line testing for ImageMagick from  https://github.com/ggordan/GutterColor/issues/47 went fine, it does create colored circles. However, it just doesn't display them correctly in ST3 gutter.
System info:
Win10x64 (1511 10586.104)
ST3 3103


[SPARK-37760][BUILD] Upgrade SBT to 1.6.0

What changes were proposed in this pull request?
This PR aims to upgrade SBT to 1.6.0 for better Java 17 compatibility.
Why are the changes needed?
SBT 1.6.0 has many improvements. (https://github.com/sbt/sbt/releases/tag/v1.6.0)

Java 17

The Scala version used to compile build.sbt is updated to Scala 2.12.15, which improves the compatibility with JDK 17+.

sbt.TrapExit is dropped due to Security Manager being deprecated in JDK 17. Calling sys.exit in run or test would shutdown the sbt session. Use forking to prevent it

Java 11

Fixes over-compilation of extended classes on JDK 11

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.

Wow, you are fast. Thank you, @HyukjinKwon .

Thank you, @sarutak .

All tests passed. Merged to master.


Register the database context - ERROR

I'm getting the error when getting the database context.
"'IMvcBuilder' does not contain a definition for 'SetCompatibilityVersion' and no extension method 'SetCompatibilityVersion' accepting a first argument of type 'IMvcBuilder' could be found (are you missing a using directive or an assembly reference?) [TodoApi]"
This happens when I add the code on the section Register the database context

Try installing the v2.1.302 SDK. https://www.microsoft.com/net/download
... and make sure the project file has the correct setup ...
https://github.com/aspnet/Docs/blob/master/aspnetcore/tutorials/first-web-api/samples/2.1/TodoApi/TodoApi.csproj


DOCS: Expand content/loaders

Extended content/concepts/loaders section

[x] Expand documentation, providing additional instructions
[x] Added examples to accompany expanded points
[x] Added mikeerickson to list of contributors (also updated output section)

@TheLarkInn Please review and let me know if you like the format used for the loaders section.  I basically expanded on what you have done elsewhere so it should flow the same.

This duplicates content from https://webpack.js.org/configuration/module/. Do we want to go into detail in the "concept" section? IMO, also https://webpack.js.org/concepts/output/ is too long/detailed for the "concept" section.

Agreed. The content is valid, but it can be laid out better. It's enough if the concept article can describe the basic idea and link to other articles that dig into greater detail. You can have ## References with links at the end.

@bebraw @simon04 my approach (and you are free to say otherwise) is to document the minimum examples to get started.
I am teaching a new JR to start using webpack and was met with friction as soon as getting to concepts/loaders as there was literally not even simple examples
It's your call, but I still believe examples where users are getting started

Have you seen https://webpack.js.org/loaders/ ? Probably the main thing missing is a reference between the concept page and that one. The concept page could contain a very basic example (the first one you added) and then defer to that article for more.

@bebraw no, I didn't even know about  this section until I read this initial reply.  I will review this section and revise my edits and provide link accordingly

@mikeerickson Ok, cool.

@bebraw One quick question before I proceed.   In review of this page example (https://webpack.js.org/loaders/) it uses the use long syntax.   Is this the "preferred" method, or is it safe to use the shorter approach as documented in my edits?
  module: {
    rules: [
      {
        test: /\.css$/,
        use: [
          { loader: 'style-loader'},
          {
            loader: 'css-loader',
            options: {
              modules: true
            }
          }
        ]
      }
    ]
  }

vs
module: {
  rules: [
    {test: /\.js$/, loader: 'babel-loader'},
    ...
  ]
}

@mikeerickson Both forms have their use I think. use is handy if you want to pass multiple loaders within a single definition. I have written a little chapter with the key ideas.

@bebraw Would it be prudent and example worthy, to demonstrate the short syntax in the loaders section (along with the use format)?   Or, just link to it from concepts/loaders and be done with it.   Just trying to make it as clear for beginners as possible.

@bebraw In short, in the Via Configuration section, provide both short and extended examples, since this is what is used in the concepts/loaders section

@mikeerickson I would go with a mini example and a "call to action" to encourage to read more from the in-depth article that digs into further details.

@bebraw cool, that is the approach I have just applied  :-)

@bebraw I have updated my PR, please have a look and let me know what you think.

@bebraw Whoops, a product of a copy/paste error.  All fixed :-)

@simon04 Do you want to have a look? I could use a second opinion here.

I'm not convinced:
I would rather take the simple, textual example from the intro paragraph ("you can use loaders to tell webpack to load a CSS file or to convert TypeScript to JavaScript") and develop a code example from there. Done in #736.
Also, it's good to know that there are various options to specify loaders, but I would just quickly sketch them and refer to the configuration page. Done in #736.

@bebraw @simon04 would you like me to adjust my changes, or just let it go (and accept #736)?  I am keen to do as you wish, just tell me what you want (since you are not convinced by my contribution).

@mikeerickson I think we'll accept #736 first and then figure out how to revise. I'll get back to you after that.

If you want, you could rebase it so that the PR retains Omitting Loader Error and Webpack Rules Shorthand Syntax sections. There's a typo in the latter (sytnax) to fix.


Please help!! ERROR IN WINDOWS

I ve followed all the instructions in README.MD but i cannot even test the sample questions.Can anyine figure out how i can solve this?

Ah, for windows you have to write python bot.py

@nanip9997 I believe that is a problem with your API as mine is working fine...

@nanip9997 If someone needs help add me on discord- BionicHuman
#3200
thehappydinoa  helped me fix my issues so I can help anyone else.

@nanip9997 @thehappydinoa  Join our discord server for help during trivia game.
https://discord.gg/87uZscw


Backup job destination

Hey there,
technically the backup location is not 'site/backups' but 'one directory up from accounts', which is why I just spend ten minutes figuring out where the hell my backups are located - in my case accounts resides below storage (along with cache and sessions), so maybe the could

change code
change wiki

.. to calrify this :smile:
As always, I'm not afraid of opening PRs and stuff!
Cheers
S1SYPHOS

added note in wiki


Malformed HTML in auto-generated page "Quantity"

It seems like this auto-generated page has some formatting issues: https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/#Quantity

I see HTML tags in the code block that I'm guessing should be parsed as HTML not code block.

/language en

/kind bug

Thanks
I think this is a duplicate of issue https://github.com/kubernetes/website/issues/35712

/triage duplicate

Indeed. Sorry, I posted quickly and forgot to search for existing issues.


Is an update using Apollo's SubscriptionClient required?

Does this library need to be updated to work with the latest version of Apollo?  If I use this lib's createNetworkInterface, the console is showing HTTPFetchNetworkInterface is being used, and the URI seems to default to /graphql.
Per the Apollo docs, to use "full websocket" (all queries, mutations, and subscriptions over ws):
import { SubscriptionClient } from 'subscriptions-transport-ws';
import ApolloClient from 'apollo-client';

const GRAPHQL_ENDPOINT = 'ws://localhost:3000/graphql';
const client = new SubscriptionClient(GRAPHQL_ENDPOINT, {
  reconnect: true,
});

const apolloClient = new ApolloClient({
    networkInterface: client,
});

I think this library provides an alternative to subscriptions-transport-ws, in collaboration with absinthe_phoenix it uses phoenix channels to do all graphQL interactions.

@bjunc Yes, as noted by @arjan, APW is just a complete alternative to subscriptions-transport-ws, in your code you should use APW's own networkInterface and give it to your ApolloClient upon creation :).

And it works great, by the way, thanks @vic 👍 Must say I have not begun using subscriptions, but doing queries / mutations over WS is already very smooth :-)

Not sure what about my setup is different than yours @arjan, but I wasn't able to get this lib working.  It might have had something to do with #14, not quite sure.
Ultimately, I rolled my own called Phoenix Websocket Network Interface which seems to be more inline with how Apollo recommended interface structure.  No subscriptions yet, but I'm using it for queries/mutations over sockets.
@vic if you check mine out, please let me know what you think.  I had originally considered forking yours, but it seemed a bit too far off from the structure I was thinking, and I wasn't ready to jump into using Ramda.

@bjunc dont think it's something related to #14 will have to fix that anyways, I'm just too far from the computer these days :) So, sorry for not responding as quick as I'd like to.
Sad to hear this lib didn't work for you, I built it just for a customer project a few months ago, and share it with the hope of it being useful to others, if it's not, you are totally free to create your own, fork or otherwise contribute to this one (as I said I'm actually a bit far from keyboard these days, so it'd be wonderful some help handling issues and PRs). And yes, haha, I used ramda mainly because I'm mostly confortable doing Elixir and thinking a bit more functional than normal Js.
How is APW not inline with the apollo recommended interface structure, perhaps you can share some info on how we can improve ?
Cheers !

No problem about the response delays, totally understandable.
Apollo 1.x convention is to expose a createNetworkInterface method, which then instantiates an NetworkInterface class (I called mine PhoenixWebSocketNetworkInterface).  That class exposes query, use (middleware), and useAfter (afterware) methods.
I'm not exactly sure what about APW wasn't working, but I was able to use your socket interfacing logic to get me pointed in the right direction (so thanks!).
All that said, Apollo 2.0 is out now, which changes the convention again.  Now, instead of "network interfaces", they use "links".  Funny enough, "links" work kind of like plugs.  You create a link chain, with each link feeding into the next, before ultimately resolving.  I was able to create a new PhoenixWebSocketLink class that uses Observers (instead of Promises).  It's working well, but I think it probably deserves its own repo due to it being pretty different than APW (and even the Apollo 1.x network interface I originally created).  The only thing I haven't tested is pubsub, but that's not really a priority for me, as I don't really have a good use-case for GraphQL based pubsub (not yet anyway).
What do you think makes the most sense for the community?

FWIW, hitting this as well. My network interfaces seem to want to hit http://localhost:3000/graphql when my Phoenix server actually listens on ws://localhost:4000/socket. I think I actually had this working on an earlier version of Apollo, maybe 1.9.1 or something, but 1.9.3 seems to have changed something that has broken this.
Anyhow, no pressure, I'm just adding my voice and experiences in case they help someone else.

@ndarilek Yeah, that's the same issue I was experiencing.  Basically, the network interface is being ignored, and defaulting to a basic HTTP network interface.
Feel free to try my Apollo 1.x based PhoenixWebSocketNetworkInterface.  It works for me, in both server (Node.js) and browser environments.
Also, I just pushed a first draft Apollo 2.0 link: PhoenixWebSocketLink.  This is also working for me in both server and browser environments.  I'll be focusing on this moving forward (as opposed to Apollo 1.x).
In neither case am I using subscriptions, but query/mutations are working great.

@bjunc Oh so, the interface has changed for apollo 2.0.
It would be nice to state in the README the last apollo version people has been able to get APW to work with. Feel free to send a PR and link to your Apollo 2.0 link if that helps people use the newest version.

@vic It seems something might have changed with Apollo around 1.9.3, which might be breaking APW even in Apollo 1.9.x.  The interface spec definitely changed with Apollo 2.0.
Here's my PhoenixWebSocketLink for Apollo 2.0.  Still prototype phase, but it's working for queries / mutations.

Yea just took a look at Apollo 2.0 yesterday and noticed the changes.
The Apollo team have published an upgrade guide that demonstrates the necessary changes to the instantiation of the client.
@bjunc  is on the right track with a new Link interface for a phoenix socket + channel.
I am going to take a look, possibly try it out and fork if I can help!
Have a pretty big react-apollo application - am interested in the potential performance improvements of 2.0

I'm really interesting in your PhoenixWebSocketLink because I need a way to work with Apollo 2.0 subscriptions in my react-native application with react-apollo.
I can't get any way to do that it's works!


Build Tensorflow from source

Motivation for this change
(Do not merge before #30433!)
What it says.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

This is based atop https://github.com/NixOS/nixpkgs/pull/30433 , so there are extra commits (you are only interested in the last two).
Possible problem with this are long compile times with CUDA (half an hour+) where previously we uesd prebuilt binaries.
Bazel is a wonderful tool for building projects -- it's certainly a genius idea to create and use a 500k LoC Java build manager tool which eats 1.5G of memory for building C++. Unfortunately, Nix is not enterprise enough so our ways to isolate builds were not supported by Basel, making building a project with it effectively impossible in Nix (which is why TensorFlow used binary distribution before). Specifically, Bazel tries to download all external project dependencies by itself, which fails because of Nix isolation. In vanilla Bazel there is no way to circumvent this (please correct me if there's a better way!).
My solution is to add a special patched version of Bazel which skips checking of fetched dependencies checksums (which depend, among other things, on build directory paths). Then we can do two-staged build as described in https://github.com/NixOS/nixpkgs/commit/9b3559f274638a911d5337cabb671a63f47e82ea.
This, along with TensorFlow's own pecularities (special scripts for everything, awful compile times because all dependent libraries like CURL or ffmpeg are built statically, incomprehensible build system scripts even considering Bazel etc.) made this PR a work of three or four straight days. I'd say this gets a solid second place in my list of my most awful packages after Telegram.
Sorry for the rant ~_~

cc @jyp -- I won't have access to NVIDIA GPU for quite some time yet so it'd be cool if you could test this.

@TravisWhitaker , you may also be interested in this.

Your bazel changes seem helpful more broadly than here too!

@copumpkin Yeah, I'd imagine this approach can be used to build any other Bazel project on Nix. If you have anything else in mind we could merge the Bazel patch separately faster.

@abbradar This looks excellent, thanks so much for doing this! I'll give this a go tomorrow for sure.

What is the build time for Tensorflow?

@edolstra ~20 minutes for pure CPU build and ~35 for CUDA-enabled one on my machine -- this is very subjective though, I didn't get any measurements.

Okay, that's not too bad :-)

Impressive. I gave up on this one awhile back for sanity reasons. Packages be warned, don't mess with @abbradar .

@abbradar Bazel fails to build with trying to access /tmp. Here is the tail of the log:
INFO: From Compiling src/main/cpp/blaze.cc:
<command-line>:0:0: warning: "_FORTIFY_SOURCE" redefined
<command-line>:0:0: note: this is the location of the previous definition
INFO: From JavacBootstrap src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libbootstrap_JarOwner.jar [for host]:
warning: Implicitly compiled files were not subject to annotation processing.
  Use -proc:none to disable annotation processing or -implicit to specify a policy for implicit compilation.
1 warning
Target //src:bazel up-to-date:
  bazel-bin/src/bazel
INFO: Elapsed time: 101.997s, Critical Path: 92.72s
WARNING: /tmp/bazel_XdUveRNx/out/external/bazel_tools/WORKSPACE:1: Workspace name in /tmp/bazel_XdUveRNx/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions.

Build successful! Binary is here: /tmp/nix-build-bazel-0.4.5.drv-0/output/bazel
Error: mkdir('/tmp/.bazel'):  bV
builder for ‘/nix/store/m0c7wz18nmwi9zh3zy54478l73qddv1y-bazel-0.4.5.drv’ failed with exit code 36
cannot build derivation ‘/nix/store/igx4pla9mmc5p475z65xmwz7x0nzvqy8-python3.6-tensorflow-1.3.1.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/73ram5ch0sp485gk6pn28wcvgg51x9zy-python3-3.6.2-env.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/73ram5ch0sp485gk6pn28wcvgg51x9zy-python3-3.6.2-env.drv’ failed
/usr/local/bin/nix-shell: failed to build all dependencies

@jyp Not sure what happens here. Do you have sandbox enabled?

@abbradar No, I am not using sandboxing. Should I?

@jyp Yeah, it's disabled by default for users because of performance reasons but practically required if you start building stuff.
nix.useSanbdox = true;

Using a single process I could build bazel, but tensorflow does not build. There is apparently a problem with libstdc++. Here is the tail of the log.
/nix/store/qxb0x9fcs60zv719yy0zm95gyd8mmgks-glibc-2.25-49-dev/include/features.h:373:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
 #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
    ^
In file included from /nix/store/iwkcli8sy3rs7ks6kw7vclhwpgjn5bgb-cudatoolkit-8.0.61-unsplit/bin/..//include/host_config.h:173:0,
                 from /nix/store/iwkcli8sy3rs7ks6kw7vclhwpgjn5bgb-cudatoolkit-8.0.61-unsplit/bin/..//include/cuda_runtime.h:78,
                 from <command-line>:0:
/nix/store/qxb0x9fcs60zv719yy0zm95gyd8mmgks-glibc-2.25-49-dev/include/features.h:373:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
 #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
    ^
ERROR: /tmp/nix-build-python3.6-tensorflow-1.3.1.drv-0/tensorflow-v1.3.1-src/output/external/jpeg/BUILD:167:1: Executing genrule @jpeg//:simd_x86_64_assemblage23 failed: bash failed: error executing command /nix/store/h404wfcz8rzzlq8vr4z7plcijwzfci72-bash-4.4-p12/bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 127.
bazel-out/host/bin/external/nasm/nasm: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
____Building complete.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
____Elapsed time: 85.798s, Critical Path: 60.86s
builder for ‘/nix/store/igx4pla9mmc5p475z65xmwz7x0nzvqy8-python3.6-tensorflow-1.3.1.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/73ram5ch0sp485gk6pn28wcvgg51x9zy-python3-3.6.2-env.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/73ram5ch0sp485gk6pn28wcvgg51x9zy-python3-3.6.2-env.drv’ failed
/usr/local/bin/nix-shell: failed to build all dependencies

Wha~...  Not sure what happens here again, haven't seen that before. What do you mean by "single process"?

@abbradar I meant not using "-j n" option.
After activating sandboxing (using /etc/nix/nix.conf, because I am not on NixOS but Fedora), I am getting the same error, but apparently at a different point:
____[1,567 / 4,608] Writing file tensorflow/core/libfunctional_ops_op_lib.pic.lo-2.params
____[1,591 / 4,769] Writing file tensorflow/core/kernels/libsegment_reduction_ops_gpu.pic.lo-2.params
____[1,653 / 4,926] Writing file tensorflow/core/kernels/liblookup_table_op.pic.lo-2.params
____[1,703 / 4,982] Compiling external/boringssl/linux-x86_64/crypto/bn/x86_64-mont5.S
____[1,817 / 4,982] Compiling external/boringssl/src/crypto/bn/add.c
____[1,828 / 4,982] Compiling external/boringssl/src/crypto/refcount_lock.c
____[1,931 / 4,982] Compiling external/boringssl/src/crypto/bn/mul.c
____[1,965 / 4,982] Compiling external/boringssl/src/crypto/hmac/hmac.c
____[2,045 / 4,987] Compiling external/curl/lib/hash.c
____[2,086 / 4,996] Writing file external/highwayhash/libsip_hash.a-2.params [for host]
____[2,098 / 5,010] Compiling external/png_archive/pngmem.c [for host]
____[2,243 / 5,017] Compiling external/grpc/src/core/lib/iomgr/load_file.c
____[2,244 / 5,017] Compiling external/grpc/src/core/lib/iomgr/timer_heap.c
____[2,245 / 5,017] Compiling external/grpc/src/core/lib/iomgr/udp_server.c
____[2,252 / 5,017] Compiling external/grpc/src/core/lib/surface/server.c
____[2,245 / 5,017] Compiling external/grpc/src/core/lib/iomgr/timer.c
ERROR: /tmp/nix-build-python3.6-tensorflow-1.3.1.drv-0/tensorflow-v1.3.1-src/output/external/protobuf/BUILD:269:1: Executing genrule @protobuf//:generate_js_well_known_types_embed failed: bash failed: error executing command /nix/store/h404wfcz8rzzlq8vr4z7plcijwzfci72-bash-4.4-p12/bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 127.
bazel-out/host/bin/external/protobuf/js_embed: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
____Building complete.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
____Elapsed time: 77.122s, Critical Path: 55.70s
builder for ‘/nix/store/igx4pla9mmc5p475z65xmwz7x0nzvqy8-python3.6-tensorflow-1.3.1.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/73ram5ch0sp485gk6pn28wcvgg51x9zy-python3-3.6.2-env.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/73ram5ch0sp485gk6pn28wcvgg51x9zy-python3-3.6.2-env.drv’ failed
/usr/local/bin/nix-shell: failed to build all dependencies

About single process: this seems to be a bug in our Bazel package then, I'll look at this later.
About your setup: do you use single-user or multi-user Nix?
Try to build it with --keep-failed and poke around in the build directory. Let's start with ldd path/to/failed/binary...

$ ldd
/tmp/nix-build-python3.6-tensorflow-1.3.1.drv-0/tensorflow-v1.3.1-src/output/execroot/tensorflow-v1.3.1-src/bazel-out/host/bin/external/protobuf/js_embed
linux-vdso.so.1 (0x00007ffe39da2000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff8f174b000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff8f1534000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff8f116e000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff8f0e65000)
/nix/store/yydnhs7migvlbl48wpsxan1yvq2icbr9-glibc-2.25-49/lib/ld-linux-x86-64.so.2
=> /lib64/ld-linux-x86-64.so.2 (0x0000560e970ae000)

On Mon, Oct 16, 2017 at 11:13 AM, Nikolay Amiantov<EMAIL_ADDRESS>
wrote:

About single process: this seems to be a bug in our Bazel package then,
I'll look at this later.
About your setup: do you use single-user or multi-user Nix?
Try to build it with --keep-failed and poke around in the build
directory. Let's start with ldd path/to/failed/binary...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/30434#issuecomment-336827674, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABsY0MAqo4FMZ5i0eHU-EFA65T-wSbRks5ssx4zgaJpZM4P5vfr
.

It seems to impurely use you host OS's libraries for some reason. That shouldn't happen with sandboxing. Do you use nix-daemon?

@abbradar Yes, I am using nix-daemon. I suppose that I should re-build bazel with sandboxing enabled. However I do not know how to invalidate entries in the nix build cache.

nix-store --delete /nix/store/foo

I managed to make my laptop's hybrid NVIDIA to work with CUDA and verified that TensorFlow works for me.

That's great :) But unfortunately I am still getting the same error after re-compiling bazel and tensorflow. How can I verify that the sandbox mode is enabled?

Try to build this:
with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "foo";
  buildCommand = ''
    cp /tmp/test-file $out
  '';
}

with touch /tmp/test-file; nix-build foo.nix. It should fail if sandboxing is enabled.

Thanks. So for some reason nix-* disregards my configuration

Did you restart nix-daemon after your change? You set build-use-sandbox = true, correct?

I'm also failing on Error: mkdir('/tmp/.bazel') I simply checked out abbradar/tensorflow and ran:
$ nix-build --option build-use-sandbox true -E 'with import ./default.nix {}; pythonPackages.tensorflow.override {cudaSupport = true; cudaCapabilities = ["3.7" "6.1"];}'

I had this problem with my impure Tensorflow derivation too (I need to build with a specific Bazel version), and hacked around it with:
bazel.overrideAttrs (a: rec
    {
        preBuild = ''
            rm -rf /tmp/.bazel
        '';

        postFixup = ''
            rm -rf /tmp/.bazel
        '';
    });

@abbradar Yes, that is what I did, with apparently no effect.

@jyp Strange! What happens to you with sudo nix-build --option build-use-sandbox true -A tensorflow (sudo is important but don't worry, it won't result in builds from root)?

Finally I managed to enable the sandbox and I'm getting yet another error:
🍃  Building Bazel with Bazel.
.WARNING: /tmp/bazel_GwJ9FX3o/out/external/bazel_tools/WORKSPACE:1: Workspace name in /tmp/bazel_GwJ9FX3o/out/external/bazel_tools/WORKSPACE (@io_bazel) does not match the name given in the repository's definition (@bazel_tools); this will cause a build error in future versions.
INFO: Found 1 target...

Target //src:bazel failed to build
INFO: Elapsed time: 13.925s, Critical Path: 0.46s

ERROR: Could not build Bazel

Hmm, can you allow /bin/sh in your sandbox? I think that's the problem.
build-sandbox-paths =  /bin/sh=/bin/sh and also add all libraries that it depends on. You can do it by running:
with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "foo";
  buildCommand = ''
    /bin/sh -c 'touch $out'
  '';
}

and fixing your /bin/sh until it works.

I've pushed an update which hopefully fixes Bazel's /tmp problems.
@edolstra , I've measured TensorFlow compile time with CUDA enabled on my machine -- 55 minutes. Seems I tend to strongly underestimate time ^_^"

@abbradar any thoughts on upstreaming your changes to make Bazel make fewer assumptions about its build environment?

@copumpkin The problem is that my patch is strongly a hack: it breaks Bazel environment isolation (which is a bad thing really but we want it since we have our own!) and breaks its checksums to achieve network isolation.
All other changes are our usual Nix package building business like s,/bin/sh,${stdenv.shell}/bin/sh,g :D

You can do it by running:
with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "foo";
  buildCommand = ''
    /bin/sh -c 'touch $out'
  '';
}

and fixing your /bin/sh until it works.

I don't understand what this means.

@jyp This derivation should work for you. If it doesn't, that means your sandbox doesn't include /bin/sh (which it should). To fix that you need to add build-sandbox-paths = /bin/sh .... to your Nix configuration, where ... are libraries which are needed by your sh. You can find them out by running ldd /bin/sh -- all those should be added to the option, separated by spaces.

@abbradar It works :) And thanks a lot for guiding me through all this!
A few remarks:

shouldn't the build also work without sandboxing? I've been using nix for years and it's the first time that I needed that.
I am guessing that the build won't work on darwin. (Does bazel even build on darwin?)
did you manage to fix the issue with /tmp?
I am still getting the messages about not using SSE and whatnot:

 2017-10-17 22:00:43.873246: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-17 22:00:43.873274: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-17 22:00:43.873282: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-10-17 22:00:43.873288: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-17 22:00:43.873295: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.

Could we enable those build flags?

We can try but this requires further crawling in Bazel's build system (eeeew) which is responsible for picking host's libraries -- and for a little gain, because when this is upstream you'd always get Bazel from Hydra;
I'd like to make this work on Darwin to avoid regression but I don't have a machine available to check. The only thing I can do is merge this and build-fix-build on Hydra. Bazel should work on Darwin I think (after all it's not like they cross-compile TensorFlow from Linux so it means its build system is working);
Not sure but may be fixed with the new patch above;
We can't do this because TensorFlow doesn't support any kind of fallback -- i.e. if you enable AVX and then run it on a machine without AVX it'd just crash.

It would be nice to be able make the flags available anyway, I am guessing that they make quite a bit of performance difference. Also, some options are pretty much safe to enable (SSE) since they were introduced on CPUs 10 years ago. (Who is using tensorflow on such old machines?)

@FRidh Fixed in https://github.com/NixOS/nixpkgs/pull/30433 -- those changes here are only because I base on that branch.

Don't clean environment variables for children processes.
Needed for Nix compiler wrappers.

You should be able to use --action_env=VARIABLE to preserve a given environment variable: https://bazel.build/designs/2016/06/21/environment.html
It might be better to enumerate NIX_* variables and add an action_env flag for each, rather than patching bazel to preserve all env vars.
Regarding downloads, generally (that is, assuming the respective build systems are well behaved) you should be able to specify a repository for anything that would need to be downloaded, which you can point at a store path when building as apart of a nix derivation. You can see where I leverage this to point Envoy's build system at third-party dependencies that are built by Nix: https://github.com/NixOS/nixpkgs/blob/30f6d630f1908d05da1d22aff80c489f3c09652d/pkgs/tools/networking/envoy/default.nix#L164-L168
If you can't prevent fetching stuff over the network, that's either a bug in Tensorflow's build system or a bug in Bazel.
I'll see if I can find time to see if I can improve the package, and report any issues upstream.

@cstrahan I've seen this article but isn't this a design document? I thought that this functionality is not implemented yet -- if I'm mistaken it's great, let's then just passthru NIX_*.
About local repositories -- I didn't know that! From the build file it seemed that it'd always just download dependencies over the network but I'm very unfamiliar with Bazel. It would be great if you could take a look at it.


Optimized projection rebuilds for single stream projections

It's a completely different set of issues for multi-stream projections, so let's put that aside for right now.
With a single stream projection, what if you could do a rebuild model that does it stream by stream instead of doing the naive "left fold" approach from zero? That would enable you to do things much more efficiently and do a lot less work against the database.

Test Cases

[ ] Rebuild single stream with Guid identity, aggregate has a Guid identity as well
[ ] Rebuild single stream with Guid identity, aggregate has a Guid-based strong typed identity
[ ] Rebuild single stream with string identity, aggregate has a string identity as well
[ ] Rebuild single stream with string identity, aggregate has a string-based strong typed identity

Punchlist

[ ] In normal mode, if start up and the progress is 0, and the high water mark is more than 10 pages long, use optimized rebuilder first
[ ] Performance test