Broken with the last commits

Upon building my project I get the following error message:
github.com/kardianos/service
......\kardianos\service\service_windows.go:247:17: undefined: windows
......\kardianos\service\service_windows.go:248:19: not enough arguments in call to ws.Config.Option.bool
have (string)
want (string, bool)
......\kardianos\service\service_windows.go:249:31: undefined: windows
It worked properly yesterday, so the last commit probably caused the issue.

Fixed.


Fix 1906

During doing it I have some doubt about consistency to 'order' results when computing stats.
For now it is ordering as (x, y). As it should be displayed. But to be more numpy compliant maybe having it y, x would be better. And manage it in the display. But I an not sure this is consistent inside the entire library.

There is also an issue with the coordinates of min and max when many points have this value.
An image can have many points with the same value, and this case leads to issues: To reproduce, try to display the stats of an image like: numpy.ones(1000).reshape(10, 100) or even numpy.ones(10**6).reshape(1000, 1000)

As it is now, for images, coord min. and coord max. provides the coordinates of a corner of the pixel.
Wouldn't it be better to return the center of the pixel? so adding a 0.5 pixel offset.

Switching back from visible range to full data range is not taken into account.

I made a PR with a few fixes.
Also, when denominator is 0, it is better to return an invalid result (infinity, NaN).
closes #1906

As it is now, for images, coord min. and coord max. provides the coordinates of a corner of the pixel.

Wouldn't it be better to return the center of the pixel? so adding a 0.5 pixel offset.
Don't know. Could be an option

There is also an issue with the coordinates of min and max when many points have this value.
An image can have many points with the same value, and this case leads to issues: To reproduce, try to display the stats of an image like: numpy.ones(1000).reshape(10, 100) or even numpy.ones(10**6).reshape(1000, 1000)

We can display only the first coordinates or a QList of coordinates. Or having both (into two classes).

To provide the center of the pixel, we should also draw the pixels at the center. At this moment, pixel 0 is represented with its center at 0.5. So, at this point I would not change anything.
When we represent an image as a scatter plot, there is an effective shift because the symbol is drawn at the actual coordinates. So, if something is to be changed at a certain point, is the visualization of the images with a shift of 0.5 pixels, the calculations will not change.

LGTM, can be merged once CI is happy

BTW, closes #1906

Everything's fine


Capture container log output

Is it possible to capture the logs from the containers?
I often want to inspect to logs of a service and right now to achieve that I need to quickly run
docker logs -f <service name>

in another terminal while the tests are running. Otherwise I can set removeContainers to false but then I want a fresh container each time I run the test.

It should be possible. Where do you want to prefer to print the logs to? Should it be disabled by default?
Feel free to prepare PR.


Reportings bugs found by OSS-Fuzz

Hi, we recently started fuzzing Chakra using OSS-Fuzz (google/oss-fuzz#925), and have discovered a number of crashes. Some of them are potentially security vulnerabilities (buffer overflows, bad address dereferences etc).
We'd like to report these bugs by asking Chakra developers/maintainers to send a PR to add their emails (needs to be associated with a Google account) to this file (example) so that you are CC'ed on current and future bug reports in our issue tracker (disclosure policy).
Our infrastructure comes with regression range testing, fix verification, de-duplication, testcase minimization, and more.
Thanks!

@oliverchang Killer! Thanks for offering to share the reports. Is there a reason to not report the security concerns via<EMAIL_ADDRESS>

@EdMaurer Our service is designed as a tool for the developers of the code that is being fuzzed, and is fully automated in that regard. We don't have anyone doing manual triage to e.g. report to<EMAIL_ADDRESS>

@oliverchang Got it. If this is intended to be an ongoing activity, I would prefer it if you would target a release branch as opposed to master. master is our least stable branch and does not generally contain the fixes for issues found through our fuzzing efforts.

Sure, we can do that. Would release/1.7 be the best branch?
That said, our workflow is optimized for fuzzing on master/development branches. Our infra is generally very useful at finding regressions quickly on master before they make it into a release branch at all. I'm not familiar with Chakra development or how this would fit into that though, so this is of course up to you :)

As Oliver said, OSS-Fuzz workflow is strictly meant for master branch fuzzing. That way, we don't have to keep changing release branch numbers in our code. If your branch is unstable, it creates a few more bugs, but once your developer uploads a fix, bugs get auto-verified and auto-closed. So, workflow wise it is all automatic and more useful. This way, you will get both stability and security bugs as soon as they are introduced.

The concern I have is around managing the triage of your discoveries. I'm hoping that if you run it on something that we've put our hardening/fuzzing effort into, you will find fewer issues that will need to be dealt with outside of our normal workflow.
You've run it on master it sounds like. How many issues did it turn up?

We've found 21 unique reproducible crashes so far. Our de-duping is generally pretty good and we have tools to help triage like e.g. regression ranges and automatic fix verification.
Would you be opposed to receiving these reports for now and giving it a trial run? If they end up being too much to triage in the long run, we can change to fuzzing release branches.

@oliverchang Yeah, that'll work. Regression ranges will be very helpful.

Hey @EdMaurer, how are reports that you've received so far?
We'd be very happy to hear any feedback you might have for us.

@oliverchang I tuned out on this topic, sorry. I'm not the best person to ask about the quality of the reports. @digitalinfinity and @curtisman have been looking at them. I've heard of a couple of cases where we had the same issue already on our books from our internal fuzzing, but that has not been the general case.


NowPlaying Improvements: seek buttons, play/pause track item syncing, replay button, shortcuts.

Here are the list of changes:

Fast-forward and rewind buttons (10 seconds)
Play/pause in the NowPlaying component is synced with the TrackItem button
Replay button for when the song is finished
shortcut keys for each button

space/enter for play/pause/replay
left for rewind
right for fast-forward

I think I would like to implement next/previous buttons as well, but that requires managing the track list state across different tabs, so I'll probably add that in a different PR. Let me know if you want anything changed.

I've just tested this on my machine and it works great!
Really love the replay feature, I admit I'm a bit annoyed that I can't replay the song in the current released version. Another small thing I'd love to have on the now playing is that, after the song is ended, we can click on any point on the seek bar to replay it at the selected point. If you want to help with that then it would be amazing (another PR is fine)!
One more useful feature is in-app volume control (as suggested in https://github.com/tranxuanthang/lrcget/issues/38). This might require some additional commands from the Rust side. But asking more from you is probably too much, so feel free to continue with whatever you have planned or enjoy working on!
Anyway, this is a solid PR. Thank you for your hard work!


starting tidal and vim in a tmux session

How do I start tidalvim in a pre-existing tmux session? Whenever I launch it via tidalvim, it always creates a session for me. Otherwise I get an error like this:
[root@gns333 ~]# tidalvim
/usr/local/bin/tidalvim: line 18: /tmp//tmux-0/default,2875,0: No such file or directory
/usr/local/bin/tidalvim: line 18: /tmp//tmux-0/default,2875,0: No such file or directory

i get the same error /usr/local/bin/tidalvim: line 17: /tmp//tmux-1000/default,2700,3: No such file or
directory
/usr/local/bin/tidalvim: line 17: /tmp//tmux-1000/default,2700,3: No such file or
directory

Same as above, when prompting tidalvim.


Fixing documentation example for KongConsumer custom_id field

What this PR does / why we need it:
There was a typo in the documentation that led to some of the discussion in #70.
Which issue this PR fixes:
fixes #70

Codecov Report

Merging #90 into master will increase coverage by 0.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
+ Coverage   32.03%   32.17%   +0.13%
==========================================
  Files          33       33
  Lines        2859     2859
==========================================
+ Hits          916      920       +4
+ Misses       1830     1826       -4
  Partials      113      113

Impacted Files
Coverage Δ

cli/ingress-controller/main.go
22.22% <0%> (+2.61%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78eb8fa...c59e884. Read the comment docs.


assessment point entries DISAPEARING from strand reporting tab

there's a clash when you have two open sessions in the same strand reporting view.
to reproduce:

open two tabs (tab A and B) in the same strand reporting view (same assessment points, same class filters)
in tab A, create some entries
after step 2, try to create an entry for the same assessment point in tab B — it should raise an error, because of the unique DB constraint (unique for student_id + assessment_point_id), and on recovery, it will "sync" the DB with tab B, effectively deleting the entries registered in step 2

closed by aa282b6


Hash hrefs are now relative to the main HTML page, not the HTML imported doc

 Issue by sethladd
Originally opened as dart-lang/sdk#14450

See repro here: https://gist.github.com/sethladd/7149570
This behavior changed w/ 0.8.5
See also this thread: https://groups.google.com/a/dartlang.org/forum/#!topic/web-ui/xwV00tyWaCI

 Comment by sigmundch

Added this to the Later milestone.

 Comment by zoechi

I think a good solution would be if there would be different forms of hrefs.
-relative to the package where it is defined
-relative to the main HTML page
-absolute (like packages/...
where some sort of syntax defines which one to use.

 Comment by clayberg

Removed this from the M9 milestone.
Added this to the 1.1 milestone.

 Comment by sigmundch

Removed this from the Later milestone.
Added this to the M9 milestone.

 Comment by jmesserly

Removed Area-Polymer label.
Added area-pkg, Pkg-Polymer labels.

 Comment by sigmundch

Removed this from the 1.1 milestone.
Added this to the 1.2 milestone.

 Comment by sigmundch

Added Library-Polymer label.

 Comment by jmesserly

Removed Library-Polymer label.

 Comment by sigmundch

Removed this from the 1.6 milestone.
Added Polymer-P-1 label.

 Comment by sigmundch

Removed Polymer-P-1 label.
Added Polymer-Milestone-Next label.

 Comment by sigmundch

Removed Priority-Unassigned label.
Added Priority-Medium label.

 Comment by kasperl

Removed this from the 1.2 milestone.
Added this to the 1.6 milestone.

 Comment by sigmundch

Removed Polymer-Milestone-Next label.
Added PolymerMilestone-Next label.

Unless I am mistaken this doesn't seem to be a polymer specific bug. If it is still an issue please refile it in the appropriate repo (sdk)? If it is polymer specific then please re-open. Thanks!


For those inexperienced with Visual Studio, and getting "Unable to find package" nuget errors

Visual studio sometimes sets the package manager to offline

I had been getting "Unable to find Raylib-cs" so I have added nuget.org (https://api.nuget.org/v3/index.json) as an online source to resolve this and installed latest stable version of Raylib. But now I am getting that its incompatiblew with netcoreapp1.1. Someone facing simmilar issues?

Duplicate of #2
Fixed in #20


Can't return to the 'Connect to public blob container' dialog after clicking 'Back' in the 'Connection Summary' dialog

Storage Explorer Version: 1.15.0-dev
Build: 20200825.4
Branch: rel/1.15.0
Platform/OS: Windows 10/ Linux Ubuntu 16.04/ MacOS Catalina
Architecture: ia32/x64
Regression From: Not a regression
Steps to reproduce:

Expand one storage account -> Blob Containers.
Select one blob container -> Copy its URL.
Open Connect Dialog -> Select 'Connect to public blob container'.
Paste the URL in the 'Connect to public blob container' dialog -> Click 'Next'.
Click 'Back' in the 'Connection Summary' dialog.
Check whether it returns to the 'Connect to public blob container' dialog.

Expect Experience:
Return to the 'Connect to public blob container' dialog.
Actual Experience:
Directly return to the 'Connect to Azure Storage' dialog.

Verified on build 20201112.8. This issue has been fixed. So close it.


Fix TSAN Race in Overriding poll_function

Fixes #14388

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

Broader question: is this the right approach? AFAICT, polling is overridden at setup and goes away at test destruction. Doesn't the fact that there is a race indicate that some thread is still active at the time of test destruction (which would be a bad thing)?

Just saw the stack trace and saw that the other thread that is active is a timer thread. It seems to me that this whole concept that we have needs a revisit. The thing is that this PollingOverride test was created before we had a dedicated timer thread, and I think it's making the previously-valid assumption that there aren't lurking library threads doing stuff behind the test's back. I'm not that comfortable with atomically switching a poll function while some other thread is in the middle of polling with a different polling function.

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences

[microbenchmarks] No significant performance differences

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

Done

[trickle] No significant performance differences

Got a green run earlier, merging

[microbenchmarks] No significant performance differences


[Question] Wild guess ETA of support 3.x release GA

Hello team,
Do you have any wild guess on when the support of 3.x will be released GA?
It seems that beta with alpha has 1 year different. That means that GA might take 1 more year?
Thank you in advance :)

We had planned on moving on to 3.6-alpha after the 3.4-beta release, but I hadn't considered that people might be waiting for a "final" release before using 3.x... In my opinion the beta release is probably GA-quality already, but if you feel a final v3.4.0 release is desirable we can certainly reconsider our plans and look into this.

I have realized that for some people/projects, the official status of a "final" release is important. I know some parties that would not touch "alpha" because of its status. The "beta" may attract more users but some project may be reluctant to migrate from ipy2 "final" to ipy3 "beta" because of the perceived maturity downgrade. But it is more than just a matter of perception. An official "final"/stable release means that (1) no new potentially unstable development takes place on top of it, and (2) bugfixes are being applied.
Given the current limited resources I believe that the IronPython project can only reasonably support at most two branches. For quite some time those branches were ironpython2/master (stable) and ironpython3/master (leading-edge). Making a "final" release of IronPython 3 will effectively mean that IronPython 2 becomes "out-of-support". This means no more of ipy2 releases, or bugfixes, even when found and fixed in ipy3. Sure, people can still submit pull request that may or may not be merged, but effectively ipy2 becomes end-of-life.
So here are some ideas: instead of ipy2, a separate support branch of 3.4 would be maintained, that only contains essential bugfixes from the leading-edge master branch of ipy3. Porting some fixes from master to this 3.4 branch should theoretically be easier than backports to ipy2 because the code is more similar. So we would have again only two branches, ironpython3/master and ironpython3/3.4.
Now the question is how to get there. I think it is fair to give "beta" some time to get feedback  on quality and bugs from folks that would not dab in "alpha", before making a "final" release. Since slozier/stdlib-3.6 is now practically the leading edge, the current master will receive only bugfixes or other improvements that do not jeopardize stability (like fixes in build, installation, usage, packaging, small and localized upgrades, etc.). I am OK with that, there is plenty of stuff of this caliber that can be done. Once stdlib-3.4 is ready for merge into master, we branch off 3.4 from the last commit before merge. This becomes the stable branch that will lead to the "final" and post-final releases of 3.4, fully replacing ipy2.

You are right that a final version of a project version makes it more desirable for others people/project to use it. Even if the beta is final-quality  the final word is still better 😄
Python 2.7.x has been marked as EOL from Jan 2020, with this in mind I think you can stop supporting the ipy2 and move forward to have a final ipy3 version.
Do you think that you can move to ipy3/3.4-final in a couple of months?

Yes, a couple of months is a reasonable estimate.


WIP: rebased position offset

Trying to make auto base work nicely with variants.

Codecov Report

Merging #120 (71b737c) into 2076-2 (3233c92) will increase coverage by 0.16%.
The diff coverage is 96.70%.

@@            Coverage Diff             @@
##           2076-2     #120      +/-   ##
==========================================
+ Coverage   90.30%   90.46%   +0.16%
==========================================
  Files         122      125       +3
  Lines        5601     5687      +86
==========================================
+ Hits         5058     5145      +87
+ Misses        543      542       -1

Impacted Files
Coverage Δ

...nclude/adm/elements/audio_block_format_objects.hpp
100.00% <ø> (ø)

include/adm/elements/audio_object.hpp
100.00% <ø> (ø)

include/adm/elements/position.hpp
100.00% <ø> (ø)

include/adm/private/rapidxml_formatter.hpp
100.00% <ø> (ø)

include/adm/private/xml_parser.hpp
100.00% <ø> (ø)

src/elements/audio_object.cpp
86.66% <ø> (ø)

src/elements/position_offset.cpp
85.71% <85.71%> (ø)

include/adm/detail/auto_base.hpp
98.18% <87.50%> (-1.82%)
:arrow_down:

include/adm/detail/print_helper.hpp
100.00% <100.00%> (ø)

include/adm/elements/position_offset.hpp
100.00% <100.00%> (ø)

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3233c92...71b737c. Read the comment docs.


fix ruff tests by running the quiet mode

The latest version of ruff (0.3.3) does not have an empty stdout when succesfull.
This can be restored by passing the --quiet flag.

@all-contributors please add @GaetanLepage for tests

do you need a release, given that it only affects tests?

This would let us update ruff on nixpkgs: https://github.com/NixOS/nixpkgs/pull/296256
We can still cherry-pick this patch. I fully understand that this might not be worth a release for you.

no worries, just made a new one


Extend VQE to include orbital optimization

This PR closes #24 and closes #55. It is actually a MR from a feature branch developed in a private qc2 fork.
@NicoRenaud, I have squashed all commits into a single one, as you have already approved it in the forked repo. Please, could you also approve the MR here ?

@NicoRenaud. Please, could you have a look at this small PR ? Thanks!


Add more infrastructure specific options to deployer

https://github.com/api3dao/airnode/pull/333#discussion_r626508755

Also worth considering letting the user specify any throttling/quotas for the testing API

Should these be specified in nodeSettings.cloudProvider in config.json, assuming they will be cloud provider-specific?

Should these be specified in nodeSettings.cloudProvider in config.json, assuming they will be cloud provider-specific?

Yes, that would probably be a good place for them.

Tracked by https://api3dao.atlassian.net/browse/AN-145


Run Selection/Line in Python Terminal - Long Text

- VSCode Version: 1.40.2
Version: 1.40.2
Commit: f359dd69833dd8800b54d458f6d37ab7c78df520
Date: 2019-11-25T14:52:45.129Z
Electron: 6.1.5
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Darwin x64 19.0.0
- OS Version: macOS Catalina Version 10.15
- VS Code Extension - Python
Name: Python
Id: ms-python.python
Description: Linting, Debugging (multi-threaded, remote), Intellisense, code formatting, refactoring, unit tests, snippets, and more.
Version: 2020.1.54965-dev
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=ms-python.python
Steps to Reproduce:

Launch Python3
Run Selection with more than ~1024 characters selected, example below.

{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}
{
    "test": "test",
    "test2": "test2"
}

Does this issue occur when all extensions are disabled?: Yes
This is a duplicate of https://github.com/microsoft/vscode-python/issues/6334 and https://github.com/microsoft/vscode/issues/38137 (and likely others).  This does not appear to be resolved and the old issues are closed so I cannot update them.
The result is that I cannot copy/paste or run/selection on large amounts of Python code in the embedded terminal.

I am seeing the same issue

Please file the issue against the Python extension you are using. Very likely here: https://github.com/Microsoft/vscode-python

Sorry my fault. Dups https://github.com/microsoft/vscode/issues/38137

@dbaeumer when I had opened this originally via the VS Code extension repository it was determined to be an upstream issue.  Do you think that's not that case anymore?
Original extension issue: https://github.com/microsoft/vscode-python/issues/6334

It is. The issue in VS Code to track this is #38137 which I marked this issue a duplicate of.

Just to add this issue affects more than python.  My ticket was closed as a duplicate of this one (#96973) but shows that the issue also occours using send text to terminal  even on bash commands

@gowerc I think there might be another older duplicate of this as well, but it's a generic bug related to a maximum amount of text you can use with the sendText API and pasting.

Thanks @Tyriar  for clarifying.  Does this mean there is unlikely to be a solution then ?

I'm sure there is a solution, it's just not something I'm focusing on at the moment. If anyone wants to investigate this, I suspect the problem lies inside the node-pty component which manages the connection between VS Code and the process inside the terminal, either in how it's created and/or socket limitations when writing.
Related: https://github.com/microsoft/vscode/issues/59135

/duplicate #38137


Update to rc1.19456.4 packages

Fixes: https://github.com/Azure/azure-sdk-for-net/issues/7671

https://github.com/Azure/azure-sdk-for-net/issues/7490

/azp run net - appconfiguration - ci


Redis Cache CLI sample scripts

@steved0x,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Hey, @steved0x, we're not using variables like that for resource group, etc. That came out of ScottGu reviews.


update and adding topics

moved the code and material to python3 and also added topics

@silent-mobius
Sorry for the late reply, been busy lately with Dashain/Tihar festival here in Nepal.
Thanks for the PR. This is awesome! :heart_eyes:
I can merge this, if we can refactor README file with following fixes:

Broken Links
Code/header formatting

just checked all the links and they seem to work fine.
Please provide correct header formatting for fix.


Update core repos tip to latest master

The repos where moved to produce netcoreapp3.0 packages. If someone wants to test local changes with a produced runtime and point to a updated coreclr/corefx but leave the core-setup submodule as is, they will find issues since we're pointing to a core-setup commit that is still restoring packages for netcoreapp2.2
cc: @weshaggard @dagood
FYI: @GrabYourPitchforks

@safern I suspect you also need to update the version of BuildTools in order to fix the other CI issues.

test Windows_NT Release please
test CentOS7.1 Release

To fix the coreclr build you need to move coreclr up to include https://github.com/dotnet/coreclr/pull/18922


Fill in a simple SETUP.md

closes #145

LGTM.  I'm happy with this being short, just a jog of what to do and a link to the detailed information online.

Yeah, I like this. Thanks!


[ARO-7345] smoke test automation for aro-operator log

Which issue this PR addresses:

Fixes https://issues.redhat.com/browse/ARO-7345
What this PR does / why we need it:

This PR adds smoke test that confirms there is no error log in aro-operator logs.
This is labelled as "smoke", so it doesn't run automatically in CI.
Test plan for issue:

Run E2E_LABEL=smoke make test-e2e locally.
Is there any documentation that needs to be updated for this PR?

N/A
How do you know this will function as expected in production?

This is a part of tests.

/azp run ci, e2e

let's see if E2E passes the new test 😅

So E2E won't run the smoke tests as is: https://github.com/azure/ARO-RP/blob/09a56b1c466c54d10fb9132e18ff3dd78ce0533b/Makefile#L6
Having said that I agree with your concerns @kimorris27. If there's some error in the aro-operator logs do we care? I think the answer is "it depends" but given the scope of where we use the smoke tests I'm happy to ship this as is. Thoughts?

So E2E won't run the smoke tests as is: https://github.com/azure/ARO-RP/blob/09a56b1c466c54d10fb9132e18ff3dd78ce0533b/Makefile#L6
Having said that I agree with your concerns @kimorris27. If there's some error in the aro-operator logs do we care? I think the answer is "it depends" but given the scope of where we use the smoke tests I'm happy to ship this as is. Thoughts?

Oh, that's right; I forgot. That makes sense to me. We can tweak the test later on if this causes too many "flakes" in the smoke tests.


⚠ Bofrost.at has degraded performance

In 739168d, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 805 ms

Resolved: Bofrost.at performance has improved in 42ac71b.


Update to new 3/4" Venturi specifications.

Commits in this PR

Update to new 3/4" Venturi specifications.

PR chain

👉 #475 Update to new 3/4" Venturi specifications. 👈 YOU ARE HERE
#483 Move flow zeroing into TVIntegrator.

Thanks a lot, Ethan!  I've updated the PR, please have a look.

@esc-works friendly ping


Make duplication checks more strict & amend documentation

See https://github.com/EBIvariation/eva-opentargets/issues/185#issuecomment-797406076.
Because of the way pipeline v2.0.0+ operates, it does guarantee uniqueness on the four fields mentioned; however, it is not actually being checked yet prior to submission. Specifically, v2.0.0 doesn't check anything at all, and v2.0.1 will check for complete evidence string duplicates only.
The checks for duplicates need to become more strict, and the documentation needs to be amended to make clear which fields we guarantee uniqueness upon.

Since running the project locally was unfeasible (through WSL2) at the end, I have done the review using the released batch of April 2021 (batch-2021-04).
Overview
The four fields that, a priori, constitute the minimal non-repeatable unit are:

studyId - Reference ClinVar record (e.g. RCV000015714). These are not repeated across the whole ClinVar full release.
targetFromSourceId - The gene affected by the variant (e.g. ENSG00000186832). Can easily be repeated.
variantFunctionalConsequenceId - The consequence of such variant (e.g. SO_0001818, which corresponds to protein_altering_variant). Can easily be repeated.
diseaseFromSourceMappedId - Associated phenotype to such variant (e.g. Orphanet_2337, which corresponds to a type of keratoderma). Can easily be repeated.

Since an evidence string is an association of a "variant" causing a "disease" as described in "this ClinVar record" through "this molecular consequence", it does make sense that no complete repetition of them shall appear within the evidence strings.
Verdict
I have found 279 evidence strings with these 4 fields repeated within the examined batch. Interestingly, all of them were repeated exactly twice, and it appears that most, if not all, differ in the source of the data: either eva or eva_somatic, which leads to other differences (e.g. datatypeId). For instance:
{"alleleOrigins": ["somatic"], "datasourceId": "eva_somatic", "datatypeId": "somatic_mutation", "clinicalSignificances": ["pathogenic"], "confidence": "no assertion criteria provided", "studyId": "RCV001201102", "targetFromSourceId": "ENSG00000100697", "variantFunctionalConsequenceId": "SO_0001583", "variantId": "14_95091292_T_C", "cohortPhenotypes": ["DICER1-related pleuropulmonary blastoma cancer predisposition syndrome"], "diseaseFromSource": "DICER1-related pleuropulmonary blastoma cancer predisposition syndrome", "diseaseFromSourceId": "C3839822", "diseaseFromSourceMappedId": "EFO_0009068"}

{"alleleOrigins": ["unknown"], "datasourceId": "eva", "datatypeId": "genetic_association", "clinicalSignificances": ["pathogenic"], "confidence": "no assertion criteria provided", "studyId": "RCV001201102", "targetFromSourceId": "ENSG00000100697", "variantFunctionalConsequenceId": "SO_0001583", "variantId": "14_95091292_T_C", "cohortPhenotypes": ["DICER1-related pleuropulmonary blastoma cancer predisposition syndrome"], "diseaseFromSource": "DICER1-related pleuropulmonary blastoma cancer predisposition syndrome", "diseaseFromSourceId": "C3839822", "diseaseFromSourceMappedId": "EFO_0009068"}

If we enforce uniqueness upon the evidence strings based on those 4 fields, how shall we resolve this dicotomy, @tskir?
Additional comments
Besides, I would like to bring to discussion another type of repetition I noticed during my screening: identical evidence strings except for the diseaseFromSourceMappedId. Given what we have mentioned above, this should not be considered as a repetition, and could or could not be intended. Nevertheless, since the diseaseFromSource is the same I wonder if it is wise to store twice the amount of information for the same "record" we described, with the only difference that the disease was mapped through one database or the other. As an example, take a look at the following evidence strings:
{"alleleOrigins": ["unknown"], "datasourceId": "eva", "datatypeId": "genetic_association", "clinicalSignificances": ["pathogenic"], "confidence": "no assertion criteria provided", "studyId": "RCV001292529", "targetFromSourceId": "ENSG00000008710", "variantFunctionalConsequenceId": "SO_0001587", "variantId": "16_2106560_C_A", "variantRsId": "rs1567186165", "cohortPhenotypes": ["Polycystic kidney disease"], "diseaseFromSource": "Polycystic kidney disease", "diseaseFromSourceId": "C0022680", "diseaseFromSourceMappedId": "MONDO_0020642"}

{"alleleOrigins": ["unknown"], "datasourceId": "eva", "datatypeId": "genetic_association", "clinicalSignificances": ["pathogenic"], "confidence": "no assertion criteria provided", "studyId": "RCV001292529", "targetFromSourceId": "ENSG00000008710", "variantFunctionalConsequenceId": "SO_0001587", "variantId": "16_2106560_C_A", "variantRsId": "rs1567186165", "cohortPhenotypes": ["Polycystic kidney disease"], "diseaseFromSource": "Polycystic kidney disease", "diseaseFromSourceId": "C0022680", "diseaseFromSourceMappedId": "Orphanet_564"}

it appears that most, if not all, differ in the source of the data: either eva or eva_somatic,

I only looked at the one example you provided: RCV001201102 on Clinvar website and they mention two lines of evidence so I think the "duplication" of the evidence string makes sense since there are two independent type of evidence

We can bring bring it up at our next meeting with Open Targets to confirm this with them.

Given what we have mentioned above, this should not be considered as a repetition, and could or could not be intended. Nevertheless, since the diseaseFromSource is the same I wonder if it is wise to store twice the amount of information for the same "record" we described

I again only looked at the example you presented and I'm wondering if this might not be a mapping error or a bug in the code:
In the manual_curation/latest_mappings.tsv file:

polycystic kidney disease is mapped in the  to http://purl.obolibrary.org/obo/MONDO_0020642
Orphanet_564  is mapped to 15 traits but none of them are polycystic kidney disease (some are very close though polycystic kidney dysplasia). I think we should look into this a bit more to ensure that this is not due to a bug or a mapping error.

I think the "duplication" of the evidence string makes sense since there are two independent type of evidence.

I see, but it still does not solve the issue, since we are considering those four fields the minimum replication block of an evidence string. If there are several sources within ClinVar that lead to two evidence strings with those fields being identical, we should either group them, choose one, or change what we consider as a replicated evidence string.

We can bring bring it up at our next meeting with Open Targets to confirm this with them.

Should we consider this issue blocked until we do so?

polycystic kidney disease is mapped in the  to http://purl.obolibrary.org/obo/MONDO_0020642
Orphanet_564  is mapped to 15 traits but none of them are polycystic kidney disease (some are very close though polycystic kidney dysplasia). I think we should look into this a bit more to ensure that this is not due to a bug or a mapping error.

It may be due a possible duplicated sources for the disease mappings, since Orphanet_564 corresponds to the Meckel syndrome. One of the characteristics of such disease is large polycystic kidneys, which in essence can also be considered polycystic kidney disease.

@M-casado Thank you for an excellent investigation!
1 — Evidence string duplication

If we enforce uniqueness upon the evidence strings based on those 4 fields, how shall we resolve this dicotomy, @tskir?

Ah, I'm terribly sorry; when reporting in this ticket (& also the one which generated it), I simply forgot about the datatypeId. The uniqueness tuple should be five fields then, not four.
The reason is that Open Targets would like to keep germline and somatic evidence separately. So, in cases where a ClinVar record contains both somatic and non-somatic (that is, some version of germline) allele origins, that record is exploded into two evidence strings. This is expected and should keep being this way.
So with that in mind, could you check again please that the evidence strings are really unique on the five fields, including datatypeId?

2 — Different diseaseFromSourceMappedIds for the same diseaseFromSource
This is super important, thanks for noticing & bringing this up.
So first of all, it is possible to legitimately have this situation. The pipeline (and the mapping database latest_mappings.tsv) support the case of mapping the same source string to multiple ontology terms. This is necessary for complex concepts such as "Coronary artery disease/myocardial infarction" which should map both to http://www.ebi.ac.uk/efo/EFO_0001645 "coronary artery disease" and http://www.ebi.ac.uk/efo/EFO_0000612 "myocardial infarction".
In this case, two distinct evidence strings will be, and must be, generated, because eventually during ingestion Open Targets will group all evidence (not only ClinVar) on the pair of (Ensembl gene ID; ontology term ID) values.
But since “Polycystic kidney disease” doesn't have multiple mappings in the database, a different factor must be at play. I'm almost certain that this happens due to how multiple trait names are currently handled. The “Polycystic kidney disease” trait in ClinVar probably has multiple names (synonyms). The pipeline will attempt to map using all of them; but, due to a quirk of how JSON schema v2 was implemented, only one (primary) name is actually stored in the evidence string.
So I suspect there's some synonym for “Polycystic kidney disease” in ClinVar, invisible in the evidence string, but producing this additional mapping through the database.
There will be an upcoming ticket to address this issue and one other related to the ontology mapping process. I'll tell you more during the upcoming meeting. And thanks again for noticing, because I already nearly forgot about all those subtle things happening

Also to address @M-casado's comments

<EMAIL_ADDRESS>Given what Daniel mentioned and how we run the pipeline (copy-pasted codeblocks), should it be a soft check (Warning when a duplicated Evidence String is found) or a hard check (Enforcing duplicated ES are discarded).

The first one may come in handier in case we need to de-bug some issues, but the second one is more straightforward for the one running the commands.

I think it should be the same type of check we have now, I would cause it “soft-hard” :)
It's essentially just an additional command which is being run as a part of a copy-pasted block which generates the file. The protocol then tells the person who's running it to ensure the file is empty, that is, there are no duplicates.
It is a soft check in the sense that it doesn't immediately trip the pipeline or discard any evidence strings.
However, it is also a hard check in the sense that the protocol doesn't (and shouldn't) allow to go any further before this is resolved. That is, there isn't a scenario where we just discard the duplicated evidence strings, because if there are any it means that there's a bug somewhere


Pre-load price data

Currently, when a dynamically priced order is created, Mostro fetches a quote from Yadio to check that the provided fiat amount yields an amount in sats within the values set in the config file as max_order_amount and min_payment_amount. This procedure requires a network call, which, based on my local measurements, takes around 2 seconds to complete. Since both limits of the range are checked, this delay is further duplicated with range orders, resulting in approximately 4 seconds for this processing alone.
Here are a couple of measurement samples:
2024-07-07T17:35:22.958910Z  INFO mostrod::app::order: >> Quote at 0: 8790 in 2.065589375s
2024-07-07T17:35:25.007952Z  INFO mostrod::app::order: >> Quote at 1: 179318 in 2.048965875s

2024-07-07T17:36:23.608319Z  INFO mostrod::app::order: >> Quote at 0: 8790 in 2.01362875s
2024-07-07T17:36:25.557908Z  INFO mostrod::app::order: >> Quote at 1: 35160 in 1.949496833s

We could improve this with range orders by getting a quote only once, extracting the price and using it to check against the opposite limit.
Or we could also introduce a new table which we'd use to store the price of BTC in all fiat currencies tracked by Yadio. There's an API endpoint for that at https://api.yadio.io/exrates/BTC. Here's a sample response:
{"BTC":{"BTC":1,"AED":211246.58,"ALL":5333413.47,"ANG":103694.46,"AOA":50102764.87,"ARS":80873960.91,"AUD":85451.86,"AZN":97772.71,"BDT":6760554.97,"BGN":103978.12,"BHD":21675.87,"BIF":165605558.36,"BMD":57513.36,"BOB":397573.36,"BRL":313303.86,"BWP":805277.93,"BYN":188291.79,"BZD":115974.14,"CAD":78487.56,"CDF":163691664.89,"CHF":51624.4,"CLP":53525416.94,"CNY":418265.92,"COP":230188367.19,"CRC":30760722.56,"CUP":19554542.91,"CZK":1348517.05,"DJF":10244507.82,"DKK":396806.94,"DOP":3401433.49,"DZD":7723431.24,"EGP":2764632.78,"ETB":5809331.65,"EUR":53190.94,"GBP":45032.1,"GEL":158281.89,"GHS":886047.86,"GNF":495443979.97,"GTQ":446858,"HKD":449253.55,"HNL":1425054.4,"HUF":20985075.92,"IDR":941424279.81,"ILS":214542.84,"INR":4802380.64,"IRR":34067464553.37,"IRT":3406746455.34,"ISK":7930517.42,"JMD":9019034.74,"JOD":40759.72,"JPY":9283438.27,"KES":7376088.61,"KGS":4925576.56,"KRW":81317513.09,"KZT":27636272.45,"LBP":5147445854.62,"LKR":17491518.62,"MAD":566762.83,"MGA":258695911.56,"MLC":66286.56,"MRU":2278491.77,"MXN":1031463.95,"MYR":271304.44,"NAD":1043175.51,"NGN":88489355.83,"NIO":2118122.78,"NOK":610540.45,"NPR":7685069.83,"NZD":93793.88,"PAB":57513.36,"PEN":218870.38,"PHP":3369413.9,"PKR":16018203.67,"PLN":229333.61,"PYG":433831076.53,"QAR":209793.44,"RON":266352.49,"RSD":6225763.87,"RUB":5139947.26,"RWF":75276353.08,"SAR":215720.71,"SEK":607061.01,"SGD":77612.79,"SYP":848322082.19,"THB":2096573.56,"TND":179572.53,"TRY":1891484.02,"TTD":389585.91,"TWD":1887128.88,"TZS":152757563.06,"UAH":2650804.46,"UGX":213193898.76,"USD":57513.36,"UYU":2304576.73,"UZS":726042065.5,"VES":2342114.49,"VND":1519222319.22,"XAF":34891045.71,"XAG":1869.04,"XAU":24.36,"XOF":34891045.71,"XPT":58.5,"ZAR":1068724.1},"base":"BTC","timestamp":1720543203583}

This response is ~1.6KB. If we schedule this to run every 5 minutes that would result in a total daily data transfer of 476 KB. Which is acceptable IMO (provided that the API doesn't rate limit us) and would yield in a ~4 second reduction in wait time for the client when posting an order.

I totally agree that this is not a good practice and it slows down the order creation, I'm in favor of your proposal of saving the fiat prices in our database and update it let's say each 5 minutes.
This database values are good to be used as a reference on order creation to check limits, but no when the taker takes the order because we need real time price, so the take order part shouldn't be changed.

Ok this has been implemented in PR 329. The original idea of introducing a database table for this has been ditched for several reasons:

A price data table wouldn't have a relationship with anything else stored in the  database
We don't need (at this point) to keep a long history of price data, which is something that a db would be good for
Keeping data in memory is even better in terms of speed

With this in place, new measurements of the quote retrieval procedure for ranged orders look like the following:
2024-07-14T03:05:58.508829Z  INFO mostrod::app::order: >> Quote at 0: 1683 in 40.5µs
2024-07-14T03:05:58.508868Z  INFO mostrod::app::order: >> Quote at 1: 3366 in 13.167µs

2024-07-14T03:06:47.208367Z  INFO mostrod::app::order: >> Quote at 0: 3366 in 59.458µs
2024-07-14T03:06:47.208414Z  INFO mostrod::app::order: >> Quote at 1: 5050 in 16.5µs

Closed by #329


Intellisense does not pickup Typescript 3 references

From @tommedema on August 27, 2018 3:12

VSCode Version: 1.26.1
OS Version: OSX High Sierra

Steps to Reproduce:

git clone git@github.com:tommedema/serverless-mono-example.git
cd serverless-mono-example && yarn install && code .
if you change a typing in package random, it is not picked up by intellisense in typescript packages that reference to it in the same workspace (e.g. sls-random); you'd have to compile first

for example, change fetchRandomNumber in random to return a Promise<string>, and sls-random still expects a Promise<number>
Note that sls-random is connected with random through Typescript 3 references:
{
  "extends": "../../tsconfig.settings.json",
  "compilerOptions": {
    "outDir": "dist",
    "rootDir": "src"
  },
  "references": [
    { "path": "../random" }
  ]
}

And therefore intellisense should pickup the referenced typings dynamically.

Does this issue occur when all extensions are disabled?: Yes
Copied from original issue: Microsoft/vscode#57242

From the TypeScript Handbook: https://www.typescriptlang.org/docs/handbook/project-references.html

Because dependent projects make use of .d.ts files that are built from their dependencies, you’ll either have to check in certain build outputs or build a project after cloning it before you can navigate the project in an editor without seeing spurious errors. We’re working on a behind-the-scenes .d.ts generation process that should be able to mitigate this, but for now we recommend informing developers that they should build after cloning.

I have a similar problem. Is this a duplicate or is my example different? https://github.com/donaldpipowitch/typescript-refactoring-references

Getting the same issue, have to compile a project for its type intellisense in other projects to update when changes are made.
It's a big problem for a repo I'm working on that has 19 different package references. When running the tsc watch, it can take over 15 sec for a typing to update.

Running into the same issue in our mono-repo.
This is labeled as VS Code Tracked, but the linked bug in their repo is closed. @mjbvz, is this something that needs to be fixed in Typescript or VS Code?

Correct me if I’m wrong, but since vscode relys on TSServer for providing information for intellesence, and the new References feature uses the output .d.ts for each project for typing; the TSServer would have to have a way to bypass the .d.ts files and directly go to the source to determine typings. I would think that this is a TS issue and not only VSCode.

I would like to bump this issue. We are running into this problem in our monorepo and it's making development more difficult in particular for newcomers.

I have a sample project that uses project reference. You need to set properly baseUrl to the top level output and this should enable type checks to work
https://github.com/wmira/react-ts-starter/tree/master/project-reference

I have a sample project that uses project reference. You need to set properly baseUrl to the top level output and this should enable type checks to work
https://github.com/wmira/react-ts-starter/tree/master/project-reference

@wmira That's the current workaround, at least what my team uses. Ideally vscode would understand types as you change them without the watch running or having to run tsc build (which I believe is the issue outlined by mjvbz).

@wmira I'm confused about your setup. Why are these properties directories that do not exist? Also, this still means that you have to compile the typscript to get references to work. As I understand the issue, this is the main problem, not that references do not work at all.

its going to get created when you compile..from the top level. baseUrl is
used by ts to resolved relative paths. E.g. do a tsc --build from the
project-reference project
On Wed, May 1, 2019, 4:32 PM Dominik Moritz<EMAIL_ADDRESS>wrote:

@wmira https://github.com/wmira I'm confused about your setup. Why are these
properties
https://github.com/wmira/react-ts-starter/blob/2ca6ed40242b2cf08e183f5966611d22c2138ebf/project-reference/core/tsconfig.json#L4
directories that do not exist? Also, this still means that you have to
compile the typscript to get references to work. As I understand the issue,
this is the main problem, not that references do not work at all.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/26913#issuecomment-488235055,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADSMEHWTI2536XYEZVRABLPTFISLANCNFSM4FTNWJZA
.

I see. But I still have to compile (and re-compile) to get working jump-to-reference in VSCode across project boundaries. Unfortunately, that's a huge UX problem.

@wmira i just gave your sample a try. However, if you delete `import { greet } from 'module1/util' in App.tsx, you'll notice that vscode can't figure out to auto import greet, back into the file.
It seems that once you include that line, that vscode can find all the symbols within module1/util fine, but before that it can't pick it up. Likewise, if you add a new file, say module/util2 and export a function hello(), vscode won't be able to find that until you've referenced it in at least one file.
Is that something that you were able to overcome with a workaround?

I definitely think this fix is high priority. I my case (yarn workspaces with nohoist and typescript modules) I haven't been able to find a good workaround.

The initial issue is fixed by #32028.

I'm not quite sure what the proposed fix actually is. Ended up using pnpm as package manger which helps resolve this issue pretty easily as it symlinks referenced projects to the importing application's node_modules folder.

Is this fixed? Seems like it isn't and this is the current behaviour:

@wmira i just gave your sample a try. However, if you delete import { greet } from 'module1/util' in App.tsx, you'll notice that vscode can't figure out to auto import greet, back into the file.
It seems that once you include that line, that vscode can find all the symbols within module1/util fine, but before that it can't pick it up. Likewise, if you add a new file, say module/util2 and export a function hello(), vscode won't be able to find that until you've referenced it in at least one file.
Is that something that you were able to overcome with a workaround?


Windows 上での PATH 区切り文字の表示

結果出力時に、ファイル名の PATH を Windows のお作法にのっとって \ で表示されていると
辛い場面があるので、/ で置き換えて出力してくれるオプションを追加して頂けないでしょうか。

While pt aims to support searching text in all sorts of languages, I think the developers would appreciate features being requested in English :smile:

@vermiculus
"It would be nice to have the option which could switch path separators (windows style <> unix/cygwin style) in search results output."
Anyway, closing this issue because I've already given up using pt, and there's no interested party other than me. Thank you for taking care this abondoned issue!

@lurdan You are hardly the only interested party – this is a good idea!


Are extended terminfo files being correctly parsed?

Code contains #define TI_ALT_MAGIC 542, which is the alternative header for terminfo "extended storage format".  This is ultimately used to correctly calculate the strings offset into the compiled terminfo database.
However, term(5) states that the alt magic is 0542, which is octal.  TI_ALT_MAGIC should be set to (int)354, or prefixed with a zero.
I don't have any extended storage format terminfo files handy, but I am happy to test if you point me to where I can get some.

See: https://github.com/nsf/termbox-go/issues/185#issuecomment-370141243
Yes indeed docs mention octal, but it's different in actual files. Weird.

Huh. Apparently it's been acknowledged by the manpage author.  I went to report it upstream but it seems to be known.
https://lists.gnu.org/archive/html/bug-ncurses/2019-01/msg00031.html


man page says Linux

@MikeFultonDev I recall you had complained about this (or something similar). Was there any action taken on it?

Ah there was:

2023-05-24  Bruno Haible<EMAIL_ADDRESS>man pages: Use man page section title from groff or the distro.
Reported by Mike Fulton<EMAIL_ADDRESS>in
https://lists.gnu.org/archive/html/bug-gnu-libiconv/2023-04/msg00027.html.
* man/*.[13]: Remove section title "Linux Programmer's Manual".

There just has not been a new version released yet.


[ZEPPELIN-6040] Run mode "docker" not working properly

What is this PR for?
Several issues have been identified with the "docker" run mode that are causing it to function improperly. Below are the specific problems encountered:
Base Image Version Issue in /scripts/docker/interpreter/Dockerfile:

The base image version is outdated, currently at 0.8.0, which is significantly behind the latest version.
The URL for downloading Spark is not working, causing failures in the build process.

Host Environment Variables Overriding Container Environment Variables:

The JAVA_HOME variable from the host is unnecessarily being passed to the container, potentially causing conflicts with the intended Java environment inside the container.
The PATH from the host is being propagated into the container, which may lead to unexpected behavior due to differing paths and binaries.
The ZEPPELIN_HOME variable from the host is also being passed, which can interfere with the Zeppelin environment within the container.

Miscellaneous Issues:

Additional issues have been identified testing Spark interpreter, I will check for it in another PR

What type of PR is it?
Bug Fix
Todos

[x] - Fix dockerfile
[ ] - Check for major interpreters

[x] md
[x] python
[x] shell
[ ] R
[ ] spark

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/6040

How should this be tested?

Build the Docker image using the Dockerfile located at /scripts/docker/interpreter/Dockerfile.
Set proper configuration variables:

zeppelin.run.mode : docker
zeppelin.docker.container.image : Use the image name for the Docker image built above.

Test the interpreters.

Questions:

Does the license files need to update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

I am currently checking the functionality of the Spark interpreter

the shipped spark version is too old, maybe you can bump it to 3.5

ping @jongyoul @pan3793

@tbonelee To make sure, do we assume the zeppelin home for containers is /opt/zeppelin by default?

@tbonelee To make sure, do we assume the zeppelin home for containers is /opt/zeppelin by default?

Yes, I have matched the zeppelin home in the scripts/docker/interpreter/Dockerfile (which is based on scripts/docker/zeppelin/bin/Dockerfile, so it refers to the zeppelin home in that Dockerfile) with the default zeppelin.docker.container.home configuration value.
In practice, when the run mode is 'docker', the Zeppelin server uploads the interpreter files to the container's zeppelin home path at runtime within the interpreter Docker container, so it might not be strictly necessary to match them. However, to prevent potential conflicts, I determined it would be better to align it with the zeppelin home path of the base image(of dockerfile scripts/docker/zeppelin/bin/Dockerfile).

Actually, Spark hardcoding SPARK_HOME to /opt/spark and HADOOP_CONF_DIR to /opt/hadoop/conf in docker image and Spark on K8s cases, so far it works well.
I don't see any potential issues on hardcoding the ZEPPELIN_HOME inside the docker image, since docker already provides good isolation, to make life easier, I think we can do similar things too.

@pan3793
Thank you for the detailed review.
I've reflected your review and also made some adjustments to the "How should this be tested?" section.
Does this align with the direction you requested?

@pan3793 kindly ping :-)

@jongyoul Just rebased and pushed!


Adding TLDR info

Resolves #66
also want to hold off on including Beth's email for a bit (discussed in the meeting today)

How about something like this:

We could add the slide image at the end of the course maybe?

I think the contents are great. As mentioned in #66 I think we should probably break it up into smaller bits if they are going to be text-on-slide, if no other reason than the text is small :)

I think the contents are great. As mentioned in #66 I think we should probably break it up into smaller bits if they are going to be text-on-slide, if no other reason than the text is small :)

OK - Are we good then with the style of this text-within-the-text version? Maybe if we decide to do slides it could be at the end or somewhere else? I wasn't sure if by "contents are great" you meant the wording or this option of putting the takeaways in the text with the styling I did. I know you wanted people to be able to copy the text too, so I was thinking we at least give that option this way. I will keep thinking about how we could do slide versions.

@carriewright11 I'm sorry! I thought this was another google slide. I love it!

@carriewright11 I'm sorry! I thought this was another google slide. I love it!

No worries :) I'm glad you like it!


Merge Shimada and Lasker Flow

Codecov Report
Attention: Patch coverage is 11.32075% with 141 lines in your changes missing coverage. Please review.

Project coverage is 32.65%. Comparing base (acd01ea) to head (521cadf).

Files with missing lines
Patch %
Lines

echodataflow/utils/xr_utils.py
15.73%
75 Missing :warning:

echodataflow/stages/subflows/mask_prediction.py
0.00%
31 Missing :warning:

echodataflow/utils/flow_utils.py
0.00%
29 Missing :warning:

echodataflow/stages/subflows/echoshader_flow.py
0.00%
2 Missing :warning:

...low/stages/subflows/echoshader_flow_predictions.py
0.00%
2 Missing :warning:

...chodataflow/stages/subflows/initialization_flow.py
50.00%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #132      +/-   ##
==========================================
+ Coverage   32.13%   32.65%   +0.51%
==========================================
  Files          48       50       +2
  Lines        3834     3810      -24
==========================================
+ Hits         1232     1244      +12
+ Misses       2602     2566      -36

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


ExtractorError ("Could not find JS function u'gr'")

When using the youtube-dl with the following argument
"http://www.youtube.com/watch?v=FOP_PPavoLA --verbose", the following verbose output is generated:
youtube-dl http://www.youtube.com/watch?v=FOP_PPavoLA
[youtube] FOP_PPavoLA: Downloading webpage
[youtube] FOP_PPavoLA: Downloading video info webpage
[youtube] FOP_PPavoLA: Extracting video information
[youtube] FOP_PPavoLA: Downloading js player en_US-vflXJM_5_
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self.parse_sig_js(code)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 855, in parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
pi@raspberrypi ~ $ youtube-dl http://www.youtube.com/watch?v=FOP_PPavoLA --verbose
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'http://www.youtube.com/watch?v=FOP_PPavoLA', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.12.13
[debug] Python version 2.7.3 - Linux-4.1.7-v7+-armv7l-with-debian-7.8
[debug] exe versions: avconv 9.14-6, avprobe 9.14-6
[debug] Proxy map: {}
[youtube] FOP_PPavoLA: Downloading webpage
[youtube] FOP_PPavoLA: Downloading video info webpage
[youtube] FOP_PPavoLA: Extracting video information
[youtube] {22} signature length 43.43, html5 player en_US-vflXJM_5
[youtube] FOP_PPavoLA: Downloading player https://s.ytimg.com/yts/jsbin/player-en_US-vflXJM_5/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 663, in extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 290, in extract
return self._real_extract(url)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 1431, in _real_extract
encrypted_sig, video_id, player_url, age_gate)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 887, in _decrypt_signature
'Signature extraction failed: ' + tb, cause=e)
ExtractorError: Signature extraction failed: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

It also happens with http://www.youtube.com/watch?v=YQHsXMglC9A.
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'http://www.youtube.com/watch?v=YQHsXMglC9A', u'--v
erbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2015.12.13
[debug] Python version 2.7.10 - Windows-2008ServerR2-6.1.7601-SP1
[debug] exe versions: ffmpeg N-76860-g72eaf72, ffprobe N-76860-g72eaf72
[debug] Proxy map: {}
[youtube] YQHsXMglC9A: Downloading webpage
[youtube] YQHsXMglC9A: Downloading video info webpage
[youtube] YQHsXMglC9A: Extracting video information
[youtube] {22} signature length 43.43, html5 player en_US-vflXJM_5_
[youtube] YQHsXMglC9A: Downloading player https://s.ytimg.com/yts/jsbin/player-e
n_US-vflXJM_5_/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
File "youtube_dl\extractor\youtube.pyo", line 877, in _decrypt_signature
File "youtube_dl\extractor\youtube.pyo", line 792, in _extract_signature_funct
ion
File "youtube_dl\extractor\youtube.pyo", line 855, in _parse_sig_js
File "youtube_dl\jsinterp.pyo", line 241, in extract_function
ExtractorError: Could not find JS function u'gr'; please report this issue on ht
tps://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-
dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include
its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report thi
s issue on https://yt-dl.org/bug . Make sure you are using the latest version; t
ype  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose fla
g and include its complete output.",)); please report this issue on https://yt-d
l.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to
update. Be sure to call youtube-dl with the --verbose flag and include its compl
ete output.
Traceback (most recent call last):
File "youtube_dl\extractor\youtube.pyo", line 877, in _decrypt_signature
File "youtube_dl\extractor\youtube.pyo", line 792, in _extract_signature_funct
ion
File "youtube_dl\extractor\youtube.pyo", line 855, in _parse_sig_js
File "youtube_dl\jsinterp.pyo", line 241, in extract_function
ExtractorError: Could not find JS function u'gr'; please report this issue on ht
tps://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-
dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include
its complete output.
Traceback (most recent call last):
File "youtube_dl\YoutubeDL.pyo", line 663, in extract_info
File "youtube_dl\extractor\common.pyo", line 290, in extract
File "youtube_dl\extractor\youtube.pyo", line 1431, in _real_extract
File "youtube_dl\extractor\youtube.pyo", line 887, in _decrypt_signature
ExtractorError: Signature extraction failed: Traceback (most recent call last):
File "youtube_dl\extractor\youtube.pyo", line 877, in _decrypt_signature
File "youtube_dl\extractor\youtube.pyo", line 792, in _extract_signature_funct
ion
File "youtube_dl\extractor\youtube.pyo", line 855, in _parse_sig_js
File "youtube_dl\jsinterp.pyo", line 241, in extract_function
ExtractorError: Could not find JS function u'gr'; please report this issue on ht
tps://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-
dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include
its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report thi
s issue on https://yt-dl.org/bug . Make sure you are using the latest version; t
ype  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose fla
g and include its complete output.",)); please report this issue on https://yt-d
l.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to
update. Be sure to call youtube-dl with the --verbose flag and include its compl
ete output.

Thank you for the report. I believe this issue to be fixed in the current version of youtube-dl, 2015.12.18. See our FAQ if you need help updating.

Thanks for the swift response.
Fix confirmed.

Ty working normally again

It is still present even after a deleting and reinstalling. http://i.imgur.com/pDutLaa.png Is this related to my python version? my youtube-dl is up to date (2015.12.18)

@zunjae post the full output of youtube-dl when run with --verbose option.

Thanks. it running!

@dstftw the error is only present while using mpsyt. I've also updated that but I still get errors
mpsyt --version
mpsyt version  : 0.2.5
youtube-dl --version
2015.12.18

@zunjae this is youtube-dl issue tracker not mpsyt. Open a new issue if you can reproduce with latest youtube-dl used directly.

hi . same bug with this video
youtube-dl https://www.youtube.com/watch?v=XrrCrtyM4Bg --verbose
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=XrrCrtyM4Bg', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.02.22
[debug] Python version 2.7.12 - Linux-4.6.0-kali1-amd64-x86_64-with-Kali-kali-rolling-kali-rolling
[debug] exe versions: ffmpeg 3.0.2-4, ffprobe 3.0.2-4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] XrrCrtyM4Bg: Downloading webpage
[youtube] XrrCrtyM4Bg: Downloading video info webpage
[youtube] XrrCrtyM4Bg: Extracting video information
[youtube] {22} signature length 43.40, html5 player en_US-vflL1__zc
[youtube] XrrCrtyM4Bg: Downloading player https://s.ytimg.com/yts/jsbin/player-en_US-vflL1__zc/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 905, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 820, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 883, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'ur'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'ur'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 905, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 820, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 883, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'ur'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 666, in extract_info
ie_result = ie.extract(url)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 316, in extract
return self._real_extract(url)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 1407, in _real_extract
encrypted_sig, video_id, player_url, age_gate)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 915, in _decrypt_signature
'Signature extraction failed: ' + tb, cause=e)
ExtractorError: Signature extraction failed: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 905, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 820, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 883, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'ur'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'ur'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

@playqwerty01 You are using an outdated version of youtube-dl. Read our FAQ if you have troubles updating.

youtube-dl --version --> 2016.04.06
[youtube] -0YdHqlKV8w: Downloading webpage
[youtube] -0YdHqlKV8w: Downloading video info webpage
[youtube] -0YdHqlKV8w: Extracting video information
[youtube] -0YdHqlKV8w: Downloading js player en_US-vflbQww0A
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 989, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 904, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 967, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

cmd = sprintf('youtube-dl --verbose --no-check-certificate --proxy "<IP_ADDRESS>:8080" %s', video_url);
system(cmd);
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--no-check-certificate', u'--proxy', u'<IP_ADDRESS>:8080', u'https://www.youtube.com/watch?v=-0YdHqlKV8w']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.04.06
[debug] Python version 2.7.12 - Linux-3.13.0-43-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 1.2.6-7, avprobe 1.2.6-7, ffmpeg 2.2.3, ffprobe 2.2.3, rtmpdump 2.4
[debug] Proxy map: {u'http': u'<IP_ADDRESS>:8080', u'https': u'<IP_ADDRESS>:8080'}
[youtube] -0YdHqlKV8w: Downloading webpage
[youtube] -0YdHqlKV8w: Downloading video info webpage
[youtube] -0YdHqlKV8w: Extracting video information
[youtube] {22} signature length 43.40, html5 player en_US-vflbQww0A
[youtube] -0YdHqlKV8w: Downloading player https://s.ytimg.com/yts/jsbin/player-en_US-vflbQww0A/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 989, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 904, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 967, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 989, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 904, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 967, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 669, in extract_info
ie_result = ie.extract(url)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 323, in extract
return self._real_extract(url)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 1562, in _real_extract
encrypted_sig, video_id, player_url, age_gate)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 999, in _decrypt_signature
'Signature extraction failed: ' + tb, cause=e)
ExtractorError: Signature extraction failed: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 989, in _decrypt_signature
video_id, player_url, s
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 904, in _extract_signature_function
res = self._parse_sig_js(code)
File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 967, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/usr/lib/python2.7/dist-packages/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'tr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

The latest version is 2016.07.17, 2016.04.06 is quite old.

worked!

Simple method
Remove the package by run the following command in terminal

sudo apt-get remove -f youtube-dl
Reinstall the package by run the following command
sudo apt-get install youtube-dl

version 2016.02.22
same error

@sietse85 2016.02.22 is too old for youtube-dl. The latest version is 2<IP_ADDRESS>.

Thanks, DL the latest version and put in /usr/loca/bin working perfect


#115: add upgrade command and fix last '%' on progress bar

Codecov Report
Attention: Patch coverage is 53.84615% with 36 lines in your changes missing coverage. Please review.

Project coverage is 65.28%. Comparing base (42b8c83) to head (63d79fd).

Files
Patch %
Lines

cmd/gbc/command/deps.go
42.42%
19 Missing :warning:

cmd/gbc/artifact/pty_writer.go
59.25%
7 Missing and 4 partials :warning:

cmd/gbc/artifact/project.go
37.50%
4 Missing and 1 partial :warning:

cmd/gbc/command/build_action.go
0.00%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #116      +/-   ##
==========================================
- Coverage   66.66%   65.28%   -1.39%
==========================================
  Files          21       23       +2
  Lines        1110     1178      +68
==========================================
+ Hits          740      769      +29
- Misses        313      351      +38
- Partials       57       58       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Metadesc/header/other small changes

I see that our metatitle for this is "Solana (SOL) Faucet | thirdweb" - it used to show up like this on Google, but it seems to be showing as "Solana Faucet" now for some reason. (tried on different browsers, incognito, etc.):

Because of this, we're missing out on a lot of traffic that goes to "sol faucet" rather than "solana faucet" - we rank #4 for "solana faucet" but not at all for "sol faucet" & could easily rank in the top 5 for sol faucet as well!
Generally seeing discrepancies across Dashboard pages between what shows up on Google, what it shows up as when shared (og:title), and what shows up when on-page. @jnsdls hope you feel better throughout the week - lmk if you'll be able to make it to our SEO biweekly or if ya wanna push!

📝 Changed routes:

/faucet/solana

(view full size)

Commit 4ea92694e3a09a8928b28dc7b8ce963e33f5ee5f (https://thirdweb-jwwqqzt8n.thirdweb-preview.com).


Add contributor list and script to update it

@EBatTiVo, I've updated the commit to include your change. Using last year contributors seems reasonnable given the age of the plugin.
At some point we might want to create the list of contributors for each version by using git tags.


NixVim improvents!

This serves as a tracking issue for all planned improvements to be made to NixVim.
Documentation

[ ] Better, per-module documentation
[ ] Documentation search
[ ] Document every option

Features

[ ] Support package option for every plugin (#58)
[ ] Support defining keybinds in every plugin
[ ] Support pinning treesitter versions (to fix bugs with the invalid nodes)
[ ] Support which-key - can use the maps option to automatically populate it
[ ] Support autocommands (initially, just extraConfigLua and extraConfigVim are fine, but I'd really like to be able to set per-plugin options for this eventually)
[x] Use vim.keybind.set to define mappings
[ ] Allow lazy-loading of plugins
[ ] Have some form of in-editor documentation (e.g., a :NixVim command which shows a page showing currently enabled options and documentation)

https://github.com/pta2002/nixvim/pull/115 solves the

[ ] Use vim.keybind.set to define mappings
item.

Oops, thanks for reminding me

What about automated tests, just to make sure that nixvim can build the configuration for all plugins. This would require something like github actions (or circleci which i can see is used) to run the tests. And then a simple test configuration for each module. The test files could just be called something like pluginName.test.nix.
The tests can also be run by contributors to verify that the module works as expected.
I think this would greatly improve the review process and make it easier for contributors to catch their errors.
@pta2002

Another improvement could be adding something like devenv and adding a pre commit hook to a formatter (rnix-lsp for example), so that it is okay if we forget to format before committing.

What about automated tests, just to make sure that nixvim can build the configuration for all plugins. This would require something like github actions (or circleci which i can see is used) to run the tests. And then a simple test configuration for each module. The test files could just be called something like pluginName.test.nix.

I quite like the idea of automated tests. I had thought about it, but I haven't really figured out how to do it. I assume there are ways though, will have to look into it. Will add to the list!

I also really like the idea of the pre-commit hook! Will look into devenv.

Any thoughts on how lazy loading could be implemented?

Any thoughts on how lazy loading could be implemented?

Hmm I am not really sure. Obviously, we could look at how lazy nvim does it and try to replicate the same.
Feel free to hack something and try. I will happily review your PR :)

Support which-key - can use the maps option to automatically populate it
Support autocommands (initially, just extraConfigLua and extraConfigVim are fine, but I'd really like to be able to set per plugin options for this eventually)

Aren't those two now solved ?
We have a module for autocommands and it is possible to add a description to each mapping that which-key can use.
@pta2002 was that what you were thinking of when writing those objectives ?

Oh yes they are done! will mark them as done, kinda forgot about this issue

Any thoughts on how lazy loading could be implemented?

With @GaetanLepage on this one - basically just look at how lazy nvim does it, and implement that. It's not necessarily simple but it shouldn't be too hard.

I'm probably not the best person to work on this as I'm fairly new to nix and haven't messed much with lua plugins either. I also don't have much time to dedicate atm. But I've had a quick look through lazy.nvim...
Lazy.nvim actually completely disables vim's plugin loading system and re-implements it itself to gain full control of the process.
I don't think disabling NeoVim's plugin loading is neccessary to lazy load plugins if we can avoid adding them to the runtimepath. If we instead allow neovim to load plugins in the runtimepath as-per normal, we would just need to ensure that any packages marked "lazy" are not added to the rtp by nix's neovim-wrapper.
If we can manage that, then NixVim just needs to add handlers for each lazy-loaded plugin's keybinds/commands/trigger events. These handlers would add the plugin to the rtp and source
files in the relevant directories.
Does neovim-wrapper currently support not adding a plugin to the runtimepath, or would that option need to be implemented upstream?
Overview of lazy.nvim's plugin loading system:
lazy.nvim's entrypoint is the M.setup() function in /lua/init.lua.
From setup() it some sanity checks and sets up caching, then gets into the meat of things loading the config, setting up the loader (Loader.setup()) and finally loading non-lazy plugins (Loader.startup()).
In Config.setup(), vim's plugin loading is disabled using vim.go.loadplugins=false
which gives lazy.nvim full control to load things how it likes and even blacklist builtin plugins such as tutorial to gain a little performance.
Loader.setup()
starts by copying the list of disabled plugins (usually used to prevent loading builtin plugins such as tutorial), then it calls Plugin.load()
which parses the config's plugin list.
Finally, Loader.setup() calls Handler.setup()
which creates event handlers for any keybinds/events/commands/etc defined in the config's plugin list. Lazy plugins are loaded later by these event handlers.
Loader.startup()
initially loads /filetype.lua for compatibility reasons, then it backs up the runtimepath, calls any configured init handlers, finally it loads the non-lazy plugins.
First the configured plugins are loaded, looping over Loader.get_start_plugins() and passing each to Loader.load(). This mutates the runtimepath, hence why the original was copied earlier.
Next Loader.startup() loops over everything in the origional runtimepath (mainly builtin plugins), passing each to Loader.packadd() which sources the rumtime.
Finally, Loader.startup() loops over the new runtimepath, and sources everything in an /after directory.

Wow ! Thank you very much @MattSturgeon for this detailed description of the working of lazy.nvim.
I have never used it myself so this stuff is unfamiliar to me.
I think that lazy loading support would be a great addition to nixvim.
However, given your description, it seems to imply some substantial development effort.
Our approach would probably be different than the one of lazy.nvim as we can perform operations at build time when generating the lua code.
I am opening a new issue to track this development:
https://github.com/pta2002/nixvim/issues/421
I suggest to keep this discussion there.

Could we also look into better lua integration?
Writing lua code as a nix string is far from ideal, and makes even a simple missing comma much harder to catch.
I'm not sure if this is even possible in nix but directly writing lua would be ideal.

@Bodleum I don't think this is something nixvim itself can handle, but if you are using treesitter to highlight your nix files, you can mark strings as lua by doing this:
someValue = /* lua */ ''
  function() --[[ lua code ]] end
'';

What if there were two ways of configuring a plugin with nixvim:

The standard nixvim way that is in use right now.
Nixvim reads a lua file with the desired configuration.

Each package could come with an option to configure with lua, which you can pass in a lua file path and nixvim will use the configuration there to configure the plugin. I'm envisaging something like how lazy.nvim does it, A table with the plugin name, opts, config, etc. Maybe even compatibility is possible? I think this would reduce the "double-configuring" feel as well.
This is something I'd be willing and keen to work on, with a bit of assistance!

You could always use builtins.readFile to achieve this

What if there were two ways of configuring a plugin with nixvim:

The standard nixvim way that is in use right now.

Nixvim reads a lua file with the desired configuration.

Each package could come with an option to configure with lua, which you can pass in a lua file path and nixvim will use the configuration there to configure the plugin. I'm envisaging something like how lazy.nvim does it, A table with the plugin name, opts, config, etc. Maybe even compatibility is possible? I think this would reduce the "double-configuring" feel as well.
This is something I'd be willing and keen to work on, with a bit of assistance!

There is kinda allready a way using extraluaconfig by listing the file by using the builtin.readFile from nix to output the file.

Hey i am pretty interested on the documentation command in vim you just do vim,command so what if nixvim had a command that loads a man page like how neovim does it already.

There's not many outstanding items on this issue, and its scope is rather broad. I think we can close this now?
I'm not sure "support defining keybinds in every plugin" is an achievable goal.
"Better, per-module documentation" is rather vague. Is this meaning the platform wrapper modules? How do we define better?
"Allow lazy-loading of plugins" is already tracked in #421
"Have some form of in-editor documentation (e.g., a :NixVim command which shows a page showing currently enabled options and documentation)" has been opened as https://github.com/nix-community/nixvim/issues/1706
All other items are marked as complete.


Make file processing with ClassDefVisitor deterministic

Related to #117 (non-determinism part)

Change queue creation and element addition while processing
Sort files before processing (Path.glob doesn't guarantee any order)

Thanks @slafs ❤


Feedback from implementing iterators

I've been working on writing an iterator that is relatively boiler-plate free, akin to how slices can do for (slice) |value| { ... } and for (slice) |value, idx| { ... }. I was pleasantly surprised with how well everything composes in general, but a bit of feedback on how it could be nicer:
Here is some code for a custom iterator. Notes at bottom.
const Iterator = struct {
  count: i32,
  current: i32,

  pub fn next(self: &Iterator) ?i32 {
    if (self.current >= self.count) return null;
    const result = self.current;
    self.current += 1;
    return result;
  }

  const WithIndex = struct {
    value: i32,
    idx: i32
  };
  pub fn nextWithIndex(self: &Iterator) ?WithIndex {
    // It'd be much more natural to write this with an early return.
    if (self.next()) |value| {
      return WithIndex {
        .value = value,
        .idx = 1
      };
    }

    return null;
  }
};

fn buildIterator(count: i32) Iterator {
  return Iterator {
    .count = count,
    .current = 0
  };
}

fn useCase() void {
  { var it = buildIterator(10); while (it.next()) |i| {
    std.debug.warn("i={}\n", i);
  } }
  // It'd be great if `|e|` could be `|value, index|`
  { var it = buildIterator(10); while (it.nextWithIndex()) |e| {
    std.debug.warn("value={}, idx={}\n", e.value, e.idx);
  } }
}

In nextWithIndex, an early return would read more cleanly, especially when code starts getting complicated and there are a lot of reasons to exit the function. However, the unwrapping if makes this hard to write ; maybe if can adjust the type for the rest of the block so it is no longer optional?

The while loops in useCase are very ugly to look at (the double } }) and require a lot more typing than the typical for (init; cond; increment) loop.

Instead of using a proxy type WithIndex, it'd be great if the |e| could unwrap directly to |value, idx|. Two quick ideas on how this could be done:

multiple return values
simple name-based pattern matching

Another option is to standardize this type of thing into the compiler so for (foo) |x| looks for a method with a specific name/result-type, ie, for (foo) |x| ==> next1() IteratorResult(u32), for (foo) |x, y| ==> next2() IteratorResult(u32, bool), etc.

Here's another minor tip, subject to your preference:
  pub fn next(self: &Iterator) ?i32 {
    if (self.current >= self.count) return null;
    defer {self.current += 1;}
    return self.current;
  }