Add docker

Added two extra Dockerfiles and associated documentation.

Another suggestion is to organize dockerfiles into a dedicated directory

@fedorov, @jcfr, Based on the comments, I'm going to propose that we settle down on 1 Dockerfile.  It will be based on the jupyter/datascience-notebook and install the example notebooks, as well as the pyradiomics command.  In the documentation, we can give examples on how to start up the notebook interface, and use the command in the Docker image.  I do realize the datascience-notebook is heavyweight, but it does come with a huge amount of helpful packages already baked in.  Having our examples (and more would be better) already baked in is a huge benefit.
We could also try to produce the smallest possible image for the command line version.
I hadn't planned on squashing the commits, it didn't seem to be the style in this repo.  Is it necessary?

Personally, I really really would hate the perfect to be the enemy of the good here. We can always improve later, but I think it is better to provide this functionality to the user early than have this PR pending much longer.

@fedorov Happy to make changes, but I also agree with you.  We can iterate on Docker support.  I'll let someone else approve the PR.

@blezek would be great if you do those changes that you can and agree with today or tomorrow, we really would like to add docker support before the end of the week in time for the QIN meeting.

+1 to making a full featured version available sooner and offering a
stripped down version later.
On Thu, Apr 6, 2017 at 7:23 AM, Andrey Fedorov<EMAIL_ADDRESS>wrote:

@blezek https://github.com/blezek would be great if you do those
changes that you can and agree with today or tomorrow, we really would like
to add docker support before the end of the week in time for the QIN
meeting.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Radiomics/pyradiomics/pull/236#issuecomment-292144391,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHsfXQhlT3g-_J-72OtkDxZsfut3uA0ks5rtMtLgaJpZM4M0ofa
.

offering a stripped down version later.

You are right size of the images is now critical problem.
That said, the other issues with the current approach is that we do not know if the created images will work or not.
To keep up with the high bar maintained so far :smile: , I would hight recommend to add test(s) checking  that the built docker images works as expected. For example:

build and publish the image as part of the CI process (e.g execute a basic notebook that would import pyradiomics and do a simple computation or even run the complete test suite). The new CircleCi 2.0 makes working with docker even easier.

If not we should at least update the release check list to ensure the images are manually tested before each release.
Finally, for any given docker images ... we should be able to tracker what is the corresponding pyradiomics SHA version. This can be done easily adding metadata. See https://microbadger.com/labels#!
For example: https://github.com/QIICR/dcmqi/blob/f621a32eed0eadac852cee9860addbffb5d92460/docker/Makefile#L65-L72
I know it is tempting to move forward .. but I really think having systematic testing and provenance tracking for the docker images will pay off.

Absolutely - testing is critical.  Let's make sure the user of the image
can verify that the image works as we intended.  A self test command that
the user can easily run would be very helpful.  Let's then add running the
tests to a notebook tutorial.
On Thu, Apr 6, 2017 at 9:21 AM, Jean-Christophe Fillion-Robin <
<EMAIL_ADDRESS>wrote:

offering a stripped down version later.
You are right size of the images is now critical problem.
That said, the other issues with the current approach is that we do not
know if the created images will work or not.
To keep up with the high bar maintained so far 😄 , I would hight
recommend to add test(s) checking that the built docker images works as
expected. For example:

build and publish the image as part of the CI process (e.g execute a
basic notebook that would import pyradiomics and do a simple computation or
even run the complete test suite). The new CircleCi 2.0 makes working with
docker even easier.

If not we should at least update the release check list to ensure the
images are manually tested before each release.
Finally, for any given docker images ... we should be able to tracker what
is the corresponding pyradiomics SHA version. This can be done easily
adding metadata. See https://microbadger.com/labels#!
For example: https://github.com/QIICR/dcmqi/blob/
f621a32eed0eadac852cee9860addbffb5d92460/docker/Makefile#L65-L72
I know it is tempting to move forward .. but I really think having
systematic testing and provenance tracking for the docker images will pay
off.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Radiomics/pyradiomics/pull/236#issuecomment-292172486,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHsfVBfzdx5PZ1ZWrNq5Ugs9v5SxK0yks5rtObzgaJpZM4M0ofa
.

A self test command that the user can easily run would be very helpful. Let's then add running the tests to a notebook tutorial.

Great. To achieve this, we could:

bundle the tests as part of the package. Good news is that it is already the case. See https://github.com/Radiomics/pyradiomics/blob/913473e13fdd9657f5ce9e1c836cd24fac61e748/MANIFEST.in#L14

install and execute nose.

bonus: systematically executing all notebook should also be done.

With that, tests can be executed by anyone installing the package. And in that particular case, since docker provides a consistent and repeatable environment. We can really ensure the tests run in the environment we provide right after we build the docker images.

Sounds awesome!
On Thu, Apr 6, 2017 at 9:36 AM, Jean-Christophe Fillion-Robin <
<EMAIL_ADDRESS>wrote:

A self test command that the user can easily run would be very helpful.
Let's then add running the tests to a notebook tutorial.
Great. To achieve this, we could:

bundle the tests as part of the package. Good news is that it is
already the case. See https://github.com/Radiomics/pyradiomics/blob/
913473e13fdd9657f5ce9e1c836cd24fac61e748/MANIFEST.in#L14
https://github.com/Radiomics/pyradiomics/blob/913473e13fdd9657f5ce9e1c836cd24fac61e748/MANIFEST.in#L14
install and execute nose.
bonus: systematically executing all notebook should also be done.
With that, tests can be executed by anyone installing the package. And in
that particular case, since docker provides a consistent and repeatable
environment. We can really ensure the tests run in the environment we
provide right after we build the docker images.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Radiomics/pyradiomics/pull/236#issuecomment-292176488,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHsfQDB9GSF9E52uOdu6coiqejrN30lks5rtOpGgaJpZM4M0ofa
.

Great.
@blezek Let us know if you have any questions. I would be happy to review further.

Hey @jcfr @pieper @fedorov & @JoostJM,
Bunch of commits, but I slimmed down to on Dockerfile that runs the example notebooks as part of the build with Python 2 and 3.  Also added a CircleCI v2 configuration that installs radiomics under Python 2 & 3, tests and executes the notebooks.
A working example build is at https://circleci.com/gh/blezek/pyradiomics/9.
If we configure DockerHub to auto build, we should have a nice pipeline going.

Hi @blezek I checked out the branch and ran 'docker build' but got the error below.
---> ebf7a1a7a6a3
Removing intermediate container 66d0c72f5b2d
Step 13 : ADD bin/Notebooks/FeatureVisualizationWithClustering.ipynb /home/jovyan/work/
lstat bin/Notebooks/FeatureVisualizationWithClustering.ipynb: no such file or directory
The file is referenced here, but looks like it's not in your branch.
https://github.com/Radiomics/pyradiomics/pull/236/files#diff-3254677a7917c6c01f55212f86c57fbfR33

Sorry @pieper, I missed it.  Try again.  Need someone to look at that file and decide if it should be added to the Docker or not.

I confirm I could build docker without problems. Thank you Dan!

I am merging this so we officially have docker support in time for the QIN meeting that starts tomorrow.

@JoostJM you will need to configure docker hub yourself, it does not work for me, perhaps because of some permission issue. You will need to create Automated build, not Repository, which you have at the moment. I cannot do this, because somehow radiomics organization is not showing up in the list when I initiate creation of a new Automated build.

I was not able to access radiomics/pyradiomics organization from Docker hub because of the third-party access restriction policy. After disabling it, I can see the organization and the repo. However, I am still not able to create automated build under radiomics organization on Docker hub.

@fedorov, I just added it. Setting it up now.

PyRadiomics docker is building.


⚠ Bofrost.es has degraded performance

In 2ef9672, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1533 ms

Resolved: Bofrost.es performance has improved in 189b032 after 10 minutes.


Remove dependency on k8s.io/kubernetes/pkg/apis/policy from kubectl

Remove dependencies from packages under k8s.io/kubernetes/pkg/kubectl/... on k8s.io/kubernetes/pkg/apis/policy by converting them to use the k8s.io/api/policy/v1beta1 package

Hi @pwittrock I'd like to pick this issue.

@alessbn, my apologies, I didn't see your comment on this issue when I started working on another PR that fixes some Pod Disruption Budget issues. https://github.com/kubernetes/kubernetes/pull/54411 fixes partially fixes this, but there're still some other files to update. I already started working on them, but as you commented here first that you'll pick the issue, I will stop working on it.

@frodenas thank you, as a new contribuitor is not easy for me pick an issue :sweat_smile:

Hi @frodenas you can keep working on the issue I'll pick up issue 95 😄


AMBARI-24676. Issues w.r.t Manage Journal Nodes (Particularly deletion) in HA enabled cluster

What changes were proposed in this pull request?
we have a cluster of 6 hosts (postfixes are 5, 6, 8, 9, 10 and 14)
this is a federated cluster with two namespaces: ns1 and ns2
Namenode HA is enabled; JNs are located on 5, 6, and 14
Added a new JN using the "Manage Journalnodes" wizard on host 8
Removed an existing JN using the "Manage Journalnodes" wizard from host 6
Relevant configurations after these steps:
dfs.namenode.shared.edits.dir=qjournal://ctr-e138-1518143905142-472105-01-000014.host.site:8485;ctr-e138-1518143905142-472105-01-000008.host.site:8485;ctr-e138-1518143905142-472105-01-000005.host.site:8485/ns1,ns2
dfs.namenode.shared.edits.dir.ns1=qjournal://ctr-e138-1518143905142-472105-01-000005.host.site:8485;ctr-e138-1518143905142-472105-01-000006.host.site:8485;ctr-e138-1518143905142-472105-01-000014.host.site:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ctr-e138-1518143905142-472105-01-000005.host.site:8485;ctr-e138-1518143905142-472105-01-000006.host.site:8485;ctr-e138-1518143905142-472105-01-000014.host.site:8485/ns2
How was this patch tested?
21822 passing (38s)
48 pending

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4034/
Test PASSed.


Update citadel.md

Project Abstract
Hello Web3 Foundation team.
We've followed all the instructions that you sent and updated our application. We've added a final delivery date and updated the price to an appropriate USD amount.
Thanks!
For which grant level are you applying?

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $50,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for > $100k Web3 Foundation Council approval)

#81

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks! Could you also add deliverables 0a-e from the example milestone in the application template to yours? These are mandatory for all grants, which probably wasn't in place yet when you applied.
Besides, could you confirm that the UI components will also be open-sourced and not depend on closed-source components, as with rest of the project?

@alxs Hello! We will add deliverables 0a-e in a few days.
Yes, our UI components will also be open-sourced as with rest of the project.

Sure. I will close the current delivery PR in the meantime.
Please keep in mind that following up and reviewing the amendments takes up resources on our side, so we would expect you to stick to the timeline this time around.

@citadeldao ping.

Hello, we've replaced # Roadmap Overview with the lines you mentioned above.
Please review our updated application. Please let us know if you need more details or have any questions.
Thanks!

I just moved them above the existing deliverables - sorry for the confusion.


Add QueryRequest::Grpc

closes #1966
Do we also want to remove / deprecate the Stargate query? Only on the contract side ofc, since we need to continue supporting it in wasmd for 1.x compatibility.

Do we also want to remove / deprecate the Stargate query? Only on the contract side ofc, since we need to continue supporting it in wasmd for 1.x compatibility.

Let's deprecate it. The migration is not fully trivial since the contract needs to change the decoding from JSON to protobuf. It's a bit nicer to give people time for that.


MGMT-10364 Remove dupplicated message

:tada: This PR is included in version 1.2.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Revert "Support a group size constraint in content clusters"

Reverts vespa-engine/vespa#25667
There are some class loading issues with this change:
java.lang.LinkageError: loader constraint violation: when resolving method 'com.yahoo.config.provision.Capacity com.yahoo.config.provision.Capacity.from(com.yahoo.config.provision.ClusterResources, com.yahoo.config.provision.ClusterResources, com.yahoo.collections.IntRange, boolean, boolean, java.util.Optional)' the class loader org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @37b1149b of the current class, com/yahoo/vespa/model/builder/xml/dom/NodesSpecification, and the class loader org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @1a15b789 for the method's defining class, com/yahoo/config/provision/Capacity, have different Class objects for the type com/yahoo/collections/IntRange used in the signature (com.yahoo.vespa.model.builder.xml.dom.NodesSpecification is in unnamed module of loader org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @37b1149b, parent loader 'bootstrap'; com.yahoo.config.provision.Capacity is in unnamed module of loader org.apache.felix.framework.BundleWiringImpl$BundleClassLoader @1a15b789, parent loader 'bootstrap')

Was fixed by https://github.com/vespa-engine/vespa/pull/25671


Update excludelist: removing libgdk-x11-2.0.so.0 per #145

Per title, removing libgdk-x11-2.0.so.0 from excludelist per issue #145.

Rather than just deleting it, please comment it out and add a comment describing why it cannot be assumed to be part of each target system. Thanks!

Well it wasn't part of Fedora 24, so if it is part of the assumption, it's a faulty assumption, rofl. This is the GNOME edition of Fedora 24 too, so of all desktop environments one would assume it'd be the one to provide this library if it is a GTK+ 2 one.

Done.

Could it be that a similarly-but-slightly-differently-named file exists on Fedora? Maybe we could symlink that instead? I still think this is a core part of GTK+ 2.0...

You're right, except it's not similarly named it's identically named:
# find /usr/lib64 -name "libgdk-x11*"
/usr/lib64/libgdk-x11-2.0.so.0
/usr/lib64/libgdk-x11-2.0.so.0.2400.31
/usr/lib64/libgdk-x11-2.0.so

So, why doesn't it use it then? Why should we bundle another one?

Don't know. But it causes this error, maybe it's related to the specific version of this library? Or maybe it's because it's looking for this library in /lib64 instead of /usr/lib64.

Please run with LD_DEBUG=libs ./the.AppImage and look for errors. Or sudo strace -eopen -f the.AppImage

LD_DEBUG=libs ./<AppImage> on Fedora 24 gave: http://paste2.org/4J7swGXk. strace gave http://paste2.org/Ph6558m9.

Based on the debug libs output I think I am right, the library provided by Fedora is a different version to that required by the AppImage.

Are you sure that you ran the strace command with sudo?

Based on the debug libs output I think I am right, the library provided by Fedora is a different version to that required by the AppImage.

Looking at which output?

first comment no, didn't realize it was necessary, one sec I'll do that: http://paste2.org/Ph6558m9. Second comment based on:

  2173:	find library=libgdk_pixbuf-2.0.so.0 [0]; searching
 2173:	 search path=/tmp/.mount_w2XqD4/usr/bin/../lib		(RPATH from file /tmp/.mount_w2XqD4/usr/bin/geany)
 2173:	  trying file=/tmp/.mount_w2XqD4/usr/bin/../lib/libgdk_pixbuf-2.0.so.0
 2173:	 search path=./lib/tls/x86_64:./lib/tls:./lib/x86_64:./lib:./lib/i386-linux-gnu/tls/x86_64:./lib/i386-linux-gnu/tls:./lib/i386-linux-gnu/x86_64:./lib/i386-linux-gnu:./lib/x86_64-linux-gnu/tls/x86_64:./lib/x86_64-linux-gnu/tls:./lib/x86_64-linux-gnu/x86_64:./lib/x86_64-linux-gnu:./lib32/tls/x86_64:./lib32/tls:./lib32/x86_64:./lib32:./lib64/tls/x86_64:./lib64/tls:./lib64/x86_64:./lib64		(LD_LIBRARY_PATH)
 2173:	  trying file=./lib/tls/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/tls/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/i386-linux-gnu/tls/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/i386-linux-gnu/tls/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/i386-linux-gnu/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/i386-linux-gnu/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/x86_64-linux-gnu/tls/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/x86_64-linux-gnu/tls/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/x86_64-linux-gnu/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib32/tls/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib32/tls/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib32/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib32/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib64/tls/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib64/tls/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib64/x86_64/libgdk_pixbuf-2.0.so.0
 2173:	  trying file=./lib64/libgdk_pixbuf-2.0.so.0
 2173:	 search cache=/etc/ld.so.cache
 2173:	  trying file=/lib64/libgdk_pixbuf-2.0.so.0

Oops no I mean this output:
      2173:	find library=libgdk-x11-2.0.so.0 [0]; searching
      2173:	 search path=/tmp/.mount_w2XqD4/usr/bin/../lib		(RPATH from file /tmp/.mount_w2XqD4/usr/bin/geany)
      2173:	  trying file=/tmp/.mount_w2XqD4/usr/bin/../lib/libgdk-x11-2.0.so.0
      2173:	 search path=./lib/tls/x86_64:./lib/tls:./lib/x86_64:./lib:./lib/i386-linux-gnu/tls/x86_64:./lib/i386-linux-gnu/tls:./lib/i386-linux-gnu/x86_64:./lib/i386-linux-gnu:./lib/x86_64-linux-gnu/tls/x86_64:./lib/x86_64-linux-gnu/tls:./lib/x86_64-linux-gnu/x86_64:./lib/x86_64-linux-gnu:./lib32/tls/x86_64:./lib32/tls:./lib32/x86_64:./lib32:./lib64/tls/x86_64:./lib64/tls:./lib64/x86_64:./lib64		(LD_LIBRARY_PATH)
      2173:	  trying file=./lib/tls/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/tls/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/i386-linux-gnu/tls/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/i386-linux-gnu/tls/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/i386-linux-gnu/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/i386-linux-gnu/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/x86_64-linux-gnu/tls/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/x86_64-linux-gnu/tls/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/x86_64-linux-gnu/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib32/tls/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib32/tls/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib32/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib32/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib64/tls/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib64/tls/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib64/x86_64/libgdk-x11-2.0.so.0
      2173:	  trying file=./lib64/libgdk-x11-2.0.so.0
      2173:	 search cache=/etc/ld.so.cache
      2173:	  trying file=/lib64/libgdk-x11-2.0.so.0

While we're on the topic of blacklisted libraries libselinux.so.1 may be necessary as Arch Linux gives the error (according to geany/geany#1303):
zenity, kdialog, Xdialog missing. Skipping ./bin//geany.wrapper.
/tmp/.mount_3N7xOv/usr/bin/geany: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory

https://github.com/probonopd/AppImages/issues/145#issuecomment-262383994

rather than including libgdk-x11, I guess X11 libs should be excluded (and here yours override the system's, and miss that symbol the system's gdk-x11 requires)

@b4n may be right here - needs to be verified by experimentation on different distros.

I also think that very common X11 libs should be excluded by default (libx11, libext6; but not the libxcb* stuff or "rare" libraries that are usually not installed or even dropped by some distros). I would also exclude glib2 and gtk+-2 and their companion libraries (libatk, libgdk*, ...), as those libraries are normally installed by default and they have a very stable API.
Is there a mechanism to easily override or modify the blacklisted libraries and/or Debian packages if that's required or wanted for a certain AppImage?

I also think that very common X11 libs should be excluded by default (libx11, libext6; but not the libxcb* stuff or "rare" libraries that are usually not installed or even dropped by some distros).

Do you know which of these can really be expected to (still) be part of each distro? (even distros using Wayland?)

I would also exclude glib2 and gtk+-2 and their companion libraries (libatk, libgdk*, ...), as those libraries are normally installed by default and they have a very stable API.

You may be right there. Feel free to send a PR but include the rationale as comments in the exclude(deb)list.

Is there a mechanism to easily override or modify the blacklisted libraries and/or Debian packages if that's required or wanted for a certain AppImage?

No, by design. So that we are forced to learn and share the insights (i.e., if we change the blacklist all AppImages benefit from it).

but not the libxcb* stuff

Well, on my Debian unstable, libcairo2, which is a dependency of GTK2 and 3 among a lot of things, depends on libxcb, libxcb-shm and libxcb-render.
Basically, I see not much reason to not exclude everything that is a direct dependency of one the the otherwise excluded things.

Do you know which of these can really be expected to (still) be part of each distro? (even distros using Wayland?)

I don't know, but I'm kind of afraid that either X is an indirect dependency of the app (e.g. through GTK), or the app is likely not to work on Wayland anyway.

Well, on my Debian unstable, libcairo2, which is a dependency of GTK2 and 3 among a lot of things, depends on libxcb, libxcb-shm and libxcb-render.
Basically, I see not much reason to not exclude everything that is a direct dependency of one the the otherwise excluded things.

It's better to include the libxcb* stuff because I've already experienced issues with apps being compiled on Ubuntu 12.04 that didn't work on newer versions because of a soname version bump of a libxcb* library.
Taking a quick look into the Ubuntu repositories[1] also reveals that not all libxcb libraries are available on all distros:
libxcb-present0: >= 14.04
libxcb-sync0: 12.04
libxcb-sync1: >= 14.04
libxcb-util0: <= 14.04
libxcb-util1: >= 16.04
...

Basically, I see not much reason to not exclude everything that is a direct dependency of one the the otherwise excluded things.

That kinda makes sense, but how do we know that the same stuff is not a dependency of some app directly, i.e., not through "the otherwise excluded things"?

Here's a first list based on what's installed on my system and what's available since Ubuntu 12.04: https://gist.github.com/darealshinji/21f368ddb61f4d3cfa303e228bb0e9db
It would of course be a good idea to check if these are all available by default on Live CDs of other distros.

https://github.com/saiarcot895/chromium-ubuntu-build/issues/12#issuecomment-303908033 might be affected by this


Doesn't work with newer S3 regions, because signatureversion v4 requirement

All AWS regions started after 2014 require signatureversion v4 when accessing S3 buckets. The fix is simple - just need to define the signatureversion to be used in the AWS.S3 constructor.
diff --git a/resize/index.js b/resize/index.js
index 007d1ac..7556886 100644
--- a/resize/index.js
+++ b/resize/index.js
@@ -2,7 +2,7 @@ var async = require("async");
 var AWS = require("aws-sdk");

 var im = require("gm").subClass({imageMagick: true});
-var s3 = new AWS.S3();
+var s3 = new AWS.S3({signatureVersion: 'v4'});

 function getImageType(objectContentType) {
   if (objectContentType === "image/jpeg") {
diff --git a/site-builder/index.js b/site-builder/index.js
index 79cae0e..37a69e1 100644
--- a/site-builder/index.js
+++ b/site-builder/index.js
@@ -1,5 +1,5 @@
 var AWS = require("aws-sdk");
-var s3 = new AWS.S3();
+var s3 = new AWS.S3({signatureVersion: 'v4'});
 var cloudfront = new AWS.CloudFront();

 var async = require('async');

I haven't tested the above with "old" AWS regions, but they should work

Thanks for the recommendation @muige! I've applied your patch in 6de470b.


Topic type filter should not be clickable for spectators

The topic type filter in Recent conversations has disabled styling for spectators, but a spectator can still click on it to open the dropdown. We should fix this.

Working on it!


Cannot test a component that is both a drag source and a drop target

I have a component in my app that is both a drag source and a drop target, exactly like in this example: https://github.com/gaearon/react-dnd/blob/master/examples/04 Sortable/Simple/Card.js#L92
When I use the test backend to start a drag operation, I get the following error:
Invariant Violation: Expected a valid source ID.

Looking at the source, it looks like it's coming from parseRoleFromHandlerId, which expects the ID to start with either S for a drag source, or T for a drop target. My component is both, and its ID is "T12", so it's not recognized as a drag source by the test backend.
From the examples, I gathered that this use-case is a supported one, but it would be nice to be able to test it.

OK, I found how to do this. I'll document it here in hopes that it helps future devs Googling the error message.
The solution is to export the intermediate item, like so:
class MyComponent extends React.Component {
  // ...
}

export const MyComponentDragSource = DragSource(/* ... */)(MyComponent);
// ^^^ export the intermediate component

export default const MyComponentDropTarget = DropTarget(/* ... */)(MyComponentDragSource);

Then you can use it in your test like so:
import MyComponent, {MyComponentDragSource} from '../MyComponent';

// ...
      // Inside test
      const root = TestUtils.renderIntoDocument(<MyComponent />);
      const backend = root.getManager().getBackend();

      const item = TestUtils.findRenderedComponentWithType(root, MyComponentDragSource);
      // Notice we find the DragSource, not the regular component:          ^^^^^^^^^^
      backend.simulateBeginDrag([item.getHandlerId()]); // Will work now
// ...

My god, thank you


Epoll info

Not sure if relevant but I found it rather interesting...
https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/
https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/

Epoll antipropaganda rather. I've had this sent to me many times and every time I go facepalm. µWS has one event-loop per thread and all sockets on one group belong to the same thread. This makes it completely separate from other threads and makes it possible to scale linearly (well, properly linear scaling requires mTCP) since there is no data sharing between threads.


The "bluee"!

Apparently there is this typo in the repository which has occurred 43 times in 20 files (probably due to copy-pasting). I don't know if it should be fixed or not.

Thanks for raising this issue, @amn41 will get back to you about it soon✨
Please also check out the docs and the forum in case your issue was raised there too 🤗

thanks for catching! @koaning can you please review :)

Ah yeah, that's an awkward typo. I might argue that it's not the end of the world, but I'll gladly accept any PR that addresses it.

hello @koaning Sir I have made a new PR https://github.com/RasaHQ/whatlies/pull/304
can you review it now please

This is now fixed.


Add REGEX.REPLACE

Work in progress implementation for REGEX.REPLACE()

Sorry, Keith. Totally forgot about this. I'll complete this soon. If I remember correctly, I was considering some breaking changes. Are you open to that, if the outcome justifies it?

I'm open to anything.


http-client-java, doc, require JDK 17

No changes needing a change description found.

You can try these changes here

🛝 Playground
🌐 Website
📚 Next docs


fix: update defaults

I just ran the script 🙂

FYI, from https://github.com/kevva/npm-conf/pull/7#pullrequestreview-84559407, it looks like you need to make the changes in lib/make.js


"Parameter is incorrect" when using WebAuthenticationCoreManagerInterop

Describe the bug
Calling WebAuthenticationCoreManagerInterop.RequestTokenForWindowAsync(hwnd, request) from a WinUI 3 Desktop app always returns with a status of ProviderError. The error message is as follows:
The parameter is incorrect.

onecoreuap\shell\cloudexperiencehost\onecore\app\msa\core\TokenRequestParams.cpp(175)\MicrosoftAccount.TokenProvider.Core.dll!00007FFA3DA5453D: (caller: 00007FFA3DA88115) Exception(1) tid(73b8) 80070057 The parameter is incorrect.
    CallContext:[\RequestTokenAsyncActivity]

The error persists regardless of whether my package is associated with the Microsoft Store.
Steps to reproduce the bug

Create a new WinUI 3 Desktop project (either single-project or WAP, neither work)
Add the following to MainWindow.xaml.cs, along with the necessary using statements:

private AccountsSettingsPane _asp;
private AccountsSettingsPane _hwnd;

public MainWindow()
{
    this.InitializeComponent();
    _hwnd = WinRT.Interop.WindowNative.GetWindowHandle(this);
}

private async void OnLoaded(object sender, RoutedEventArgs e)
{
    _asp = AccountsSettingsPaneInterop.GetForWindow(_hwnd);

    _asp.AccountCommandsRequested += Asp_AccountCommandsRequested;
    await AccountsSettingsPaneInterop.ShowAddAccountForWindowAsync(_hwnd);
}

private async void Asp_AccountCommandsRequested(AccountsSettingsPane sender, AccountsSettingsPaneCommandsRequestedEventArgs args)
{
    var deferral = args.GetDeferral();

    var msaProvider = await WebAuthenticationCoreManager.FindAccountProviderAsync(
        "https://login.microsoft.com", "consumers");

    var command = new WebAccountProviderCommand(msaProvider, GetMsaTokenAsync);
    args.WebAccountProviderCommands.Add(command);

    deferral.Complete();
}

private async void GetMsaTokenAsync(WebAccountProviderCommand command)
{
    WebTokenRequest request = new(command.WebAccountProvider, "wl.basic");

    WebTokenRequestResult result = await WebAuthenticationCoreManagerInterop.RequestTokenForWindowAsync(_hwnd, request);

    if (result.ResponseStatus == WebTokenRequestStatus.Success)
    {
        string token = result.ResponseData[0].Token;
    }
}

Expected behavior
Successfully retrieve account information using the Web Account Manager.
Screenshots
No response
NuGet package version
WinUI 3 - Windows App SDK 1.1.3
Windows app type

[ ] UWP
[X] Win32

Device form factor
Desktop
Windows version
Windows Insider Build (xxxxx)
Additional context
It appears that a similar issue #5827 was reported last year, but closed because it use WinUI 3 UWP.

The trace output line (thank you!) points to a check that the app provides BOTH an application callback URI and a client ID.  The code above doesn't set ClientId - https://docs.microsoft.com/en-us/uwp/api/windows.security.authentication.web.core.webtokenrequest.clientid?view=winrt-22621#windows-security-authentication-web-core-webtokenrequest-clientid so maybe start there? There's a three-parameter form of the WebTokenRequest constructor that takes the provider, scope, and client ID together.  See also this sample:
https://github.com/microsoft/Windows-universal-samples/blob/ad9a0c4def222aaf044e51f8ee0939911cb58471/Samples/WebAccountManagement/cs/SingleMicrosoftAccountScenario.xaml.cs#L179-L189

I have seen a similar issue when accessing onedrive files from a WinUI desktop app compared to a UWP app even when using a manifest file with the same package id. I could get it working by copying some code from the msal library:
webTokenRequest.Properties.Add("api-version", "2.0"); // request V2 tokens over V1
webTokenRequest.Properties.Add("oauth2_batch", "1"); // request tokens as OAuth style name/value pairs
webTokenRequest.Properties.Add("x-client-info", "1"); // request client_info

However, I am not sure why this is necessary.

The trace output line (thank you!) points to a check that the app provides BOTH an application callback URI and a client ID. The code above doesn't set ClientId - https://docs.microsoft.com/en-us/uwp/api/windows.security.authentication.web.core.webtokenrequest.clientid?view=winrt-22621#windows-security-authentication-web-core-webtokenrequest-clientid so maybe start there? There's a three-parameter form of the WebTokenRequest constructor that takes the provider, scope, and client ID together. See also this sample:
https://github.com/microsoft/Windows-universal-samples/blob/ad9a0c4def222aaf044e51f8ee0939911cb58471/Samples/WebAccountManagement/cs/SingleMicrosoftAccountScenario.xaml.cs#L179-L189

The sample you linked has a comment that says "ClientID is ignored by MSA". Even when I do set a client ID, I get this error:
The application requesting authentication tokens is either disabled or incorrectly configured.

onecoreuap\shell\cloudexperiencehost\onecore\app\msa\core\TokenProviderExecutor.cpp(1037)\MicrosoftAccount.TokenProvider.Core.dll!00007FFFE91E0FA0: (caller: 00007FFFE91BED19) Exception(4) tid(5d58) 80860003 The application requesting authentication tokens is either disabled or incorrectly configured.


New Textobj: injected block

Textobj requested
I have a lot of code blocks that are tracked by the injects.scm. For example SQL in R,Julia or Python. It would be nice the just select the injected code block.
Not sure is this is achievable at the moment because as far as I know there is not way to query the injected block using a tree-sitter query. But maybe I am wrong here.
Inner/Outer Difference
Not sure, maybe the direct parent?
Vimscript plugin already implementing the textobj
No response
Filetypes
all
Checklist

[X] The textobj would be useful to more users than just me.
[X] There is no lua plugin already implementing the textobj in an adequate manner.
[X] I read the docs and there is no such textobj already.

I am not sure I understand. What exactly is requested? This plugin already has a text object for code blocks, though it does not use treesitter for that.
Generally, treesitter-based objects are great, but not the goal for this plugin, since there is already treesitter-textobjects for that.
Or is are injected code blocks something different here? I am not very familiar with python or R, so simply might not be understanding this correctly

Hmm,ok - even though I use this plugin I was always thinking that it uses tree-sitter. So my issue is that I have injected code blocks like this:

So I was thinking that a textobject would be nice that selects injected code section (in this example the SQL statement). However, it should be agnostic of the filetype and the parent node. I have created these injections for myself by matching on the function name (query_impala). So this is obviously not useful for other people. But generically selecting injected code blocks is in my opinion.
I can create this feature request in treesitter-textobjects if that makes more sense.

yeah, this plugin only uses pattern matching with some extras. It's mostly for use cases where treesitter does not make sense (e.g. the near-end-of-line textobj) or where treesitter textobj aren't there yet. (see for example #20)
Your suggestions makes more sense for treesitter-textobjects, yes


Use Eclipse folder as process working directory

Type of request

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Refactoring
[ ] Documentation or documentation changes

Related Issue(s)

Lombok adds an agent path to the vmargs. By default it is possible to define a relative path to the Lombok plugin within the installation:
-javaagent:plugins/org.projectlombok.agent_1.18.22/lombok.jar
So this path is correctly working it is necessary that Eclipse is started from it's folder as process working directory. Otherwise Eclipse is not starting as the relativ path cannot be resolved.
Concept
Instead of using the default process working directory (by default user.dir). it is possible to provide the directory where a process is started from.
Technical information
Platform / target area

All platforms.
known side-effects
Eclipse process is started and executed from it's original directory, instead of the user directory.
How has this Tested?

Simply start Eclipse with a Lombok installation, containing the following vmarg: -javaagent:plugins/org.projectlombok.agent_1.18.22/lombok.jar
Check TortoiseGit is still working as expected by clicking on Log or Status

Checklist:

[x] All the commits are signed.
[x] My code follows the code style of this project.
[x] I agree with die CLA.
[x] I have read the CONTRIBUTING docs.
[x] I have added/updated necessary documentation (if appropriate).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


oncontextmenu

Oncontextmenu - what display context menu?
Example:
`some content no context menu
No context menu inALL BODY!`

+1
Per my testing of the current versions, latest version of Chrome, Firefox, Chrome Android and Firefox on Android support contextmenu event but iOS Safari sadly does not.

Available at https://caniuse.com/#feat=mdn-api_element_contextmenu_event


Define our approach to support our Open Source repos during week of aug 9 - aug 13

🏖 🌞 🥳 🍹
Acceptance Criteria

[x] see Austin's  internal post and pinned channel message  for guidance.
[x] draft our approach & review with team
[ ] share internally
[ ] pin a message in each of our hero channels.

I created a draft post for this and asked for feedback from @polfliet  ans @zstix

This is done.


return to the original state when task fails

When I run the pipeline and the task fails for any reason the triggers on the data factory that will be deployed keep stopped.
There is a way to return the data factory to original state?

No, there is no way to return to the previous state with the same set of objects.
What you can do is redeploy from the previous version of your (working) code.
I assume you built a Build pipeline which generates and versiones all your ADF artifacts before you start deploying them.
If you're not sure how to do that, check out my YouTube channel and this playlist:
Kamil Data Geek - Azure explained / ADFTools playlist


CmdLineEnter/CmdLineLeave/CmdwinEnter/CmdwinLeave for mode ? are triggered when interacting with mode :

Steps to reproduce
Start Vim with vim -Nu file-below:
augroup Foobar
  autocmd!
  autocmd CmdLineEnter ? echom 'CmdLineEnter ?' . localtime()
  autocmd CmdLineLeave ? echom 'CmdLineLeave ?' . localtime()
  autocmd CmdwinEnter ? echom 'CmdwinEnter ?' . localtime()
  autocmd CmdwinLeave ? echom 'CmdwinLeave ?' . localtime()
augroup END

and then play with q:/Enter and :Ctrl-F/Ctrl-C. You'll see that all autocmds above will be triggered, whereas they shouldn't, as they are for backward search command line mode, ?, not for ex command line mode, :.

Expected behaviour
The autocmds in the minimal vimrc above shouldn't be triggered.
Version of Vim
9.0.1182
Environment
Operating system: up-to-date ArchLinux
Terminal: urxvt
Value of $TERM: rxvt-unicode-256color
Shell: bash 5.1.16(1)-release
Logs and stack traces
No response

? matches any single character, and there is no ? cmdline mode

Sorry, there is a ? cmdline mode, it needs to be matched using \?

All of CmdlineChanged, CmdlineEnter, CmdlineLeave, CmdwinEnter, CmdwinLeave have this text in the doc:
<afile> is set to a single character,
indicating the type of command-line.
|cmdwin-char|

and cmdwin-char reads:
The character used for the pattern indicates the type of command-line:
	:	normal Ex command
	>	debug mode command |debug-mode|
	/	forward search string
	?	backward search string
	=	expression for "= |expr-register|
	@	string for |input()|
	-	text for |:insert| or |:append|

Where should I understand ? needs to be pretended by \? Furthermore, the autocmds above are no triggered by /, for instance.

Please check :h file-pattern, which contains this somewhere below

The pattern is interpreted like mostly used in file names:

matches any sequence of characters; Unusual: includes path
separators
?	matches any single character
?	matches a '?'
.	matches a '.'
~	matches a '~'
,	separates patterns
,	matches a ','
{ }	like ( ) in a |pattern|
,	inside { }: like | in a |pattern|
}	literal }
{	literal {
\{n,m}  like {n,m} in a |pattern|
\	special meaning like in a |pattern|
[ch]	matches 'c' or 'h'
[^ch]   match any character but 'c' and 'h'

Note that for all systems the '/' character is used for path separator (even
for MS-Windows).  This was done because the backslash is difficult to use in a
pattern and to make the autocommands portable across different systems.

That's why, the example at :h 'incsearch' contains these patterns:

If you don't want to turn 'hlsearch' on, but want to highlight all
matches while searching, you can turn on and off 'hlsearch' with
autocmd.  Example: >
augroup vimrc-incsearch-highlight
autocmd!
autocmd CmdlineEnter /,? :set hlsearch
autocmd CmdlineLeave /,? :set nohlsearch
augroup END

so if you correct the pattern, does the issue still persist?

Thanks for the explanation!

so if you correct the pattern, does the issue still persist?

No, the issue is fixes as you foresaw.
However, I have one last question on this topic: if ? was matching, in my original autocmds, a single character, then why were those autocmds triggered by both ? and :, but not by /?

sorry, not sure.


Build failure: nodejs_22 on x86_64-darwin

Steps To Reproduce
Steps to reproduce the behavior:

build nodejs_22 (currently 22.9.0)

Build log
Compilation succeeds, but the build fails on tests, ending with:
Failed tests:
out/Release/node /private/tmp/nix-build-nodejs-22.9.0.drv-0/node-v22.9.0/test/parallel/test-inspector-ip-detection.js
make: *** [Makefile:560: test-ci-js] Error 1

Full log in this gist.
Additional context
Notify maintainers
@aduh95
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
 - system: `"x86_64-darwin"`
 - host os: `Darwin 24.0.0, macOS 10.16`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Lix, like Nix) 2.91.1
System type: x86_64-darwin
Additional system types:
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /Users/dustinpomerleau/.config/nix/nix.conf:/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/cqvrm2h1501ambaw9l2509dq5c9ch0b1-lix-2.91.1/share`
 - nixpkgs: `/nix/store/69bhrhj8imx84l6zvdz0zmh3qc8ykwvd-source`

Add a :+1: reaction to issues you find important.

I can confirm on my Intel Mac. I also tried updating the nix package to nodejs 22.10.0, and that also fails some test in test-ci-js group, but not the same test that fails for 22.9.0.

Can you share what test is failing and its output please?

I am attaching the build log for Nodejs 22.10.0 on Nixpkgs commit ID e87b505f5121ae4cd00f3c70987b690b737fc2b3.
nodejs-22.10.0-build.log

not ok 3714 parallel/test-tick-processor-arguments
  ---
  duration_ms:<PHONE_NUMBER>0
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

Is this test reliably failing for you? It looks like a flake to me

I ran it again this morning, now with Nixpkgs commit 161ba5b87e6a0c048de6fdfc9bbda69f4a290969, and I am attaching the latest log: nodejs-22.10.0.log
The same test failed.
I am running it for the third time to see.

Third run. Several tests failed, including test-tick-processor-arguments. Latest log is attached.
nodejs-22.10.0-build.log

🤷♂ it would probably make sense to add that test to the list of skip tests for your architecture.

Arm64-darwin failed, too.

I have the same issue. Attaching my full build log: nodejs_build_output.txt
System details:

system: "x86_64-darwin"
host os: Darwin 24.1.0, macOS 10.16
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.24.10
channels(root): "nixpkgs"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs


Relax dependency version constraints.

It's good practice for a library not to depend on exact versions of any other libraries, so that downstream programs are able to update dependencies they have in common at their leisure.
This is described in the Cargo documentation at:
http://doc.crates.io/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
An example of the problem an exact version depedency creates can be seen at https://gist.github.com/jimmycuadra/ac26eca5f11960c357d6fba841640f2f.
This patch modifies Cargo.toml to allow semver-compatible updates to clap's dependencies.

Coverage remained the same at 91.647% when pulling a32389b5adf5baf7d827036903938228ca668b5c on jimmycuadra:relax-dependency-constraints into d29d7cc0e2b5e2cf52f2dd337bb42b605580a448 on kbknapp:master.

This was originally because all those crates are sub 1.0 to avoid breaks when the minor is bumped in said crates. I found out recently via the subreddit that cargo actually treats sub 1.0 crates slightly differently and doesn't bump the minor automatically.
Long story short, I'm good with this change now ;)
@homu r+

:pushpin: Commit a32389b has been approved by kbknapp

:hourglass: Testing commit a32389b with merge 63b8647...


pv_forecast is not aligning with JSON input for naive-mpc-optim command

EMHASS is returning a pv_forecast that doesn't align with the input data. Currently it's outputting 2,270.57 W and I can't see any reason why it's not outputting 839W ... all other output time intervals are aligned with the input
Also note that it seems to be jumping around at random over time and I can't reverse engineer what it's doing. But it does seem that the EMHASS optimisation is using this pv_forecast value and anticipating more charge than what my pv can deliver
I am using the Hybrid Inverter option but have no PV curtailment
Input for the naive-mpc-optim command
{ "pv_power_forecast":[839, 1115, 1576, 1892, 1981, 2149, 2366, 2524, 2566, 2481, 2310, 2118, 1910, 1636, 1316, 971, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],"load_cost_forecast":[0.13, 0.13, 0.16, 0.16, 0.15, 0.1, 0.1, 0.12, 0.12, 0.15, 0.14, 0.41, 0.42, 0.41, 0.42, 0.49, 0.52, 0.69, 0.69, 0.72, 0.81, 0.69, 0.69, 0.43, 0.4, 0.33, 0.24, 0.33, 0.24, 0.37, 0.34, 0.34, 0.28, 0.22, 0.21, 0.19],"prod_price_forecast":[0.03, 0.04, 0.06, 0.04, 0.03, -0.01, -0.01, 0.01, 0.01, 0.03, 0.02, 0.32, 0.33, 0.32, 0.33, 0.39, 0.42, 0.58, 0.58, 0.61, 0.68, 0.58, 0.58, 0.31, 0.28, 0.22, 0.13, 0.22, 0.14, 0.25, 0.23, 0.22, 0.17, 0.11, 0.11, 0.09],"prediction_horizon":36,"soc_init":0.53,"soc_final":0.15 }

I just changed my pv_forecast input algorithm to make the first value 0 and it's still giving back 2022.72 W

Now it's giving a massive error but the second it ticked over 1131, the error reduced to less than 100 and has now increased over the time period.
Data Input
{ "pv_power_forecast":[2418, 2456, 2371, 2290, 2214, 2030, 1752, 1441, 1135, 895, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],"load_cost_forecast":[0.04, 0.04, 0.1, 0.1, 0.1, 0.39, 0.4, 0.41, 0.42, 0.48, 0.47, 0.52, 0.69, 0.58, 0.54, 0.53, 0.58, 0.31, 0.23, 0.26, 0.23, 0.37, 0.21, 0.37, 0.27, 0.24, 0.21, 0.21, 0.19, 0.18, 0.18, 0.18, 0.18],"prod_price_forecast":[-0.07, -0.07, -0.01, -0.01, -0.01, 0.31, 0.32, 0.32, 0.33, 0.38, 0.38, 0.42, 0.58, 0.48, 0.44, 0.43, 0.48, 0.2, 0.13, 0.15, 0.13, 0.25, 0.11, 0.26, 0.16, 0.14, 0.11, 0.11, 0.09, 0.08, 0.08, 0.08, 0.08],"prediction_horizon":33,"soc_init":0.64,"soc_final":0.15 }

Please check your alpha and beta values as well as the method_ts_round:
https://emhass.readthedocs.io/en/latest/forecasts.html#now-current-values-in-forecasts
https://emhass.readthedocs.io/en/latest/config.html#retrieve-hass-data-configuration
I have:
method_ts_round set to first.
"alpha": 1,
"beta": 0,

Please check your alpha and beta values as well as the method_ts_round: https://emhass.readthedocs.io/en/latest/forecasts.html#now-current-values-in-forecasts https://emhass.readthedocs.io/en/latest/config.html#retrieve-hass-data-configuration
I have:
method_ts_round set to first. "alpha": 1, "beta": 0,

Thankyou ... I don't know where this thread can be moved to to help others ... but, if anyone else has this issue here is my rest command change that solved it:
post_mpc_optim_solcast:   url: http://homeassistant.local:5000/action/naive-mpc-optim   method: POST   content_type: 'application/json'   payload: >-     { "pv_power_forecast":{{states('sensor.solcast_24hrs_forecast')}},"load_cost_forecast":{{states('sensor.amber_gen_price_forecast')}},"prod_price_forecast":{{states('sensor.amber_prod_price_forecast')}},"prediction_horizon":{{states('sensor.solcast_24hrs_forecast')|from_json|count}},"soc_init":{{(states('sensor.med_battery_charge')|float(0)/100)|round(2)}},"soc_final":{{float(0.15)|round(2)}},"alpha":{{float(1)}},"beta":{{float(0)}} }
With this rest command, I'm now getting:

if alpha = 0 and beta = 1 then this is then I got the below output for this input


Note on change to UUIDs

Due to the change from sequence IDs to UUIDs, the local feature doesn't make sense anymore, and it should be removed.

Fixed by b80c152ecd68f24e9535b95cc3697df9a246de2e


Design

We should work out a couple of sketches.

Good Idea. You can upload any you have drawn in a folder. (Pull request?)

Just a quick mock up, similar to what you have done.

Here's my TI30XIIS:

What do you think of the changes?

We should come up with a standardized design for the s-b-s milestone.

All buttons that I add will now be just 'dummy buttons.' They will become active once #23 is ready.
Another idea is to render s-b-s in another box. The result/current formula can be in the current text box, and then steps can be shown elsewhere(once again, requires #23 ).

Whoa, this new green theme is awesome!
@chexbox what do you think?

Sorry for not seeing all these comments. They look great!

It looks like this has been inactive and off-topic. Make an "Announcement" issue for now on; Closing


Embedding vectors

Can I output  the embedding vectors (contex vectors) as a file ?

Same question, how can I get the embedding vectors?

Hi @LiuChuang0059, @baoxin1100,
Thank you for your attention. I have added the code for exporting embedding vectors. For the unsupervised node classification task, our code will save the embedding vectors to the directory specified by save-dir now.


🐛 [BUG]: Can no longer add classes to MiniMap in order to place other elements in the same Panel

Is there an existing issue for this?

[X] I have searched the existing issues and this is a new bug.

Current Behavior
Prior to<EMAIL_ADDRESS>I was able to write code similar to the following (using Bootstrap utilities to reset margin and position):
<Panel>
  <MiniMap class="position-static m-0" />
  <!-- other stuff -->
</Panel>

in order to put the minimap inside the same panel as other components. I used this for an effect like so:

In 1.3.0, classes are no longer applied to the Panel, but instead to the svg inside the Panel. Additionally, there is no way to add additional contents to the Panel such as via a slot or to use the MiniMap without a wrapping Panel.
Expected Behavior
I should be able to include a minimap alongside additional content in the same floating panel.
Steps To Reproduce
see code above
Relevant log output
No response
Anything else?
No response

Mh... You could just do Controls position="bottom-right" to have them on the same position.
Is it a hard requirement to put these in the same Panel? 🤔
This change could be reverted though but if a different solution is possible, I'd prefer that 😅

Well yeah, but wrapping it in a Panel isn't preventing Panels overlapping.
You put a third Panel around 2 components that already use a panel, so I'm not sure how "overlap" is the issue here?

I'mma be real with you, looking back at this I'm not even sure why I changed it in the first place.
This change will be reverted 🤷♂

Reverted in 1.29.0


NonuniformLineScan cannot fulfill test_periodic_flag

NonuniformLineScan always returns
https://github.com/ContactEngineering/SurfaceTopography/blob/34816d954c4886440ba10eadda4a4b8b721526f3/SurfaceTopography/NonuniformLineScan.py#L91-L95
on the is_periodic flag and a reader returning such an object on the topography method can never fulfill
https://github.com/ContactEngineering/SurfaceTopography/blob/34816d954c4886440ba10eadda4a4b8b721526f3/test/IO/test_io.py#L382-L392
Is that intended?

This looks strange. So the test always fails?

This is bogus. It should not be possible to just overide the periodic flag. For most readers.

The readers just pass through the periodic flag. I think this is not what we want. We could think about a pipeline function to modify the periodicity, but the reader itself should only allow specifying properties that are not in the file. Basically all readers for experimental instruments yield nonperiodic data.

Changing this would a behavioral (API) change of the whole system.
@mcrot - How does TopoBank deal with periodicity?

The question on whether the topography method should or should not allow overriding the periodic flag left aside, why should a NonuniformLineScan not allow periodicity per se? I can certainly imagine reading some periodic (likely computer-generated) profile on a non-uniform grid.

That is right, but we currently do not support this

This would require the NonuniformTopography to store additional information on the length of the scan. Right now it is identical to the distance between leftmost and rightmost data point. I'd be happy to add this, but not without a use-case.

Changing this would a behavioral (API) change of the whole system.
@mcrot - How does TopoBank deal with periodicity?

When uploading a topography, the user sees this checkbox:

So for only for 2D measurements and if no sizes are given in the original file, the user can decide to consider
a measurement as periodic. In other cases, the checkbox cannot be enabled.
Additionally, only detrend_mode "center" can be used if periodicity is enabled.
So the default is to store False for the is_periodic flag of a measurement, but in some cases it can be set to True by the user.
This flag is then used every time, when an instance of a SurfaceTopography topography is generated.
During analyses, SurfaceTopography makes use of it, not Topobank.

Would it be a problem if we restricted it?

For https://github.com/ContactEngineering/SurfaceTopography/pull/226 that means, I undo the modifications on NonUniformLineScan introduced in https://github.com/ContactEngineering/SurfaceTopography/pull/226/commits/6984072e8a9173f7020abb79ddfa6bbdfe330855 and simply remove test_periodic_flag  from the tests, alright?

I'll push my changes in a second

Issue handled by https://github.com/ContactEngineering/SurfaceTopography/commit/e5e5a70fc491794f4f177ec9655da3eb0e57cb16
If at some point a use case for periodicity in the NonuniformLineScan arises, begin with reintroducing the modifications undone by https://github.com/ContactEngineering/SurfaceTopography/commit/e6adaa300d2ea7b9580e4e115746d51947b1f066 and https://github.com/ContactEngineering/SurfaceTopography/commit/b6361e5db8091b2f3afa58628b3082fdd63e8c1e


Froze WSL with scp of large folder from DriveFS

I somehow froze WSL and was unable to kill the Lxssmanager service (except by restarting the PC)  attempting to do
scp -r /mnt/c/blah/ user@my.vps:~/blah/
The folder was about 18GB, and as soon as I entered the command, it froze with no data transfered.
Disclosure: I did this from within tmux. Not sure if it is related, will test in 8 hours when the folder is done uploading (using WinSCP, it is a bit important)

This was actually related to https://github.com/Microsoft/BashOnWindows/issues/1085 which provides reproduction steps.  Closing this one.


Create programming guide for ObjectN encoding

CONCEPTS:

OT (object type):
This is a vanilla enum of (generalized) singular MTG object types.
Examples: OTInstant, OTActivatedAbility, OTCreature

OTKN (object type kind N):
While the type OTKN is not literally in the code, this is a family of OTK0, OTK1, ... OTK12 type aliases for type level [OT] of the corresponding length. By convention the OTs in the lists are sorted alphabetically. This is to ensure unique representations. (Failure to do so is a bug, and if such a type is actually used by the engine, I think it wouldn't compile due to missing class instances or GADT constructors for the non-sorted type list.) OTK0 doesn't actually contain object type information in it. This is useful for doing type erasure.
Examples: OTK1 'OTSorcery, OTK1 'OTLand, OTK2 'OTArtifact 'OTCreature

OTN (object type N):
This is a GADT parameterized generically on OTKN and has corresponding constructors for them. This is the composite object type form of OT. Convenience type aliases are provided for each of the concrete OTN otk families: OT0, OT1 a, OT2 a b, etc. The corresponding OTNconstructors have the namesOT0, OT1, ..., OT12` as well. The constructors are each nullary and just store relevant class instance dictionary information for each of the encoded object types.
Examples: OT1 'OTEnchantment, OT2 'OTCreature 'OTPlayer

ObjectN (object N)
This is a GADT parameterized by OTN types and represents a hierarchical encoding of how to lift a (composite) object of fewer types into one of more types. For example, one might want to lift a "creature" into a "creature and/or planeswalker" representation ("and/or" is used because the encoding doesn't care about how one interprets its structure, but in practice this would mean either "and" xor "or" depending on the specific use case). Furthermore one might want to further lift that into a "permanent" representation (type OTPermanent = OT5 'OTArtifact 'OTCreature 'OTEnchantment 'OTLand 'OTPlaneswalker) and so on.
Lifting is done in one of two fashions through their corresponding constructors:
(1) Lifting an object of a singular a :: OT into ObjectN (OT3 a b c) or some permutation of [a, b, c] where a is not at the front of the OTKN. This pattern generalizes to other OTKN N-sizes.
(2) Lifting a composite object of ObjectN (OT3 a b c) to ObjectN (OT4 a b c d) or whatever the correct alphabetically sorted permutation of a b c d would be. This pattern generalizes to other OTKN N-sizes.

ToObjectN'
While the type class ToObjectN' is not literally in the code, there is a family of ``ToObject1', ToObject2', ...ToObject12'type classes that promoteObject atoObjectN a b c d` so-to-speak

ToObjectN
Similar to ToObjectN' but lifts "subset" ObjectN into superset ObjectN encodings. So for example lifting a ObjectN a b into an ObjectN a b c or ObjectN a b c d and so forth.


S3 gateway fails to connect to Filer on non-standard port when running together

Describe the bug
When Filer runs together with S3 and Filer has non-standard gRPC port, managed by third-party system (Nomad in my case), S3 cannot connect to Filer, assuming default gRPC port being 10000+port:
port "filer" {
        static       = 8280
        to           = 8280
        host_network = "internal"
}
// Static port for gRPC is set for illustrative purposes, usually it is allocated dynamically.
port "grpc" {
        static       = 18380
        to           = 18380
        host_network = "internal"
}

I1101 15:47:51.437282 s3.go:203 wait to connect to filer <IP_ADDRESS>:8280 grpc address <IP_ADDRESS>:18280
I1101 15:47:52.439709 s3.go:203 wait to connect to filer <IP_ADDRESS>:8280 grpc address <IP_ADDRESS>:18280
I1101 15:47:53.442039 s3.go:203 wait to connect to filer <IP_ADDRESS>:8280 grpc address <IP_ADDRESS>:18280

System Setup

List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".

# weed filer \
          -master=seaweedfs-master.service.consul:9333 \
          -ip=${NOMAD_IP_filer} \
          -port=${NOMAD_PORT_filer} \
          -port.grpc=${NOMAD_PORT_grpc} \
          -ip.bind=<IP_ADDRESS> \
          -dataCenter=dc1 \
          -rack=rack0 \
          -metricsPort=${NOMAD_PORT_metrics} \
          -s3=true \
          -s3.port=${NOMAD_PORT_s3} \
          -s3.port.grpc=${NOMAD_PORT_s3grpc} \
          -s3.config=${NOMAD_ALLOC_DIR}/s3.conf \
          -s3.dataCenter=dc1

OS version

# uname -a
Linux b94fda572829 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 Linux

output of weed version

# weed version
version 30GB 3.58 d1e83a3b4 linux amd64

if using filer, show the content of filer.toml

[leveldb2]
enabled = true
dir = "."					# directory to store level db files

Expected behavior
S3 gateway server connects to Filer's correct gRPC port.

Thanks for reporting and fixing the issue!!!


Setup RuboCop

To reduce efforts on code format and code review, let's introduce RuboCop.
This PR fixes some warning-level code offenses.
Other minor offenses will be addressed in subsequent PRs.

I believe this eases us to maintain code quality.
We may configure RuboCop rules if we find them not so useful.


[Feature Request] Enable the Sync (Image Slice Sync) function when the viewer starts

What feature or change would you like to see made?
Doctors want to enable the Sync (Image Slice Sync) function when the viewer starts. Now We have to click the 'Image Slice Sync' from the toolbar to be enabled. Please add this function to make doctor operations faster
Why should we prioritize this feature?
Operate quickly.
With MR, CT studies, this function is usually used.

I think we have it, no?
add this to your toolbar
createButton({
          id: 'ImageSliceSync',
          icon: 'link',
          label: 'Image Slice Sync',
          tooltip: 'Enable position synchronization on stack viewports',
          commands: {
            commandName: 'toggleSynchronizer',
            commandOptions: {
              type: 'imageSlice',
            },
          },
          listeners: {
            [EVENTS.VIEWPORT_NEW_IMAGE_SET]: {
              commandName: 'toggleImageSliceSync',
              commandOptions: { toggledState: true },
            },
          },
          evaluate: [
            'evaluate.cornerstone.synchronizer',
            {
              name: 'evaluate.viewport.supported',
              unsupportedViewportTypes: ['video', 'volume3d'],
            },
          ],
        }),

@sedghi Yes we already have it but it isn't enabled/activated automatically when going to the image viewer page.
Reproceduce:
Test on the study:
https://viewer-dev.ohif.org/viewer?StudyInstanceUIDs=<IP_ADDRESS>.4.1.145<IP_ADDRESS>.7310.5101.860473186348887719777907797922

Select an MR study with multi-series
Select the Basic Viewer button
Choose a 2x2 layout from the toolbar
Scroll on a series, other related series don't scroll either => Not work because the Sync is not enabled/activated yet
From the More dropdown, click 'Image Slice Sync' to enable/active
Scroll on a series, other related series scroll also => Works

I want step #4 work also.
Could you add it?
I tried to add ‘ImageSliceSync’ to ‘enabled’ after ‘ReferenceLines’ at
File: \modes\longitudinal\src\initToolGroups.js
but it doesn't work.
function initDefaultToolGroup(
   ...
  const tools = {
     ...
    enabled: [
      { toolName: toolNames.ImageOverlayViewer },
      { toolName: toolNames.ReferenceLines },
      { toolName: toolNames.ImageSliceSync  },
      {
        toolName: SRToolNames.SRSCOORD3DPoint,
      },


Add User Interface section under Overview of InternBuddy

Currently, there are no sections in the user guide dedicated to showing the users the components of InternBuddy's user interface. This could result in confusion when referring to the UI elements in the user guide.
Let's add a dedicated section with an annotated image of InternBuddy that concisely informs the user what are the components in the InternBuddy app.

Codecov Report

Merging #175 (b37edf5) into master (35a3f73) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #175   +/-   ##
=========================================
  Coverage     72.60%   72.60%
  Complexity      806      806
=========================================
  Files           121      121
  Lines          2610     2610
  Branches        371      371
=========================================
  Hits           1895     1895
  Misses          581      581
  Partials        134      134

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35a3f73...b37edf5. Read the comment docs.


Latest release breaks Cypress Custom Commands

Hello,
When I use<EMAIL_ADDRESS>every call to my Custom Commands defined in cypress/support/commands.js (imported in cypress/support/index.js as expected) are not recognised and I get error message "There is not Cypress Command issued for this test".
If I switch back to<EMAIL_ADDRESS>everything works as expected.
FYI I'm using<EMAIL_ADDRESS>in both cases.

This is very weird,as I don't see what could possibly cause this between the 0.6.0 -> 1.0.0 change.. would you be able to make a reproduction? That would help us a lot in fixing this problem
Thanks for reporting!

I'm running<EMAIL_ADDRESS>and cypress 3.0.1 with custom commands and everything works fine. Perhaps updating cypress may help in this?

I'm going to close this for now to clear things up a bit, @vvanmol if you still have this problem after updating to the newest cypress please reopen, reproduction would be great (and shouldn't be hard).

Hello.
Sorry for the late reply...  I couldn't get back on this yet.
I'll try to reproduce it and if I can I'll re-open this ticket and provide more information.
Thanks !


exporter/stats/stackdriver: name metrics consistently

OpenCensus Java names the metric descriptors in the form of:
OpenCensus/grpc.io/rountrip_latency/cumulative

We use lowercase "opencensus" as a prefix. Fix the inconsistency problem.

Thanks for creating this issue @rakyll, in deed. For posterity here is what @rakyll is referring to https://github.com/census-instrumentation/opencensus-java/blob/2b464864e3dd3f80e8e4c9dc72fccc225444a939/exporters/stats/stackdriver/src/main/java/io/opencensus/exporter/stats/stackdriver/StackdriverExportUtils.java#L89
vs what the Go client does
https://github.com/census-instrumentation/opencensus-go/blob/2237b123f1e72228dd0bbe11fb038462d15f6fbb/exporter/stats/stackdriver/stackdriver.go#L329
PR coming up shortly.


Receive Request URI now starts with vertcoin: not litecoin:

Would changing ./lib/util.py:463 be better way of fixing this issue?

Oops sorry, that's my bad. I don't know how I missed that. I fixed it now.


Link points to 365 admin rather than teams admin

The link admin.microsoft.com should be admin.teams.microsoft.com

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 76dc173c-0c98-1b94-a2f7-0a34781c985a
Version Independent ID: a1c44ccb-67f3-f98e-6569-f938e88ddc63
Content: Set the phone numbers included on invites - Microsoft Teams
Content Source: Teams/set-the-phone-numbers-included-on-invites-in-teams.md
Service: msteams
GitHub Login: @tonysmit
Microsoft Alias: tonysmit

@trajano  Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

@trajano - I've removed the URL that was broken and republished the topic. Changes should be live in a few hours.


Change local_path to save_path in get_bsrn

[x] Pull request is nearly complete and ready for detailed review.
[x] Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

This pull request changes the input argument local_path to save_path in get_bsrn as this is a more descriptive name.
I am making this pull request now as this needs to be merged before 0.9 as otherwise it will be a breaking change.

@kanderso-nrel Wanna add labels? :)

@AdamRJensen as long as we're thinking about clarity... what do you think about editing the parameter description to make it explicit that save_path must be a directory (not a file)?

I don't recall any other iotools functions that will write data to disk. Am I forgetting something? While I'm not opposed to throwing it into the wild and seeing what the response is, I'm leery of requests to add this feature to every fetch function and there is something to be said for API consistency. Why not let users get the raw files themselves? We're still giving them a useful parser.

It's true that you haven't seen it in the other functions, but I think that is a mistake!
Particularly for research projects (which need to be reproducible), I find it essential to have the raw files available locally. The main reason for this is that many data suppliers are not always consistent (e.g., BSRN updates files when mistakes are found and CAMS computes irradiance on the fly using the most recent inputs). Sure users could save the returned dataframe, but there's no straight method for packing the associated metadata.
The alternative is for users to get the files themselves, but then all the super nice retrieval functions are obsolete. Personally, this would reduce the value of the iotools by ~40% and add all the hassle the retrieval functions are trying to remove.
If it's because you're worried about requests to add it, then I'd be happy to implement this feature for all the retrieval functions or at least the ones it makes sense for.
I'm very interested in hearing other opinions on this? Do pvlib users not save data, and if you do, how do you do it?

I'd be ok with functions like download_bsrn that do just that one thing: download a file. I dislike the idea of saving files as a side effect of fetching the data and putting it into python data structures.
What other libraries can we look to for patterns? I found only one example of siphon supporting download to a file:
https://github.com/Unidata/siphon/blob/feb7f60260d157c1c282469f755c3998ef6a6e0f/siphon/catalog.py#L598-L611

but then all the super nice retrieval functions are obsolete

I typically value the parsers more than the fetchers, so maybe we're coming in with different perspectives. The BSRN parser that you wrote is a great example of this - the file is horrible to parse but not difficult to get.
In the interest of not holding up a release, I see two options:

Merge this change (it's certainly an improvement). Comes with the risk of introducing a deprecation in the next release cycle.
Revise this PR to remove this feature from this function. Eliminates risk of introducing a feature that may deprecated, but doesn't allow for @AdamRJensen (or others) to use this feature.

Then we take up discussion elsewhere for the next release cycle.

I like the pvlib pattern of functions only doing one thing,  so my preference is for a separate download_bsrn() I agree some files like ECMWF files are so cumbersome to retrieve that a separate step to save the files for repeatability is very useful,  but for other files it's an easy step for users to save by themselves

I don't recall any other iotools functions that will write data to disk.

They're not iotools functions yet, but #1264 and #1274 (WIP ERA5 and MERRA2 PRs) also have this local_path option, so BSRN would have some company if we decide to go forward with this.
I see the point about functions doing one thing.  But I'm also not enthusiastic about adding download to our already somewhat cluttered get/read/parse API.

Comes with the risk of introducing a deprecation in the next release cycle.

If you think that chance is nontrivial, and the way forward isn't clear, I'd lean towards removing the feature for now until the way forward becomes clear.  Deprecations are a pain for us and for users.

I know that other solar related libraries save local data, e.g., SolarData and irradpy, so it seems others are also considering this useful.
While it is easy to manually download a single file from one of the services, this is not the case in many use cases e.g., simulating PV systems using irradiance data from multiple sources or getting all BSRN files for the past 5 years for Europe. Besides ECMWF files as @mikofski mentioned, ERA5 and MERRA2 files can also not be downloaded from a user interface but requires API usage.
Taking get_bsrn as an example, the option of saving local data adds two lines of documentation and three lines of code. As previously mentioned, having a download function as an alternative would add a bunch of functions to the already cluttered iotools, and instead of the 5 lines currently used to support the feature, it would require hundreds of lines of duplicated code or defining a private fetch function that both get_ and download_ could use - sounds like a hassle to me.
A thing that I don't like about the option of just saving the data and metadata objects is that our parser functions carry out an 'irreversible' act i.e., you cannot recreate the data files (also metadata that isn't parsed is lost). Having the raw data files locally makes it reversible in a way, if that makes sense.

If you think that chance is nontrivial, and the way forward isn't clear, I'd lean towards removing the feature for now until the way forward becomes clear. Deprecations are a pain for us and for users.

I do think the chance is nontrivial. I'm less concerned about deprecation because it feels like we might need a larger reorganization of the iotools package to support this feature across more modules. And it feels like we're getting to the point where a separate repository/package may be preferable. So I'm still ok with keeping the save_path as an experiment or removing the feature. I don't think anyone's mind is going to be changed with further discussion right now.

having a download function as an alternative would add a bunch of functions to the already cluttered iotools, and instead of the 5 lines currently used to support the feature, it would require hundreds of lines of duplicated code or defining a private fetch function that both get_ and download_ could use - sounds like a hassle to me.

I don't think this is necessarily so. For example, imagine how should a user reuse a file that they had downloaded? Would they rerun the retrieval function, downloading another copy? Can't we move the parsing code to a separate function? This isolates each step into its own function to be grouped as needed. EG:

download_bsrn(lat, lon, ...) - only retrieves the file and saves it to file, perhaps optionally, and returns the file buffer? Note: it doesn't parse the contents.
read_bsrn(file) - read a saved file and call the parsing function to output a dataframe.
parse_bsrn(fbuf) - parse the contents of the file buffer to a data frame
get_bsrn(lat, lon, ...) - calls download and parsing functions to output a dataframe. Maybe this has optional localpath arg as suggested?

@mikofski just to be clear, are you also thinking of a _get_raw_bsrn(lat, lon, ...) that would be used by both download_bsrn and get_bsrn?
def _get_raw_bsrn(lat, lon, ...):
    # get the data
    return data

def download_bsrn(filepath_or_buffer, lat, lon, ...):
    data = _get_raw_bsrn(lat, lon, ...)
    with open(filepath_or_buffer, 'w') as f:
        f.write(data)

def get_bsrn(lat, lon, ...):
    data = _get_raw_bsrn(lat, lon, ...)
    parsed_data, metadata = parse_bsrn(data)  # might need to be buffered
    return parsed_data, metadata

def read_bsrn(filepath):
    with open(filepath) as f:
        parsed_data, metadata = parse_bsrn(fbuf)
        return parsed_data, metadata

def parse_bsrn(fbuf):
    # do the parsing
    return parsed_data, metadata

That could work.
Here are some attempts to simplify the iotools API by putting slightly more responsibility on the user...
Alternative 1:
def get_raw_bsrn(lat, lon, ...):
    # get the data
    # plain text easier for most users when API returns plain text
    # but bad fit for consistency with APIs that return binary data
    return buffered_data

def read_bsrn(filepath_or_buffer):
    # https://docs.python.org/3/library/contextlib.html#contextlib.nullcontext
    if isinstance(filepath_or_buffer, (str, Path)):
        cm = open(filepath_or_buffer)
    else:
        cm = nullcontext
    with cm as buffered_data:
        parsed_data, metadata = parse_bsrn(buffered_data)
        return parsed_data, metadata

def parse_bsrn(buffered_data):
    # do the parsing
    return parsed_data, metadata

# my code
buffered_data = get_raw_bsrn(lat, lon, ...)
parsed_data, metadata = parse_bsrn(buffered_data)

# adam's code
buffered_data = get_raw_bsrn(lat, lon, ...)
with open('myfile', 'w') as f:
    # or copyfileobj https://stackoverflow.com/a/3253819/2802993
    f.write(buffered_data.read())
parsed_data, metadata = read_bsrn(myfile)

Alternative 2
def get_raw_bsrn(lat, lon, ...):
    # get the data
    # plain text easier for most users when API returns csv
    # but bad fit for consistency with APIs that return binary data
    return data.read()

def read_bsrn(filepath_or_buffer):
    # https://docs.python.org/3/library/contextlib.html#contextlib.nullcontext
    if isinstance(filepath_or_buffer, (str, Path)):
        cm = open(filepath_or_buffer)
    else:
        cm = nullcontext
    with cm as buffered_data:
        parsed_data, metadata = parse_bsrn(buffered_data)
        return parsed_data, metadata

def parse_bsrn(data):
    buffered_data = StringIO(data)
    # do the parsing
    return parsed_data, metadata

# my code
data = get_raw_bsrn(lat, lon, ...)
parsed_data, metadata = parse_bsrn(data)

# adam's code
data = get_raw_bsrn(lat, lon, ...)
with open('myfile', 'w') as f:
    f.write(data)
parsed_data, metadata = read_bsrn(myfile)

Reiterating the comment in the get_raw_bsrn function: Alternative 2 might look better until you consider consistency implications for PRs like #1264 #1274.
To support direct calls to APIs in read_bsrn and friends, we could consider copying or directly using some of pandas.io.common. It's not documented in the reference guide so it's unclear if it's considered public and thus safe for us to rely on. Obviously pandas has put a ton of work into making read_csv and friends as seamless as possible. Is it reasonable to expect the same for pvlib.iotools? I'd say no.

Just stumbled upon SunPy's Fido module which bears a strong resemblance to pvlib's iotools. The Fido functions have a keyword path that allows for files to be written to the disk, e.g.:
# Once you have located your files via a Fido.search you can download them via Fido.fetch:
downloaded_files = Fido.fetch(results, path='/ThisIs/MyPath/to/Data/{file}')

Some explanation from SunPy:

This downloads the query results into the directory /ThisIs/MyPath/to/Data, naming each downloaded file with the filename {file} obtained from the client. You can also use other properties of the returned query to define the path where the data is saved.

It's not completely parallel, but I figured I'd mention it here in regards to the previous debate:

I'd be ok with functions like download_bsrn that do just that one thing: download a file. I dislike the idea of saving files as a side effect of fetching the data and putting it into python data structures.
What other libraries can we look to for patterns? I found only one example of siphon supporting download to a file:
https://github.com/Unidata/siphon/blob/feb7f60260d157c1c282469f755c3998ef6a6e0f/siphon/catalog.py#L598-L611


SONAR-23150 Group all Openshift-specific resources under the same parameter

to help testing i would suggest updating the openshift-values.yaml to include the route settings.

This makes a lot of sense, thanks for the suggestion!


robotkernel status

Hi there!
cc. @datakurre @bollwyvl
We've been working for the past months on xeus-robot, which is more or less a robotkernel that works with the new JupyterLab 3 debugger (which is not possible with the kernel wrapper approach as of today). As part of this work, we've put together the robotframework-interpreter python library which implements all the robotkernel features, and make them usable as a library for xeus-robot. This work would not have been possible without the effort put in robotkernel!
https://user-images.githubusercontent.com/21197331/104568942-facc5680-5650-11eb-97cd-34695417476e.mp4
I am opening this issue to start discussing the future of those two projects. Using robotframework-interpreter in robotkernel would drastically reduce the size of the robotkernel codebase while keeping all the features.

@martinRenou Thanks for the heads up! Awesome news! I'm looking forward for you to clean up our legacy at robotframework-interpreter, and then see what's the place left for robotkernel in the future. I am also happy to hear that you were able to re-use parts to get started with debuggable robot!
Unfortunately, I am not currently able to do much development on robotkernel before the summer personally. Then I'll take a fresh look on the ecosystem. The robot ecosystem has advanced a lot, and new approaches for both browser and desktop testing and automation with robotframework...

@martinRenou Thanks for the heads up! Awesome news! I'm looking forward for you to clean up our legacy at robotframework-interpreter, and then see what's the place left for robotkernel in the future. I am also happy to hear that you were able to re-use parts to get started with debuggable robot!
Unfortunately, I am not currently able to do much development on robotkernel before the summer personally. Then I'll take a fresh look on the ecosystem. The robot ecosystem has advanced a lot, and new approaches for both browser and desktop testing and automation with robotframework...

I have demoed the xrobot kernel with the debugger in lab 3, really works
pretty slick!
I have noticed that the suite model doesn't seem as aware of the notebook
execution model (e.g by tracking cell id), which both this kernel and
irobotframework (which I also don't get to work on much) share. If I did
get to work on that, it would certainly be in other feature areas, and
would love to be able to share xeus code.
I don't really think there is a "2021 jupyter/robot" roadmap... Maybe there
should be!

I have demoed the xrobot kernel with the debugger in lab 3, really works
pretty slick!
I have noticed that the suite model doesn't seem as aware of the notebook
execution model (e.g by tracking cell id), which both this kernel and
irobotframework (which I also don't get to work on much) share. If I did
get to work on that, it would certainly be in other feature areas, and
would love to be able to share xeus code.
I don't really think there is a "2021 jupyter/robot" roadmap... Maybe there
should be!

Thanks for your responses :)

I have noticed that the suite model doesn't seem as aware of the notebook
execution model (e.g by tracking cell id), which both this kernel and
irobotframework (which I also don't get to work on much) share

What do you mean by "not aware of the notebook execution model"?
Concerning the execution model, xeus-robot is slightly different from robotkernel, but this is very subtle. Basically, we do not recompile the entire AST of the Notebook when we execute a new cell, we just compile the AST of the code in the cell, and manually append the new keywords/imports/test cases in the test suite. This is pretty ugly but it works nicely. And I suppose it could be improved by making some contributions in robotframework directly.

Thanks for your responses :)

I have noticed that the suite model doesn't seem as aware of the notebook
execution model (e.g by tracking cell id), which both this kernel and
irobotframework (which I also don't get to work on much) share

What do you mean by "not aware of the notebook execution model"?
Concerning the execution model, xeus-robot is slightly different from robotkernel, but this is very subtle. Basically, we do not recompile the entire AST of the Notebook when we execute a new cell, we just compile the AST of the code in the cell, and manually append the new keywords/imports/test cases in the test suite. This is pretty ugly but it works nicely. And I suppose it could be improved by making some contributions in robotframework directly.

 I think i remember what surprised me was xrobot's behavior in C2:

C2: apply S0, S1, run T0, run T1

I'm not sure there's a right answer, as the robot syntax itself seldom uses a "stream"-based model, and only acts against "full" documents.

 I think i remember what surprised me was xrobot's behavior in C2:

C2: apply S0, S1, run T0, run T1

I'm not sure there's a right answer, as the robot syntax itself seldom uses a "stream"-based model, and only acts against "full" documents.

Any comment is very welcome!

... I think i remember what surprised me was xrobot's behavior in C2:
C2: apply S0, S1, run T0, run T1

We'd like to stay close to robotkernel's behavior, and we actually have this behavior with tests so we should have the same with tasks. I will look into it. Thanks!

and only acts against "full" documents.

Yep. That was the reason for some ugly workarounds to make the debugger and file mapping work.

Any comment is very welcome!

... I think i remember what surprised me was xrobot's behavior in C2:
C2: apply S0, S1, run T0, run T1

We'd like to stay close to robotkernel's behavior, and we actually have this behavior with tests so we should have the same with tasks. I will look into it. Thanks!

and only acts against "full" documents.

Yep. That was the reason for some ugly workarounds to make the debugger and file mapping work.


[JBPM-5019] DisposeCommand is not part of the public API

Merged by https://github.com/droolsjbpm/droolsjbpm-knowledge/commit/74f3104bd


[SPARK-17246][SQL] Add BigDecimal literal

What changes were proposed in this pull request?
This PR adds parser support for BigDecimal literals. If you append the suffix BD to a valid number then this will be interpreted as a BigDecimal, for example 12.0E10BD will interpreted into a BigDecimal with scale -9 and precision 3. This is useful in situations where you need exact values.
How was this patch tested?
Added tests to ExpressionParserSuite, ExpressionSQLBuilderSuite and SQLQueryTestSuite.

cc @JoshRosen

Test build #64445 has started for PR 14819 at commit fda100f.

Test build #64445 has finished for PR 14819 at commit fda100f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64445/
Test PASSed.

Does other database do this?

@rxin Hive supports this.

Test build #64460 has started for PR 14819 at commit e889a25.

Test build #64460 has finished for PR 14819 at commit e889a25.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64460/
Test PASSed.

Thanks - merging in master/2.0. This is an omission from the 2.0 parser change.