Runtime errors on PPC (big endian)

I adapt the OpenWrt firmware for our railway access points. The access point hardware is based on PowerPC CPUs of the type QorIQ T1023(32Bit) and T2081(64Bit). For the next OpenWrt merge to version v22.03.05 the ucode package for the firewall (fw4) had to work.
In the first attempts on x86 and an ARM platform, the 167 tests of the ucode Test-Suite ran error-free.
But the ucode output generated on the PPC target was faulty, causing the FW4 not to start correctly.
Unfortunately, I couldn't run the test suite on the PPC platform at first, because it only provides busybox-shell and busybox-utils and not the full versions of bash / diff / truncate that are required.
That's why I first adapted the test suite script 'run_tests.sh'. My 'OpenWrt' version 'run_tests_openwrt.sh' now uses readlink-coreutils, truncate-coreutils and diff-gnu and can run in the busybox shell. However, many of the 167 tests were incorrect.
I was able to localize the error on the function 'ucv_compare' from the source file types.c.
Here the (int) return value of the strcmp () function is converted into a character (int8_t). On an x86/arm this always seems to work fine. However, on a PPC platform (big endian), the conversion leads to unpredictable results.
I patched the spot like this:
diff --git a/types.c b/types.c
index 5758f74..eebf1b1 100644
--- a/types.c
+++ b/types.c
@@ -1999,6 +1999,7 @@ ucv_compare(int how, uc_value_t *v1, uc_value_t *v2, int *deltap)
 	int64_t n1, n2;
 	double d1, d2;
 	int8_t delta;
+	int ret;

 	/* at least one operand is null and we compare for equality or inequality ... */
 	if ((!v1 || !v2) && (how == I_EQ || how == I_NE)) {
@@ -2007,7 +2008,11 @@ ucv_compare(int how, uc_value_t *v1, uc_value_t *v2, int *deltap)

 	/* ... otherwise if both operands are strings, compare bytewise ... */
 	else if (t1 == UC_STRING && t2 == UC_STRING) {
-		delta = strcmp(ucv_string_get(v1), ucv_string_get(v2));
+		/* Do not cast return value of stncmp as int8_t on PPC (big-endian) ! */
+		ret = strcmp(ucv_string_get(v1), ucv_string_get(v2));
+		if (ret == 0) delta = 0;
+		if (ret > 0) delta = 1;
+		if (ret < 0) delta = -1;
 	}

 	/* handle non-string cases... */

With this patch, at least 157 out of 167 tests in the test suite were error-free.
I then found a similar error in 'lib/fs.c'. Here the (int) return value of isatty() is directly converted into a 'ucv_boolean_new(isatty(fd))'. This also does not work correctly on the PPC. The following patch:
diff --git a/lib/fs.c b/lib/fs.c
index 831137d..461810d 100644
--- a/lib/fs.c
+++ b/lib/fs.c
@@ -356,7 +356,7 @@ static uc_value_t *
 uc_fs_isatty(uc_vm_t *vm, size_t nargs)
 {
 	FILE **fp = uc_fn_this("fs.file");
-	int fd;
+	int fd, ret;

 	if (!fp || !*fp)
 		err_return(EBADF);
@@ -366,7 +366,12 @@ uc_fs_isatty(uc_vm_t *vm, size_t nargs)
 	if (fd == -1)
 		err_return(errno);

-	return ucv_boolean_new(isatty(fd));
+	/* Do not convert isatty return value into bool on PPC (big-endian) */
+	ret = isatty(fd);
+	if (ret == 1)
+		return ucv_boolean_new(true);
+	else
+		return ucv_boolean_new(false);
 }

 static uc_value_t *
--

With this patch, all 167 tests in the Test-Suite were error-free.
I added another patch in types.c that didn't trigger a Test-Suite error. The difference of an INT pointer subtraction can definitely be larger than: int8_t delta;
diff --git a/types.c b/types.c
index 4391880..bfda5d8 100644
--- a/types.c
+++ b/types.c
@@ -2000,6 +2000,7 @@ ucv_compare(int how, uc_value_t *v1, uc_value_t *v2, int *deltap)
 	double d1, d2;
 	int8_t delta;
 	int ret;
+	intptr_t diff;

 	/* at least one operand is null and we compare for equality or inequality ... */
 	if ((!v1 || !v2) && (how == I_EQ || how == I_NE)) {
@@ -2021,7 +2022,11 @@ ucv_compare(int how, uc_value_t *v1, uc_value_t *v2, int *deltap)
 		/* ... both operands are of the same, non-scalar type... */
 		if (t1 == t2 && !ucv_is_scalar(v1)) {
 			/* ... compare memory addrs */
-			delta = (intptr_t)v1 - (intptr_t)v2;
+			/* Do not cast (long int) difference to int8 type. */
+			diff = (intptr_t)v1 - (intptr_t)v2;
+			if (diff == 0) delta = 0;
+			if (diff > 0) delta = 1;
+			if (diff < 0) delta = -1;
 		}

 		/* ... operands are of different type or at least one is scalar... */
--

The patches could also be interesting for other architectures, but at least for OpenWrt ./target/linux/mpc85xx'.
If desired I can provide my test script: test/custom/run_tests_openwrt.sh.
Here is the version for which we use the patches.
PKG_SOURCE_DATE:=2023-06-06
PKG_SOURCE_VERSION:=c7d84aae09691a99ae3db427c0b2463732ef84f4
PKG_MIRROR_HASH:=97339d563e988d57ed02f5fd27722c908d616c2cec8b1f0479f42bc53dfd0aa1
PKG_MAINTAINER:=Jo-Philipp Wich<EMAIL_ADDRESS>

Thanks for the quick reaction and for adjusting the patches. I was able to compile and run it out on PPC without any problems. (pulled Merge: 4bee0ef 093684d)
In the meantime, the custom test cases suite have probably also changed, which means that the following test cases are now faulty.
25_ltrim ................................ OK
26_rtrim ................................ OK
27_sprintf .............................. OK
28_printf ............................... OK
29_require .............................. !
Testcase #4: Expected stderr did not match:
--- Expected stderr
+++ Resulting stderr
@@ -1,7 +1,7 @@
 Runtime error: Unable to compile source file './files/require/test/broken.uc':

   | Syntax error: Expecting label
-  | In line 3, byte 1:
+  | In line 2, byte 10:
   |
   |  `return {`
   |           ^-- Near here
---
29_require .............................. FAILED (1/4)
30_iptoarr .............................. OK
...OK
34_json ................................. OK
35_include .............................. !
Testcase #6: Expected stderr did not match:
--- Expected stderr
+++ Resulting stderr
@@ -1,7 +1,7 @@
 Runtime error: Unable to compile source file './files/broken.uc':

   | Syntax error: Expecting label
-  | In line 4, byte 1:
+  | In line 3, byte 11:
   |
   |  `    return {`
   |   Near here --^
---
35_include .............................. FAILED (1/7)
36_render ............................... !
Testcase #6: Expected stderr did not match:
--- Expected stderr
+++ Resulting stderr
@@ -1,7 +1,7 @@
 Runtime error: Unable to compile source file './files/broken.uc':

   | Syntax error: Expecting label
-  | In line 4, byte 1:
+  | In line 3, byte 11:
   |
   |  `    return {`
   |   Near here --^
---
36_render ............................... FAILED (1/6)
...OK
62_loadfile ............................. !
Testcase #7: Expected stderr did not match:
--- Expected stderr
+++ Resulting stderr
@@ -1,7 +1,7 @@
 Runtime error: Unable to compile source file './files/test6.uc':

   | Syntax error: Expecting expression
-  | In line 2, byte 1:
+  | In line 1, byte 5:
   |
   |  `1 +`
   |      ^-- Near here
---
62_loadfile ............................. FAILED (1/7)
...OK
13_split_by_string_leading_trailing ..... OK
14_incomplete_expression_at_eof ......... !
Testcase #1: Expected stderr did not match:
--- Expected stderr
+++ Resulting stderr
@@ -1,5 +1,5 @@
 Syntax error: Expecting expression
-In line 2, byte 1:
+In line 1, byte 7:

  `{% 1+`
        ^-- Near here
---
14_incomplete_expression_at_eof ......... FAILED (1/1)
...OK
24_compiler_local_for_loop_declaration .. OK
25_lexer_shifted_offsets ................ !
Testcase #1: Expected stderr did not match:
--- Expected stderr
+++ Resulting stderr
@@ -1,5 +1,5 @@
 Error
-In line 3, byte 13:
+In line 3, byte 12:

  `    die("Error");`
   Near here -----^
---
25_lexer_shifted_offsets ................ FAILED (1/1)
...
Ran 167 tests, 161 okay, 6 failures

However, a run with the old custom test cases from (PKG_SOURCE_VERSION:=c7d84aae09691a99ae3db427c0b2463732ef84f4) is error-free. I therefore assume that the PPC specific patches work well (also the (isatty(fd) == 1).
Maybe the custom test suite should possibly be adjusted.

which means that the following test cases are now faulty.

This is happening on PPC?

Maybe the custom test suite should possibly be adjusted.

FYI that custom test suite passes on x86/64 https://github.com/jow-/ucode/actions/runs/5656546401/job/15324077119#step:3:1805

Yes, that's true on my x86/64, that was my first test, no errors occur.
The errors are only apply on a PPC T1023 with OpenWrt toolchain-powerpc_e500mc_gcc-8.4.0_glibc.
I have yet to compare the differences.

In my tests yesterday I accidentally had an old version of libucode.so. As a result, the function 'update_line()' from lexer.c was not up to date, which led to the line counting error. With the newly compiled version, 167 tests are now also okay on the PPC. Thank you for the effort.


Add "Can't-Add-To-List Message" Functionality on Dragover

I have a few different lists that, depending on some other settings, will or will not accept certain dnd-types. I would like to display some sort of notification to the user why they can't add certain items to certain lists as they drag the item over the un-accepting list. I'm using Bootstrap's "popover" functionality to display notifications.
Does anyone have any idea how to accomplish this?

Instead of using dnd-allowed-types, you have to write your own dnd-dragover callback that shows the message and returns false if the type is not allowed.


Png flags do not have the same height

By example Autralia and France do not have the same height

Instead of using png's isn't it possible to use the native flag emoji's?

hey @michael-ottink
it's not, at the beginning this package was done with emojis, but there was some problem with iOS

closing this one
Also, I just discovered the Australian flag it's just wider than a usual flag! Good to know


git-credential-gopass: init at 1.12.0

Motivation for this change
git-credential-gopass was split from gopass and now needs it's own little package.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Thanks a lot for this! :sparkles: I legitimately thought it was just gone.

Thanks a lot for this! sparkles I legitimately thought it was just gone.

Me too for a very sad, brief moment :sweat_smile:
I hope checks out now.

Thanks a lot for this! sparkles I legitimately thought it was just gone.

Me too for a very sad, brief moment sweat_smile
I hope checks out now.

Currently building, but it all checks out. Btw, just a quick tip for contributing to nixpkgs (if u do in the future) is we prefer PRs to be on unique branches other than master.

Ah sorry forgot that when copying from gopass/default.nix...

Thanks for contributing :sparkles:

Thanks for the fast feedback! :tada:

The commit history should have been cleaned up before this was merged.

The commit history should have been cleaned up before this was merged.

It was squash merged.

It was squash merged.

So the adding the maintainer is not an extra commit? Also The maintainer is not sorted alphabetically. Going to fix that in a fix collection PR.


Implementation of the dynamic MPF feature

This PR implements the dynamic MPF feature as proposed and demonstrated in 1 and 2.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

550 of 550 (100.0%)  changed or added relevant lines in 24 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>3:
0.0%

Covered Lines:
577

Relevant Lines:
577

💛  - Coveralls

Some tests appear to be more flaky than expected. Will revisit this later this week in parallel to the documentation reviews of this PR


Automatically support limit/next searches

Current api implementation ignores limit/next return values. Large result sets should be automatically handled by the library.

Actually, this may be more of a question: I have not found any way to even retrieve paged results without reimplementing the internal logic. Even calling e.g. getUserPlaylist with options for offset/limit it doesn't work as spotify wants to append a /tracks to the uri.

Hi,
We aim to keep this library as simple as possible and leaving these sort of things up to the user.
I'm not sure if getUserPlaylist was just an example or not, but there's also getUserPlaylistTracks which handles limit/offset. But if you happen to find a method that should support limit/offset but doesn't have it documented, PRs are always appreciated!

I wasn't even aware of getUserPlaylistTracks, thank you!


Test Update tutorial-manage-vm.md

@tysonn : Thanks for your contribution! The author, @cynthn, has been notified to review your proposed change.


Aditions to AVLTree

Checklist

[x] I've looked at the contribution guidelines.
[x] This pull request is complete and ready for review.

Description

I've added methods that return the key from the node in order, preorder and postorder, as well as a method that executes a closure in the same traversal order previously mentioned.

Could you change the indents back to 2 spaces?

sure, xcode did it on itself, sorry

👍


Building on freebsd with clang++

Hi, found strange issue with mhash on freebsd.
First of all I know that there is a node-gyp issue, described here https://github.com/Sembiance/mhash/issues/11 and I have created symlink to clang++:
ls -la /usr/bin/g++
lrwxr-xr-x  1 root  wheel  16 19 feb 09:56 /usr/bin/g++ -> /usr/bin/clang++

But now, I've got another issue, here is a build-log: https://gist.github.com/josser/d3e4524be46afccc5227
I'm not familiar with c++ so if there are any things I missed, feel free to ask.
Thanks, and sorry for my bad english :)

Hello
I took a look at the gist and don't know what exactly might be wrong. It's clearly a bug in the way the mhash lib is being compiled but I don't know what exactly is wrong without further digging.
So I downloaded the FreeBSD 10.1 amd64 virtual disk image and I was going to test this out. Sadly I couldn't even get node to compile and install, even after symlinking g++ to clang++.

Use ports! )
cd /usr/ports/www/node
make install clean

Ok, I have some new information about this issue.
The error goes from ../deps/mhash/include/mutils/mhash.h:118:1: error: unknown type name '_Bool'
You may found this link useful: https://llvm.org/bugs/show_bug.cgi?id=21410
This is actual part from the file which lead to an error:
/*
 * Due to buggy implementations of the boolean headers in some systems,
 * we have to detect the boolean values seperately from the boolean type.
 * A big thank you to Heiko Lehmann for spotting the bug. Unfortunately,
 * this fix may not be enough, as it is only going to work on things the
 * precompiler knows about.
 *
 * Why go to all this trouble over something that should have a standard
 * value anyway? Because.
 */
#if defined(HAVE__BOOL)
#define mutils_boolean _Bool
#else
typedef char mutils_boolean;
#endif

Than I went to ports tree and check, how guys from freebsd compiles this library. And here we go:
http://svnweb.freebsd.org/ports/head/security/mhash/files/patch-include-mutils-mutils.h?view=markup
This is patch which they apply on top of the mhash source before building it for freebsd.
As you can see, there is nothing about mutils_boolean, so I downloaded source and tried to build it myself.
Mystically it builds fine without any errors.
So here I'm stuck because my knowledge of C/C++ is to weak.
The only thing I understand is that issue is in _Bool type and some difference in C/C++:
Here is example:
#include <stdbool.h>

int main(){
    _Bool y;
    return 0;
}

If I do:
$clang test.c
compiles fine, but compiling it as c++:
$clang test.cc
gives compilation error: test.cc:4:5: error: unknown type name '_Bool'
Hope this can help and sorry for my bad english again :)

Maybe I should have node-mhash only compile mhash if not found on the system. This way folks can install mhash separately and node-mhash will just use the system lib.

I have found a way to build it with this hack:
#define __GNUC__ 4

#include <stdbool.h>

int main(){
    _Bool y;
    return 0;
}

$clang test.cpp
works without any errors. Maybe you can add this but I don't know possible drawbacks

I am running into the same issue with _Bool on Ubuntu. With g++ it works fine, but with clang++ it fails since _Bool isn't defined. As far as I can tell the issue is that clang does only define _Bool in C mode or in GNUC mode, but not when you use C++:
http://clang.llvm.org/doxygen/stdbool_8h_source.html
The mhash_config.h file can't catch the issue as both g++ and clang++ use the file provided by the precompiled package, which is probably compiled with gcc.
The proper fix as I understand it would be to use bool instead of _Bool, but the comment in mutils/mutils.h suggest that that might cause trouble elsewhere. A brute force fix would be to just check for a _Bool define:
#ifdef _Bool
#  define mutils_boolean _Bool
#else
#  define mutils_boolean bool
#endif

Did anyone find a workaround this issues?
I'm having the same kind of trouble building on alpine:
...
make[1]: Leaving directory '/code/node_modules/mhash/deps/mhash'
  TOUCH Release/obj.target/libmhash.stamp
Release/obj.target/libmhash.stamp Stats {
  dev: 44,
  mode: 33188,
  nlink: 1,
  uid: 0,
  gid: 0,
  rdev: 0,
  blksize: 4096,
  ino: 13262690,
  size: 0,
  blocks: 0,
  atime: 2017-07-07T08:04:51.000Z,
  mtime: 2017-07-07T08:04:51.000Z,
  ctime: 2017-07-07T08:04:51.000Z,
  birthtime: 2017-07-07T08:04:51.000Z }
  CXX(target) Release/obj.target/mhash/mhash.o
In file included from ../deps/mhash/include/mutils/mtypes.h:24:0,
                 from ../deps/mhash/include/mhash.h:8,
                 from ../mhash.cc:7:
../deps/mhash/include/mutils/mutils.h:100:24: error: '_Bool' has not been declared
 #define mutils_boolean _Bool
                        ^
../deps/mhash/include/mutils/mutils.h:250:70: note: in expansion of macro 'mutils_boolean'
 mutils_word32 *mutils_word32nswap(mutils_word32 *x, mutils_word32 n, mutils_boolean destructive);
...

node -v v6.10.3
node-gyp -v v3.4.0

I've submitted a PR that fixed my build #20

Should be fixed with pull request #20 having been merged.


Failed to define a PVC size using kale deployment panel

Hi there,
first of all thanks for developing such a great and useful tool!
I installed kale in my kubeflow notebook server based on GKE (with snapshot created) and cloned the titanic example to give it a try.
The pipeline can be successfully complied, and uploaded, however the loaddata component cannot be completed as there comes a warning of
This step is in Pending state with this message: Unschedulable: pod has unbound immediate PersistentVolumeClaims (repeated 2 times).

And here's the log file
11-20 17:50 | kubeflow-kale |  DEBUG: ------------- Kale Start Run -------------
11-20 17:50 | kubeflow-kale |  INFO: Pipeline code saved at kfp_titanic-ml-pipeline.kfp.py
11-20 17:50 | kubeflow-kale |  INFO: Deployment Successful. Pipeline run at None/#/runs/details/52a84d9e-b1bc-41cf-af40-cc87785c5b7f

It would be very appreciated if someone could maybe kindly point me out whether I configured the volumes correctly. Thanks!

Hey @Felihong ! Thank you for trying out Kale. I would suggest to try out Kale from the kubecon-workshop (same thing for the JuyterLab extension). There are loads of new features there that we will merge into master in the next weeks. Please note that we have been developing targeting MiniKF, so there might be issues when running Kale outside MiniKF.

In any case, please provide any detailed issue you may find using the versions under kubecon-workshop branches, so that we can track them and solve any issue when running Kale in a Kubeflow cluster.

Thanks for the timely reply!
So I installed the kubecon-workshop branch of kubeflow-kale using pip install git+https://github.com/kubeflow-kale/kale.git@kubecon-workshop.
However to install the jupyterlab extension using pip install git+https://github.com/kubeflow-kale/jupyterlab-kubeflow-kale.git@kubecon-workshop, I got an error as below:
Collecting git+https://github.com/kubeflow-kale/jupyterlab-kubeflow-kale.git@kubecon-workshop
  Cloning https://github.com/kubeflow-kale/jupyterlab-kubeflow-kale.git (to revision kubecon-workshop) to /tmp/pip-req-build-3y7xrwyy
  Running command git clone -q https://github.com/kubeflow-kale/jupyterlab-kubeflow-kale.git /tmp/pip-req-build-3y7xrwyy
  Running command git checkout -b kubecon-workshop --track origin/kubecon-workshop
  Switched to a new branch 'kubecon-workshop'
  Branch 'kubecon-workshop' set up to track remote branch 'kubecon-workshop' from 'origin'.

ERROR: Command errored out with exit status 1:
     command: /opt/conda/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-3y7xrwyy/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-3y7xrwyy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-3y7xrwyy/pip-egg-info
         cwd: /tmp/pip-req-build-3y7xrwyy/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/conda/lib/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-3y7xrwyy/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Any ideas?

Oops I just saw I should use jupyter labextension install  instead, I'm wondering how should I specify the branch in this case?

I just followed the instructions in the Contributing part, the extension has been properly installed. Thanks!

Hi, here are some issues I found under the kubecon-workshop branch:

Failed to Compile and Run  example notebook candies_sharing
After I defined the experiment name and pipeline name in the panel there comes an error saying the experiment name is empty:

(400) Reason: Bad Request HTTP response headers:
HTTPHeaderDict({'content-type': 'application/json', 'trailer': 'Grpc-Trailer-Content-Type', 'date': 'Fri, 22 Nov 2019 09:33:56 GMT', 'x-envoy-upstream-service-time': '0', 'server': 'envoy', 'transfer-encoding': 'chunked'})
HTTP response body: {
"error":"Validate experiment request failed.: Invalid input error: Experiment name is empty. Please specify a valid experiment name.",
"message":"Validate experiment request failed.: Invalid input error: Experiment name is empty. Please specify a valid experiment name.","code":3,
"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Experiment name is empty. Please specify a valid experiment name.","error_details":"Validate experiment request failed.: Invalid input error: Experiment name is empty. Please specify a valid experiment name."}]}

In the generated candies-sharing-urgrg.kale.py script the experiment name seems failed to be generated:
if __name__ == "__main__":
    pipeline_func = auto_generated_pipeline
    pipeline_filename = pipeline_func.__name__ + '.pipeline.tar.gz'
    import kfp.compiler as compiler
    compiler.Compiler().compile(pipeline_func, pipeline_filename)

    # Get or create an experiment and submit a pipeline run
    import kfp
    client = kfp.Client()
    experiment = client.create_experiment('')

    # Submit a pipeline run
    run_name = 'candies-sharing-urgrg_run'
    run_result = client.run_pipeline(
        experiment.id, run_name, pipeline_filename, {})

The pipeline can be successfully uploaded but also failed to run.

Pending state when running notebook titanic-dataset-ml
It seems like the old issue of unbound PVC is not solved with the workshop branch, the component loaddata cannot be finished:

This step is in Pending state with this message: Unschedulable: pod has unbound immediate PersistentVolumeClaims (repeated 2 times)

PS: No volumes mounts are defined in both experiments.

Hey @Felihong , I suspect you are running Kale in your own Kubeflow cluster and not in MiniKF, is that right? Currently we have everything working in MiniKF and in the next week's we will work to expand support to full Kubeflow cluster!
Can you provide more information about your environment?

Hi @StefanoFioravanzo , yes I'm running kale in a kubeflow cluster which is deployed on GKE, and it will be cool if kale could be expanded to full kubeflow cluster. Thank you and I'm very glad to help!
And here are some specifications of the pod where the notebook is running:
apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/status: '{"version":"5f3ae3613c7945ef767cb9fd594596bc001ff3ab915f12e4379c0cb5648d2729","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs"],"imagePullSecrets":null}'
  generateName: kale-test
  labels:
    app: kale-conda-test
    controller-revision-hash: kale-test-5f6587c7d5
    notebook-name: kale-test
    statefulset: kale-test
    statefulset.kubernetes.io/pod-name: kale-test-0
  name: kale-test-0
  namespace: [USER_NAMESPACE]
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: kale-test
    uid: 1cc5023c-0d37-11ea-ba66-42010a84025a
  resourceVersion: "2762529"
  selfLink: /api/v1/namespaces/[USER_NAMESPACE]/pods/kale-test-0
  uid: 1d47af68-0d37-11ea-ba66-42010a84025a
spec:
  containers:
  - env:
    - name: NB_PREFIX
      value: /notebook/[USER_NAMESPACE]/kale-test
    image: [IMAGE_NAME]
    imagePullPolicy: IfNotPresent
    name: kale-test
    ports:
    - containerPort: 8888
      name: notebook-port
      protocol: TCP
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /dev/shm
      name: dshm
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-editor-token-6qblc
      readOnly: true
    workingDir: /home/jovyan
  - args:
    - proxy
    - sidecar
    - --domain
    - $(POD_NAMESPACE).svc.cluster.local
    - --configPath
    - /etc/istio/proxy
    - --binaryPath
    - /usr/local/bin/envoy
    - --serviceCluster
    - kale-test.$(POD_NAMESPACE)
    - --drainDuration
    - 45s
    - --parentShutdownDuration
    - 1m0s
    - --discoveryAddress
    - istio-pilot.istio-system:15010
    - --zipkinAddress
    - zipkin.istio-system:9411
    - --connectTimeout
    - 10s
    - --proxyAdminPort
    - "15000"
    - --concurrency
    - "2"
    - --controlPlaneAuthPolicy
    - NONE
    - --statusPort
    - "15020"
    - --applicationPorts
    - "8888"
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ISTIO_META_CONFIG_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ISTIO_META_INTERCEPTION_MODE
      value: REDIRECT
    - name: ISTIO_METAJSON_LABELS
      value: |
        {"app":"kale-conda-test","controller-revision-hash":"kale-test-5f6587c7d5","notebook-name":"kale-test","statefulset":"kale-test","statefulset.kubernetes.io/pod-name":"kale-test-0"}
    image: docker.io/istio/proxyv2:1.1.6
    imagePullPolicy: IfNotPresent
    name: istio-proxy
    ports:
    - containerPort: 15090
      name: http-envoy-prom
      protocol: TCP
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /healthz/ready
        port: 15020
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 2
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 128Mi
      requests:
        cpu: 10m
        memory: 40Mi
    securityContext:
      readOnlyRootFilesystem: true
      runAsUser: 1337
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-editor-token-6qblc
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: kale-test-0
  initContainers:
  - args:
    - -p
    - "15001"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - "8888"
    - -d
    - "15020"
    image: docker.io/istio/proxy_init:1.1.6
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 10m
        memory: 10Mi
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  nodeName: [NODE_NAME]
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 100
  serviceAccount: default-editor
  serviceAccountName: default-editor
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir:
      medium: Memory
    name: dshm
  - name: default-editor-token-6qblc
    secret:
      defaultMode: 420
      secretName: default-editor-token-6qblc
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio.default-editor

Hi @Felihong ,
Regarding the unbound PVC, Kale creates PVCs setting access mode to ReadWriteMany. Does your cluster support RWX PVCs?

Hey @elikatsis ,
thanks for the info!
I'm using the volume type gcePersistentDisk which is defined as standard StorageClass by default. Unfortunately it seems like it doesn't support RWX access mode yet https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes
I guess in this case I should create another StorageClass and use some type like NFS?
Btw should this volume be data volume or workspace volume which can be created in the kubeflow UI? I'm kind of confused...
Thank you in advance!

I guess in this case I should create another StorageClass and use some type like NFS?

That would work. But you should also set it as default storage class because it is the default storage class that gets chosen.
We could add features to choose such options in the future.

Btw should this volume be data volume or workspace volume which can be created in the kubeflow UI? I'm kind of confused...

Volumes mounted on pipeline steps don't know anything about workspace volume or data volumes.
Let's say that they are all considered data volumes.
So, the marshal volume, which is the one you have issues with, is a filesystem where data passed between steps are saved to and loaded from.
There is only one special case: If you use the default notebook image, along with your notebook server's workspace volume mounted under the same mount point (/home/jovyan), then any installed library will be present during a step's execution.

@elikatsis Thank you so much for the info!
I tried to create StorageClass provided by glutserfs and quobyte and set them as default class. According to the documents both volume plugins support RWX mode. https://kubernetes.io/docs/concepts/storage/persistent-volumes/
kubectl get storageclass
NAME             PROVISIONER               AGE
slow (default)   kubernetes.io/glusterfs   22m
standard         kubernetes.io/gce-pd      20h

However pods stay in pending when setting the access mode to RWX.
Events:
  Type     Reason             Age                From                Message
  ----     ------             ----               ----                -------
  Warning  FailedScheduling   34s (x2 over 34s)  default-scheduler   pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
  Normal   NotTriggerScaleUp  10s (x3 over 31s)  cluster-autoscaler  pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 node(s) had taints that the pod didn't tolerate

I'm not sure if I set the StorageClass somehow wrong or maybe I missed some points there. Since it is necessary to set the mode to RWX to get kale started, which is however currently not supported by GKE's default, I thought maybe you have some ideas?
Thank you!

Hi there 👋,
so my pod is using now a google Filestore backed PV with RWX access mode, which is dynamically provisioned.
kubectl get storageclass
NAME                   PROVISIONER                                   AGE
nfs-client (default)   cluster.local/nfs-cp-nfs-client-provisioner   39m
standard               kubernetes.io/gce-pd                          4d21h

The problem now seems different as before when I test run the example pipeline candy sharing.
In the first component kale-marshal-volume I can see the pipeline is now bounded successfully to a volume:
kale-marshal-volume-manifest

map[apiVersion:v1 metadata:map[name:candies-sharing-5zg0x-wmr5v-kale-marshal-pvc namespace:kubeflow
selfLink:/api/v1/namespaces/kubeflow/persistentvolumeclaims/candies-sharing-5zg0x-wmr5v-kale-marshal-pvc uid:5eff311e-14ea-11ea-a64c-42010a84021c resourceVersion:3100421 creationTimestamp:2019-12-02T09:59:03Z
annotations:map[pv.kubernetes.io/bind-completed:yes pv.kubernetes.io/bound-by-controller:yes volume.beta.kubernetes.io/storage-provisioner:cluster.local/nfs-cp-nfs-client-provisioner] finalizers:[kubernetes.io/pvc-protection]]
spec:map[resources:map[requests:map[storage:1Gi]] volumeName:pvc-5eff311e-14ea-11ea-a64c-42010a84021c storageClassName:nfs-client volumeMode:Filesystem accessModes:[ReadWriteMany]]
status:map[phase:Bound accessModes:[ReadWriteMany] capacity:map[storage:1Gi]] kind:PersistentVolumeClaim]

kale-marshal-volume-name
candies-sharing-5zg0x-wmr5v-kale-marshal-pvc

kale-marshal-volume-size
1Gi

However in the second component sack, I met an error somehow related to snapshot:
Traceback (most recent call last):
    File "<string>", line 36, in <module>
    File "<string>", line 16, in sack
    File "/opt/conda/lib/python3.7/site-packages/kale/utils/pod_utils.py", line 171, in snapshot_pipeline_step
       from rok_gw_client.client import RokClient
ModuleNotFoundError: No module named 'rok_gw_client'

I also tried to manually define the volume and also created/defined a snapshot for that, the error stays the same. Any ideas?
Thank you!

@Felihong It looks like you are now able to correctly provision and bind a volume, that is great. The issue now is that we have been building images using the rok_gw_client library, that is not publicly available. @elikatsis we should make sure that rok_gw_client is not a hard dependency, and if it fails to import then rok integration is disabled.

Opened #20 to track this

@Felihong, I'm glad you setup a storage class that can provide RWX PVCs. [I'd been trying to find out what's up with the glusterfs issue but couldn't find any info.]
I've also figured out what is wrong with the pipeline name and experiment name issue you've mentioned in your first comments. Thank you for reporting it! A fix for that will be included in upcoming releases.
The rok_gw_client issue you mention should only occur if you have the Take Rok snapshots before each step UI option switched on. Try turning it off when deploying the pipeline.

@elikatsis Thanks for pointing out! Looking forward to the new releases :)
Regarding the rod_gw_client issue, I didn't managed to locate the Take Rok snapshots before each step option in the extension. Do you mean the KALE DEPLOYMENT PANEL?
The truth is I just defined the name of experiment and pipeline, no volumes are from my side defined. (I don't even have a snapshot API enabled in my cluster).
Is there a way to do some edits in my local installed pod_utils.py script to disable the snapshot?

Regarding the rod_gw_client issue, I didn't managed to locate the Take Rok snapshots before each step option in the extension. Do you actually mean the KALE DEPLOYMENT PANEL?

Yes, in KALE DEPLOYMENT PANEL you should set your volumes pane like this:

I believe it should work if you set these options as such.
Rok is imported lazily, only when it is called, so if you disable all related options [which should be possible at this moment], then it should work. Please report back if you try it and it doesn't work.

Is there a way to do some edits in my local installed pod_utils.py script to disable the rok integration?
That would not be very easy. You would only modify the current container's filesystem, not the docker image which is used for the pods.
You would have to build a new docker image with custom Kale installation and pass that to all steps via Additional Settings.
But if you, let's say, delete lines that seem related to Rok and snapshotting, that would be different than disabling features so something could break.

Hi @elikatsis , thank you so much for your suggestion!
I now noticed it is my container not up-to-date that I can't get the latest extension version.
Good news is I tried with this image gcr.io/arrikto-public/tensorflow-1.14.0-notebook-cpu:kubecon-workshop according to https://codelabs.developers.google.com/codelabs/cloud-kubeflow-minikf-kale/#3 in my cluster to run the base-example, and it works perfectly! 😊
Regarding my notebook image, I used the following commands to integrate and build kale and kale jupyterlab extension (based on jupyterlab 1.1.1) in my Dockerfile:
RUN pip install git+https://github.com/kubeflow-kale/kale.git@kubecon-workshop

RUN git clone https://github.com/kubeflow-kale/jupyterlab-kubeflow-kale.git \
    && cd jupyterlab-kubeflow-kale \
    && jlpm install \
    && jlpm run build \
    && jupyter labextension install .

It works but obviously doesn't lead me to the latest version. Did I missed some points here?
And would that be possible to share me some details (Dockerfile etc.) about image gcr.io/arrikto-public/tensorflow-1.14.0-notebook-cpu:kubecon-workshop?
Thanks!

JupyterLab Extension development also lives in a kubecon-workshop branch. Using those commands you have installed the master version.
gcr.io/arrikto-public/tensorflow-1.14.0-notebook-cpu:kubecon-workshop image has
this as base image: gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v-base-ef41372-1177829795472347138.
Then we install Rok, latest KFP and Kale, Kale JupyterLab extension from kubecon-workshop branch.
Finally, we run jupyter lab instead of jupyter notebook.

Hi @elikatsis , you are right about the branch, I mis-pulled master branch.
Now I'm using the kubecon-workshop branch and everything works just fine!
Thank you!


Move redux logging to a different file

Some of the nightly flake tests are failing and I can't repro on my local box. This change will move the logging for the react side into a separate file so I can see only what's happening in the extension.
The other log will still contain time stamps, so it should be possible to correlate it with the main extension log.

Codecov Report

Merging #10605 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #10605      +/-   ##
==========================================
- Coverage   60.79%   60.77%   -0.02%
==========================================
  Files         579      579
  Lines       31412    31412
  Branches     4469     4469
==========================================
- Hits        19097    19091       -6
- Misses      11344    11348       +4
- Partials      971      973       +2

Impacted Files
Coverage Δ

src/datascience-ui/react-common/arePathsSame.ts
75.00% <0.00%> (-12.50%)
:arrow_down:

src/client/common/utils/platform.ts
64.70% <0.00%> (-11.77%)
:arrow_down:

src/client/linters/pydocstyle.ts
86.66% <0.00%> (-2.23%)
:arrow_down:

src/client/datascience/debugLocationTracker.ts
76.56% <0.00%> (-1.57%)
:arrow_down:

src/client/common/process/proc.ts
14.49% <0.00%> (-0.73%)
:arrow_down:

...bugger/extension/configuration/resolvers/attach.ts
81.66% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 324824a...1046292. Read the comment docs.


Rabit Poll Timeout Error Flakiness when using Approx Tree Method

Using the "approx" tree method during distributed XGBoost training seems rather flaky. By my observation, the exact same training job with the exact same data and configurations will fail around 50% of the time when using this method.
When it does fail, it will get to the part where the RabitTracker receives the start signal from all of the workers and print "All Workers have started". However, it will then just hang there for 30 minutes and eventually fail with some opaque Rabit poll timeout error.
The "hist" tree method is far more stable and almost always works, but unfortunately consumes vastly more memory than the "approx" method so I'd like to stick to the approx method if possible.
Dataset contains ~6500 columns and 40000 rows. This is for testing purposes, but I've observed the same flaky behavior in the full size dataset as well.
Environment

Spark 3.1.1
xgboost4j 1.4.1
xgboost4j-spark 1.4.1
YARN
2.0.19-debian10 via Google DataProc
Scala 2.12

Spark Config:

spark.yarn.am.memory":"1024m"
spark.yarn.am.memoryOverhead":"1024m"
spark.dynamicAllocation.enabled":"false"
spark.kryoserializer.buffer.max":"1048"
spark.task.cpus":"4"
spark.executor.memory":"45g"
spark.executor.memoryOverhead":"5g"
spark.executor.cores":"8"
spark.executor.heartbeatInterval":"1000000"
spark.network.timeout":"2000000"
spark.driver.memory":"26g"
spark.driver.cores":"8"
spark.sql.shuffle.partitions":"1000"
spark.default.parallelism":"1000"
spark.sql.parquet.enableVectorizedReader":"false"
spark.memory.fraction":"0.8"
spark.executor.instances":"8"

Learning Algorithm Parameters

num_workers: 16
nthread: 4
importance_type: total_gain
colsample_bytree: 0.8
objective: "binary:logistic"
alpha: 0.1
lambda: 1.2
subsample: 0.8
verbosity: 3
max_depth: 8
learning_rate: 0.25
min_child_weight: 5.2
gamma: 0.15
num_round: 10
scale_pos_weight: 1.52
tree_method: "approx"
random_state: 666
eval_metric: "logloss"

Logs:
21/09/22 16:36:13 WARN XGBoostSpark: train_test_ratio is deprecated since XGBoost 0.82, we recommend to explicitly pass a training and multiple evaluation datasets by passing 'eval_sets' and 'eval_set_names'
[INFO] [09/22/2021 16:36:14.082] [RabitTracker-akka.actor.default-dispatcher-2] [akka://RabitTracker/user/Handler] Tracker listening @ <IP_ADDRESS>:44077
[INFO] [09/22/2021 16:36:14.083] [RabitTracker-akka.actor.default-dispatcher-2] [akka://RabitTracker/user/Handler] Worker connection timeout is 5 minutes.
21/09/22 16:36:14 INFO XGBoostSpark: starting training with timeout set as 1800000 ms for waiting for resources
21/09/22 16:36:14 WARN org.apache.spark.scheduler.DAGScheduler: Broadcasting large task binary with size 1672.6 KiB
[INFO] [09/22/2021 16:36:56.078] [RabitTracker-akka.actor.default-dispatcher-4] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 0]
[INFO] [09/22/2021 16:36:56.080] [RabitTracker-akka.actor.default-dispatcher-4] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 1]
[INFO] [09/22/2021 16:36:56.086] [RabitTracker-akka.actor.default-dispatcher-5] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 2]
[INFO] [09/22/2021 16:36:56.088] [RabitTracker-akka.actor.default-dispatcher-5] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 3]
[INFO] [09/22/2021 16:36:56.088] [RabitTracker-akka.actor.default-dispatcher-7] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 4]
[INFO] [09/22/2021 16:36:56.089] [RabitTracker-akka.actor.default-dispatcher-3] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 0) has started.
[INFO] [09/22/2021 16:36:56.098] [RabitTracker-akka.actor.default-dispatcher-7] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 5]
[INFO] [09/22/2021 16:36:56.098] [RabitTracker-akka.actor.default-dispatcher-7] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 1) has started.
[INFO] [09/22/2021 16:36:56.099] [RabitTracker-akka.actor.default-dispatcher-7] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 6]
[INFO] [09/22/2021 16:36:56.099] [RabitTracker-akka.actor.default-dispatcher-7] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 7]
[INFO] [09/22/2021 16:36:56.100] [RabitTracker-akka.actor.default-dispatcher-7] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 2) has started.
[INFO] [09/22/2021 16:36:56.101] [RabitTracker-akka.actor.default-dispatcher-16] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 8]
[INFO] [09/22/2021 16:36:56.107] [RabitTracker-akka.actor.default-dispatcher-21] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 3) has started.
[INFO] [09/22/2021 16:36:56.108] [RabitTracker-akka.actor.default-dispatcher-21] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 9]
[INFO] [09/22/2021 16:36:56.108] [RabitTracker-akka.actor.default-dispatcher-21] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 10]
[INFO] [09/22/2021 16:36:56.112] [RabitTracker-akka.actor.default-dispatcher-11] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 4) has started.
[INFO] [09/22/2021 16:36:56.114] [RabitTracker-akka.actor.default-dispatcher-22] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 11]
[INFO] [09/22/2021 16:36:56.114] [RabitTracker-akka.actor.default-dispatcher-22] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 12]
[INFO] [09/22/2021 16:36:56.116] [RabitTracker-akka.actor.default-dispatcher-2] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 5) has started.
[INFO] [09/22/2021 16:36:56.119] [RabitTracker-akka.actor.default-dispatcher-31] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 6) has started.
[INFO] [09/22/2021 16:36:56.125] [RabitTracker-akka.actor.default-dispatcher-31] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 7) has started.
[INFO] [09/22/2021 16:36:56.126] [RabitTracker-akka.actor.default-dispatcher-12] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 8) has started.
[INFO] [09/22/2021 16:36:56.129] [RabitTracker-akka.actor.default-dispatcher-21] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 9) has started.
[INFO] [09/22/2021 16:36:56.130] [RabitTracker-akka.actor.default-dispatcher-27] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 10) has started.
[INFO] [09/22/2021 16:36:56.132] [RabitTracker-akka.actor.default-dispatcher-25] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 11) has started.
[INFO] [09/22/2021 16:36:56.134] [RabitTracker-akka.actor.default-dispatcher-14] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 12) has started.
[INFO] [09/22/2021 16:36:56.149] [RabitTracker-akka.actor.default-dispatcher-33] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 13]
[INFO] [09/22/2021 16:36:56.151] [RabitTracker-akka.actor.default-dispatcher-5] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 13) has started.
[INFO] [09/22/2021 16:36:56.169] [RabitTracker-akka.actor.default-dispatcher-25] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal [rank: 14]
[INFO] [09/22/2021 16:36:56.171] [RabitTracker-akka.actor.default-dispatcher-25] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 14) has started.
[INFO] [09/22/2021 16:36:56.181] [RabitTracker-akka.actor.default-dispatcher-25] [akka://RabitTracker/user/Handler] Received start signal from fa-test-scala-trackerconf-3-w-1.c.ds-risk-prod.internal [rank: 15]
[INFO] [09/22/2021 16:36:56.184] [RabitTracker-akka.actor.default-dispatcher-20] [akka://RabitTracker/user/Handler] Worker <IP_ADDRESS> (rank: 15) has started.
[INFO] [09/22/2021 16:36:56.184] [RabitTracker-akka.actor.default-dispatcher-20] [akka://RabitTracker/user/Handler] All workers have started.
[INFO] [09/22/2021 17:06:56.409] [RabitTracker-akka.actor.default-dispatcher-4] [akka://RabitTracker/user/Handler] Received shutdown signal from 11
21/09/22 17:06:56 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 10.0 in stage 41.0 (TID 9758) (fa-test-scala-trackerconf-3-w-0.c.ds-risk-prod.internal executor 8): ml.dmlc.xgboost4j.java.XGBoostError: [17:06:56] /workspace/rabit/include/rabit/internal/socket.h:630: Poll timeout
Stack trace:
[bt] (0) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x53) [0x7f5f57f31843]
[bt] (1) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(rabit::engine::AllreduceBase::TryAllreduceTree(void*, unsigned long, unsigned long, void ()(void const, void*, int, MPI::Datatype const&))+0xa16) [0x7f5f58214166]
[bt] (2) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(rabit::engine::AllreduceBase::TryAllreduce(void*, unsigned long, unsigned long, void ()(void const, void*, int, MPI::Datatype const&))+0x12) [0x7f5f58216c52]
[bt] (3) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(rabit::engine::ReduceHandle::Allreduce(void*, unsigned long, unsigned long, void ()(void), void*)+0xa4) [0x7f5f58226c14]
[bt] (4) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(rabit::SerializeReducer<xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WXQSummary<float, float> >::SummaryContainer>::Allreduce(xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WXQSummary<float, float> >::SummaryContainer*, unsigned long, unsigned long, void ()(void), void*)+0xc0) [0x7f5f581a1fb0]
[bt] (5) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(xgboost::tree::CQHistMaker::ResetPosAndPropose(std::vector<xgboost::detail::GradientPairInternal, std::allocator<xgboost::detail::GradientPairInternal > > const&, xgboost::DMatrix*, std::vector<unsigned int, std::allocator > const&, xgboost::RegTree const&)+0x1c4c) [0x7f5f581a76cc]
[bt] (6) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(xgboost::tree::GlobalProposalHistMaker::ResetPosAndPropose(std::vector<xgboost::detail::GradientPairInternal, std::allocator<xgboost::detail::GradientPairInternal > > const&, xgboost::DMatrix*, std::vector<unsigned int, std::allocator > const&, xgboost::RegTree const&)+0x11c) [0x7f5f581a80dc]
[bt] (7) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(xgboost::tree::HistMaker::UpdateTree(std::vector<xgboost::detail::GradientPairInternal, std::allocator<xgboost::detail::GradientPairInternal > > const&, xgboost::DMatrix*, xgboost::RegTree*)+0xb5) [0x7f5f58191295]
[bt] (8) /hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1632327675372_0001/container_1632327675372_0001_01_000008/tmp/libxgboost4j3745035270672734986.so(xgboost::tree::HistMaker::Update(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::DMatrix, std::vector<xgboost::RegTree*, std::allocatorxgboost::RegTree* > const&)+0xb8) [0x7f5f5818c1b8]
at ml.dmlc.xgboost4j.java.XGBoostJNI.checkCall(XGBoostJNI.java:48)
at ml.dmlc.xgboost4j.java.Booster.update(Booster.java:172)
at ml.dmlc.xgboost4j.java.XGBoost.trainAndSaveCheckpoint(XGBoost.java:218)
at ml.dmlc.xgboost4j.java.XGBoost.train(XGBoost.java:300)
at ml.dmlc.xgboost4j.scala.XGBoost$.$anonfun$trainAndSaveCheckpoint$5(XGBoost.scala:66)
at scala.Option.getOrElse(Option.scala:121)
at ml.dmlc.xgboost4j.scala.XGBoost$.trainAndSaveCheckpoint(XGBoost.scala:62)
at ml.dmlc.xgboost4j.scala.XGBoost$.train(XGBoost.scala:106)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.buildDistributedBooster(XGBoost.scala:416)
at ml.dmlc.xgboost4j.scala.spark.XGBoost$.$anonfun$trainForNonRanking$1(XGBoost.scala:499)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:863)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:863)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:386)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1423)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Can you post an example program with which we can reproduce the problem?

@hcho3
For privacy reasons I can't give the exact same data, but we can create synthetic data like this to mimic the dataset I used:
import pandas
import random
num_features = 7000
num_rows = 30000
data = [[random.random() for _ in range(num_features)] + [random.randint(0,1)] for _ in range(num_rows)]
columns = [f"feature{i}" for i in range(num_features)] + ["label"]
df = pandas.DataFrame(data, columns=columns)
df = pandas.to_parquet("path_to_your_parquet_file.pq")

Using approx method and the same conditions as I described in this post, I got 2/5 successful runs using the synthetic data.
For the training code (in Scala), here's a pared down version of my work code. Nothing fancy:
val dataPath = "path_to_your_parquet_file.pq"
val trainDF = spark.read.parquet(dataPath)

val featureColumns = Range(0, 7000).map(x => s"feature${x}")
val vectorAssembler = new VectorAssembler().setInputCols(featureColumns)
                                           .setOutputCol("features")
                                           .setHandleInvalid("keep")
val sparseToDenseTransformer = new SparseToDenseTransformer().setInputCol("features")
                                                             .setOutputCol("denseFeatures")
val xgboostTrainer = new XGBoostClassifier(xgboostParams).setLabelCol("label")
                                                         .setFeaturesCol("denseFeatures")
val pipeline = new Pipeline().setStages(Array(
  vectorAssembler,
  sparseToDenseTransformer,
  xgboostModel
))
val pipelineModel = pipeline.fit(trainDF)

@tristers-at-square Thanks. Let me see if I can reproduce this issue locally using a single machine.

@hcho3  Thanks!

@hcho3 If you manage to reproduce it, could you please also test https://github.com/dmlc/xgboost/pull/7214 ?

I'm running into a similar issue when using Spark on k8s with the approx method that does not seem to trigger when using the hist method. I'm using a SNAPSHOT xgboost compiled from the master branch within the last few days with a SNAPSHOT of spark (~3.3.x). Java 11.0.13 and Python 3.8 are used on all the hosts.
After I start the job, it takes only a few minutes to get to the forEachPartition step. In this job, nWorkers is set to 250, there are 3 cores per executor, nthread is set to 3, and spark.task.cpus is also set to 3.

From the logs, the driver acknowledges the correct number of nodes are available and appears to start the training process.

However, just a few minutes in, the CPU usage goes to 0% and the memory usage does not change further.

This suggests to me that the training process is hung and no useful work is being done. The driver CPU usage also goes down and is mostly flat, but does not quite go to zero.
After about 30 minutes, the job gives up and the logs report the following error:
 WARN TaskSetManager: Lost task 42.0 in stage 11.0 (TID 2064) (<IP_ADDRESS> executor 70): ml.dmlc.xgboost4j.java.XGBoostError: [20:54:20] /opt/xgboost/rabit/include/rabit/internal/socket.h:630: Poll timeout
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
Stack trace:
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (0) /tmp/libxgboost4j11483072497966676486.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x59) [0x7f3b835b72c5]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (1) /tmp/libxgboost4j11483072497966676486.so(rabit::utils::PollHelper::Poll(std::chrono::duration<long, std::ratio<1l, 1l> >)+0x141) [0x7f3b839129d5]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (2) /tmp/libxgboost4j11483072497966676486.so(rabit::engine::AllreduceBase::TryAllreduceTree(void*, unsigned long, unsigned long, void (*)(void const*, void*, int, MPI::Datatype const&))+0x474) [0x7f3b8390f584]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (3) /tmp/libxgboost4j11483072497966676486.so(rabit::engine::AllreduceBase::TryAllreduce(void*, unsigned long, unsigned long, void (*)(void const*, void*, int, MPI::Datatype const&))+0x71) [0x7f3b8390f10d]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (4) /tmp/libxgboost4j11483072497966676486.so(rabit::engine::AllreduceBase::Allreduce(void*, unsigned long, unsigned long, void (*)(void const*, void*, int, MPI::Datatype const&), void (*)(void*), void*)+0x93) [0x7f3b83912d2b]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (5) /tmp/libxgboost4j11483072497966676486.so(rabit::engine::ReduceHandle::Allreduce(void*, unsigned long, unsigned long, void (*)(void*), void*)+0x82) [0x7f3b8391f2a0]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (6) /tmp/libxgboost4j11483072497966676486.so(rabit::SerializeReducer<xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WXQSummary<float, float> >::SummaryContainer>::Allreduce(xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WXQSummary<float, float> >::SummaryContainer*, unsigned long, unsigned long, void (*)(void*), void*)+0xfe) [0x7f3b83887dcc]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (7) /tmp/libxgboost4j11483072497966676486.so(xgboost::tree::CQHistMaker::ResetPosAndPropose(std::vector<xgboost::detail::GradientPairInternal<float>, std::allocator<xgboost::detail::GradientPairInternal<float> > > const&, xgboost::DMatrix*, std::vector<unsigned int, std::allocator<unsigned int> > const&, xgboost::RegTree const&)+0xa26) [0x7f3b8388224e]
Error
2021-11-23 12:54:20.982 PST
spark-kubernetes-driver
 [bt] (8) /tmp/libxgboost4j11483072497966676486.so(xgboost::tree::GlobalProposalHistMaker::ResetPosAndPropose(std::vector<xgboost::detail::GradientPairInternal<float>, std::allocator<xgboost::detail::GradientPairInternal<float> > > const&, xgboost::DMatrix*, std::vector<unsigned int, std::allocator<unsigned int> > const&, xgboost::RegTree const&)+0x1a2) [0x7f3b83883c4c]

I've attached a thread dump from the driver recorded about 15 minutes after the nodes have nominally started in case it may help in finding the root cause.
driver_threaddump.log

I can't reproduce the issue, it would be great if someone can test the nightly build.  Feel free to ping me if you need any help. ;-)

Closing since the approx is rewritten based on the hist tree method.  Now they share a lot of common code structures.


Production fix: validate-env for graphql url

In production, we use a local path /_hasura/v1/graphql to access the hasura graphql server via https (or http).

thanks @art-w!


Unknown reason Terminated

Hi, thank you for your sharing code~!
following your tutorial train on hockey dataset, l meet a Unknown reason Terminated.
any suggestions will be welcomed.
the following is training log:
python3 Train.py
Using Network:  src.net.G2D19_P2OF_ResHB_1LSTM
2019-03-04 22:10:58.491399: W tensorflow/core/graph/graph_constructor.cc:1265] Importing a graph with a lower producer version 24 into an existing graph with producer version 27. Shape inference will have run different parts of the graph with different producer versions.
In OpticalFlow:
pool2.shape =  (?, 112, 112, 32)
after transpose, out.shape =  (?, 112, 112, 32, 2)
before Conv2, out.shape =  (?, 112, 112, 64)
before Fc, out.shape =  (?, 28, 28, 64)
Fc final.shape =  (?, 1024)
opticalFlowOut.shape =  (?, 1024)
In CNN:
darknetOutput.shape =  (?, 7, 7, 1024)
after transpose, out.shape =  (?, 7, 7, 1024, 2)
before ConcatConv, out.shape =  (?, 7, 7, 2048)
before Fc, out.shape =  (?, 7, 7, 128)
Fc out.shape =  (?, 1024)
In Concat:
before Fc, out.shape =  (?, 2048)
before LSTM, shape =  (?, ?, 1024)
WARNING:tensorflow:From /media/guo/DataUbuntu/ViolenceDetection/src/Classifier.py:34: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See tf.nn.softmax_cross_entropy_with_logits_v2.
Reading Training set...
Done.
Reading Validation set...
Done.
Reading Test set...
Done.
2019-03-04 22:11:19.952292: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-03-04 22:11:20.146532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 11.12GiB
2019-03-04 22:11:20.477948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:02:00.0
totalMemory: 11.91GiB freeMemory: 11.75GiB
2019-03-04 22:11:20.481819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-03-04 22:11:21.286405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-04 22:11:21.286446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1
2019-03-04 22:11:21.286455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y
2019-03-04 22:11:21.286460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N
2019-03-04 22:11:21.286937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10761 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-03-04 22:11:21.287347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 11370 MB memory) -> physical GPU (device: 1, name: TITAN X (Pascal), pci bus id: 0000:02:00.0, compute capability: 6.1)
Path to save mode:  outputs/
Start Training...
Averaged GetBatch time:  0.0006592693328857422
Epoch: 1 ==================================================================================================================
Back Propergation time measurement:
duration:  244.98 s/epoch
average:  1.96 s/batch
 validation:
     loss: 0.4407      frame accuracy: 0.8793      best frame threshold: 22      video accuracy: 0.9200      duration: 59.85(s)

 train:
     loss: 0.4232      frame accuracy: 0.8943      given frame threshold: 22      video accuracy: 0.9160      duration: 284.97(s)

Terminated
environment information:
tensorflow 1.12.0
Cuda compilation tools, release 9.0, V9.0.176
cudnn 7.1.4

Hi,
Could you give me more information about your hardware?  Moreover, could you make sure your Main Memory can handle the burden of loading the videos?  (In my computer, it will consume nearly 30GB of Main Memory!)
Also, it seems that the whole process stuck in evaluating the Test Set.  Do you have any clue about this?  (e.g. do you add many or very large video as  your Test Set?).  Normally, the user have the issue of the loading thread may cause the training step failed.  However, you seems stuck in the evaluation...


Cancelled Games on a Double Header Day Fails to Load

Report:
Double header - -one game cancelled and screen dumps errors....
NOTE:
Since double headers are pretty rare, I did not put too too much effort here but they are supported. If the first ends without a 'Final' status'  that could be a problem...Will check this out.

If a game is cancelled it may show up as postponed:
> Game: {'doubleheader': 'N', 'away_pitcher_note': '', 'game_date': '2023-06-21', 'inning_state': '', 'away_probable_pitcher': '', 'home_rec': '38-35', 'away_rec': '47-26', 'game_num': 1, 'game_datetime': '2023-06-21T22:40:00Z', 'current_inning': '', 'game_type': 'R', 'away_score': '0', 'away_name': 'Atlanta Braves', 'summary': '2023-06-21 - Atlanta Braves @ Philadelphia Phillies (Postponed)', 'away_id': 144, 'home_name': 'Philadelphia Phillies', 'status': 'Postponed', 'venue_id': 2681, 'venue_name': 'Citizens Bank Park', 'home_probable_pitcher': '', 'home_pitcher_note': '', 'home_score': '0', 'home_id': 143, 'game_id': 717664}
That is treated as:
      else:  #"Pre Game / Delayed / Postponed"
                self.show_scheduled()

Speaking with the user they have an old version. Once upgraded this should no longer occur.
Closing.


Autocomplete trigger

Readme says that ctrl + space is the key combination for autocomplete feature. Can we change it to do autocomplete without ctrl + space ?

You can change to any key combination you like. Take a look at "Settings" -> "Keybinding", search for "autocomplete". You could then set your own keybinding using your keymap file.


JSDoc and spread

All examples are valid for Closure Compiler.
Expects valid type instead of ...?
/**
 * @param {...?} args
 */
function decorate(args) {

}

Expects valid type instead of function(...?)
/**
 * @return {function(...?)}
 */
function decorate() {
	return function () {
		[].slice.call(arguments).forEach(function () {

		});
	};
}

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/25079061-jsdoc-and-spread?utm_campaign=plugin&utm_content=tracker%2F281640&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F281640&utm_medium=issues&utm_source=github).

Are you sure about ...?? Looks like you need to use ...*.

Are you sure about ...?? Looks like you need to use ...*

Yep.
The ALL type * Indicates that the variable can take on any type.
The UNKNOWN type ? Indicates that the variable can take on any type, and the compiler should not type-check any uses of it.
https://developers.google.com/closure/compiler/docs/js-for-compiler

<EMAIL_ADDRESS>still has this bug :cry:

I've left an issue in jsdoctypeparser, hope to find some time in a week or two.

Okay, my turn now ;-)

+1

pleeeese, very need this fix :sob:

At this point only major and CST related bugs will be fixed.


Development: Speedup client tests for exercise assesment dashboard

Checklist
General

[x] I chose a title conforming to the [naming conventions for pull requests]

Client

[ ] I followed the coding and design guidelines.

Motivation and Context

This PR speeds up the client tests in exercise-assesment-dashboard.component.spec.ts.

Description

Mocked what possible, removed another module import and replaced a few toEquals() with toBe()
Review Progress
Code Review

[ ] Review 1
[ ] Review 2

I think you could improve the provider mocking. Maybe this will speed up things. => Take a look at the constructor of the component


[0.4.4] Freezing on Windows after switching projects multiple times

Today I have managed to get ecode to freeze (lock-up) several times.
This has never happened before.
The only thing I've been doing differently today is switching between folders (projects) several times.

I'm not sure what other information I can provide to help find the bug.
Thanks,

That's probably because it's still scanning all folders while you open new folders. I think i forgot to stop the scanning when changing folders (the scan is really fast in SSDs and never thought about it).

Yep, that might be it.
I'm running on a fast NVME SSD though... but still it sounds likely as there are a ton of files in some of these project subfolders.
Thanks,

Sure, where do I find the logs?

Linux: uses XDG_CONFIG_HOME, usually translates to ~/.config/ecode/ecode.log
macOS: uses Application Support folder in HOME, usually translates to ~/Library/Application Support/ecode/ecode.log
Windows: uses APPDATA, usually translates to C:\Users{username}\AppData\Roaming\ecode\ecode.log

ecode.log

Just happened again while I was typing..

Hmmm...

I think it's the LSP. LSP is broken in 0.4.4 on Windows, please use this build:
ecode-windows-0.4.4-x86_64.zip. It should fix the issue...

Let me know if you get diagnostics and code completion in the new build. Something weird is going on with Omnisharp. Also if you can run ecode with debug log will be really helpful, debug logs are enabled by running ecode.exe -ldebug.

ecode.log
Here's a new log...seems to be an exception - not sure if that's the cause of the freezing or something else.
I can remove Omnisharp to see if it still has this issue..  It's easy to reproduce for me with this project.
Thanks,

Yes, without Omnisharp the freeze doesn't seem to happen.  Aslo I noticed a bunch of zombie Omnisharp.exe processes that must have been related to each time I force closed ecode.
I hope this helps but not sure what you can do to fix it :-)
Let me know if you need any more information.
Cheers,

It seems I don't gracefully handle a stuck LSP, my theory is that all the pool threads are being frozen by the LSP. I'll check out how to reproduce it. Your particular case should be easy to fix, the OmniSharp installation is lacking MSBuild that it's required to properly function. You need to install MSBuild.

I won't close the issue until I'm 100% sure a freeze can happen. I've been trying to reproduce the bug but I wasn't able, but, I was able to reproduce the zombie processes, and it's fixed for sure. I tested with your same OmniSharp setup (I mean, even without MSBuild installed), and worked fine (of course OmniSharp won't work), OmniSharp with MSBuild installed works nicely as far as I could test, but I'm not a C# user so I'm not sure how good it is.
I new release with a ton of fixes will be released very soon (may be in a few hours I can build all the platforms).

Thanks I will test and let you know.

I haven't had the freeze since 0.4.5 so I guess it must be solved 👍

Awesome! Thank you for testing it. I'll close the issue, and if you see something wrong just let me know and I'll re-open it.


setup.sh

Gives this error over and over.
Fetching... this can take some time...
the input device is not a TTY

okay, your connection to GCP doesnt have a PTY opened. You can either force the connection to open a PTY, or change the line 83 of setup.sh and remove the "-ti" from the commandline.

okay, your connection to GCP doesnt have a PTY opened. You can either force the connection to open a PTY, or change the line 83 of setup.sh and remove the "-ti" from the commandline.

Thanks I'm trying now.


Difference between "variables:" and "parameters:"

The YAML schema doc doesn't mention "variables:" at all. Is this Variables article outdated?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dd7e0bd3-1f7d-d7b6-cc72-5ef63c31b46a
Version Independent ID: dae87abd-b73d-9120-bcdb-6097d4b40f2a
Content: Pipeline variables - Azure Pipelines
Content Source: docs/pipelines/process/variables.md
Product: devops
GitHub Login: @andyjlewis
Microsoft Alias: alewis

The "variables:" is mentioned in multiple places in the YAML schema doc. I am not sure what you mean.

You're right, I'm not sure why I didn't see that. My bad!

I think it would be good to have a summary somewhere that says what the difference between variables and parameters is, mentioning for example that parameters are not directly accessible as environment variables, but variables are.


Tiler terminates with status 143

Often tiler terminates with status 143.
[Thu Jan 29 10:12:21 2015] init: node-windshaft main process (25442) terminated with status 143
[Wed Feb  4 17:31:45 2015] init: node-windshaft main process (30056) terminated with status 143
[Mon Feb  9 16:01:52 2015] init: node-windshaft main process (4733) terminated with status 143
[Wed Feb 11 11:31:38 2015] init: node-windshaft main process (10553) terminated with status 143
[Wed Feb 11 12:02:02 2015] init: node-windshaft main process (8704) terminated with status 143
[Wed Feb 11 14:31:06 2015] init: node-windshaft main process (21510) terminated with status 143
[Wed Feb 11 15:00:57 2015] init: node-windshaft main process (23083) terminated with status 143
[Wed Feb 11 15:30:41 2015] init: node-windshaft main process (4015) terminated with status 143
[Fri Feb 13 17:31:52 2015] init: node-windshaft main process (17477) terminated with status 143
[Wed Feb 18 12:01:00 2015] init: node-windshaft main process (24837) terminated with status 143
[Thu Feb 19 12:02:09 2015] init: node-windshaft main process (12884) terminated with status 143
[Fri Feb 20 11:31:57 2015] init: node-windshaft main process (6249) terminated with status 143
[Tue Feb 24 20:52:24 2015] init: node-windshaft main process (25399) terminated with status 143
[Wed Mar 11 16:40:06 2015] init: node-windshaft main process (20886) terminated with status 143
[Fri Mar 20 12:29:32 2015] init: node-windshaft main process (16349) terminated with status 143
[Wed Apr  1 12:01:59 2015] init: node-windshaft main process (14308) terminated with status 143
[Tue Apr  7 09:01:36 2015] init: node-windshaft main process (11581) terminated with status 143
[Wed Apr  8 09:31:13 2015] init: node-windshaft main process (2982) terminated with status 143
[Wed Apr  8 11:42:09 2015] init: node-windshaft main process (11135) terminated with status 143
[Thu Apr  9 13:40:53 2015] init: node-windshaft main process (7626) terminated with status 143
[Fri Apr 10 09:07:26 2015] init: node-windshaft main process (5255) terminated with status 143
[Mon Apr 13 10:10:25 2015] init: node-windshaft main process (8247) terminated with status 143
[Tue Apr 14 16:42:01 2015] init: node-windshaft main process (22352) terminated with status 143
[Wed Apr 15 14:32:18 2015] init: node-windshaft main process (22164) terminated with status 143
[Thu Apr 16 16:36:17 2015] init: node-windshaft main process (27538) terminated with status 143
[Fri Apr 24 07:04:23 2015] init: node-windshaft main process (32000) terminated with status 143

Supersedes https://github.com/CartoDB/Windshaft/issues/327.


Implementing my own custom optimizer

I was following the instructions on the read me but I still got issues. This is the issue I currently have:
Exception has occurred: ValueError
Optimizer type <class '__main__.TrainableSGD'> not supported by higher yet.

but I followed the instructions in the readme, which makes me feel this is a bug or higher doesn't actually allow my own custom optimizer.
The specific thing my optimizer is doing doesn't really matter but I will paste the code just for reference (note it's just really silly optimizer):
class MySGD(Optimizer):

    def __init__(self, params):
        defaults = {}
        super().__init__(params,defaults)

class TrainableSGD(DifferentiableOptimizer):

    def _update(self, grouped_grads, **kwargs):
        old_eta = self.__old_eta
        eta = self.__eta
        # start differentiable & trainable update
        eta = eta(old_eta)
        zipped = zip(self.param_groups, grouped_grads)
        for group_idx, (group, grads) in enumerate(zipped):
            for p_idx, (p, g) in enumerate(zip(group['params'], grads)):
                if g is None:
                    continue
                #group['params'][p_idx] = _add(p, -group['lr'], g)
                group['params'][p_idx] = p + eta*g
        # fake returns
        self._old_eta = eta

Full script:
'''
Single task MAML:

MAML: min_{theta} sum_t L^val( theta - eta* Grad L^train(theta) )

T-step MAML: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) )
outerloop: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) ) ~ min_{theta} sum_t L^val( argmin L^train(theta) )
Innerloop: theta^{T} - eta* Grad L^train(theta^{T}) ~ argmin L^train(theta)

single task MAML: min_{theta} L^val( theta - eta* Grad L^train(theta) )

based on MAML example: https://github.com/facebookresearch/higher/blob/master/examples/maml-omniglot.py
'''

import torch
import torch.nn as nn
from torch.optim.optimizer import Optimizer

import higher
from higher.optim import DifferentiableOptimizer
from higher.optim import DifferentiableSGD

import torchvision
import torchvision.transforms as transforms

from torchviz import make_dot

import copy

import itertools

from collections import OrderedDict

#mini class to add a flatten layer to the ordered dictionary
class Flatten(nn.Module):
    def forward(self, input):
        '''
        Note that input.size(0) is usually the batch size.
        So what it does is that given any input with input.size(0) # of batches,
        will flatten to be 1 * nb_elements.
        '''
        batch_size = input.size(0)
        out = input.view(batch_size,-1)
        return out # (batch_size, *size)

def get_cifar10():
    transform = transforms.Compose(
        [transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                            shuffle=True, num_workers=2)

    testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                        download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                            shuffle=False, num_workers=2)
    return trainloader, testloader

class MySGD(Optimizer):

    def __init__(self, params):
        defaults = {}
        super().__init__(params,defaults)

class TrainableSGD(DifferentiableOptimizer):

    def _update(self, grouped_grads, **kwargs):
        old_eta = self.__old_eta
        eta = self.__eta
        # start differentiable & trainable update
        eta = eta(old_eta)
        zipped = zip(self.param_groups, grouped_grads)
        for group_idx, (group, grads) in enumerate(zipped):
            for p_idx, (p, g) in enumerate(zip(group['params'], grads)):
                if g is None:
                    continue
                #group['params'][p_idx] = _add(p, -group['lr'], g)
                group['params'][p_idx] = p + eta*g
        # fake returns
        self._old_eta = eta

# get dataloaders
trainloader, testloader = get_cifar10()
criterion = nn.CrossEntropyLoss()

#
child_model = nn.Sequential(OrderedDict([
        ('conv1', nn.Conv2d(in_channels=3,out_channels=2,kernel_size=5)),
        ('relu1', nn.ReLU()),
        ('Flatten', Flatten()),
        ('fc', nn.Linear(in_features=28*28*2,out_features=10) )
    ]))
hidden = torch.randn(size=(1,1),requires_grad=True)
eta = nn.Sequential(OrderedDict([
    ('fc', nn.Linear(1,1)),
    ('sigmoid', nn.Sigmoid())
]))
other = MySGD(child_model.parameters())
inner_opt = TrainableSGD(other, child_model.parameters())
#meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_params = itertools.chain(eta.parameters(),[hidden])
meta_opt = torch.optim.Adam(meta_params, lr=1e-3)
# do meta-training/outer training minimize outerloop: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) )
nb_outer_steps = 2 # note, in this case it's the same as number of meta-train steps (but it's could not be the same depending how you loop through the val set)
for outer_i, (outer_inputs, outer_targets) in enumerate(testloader, 0):
    meta_opt.zero_grad()
    if outer_i >= nb_outer_steps:
        break
    # do inner-training/MAML; minimize innerloop: theta^{T} - eta* Grad L^train(theta^{T}) ~ argmin L^train(theta)
    nb_inner_steps = 3
    inner_opt.__old_eta = hidden
    inner_opt.__eta = eta
    with higher.innerloop_ctx(child_model, inner_opt) as (fmodel, diffopt):
        for inner_i, (inner_inputs, inner_targets) in enumerate(trainloader, 0):
            print(f'inner_i = {inner_i}')
            print(f'et^<{inner_i-1}> = {diffopt.__old_eta}')
            if inner_i >= nb_inner_steps:
                break
            logits = fmodel(inner_inputs)
            inner_loss = criterion(logits, inner_targets)
            #print(f'inner i = {inner_i}')
            #print(f'inner_loss^<{inner_i}>: {inner_loss}')
            old_eta = diffopt.step(inner_loss) # changes params P[t+1] using P[t] and loss[t] in a differentiable manner
        # compute the meta-loss L^val( theta^{T} - eta* Grad L^train(theta^{T}) )
        outer_outputs = fmodel(outer_inputs)
        meta_loss = criterion(outer_outputs, outer_targets) # L^val
        #grad_of_grads = torch.autograd.grad(outputs=meta_loss, inputs=fmodel.parameters(time=0)) # dmeta_loss/dw0
        print(f'outer_i = {outer_i}')
        print(f'-> outer_loss/meta_loss^{outer_i}: {meta_loss}')
        print(f'hidden.grad = {hidden.grad}')
        print(f'eta.fc.weight = {eta.fc.weight}')
        meta_opt.step() # meta-optimizer step: more or less theta^<t> := theta^<t> - meta_eta * Grad L^val( theta^{T} - eta* Grad L^train(theta^{T}) )

So sorry, but it seems I forgot to add one of the most important steps from that little guide. Please see the last step (step 7) of the newly expanded guide.

Closing the issue since this addresses the main problem. Note that you are doing things in a way which seems a bit strange. TrainableSGD should be the differentiable version of a functionally equivalent class which inherits from torch.optim.Optimizer (which you should also write). Let's call it FixedTrainableSGD for sake of argument. Once you've implemented both classes, you should register the pair with a call to higher.register_optim(FixedTrainableSGD, TrainableSGD).

Closing the issue since this addresses the main problem. Note that you are doing things in a way which seems a bit strange. TrainableSGD should be the differentiable version of a functionally equivalent class which inherits from torch.optim.Optimizer (which you should also write). Let's call it FixedTrainableSGD for sake of argument. Once you've implemented both classes, you should register the pair with a call to higher.register_optim(FixedTrainableSGD, TrainableSGD).

It looks strange because I don't actually care what the original optimizer was at all. So I created an empty one because I don't need it. I only need the Differentiable (and trainable one in this case). So that's what I implemented.
How would you have implemented it? My goal is to have a trainable step-size (where the trainable step-size is implemented by a nn and not just a torch.tensor).

Also, the step code for higher doesn't allow me to pass things to my _update...so I was forced to mutate my class to force passing stuff around. I know it's not the best or most elegant way to do this but I didn't know how else to do it if not change higher itself...

Note that you are doing things in a way which seems a bit strange. TrainableSGD should be the differentiable version of a functionally equivalent class which inherits from torch.optim.Optimizer (which you should also write)

if they are not functionally equivalent, would that be a problem? I never actually use none-differentiable version of it.


build(deps): bump github.com/cosmos/cosmos-sdk from v0.50.9 to v0.50.11

Summary by CodeRabbit

New Features

Upgraded dependencies to newer versions, potentially enhancing performance and introducing new features.

Bug Fixes

Updated dependencies may include bug fixes that improve overall stability.

mg


Looking for new Maintainer

let me know if you're interested in maintaining this repo!

not sure if he'd be up for it, but @Redmega has been pretty active 😅

Haha, I'm just waiting on @KyleAMathews to approve the PRs
This is a nifty repo and I'd be more than willing to maintain it.

haha. totally forgot about @KyleAMathews . I'm sorry. will add you as well @Redmega .thanks everyone for keeping this alive.

Would it be bad form to accept my own PRs now?

Planning on reviewing it tomorrow. Generally good to get two sets of eyes
on something.
On Sun, Feb 26, 2017, 10:42 PM Angel J Piscola<EMAIL_ADDRESS>wrote:

Would it be bad form to accept my own PRs now?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stoeffel/react-motion-drawer/issues/14#issuecomment-282640541,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEVh2GhN40aP4NZlxMvL3LJRkC4h9LFks5rgnA7gaJpZM4LN3TA
.

Yeah, that's what I was waiting for. Sounds good, I'll hold off then.


Breaks for multi-aws-account single okta application setup.

Problem Statement
Overview
The Okta setup documentation now has guidance on how to configure a single Okta application to refer to multiple downstream AWS accounts (http://saml-doc.okta.com/SAML_Docs/How-to-Configure-SAML-2.0-for-Amazon-Web-Service#scenarioB).  This breaks the cli / api assumptions made in gimme-aws-creds in 2 key areas - the role name and the identityProviderArn.  I am trying to work out how to best implement a fix.
Incorrect role name
First a quick example -
[root@fe1728b00c12 okta-src]# gimme-aws-creds -u<EMAIL_ADDRESS>Password for<EMAIL_ADDRESS>Authentication Success! Getting AWS Accountsdone

Pick an app:
[ 0 ] FOOBAR AWS Accounts
Selection: 0
Pick a role:
[ 0 ] [foobarcontractors] -- okta-xonk-admin
[ 1 ] [foobar-it] -- okta-it-admin
Selection: 0
Traceback (most recent call last):
<snip>
  File "/usr/lib/python3.4/site-packages/botocore/client.py", line 599, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the AssumeRoleWithSAML operation: Request ARN is invalid

Note that [foobarcontractors] and [foobar-it] are independent aws accounts with separate account ids.  These are not 2 roles within a single AWS account, rather they are 2 accounts with 1 saml role each.
I added a quick debug print to see what we are getting during the _get_sts_creds function -
print("Attempting STS with Role={}, Principal={}, Duration={}".format(self.role_arn,self.idp_arn,duration))

yields
Attempting STS with Role=arn:aws:iam::130894992587:role/[foobarcontractors] -- okta-xonk-admin, Principal=arn:aws:iam::123456789012:saml-provider/Okta, Duration=3600

Clearly this is broken due to the account string prefix.    This leads to the next issue -
Incorrect AWS Account ID
This one is a bit more thorny and will require some more digging.  In _get_aws_account_info we enumerate the API call - {{url}}/api/v1/apps?filter=user.id+eq+"##USERID##"&expand=user/##USERID##.  Inside the returned json, there is a single value for final_result[0]['settings']['app']['identityProviderArn'] = arn:aws:iam::123456789012:saml-provider/[Userdefinedstring].  Sadly this is AWS account id specific.  As such, even if we correctly regex sub the role-name above, the identityProviderArn will have the incorrect account id and the derived role arn account number will be incorrect and the SAML assertion will fail.  So we need to find some other way to infer aws account id.
If we just use the Okta website we can see the AWS account ids (note I had to censor the details of account numbers, hopefully doesn't affect clarity) -

However, these account ID values don't exist in any of the datastructures we have in gimme-aws-creds currently.

Possible solutions
So, for the AWS Account ids I have a range of options:
Role name hints
To prefix this, I hate this idea, but through documentation you could require that users provide the target AWS account ID somewhere within the role name string.  i.e. [foobar-it] -- okta-it-admin becomes [foobar-it] --<PHONE_NUMBER>12-okta-it-admin.  We could use this hint to locate aws account id if found.
Give up on multi-account and just document the limitation
I am actually kind of ok on this, but it is a bit unsatisfying.  It is a similar amount of work to create a new Okta application as it is to tie multiple AWS accounts under a single application.  As such we could just document the limitation and require single AWS accounts under each Okta application, with as many roles for that account as you like.  We then will have the correct identity provider in the final_result for each application.
Option X
So for this, we would need to go digging.  I don't know where to find the account number information displayed on the graphical saml page - but it has to be somewhere.  I think based on the successful auth we should be able to curl and parse the saml page to get the roles and account ids for example.
Any thoughts on the above, and relative interest in making the script work vs just documenting lack of support for multi-account setups?

In terms of Option X - quick dirty diff on gimme_aws_creds/okta.py
+import base64
+import xml.etree.ElementTree as ET

    def get_saml_response(self, url):
<snip>
         for inputtag in saml_soup.find_all('input'):
             if inputtag.get('name') == 'SAMLResponse':
                 saml_response = inputtag.get('value')
+                role_pairs = []
+                root = ET.fromstring(base64.b64decode(saml_response))
+                for saml2_attribute in root.iter('{urn:oasis:names:tc:SAML:2.0:assertion}Attribute'):
+                    if (saml2_attribute.get('Name') == 'https://aws.amazon.com/SAML/Attributes/Role'):
+                        for saml2_attribute_value in saml2_attribute.iter('{urn:oasis:names:tc:SAML:2.0:assertion}AttributeValue'):
+                            role_pairs.append(saml2_attribute_value.text)
+                print (role_pairs)

yields -
role_pairs =
['arn:aws:iam::20<SNIP>68:saml-provider/Okta,arn:aws:iam::20<SNIP>68:role/okta-<SNIP>in',
'arn:aws:iam::92<SNIP>22:saml-provider/Okta,arn:aws:iam::92<SNIP>22:role/okta-<SNIP>in',
'arn:aws:iam::13<SNIP>87:saml-provider/Okta,arn:aws:iam::13<SNIP>87:role/okta-<SNIP>in']

So we definitely have the data.  Just need to work out how to key our identity provider and role based on the saml assertion from beautiful soup and not make assumptions based on the Okta api return.

Hi @onetinov - We currently create one Okta app per AWS account, so we have not run into your particular use case but we do want to support it.  We are reviewing the possible options, thanks for your suggestions. Feel free to submit a PR if you get something working.

Ok - I took a stab at it.  Let me know what you think - https://github.com/Nike-Inc/gimme-aws-creds/pull/19

Incorporated the feedback, let me know if you think we should do more.  I believe this addresses the issue at hand, but as a new/separate topic, it now has me really wanting to eliminate the API key requirement entirely - it feels a bit icky to me to have to give my users an API key to stash in their environment and if it could just be based on their username/password I would be happier.
However, that almost feels like a real fork, although I hate to duplicate code - especially now this is working so well.  Maybe if we provided a config option to hard code the org_url and the app_url, we would no longer need the api_key enumeration of a user's SAML destinations.
Anyway, let me know what you think!

We felt the same way, and actually ended up building a server component for this that handles all the Okta API calls.  I think was on it's way to becoming open source, but I'm not sure if that's still the case.
Being able to hard-code the URLs in the config is a really good way to go about this for those not wanting to leverage the server component, great idea.  We should add that to our list.

Changes merged, thanks!  Feel free to open an issue regarding the replacement of the API key envar.


Fixed image orientation based on EXIF info when compressing image with list

Getting EXIF in such manual way is caused by ExifInterface incompatibility with API versions lower than 24 (flutter_compress_image is at version 16)
However, this manual method works fine and is much simpler:
val exifRotate = Exif.getRotationDegrees(arr)

vs
val exifInterface = ExifInterface(ByteArrayInputStream(arr))
val orientation = exifInterface.getAttributeInt(ExifInterface.TAG_ORIENTATION, 1)
var exifRotate = 0

when (orientation) {
    ExifInterface.ORIENTATION_ROTATE_90 -> exifRotate = 90
    ExifInterface.ORIENTATION_ROTATE_180 -> exifRotate = 180
    ExifInterface.ORIENTATION_ROTATE_270 -> exifRotate = 270
}

Possibly it would be also good for compressAssetImage, compressAndGetFile and compressWithFile to support this orientation fix

So this fix is only for Android.
And it's not about keeping EXIF, it's just about "correcting" the angle.

@CaiJingLong on iOS there is no such problem
And yes, it just corrects the angle

LGTM
Thanks for job.

I think a parameter should be added to let the user decide whether to turn on the "Auto Correction Angle" option, which consumes slightly more performance, especially for File.
If the user chooses not, the code to get the angle is not called.
What do you think?

In addition, if you set this option, you think it's better to set it to true or false.
True meets the general expectations, but for users who used the library in the past, they might rotate repeatedly by finding their own angles and setting rotate attributes.
If set to false, there is no breaking change.

@CaiJingLong okay, I’ll add such param on Monday


Question about LTSC

Are you going to update the LTSC version to version 5.x so that it is easier to use?  🤔
Thanks for your work 👍

Hello. Think I going to do it on Monday (without any release page) or with the 5.2 release.

Thanks a lot for the quick answer !

I always make a quick rrply when in the Moscow undeground))

@farag2   Do you have some news about the LTSC version ?
Regards :octocat:

Have not free time. Only after October 28.

Please clarify - is LTSC ONLY supported in 5.0 script and none of scripts since then work with LTSC?

Please clarify - is LTSC ONLY supported in 5.0 script and none of scripts since then work with LTSC?


Unable to recv() from connection in v10

Hi,
When using websockets 10, the following connection will never return recv(), whereas it works fine in 9.1
import logging
logger = logging.getLogger('websockets')
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())
async with websockets.connect("wss://fstream.binance.com:443/ws/stream/") as websocket:
        await websocket.send('{"method": "SUBSCRIBE", "id": 0, "params": ["btcusdt@aggTrade"]}')
        while True:
                print(await websocket.recv())

This looks very similar to #1050. Closing as a duplicate for now. Let's reopen if you find cues suggesting that it's a different case or that websockets is the culprit. I'm pretty sure the problem is on the server side though.

A 5 minutes investigation shows exactly the same symptoms, so I suspect binance and bitpanda use the same software and suffer the same bugs.