Fix code climate

Codecov Report

Merging #68 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #68   +/-   ##
=======================================
  Coverage   97.17%   97.17%
=======================================
  Files          19       19
  Lines         248      248
=======================================
  Hits          241      241
  Misses          7        7

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39f13ae...23724f9. Read the comment docs.


Automatically restart Spark and HDFS master a few times if there is an issue starting them up

This PR band-aids a couple of related issues that have plagued Flintrock for a while: Spark and HDFS sometimes get stuck on master startup, especially after a cluster restart.
The issue is somehow related to EC2's network stack, but I don't have any solid leads on the root cause. Instead, I've implemented a good-enough workaround that finally gets the full acceptance suite passing on first try. It's been a while since I've been able to do that.
TODO:

[ ] Abstract out repeated retry logic across HDFS and Spark (?)
[ ] Try removing waits added in #160

This is probably as good a solution as I'm going to find for now, so I'm marking this PR as a fix for these two issues:
Fixes #129.
Fixes #157.

@kruhly - Following on to the discussion in #129, you may want to test out this PR and see if it fixes the issues you were seeing. It seems to do so for me, though I will try to tweak things a bit more before merging this in.

Having trouble installing the dev version to test.

Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found

Can you share:

How you are installing the dev version
More of the stack trace, if any
What your environment is (OS, Python version, etc.)

Following instructions from CONTRIBUTING.md under Setup heading
but replaced virtual environment name venv -> fvenv
OS: Ubuntu 14.04
Python: 3.43

Installing collected packages: cryptography, Flintrock, idna, asn1crypto, six, cffi, pycparser
Running setup.py install for cryptography
Running command /home/rjk/odesk/flintrock/fvenv/bin/python3 -c "import setuptools, tokenize;file='/home/rjk/odesk/flintrock/fvenv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-8qfmnyqj-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/rjk/odesk/flintrock/fvenv/include/site/python3.4
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^
compilation terminated.
Traceback (most recent call last):
File "/usr/lib/python3.4/distutils/unixccompiler.py", line 116, in _compile
extra_postargs)
File "/usr/lib/python3.4/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/python3.4/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python3.4/distutils/spawn.py", line 162, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/rjk/odesk/flintrock/fvenv/lib/python3.4/site-packages/setuptools/command/bdist_egg.py", line 161, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/home/rjk/odesk/flintrock/fvenv/lib/python3.4/site-packages/setuptools/command/bdist_egg.py", line 147, in call_command
    self.run_command(cmdname)
  File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/rjk/odesk/flintrock/fvenv/lib/python3.4/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/usr/lib/python3.4/distutils/command/install_lib.py", line 109, in build
    self.run_command('build_ext')
  File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/rjk/odesk/flintrock/fvenv/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 75, in run
    _build_ext.run(self)
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
    self.build_extension(ext)
  File "/home/rjk/odesk/flintrock/fvenv/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/lib/python3.4/distutils/command/build_ext.py", line 503, in build_extension
    depends=ext.depends)
  File "/usr/lib/python3.4/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib/python3.4/distutils/unixccompiler.py", line 118, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

According to Stack Overflow, you need to install libffi-dev, and perhaps some related packages too.

Installing libffi-dev fixed the problem. I will do some testing tomorrow.
Thanks
On Mon, Jul 24, 2017 at 10:50 PM, Nicholas Chammas<EMAIL_ADDRESS>
wrote:

According to Stack Overflow
https://stackoverflow.com/questions/38109637/package-libffi-was-not-found-in-the-pkg-config-search-path-redhat6-5,
you need to install libffi-dev, and perhaps some related packages too.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nchammas/flintrock/pull/204#issuecomment-317412506,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADi6waBAMzV6rkZOu95ISohpGtixCvglks5sRJMZgaJpZM4Ogv2g
.

--
Robert Kruhlak
Geelong, VIC
Australia
(M) +61 (0)481 091 244
(E<EMAIL_ADDRESS>

I did 5 launches  with 1 master, 1 worker and  t2.micro with short time intervals in between today.
2 launches finished ~ 2.3 minutes with health report of 0 workers, logs showed successful configuration and  registered worker. Spark-ui showed registered worker
1 launch finished ~8 minutes. Health report showed 1 worker as expected. Looks like it recovered as expected.
1 launch stuck on configuring master. Timed out waiting for spark master to come up and asks to terminate cluster. AWS console shows master and worker running. Two spark log files  .out and .out.1. .out.1 log file shows binding exception. .out log file shows successful configuration of master and register of worker.
1 launch timed out waiting for spark master to come up and asks to terminate cluster. AWS console shows master and worker running. Three log files .out, .out.1, .out.2 all have the binding exception. spark-ui can't be reached. Manual call to start-all.sh, starts master and returns that worker is running and to stop first. spark-ui comes up with master only then serveral minutes later the worker appears.
Hope this helps.

2 launches finished ~ 2.3 minutes with health report of 0 workers

Yeah, sometimes the health report runs before all the workers are fully registered. It's an annoying issue but relatively harmless.

1 launch finished ~8 minutes. Health report showed 1 worker as expected. Looks like it recovered as expected.

So Flintrock showed the "Trying again..." message a few times followed by success? That's good.

1 launch stuck on configuring master. Timed out waiting for spark master to come up and asks to terminate cluster. AWS console shows master and worker running. Two spark log files .out and .out.1. .out.1 log file shows binding exception. .out log file shows successful configuration of master and register of worker.
1 launch timed out waiting for spark master to come up and asks to terminate cluster. AWS console shows master and worker running. Three log files .out, .out.1, .out.2 all have the binding exception. spark-ui can't be reached. Manual call to start-all.sh, starts master and returns that worker is running and to stop first. spark-ui comes up with master only then serveral minutes later the worker appears.

That's a bummer. This can definitely be addressed by waiting longer or trying more times, but at this point it feels like a very bad problem. And you're seeing this on launches, not restarts. My issues have typically been with restarts and only rarely with launches.
What region are you working in?
Are all the destroys/launches targeted at a cluster with the same name? What about if you try altering the name every time? I wonder if this problem is related to some kind of lease expiration/reacquisition on named resources in AWS.
From your sample of 5 launches you seem to be experiencing the problem much worse than even I am. :(

Reply in text
So Flintrock showed the "Trying again..." message a few times followed by

success? That's good.

I did not think to note the details as it was successful.

1 launch stuck on configuring master. Timed out waiting for spark master
to come up and asks to terminate cluster. AWS console shows master and
worker running. Two spark log files .out and .out.1. .out.1 log file shows
binding exception. .out log file shows successful configuration of master
and register of worker.
1 launch timed out waiting for spark master to come up and asks to
terminate cluster. AWS console shows master and worker running. Three log
files .out, .out.1, .out.2 all have the binding exception. spark-ui can't
be reached. Manual call to start-all.sh, starts master and returns that
worker is running and to stop first. spark-ui comes up with master only
then serveral minutes later the worker appears.
That's a bummer. This can definitely be addressed by waiting longer or
trying more times, but at this point it feels like a very bad problem. And
you're seeing this on launches, not restarts. My issues have typically been
with restarts and only rarely with launches.

Seeing it on launch and restarts but tested the PR using launches because
we are currently using launch in our pipeline.

What region are you working in?

us-east-1

Are all the destroys/launches targeted at a cluster with the same name?
What about if you try altering the name every time? I wonder if this
problem is related to some kind of lease expiration/reacquisition on named
resources in AWS.
Launches are with different names each time but the cluster parameters are
the same which I don't think rules out the reuse of AWS resources.

From your sample of 5 launches you seem to be experiencing the problem much

worse than even I am. :(

The test was not a typical workflow because our batches vary in size and
arrival time which changes the cluster parameters and time between
launches. Fortunately, we don't see this poor of performance normally.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nchammas/flintrock/pull/204#issuecomment-317775183,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADi6wTWI8tlVKslk6qUXRJBVCsHnp7j-ks5sRgotgaJpZM4Ogv2g
.

--
Robert Kruhlak
Geelong, VIC
Australia
(M) +61 (0)481 091 244
(E<EMAIL_ADDRESS>

@kruhly - I upped the number of attempts Flintrock will make to start the Spark and HDFS masters. Could you repeat your test of this branch and let me know how it goes?
I'm curious as to how bad the problem is for you, and whether with enough attempts things work out in the end. If they do, perhaps we should add an option for cases like these where the user is okay with waiting a while for Flintrock to retry starting the masters. But it's nonetheless maddening that this happens and I don't really have good leads as to why...
I have one additional thing I may try (and I'll ask you to repeat your test again, if that's OK 🙏), but I'll wait to see what you find first regarding the additional attempts.

Will do.

Ran 5 tests using the same settings as the previous tests without an issue after pulling the changes :-)
2 launches finished around 2 minutes with no signs of trying again
3 launches finished around 3:30 minutes with no signs of trying again
I will try another round tomorrow.

Huh... I'm baffled as to why the latest changes would have any impact then, since I just upped the number of retries and it sounds like you're not needing to use them. Maybe the bad run from a couple of days ago was a fluke?

I've set the attempt limit back down to where I'd ideally want to keep it. If this branch still works for you (which it seems like it should), then I think we have a good-enough solution for now. Let me know, and I'll make some final cleanups and merge this in.
On my side, I'm able to go through the full acceptance test suite without any errors, which is good.

Maybe the phase of the moon or pressure on AWS resources?
I ran 6 more launches with the same configuration as before. All successful.
3 launches first attempt
3 launches second attempt
I'm fine with the lower attempts. If AWS is having a bad day, we will just have to take a break ... Thanks

That's great news. I'll clean up this PR a bit and merge it in. Thanks for testing it out @kruhly! Your feedback was helpful in confirming that I've addressed the issue.


Error with start bridge

when execute  node scripts/cli.js registerSlashCommands :
node:internal/modules/cjs/loader:1093
  throw err;
  ^

Error: Cannot find module '/root/bolt/scripts/cli.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1090:15)
    at Module._load (node:internal/modules/cjs/loader:934:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v19.9.0

Hey there @raymand211092, are you running the script while inside the bolt directory?

Yes...i followed the steps:
root@srv112118-206152:~# cd bolt/
root@srv112118-206152:~/bolt# ls
app  changelog.md  docker-compose.yml  docs  legalese.md  LICENSE  node_modules  package.json  package-lock.json  README.md
root@srv112118-206152:~/bolt# node scripts/cli.js registerSlashCommands
node:internal/modules/cjs/loader:1093
throw err;
^
Error: Cannot find module '/root/bolt/scripts/cli.js'
at Module._resolveFilename (node:internal/modules/cjs/loader:1090:15)
at Module._load (node:internal/modules/cjs/loader:934:27)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
at node:internal/main/run_main_module:23:47 {
code: 'MODULE_NOT_FOUND',
requireStack: []
}
Node.js v19.9.0
root@srv112118-206152:~/bolt#

@raymand211092 I don't think I can repro this locally, could you try to run Bolt using docker-compose and check to see if you still face the issue then?


[docs] Svelte tutorial is not searchable

Describe the bug
Hi teams
ctrl/command + f or just type not works for select/options at https://svelte.dev/tutorial/
It's not a short list. I feel it's hard to find a specific section.

Can team think about make select/options a plain list, like Go or just like https://svelte.dev/docs
https://go.dev/tour/welcome/1

Reproduction
see https://svelte.dev/tutorial/
Logs
No response
System Info
Browsers:
    Chrome: 100.0.4896.127
    Safari: 14.1.2

Severity
annoyance

it is now searchable


Set default value for monitor healthcheck

Checklist

[x] The PR has a meaningful title. It will be used to auto-generate the
changelog.
The PR has a meaningful description that sums up the change. It will be
linked in the changelog.
[x] PR contains a single logical change (to build a better changelog).
[ ] Update the documentation.
[x] Categorize the PR by adding one of the labels:
bug, enhancement, documentation, change, breaking, dependency
as they show up in the changelog.
[ ] Link this PR to related issues or PRs.

Thinking about setting the other healthCheck parameters as well 🤔


Feature/deprecate binary cross entropy loss

This PR is opened to resolve an issue found by @tomaarsen. Thanks @tomaarsen and nice PR #187, I use it as the template!
Pull request overview
Major changes

Deprecate torch.nn.BCELoss for binary classification and replace it by torch.nn.CrossEntropyLoss as 2-class classification.
Update docstring
Update test_modeling.py

Minor changes

Add eps for numerical stability when scaling logins with temperature

Details
Problem
As @tomaarsen mentioned here:

As it turns out, fix 2 is a bit tricky due to the different requirements of the different loss functions. The input (i.e. the prediction) is always float32, so that's all sorted. Then, the BCELoss wants the target (i.e. the labels) to be the same dtype as the input, so also float32. This is why I've opted to make fix 2.
However, the other loss used by the SetFitHead, CrossEntropyLoss, wants the target to be float64 whenever the target has a different shape than the input.

BCELoss and CrossEntropyLoss require different data types respectively, which increases complexity to cast the labels into correct data type and makes the code harder to read.
Solution
Therefore, this PR deprecates BCELoss and use CrossEntropyLoss for binary classification.
Result
I run some experiments on CR and EnronSpam datasets to check whether the performance is changed.
Here is the results with N=8 as Table 2 in the paper:

Head
CR
EnronSpam
Amazon-CF

sklearn (from paper)
88.5 (1.9)
90.1 (3.4)
40.3 (11.8)

pytorch (BCE)
88.8 (1.3)
90.0 (3.3)
76.4 (11.2)

pytorch (CrossEntropy)
88.8 (1.4)
90.3 (3.2)
73.1 (10.5)

(Hyperparamters: batch size = 16, L2 weight (weight decay) = 0, head learning rate = 1e-2, keep body frozen)
From the results, their performance are similar except Amazon-CF. It looked strange to me and after several runs, I still got similar results at the end. Here is the notebook I used to run experiments for pytorch (CrossEntropy).
Conclusion
The advantages of this change (copied from here):

Solve the issue about different data types for different loss functions.
Won't forget to test either in the future when other features add in.
Make #179 easier.

Still need to validate whether the results of Amazon-CF is correct. @lewtun could you have a check on the notebook to validate the experiments? Thanks!

Thanks for you feedbacks, @tomaarsen! That really helps validate the experiments!
I checked the test dataset of Amazon-CF and it matched your guess. The proportion is 10% (503 samples) for 1 vs 90% (5497 samples) for 0. Now I'm thinking why in this PR I didn't get similar results as what we see now. Maybe something is changed? Will look into it.

@blakechi I had gotten a hunch about the results and the class imbalance. No reputable researchers would evaluate a classification task with a 90:10 class inbalance using accuracy, and most certainly no reasonable data scientist would then only score a ~43% on the task, hah!
The discrepancy is shown here:
https://github.com/huggingface/setfit/blob/fa1021d2355f0cb3a2c85732ee7ffe44b0cef0d1/src/setfit/utils.py#L27-L34
In the paper, amazon_counterfactual_en is evaluated using the matthews correlation coefficient, rather than accuracy. See Table 6, appendix A for the details.
However, in the script, if you supply just a list of datasets, each will be evaluated using just accuracy. Your earlier test used the test dataset from the previous snippet, causing the AmazonCF task to be evaluated using MCC.

I've re-ran the experiments for the LogisticRegression head and the differentiable SetFitHead with these commands:
python scripts/setfit/run_fewshot.py --sample_sizes=8 --lr=0.01 --keep_body_frozen --datasets amazon_counterfactual_en --batch_size 4

python scripts/setfit/run_fewshot.py --sample_sizes=8 --classifier=pytorch --lr=0.01 --keep_body_frozen --datasets amazon_counterfactual_en --batch_size 4

with the following change to scripts/setfit/run_fewshot.py:
diff --git a/scripts/setfit/run_fewshot.py b/scripts/setfit/run_fewshot.py
index 088e25d..ae03992 100644
--- a/scripts/setfit/run_fewshot.py
+++ b/scripts/setfit/run_fewshot.py
@@ -91,7 +91,7 @@ def main():
     elif args.is_test_set:
         dataset_to_metric = TEST_DATASET_TO_METRIC
     else:
-        dataset_to_metric = {dataset: "accuracy" for dataset in args.datasets}
+        dataset_to_metric = {dataset: "matthews_correlation" for dataset in args.datasets}

     # Configure loss function
     loss_class = LOSS_NAME_TO_CLASS[args.loss]

This has resulted in these outputs:

Head
Amazon CF (MCC)

LogisticRegression
44.4 (8.6)

SetFitHead w. CrossEntropyLoss
44.6 (16.9)

This is much more along the line of what we would expect. With other words, this PR seems to work like intended, without any fun additional surprises.

Tom Aarsen

Great finding! Ya, that explains why the numbers are so different. Appreciated for the experiments as well!

Put this PR on hold since #207 might make some changes that overlaps.


turn off the flaker nagger

Temporary disable the code that nags owners of test flakes

This change is

/lgtm

/lgtm


switch to 3.3V scale on ADC inputs

All inputs switched to 3.3V scale.  Resistor values modified to maintain filter values and suitable impedance for ADC sampling

This circuit will do as is, but changing the series R to 1k will give you more voltage range (0-3V instead of 0-2.5V) and a slightly weaker cutoff (-1.2 dB @ 100 Hz vs originally -1.7 dB @ 100 Hz).
If you want the stronger cutoff bump up Cout to 1.8u or 2.2u.

Unrelated - these sensors can only be powered by 5V.

Also keep in mind that BOM changes can be made after the board goes to fab, so we can tune these values once these are on our desks.  If there's a component option that we should add pads for, let me know.

This circuit will do as is, but changing the series R to 1k will give you more voltage range (0-3V instead of 0-2.5V) and a slightly weaker cutoff (-1.2 dB @ 100 Hz vs originally -1.7 dB @ 100 Hz).
If you want the stronger cutoff bump up Cout to 1.8u or 2.2u.

That's a typo in the part selection, the lower resistor should be 3K.  Good catch.

corrected/implemented

@inceptionev  can we close this issue? Looks like you resolved it.


New Node Deploy: Error creating machine: Error installing Docker: , requeuing

Rancher Server Setup

Rancher version: 2.5.9
Installation option (Docker install/Helm Chart): Helm Chart

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE1, v1.20.9

Proxy/Cert Details: Self Signed

Information about the Cluster

Kubernetes version: v1.20.11
Cluster Type (Local/Downstream): Downstream

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Infrastructure Provider (vSphere DataCenter Template Deploy)

Describe the bug
Trying to add nodes to an existing cluster. it always fails with installing Docker.
The error message does not give a lot of context and details: (I've tried DEBUG logging, but haven't found much - INFO logging below)
2021/09/27 04:41:19 [ERROR] error syncing 'c-zt6mn/m-nnq5w': handler node-controller: Error creating machine: Error installing Docker: , requeuing
To Reproduce

Create a node Template with vSphere, using Ubuntu 20.04 CloudImage OVA, Converted to Template
Leave cloud-init blank or use a cloud-init script, it does not matter
We have tried a very basic template - everything default.
Add a node to an existing cluster, or create a new cluster with the Template

Result
Expected Result
The node to be deployed successfully
Screenshots
N/A
Additional context
It installs Docker from https://releases.rancher.com/install-docker/20.10.sh on Ubuntu 20.04 Cloud Image (used an older one that worked before, and also tried the latest Daily from yesterday)
None of the existing node pools work anymore to deploy images, they used to work fine. I think it started after the 2.5.9 update.
I can SSH into the deployed node when the error displays and before the node gets destroyed. I can access Docker e.g. with "docker ps", and I can access it with "docker version", which tells me 20.10.7 has been installed.
After a few minutes it kills the node and tries again, failing all again.

2021/09/27 04:35:33 [INFO] [node-controller-rancher-machine] Creating machine...
2021/09/27 04:35:33 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Generating SSH Keypair...
2021/09/27 04:35:34 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Cloning VM from VM or Template: /EU-CLOUD/vm/_TEMPLATES/ubuntu-20.04-server-cloudimg-amd64...
2021/09/27 04:35:34 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Finding datastore /EU-CLOUD/datastore/eu-esx-01-nv3
2021/09/27 04:35:37 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Adding network: /EU-CLOUD/network/EU-CLOUD-ESX-01
2021/09/27 04:35:37 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Resizing disk disk-1000-0 up from 10485760Kb to 102400000Kb
2021/09/27 04:35:37 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Setting disk.enableUUID to TRUE
2021/09/27 04:35:37 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Creating cloud-init.iso
2021/09/27 04:35:37 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Uploading cloud-init.iso
2021/09/27 04:35:38 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Mounting cloudinit user-data.iso
2021/09/27 04:35:39 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Waiting for VMware Tools to come online...
2021/09/27 04:36:19 [INFO] [node-controller-rancher-machine] Waiting for machine to be running, this may take a few minutes...
2021/09/27 04:36:19 [INFO] [node-controller-rancher-machine] Detecting operating system of created instance...
2021/09/27 04:36:19 [INFO] [node-controller-rancher-machine] Waiting for SSH to be available...
2021/09/27 04:36:20 [INFO] [node-controller-rancher-machine] Detecting the provisioner...
2021/09/27 04:36:21 [INFO] [node-controller-rancher-machine] Provisioning with ubuntu(systemd)...
2021/09/27 04:36:32 [INFO] [node-controller-rancher-machine] Installing Docker from: https://releases.rancher.com/install-docker/20.10.sh
2021/09/27 04:41:18 [INFO] [node-controller-rancher-machine] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2021/09/27 04:41:18 [INFO] [node-controller-rancher-machine]
2021/09/27 04:41:19 [INFO] Generating and uploading node config eu-k8s-staging-1-w7-1
2021/09/27 04:41:19 [ERROR] error syncing 'c-zt6mn/m-nnq5w': handler node-controller: Error creating machine: Error installing Docker: , requeuing
2021/09/27 04:42:09 [INFO] [nodepool] deleting np-r5nwg
2021/09/27 04:42:10 [INFO] Removing node eu-k8s-staging-1-w7-1
2021/09/27 04:42:10 [INFO] [node-controller-rancher-machine] About to remove eu-k8s-staging-1-w7-1
2021/09/27 04:42:10 [INFO] [node-controller-rancher-machine] WARNING: This action will delete both local reference and remote instance.
2021/09/27 04:42:12 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Removing cloud-init.iso
2021/09/27 04:42:12 [INFO] [node-controller-rancher-machine] Successfully removed eu-k8s-staging-1-w7-1
2021/09/27 04:42:12 [INFO] [node-controller-rancher-machine] (eu-k8s-staging-1-w7-1) Closing plugin on server side
2021/09/27 04:42:12 [INFO] Removing node eu-k8s-staging-1-w7-1 done

I also managed to snag some more logs when using TRACE most state INFO level however - weird):
2021/09/27 05:24:03 [DEBUG] Wrote ping
2021/09/27 05:24:03 [DEBUG] Wrote ping
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] SSH cmd err, output: exit status 255: docker: not found
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] # Executing docker install script, commit: 7cae5f8b0decc17d6571f9f52eb840fbc13b2737
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c apt-get update -qq >/dev/null
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] Warning: apt-key output should not be parsed (stdout is not a terminal)
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /etc/apt/sources.list.d/docker.list
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c apt-get update -qq >/dev/null
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] INFO: Searching repository for VERSION '20.10.7'
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] INFO: apt-cache madison 'docker-ce' | grep '20.10.7.*-0~ubuntu' | head -1 | awk '{$1=$1};1' | cut -d' ' -f 3
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + [ -n 5:20.10.7~3-0~ubuntu-focal ]
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c apt-get install -y -qq --no-install-recommends docker-ce-cli=5:20.10.7~3-0~ubuntu-focal >/dev/null
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: unable to initialize frontend: Dialog
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: falling back to frontend: Readline
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: unable to initialize frontend: Readline
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: (This frontend requires a controlling tty.)
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: falling back to frontend: Teletype
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] dpkg-preconfigure: unable to re-open stdin:
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] + sudo -E sh -c apt-get install -y -qq --no-install-recommends docker-ce=5:20.10.7~3-0~ubuntu-focal >/dev/null
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: unable to initialize frontend: Dialog
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: falling back to frontend: Readline
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: unable to initialize frontend: Readline
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: (This frontend requires a controlling tty.)
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] debconf: falling back to frontend: Teletype
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine] dpkg-preconfigure: unable to re-open stdin:
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:07 [INFO] [node-controller-rancher-machine]
2021/09/27 05:24:07 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:08 [DEBUG] Wrote ping
2021/09/27 05:24:08 [DEBUG] Wrote ping
2021/09/27 05:24:09 [INFO] [node-controller-rancher-machine] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2021/09/27 05:24:09 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:09 [INFO] [node-controller-rancher-machine]
2021/09/27 05:24:09 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:11 [INFO] Generating and uploading node config eu-k8s-staging-1-m1-2
2021/09/27 05:24:11 [DEBUG] getNodeTemplate parsed [cattle-global-nt:nt-s67sj] to ns: [cattle-global-nt] and n: [nt-s67sj]
2021/09/27 05:24:11 [TRACE] REST GET apis/management.cattle.io/v3/cattle-global-nt/nodetemplates/nt-s67sj
2021/09/27 05:24:11 [DEBUG] Cleaning up [/opt/jail/c-zt6mn/management-state/node/nodes/eu-k8s-staging-1-m1-2]
2021/09/27 05:24:11 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController calling handler node-controller-sync c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController calling handler rke-worker-upgrader c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [ERROR] error syncing 'c-zt6mn/m-j8q9t': handler node-controller: notifying bugsnag: [Error creating machine: Error installing Docker: ], requeuing
2021/09/27 05:24:11 [TRACE] NodeController calling handler cluster-provisioner-controller c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController calling handler cluster-stats c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController calling handler nodepool-provisioner c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController calling handler node-controller c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController c-zt6mn/m-j8q9t completed with dropped err: Error
2021/09/27 05:24:11 [TRACE] NodeController calling handler node-controller-sync c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodeController calling handler rke-worker-upgrader c-zt6mn/m-j8q9t
2021/09/27 05:24:11 [TRACE] NodePoolController calling handler nodepool-provisioner c-zt6mn/np-z8x8b
2021/09/27 05:24:11 [TRACE] ClusterController calling handler user-controllers-controller c-zt6mn
2021/09/27 05:24:11 [TRACE] ClusterController calling handler mgmt-cluster-rbac-delete c-zt6mn
2021/09/27 05:24:11 [DEBUG] [nodepool] bad node found: m-j8q9t
2021/09/27 05:24:11 [TRACE] REST GET apis/management.cattle.io/v3/c-zt6mn/nodes/m-j8q9t
2021/09/27 05:24:11 [TRACE] ClusterController calling handler cluster-agent-controller-cleanup c-zt6mn
2021/09/27 05:24:11 [TRACE] ClusterController calling handler certificate-expiration c-zt6mn

It looks like the error was misleading. Docker was installed correctly, there was a network issue (wrong MTU) which prevented proper communication from the deployed VM to rancher, while it seems SSH was working, other things were not.


[Bug] fixed issue 2052

[Bug] fixed issue 2052
What changes were proposed in this pull request
Issue Number: close #2052

Brief change log

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): no

cc @wangsizhu0504  PTAL thanks


Update default version to latest LTS

@PierreR Could you please also update the test classes to check on the latest LTS?

My original intend was to make two separate PR. I hope it is alright as it is.

@PierreR Fine by me if everything is in the same PR.

Thanks for your contribution.


Ztan4

Update the User class to database version:

add commit() to update user data on database
add fetch() to fetch user data from database

Modified sign-up activity and add a simple user info page for testing purpose
Known issue:

Sometime the User.fetch() may fail to fetch data from database. Based on some testing, I doubt that it is the threading that is causing this problem, so I added a sleep() inside fetch trying to improve it. I am still looking for better solution but it seems a bit difficult. Contact me for more details about this bug, I am more than glad if somebody can help me solve this out.


New prop (showCaret) proposal for the tooltip component.

Thanks for putting together this brilliant component set. I would like to propose a new prop on Tooltip component - showCaret. It's a boolean prop which will show caret along with tooltip for better UI and ease of use to anyone.
Preview

Ran locally lint/test which passed for me. Not sure what travis is trying.

ok fixed the ts issue which Travis CI was complaining.


Updated readme to streamline instruction, changed RPI setup script to not add unnecessary deps, and fixed script pre-flight checks.

setup_host_rpi.sh

General Updates
  Header Updates:
      Added script metadata including script name, author, date, and description.
      Enhanced clarity regarding the purpose of the script (specific to Raspberry Pi with Spotify CarThing).

  Comment Improvements:
      Standardized and improved comments for better readability.
      Reformatted usage notes for functions.

  Enhanced Error Handling:
      Introduced checks for Raspbian Bullseye compatibility.
      Enhanced network interface detection with fallback mechanisms.

  Dynamic Network Interface Handling:
      Added support for dynamically identifying and selecting network adapters if multiple NICs are detected.
      Improved logic for setting the WAN_INTERFACE.

Feature Additions
  Developer Mode:
      Added an interactive prompt to enable developer mode for remote access.
      Introduced port forwarding (e.g., SSH, VNC, debugging) if developer mode is enabled.

  Dependency Installation:
      Automatically installs required dependencies (iptables, iptables-persistent).

  System Updates:
      Added a system update process to ensure packages are up-to-date.

  Dev Mode Configuration:
      Conditionally forwards ports (2022, 5900, 9222) based on developer mode status.

Improved IP Tables Handling
  Rule Management:
      Added messages to indicate when rules are being cleared, rewritten, or persisted.
      Consolidated NAT masquerading rules for enhanced clarity.

  Dynamic Configurations:
      Forwarding rules now dynamically apply to the selected network adapter.

USB and Network Configuration
  Enhanced USB Detection:
      Updated logic to check for an inactive usb0 network interface using a specific IP pattern (192.168.1).

  Custom Configurations:
      Refined USB network settings and ensured they are saved in interfaces.d.

  Superbird Hostname Addition:
      Automatically appends superbird to /etc/hosts if not already present.

Logging and User Interaction
  Progress Indicators:
      Added clear console messages for each major step to improve user feedback.
  Interactive Choices:
      Prompted user interaction for network interface selection and developer mode setup.

Minor Changes
  Standardized function naming and descriptions.
  Minor adjustments to ensure compatibility with modern Raspberry Pi configurations.

README.md

Streamlined Instructions:
  Consolidated detailed platform-specific setup steps into simplified instructions for flashing the CarThing.

Introduction of Terbium:
  Added Terbium as the recommended web-based tool for flashing the CarThing, reducing reliance on the command-line superbird-tool.

Revised "How to Use" Section:
  Simplified prerequisites for running Nocturne on CarThing.
  Expanded supported host devices to include Raspberry Pi, Windows, Mac, and Linux.

Flashing Instructions Reorganized:
  Grouped flashing methods into collapsible sections with a preference for the Terbium method.
  Included a brief mention of using superbird-tool for advanced users.

Updated Raspberry Pi Setup:
  Streamlined instructions for setting up Raspberry Pi with a focus on the latest tools and configuration steps.

Platform-Specific Notes:
  Marked MacOS and Linux as "Not Recommended" with placeholders for future details.
  Provided a basic unsupported method for Windows, warning users about potential issues.

Enhanced UI Setup Instructions:
  Directed users to follow UI setup steps in the linked Nocturne UI repository.

Miscellaneous:
  Removed verbose, redundant instructions to improve readability.
  Updated links and clarified paths for downloading and setting up files.

Check out the latest commits. It addresses a lot of the issues and nitpicks you had. I left a lot of the comments however. My general rule of thumb is more comments is better than none, especially if you're wanting to allow the end user to understand what the script does without them having to be a pro at bash/shell scripting.

My general rule of thumb is more comments is better than none

My rule of thumb is that redundant comments are worse than none, especially when they can later become out of sync with the code and misleading. For example, simply saying "VARIABLES" above a section of several variables doesn't really add any new information that isn't already obvious just from looking at it. Likewise, putting "update system" immediately above apt update, or "Allow IP forwarding" immediately before an echo command saying "Enabling IP forwarding" doesn't really add any information that isn't obvious from the script's text.
It might not be a big deal but I think it's important to keep comments clean. If you have too many comments that feel useless, your eyes start to gloss over them. This is a problem because comments should always command attention.
I will re-review in a little bit, unless you are still actively working on something.


how to debug in gui

Use case
i download the latest jmeter source，then how to debug the gui？
Possible solution
No response
Possible workarounds
No response
JMeter Version
5.6.3
Java Version
17
OS Version
windows

Install IntelliJ IDEA, and use run with debugger.


Added duplicateOffscreen directive

Context
US115498
Design Document
Originally named LongLoadDirective, duplicateOffscreen takes a content node and duplicates it offscreen, invisible to screen readers. The duplicated item sits just after the original content node in the DOM.
This directive will be used to assist with copying elements that take a long time to load, such as course images. It is one of the requirements for Drag N Drop lists.

Quality

the directives folder has been added to the karma configuration file
a unit test that tests some basics for the directive was added
Manual browser testing with VoiceOver

[x] Chrome
[x] Legacy Edge
[ ] CrEdge
[x] Safari
[x] Firefox

An alternative solution was found by @m6jones here: https://codepen.io/m6jones/pen/LYGRxzw?editors=0010


Error in "predictions = multi_atlas_vae_model.predict_labels(return_pandas=True) "

I am running the across atlas integration of my own data and secceed in initializing a merge_data scAtlasVAE model.
after training and getting latent embedding, I would like to predict labels,
but I meet a ERROR attached below:

IndexError                                Traceback (most recent call last)
Cell In[97], line 2
1 # 7. 获取预测结果并存储到 adata.uns 和 adata.obs 中
----> 2 predictions = multi_atlas_vae_model.predict_labels(return_pandas=True)
3 predictions_logits = multi_atlas_vae_model.predict_labels(return_pandas=False)
File ~/mambaforge/envs/scatlasvae/lib/python3.8/site-packages/torch/autograd/grad_mode.py:27, in _DecoratorContextManager.call..decorate_context(*args, **kwargs)
24 @functools.wraps(func)
25 def decorate_context(*args, **kwargs):
26     with self.clone():
---> 27         return func(*args, **kwargs)
File ~/mambaforge/envs/scatlasvae/lib/python3.8/site-packages/scatlasvae/model/_gex_model.py:1507, in scAtlasVAE.predict_labels(self, n_per_batch, return_pandas, show_progress)
1505     X, _, batch_index, _, _, _, _ = self._prepare_batch(batch_indices)
1506 else:
-> 1507     X, _, batch_index, _, _, _, _ = future.result()
1509 if e+1 < len(X):
1510     future = executor.submit(self._prepare_batch, X[e+1])
File ~/mambaforge/envs/scatlasvae/lib/python3.8/concurrent/futures/_base.py:437, in Future.result(self, timeout)
435     raise CancelledError()
436 elif self._state == FINISHED:
--> 437     return self.__get_result()
439 self._condition.wait(timeout)
441 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
File ~/mambaforge/envs/scatlasvae/lib/python3.8/concurrent/futures/_base.py:389, in Future.__get_result(self)
387 if self._exception:
388     try:
--> 389         raise self._exception
390     finally:
391         # Break a reference cycle with the exception in self._exception
392         self = None
File ~/mambaforge/envs/scatlasvae/lib/python3.8/concurrent/futures/thread.py:57, in _WorkItem.run(self)
54     return
56 try:
---> 57     result = self.fn(*self.args, **self.kwargs)
58 except BaseException as exc:
59     self.future.set_exception(exc)
File ~/mambaforge/envs/scatlasvae/lib/python3.8/site-packages/scatlasvae/model/_gex_model.py:1799, in scAtlasVAE._prepare_batch(self, batch_indices)
1797 def _prepare_batch(self, batch_indices):
1798     P = None
-> 1799     batch_data = self._dataset[batch_indices.cpu().numpy().astype(int)]
1800     batch_index, label_index, additional_label_index, additional_batch_index = None, None, None, None
1802     if self.low_memory_initialization:
IndexError: index 142053 is out of bounds for axis 0 with size 22387
could you tell me which part of the code should I re-debug?
millions of thanks!

Could you print the version of your current installation via import scatlasvae; print(scatlasvae.__version__)
And use to upgrade to the latest version
pip install scatlasvae==<IP_ADDRESS> or pip install –upgrade scatlasvae

my version has been to <IP_ADDRESS>.
import scatlasvae
print(scatlasvae.version)
<IP_ADDRESS>

Please try pip install –upgrade scatlasvae to upgrade to <IP_ADDRESS> and try to run predict_labels. This should solve the problem.  Sorry for bugs in the previous version

thanks. it works

Please try pip install –upgrade scatlasvae to upgrade to <IP_ADDRESS> and try to run predict_labels. This should solve the problem. Sorry for bugs in the previous version

thanks. it works


Allow users to specify MoveOptions to FileIO

Allow users to specify strategy to employ when destination file exists.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @chamikaramj

Hi Alexey, Pranav sent a design doc: https://lists.apache.org/thread/543590mg41rytt8r3hk6nj4nq7n40y1j
So this PR is on hold.

@chamikaramj Thanks for letting me know! Then it would be helpful to keep it as a draft PR or just add [WIP] suffix to title.


GAN example guide.

Hello,
I'm interested in example/gan but the only script I found was a dcgan.py.
Is there anyone could bring a README for this example? Several hard codes are found in the dcgan.py script and I found it hard to run with.
Thanks very much.

That sample has already been simple and clear as far as I am concerned...
If you could point out where you feel confused maybe I could help~

As I was trying to simply run the script and hope to see something, I got an error like below
Traceback (most recent call last):
  File "dcgan.py", line 152, in <module>
    X_train, X_test = get_mnist()
  File "dcgan.py", line 56, in get_mnist
    mnist = fetch_mldata('MNIST original')
  File "/usr/lib/python2.7/dist-packages/sklearn/datasets/mldata.py", line 157, in fetch_mldata
    matlab_dict = io.loadmat(matlab_file, struct_as_record=True)
  File "/usr/lib/python2.7/dist-packages/scipy/io/matlab/mio.py", line 126, in loadmat
    matfile_dict = MR.get_variables(variable_names)
  File "/usr/lib/python2.7/dist-packages/scipy/io/matlab/mio5.py", line 288, in get_variables
    res = self.read_var_array(hdr, process)
  File "/usr/lib/python2.7/dist-packages/scipy/io/matlab/mio5.py", line 248, in read_var_array
    return self._matrix_reader.array_from_header(header, process)
  File "mio5_utils.pyx", line 616, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:6021)
  File "mio5_utils.pyx", line 645, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy/io/matlab/mio5_utils.c:5448)
  File "mio5_utils.pyx", line 713, in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex (scipy/io/matlab/mio5_utils.c:6445)
  File "mio5_utils.pyx", line 417, in scipy.io.matlab.mio5_utils.VarReader5.read_numeric (scipy/io/matlab/mio5_utils.c:3949)
  File "mio5_utils.pyx", line 353, in scipy.io.matlab.mio5_utils.VarReader5.read_element (scipy/io/matlab/mio5_utils.c:3657)
  File "streams.pyx", line 324, in scipy.io.matlab.streams.FileStream.read_string (scipy/io/matlab/streams.c:4453)
IOError: could not read bytes

It seems like getting data is not correct.
Fortunately there are some useful codes in image-classification/train_mnist.py and I have to change some codes in dcgan,py so as to read mnist correctly. It's working fine now and I could see amazing visualization. Thanks a lot.


Stop syncing main to master

I think it's been long enough now that I can drop the code that syncs the main branch to master.
I originally added this for people who might be using datasette publish ... --branch master - which might only have been me anyway!

Here's the issue where I started doing this:

#849


Implement  num_traits::identities::{One,Zero}

I see that One and Zero are implemented, but it appears that this is not done so in a way that satisfy num_traits::identifies. When trying to use num_traits::pow::pow, I run into this compile time error
the trait `num_traits::identities::One` is not implemented for `&num_bigint::biguint::BigUint

Note that I am aware that I can now use the new Pow method from 0.2.3 (thank you!), But when my exponents are small, I'd like to be able to use what I am guessing would be more efficient functions.

Never mind. I hadn't grokked how to call pow. (I am new to Rust.)


Loss and grad.norm are 0?

Hello! I was wondering if I could get some insight into the issue I'm having right now? I pass in a video data set in the structure:
Videos
-CPR
--CPRvideos.mp4
-PPV
--PPVvideos.mp4
-ETT
--CPRvideos.mp4
...
and I pass in the JSON (eval and train) in this format
[
    {
        "video": "CPR/CPR5(1)(1).mp4",
        "conversations": [
            {
                "from": "human",
                "value": "<video>Please provide a detailed description of the video."
            },
            {
                "from": "gpt",
                "value": "This video shows a healthcare provider performing chest compressions on a newborn baby. The compressions are being delivered at the appropriate depth and rate for neonatal resuscitation."
            }
        ]
    },

    ... rest of the videos

    ]

Your script is fantastic and has gotten us far, however I am wondering what potentially causing loss and gradnorm being 0 through each epoch?

Right okay, thank you so much I'll get back to you!

Hi, zjysteven
After running your colab script, it appears I'm getting 0 loss and 0 grad_norm

	base_model.model.language_model.model.layers.31.mlp.gate_proj.lora_A.default.weight
	base_model.model.language_model.model.layers.31.mlp.gate_proj.lora_B.default.weight
	base_model.model.language_model.model.layers.31.mlp.up_proj.lora_A.default.weight
	base_model.model.language_model.model.layers.31.mlp.up_proj.lora_B.default.weight
	base_model.model.language_model.model.layers.31.mlp.down_proj.lora_A.default.weight
	base_model.model.language_model.model.layers.31.mlp.down_proj.lora_B.default.weight
Loading data...
Parameter Offload: Total persistent parameters: 12137472 in 665 params
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: Tracking run with wandb version 0.18.7
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
  0% 0/175 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (530 > 512). Running this sequence through the model will result in indexing errors
/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
Token indices sequence length is longer than the specified maximum sequence length for this model (584 > 512). Running this sequence through the model will result in indexing errors
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
Token indices sequence length is longer than the specified maximum sequence length for this model (542 > 512). Running this sequence through the model will result in indexing errors
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 3.3333333333333333e-06, 'epoch': 0.03}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 6.666666666666667e-06, 'epoch': 0.06}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1e-05, 'epoch': 0.09}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.3333333333333333e-05, 'epoch': 0.11}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.6666666666666667e-05, 'epoch': 0.14}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2e-05, 'epoch': 0.17}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.9998272238333606e-05, 'epoch': 0.2}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.99930895503665e-05, 'epoch': 0.23}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.99844537269886e-05, 'epoch': 0.26}
  5% 9/175 [00:56<09:44,  3.52s/it]W1123 23:41:16.124000 7112 torch/distributed/elastic/agent/server/api.py:704] Received Signals.SIGINT death signal, shutting down workers
W1123 23:41:16.125000 7112 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 7138 closing signal SIGINT
Traceback (most recent call last):

Sorry for the inconvenience but we've been trying for ages any assitance would be massively appreciated!
Thanks!!

Can you also run the unmodified example_video.sh? The colab hasn't been updated for a while.

From: Zarif @.>
Sent: Saturday, November 23, 2024 6:44:16 PM
To: zjysteven/lmms-finetune @.>
Cc: Jingyang Zhang @.>; Comment @.>
Subject: Re: [zjysteven/lmms-finetune] Loss and grad.norm are 0? (Issue #55)
Hi, zjysteven
After running your colab script, it appears I'm getting 0 loss and 0 grad_norm
image.png (view on web)https://urldefense.com/v3/__https://github.com/user-attachments/assets/c10dad95-c081-49e7-a8aa-ee9b6e37d45a__;!!OToaGQ!tmljA_0LnEbE0y04epVXtOfoTG6reQUvIfXM76nwNEXz-8kFXrEuqGHRE0DB5PrIJo7Abm6Z1n-yTDXP1UEOiLnQMNOATPI$
    base_model.model.language_model.model.layers.31.mlp.gate_proj.lora_A.default.weight
    base_model.model.language_model.model.layers.31.mlp.gate_proj.lora_B.default.weight
    base_model.model.language_model.model.layers.31.mlp.up_proj.lora_A.default.weight
    base_model.model.language_model.model.layers.31.mlp.up_proj.lora_B.default.weight
    base_model.model.language_model.model.layers.31.mlp.down_proj.lora_A.default.weight
    base_model.model.language_model.model.layers.31.mlp.down_proj.lora_B.default.weight

Loading data...
Parameter Offload: Total persistent parameters: 12137472 in 665 params
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-corehttps://urldefense.com/v3/__https://wandb.me/wandb-core__;!!OToaGQ!tmljA_0LnEbE0y04epVXtOfoTG6reQUvIfXM76nwNEXz-8kFXrEuqGHRE0DB5PrIJo7Abm6Z1n-yTDXP1UEOiLnQmrRc4m4$ for more information.
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: Tracking run with wandb version 0.18.7
wandb: W&B syncing is set to offline in this directory.
wandb: Run wandb online or set WANDB_MODE=online to enable cloud syncing.
0% 0/175 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (530 > 512). Running this sequence through the model will result in indexing errors
/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
Token indices sequence length is longer than the specified maximum sequence length for this model (584 > 512). Running this sequence through the model will result in indexing errors
use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False.
Starting from v4.46, the logits model output will have the same type as the model (except at train time, where it will always be FP32)
Token indices sequence length is longer than the specified maximum sequence length for this model (542 > 512). Running this sequence through the model will result in indexing errors
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 3.3333333333333333e-06, 'epoch': 0.03}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 6.666666666666667e-06, 'epoch': 0.06}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1e-05, 'epoch': 0.09}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.3333333333333333e-05, 'epoch': 0.11}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.6666666666666667e-05, 'epoch': 0.14}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2e-05, 'epoch': 0.17}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.9998272238333606e-05, 'epoch': 0.2}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.99930895503665e-05, 'epoch': 0.23}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 1.99844537269886e-05, 'epoch': 0.26}
5% 9/175 [00:56<09:44,  3.52s/it]W1123 23:41:16.124000 7112 torch/distributed/elastic/agent/server/api.py:704] Received Signals.SIGINT death signal, shutting down workers
W1123 23:41:16.125000 7112 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 7138 closing signal SIGINT
Traceback (most recent call last):
Sorry for the inconvenience but we've been trying for ages any assitance would be massively appreciated!
Thanks!!
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/zjysteven/lmms-finetune/issues/55*issuecomment-2495694072__;Iw!!OToaGQ!tmljA_0LnEbE0y04epVXtOfoTG6reQUvIfXM76nwNEXz-8kFXrEuqGHRE0DB5PrIJo7Abm6Z1n-yTDXP1UEOiLnQd68qlIo$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AEWZDG7YHIQX4H5O3CM7THL2CEHNBAVCNFSM6AAAAABSLQQU6GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJVGY4TIMBXGI__;!!OToaGQ!tmljA_0LnEbE0y04epVXtOfoTG6reQUvIfXM76nwNEXz-8kFXrEuqGHRE0DB5PrIJo7Abm6Z1n-yTDXP1UEOiLnQcncI6-o$.
You are receiving this because you commented.Message ID: @.***>

I believe I did? I ran ir directly from this github: here I'll copy paste it here
NUM_GPUS=1
DISTRIBUTED_ARGS="
    --nnodes=1 \
    --nproc_per_node ${NUM_GPUS} \
    --rdzv_backend c10d \
    --rdzv_endpoint localhost:0
"

# arguments that are very likely to be changed
# according to your own case
MODEL_ID=llava-next-video-7b                            # model id; pick on by running `python supported_models.py`
TRAIN_DATA_PATH=./example_data/ego4d_video_train.json   # path to the training data json file
EVAL_DATA_PATH=./example_data/ego4d_video_eval.json     # path to the evaluation data json file (optional)
IMAGE_FOLDER=./example_data/images                      # path to the image root folder; if provided, the image paths in the json should be relative
VIDEO_FOLDER=./example_data/videos                      # path to the video root folder; if provided, the video paths in the json should be relative
NUM_FRAMES=8                                            # how many frames are sampled from each video

TRAIN_VISION_ENCODER=False                              # whether train the vision encoder
USE_VISION_LORA=False                                   # whether use lora for vision encoder (only effective when `TRAIN_VISION_ENCODER` is True)
TRAIN_VISION_PROJECTOR=False                            # whether train the vision projector (only full finetuning is supported)

USE_LORA=True                                           # whether use lora for llm
Q_LORA=False                                            # whether use q-lora for llm; only effective when `USE_LORA` is True
LORA_R=8                                                # the lora rank (both llm and vision encoder)
LORA_ALPHA=8                                            # the lora alpha (both llm and vision encoder)

RUN_ID=${MODEL_ID}_lora-${USE_LORA}_qlora-${Q_LORA}     # a custom run id that determines the checkpoint folder and wandb run name

DS_STAGE=zero3                                          # deepspeed stage; < zero2 | zero3 >
PER_DEVICE_BATCH_SIZE=2                                 # batch size per GPU
GRAD_ACCUM=1                                            # gradient accumulation steps
NUM_EPOCHS=5                                            # number of training epochs

LR=2e-5                                                 # learning rate
MODEL_MAX_LEN=512                                       # maximum input length of the model

torchrun $DISTRIBUTED_ARGS train.py \
    --model_id $MODEL_ID \
    --data_path $TRAIN_DATA_PATH \
    --eval_data_path $EVAL_DATA_PATH \
    --image_folder $IMAGE_FOLDER \
    --video_folder $VIDEO_FOLDER \
    --num_frames $NUM_FRAMES \
    --output_dir ./checkpoints/$RUN_ID \
    --report_to wandb \
    --run_name $RUN_ID \
    --deepspeed ./ds_configs/${DS_STAGE}.json \
    --bf16 True \
    --num_train_epochs $NUM_EPOCHS \
    --per_device_train_batch_size $PER_DEVICE_BATCH_SIZE \
    --per_device_eval_batch_size $PER_DEVICE_BATCH_SIZE \
    --gradient_accumulation_steps $GRAD_ACCUM \
    --eval_strategy "epoch" \
    --save_strategy "epoch" \
    --save_total_limit 1 \
    --learning_rate ${LR} \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length $MODEL_MAX_LEN \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --train_vision_encoder $TRAIN_VISION_ENCODER \
    --use_vision_lora $USE_VISION_LORA \
    --train_vision_projector $TRAIN_VISION_PROJECTOR \
    --use_lora $USE_LORA \
    --q_lora $Q_LORA \
    --lora_r $LORA_R \
    --lora_alpha $LORA_ALPHA
    ```

Ok I will try to see if I can reproduce asap

I am thoroughly impressed at how willing you are to help, thank you so much!!

Hi @Nicerice96 thank you for reporting this. I can reproduce it and I've found the fix and cause. Just set a large enough MODEL_MAX_LEN, say 4096, and then you will see loss being not 0. If you are curious about what happened, you can continue to read.
So basically it's related to an earlier update in the transformers library that I described in https://github.com/zjysteven/lmms-finetune/issues/43#issue-2593395378. The huggingface maintainers added a related config for llava-next-video 4 days ago, which triggered the new processing behavior of expanding the  token at preprocessing stage rather than within the LLM's forward pass as before. As a result, having a small MODEL_MAX_LEN will actually include none of your input texts because there are already a lot of  tokens there that reach MODEL_MAX_LEN. Having a large enough MODEL_MAX_LEN thus can solve the problem.
Again, the script for inspecting the output of the collator that I shared here https://github.com/zjysteven/lmms-finetune/issues/55#issuecomment-2495675125 could be very helpful, which is exactly how I identify the cause of this bug.

Hey zjysteven,
OHHHHH, yeah I did notice that on their repo, the tokenizer.config was updated. Thought nothing of it, but that makes a lot more sense, now that you've described it. Okay, thank you so much!!
Thanks,

No problem. Closing now.


Make progress on CI and CD adoption

We didn't really specify well what this means. Let's discuss this to refine it.

Implemented


Support for Shimmie based sites?

I've tried to use two Shimmie based sites, but the program asks for an API key even though they don't have one, and no images load. Another program was able to fetch the images without an account/key.

Shimmie doesn't seem to follow danbooru's Xml API like gelbooru and moebooru does - so it will not work with ahoveiwer.


Footer text gets bigger on iOS when phone is horizontal

Portrait:

Landscape:

https://stackoverflow.com/questions/2710764/preserve-html-font-size-when-iphone-orientation-changes-from-portrait-to-landsca#2711132

Fixed in e6bf0b2


incorrect image URL for different versions' documentation

Actual Behaviour
see https://github.com/apache/apisix-website/blob/master/scripts/sync-docs.js#L136

https://apisix.apache.org/docs/apisix/getting-started
https://apisix.apache.org/docs/apisix/2.9/getting-started

fixed by #697


[cpprestsdk] Update version to fix uwp build

When building cpprestsdk:x86-uwp, some errors ocurred:
c1xx: error C2338: two-phase name lookup is not supported for C++/CLI, C++/CX, or OpenMP; use /Zc:twoPhase-

So I update version to fix this issue.
Related: #7540 #1227。
Note: All features passed in the following triplets test:

x86-windows
x64-windows
x64-windows-static

Needs to wait #10116 merge.

/azp run

(I'm going through outstanding cpprestsdk PRs now, then I'll mint v2.10.15 which should unblock this)

@BillyONeal, thanks for the information!
@JackBoosY, please update cpprestsdk to release revision v2.10.15.

@PhoebeHui There is no v2.10.15 yet; that's what I'm saying, I'm going to drain the PRs against cpprestsdk and then mint it.

@BillyONeal If cpprestsdk has been updated, please let me know.
Thanks.

@JackBoosY Update: I'm waiting on a customer to confirm a data race fix and then I'll mint 2.10.15 :)

Just minted https://github.com/microsoft/cpprestsdk/releases/tag/v2.10.15

Because this PR is blocked by that Brotli thing I made a separate PR which does only the version bump: https://github.com/microsoft/vcpkg/pull/10191/

Duplicated to https://github.com/microsoft/vcpkg/pull/10191

@PhoebeHui You might still want to merge the other changes here (e.g. vcpkg_check_features) -- I only did the version number bump in my change.

@BillyONeal That's not important, I will change them in the next version update.


Esper Build Link errors

Esper build links improperly generate for any esper with nodes 54 and higher selected, seemingly in the generation of the Hex value for the link. Filling a 3* esper's grid fully causes only node 4 to be populated.

From tmtl/erim:
hm. this error just gets strange.
with odin 3*, you can select all but node 39 without selecting anything on the top and bottom rows. that link shares fine - https://ffbeequip.com/espers.html?server=GL&o#Odin|3|60|fffffffffdffe000
the next chain of nodes is 54 -> 53 -> 52.
select 54, and it's removed in the shared link.
select 54 and 53, and they are removed in the shared link, but 52 is added.
select all three, and the entire outer layer is removed, except for nodes 38 and 39. note that node 39 was not selected, as it's access comes from the top row.

everything selected except for nodes 52 and 53 generates https://ffbeEquip.com/espers.html?server=GL&o#Odin|3|60|ffffffffffffe800
from the earlier link, this adds nodes 39 and 53, which can't be selected on their own. everything else on the top and bottom row is unselected.
everything selected except for node 52 has the same result as selecting only nodes 53 and 54 in the top and bottom rows.
i added all of the nodes on the bottom row one at a time, and they were just removed.
back to the first link-
selecting nodes 54 and 55, the link removes them and adds 53.

From tmtl/erim
https://stackoverflow.com/questions/40608111/why-is-18446744073709551615-1-true
interesting.
https://stackoverflow.com/questions/48981762/mysql-bigint-max-value-stored-in-javascript-variable
there's the problem.
"That value cannot be represented accurately in a JavaScript number. You can store it as a string, though that's probably not very useful."
"Part of the 64 bits are used as an exponent. For exact integer values you only have the range of ±2^53-1. "

"JavaScript numbers are always 64-bit floating point."
Trying to convert the whole thing into an integer was a mistake. Particularly if there were more than 64 panels, it would have never worked. You need to review both hex2bin() and bin2hex() as both rely on this method of conversion. I suggest checking if there is no built-in (or a simple code snippet) that can do the job correctly on arbitrarily long strings.


[Packaging] Consider uploading to winget repository

Consider submitting packages to winget repository as an alternative to Scoop or Chocolatey.
https://learn.microsoft.com/en-us/windows/package-manager/package/repository

Update, placed in wrong issue page.


Add support for single thread info

You're doing great work, can you also add support for single thread search using the link: https://www.threads.net/t/CuVvNbmtr06

Of course, working on it :)

Sure, will work on it 👍

@codespikey Just released 1.0.6 with the ability to read from Thread URLs.
const postID = await threadsAPI.getPostIDfromURL('https://www.threads.net/t/CuVvNbmtr06');
const post = await threadsAPI.getThreads(postID);

Super simple!


Add Failure details into comment

Fix #133

@ArtiomTr I've replaced Webpack with esbuild, hope that's fine.
The build time went from ~50sec to ~1sec.


TS-4475 Fix Log Collation Crash (back-port).

@shinrich @oknet -- Susan, this PR is just a back-port to 6.2.x of the Log Collation crash fix. I believe Oknet already +1 the back-port as part of the previous PR #831 . I think this should be back-ported as this is a crash bug and 6.2.x is a LTS release.

@pbchou Please add the version to the Backport field in the corresponding JIRA.

No conflicts in all those merges?

Confirmed no conflicts.


Destroying servers directly instead of removing all the NICs first

Is it possible to delete servers directly and skipping the deletion of the NICs? In CloudControl the server can be deleted regardless of how many NICs are still attached to it
Just a minor enhancement suggestion, if it requires too much effort please drop it.

Do you mean when the NICs are separate resources? If so, sorry but Terraform doesn't work like that - the network adapter resource doesn't know what the server resource is doing...
On 10 Dec. 2016, at 4:24 pm, Ming Sheng <notifications<EMAIL_ADDRESS>wrote:
Is it possible to delete servers directly and skipping the deletion of the NICs? In CloudControl the server can be deleted regardless if the NICs are still attached.
Just a minor enhancement suggestion, if it requires too much effort please drop it.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/DimensionDataResearch/dd-cloud-compute-terraform/issues/79, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezMYC1vhKQktHcpWyAcVwjXufVSU5ks5rGjd4gaJpZM4LJkF3.
itevomcid

Alright, understood.


Auto merge error

https://github.com/solana-labs/token-list/pull/28664
I got an auto merge error:
failed to verify website: https://www.solavegas.com/: failed to verify https://www.solavegas.com/ using HEAD request: context deadline exceeded

Send a message to the customer care
https://accounts.livechat.com/join?access_token=dal%3ATfsLU6_sss2yRHkQ13easy9zIoQ


Cannot build Android app with Tabris 3.8 on paid cloud build service

Problem description
I cannot build a Tabris 3.8 app on the cloud build service for Android. iOS builds fine. Android can still build a 3.7.2 app without a problem.
Expected behavior
App to build successfully.
Error message

We are looking on it right now. With tabris.js 3.8 the Cordova platform and android tooling have been updated. As a result the produced bundle file name has changed from "app.aab" to "app-release.aab" or "app-debug.aab" .

The issue has been resolved. Please trigger your builds again.

Thanks, I can confirm this is working for us now :+1:


Command line installer

Command line installers are very useful for development or hosting companies trying to automate the deployment process.
Some good examples: Prestashop: http://doc.prestashop.com/display/PS15/Installing+PrestaShop+using+the+command+line
Magento: http://devdocs.magento.com/guides/v2.0/install-gde/install/cli/install-cli-install.html

Not spend time to develop  something that 10% of users will maybe use it.

Might be implemented in the near future. I will reopen it and see if I will get time to add that
@aquilax

Happy day @aquilax! 🎉 Better late than never, I'm creating a command line installer => https://github.com/pH7Software/pH7-Social-Dating-CMS/pull/949
It will be available in the next release 🙂


Remove button divs

Short Description

It is, especially for accessibility reasons, not recommended to have divs within buttons.
This removes the internal divs and instead uses multiple buttons.

Changes

Replace h5 with span
Move button further into the component -> have multiple buttons: Text + icon
Do not hide elements, instead make them very small (if we hide them, the focus goes to body when clicking the button, as the button that would have focus is removed)
disable tabbing + screen reading of the icon buttons (the text-button should be enough)
Align details component to not visually overflow on the left side in the form (row had negative margin)

Feedback

Do you think the solution with multiple buttons makes sense?
Do you know any other place where we are using divs in buttons? I scanned all templates for warnings (which we got for details and accordion) but couldn't find any more)

Also: TIL about focus-within - so cool!

Nevertheless, I can not reproduce the focus problem when using display:none in Chrome or Safari. Maybe, we can talk about that again?

We talked about it and the issue is resolved.

However, if the buttons would have any background-color you could still see them (not a problem for now)

We decided to explicitly not show width or background color.


forget: is --prune executed with --dry-run?

Output of restic version
restic 0.14.0 compiled with go1.19 on linux/amd64

How did you run restic exactly?
$HOME/bin/restic --repository-file $HOME/bin/.restic_repository --password-file $HOME/bin/.restic_password forget --keep-within-hourly 24h --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 12m --keep-within-yearly 3y --tag "" --host $HOSTNAME --prune --dry-run

What backend/server/service did you use to store the repository?
sftp
Expected behavior
prune not executed because of --dry-run

Actual behavior
not sure if prune is executed or not - output is not really clear.
Would have removed the following snapshots:
{55803c2c fdb6bcc0}

2 snapshots have been removed, running prune
loading indexes...
loading all snapshots...
finding data that is still in use for 4 snapshots
[0:09] 100.00%  4 / 4 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:02] 100.00%  1407 / 1407 packs processed

to repack:         30606 blobs / 17.197 MiB
this removes:         39 blobs / 105.196 KiB
to delete:             0 blobs / 0 B
total prune:          39 blobs / 105.196 KiB
remaining:        254628 blobs / 22.621 GiB
unused size after prune: 122.595 MiB (0.53% of remaining size)

forget says "Would have removed the following snapshots" but prune sounds like it was executed.
Of course this is not a major issue but still a little bit inconsistent. 🙂

Steps to reproduce the behavior
see above

Do you have any idea what may have caused this?
Do you have an idea how to solve the issue?
better text 🙂
Did restic help you today? Did it make you happy in any way?

What do you think about #4117? Would that have helped or do you have some other message in mind?

Is think this makes it clear enough.
I know it's a little bit pedantic but maybe the output "2 snapshots have been removed, running prune" should also be changed. 🙂

Here you go: https://github.com/restic/restic/pull/4117/commits/6bbb9ef52e6917b5fa97c001ed14716857d92c4a

looks fine for me in 0.15.0 👍


optional overwrite for generate

related to #8

Renamed --install flag to --skip-install and added a couple of test cases.

New option added, request buttons -r, --request-buttons (only for GET requests). Check this gif and last request of the npm run docs:build deployed.
I'm not sure about the location of the location of the vue components (now at src/utils/components/). Maybe add the components as package so we could do the tests of the components separated from the cli.

Feel free to revisit what's leftover in a follow-up PR.

please check #12


Remove mz-fs and glob-promise

Fixes #3977

Just to be clear: promisify had to be implemented by us because node 6 doesn't support promisify. Once we drop node 6 support we can remove it and use util.promisify

Great work @Samatar26 ❤ I made a few changes 👍


New FormEncodingBuilder.addEncoded() API.

This is a behavior change: previously we'd encode '+' as plus
and encode characters that don't need it, including ','. Now
we use the same rules as HttpUrl.

LGTM

Closes #1215


merge from template-arlac77-github

package.json

chore(package): add true (publishConfig.provenance)

.github/workflows/ci.yml

chore: add coverallsapp/github-action@v2 (jobs.test-node.steps.uses)
chore: (jobs.release.permissions)
chore: add coverallsapp/github-action@v2 (jobs.finish.steps.uses)
chore: (permissions)

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 98.485%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
61

Relevant Lines:
61

💛  - Coveralls


the plugin got from npm does not work well

I use npm to get the newest plugin. However, it's still an old version with usage of "```", not the "block" way in master branch.

I have released v0.0.5 from master. Hope this works for you.


gdextension: singleton causes segfault on exit (although works in runtime)

Godot version
v4.0.alpha.custom_build [4463dd9d8] (2022-06-17)
v4.0.alpha.custom_build [ebd966ace] (2022-08-23)
System information
Arch on 5.15.46-1-lts
Issue description
"although works in runtime" means MySingleton still behaves correctly. MySingleton.hello_singleton() correctly prints text.
Code Snippet:
void gdextension_initialize(ModuleInitializationLevel p_level)
{
	if (p_level == MODULE_INITIALIZATION_LEVEL_SCENE)
	{
		ClassDB::register_class<MyNode>();
		ClassDB::register_class<MySingleton>();

		_my_singleton = memnew(MySingleton);
		Engine::get_singleton()->register_singleton("MySingleton", _my_singleton);
	}
}

void gdextension_terminate(ModuleInitializationLevel p_level)
{
	if (p_level == MODULE_INITIALIZATION_LEVEL_SCENE)
	{
		Engine::get_singleton()->unregister_singleton("MySingleton");
		memdelete(_my_singleton); // Segmentation fault with and without this line.
	}
}

Log:
Godot Engine v4.0.alpha.custom_build.4463dd9d8 - https://godotengine.org
Vulkan API 1.2.0 - Using Vulkan Device #0: AMD - AMD RADV SIENNA_CICHLID

WARNING: FBX file import is enabled, but no FBX2glTF path is configured. FBX files will not be imported.
     at: _editor_init (modules/gltf/register_types.cpp:92)

*** clicked exit ***

Segmentation fault (core dumped)

Valgrind:
==202520== Jump to the invalid address stated on the next line
==202520==    at 0xCB7D920: ???
==202520==    by 0x6DE1226: core_bind::Engine::~Engine() (core_bind.h:622)
==202520==    by 0x6DDD84E: void memdelete<core_bind::Engine>(core_bind::Engine*) (memory.h:108)
==202520==    by 0x6DD1E2F: unregister_core_types() (register_core_types.cpp:345)
==202520==    by 0x2316E8D: Main::cleanup(bool) (main.cpp:2994)
==202520==    by 0x22AE913: main (godot_linuxbsd.cpp:70)
==202520==  Address 0xcb7d920 is not stack'd, malloc'd or (recently) free'd
==202520==
==202520== Invalid read of size 1
==202520==    at 0xB1AE72E: x86_64_fallback_frame_state (md-unwind-support.h:63)
==202520==    by 0xB1AE72E: uw_frame_state_for (unwind-dw2.c:1271)
==202520==    by 0xB1B04AA: _Unwind_Backtrace (unwind.inc:303)
==202520==    by 0xB2D78D2: backtrace (backtrace.c:78)
==202520==    by 0x22AF292: handle_crash(int) (crash_handler_linuxbsd.cpp:56)
==202520==    by 0xB1F78DF: ??? (in /usr/lib/libc.so.6)
==202520==    by 0xCB7D91F: ???
==202520==    by 0x6DE1226: core_bind::Engine::~Engine() (core_bind.h:622)
==202520==    by 0x6DDD84E: void memdelete<core_bind::Engine>(core_bind::Engine*) (memory.h:108)
==202520==    by 0x6DD1E2F: unregister_core_types() (register_core_types.cpp:345)
==202520==    by 0x2316E8D: Main::cleanup(bool) (main.cpp:2994)
==202520==    by 0x22AE913: main (godot_linuxbsd.cpp:70)
==202520==  Address 0xcb7d920 is not stack'd, malloc'd or (recently) free'd
==202520==
==202520==
==202520== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==202520==  Access not within mapped region at address 0xCB7D920
==202520==    at 0xB1AE72E: x86_64_fallback_frame_state (md-unwind-support.h:63)
==202520==    by 0xB1AE72E: uw_frame_state_for (unwind-dw2.c:1271)
==202520==    by 0xB1B04AA: _Unwind_Backtrace (unwind.inc:303)
==202520==    by 0xB2D78D2: backtrace (backtrace.c:78)
==202520==    by 0x22AF292: handle_crash(int) (crash_handler_linuxbsd.cpp:56)
==202520==    by 0xB1F78DF: ??? (in /usr/lib/libc.so.6)
==202520==    by 0xCB7D91F: ???
==202520==    by 0x6DE1226: core_bind::Engine::~Engine() (core_bind.h:622)
==202520==    by 0x6DDD84E: void memdelete<core_bind::Engine>(core_bind::Engine*) (memory.h:108)
==202520==    by 0x6DD1E2F: unregister_core_types() (register_core_types.cpp:345)
==202520==    by 0x2316E8D: Main::cleanup(bool) (main.cpp:2994)
==202520==    by 0x22AE913: main (godot_linuxbsd.cpp:70)

Note: For those curious the MRP is a copy of my (experimental) self-contained gdextension template
Steps to reproduce

Download and extract the MRP (It is a test godot project bundled with C++ code)
Run scons in the MRP
Open Godot, import and edit the MRP project
Scene→Quit

Minimal reproduction project
gdextension-segfault.zip

Edit: Error is fixed now and it was unrelated.

My GDExtension is fairly similar and with the latest master @ eda6800f5dc01981f3626580d91a619b496acb5f I do not have this issue. Perhaps it was fixed?

I am still getting a segfault in v4.0.alpha.custom_build [ebd966ace]
I just regenerated the extension api and gdextension too.
Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x00007fffe668d840 in ?? ()
(gdb) bt
#0  0x00007fffe668d840 in ?? ()
#1  0x000055555c90d148 in Object::~Object (this=0x55555f620e70) at core/object/object.cpp:1825
#2  0x000055555c2454a7 in core_bind::Engine::~Engine (this=0x55555f620e70) at ./core/core_bind.h:632
#3  0x000055555c240fff in memdelete<core_bind::Engine> (p_class=0x55555f620e70) at ./core/os/memory.h:109
#4  0x000055555c235920 in unregister_core_types () at core/register_core_types.cpp:358
#5  0x00005555577b9619 in Main::cleanup (p_force=false) at main/main.cpp:3189
#6  0x000055555774265a in main (argc=4, argv=0x7fffffffe048) at platform/linuxbsd/godot_linuxbsd.cpp:74

v4.0.beta.custom_build [afe1c89fb]
I have the same crash in these lines:
https://github.com/godotengine/godot/blob/a16d3625c3b2e82c88cb29e897523aa53f78ec7c/core/object/object.cpp#L1830-L1834
godot.windows.editor.dev.x86_64.exe!Object::~Object() Line 1834	C++
[External Code]
godot.windows.editor.dev.x86_64.exe!memdelete<core_bind::ResourceSaver>(core_bind::ResourceSaver * p_class) Line 112	C++
godot.windows.editor.dev.x86_64.exe!unregister_core_types() Line 372	C++
godot.windows.editor.dev.x86_64.exe!Main::cleanup(bool p_force) Line 3419	C++
godot.windows.editor.dev.x86_64.exe!widechar_main(int argc, wchar_t * * argv) Line 183	C++
godot.windows.editor.dev.x86_64.exe!_main() Line 203	C++
godot.windows.editor.dev.x86_64.exe!main(int argc, char * * argv) Line 217	C++
[External Code]

In my case, this happens if a ResourceFormatSaver was added.

Following @nathanfranke's example on 4.0-stable, with my project, I have an issue with singletons in GDExtension that result in: Failed to retrieve non-existent singleton 'Engine'.
Immediately crashes when trying to run the test project.

Experiencing the same with 4.03-stable (built from source and generated new header/JSON just to be sure).
Unfortunately I don't get any readable backtrace (VS2022):
================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.0.3.stable.custom_build (5222a99f5d38cd5346254cefed8f65315bca4fcb)
Dumping the backtrace.
[0] <couldn't map PC to fn name>
[1] <couldn't map PC to fn name>
[2] <couldn't map PC to fn name>
[3] <couldn't map PC to fn name>
[4] <couldn't map PC to fn name>
[5] <couldn't map PC to fn name>
[6] <couldn't map PC to fn name>
[7] <couldn't map PC to fn name>
[8] <couldn't map PC to fn name>
[9] <couldn't map PC to fn name>
-- END OF BACKTRACE --
================================================================

This seems to happen with the editor interface only. If I run the project from the editor, that instance exists just fine. Only the editor instance is crashing.

To anyone still experiencing this issue, would be great if you could try the PR and see if it fixes the issue for you.


Generate docs

Generate/check docs for mdtk use (including the readme).

update package schematic diagram
base README
mdtk/README describing packages (and contribution? - maybe put this in base...)

Done - raised new issue about readthedocs and listing contributions


Feature request: API to open document in a non-persistent tab

I am creating an extension to interact with reverse engineering tools like IDA Pro and Hopper. My extension will provide a list of functions in the sidebar, and when clicked they will open a document of the disassembly or pseudocode for that function.
I would like it to behave the same way the file explorer does: when clicked, a "non-persistent" tab is opened. And when a new function is clicked, it replaces the old non-persistent tab with its own contents, etc. Currently, calling window.openTextDocument() opens a persistent document.

showTextDocument(document: TextDocument, options?: TextDocumentShowOptions): Thenable<TextEditor>
One of the TextDocumentShowOptions is preview.

I am truly dumb. I had recently refactored my code so that the call to openTextDocument was entirely separate from my own call to showTextDocument with preview explicitly set to false. Pretty embarrassed about this; sorry again. And thank you.


Very slow loading suggestions with file more than 1000 line

with large file with more than 1000 line of code, the suggestion loaded very slowly.

The last version 0.25.6 also have this problem. When you press $ , the world will complete stop..
:(


fix debugger tests with new EMITTING probe status

Motivation
Following https://github.com/DataDog/dd-trace-java/pull/6365 a new probe status was introduced, requiring to update those tests
Changes

Workflow

⚠ Create your PR as draft ⚠
Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
Mark it as ready for review

Test logic is modified? -> Get a review from RFC owner. We're working on refining the codeowners file quickly.
Framework is modified, or non obvious usage of it -> get a review from R&P team

:rocket: Once your PR is reviewed, you can merge it!
🛟 #apm-shared-testing 🛟
Reviewer checklist

[ ] Relevant label (run-parametric-scenario, run-profiling-scenario...) are presents
[ ] No system-tests internal is modified. Otherwise, I have the approval from R&P team
[ ] CI is green, or failing jobs are not related to this change (and you are 100% sure about this statement)
[ ] A docker base image is modified?

[ ] the relevant build-XXX-image label is present
[ ] To R&P team: locally build and push the image to hub.docker.com

[ ] A scenario is added (or removed)?

[ ] Get a review from R&P team
[ ] Once merged, add (or remove) it in system-test-dasboard nightly

There are tests failures. Probably you should split the method "test_method_probe_snaphots" and use the decorators @missing_feature setting the version of the tracer that releases this feature. Be careful, as it will impact in the test for other lang/tracers.

Thanks for the hint. Fixed


Error in conversion of raw -> mzML in project PXD018682

The project PXD018682 when is converted from RAW -> mzML only captured profile MS2 spectra and not the centroid. This is probably an error in the TRFP.
One of the files examples is here:
03.raw

@timosachsenberg @julianus The error will remain for a while because is in the API. It will take some time to be solved. I will add this as a low-priority issue.

@ypriverol Can this be closed with the upgrade to the new TRFP version?


[REVIEW ONLY] Add A/B test of format boosts

In the B variant, remove the down-weighting of content with formats used in the government index. Replace it with an up-weighting of guidance content.
The government index down-weighting was originally added when there was a clear split between specialist content int the government index and mainstream content in the mainstream index. This distinction is no longer as clear because new specialist content has been added to the mainstream index.
The B variant uses the navigation_document_supertype field to boost guidance content. This should push truly mainstream content (as opposed to content in the mainstream index) higher in search results, helping mainstream users find content faster.
A boost of 2.5 was chosen because it is the inverse of the 0.4 boost that is applied to government index formats in the A variant.
https://trello.com/c/TjolBPDX/126-have-new-format-weightings-ready-to-use
Marked as REVIEW ONLY until we've run this against the healthcheck and updated the base branch to master.

Apparently Github gets confused if you delete a branch that's been used as a base for a PR, so this PR has been replaced by #806. 🙃


Inappropriate (?) panic from ble::connection::ConnectionState::disconnect() in response to NRF_ERROR_INVALID_STATE

The code in ConnectionState::disconnect() calls disconnect_with_reason() which unwraps any error result from the call to sd_ble_gap_disconnect().  That code in bindings.rs is documented as potentially returning success, or one of 3 possible errors.  Two of the errors may legitimately be considered internal failures that should never happen, but it appears the third, NRF_ERROR_INVALID_STATE can occur in normal operation, though under unusual circumstances. To put it another way, it appears it can occur in a way that can't be anticipated, but doesn't represent a system failure deserving a panic.  Unfortunately it currently panics because of the unwrap(), and I'm not sure there would be any way to avoid it.
#[doc = " @retval ::NRF_SUCCESS The disconnection procedure has been started successfully."]
#[doc = " @retval ::NRF_ERROR_INVALID_PARAM Invalid parameter(s) supplied."]
#[doc = " @retval ::BLE_ERROR_INVALID_CONN_HANDLE Invalid connection handle supplied."]
#[doc = " @retval ::NRF_ERROR_INVALID_STATE Disconnection in progress or link has not been established."]
#[inline(always)]
pub unsafe fn sd_ble_gap_disconnect(conn_handle: u16, hci_status_code: u8) -> u32 {

We've observed this once while a unit was connected to a host via WebBluetooth, and the host eventually had a browser crash which led to this panic occurring on the device.
This one error should probably be caught and turned into a different valid error response.  Possibly we should just map it to the existing DisconnectedError, since it's probably very rare and that covers the second half of the "disconnection in progress or link has not been established" that InvalidState supposedly covers.  The "disconnection in progress" part may just be a report on what amounts to an internal (safe) race condition between trying to disconnect while something (like the peer) has already triggered a disconnect which is currently in progress.
Anyone have thoughts on this?

Yes, I agree that DisconnectedError should be returned in that case. nrf_softdevice tries to ensure the connection is in a valid state before calling sd_ble_gap_disconnect, but if an internal disconnection happens and ConnectionState::disconnect() is called before the disconnection event is processed you could get that error.