Runtime upload to VPC does not occur

After upgrading to version 2.5.7, I found that the latest version of lithops is not uploaded to the VPC (the old version remains). I tried deleting the VPC to create a new one. In this case, the lithops process does not start on the VPC at all. I tried the previous version (2.5.6) - same result. The last version we used that didn't have this problem was 2.5.0. It seems that the reason is the recent changes to this function. We are using consume mode for VPC.
Stacktrace on our side:
2021-11-23 11:36:25,763 - INFO - lithops.config[Thread-1] - config.py:131 - Lithops v2.5.7
................................
2021-11-23 11:36:32,054 - INFO - engine.lithops-wrapper[Thread-1] - executor.py:205 - executor.map(_load_ds, 1 items, 32768MB, attempt 1)
2021-11-23 11:36:32,054 - DEBUG - engine.lithops-wrapper[Thread-1] - executor.py:331 - Selected executor ibm_vpc
2021-11-23 11:36:32,061 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:112 - ExecutorID 317913-1 | JobID M000 - Selected Runtime: metaspace2020/metaspace-lithops:1.9.3
2021-11-23 11:36:32,408 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:120 - Runtime metaspace2020/metaspace-lithops:1.9.3 is not yet installed
2021-11-23 11:39:35,209 - ERROR - engine.lithops-wrapper[Thread-1] - executor.py:263 - _load_ds raised an exception. Failed activation(s): [] ID(s): []
NoneType: None
2021-11-23 11:39:35,214 - ERROR - lithops-daemon[Thread-1] - queue.py:432 -  [x] Failed: {"ds_id": "2021-08-27_16h26m18s", "ds_name": "dataset_05", "action": "annotate", "del_first": true, "force": true, "email"<EMAIL_ADDRESS>2021-11-23 11:39:35,215 - WARNING - lithops-daemon[Thread-1] - lithops.py:51 - Lithops annotation failed, retrying.
Traceback (most recent call last):
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/lithops.py", line 81, in _callback
    self._manager.annotate_lithops(ds=ds, del_first=msg.get('del_first', False))
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/dataset_manager.py", line 114, in annotate_lithops
    ServerAnnotationJob(executor, ds, perf).run()
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/annotation_job.py", line 322, in run
    self.results_dfs, self.png_cobjs = self.pipe(**kwargs)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/pipeline.py", line 90, in __call__
    self.load_ds(use_cache=use_cache)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/cache.py", line 81, in wrapper
    return f(self, *args, **kwargs)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/pipeline.py", line 131, in load_ds
    ) = load_ds(self.executor, self.imzml_cobject, self.ibd_cobject, self.ds_segm_size_mb)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/load_ds.py", line 141, in load_ds
    imzml_reader, ds_segments_bounds, ds_segms_cobjs, ds_segm_lens = executor.call(
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 364, in call
    return self.map(
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 269, in map
    raise exc
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/annotation_lithops/executor.py", line 302, in run
    futures = executor.map(
TimeoutError: Readiness probe expired on VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/queue.py", line 430, in get_message
    self._callback(msg)
  File "/opt/dev/metaspace/metaspace/engine/sm/engine/daemons/lithops.py", line 104, in _callback
    raise AnnotationError(ds_id=msg['ds_id'], traceback=format_exc(chain=False)) from e
sm.engine.errors.AnnotationError: Annotation failed (ds_id=2021-08-27_16h26m18s)

@JosepSampe can you look on it?

I ran some tests with Lithops from scratch and for now I'm unable to replicate the issue.

TimeoutError: Readiness probe expired on VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)

Initially, lithops tries to create an ssh connection to the VM and then it proceeds to install all the necessary to it. This message means lithops is unable to ssh to the VM, I think the code raises the exception on this line, so it is normal that it never installs the service there.
Could you verify that the VM is started? and the ssh key is correct?
We recently added a new logic that verifies the ssh keys in IBM VPC are correct: https://github.com/lithops-cloud/lithops/blob/master/lithops/standalone/backends/ibm_vpc/ibm_vpc.py#L559-L577. Might it fail here?
Could you run a simple test by changing this line and add self.backend.master.wait_ready(verbose=True)?

We did not change the ssh key and I tested the ability to manually connect via ssh. I enable verbose=True and DEBUG level for logging. How else can I figure out what the problem is?
2021-11-25 11:17:15,216 - INFO - lithops.config[Thread-1] - config.py:131 - Lithops v2.5.7
2021-11-25 11:17:15,216 - DEBUG - lithops.config[Thread-1] - config.py:212 - Loading Standalone backend module: ibm_vpc
2021-11-25 11:17:15,216 - DEBUG - lithops.config[Thread-1] - config.py:254 - Loading Storage backend module: ibm_cos
2021-11-25 11:17:15,216 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:37 - Creating IBM COS client
2021-11-25 11:17:15,217 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:57 - Set IBM COS Endpoint to https://s3.eu-de.cloud-object-storage.appdomain.cloud
2021-11-25 11:17:15,217 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:60 - Using access_key and secret_key
2021-11-25 11:17:15,222 - INFO - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:97 - IBM COS client created - Region: eu-de
2021-11-25 11:17:15,222 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1] - ibm_vpc.py:46 - Creating IBM VPC client
2021-11-25 11:17:15,222 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1] - ibm_vpc.py:56 - Setting VPC endpoint to: https://eu-de.iaas.cloud.ibm.com
2021-11-25 11:17:15,223 - INFO - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1] - ibm_vpc.py:75 - IBM VPC client created - Region: eu-de
2021-11-25 11:17:15,224 - DEBUG - lithops.standalone.standalone[Thread-1] - standalone.py:60 - Standalone handler created successfully
2021-11-25 11:17:15,224 - DEBUG - lithops.invokers[Thread-1] - invokers.py:92 - ExecutorID aecfec-3 - Invoker initialized. Max workers: 1
2021-11-25 11:17:15,225 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1] - ibm_vpc.py:250 - Initializing IBM VPC backend (consume mode)
2021-11-25 11:17:15,225 - DEBUG - lithops.executors[Thread-1] - executors.py:162 - Function executor for ibm_vpc created with ID: aecfec-3
2021-11-25 11:17:15,225 - DEBUG - lithops.config[Thread-1] - config.py:254 - Loading Storage backend module: ibm_cos
2021-11-25 11:17:15,225 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:37 - Creating IBM COS client
2021-11-25 11:17:15,225 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:57 - Set IBM COS Endpoint to https://s3.eu-de.cloud-object-storage.appdomain.cloud
2021-11-25 11:17:15,225 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:60 - Using access_key and secret_key
2021-11-25 11:17:15,229 - INFO - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:97 - IBM COS client created - Region: eu-de
2021-11-25 11:17:15,230 - DEBUG - lithops.config[Thread-1] - config.py:254 - Loading Storage backend module: ibm_cos
2021-11-25 11:17:15,230 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:37 - Creating IBM COS client
2021-11-25 11:17:15,230 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:57 - Set IBM COS Endpoint to https://s3.eu-de.cloud-object-storage.appdomain.cloud
2021-11-25 11:17:15,230 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:60 - Using access_key and secret_key
2021-11-25 11:17:15,233 - INFO - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1] - ibm_cos.py:97 - IBM COS client created - Region: eu-de
2021-11-25 11:17:15,968 - INFO - engine[Thread-1] - job.py:67 - Storing job metadata
2021-11-25 11:17:16,583 - INFO - engine.lithops-wrapper[Thread-1] - executor.py:205 - executor.map(_load_ds, 1 items, 32768MB, attempt 1)
2021-11-25 11:17:16,584 - DEBUG - engine.lithops-wrapper[Thread-1] - executor.py:331 - Selected executor ibm_vpc
2021-11-25 11:17:16,590 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:112 - ExecutorID aecfec-3 | JobID M000 - Selected Runtime: metaspace2020/metaspace-lithops:1.9.3
2021-11-25 11:17:16,591 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:397 - Runtime metadata not found in local cache. Retrieving it from storage
2021-11-25 11:17:16,591 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:400 - Trying to download runtime metadata from: ibm_cos://metaspace-env-staging-temp/lithops.runtimes/2.5.7/ibm_vpc/02c7_ad2243d5-20c6-4d41-bc24-cf0e316dea32/metaspace2020-metaspace-lithops-1.9.3.meta.json
2021-11-25 11:17:16,809 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:419 - Runtime metadata not found in storage
2021-11-25 11:17:16,809 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:120 - Runtime metaspace2020/metaspace-lithops:1.9.3 is not yet installed
2021-11-25 11:17:16,809 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:337 - Checking if VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>) is ready
2021-11-25 11:17:18,209 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1-ex] - ibm_vpc.py:773 - Starting VM instance metaspace-staging-vpc-128
2021-11-25 11:17:19,440 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1-ex] - ibm_vpc.py:783 - VM instance metaspace-staging-vpc-128 started successfully
2021-11-25 11:17:19,440 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1-ex] - ibm_vpc.py:614 - Waiting VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>) to become ready

Do not appear any other line after the last line?
Try removing all this block: https://github.com/lithops-cloud/lithops/blob/master/lithops/standalone/backends/ibm_vpc/ibm_vpc.py#L559-L577

Yes, this is the last line from the lithops. Next comes an exception from our code, as in the example above.

Try removing all this block: https://github.com/lithops-cloud/lithops/blob/master/lithops/standalone/backends/ibm_vpc/ibm_vpc.py#L559-L577

In this case everything is fine:
2021-11-25 11:34:29,889 - INFO - engine[Thread-1] - job.py:67 - Storing job metadata
2021-11-25 11:34:30,574 - INFO - engine.lithops-wrapper[Thread-1] - executor.py:205 - executor.map(_load_ds, 1 items, 32768MB, attempt 1)
2021-11-25 11:34:30,574 - DEBUG - engine.lithops-wrapper[Thread-1] - executor.py:331 - Selected executor ibm_vpc
2021-11-25 11:34:30,581 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:112 - ExecutorID aad558-1 | JobID M000 - Selected Runtime: metaspace2020/metaspace-lithops:1.9.3
2021-11-25 11:34:30,581 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:397 - Runtime metadata not found in local cache. Retrieving it from storage
2021-11-25 11:34:30,581 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:400 - Trying to download runtime metadata from: ibm_cos://metaspace-env-staging-temp/lithops.runtimes/2.5.7/ibm_vpc/02c7_ad2243d5-20c6-4d41-bc24-cf0e316dea32/metaspace2020-metaspace-lithops-1.9.3.meta.json
2021-11-25 11:34:30,876 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:419 - Runtime metadata not found in storage
2021-11-25 11:34:30,876 - INFO - lithops.invokers[Thread-1-ex] - invokers.py:120 - Runtime metaspace2020/metaspace-lithops:1.9.3 is not yet installed
2021-11-25 11:34:30,876 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:337 - Checking if VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>) is ready
2021-11-25 11:34:31,163 - DEBUG - lithops.util.ssh_client[Thread-1-ex] - ssh_client.py:37 - <IP_ADDRESS> ssh client created
2021-11-25 11:34:31,632 - INFO - lithops.standalone.standalone[Thread-1-ex] - standalone.py:414 - Installing Lithops in VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)
2021-11-25 11:34:31,633 - DEBUG - lithops.utils[Thread-1-ex] - utils.py:227 - Creating function handler zip in /opt/dev/metaspace/metaspace/engine/lithops_standalone.zip
2021-11-25 11:34:31,682 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:421 - Uploading lithops files to VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)
2021-11-25 11:34:32,075 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:431 - Executing lithops installation process on VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)
2021-11-25 11:34:32,075 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:432 - Be patient, initial installation process may take up to 3 minutes
2021-11-25 11:34:34,530 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:101 - Validating lithops version installed on master matches 2.5.7
2021-11-25 11:34:34,643 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:123 - Validating lithops lithops master service is running on VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)
2021-11-25 11:34:34,807 - INFO - lithops.standalone.standalone[Thread-1-ex] - standalone.py:140 - Waiting Lithops service to become ready on VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>)
2021-11-25 11:34:34,981 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:146 - VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>) ready in 0.17 seconds
2021-11-25 11:34:34,981 - DEBUG - lithops.standalone.standalone[Thread-1-ex] - standalone.py:345 - Extracting runtime metadata information
2021-11-25 11:34:50,538 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:430 - Uploading runtime metadata to: ibm_cos://metaspace-env-staging-temp/lithops.runtimes/2.5.7/ibm_vpc/02c7_ad2243d5-20c6-4d41-bc24-cf0e316dea32/metaspace2020-metaspace-lithops-1.9.3.meta.json
2021-11-25 11:34:50,903 - DEBUG - lithops.storage.backends.ibm_cos.ibm_cos[Thread-1-ex] - ibm_cos.py:121 - PUT Object lithops.runtimes/2.5.7/ibm_vpc/02c7_ad2243d5-20c6-4d41-bc24-cf0e316dea32/metaspace2020-metaspace-lithops-1.9.3.meta.json - Size: 6.6KiB - OK
2021-11-25 11:34:50,904 - DEBUG - lithops.storage.storage[Thread-1-ex] - storage.py:436 - Storing runtime metadata into local cache: /home/ubuntu/.lithops/cache/lithops.runtimes/2.5.7/ibm_vpc/02c7_ad2243d5-20c6-4d41-bc24-cf0e316dea32/metaspace2020-metaspace-lithops-1.9.3.meta.json

@kpavel @JosepSampe do you know what happens?

@gilv This logic is wrong and raises an exception even if the ssh keys are correct. This was not added by me, so now I'm not sure why it fails. For now I disabled it in #860 until we found an alternative.

@sergii-mamedov What operating system do you have? Can you please check if you have ssh-keygen available there?

@kpavel
# Ubuntu 16.04.6
$ uname -a
Linux 4.4.0-1128-aws #142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux LTS
$ which ssh-keygen
/usr/bin/ssh-keygen

@sergii-mamedov The code @JosepSampe removed is very important. It actually came to solve the issue when user has wrong ssh keypair configuration, validating that private/public keys are real pair and if not -> raise error for user with self explaining error message. In case it been failing in that section i would really prefer to solve it, but i may need your help as i can't reproduce it in my environment.
If you run ssh-keygen -y -f /home/ubuntu/.ssh/ibm_cloud_vpc, do you get output identical to your public key?

If you run ssh-keygen -y -f /home/ubuntu/.ssh/ibm_cloud_vpc, do you get output identical to your public key?

Yep, except comment after == on the end of the line (in you case)

@sergii-mamedov Ok... I think i may found why the original error wasn't raised. Can I ask you to modify original release code, that has the failing validation section removed by Josep? Just add those 2 lines to let original exception raise with original error message:
https://github.com/lithops-cloud/lithops/pull/862/files

Just as a suggestion: Lithops already shows this message when the private key is not valid:
2021-11-25 23:59:35,670 [DEBUG] lithops.standalone.backends.aws_ec2.aws_ec2 -- SSH to <IP_ADDRESS> failed (publickey): Authentication failed.
I wonder:  If instead of relying on platform specific logic to verify the ssh key, which has much more overhead, wouldn't doing this be enough and compatible for all backends ?

@kpavel
lithops.standalone.standalone.LithopsValidationError: Private ssh key /home/ubuntu/.ssh/ibm_cloud_vpc and public key another-id-rsa on master VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>) are not a pair

We have a few public keys on VPC.

@sergii-mamedov Is it possible you have multiple private ssh keys in the /home/ubuntu/.ssh dir? If so, maybe this one (/home/ubuntu/.ssh/ibm_cloud_vpc) is not valid, and another one in the /home/ubuntu/.ssh dir is the valid.

ssh -i /home/ubuntu/.ssh/ibm_cloud_vpc -o IdentitiesOnly=yes -F /dev/null<EMAIL_ADDRESS>
yes, it is works
On VPC we have 3 public key. On our machine, where lithpos starts, we have two privat keys: ibm_cloud_vpc and another one.

@sergii-mamedov
If you run: ssh-keygen -y -f /home/ubuntu/.ssh/ibm_cloud_vpc | cut -d' ' -f 2 do you have as output your ssh public key?

@kpavel yep

@sergii-mamedov
Can i ask you modify one line of code to have thrown exception contain also contents of the compared public keys?
                raise LithopsValidationError(
                    f"Private ssh key {key_filename} and public key "
                    f"{key_name} on master {self} are not a pair because {public_res} not equal to {private_res}. "
                    f"the public is from {self.ibm_vpc_client.get_key(key_id).get_result()['public_key']}")
                    ```
https://github.com/lithops-cloud/lithops/blob/master/lithops/standalone/backends/ibm_vpc/ibm_vpc.py#L572

I cut ssh key
lithops.standalone.standalone.LithopsValidationError: Private ssh key /home/ubuntu/.ssh/ibm_cloud_vpc and public key first-id-rsa on master VM instance metaspace-staging-vpc-128 (<IP_ADDRESS>) are not a pair because AAAAB3Nza......Ni37PW8HrtIw== not equal to /bin/sh: 1: ssh-keygen: not found
/bin/sh: 1: cut: not found. the public is from ssh-rsa AAAAB3Nz...Ni37PW8HrtIw==<EMAIL_ADDRESS>
it looks like lithops tries the private key with the first public key on the master VM and immediately gets an exception

@sergii-mamedov
I don't think that VPC allows you to have more than a single public key per VM instance.
But the issue i see is that both ssh-keygen and cut are missing from /bin/sh shell
Can if you run to get to sh shell:
sh
And then try to run from that shell cut or ssh-keygen?

ubuntu@ip-172-31-35-172:~$ which sh
/bin/sh
ubuntu@ip-172-31-35-172:~$ /bin/sh
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa): ^C
$ cut
cut: you must specify a list of bytes, characters, or fields
Try 'cut --help' for more information.
$

I tried on machine, where we launch lihtops:
$ ipython
Python 3.8.6 | packaged by conda-forge | (default, Nov 27 2020, 19:31:52)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import subprocess

In [3]: subprocess.getoutput([f'ssh-keygen -y -f /home/ubuntu/.ssh/ibm_cloud_vpc | cut -d " " -f 2'])
Out[3]: 'AAAAB3NzaC1yc2EAAAADAQABAAACAQChj/kuF457SkwpU6ZE/cRInuS0LheGPctRt4Esa5PZ9PXHfdpVO0eYrtS3r9s875Bg6xAn94xqfBmyynCmAOjdjzFmz7p04kOhqMHAwn58812VLJyOnG7RB1on5GnahUqNhHs1RaE8EhXvfJXFqfrhc7a2QbftXxgfXz/Yk3Wrr1tw/d5vhC0RYCl8qKyjABJKgkmX82K/Fr9Uer1v/6XUnOTaAo2WN/ElQRRp916ej3fGCuFXYfKeUNrBl0C1bsQ1BAtkLRvbeA6MpYeUeksW/fdfOKY7RPJ0OGeqpaFYUWz+qWQvOk33Hqhu6L5TRVm1lSV9+2iZAhM3eEDLUvbK+yfNCMhJ2o9K/0gX3yVwnmP4rmXjN15YudZCego/ZuBcwznaXRTSbSFKzD/fah65+YETxxNn99mf2m1USX4frsGO2hWrW4OtLlh5DlXjWnBikB0Stwe2bRjlEVyYyRyzBlOrmu3UWyNsWOjDs5mMyOgmS++Q/9LAdXO/XBuJRZ5K/5MIFaBgn4gmdTmV1GLL4BSfX9HC/fByQiwx0RIU5CD3igmaYcJwm88g0wYmghqTLUJbDOdARvS2EhTgKE3tnHCYpSFXZ3iXBQGgsuI1EU21bSp0BwkZG8E+VILpKzp0igC4996+Zbln9rykUjVmCt7oa/zl1I6tDIR3/Zv73Q=='

@sergii-mamedov Have another idea. I would like to try to specify full paths to the binaries.
Can you please replace the line with private_res = subprocess.getoutput([f"/usr/bin/ssh-keygen -y -f {key_filename} | /usr/bin/cut -d' ' -f 2"])
https://github.com/lithops-cloud/lithops/blob/master/lithops/standalone/backends/ibm_vpc/ibm_vpc.py#L569

@sergii-mamedov
The issue should be resolved by PR #865

@kpavel
I tried the master branch. I see the same problem. I tried deleting all public keys (~/.ssh/authorized_keys) except the one using lithops - same error. Apparently, lithops takes information and public keys not from this file, but directly from the VM metadata.

I will try to recreate the VM with one key, then add the other two to the ~/.ssh/authorized_key file and test this hypothesis. Will it help you?

I will try to recreate the VM with one key, then add the other two to the ~/.ssh/authorized_key file and test this hypothesis.
In this case, everything works without problems. Apparently the problem is still on the side of IBM.
@kpavel

@sergii-mamedov
Thanks for the update. Working on a fix.

@sergii-mamedov
Added support for multiple public keys validation in #867

@sergii-mamedov
Should be fixed in master now

@kpavel
It is sucked here:
2021-11-30 13:16:20,646 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1-ex] - ibm_vpc.py:619 - SSH to None failed (publickey): 'key_id'
2021-11-30 13:16:25,907 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1-ex] - ibm_vpc.py:619 - SSH to None failed (publickey): 'key_id'
2021-11-30 13:16:31,076 - DEBUG - lithops.standalone.backends.ibm_vpc.ibm_vpc[Thread-1-ex] - ibm_vpc.py:619 - SSH to None failed (publickey): 'key_id'

I think that I will recreate the VM with one key, since we already lost a lot of time for debugging.

@kpavel Consume mode does not have key_id in config since the VM is manually created trough the dashboard

@JosepSampe According to lithops documentation it is mandatory:

@sergii-mamedov Getting that message means it passed the ssk keypair validation
@JosepSampe Will fix it asap

@kpavel We are using consume mode. In this case, we specify the file name.

@sergii-mamedov Revalidated with my instance running in consume mode with multiple ssk keys. It works.
@JosepSampe Fixed the non mandatory ssh key id during validation. validation actually doesn't need to know key id.

@kpavel This time everything works without problems. Thanks.

@kpavel This time everything works without problems. Thanks.

Excellent. Sorry it took that long.


Update linear scaling graphic

Also a few copy edits.

This change is

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/7b18e0bfbd130998e1244752fdf7e8d3876040b3/
Edited pages:

v19.2/performance.md


Remove Of ISVStudio

Hi
After removing of isvstudio, Do we still merge our changes like before into the dev PowerPlatform. In the new process it says to zip the connector to blob storage and create sas url.
Thanks
Chandra

I got reply from Mail that github pull request process remains same.


Provide call that lifts http4s client into trace context without creating new span

There is a possible minimal implementation of tracing for a http4s client, which is to lift the context to one allowing tracing, and also forwards the tracing headers; but otherwise does not create a new span or decorate the current span.
Reasons?

Firstly, it's a valid use case to not want the client to create a span. Maybe there are issues with forwarding trace data for this particular service, or the extra span adds unneeded extra information. However, you will still want to forward headers, so that downstream and upstream services can still be connected for the larger trace.
Secondly, the creation of span attributes can be something that users want to customize. E.g. if you want to match OpenTelemetry attribute names, amongst other things. By putting the monad lifting into one place, it makes adding custom behavior much simpler, as you can just use Trace[F].span etc. inside your application code. It's basically a nice building block for custom client tracing schemes.

Example code - doesn't have a Ctx parameter, but shows the general idea:
  def liftTrace[F[_]: MonadCancelThrow, G[_]: MonadCancelThrow](
      client: Client[F],
      toHeaders: ToHeaders = ToHeaders.standard
  )(implicit P: Provide[F, G, Span[F]]): Client[G] =
    Client { (request: Request[G]) =>
      Resource.eval(P.ask).flatMap { span =>
        val traceHeaders = toHeaders
          .fromContext(span.context)
          .values
          .map(Header.Raw.apply)
          .toList

        client
          .run(
            request
              .transformHeaders(_ ++ Headers(traceHeaders))
              .mapK(P.provideK(span))
          )
          .mapK(P.liftK)
          .map(_.mapK(P.liftK))
      }
    }

IMO any outgoing call should have an explicit span. But I accept that users can decide for themselves.
PR is welcome.


Provide extensible user forms

From chicchiricco on September 03, 2010 14:53:59
In order to give user access to workflow actions, providing input.
Original issue: http://code.google.com/p/syncope/issues/detail?id=41

From chicchiricco on January 20, 2011 00:33:17
Status: Duplicate
Mergedinto: 75


SCAssetExportSession throwing exception

Almost 1 in every 6 times, the SCAssetExportSession throws an exception:
Thread : Fatal Exception: NSInvalidArgumentException
0  CoreFoundation                 0x0000000182a9c2d8 __exceptionPreprocess
1  libobjc.A.dylib                0x00000001942c80e4 objc_exception_throw
2  AVFoundation                   0x0000000181299d14 -[AVAssetWriter initWithURL:fileType:error:]
3  AVFoundation                   0x00000001812999bc +[AVAssetWriter assetWriterWithURL:fileType:error:]
4  flare                          0x00000001001f0254 -[SCAssetExportSession exportAsynchronouslyWithCompletionHandler:] (SCAssetExportSession.m:448)
5  flare                          0x00000001000634c0 -[FLReviewViewController postVideo] (FLReviewViewController.m:239)
6  flare                          0x0000000100062b7c -[FLReviewViewController postButtonPressed:] (FLReviewViewController.m:151)
7  UIKit                          0x0000000187511404 -[UIApplication sendAction:to:from:forEvent:]
8  UIKit                          0x00000001874fa4e0 -[UIControl _sendActionsForEvents:withEvent:]
9  UIKit                          0x0000000187510da0 -[UIControl touchesEnded:withEvent:]
10 UIKit                          0x0000000187510a2c -[UIWindow _sendTouchesForEvent:]
11 UIKit                          0x0000000187509f68 -[UIWindow sendEvent:]
12 UIKit                          0x00000001874dd18c -[UIApplication sendEvent:]
13 UIKit                          0x000000018777e324 _UIApplicationHandleEventFromQueueEvent
14 UIKit                          0x00000001874db6a0 _UIApplicationHandleEventQueue
15 CoreFoundation                 0x0000000182a54240 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
16 CoreFoundation                 0x0000000182a534e4 __CFRunLoopDoSources0
17 CoreFoundation                 0x0000000182a51594 __CFRunLoopRun
18 CoreFoundation                 0x000000018297d2d4 CFRunLoopRunSpecific
19 GraphicsServices               0x000000018c1936fc GSEventRunModal
20 UIKit                          0x0000000187542fac UIApplicationMain
21 flare                          0x000000010007ea58 main (main.m:16)
22 libdyld.dylib                  0x0000000194946a08 start

I configure the session like so:
- (void)exportVideo:(void(^)(NSURL *, NSError *))completion {
    AVAsset *asset = self.recordSession.assetRepresentingSegments;
    NSURL *outputUrl = self.recordSession.outputUrl;

    SCAssetExportSession *assetExportSession = [[SCAssetExportSession alloc] initWithAsset:asset];
    assetExportSession.outputUrl = outputUrl;
    assetExportSession.outputFileType = AVFileTypeMPEG4;
    assetExportSession.videoConfiguration.preset = SCPresetHighestQuality;
    assetExportSession.audioConfiguration.preset = SCPresetMediumQuality;

    @try {
        [assetExportSession exportAsynchronouslyWithCompletionHandler:^{
            [MBProgressHUD hideAllLoadingIndicators];

            if (completion) {
                completion(assetExportSession.outputUrl, assetExportSession.error);
            }
        }];
    }
    @catch (NSException *exception) {
        [MBProgressHUD hideAllLoadingIndicators];
        NSError *exportExceptionError = // Create an error

        if (completion) {
            completion(nil, exportExceptionError);
        }
    }
}

The SCRecordSession is also impossible to retry (the exception is thrown over and over again).
The strange thing is that the SCRecordSession is only one segment, and should not be too complicated to export. Can you please provide insight? I'm trying to use the library for a client and this is blocking.

I started to use a custom output url (also in the temp dir) and I'm seeing the error less. I'm not sure if that was actually the cause, and would still appreciate feedback.

Could you share the additional informations held by this exception? Seems like you are using SCAssetExportSession correctly.

Yes.
*** -[AVAssetWriter initWithURL:fileType:error:] invalid parameter not satisfying: outputURL != ((void *)0)

It's an NSInvalidArgumentException.
Let me know if you need more details.

That helps! If you don't set a fileType inside the SCRecordSession, it tries to figure out one based on the info it has from SCRecorder. I suppose at that point you detached this recordSession from the recorder, which means it doesn't know which fileType to use anymore. It could be improved, as if the recordSession already has segments, it should be able to figure out a fileType without asking the recorder. The fix for now is to manually set a fileType inside your recordSession. Tell me if that works.

@rFlex AWESOME. I'm going to implement your change this evening, I'll report back here. Thanks!

@rFlex that solution worked for most cases. I'm now having the same issue exclusively with videos that are recorded to the maximum duration. Any ideas?


App freezes

Angular App freezes if left unused for about an hour this is because of ngx-virtual scroll.
If disables it works fine.

Please create a stackblitz reproducing the problem. Or, show how it can be reproduced using https://rintoj.github.io/ngx-virtual-scroller/samples
I can't troubleshoot without a way to reproduce it.
Please re-open when you have reproduction instructions.


test: Cypress - Automated tests for Basic Arango/Curl Import datasources

Description

This PR includes tests for Arango ds - along with curl import & querying & usage of various fields in the query page

Type of change

New script

How Has This Been Tested?

Cypress CI runs

Checklist:
QA activity:

[X] Added Test Plan Approved label after reviewing all Cypress test

/ok-to-test sha=a6bc6f4

/ok-to-test sha=73370db

/ok-to-test sha=ae00fe8

/ok-to-test sha=8f5a194


[Docs]: Adding Detailed Contribution Guide

Describe the feature
There are several patterns used across this repository, which can make contributing difficult for newcomers without a guide. It's better to have a dedicated and detailed contribution guide to attract more contributors.
Additional information

[X] I intend to submit a PR for this feature.
[ ] I have already implemented and/or tested this feature.

It's worth adding this, too


react-dates keypresses are suppressed

The Problem
When using airbnb/react-dates (SingleDatePicker) as custom component I lose the ability to manually type in date.
I'm not sure if it's a bug or just a feature from rrf that supresses form input to force this to be handled through redux.
See github test repos below for example code
react-dates original functionality can be tested here : http://airbnb.io/react-dates/?selectedKind=SingleDatePicker (SDP)&selectedStory=default&full=0&down=1&left=1&panelRight=0&downPanel=kadirahq%2Fstorybook-addon-actions%2Factions-panel
(click in input field and start typing)
Steps to Reproduce
Wrap react-dates component in a Control element
Expected Behavior
Able to click on input field and start typing
Actual Behavior
Keypresses are not sent/registered in react-dates component
Reproducible Code Example
https://github.com/bovan/test-rrf-react-dates

Here's how to get it working:
<Control component={SingleDatePicker}
  model="user.date"
  mapProps={{
    date: (props) => props.modelValue,
    focused: ({ fieldValue }) => fieldValue.focus,
    onDateChange: (props) => props.onChange,
    onFocusChange: (props) => ({ focused }) => focused
      ? props.onFocus()
      : props.onBlur()
  }}
  controlProps={{
    numberOfMonths: 1
  }}
/>

You can pass any control-specific props inside controlProps.
For onFocusChange, since it returns { focused: true } or { focused: false }, you have to help RRF interpret what that means (since React-Dates has no notion of "blurring" I suppose). That's why there's this:
onFocusChange: (props) => ({ focused }) => focused
      ? props.onFocus()
      : props.onBlur()

Hope that helps! I tested it in your repo and it works fine.

Thanks for help, I've tested a bit and manual input seems to work like in your example (where you put SingleDatePicker) as component.
unfortunately, to remove all react-errors I have to wrap the component like
const dateField = (props) => {
            const fieldProps = omit(props, ["model", "name", "onBlur", "onChange", "onFocus", "onKeyPress", "value"]);
            return <SingleDatePicker {...fieldProps}/>;
        };

and use that as component, and then it stops working again. even just using const dateField = (props) => <SingleDatePicker {...props }/>; seems to be enough to stop it from working
I'll continue to dig into this, any tips would be greatly appreciated, thanks again.

Which react errors are you getting? Can you open a separate issue for that?

sorry, thought the description was enough (probably as I had just read through 5 similar issues)
it's the unknown props warning warning.js?3efc:36 Warning: Failed prop type: SingleDatePicker: unknown props found: name, onChange, onBlur, onFocus, onKeyPress, value
This is when I use <Control component={SingleDatePicker} />
If this isn't supposed to happen I can open an issue


Missing CODE of CONDUCT

This repository has no CODE of CONDUCT published.
Suggestion
Add
## Code of Conduct

This project has adopted the [Contributor Covenant](https://www.contributor-covenant.org/) in version 2.0 as our code of conduct. Please see the details in our [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). All contributors must abide by the code of conduct.

to the README file in this repository and copy https://github.com/corona-warn-app/cwa-documentation/blob/main/CODE_OF_CONDUCT.md to this repository.

@MikeMcC399 thank you! Due to this not being an active dev repo this was overlooked, but should absolutely be added. I will PR this in a minute


possible race condition in msvc tools cache

This is not a well investigated issue, just dropping something here to get it "written down".
A couple of builds on Windows have failed with what looks like corruption in the msvc cache file. This isn't confirmed to be consistently reproducible, but has been seen at least twice now, with an error that looks like this (this one is a real capture, the other one I saw wasn't saved, I just retried and it worked - which, if it was a race, would make sense, because unlike the CI setup where the cache is rebuilt for each new image spun up, mine stays around):
[00:17:38] 605/1266 (47.79%) C:\Python36\python.exe test\MSVC\MSVC_SPECTRE_LIBS.py
[00:17:38] C:\projects\scons\scripts\scons.py returned 2
[00:17:38] STDOUT =========================================================================
[00:17:38]
[00:17:38] STDERR =========================================================================
[00:17:38] JSONDecodeError: Expecting ',' delimiter: line 382 column 125 (char 24284):
[00:17:38]   File "C:\Users\appveyor\AppData\Local\Temp\1\testcmd.4240.tmk0iejo\SConstruct", line 3:
[00:17:38]     env = Environment(MSVC_VERSION='11.0', MSVC_SPECTRE_LIBS=None, tools=['msvc'])
[00:17:38]   File "C:\projects\scons\SCons\Environment.py", line 1030:
[00:17:38]     apply_tools(self, tools, toolpath)
[00:17:38]   File "C:\projects\scons\SCons\Environment.py", line 116:
[00:17:38]     _ = env.Tool(tool)
[00:17:38]   File "C:\projects\scons\SCons\Environment.py", line 1911:
[00:17:38]     tool(self)
[00:17:38]   File "C:\projects\scons\SCons\Tool\__init__.py", line 265:
[00:17:38]     self.generate(env, *args, **kw)
[00:17:38]   File "C:\projects\scons\SCons\Tool\msvc.py", line 298:
[00:17:38]     msvc_setup_env_once(env, tool=tool_name)
[00:17:38]   File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 1110:
[00:17:38]     msvc_setup_env(env)
[00:17:38]   File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 1259:
[00:17:38]     d = msvc_find_valid_batch_script(env,version)
[00:17:38]   File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 1162:
[00:17:38]     d = script_env(env, vc_script, args=arg)
[00:17:38]   File "C:\projects\scons\SCons\Tool\MSCommon\vc.py", line 1009:
[00:17:38]     script_env_cache = common.read_script_env_cache()
[00:17:38]   File "C:\projects\scons\SCons\Tool\MSCommon\common.py", line 125:
[00:17:38]     envcache_list = json.load(f)
[00:17:38]   File "C:\Python36\lib\json\__init__.py", line 299:
[00:17:38]     parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
[00:17:38]   File "C:\Python36\lib\json\__init__.py", line 354:
[00:17:38]     return _default_decoder.decode(s)
[00:17:38]   File "C:\Python36\lib\json\decoder.py", line 339:
[00:17:38]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[00:17:38]   File "C:\Python36\lib\json\decoder.py", line 355:
[00:17:38]     obj, end = self.scan_once(s, idx)

If it was a race, we could see one test creating the cache and another test come along, see the cache exists and proceed to read it, but catch it before it's all written.  But I think many tests would already have run which would have caused the cache to be created, so maybe it's a different kind of race.
Perhaps the code in Tool/MSCommon/common.py should be catching JSONDecodeError and doing something better with it than dying?

@jcbrill is this anything you've ever seen?

@mwichmann I have seen it before myself.
I suspect it can be corrupted when running multiple processes since the cache is read once but written every time a news entry is added possibly exacerbated by buffered I/O.  Likely worse if the default cache location is the user folder rather than per-project.
Project-local cache files might be worth testing.  Periodically, I wonder what it would take to use Windows API synchronization primitives to synchronize access.
It is probably better just to clear the cache and carry on rather than exit as the real purpose is just to save some time.

Perhaps multiple writes stepping on each other which corrupts the file for the next read?

one could effectively "lock" by writing to a tempfile and then moving to the real name... though on Windows that could trigger the infamous problem of failing because the file is busy I suppose.
Maybe just ignore the cache if we got an error is enough, as you say - that policy already exists somewhere in that code.

Another problem with writes from multiple processes is that of information loss:  the cache is only read once but written multiple times.  A write could cause the cache file to "lose" entries from writes from another process (i.e., the file cache is not "merged" when updated/written and only written from the memory of the current process).

Sigh.  This thing started its life as a quick hack because running the full set of tests was too slow (we fire off an scons "run" something like 3,000 times, that's a lot of times to pay the tax of detecting the MSVC bits - in many cases x2 because of the default environment as well).  Most normal uses are to start a build... once. It feels like the Hydra is growing ever more heads here...

It is still a good idea for running the test suite.
One possibility is to generate the cache file based on all versions/combinations of visual studio command-line invocations outside of SCons and then change the cache to be "read-only" in SCons (i.e., don't write the cache file out even if entries are not in the cache).
This would only require generating a new cache file whenever any visual studio installations are updated.  Effectively, pre-populate the cache file outside of SCons.

Encountered another case of this.  I fiddled with my installs - I was running a bit low on disk space and wanted to reduce the number of SDKs installed, and ended up removing one that had a path stored for finding headers (this would be a Windows Kits path) for the default compiler,  so all builds failed due to missing headers or libs. Not sure it's worth adding yet another check for that.
After removing the cachefile, a full test run worked, but one test his a similar issue.  Probably the details are unimportant - it was a -j 12 build so lots of chance for any concurrency problems that could exist. I'll paste the fail here anyway:
610/1266 (48.18%) C:\Users\mats\AppData\Local\Programs\Python\Python311\python.exe test\MSVC\MSVC_UWP_APP.py
C:\Users\mats\Documents\github\scons\scripts\scons.py returned 2
STDOUT =========================================================================

STDERR =========================================================================
JSONDecodeError: Expecting value: line 1 column 1 (char 0):
  File "C:\Users\mats\AppData\Local\Temp\testcmd.16320.7o_sotpq\SConstruct", line 3:
    env = Environment(MSVC_VERSION='14.2', MSVC_UWP_APP=True, tools=['msvc'])
  File "C:\Users\mats\Documents\github\scons\SCons\Environment.py", line 1030:
    apply_tools(self, tools, toolpath)
  File "C:\Users\mats\Documents\github\scons\SCons\Environment.py", line 116:
    _ = env.Tool(tool)
  File "C:\Users\mats\Documents\github\scons\SCons\Environment.py", line 1911:
    tool(self)
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\__init__.py", line 265:
    self.generate(env, *args, **kw)
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\msvc.py", line 298:
    msvc_setup_env_once(env, tool=tool_name)
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\vc.py", line 1110:
    msvc_setup_env(env)
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\vc.py", line 1259:
    d = msvc_find_valid_batch_script(env,version)
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\vc.py", line 1162:
    d = script_env(env, vc_script, args=arg)
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\vc.py", line 1009:
    script_env_cache = common.read_script_env_cache()
  File "C:\Users\mats\Documents\github\scons\SCons\Tool\MSCommon\common.py", line 125:
    envcache_list = json.load(f)
  File "C:\Users\mats\AppData\Local\Programs\Python\Python311\Lib\json\__init__.py", line 293:
    return loads(fp.read(),
  File "C:\Users\mats\AppData\Local\Programs\Python\Python311\Lib\json\__init__.py", line 346:
    return _default_decoder.decode(s)
  File "C:\Users\mats\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 337:
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\mats\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 355:
    raise JSONDecodeError("Expecting value", s, err.value) from None

FAILED test of C:\Users\mats\Documents\github\scons\scripts\scons.py
        at line 673 of C:\Users\mats\Documents\github\scons\testing\framework\TestCommon.py (_complete)
        from line 775 of C:\Users\mats\Documents\github\scons\testing\framework\TestCommon.py (run)
        from line 458 of C:\Users\mats\Documents\github\scons\testing\framework\TestSCons.py (run)
        from line 109 of C:\Users\mats\Documents\github\scons\test\MSVC\MSVC_UWP_APP.py

Test execution time: 58.3 seconds

For your amusement (or horror).
There are two issues with the msvc cache: exclusive access and information loss.
The recent file locking changes target exclusive access.
The information loss problem is exacerbated by the cache discipline: read-once, write-many.
In the abstract, consider the following cache behavior of multiple processes with varying execution duration simulating two simultaneous jobs:
  R-------W--------W--------W  [Process A]
  R--W                         [Process B]
       R----W                  [Process C]
              R--W             [Process D]
                     R---W     [Process E]

Assuming all processes write unique mvsc entries, the cache contains the 3 entries from process A (i.e., the cached information from processes B, C, D, and E have been overwritten by the last Process A write).
Instead of exclusive-read and exclusive-write, the discipline should be exclusive-update (a read immediately followed by write, if necessary).
The same example:
  U-------U--------U--------U  [Process A]
  U--U                         [Process B]
       U----U                  [Process C]
              U--U             [Process D]
                     U---U     [Process E]

Assuming all processes write unique msvc entries, the cache contains 7 entries as expected.
The json format as currently employed has a number of limitations that can be overcome by employing a very simple custom cache reader and using a record-oriented json data file format with a json header.
For example:
{ ... }  <- header record
#        <- single character record delimiter
{ ... }  <- cache entry
#        <- single character record delimiter
{ ... }  <- cache entry
#        <- single character record delimiter

A json structure can be loaded every time a # is seen.
The header can contain file format version information and the number of records (this needs to be back-patched though) and cache entries can contain a variable that contains paths to verify upon load:
{"cache_version": 1, "cache_count": 2}
#
{ ... "VERIFY_PATHS": [...] ...}
#
{ ... "VERIFY_PATHS": [...] ...}
#

Notes:

The cache_version field in the header can be used to discard prior incompatible formats without attempting to load the rest of the file.  Basically, only two lines are read rather than the whole file.  This is most frequently a problem when developing a modified cache format and the main branch uses a different format.

The contents of the VERIFY_PATHS contain paths that are known to exist at the time the cache entry is written.  MSVC frequently populates the enviroment variables of interest (e.g., PATH, INCLUDE, LIB, and LIBPATH) with non-existent paths.  The VERIFY_PATHS could be the union of INCLUDE, LIB, and LIBPATH.  This covers just about everything that may no longer exist: SDK version, host/target combinations, msvc toolsets, etc.  Once the json record is loaded, the paths can be evaluated and if any are missing, the record is discarded and upon completion of the read, a write is immediately initiated.
The VERIFY_PATHS allows the cache to become self-healing in the face of msvc updates.  It is not foolproof though, as there are msvc updates that are problematic to detect.

In multi-process contention, there are a number of shortcuts that can be employed to determine if the cache has changed since the last read.  If there have been no changes, simply append the new record to the end of the file. In a single process scenario, cache updates degenerate to simple single-record "append" rather than a full cache dump each time.

The book-keeping is non-trivial but not impossible.

My own "stress-tests" for MSVC code changes involve excruciatingly long run times in some cases which is a vastly different scenario than the scons tests.
Due to the combinatorial explosion of msvc installations and possible argument combinations, a current test run this morning created 1744 environments and had a cache count of 1697 entries and took more than 50 minutes to build the cache and compile a simple "hello, world" program for all 1744 environments.  That scenario only used the major sdk versions (e.g., for windows 11, 10, 8.1) and not all sdk versions installed.  (Although it did have at least one of every msvc version going back to 6.0.)

Well, in my defense, I did mention the information loss issue (much more briefly) in the locking PR... just don't know any way to deal with it.  We have the same potential issue (which occasionally does bite people who try concurrent use of the same build tree) with the sconsign signature "database": we read everything on startup, then the file remains in that state while the in-memory copy changes, then it's written out in the end.  So you can have two divergent in-memory copies, and somebody's write is going to win and somebody's is going to lose.  For sconsign of a very large build tree there's a performance hit of "read everything even if you don't need it" and "write everything even if (almost?) nothing changed".  I've suggested that one should become a real database someday, but I've not pushed the change to use pysqlite for that, although it worked in limited trials when I prototyped it.  I think that may be overkill for the less formal msvc cache.

Well, in my defense, I did mention the information loss issue (much more briefly) in the locking PR... just don't know any way to deal with it.

I was not being critical.  I apologize.
The solution is to read the cache file before every write while the lock is held.  A cache file format change makes this easier.
Code blob with load, dump, update, version check, and path verification protected by file locking:
https://github.com/jcbrill/mswindev/blob/0abd9e276e420212cf8ec906a4634aa1040f5a7f/mswindev/vsdetect.py#L12794-L13395
_cache_update_file (line 13314 is the implementation of the exclusive read and write).
Ignoring logging and timing code, it is pretty straightforward: load the file and dump the file (subject to internal bookkeeping).
Tested during development with 4 processes running simultaneously with the same stress-test sconstruct (i.e., running in lock-step).

... for large cache files, there is a performance hit of reading the file, converting the objects to json and then discarding the records that are already in the cache.  Separating the key from the full entry would allow discarding the full entry without conversion if the key was already in the process memory cache at the cost of reading the file.
For msvc, the cost tradeoff is file processing versus running one more msvc batch files which can be considered really expensive  (or there would be no need for a cache in the first place).

... for large cache files, there is a performance hit of reading the file, converting the objects to json and then discarding the records that are already in the cache. Separating the key from the full entry would allow discarding the full entry without conversion if the key was already in the process memory cache at the cost of reading the file.
For msvc, the cost tradeoff is file processing versus running one more msvc batch files which can be considered really expensive (or there would be no need for a cache in the first place).

Can you upload your largest cache json file? So we can do a little perf testing with it?

Can you upload your largest cache json file? So we can do a little perf testing with it?

I would happy to generate a large cache file.
To be clear, this does not represent a real-world situation nor is it representative of the scons test suite.
I use my own research tool to generate all possible combinations of arguments: MSVC_VERSION, HOST_ARCH, TARGET_ARCH, MSVC_TOOLSET_VERSION, MSVC_SPECTRE_LIBS, MSVC_UWP_APP, and MSVC_SDK_VERSION to test scons detection changes, argument validation, script errors, and that all expected combinations actually compile a simple hello world application.
Note: HOST_ARCH is employed to make sure every accessible batch file is called even though it won't be used directly in SCons environment construction as an end-user should not be changing HOST_ARCH.  A new variable (MSVC_TOOL_ARCH) would not be the worst thing in the world.  This could be useful if a compiler generation bug is suspected.
I started a run using ALL combinations which is going to result in an astronomical number of environments due to the number of SDK versions I have installed on my development box and the number of msvc toolsets for each modern msvs installation.
At present, the test sconstruct is still in 2019 (14.2) and has generated 2783 environments (fewer cache entries) and the cache file on disk is 10.1MB and growing.
By the time this finishes, the cache file will likely be enormous.  I'm running against a personal branch containing msvc "fixes" and enhancements.
Is this what you really want?

@jcbrill - yes. Your huge json would represent the unrealistically large cache file. So if we want to see how fast/slow read/update/write is vs read...... write... it'd be a reasonable example.

Ok will do.
FWIW, in the case of a single process (i.e., no contention), read/update/write is going to be significantly faster as there really is no reason to load the entire file (the file modification timestamp and record count in the header has not changed since the last write) and the one new record each write cycle can be appended to the file rather than dumping the entire data structure. There is a non-trivial bookkeeping cost but it should be far less than dumping the entire data structure to disk for each new entry.  Reads are cheap (no need to read whole file) and writes are far less expensive.

@mwichmann There may be a benefit to explicitly calling flush on the cache file handle and possibly calling os.fsync as well after the json dump call and prior to the context manager taking over.
Annotated snippet:
        with SCons.Util.FileLock(CONFIG_CACHE, timeout=5, writer=True), p.open('w') as f:
            # Convert the cache dictionary to a list of cache entry
            # dictionaries. The cache key is converted from a tuple to
            # a list for compatibility with json.
            envcache_list = [
                {'key': list(key), 'data': data} for key, data in cache.items()
            ]
            json.dump(envcache_list, f, indent=2)
            f.flush()                   # <- explicit flush
            os.fsync(f.fileno())        # <- explicit sync

Not sure about os.sync.  In my own work, the os.fsync() call is based on a configuration value and it is not enabled currently.  I don't remember why, either it didn't seem to make a difference or was expensive or both.

sigh.
this stuff has its history in 40 year old (and older) UNiX systems...
you don't need the flush, which has to do with getting the data out of Python and into the OS... https://docs.python.org/3/library/io.html#io.IOBase.close
fsync might be a different story, that has to do with getting the data out of OS buffers and onto a disk.

If using fsync, recommended to flush buffered files first: https://docs.python.org/3/library/os.html#os.fsync

Well, the point was that closing the file assures the flush(), which the context manager around the open/json.dump sequence should guarantee.  But hey... who ever knows.

Cache file (22.7mb json, 477kb zip):
scons_msvc_cache.zip
Log ouput:
Running scons (16:24:48.36) ...
...
Total: builds: 6644, cache: 6477
...
Scons finished (20:46:40.12)


[monitor] Precise Typechecking

Packages impacted by this PR

@azure/monitor-opentelemetry-exporter
@azure/monitor-ingestion
@azure/monitor-query

Issues associated with this PR
N/A
Describe the problem that is addressed by this PR
Tests are not being typechecked and linting isn't configured correctly. This PR migrates those libraries to precise typechecking setup introduced in https://github.com/Azure/azure-sdk-for-js/pull/31786
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Are there test cases added in this PR? (If not, why?)
N/A
Provide a list of related PRs (if any)
https://github.com/Azure/azure-sdk-for-js/pull/31786
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[x] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

API change check
API changes are not detected in this pull request.


mqtt-bridge: add option in Helm chart for connection with Azure IoT Hub

Is your feature request related to a problem? Please describe.
Connecting to Azure IoT Hub using e.g., Node-RED, can be unreliable, especially when having a bad internet connection. It would be great to use mqtt-bridge, but it does not allow authentication via Username / Password yet.
Describe the solution you'd like
Enable Username / Password authentication in Azure IoT Hub. Furthermore, add an extra option just for Azure IoT Hub that asks for things like "SAS token" and then automatically converts them to MQTT connection parameters.

Resolved, mqtt-bridge allows for passwords and certificates

nvm, the SAS option is not yet resovled


feat(types): types generator and CookieID string literal

PR for #209
📚 Description
Typing for a projects' cookies.
A nice to have feature.
Allowing the users to type their cookie references.
And might be able to be used as typing for cookiesEnabledIds etc..
📝 Checklist

[x] All commits follow the Conventional Commit format
[x] The PR's title follows the Conventional Commit format
[ ] Can build successfully
[x] Does not introduce breaking changes

I tried making sure the module would could build correctly.
But the following error was thrown.
 ERROR  Cannot find module '/home/tbj/Documents/nuxt-cookie-control/dist/runtime/types' imported from /home/tbj/Documents/nuxt-cookie-control/dist/module.mjs                                              11:56:23 AM
  at finalizeResolution (node:internal/modules/esm/resolve:264:11)
  at moduleResolve (node:internal/modules/esm/resolve:924:10)
  at defaultResolve (node:internal/modules/esm/resolve:1148:11)
  at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:390:12)
  at ModuleLoader.resolve (node:internal/modules/esm/loader:359:25)
  at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:234:38)
  at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:87:39)
  at link (node:internal/modules/esm/module_job:86:36)

Tried checking out to the current master commit, and the same error was thrown.
So i don't believe this is caused by anything i committed.


fix(cli): Allow setting admin password from CLI

This pull request allow users to specify an hash for the admin password from the command line. Fixes #428.
A pull request to update the documentation is coming soon but in the meantime a couple of caveats:

The password can be generate using the command htpasswd -nb -B admin <password>
If portainer is run from the binary escape the $ in the password hash with \ to avoid shell expansion
If portainer is run from inside a container escape the $ in the password hash with \\ to avoid expansion from the shell outside and inside the container.

Hi @dedalusj ! Thanks for the PR !
Any particular reason that you've decided to use an already hashed password as a CLI argument? I do think that it would be a lot simpler if the password was use in plain-text.
E.g. portainer --admin-password admin123
I don't really like all the $ char escaping part, it adds some complexity.

Mostly security. I wouldn't want the plain text password to be exposed to the bash history for example.
The other use case is to run portainer via a compose file and I wouldn't want the plain text password exposed there either, especially because compose files are often committed in the repositories.

Great idea to do this, and the timing is right for me.
To generate the password hash you are using htpasswd, it would be helpful if this dependency were in the Docker image so that we could use the image with a different entry point to generate credentials.
The docs and the code might also want to support loading the hashed password from a file, so that there's no need to deal with shell escaping.

The portainer container is built from scratch. Shipping htpasswd in the container require the use of a Docker image with a package manager which would increase the image size. Part of the appeal of portainer is its small Docker image, I wouldn't want to give that up.
I will add instructions to the docs on how to use another container to produce the password so that it's production is platform independent.
I'll add support for loading the hashed password from a file in the next couple of days.

Thanks @dedalusj I agree with you for the security points. I'll ping you in the docs issue to update the documentation.

@vielmetti I'm not sure about adding the password from file flag. I'd like to keep the CLI simple too, having too many CLI options will introduce more complexity. Having one way to set the password from the CLI should be enough, I do not think that the password from file use case would be commonly used.
@dedalusj think you can remove the password from file part? Or implement it as a second PR? This way we could rework/integrate it later if needed.

Good enough @deviantony - the first PR should have the simplest thing that works, and doesn't need two code paths when one will suffice.

I removed the password from file functionality. I'll open another PR if necessary once the API for this has been accepted, don't want to update a second PR if changes are needed here.
Same for the docs I'll update them once this PR has been accepted.

My tests are OK. I'll merge it soon.
@dedalusj see my update in the documentation repository first.

Thanks @dedalusj :+1:
Check my update on the documentation PR.

Docker image pulled from portainer still not able to identify --admin-password. Could you please create a new image with this change.

@AchuthSankar this will be available in the next release (should be available this week).

@deviantony it seems the docker image is still not published

@puncha It will be part of release 1.13. ETA is end of the week / beginning of next week.


[Bug Report]

What is the issue you are having?
Hi, so thx for your work that i cannot play without now cause it's simply too nice ;)
So i have an issue with, for the moment, only one char.
I have tested without addon, no problem, with elvui, np and it's when i turn on Merathilis that i have the issue.
In chat, general, it spam an issue message which says : "oUF_GCD: Missing Spell data"
it spam everytime i made an action (attack, but also open a chest, etc)
Thx for your help and have a nice continuation ;)
phil
What is the expected behavior?
In chat, no message spam
What actually happened?
In chat, general, it spam an issue message which says : "oUF_GCD: Missing Spell data"
Suggested Solution/Workaround
command /muierror does nothing
maybe an option i turn on
Errors
In chat, general, it spam an issue message which says : "oUF_GCD: Missing Spell data"

ElvUI Changelog/Troubleshooting

[X] I have confirmed that I have read the Changelog and done the Troubleshooting Steps.
[X] I have disabled all addons except the AddOns called 'ElvUI' and 'ElvUI OptionsUI' and 'ElvUI_MerathilisUI' by using /muidebug on? This is important in order to rule out that any of your other addons are causing the issue. If you manage to pinpoint a conflict between ElvUI and/or MerathilisUI and another addon, make sure you include that information.
[X] I have verfied my issue doesn't exist in the default UI.

Which Race, Class, Specialisation are you on?

So i'm on paladin horde blood elf and i saw it's an addon skin which has pb, cause when i desactive addon skins, no more pb.

So i tried with all skins and it appears there is no problem when the skin of Ace is desactivated in MarathilisUi > Skins/Addons > AddonSkins > uncheck Ace3 and it is no more message/spam

the error should not be related to a skin at all.

ok don't know, but just when i disable it, it's not message/spam again in the chat


Add cpp_compat for C headers

This affects the headers for the lakers_shared and lakers_c libraries.
Closes #230

Looks good, thank you!


source is a keyword in ballerina

Description:
source is a keyword in Ballerina lang at the moment. This is a widely used variable name and it should not be a keyword.
Steps to reproduce:
public type FileConfig record {|
    string source;
    string target;
    boolean isBallerinaConf = false;
|};

will result in an error.
Affected Versions:
jballerina-0.992.0-m2-SNAPSHOT
OS, DB, other environment details and versions:
Related Issues (optional):

Suggested Labels (optional):
Bug
Suggested Assignees (optional):

IMO, even though "source" is commonly used as a variable name, it makes sense to have "source" as a keyword to represent the source-only-ness of a construct (which is what it is used for now).
On the other hand, identifier literals are now easier to write ('source) compared to 0.991.0, so IMO having source as a variable/field name is still possible without much of a hassle. WDYT?
public type FileConfig record {|
    string 'source;
    string target;
    boolean isBallerinaConf = false;
|};

public function main() {
    FileConfig fc = { 'source: "s", target: "t" };
}

Please raise an issue in https://github.com/ballerina-platform/ballerina-spec if you have further concerns regarding the same, or have suggestions for an alternate keyword.


Elitism not sorted in next generation

By adding the elitist at the beginning, next generation will have the best at the beginning instead of being mixed up with the mutated ones.

Thanks for adding a PR @guzuligo ! :fireworks:
Would you be able to explain your thinking on the change a bit more?
Elitism selects the best members relative to the previous generation and so there's no guarantee that they will be the best in the current generation which is why they've been added to the end of the members array until now

Sure.
Have a look at the score. If you have like 5 elitist, you will notice that the fittest elitist is the 5th one.
The problem is when you are monitoring the progress and need to see the fittest one.
For example, I create several rooms where AI tries to do a certain task. If I set the elitism to 1, I just need to monitor the first. If in the middle of the training I decided to change the elitism to 10, I have to move the camera to the 10th to check the progress as the 10th is the one that made better progress in previous generation.
Using unshift simply guarantees that the one with the highest score in previous generation is at index zero, which saves me doing workarounds. 👍

Makes sense, all tests passed -- merging in!
Thanks again for contirbuting


Fetch or Update Custom Objects

Hi there, I need to fetch and update the details of custom created objects as shown in the figure.. kindly help me how can I achieve it

@MShoaibAkram There is no current support for custom objects. My own HubSpot plan doesn't include custom objects, so it's not something I was able to build and test.
Do you feel up for taking a stab at building this? I'm open to a PR, if you wanted to work on it and provided test coverage.
From reading the API docs, this is going to be a bit trickier than what I've done with built-in objects. I imagine we're going to need a base CustomObject model in this package that you would then extend in your own app. In my mind, you'd end up with Coupons and Materials as classes in your own app codebase, extending CustomObject in this package, and you'd then need to specify things like requiredProperties and searchableProperties and so forth. I'm not sure how relationships would work.
It's going to take a good deal of work to build out this part of the package, and without a HubSpot Enterprise account, I can't do much to help, unfortunately.
(I wonder if HubSpot ever provides Enterprise account features for developers building out SDKs for the community?)


Fixes broken ClassLoaderFromStackOfBrooklynClassLoadingContextTest.testLoadClassFromBundle test

Updates test jars to fix broken ClassLoaderFromStackOfBrooklynClassLoadingContextTest.testLoadClassFromBundle test and other tests

Thanks @nakomis - LGTM.
For the record, the jars needed recompiled because their MANIFEST.mf depended on brooklyn [0.12,1.0). When we bumped brooklyn version to 1.0.0-SNAPSHOT, it meant all tests using any of these test bundles failed.


Generating Schemas for Particle Accelerator with schema2kotlin

Modifies schema2kotlin tool to add generated schemas to Entity_Specs. Schemas can be accessed from the specs statically as a .schema property.
This PR also introduces a SchemaRegistry. All generated schemas are automatically registered in the registry.
Currently, collections of references are not supported.

Note:
According to the Kotlin docs,

A companion object is initialized when the class is loaded (typically the first time it's referenced by other code that is being executed), in a thread-safe manner.

source

Maybe it's time to add a collection to the schema in goldens?

Or, leave out collections support in schemas completely for now, since the rest of generated code also doesn't support collections?

Note:
According to the Kotlin docs,

A companion object is initialized when the class is loaded (typically the first time it's referenced by other code that is being executed), in a thread-safe manner.

source

Or, leave out collections support in schemas completely for now, since the rest of generated code also doesn't support collections?

This is a good question. I'm not sure of the status of collection codegen. Maybe @csilvestrini knows?

I'll wait on merging this PR; a manual copybara is required to fix internal breakages (root cause: d15008e).

I just meant: change the name of this suite of tests. It's testing the mapOf function only, it's not testing other stuff.

I held off and decided to reorganize things in a future PR. It's ready to go, the branch depends on this landing first: https://github.com/alxrsngrtn/arcs/tree/kt-gen-utils

Passes presubmit 🎊 🎉


Add set_data() to ImageWidget

This would change the current data arrays in the image widget. For example if input and mcorr movies are shown for a batch item, "set_data([new_input, new_mcorr])" would change the data arrays shown in the viewer. Useful with DataGrid to display DataFrames and select rows.
Allow this to work as long as:

The number of data arrays is the same as the current number of data arrays, this corresponds to the Gridplot shape.
The ndim of the new data arrays matches

Allow set data to take kwargs such as:

reset_frame_index: bool , which sets @current_frame index back to 0 if True
reset_vmin_vmax, True by default.

Also force setting the graphic data for each subplot to make sure they update the current frame.

done in #216


Fix useless access modifiers(private) in lib/miq_automation_engine

Purpose or Intent
private for class methods doesn't work.  These class methods following the private keyword are still public.  You need to either use private_class_method or move the methods to private instance methods on the singleton class.  The attached are changes for lib/miq_automation_engine.
You probably want to review each commit one by one and append ?w=1 to the URL so you can skip whitespace changes, like https://github.com/ManageIQ/manageiq/commit/032491f7e4456582fdaf29ee4b56a3040bd05d32?w=1
Found using rubocop --only UselessAccessModifier
Links

Follow up to #10960

Not doing the rubocops here since I just moved the code.


doc fix typos: fileystem -> fileystem

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Can't sign CLA. Unable to select a version.

Do you have JS disabled or anything like that? When I go to that page the box fills with the CLA shortly after the page loads. Can you try in a different browser?

It worked today. Seemed like cla-assistant.io had some issues, sometimes i got error "No cla found for kolide/fleet" also.
Anyhow, it worked today.


DeploymentOptions setMaxWorkerExecuteTime

I do not see that the method DeploymentOptions.setMaxWorkerExecuteTime has any effect.
Looking into the code I do not see that the method getMaxWorkerExecuteTime is invoked anywhere and I do not see any other way how the attribute maxWorkerExecuteTime can be used.
In contrast, the attribute maxWorkerExecuteTime in the VertxOptions class is used properly.

hi, you need to use a named thread pool to make it work

with workerPoolName

hi, thanks a lot. Could you perhaps provide an example? I did set the worker pool name in the deployment options but it also did not help and the max worker execute time set on the deployment options was not used. Here is my code where I set the max execute time on VertxOptions as it did not work when applied to the DeploymentOptions:
Vertx vertx = Vertx.vertx(new VertxOptions().setWorkerPoolSize(10).setMaxWorkerExecuteTime(180L * 1000 * 1000000));
DeploymentOptions deploymentOptions = new DeploymentOptions().setWorkerPoolName("generic-request-processor-pool")
.setWorker(true).setMultiThreaded(true);
vertx.deployVerticle(new GenericRequestProcessor(), deploymentOptions);

it should be:
DeploymentOptions deploymentOptions = new DeploymentOptions().setWorkerPoolName("generic-request-processor-pool").setMaxWorkerExecuteTime(limit);

I tried it - this did not work for me. Again, the source code does not look like the property maxWorkerExecuteTime can be used.

What @b-liberman said is true (As of 3.4.2), only the maxWorkerExecuteTime being used is in VertxOption.
The maxWorkerExecuteTime in DeploymentOption is not used in any context except for the DeploymentOptionsConverter, which used to convert Json configurations.

as the issue is verified by others, can it be re-opened, please?

@b-liberman is right.
in the DeploymentManager, line 468 (version 3.5.0):
WorkerExecutorImpl workerExec = poolName != null ? vertx.createSharedWorkerExecutor(poolName, options.getWorkerPoolSize()) : null;
Deploy new verticle dose not use the 3rd arg of createSharedWorkerExecutor(), so the VertxOptions.getMaxWorkerExecuteTime() will take effect.

Reopening. This test fails in master:
  @Test
  public void testMaxExecuteWorkerTime() throws Exception {
    String poolName = TestUtils.randomAlphaString(10);
    long maxWorkerExecuteTime = NANOSECONDS.convert(3, SECONDS);
    DeploymentOptions deploymentOptions = new DeploymentOptions()
      .setWorkerPoolName(poolName)
      .setMaxWorkerExecuteTime(maxWorkerExecuteTime);
    vertx.deployVerticle(new AbstractVerticle() {
      @Override
      public void start(Future<Void> startFuture) throws Exception {
        vertx.executeBlocking(fut -> {
          try {
            SECONDS.sleep(5);
            fut.complete();
          } catch (InterruptedException e) {
            fut.fail(e);
          }
        }, startFuture);
      }
    }, deploymentOptions, onSuccess(did -> {
      testComplete();
    }));
    await();
    String timeLimitMsg = "time limit is " + MILLISECONDS.convert(maxWorkerExecuteTime, NANOSECONDS);
    assertTrue(Files.lines(Paths.get(System.getProperty("java.io.tmpdir"), "vertx.log"))
      .anyMatch(line -> {
        if (line.contains("has been blocked for")
          && line.contains(timeLimitMsg)
          && line.contains(poolName)) {
          return true;
        }
        return false;
      }));
  }

As a workaround, create a SharedWorkerExecutor before deploying the verticle with io.vertx.core.Vertx#createSharedWorkerExecutor(java.lang.String, int, long)

Will be fixed in 3.5.1


[RHCLOUD-20860] Refactor set OrgIds and EBS account numbers from the database

Instead of just forwarding the incoming headers, this change ensures
that the "EBS account number" and the "OrgId" are set in the context
using the contents from the database.
This comes from one issue we have been having with the notifications
service: the service was erroring because we were just sending the
incoming "EBS account number" header, even though we had the associated
"OrgId" on the database.
So we have decided to simply use the incoming headers as a validation
—make sure that an EBS account number and/or OrgId are present—, and
then use the database contents to send the headers to the rest of the
services or to the Kafka queues.
Links
[RHCLOUD-20860]

This is great, thanks for doing it!
2 things:

in notifier middleware I am passing XRHID identity to EmitAvailabilityStatusNotification is it also possible to populate OrgID in context's h.PARSED_IDENTITY (*identity.XRHID) struct instance ? Other way is that we can do is that we can populate OrgID in  identity.XRHID struct instance which passed to EmitAvailabilityStatusNotification but my preference is to do in tenancy middleware. Let me know what do you think.
This is not passing OrgID(in case we have api request without OrgID but orgID is in DB ) to LoggerFields because tenancy middleware is not before ParseHeaders in e.Group in routes.go. Maybe we can put Tenancy middleware before  LoggerFields ? cc @lindgrenj6

Yeah we could throw LoggerFields right in the tenancyMiddleware group at the top - and add it where needed throughout the app. I think the only place that isn't covered by those is check avialability and get /endpoints/:id from my quick grep.

1. in notifier middleware I am passing` XRHID identity` to `EmitAvailabilityStatusNotification` is it also possible to populate `OrgID` in context's `h.PARSED_IDENTITY (*identity.XRHID)` struct instance ? Other way is that we can do is that we can populate OrgID in  `identity.XRHID` struct instance which passed to `EmitAvailabilityStatusNotification` but my preference is to do in tenancy middleware. Let me know what do you think.

You are right, yeah. We talked about it and I forgot implementing it... I've just pushed that!

2. This is not passing OrgID(in case we have api request without OrgID but orgID is in DB ) to `LoggerFields` because tenancy middleware is not before `ParseHeaders` in `e.Group` in routes.go. Maybe we can put Tenancy middleware before  `LoggerFields` ? cc @lindgrenj6

Yeah we could throw LoggerFields right in the tenancyMiddleware group at the top - and add it where needed throughout the app. I think the only place that isn't covered by those is check avialability and get /endpoints/:id from my quick grep.

All right, I've rearranged the middlewares!


Upgrade to SQLDelight 2.0.0

The old version depended on co.touchlab:sqliter-driver:1.0.10
The new version depends on co.touchlab:sqliter-driver:1.2.3
Unfortunately these dependencies are mutually-incompatible.

Tracking test flake here:
https://github.com/cashapp/zipline/issues/1170


[Small Fix] Edited construct usage to match script name

Just a tiny fix (not sure if this is too small of a change to submit a PR, let me know if it is).

Looks good, thanks!


Add More Visible Status LED to Frog Sensor

One of the common complaints about the Frog Sensor is that it is difficult to quickly determine the status of the sensor (is there an error? Is it connected to wifi, etc).
It's been suggested that the simplest first effort at solving this would be to add a status LED that is visible from the outside of the enclosure to display the Frog Status.

It seems like the most promising option would be to add one of these WS2813 LEDs. This one from SEED studios comes on a board and has a nice i2c connector already.
https://www.mouser.com/ProductDetail/Seeed-Studio/104020169?qs=BJlw7L4Cy7%2FQJGgijlqWzg%3D%3D

Now a mounting location just needs to be decided. Perhaps up in the sensor shroud so that light comes out of the openings could be nice. I'll try to see if it fits in CAD.

As discussed on Discord, @lwbantoto is going to take over this task :). Thanks!

Commenting @keenanjohnson

It's my belief that the local user interface will be more useful than an LED. Closing as won't fix for now.


Display topic input only in stream narrow

Automated message from Dropbox CLA bot
@kunall17, it looks like you've already signed the Dropbox CLA. Thanks!


[image_picker] Fix pickMultiImage throwing an error on iOS 13 and below.

pickMultiImage currently throws an exception on iOS 13 and below, due to it returning a string rather than a list of strings as it should. This PR fixes this behaviour.
Relevant issue:

flutter/flutter#100132

Pre-launch Checklist

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[x] I read and followed the [relevant style guides] and ran [the auto-formatter]. (Unlike the flutter/flutter repo, the flutter/plugins repo does use dart format.)
[x] I signed the [CLA].
[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated pubspec.yaml with an appropriate new version according to the [pub versioning philosophy], or this PR is [exempt from version changes].
[x] I updated CHANGELOG.md to add a description of the change, [following repository CHANGELOG style].
[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making, or this PR is [test-exempt].
[x] All existing and new tests are passing.

Oops, I meant to assign that bug to myself yesterday; I'm trying out a Pigeon migration for this implementation as a proof-of-concept for Pigeon preventing this kind of problem. I'm going to hold off on reviewing this for another day or to see how that goes, since I expect it will be a more future-proof fix. If it doesn't work out we can just patch it this way.

Closing in favor of https://github.com/flutter/plugins/pull/5285. Sorry for the duplicated effort here.


Fade In delay in fadeViewHelper

Hi thanks for the library, i want to ask is there is a way to add a fade in delay at the fadeViewHelper, it actually has a .setFadeOutDelay and a .setAnimationDuration but there is no option for fade in, like a .setFadeInDelay, thanks.

Hi, if that class does not do what you want you can add your own custom UI to have full control.


Cannot use demo api

https://user-authentication-api.now.sh/api demo api returns the following

are you still getting this? I think it might have been from a cold start on now that went wrong or something. I was able to load up the API just now.

I get this now

Yes now it works.


Handle Encrypted Keys

Auto generate password and store it encrypted in a pass file/add to Snowflake config private_key_file_pwd
In file forgeconfig/src/security/rsa_keypair_generator.py
https://github.com/kameshsampath/snow-builder-collection/blob/e43c1e200d41e9a0156b56cd259a6ca015ac7397/forgeconfig/src/security/rsa_keypair_generator.py#L53

@kameshsampath Maybe never decrypt the key in the file system, but only in memory on the container, then generate a JWT token based on it and update the users config.toml with the token for auth


Correct way of binding to check boxes and radio buttons

I wanted to make sure that I was binding to check boxes and radio buttons correctly. Please see this jsFiddle: http://jsfiddle.net/nareshbhatia/jjdm5/
Specifically:

The HTML markup for a group of check boxes should have the same class. Same is true for a group of radio buttons.
A group of radio buttons should, in addition, have the same name attribute. This is not required for check boxes.

@delambo I found this issue after having issues binding to my radio group's name...
This does not work:
bindings: {
  'input[name=id_verification_required]': 'id_verification_required'
}

However, after reading @nareshbhatia's comment, an update to using a class selector does:
bindings: {
  '.idvr': 'id_verification_required'
}

Any idea why the name selector doesn't work, yet the class selector does? This doesn't seem to be documented anywhere. I am happy to open a PR if someone can explain why the name selector fails.

Disregard the above comment. I just realized I had a hidden field with the name attribute set to id_verification_required.


[decancer] Dehoist

Adds a mass decancer command with checks and handling, and a limit of 5000 users at once.

stonks PR, thanks!


libstfl dependency looks unmaintained

Newsboat version (copy the output of newsboat -v or the first line of git show):
Newsboat 2.25.0 - https://newsboat.org/
Copyright (C) 2006-2015 Andreas Krennmair
Copyright (C) 2015-2021 Alexander Batischev
Copyright (C) 2006-2017 Newsbeuter contributors
Copyright (C) 2017-2021 Newsboat contributors
Newsboat is free software licensed under the MIT License. (Type `newsboat -vv' to see the full text.)
It bundles:

JSON for Modern C++ library, licensed under the MIT License: https://github.com/nlohmann/json
optional-lite library, licensed under the Boost Software License: https://github.com/martinmoene/optional-lite
expected-lite library, licensed under the Boost Software License: https://github.com/martinmoene/expected-lite

Newsboat 2.25.0
System: Darwin 20.6.0 (x86_64)
Compiler: g++ Apple LLVM 12.0.5 (clang-1<IP_ADDRESS>)
ncurses: ncurses 5.7.20081102 (compiled with 5.7)
libcurl: libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.41.0 (compiled with 7.64.1)
SQLite: 3.32.3 (compiled with 3.32.3)
libxml2: compiled with 2.9.4
Config file (copy from ~/.newsboat/config or ~/.config/newsboat/config):
Not applicable
Steps to reproduce the issue:

Compile newsboat from source: it requires the libstfl dependency

The libstfl package seems not maintained anymore, and the upstream website is gone: http://www.clifford.at/stfl/

Other info you think is relevant:
I am a maintainer of Homebrew, the missing package manager for MacOS (and Linux).
We might need to drop support for newsboat in the future if no alternative to libstfl is found.
See https://github.com/Homebrew/homebrew-core/pull/89981 for reference.

Thanks for opening an issue! We'll maintain our own fork for now: https://github.com/newsboat/stfl We intend to move to some Rust crate eventually, see #232, but there's no plan or timeline for that just yet.
I guess this resolves the situation for now, so closing.


Is there a release schedule?

📝 Feedback
Is there a defined release schedule for react-spectrum / react-aria / react-stately?
The last release was nearly a month ago, but there have been a bunch of useful additions and bugfixes.

Hey @shrugs,
Good question. We haven't quite worked out a consistent release schedule yet as we are still ironing out the process. However, we are in the middle of prepping the latest release. It should be out this week.

Release is out! https://react-spectrum.adobe.com/releases/2020-08-18.html

awesome! 🎉  updating my apps now
feel free to close this issue or use it as a way to track the addition of a more defined release schedule, whatever the maintainers would like