CPU Dockers not working "groups: cannot find name for group ID 1000"

I used the commands from the Advanced Tutorial  on both Python 3.6 and 3.7 CPU Docker images.  I keep getting a group error.
I'm running Unbuntu  20.04.1 LTS on Vitrual Box on Mac Mohave OS.
richard@richard-VirtualBox:~$ sudo docker run -it --init --ipc=host --rm -u=$(id -u):$(id -g) divkal/cdt-py3.7:latest /bin/bash
groups: cannot find name for group ID 1000
I have no name!@799281c9b236:/$ whoami
whoami: cannot find name for user ID 1000
I have no name!@799281c9b236:/$

Hello ! This error stems from the docker image that does not possess your /etc/group and /etc/passwd (as it is personal and configuration-dependent). A way to solve your error message is to add the -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro to your docker run arguments, thus making :
docker run -it --init --ipc=host --rm -u=$(id -u):$(id -g) -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro  divkal/cdt-py3.7:latest /bin/bash

Please note :
divkal/cdt-py3.7:latest is not the latest docker image ( the doc hasn't be updated on this one, sorry!) but use fentechai/cdt:latest instead
I'll keep this issue open to remind myself to update the doc.
Best,
Diviyan

Hello ! This error stems from the docker image that does not possess your /etc/group and /etc/passwd (as it is personal and configuration-dependent). A way to solve your error message is to add the -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro to your docker run arguments, thus making :
docker run -it --init --ipc=host --rm -u=$(id -u):$(id -g) -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro  divkal/cdt-py3.7:latest /bin/bash

Please note :
divkal/cdt-py3.7:latest is not the latest docker image ( the doc hasn't be updated on this one, sorry!) but use fentechai/cdt:latest instead
I'll keep this issue open to remind myself to update the doc.
Best,
Diviyan

Ok, I pulled the fentechai/cdt:latest and ran your command command.
richard@richard-VirtualBox:~$ sudo docker run -it --init --ipc=host --rm -u=$(id -u):$(id -g) -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro  fentechai/cdt:latest /bin/bash
[sudo] password for richard:
richard@6ae42756957d:/$ whoami
richard
richard@6ae42756957d:/$
No more group error, but I can't get to an ipython console or workspace directory as stated in the tutorial.
richard@6ae42756957d:/$ cd workspace
bash: cd: workspace: No such file or directory
richard@6ae42756957d:/$ ipython
bash: ipython: command not found
richard@77b4480a3dd5:/$ find / -type d -name "workspace"
find: '/root': Permission denied
find: '/etc/polkit-1/localauthority': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/log/private': Permission denied
find: '/var/log/unattended-upgrades': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/var/lib/polkit-1': Permission denied
find: '/var/lib/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/private': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/tmp/tmpw_248u_h': Permission denied
find: '/tmp/tmpa7jg7wqk': Permission denied
richard@77b4480a3dd5:/$ find / -type d -name "ipython"
find: '/root': Permission denied
find: '/etc/polkit-1/localauthority': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/log/private': Permission denied
find: '/var/log/unattended-upgrades': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/var/lib/polkit-1': Permission denied
find: '/var/lib/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/private': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/tmp/tmpw_248u_h': Permission denied
find: '/tmp/tmpa7jg7wqk': Permission denied
richard@6ae42756957d:/$ find / -type f -name "ipython*"
find: '/root': Permission denied
find: '/etc/polkit-1/localauthority': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/log/private': Permission denied
find: '/var/log/unattended-upgrades': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/var/lib/polkit-1': Permission denied
find: '/var/lib/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/private': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/tmp/tmpw_248u_h': Permission denied
find: '/tmp/tmpa7jg7wqk': Permission denied
Thank you,
Richard

Ok, I pulled the fentechai/cdt:latest and ran your command command.
richard@richard-VirtualBox:~$ sudo docker run -it --init --ipc=host --rm -u=$(id -u):$(id -g) -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro  fentechai/cdt:latest /bin/bash
[sudo] password for richard:
richard@6ae42756957d:/$ whoami
richard
richard@6ae42756957d:/$
No more group error, but I can't get to an ipython console or workspace directory as stated in the tutorial.
richard@6ae42756957d:/$ cd workspace
bash: cd: workspace: No such file or directory
richard@6ae42756957d:/$ ipython
bash: ipython: command not found
richard@77b4480a3dd5:/$ find / -type d -name "workspace"
find: '/root': Permission denied
find: '/etc/polkit-1/localauthority': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/log/private': Permission denied
find: '/var/log/unattended-upgrades': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/var/lib/polkit-1': Permission denied
find: '/var/lib/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/private': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/tmp/tmpw_248u_h': Permission denied
find: '/tmp/tmpa7jg7wqk': Permission denied
richard@77b4480a3dd5:/$ find / -type d -name "ipython"
find: '/root': Permission denied
find: '/etc/polkit-1/localauthority': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/log/private': Permission denied
find: '/var/log/unattended-upgrades': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/var/lib/polkit-1': Permission denied
find: '/var/lib/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/private': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/tmp/tmpw_248u_h': Permission denied
find: '/tmp/tmpa7jg7wqk': Permission denied
richard@6ae42756957d:/$ find / -type f -name "ipython*"
find: '/root': Permission denied
find: '/etc/polkit-1/localauthority': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/log/private': Permission denied
find: '/var/log/unattended-upgrades': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/var/lib/polkit-1': Permission denied
find: '/var/lib/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/private': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/tmp/tmpw_248u_h': Permission denied
find: '/tmp/tmpa7jg7wqk': Permission denied
Thank you,
Richard

Ah yes, I forgot to recreate the link in the docker (we don't have a .bashrc, so the path links are not there)
To have a python console, run usr/bin/python3, and to have an ipython, you must install it first /usr/bin/python3 -m pip install ipython
Hope that solves your issue,
Best,
Diviyan

Ah yes, I forgot to recreate the link in the docker (we don't have a .bashrc, so the path links are not there)
To have a python console, run usr/bin/python3, and to have an ipython, you must install it first /usr/bin/python3 -m pip install ipython
Hope that solves your issue,
Best,
Diviyan

The usr/bin/python3 got me started.
For installing ipython, there seems to be a permission issue:
/usr/bin/python3 -m pip install ipython
WARNING: The directory '/home/richard/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
It downloads, but then I get
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/home/richard'
Check the permissions.
Adding --no-cache-dir gets rid of the WARNING, but I still get the same ERROR.
/usr/bin/python3 -m pip install ipython --no-cache-dir
I also tried adding --user  with no luck.
/usr/bin/python3 -m pip install ipython --no-cache-dir --user
Thank you,
Richard

The usr/bin/python3 got me started.
For installing ipython, there seems to be a permission issue:
/usr/bin/python3 -m pip install ipython
WARNING: The directory '/home/richard/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
It downloads, but then I get
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/home/richard'
Check the permissions.
Adding --no-cache-dir gets rid of the WARNING, but I still get the same ERROR.
/usr/bin/python3 -m pip install ipython --no-cache-dir
I also tried adding --user  with no luck.
/usr/bin/python3 -m pip install ipython --no-cache-dir --user
Thank you,
Richard

Ah right ! Sorry, I think we need to create a new Docker image ;
/ ## Dockerfile
FROM fentechai/cdt:latest

RUN /usr/bin/python3 -m pip install ipython
CMD /bin/bash

#Command line execution
docker build  -t richardshea/cdt:latest .

And then run your docker image.
An alternative would be to run the original docker container as root :
docker run -it --init --ipc=host --rm  fentechai/cdt:latest /bin/bash

I should definitely be more explicit on the documentation....


False Positive with word Ping in 932115

Description
Audit Logs / Triggered Rule Numbers
False Positive with \nPing Pong and &Ping Pong in RCE rule 932115.
[2021-06-21 09:42:54.229922] [-:error] <IP_ADDRESS>:12345 xyz [client <IP_ADDRESS>] ModSecurity: Warning. Pattern match "(?i)(?:;|\\\\{|\\\\||
\\\\|\\\\||&|&&|\\\\n|\\\\r|`)\\\\s*[\\\\(,@\\\\'\\"\\\\s]*(?:[\\\\w'\\"\\\\./]+/|[\\\\\\\\'\\"\\\\^]*\\\\w[\\\\\\\\'\\"\\\\^]*:.*\\\\\\\\|[\\\\^\\\\.\\\\w '\\"/\\\\\\\\]*\\\\\\\\)?
[\\"\\\\^]*(?:s[\\"\\\\^]*(?:y[\\"\\\\^]*s[\\"\\\\^]*(?:t[\\"\\\\^]*e[\\"\\\\^]*m[\\"\\\\^]*(?:p[\\"\\\\^]*r[\\"\\\\^]*o[\\"\\\\^]*p[\\"\\\\^]*e ..." at
ARGS:myarg. [file "/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "294"] [id "932115"] [msg "Remote Command
Execution: Windows Command Injection"] [data "Matched Data: \\x0d\\x0aPing found within ARGS:myarg: Fussball spielen\\x0d
\\x0aPing Pong"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-
windows"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"]
[hostname "x.ch"] [uri "/myuri"] [unique_id "xyz"]

https://regex101.com/r/YJwfiJ/1
Your Environment

CRS version (e.g., v3.2.0): 3.3.0
Paranoia level setting: 1
ModSecurity version (e.g., 2.9.3): 2.9.3

Confirmation
[x] I have removed any personal data (email addresses, IP addresses,
passwords, domain names) from any logs posted.

The &Ping Pong does not trigger any rules for me. Yet &ping pong triggers 932150.
The link break brings the following:
$ curl localhost -d 'foo=
Ping Pong'
932115 Remote Command Execution: Windows Command Injection

$ curl localhost -d 'foo=
ping pong'
932105 Remote Command Execution: Unix Command Injection
932115 Remote Command Execution: Windows Command Injection
932150 Remote Command Execution: Direct Unix Command Execution

The difference with lowercase is fairly striking. As is the number of rules bring triggered. I suggest to replace Ping Pong with table tennis. That fares a lot better from a FP perspective.

Honestly, I do not really know what to do here outside of ignoring the problem. Any ideas?

Yes, this problem may be too rare. We can close this issue here. I just wanted to report the problem but then didn't find the time to resolve it ;-)

CRS issue chat meeting: we don't like a chained rule that excludes "ping pong".
In addition, this could potentially open a bypass.
We think we should leave this for quieter times.


Added multibanco payment method

Description
Added multibanco payment method
Tested scenarios
Make a purchase using the multibanco payment method
Fixed issue:
N/A

@rikterbeek

Hi @lfluvisotto,
Many thanks for the PR! We will investigate this and test it and get back to you, but it looks great.
Regards,
Rik
Adyen

Hi @lfluvisotto,
The code looks really good thank you. A couple of methods that are not needed because it does not support recurring/subscription. Can you remove these methods see above. As well Multibance comes back as well as HPP method. Could you add the check here (example SEPA direct debet see: https://github.com/Adyen/adyen-magento/blob/develop/app/code/community/Adyen/Payment/Model/Observer.php#L149-L152) to disable it when the API method is enabled?
Regards,
Rik
Adyen

Hi @lfluvisotto
Yes indeed perfect!
Regards,
Rik
Adyen

Hey @lfluvisotto
Thanks for adding multibanco - the PR looks fine!
One minor improvement can be that instead of calculating the comprafacil.deadline_date two times (https://github.com/Adyen/adyen-magento/pull/856/files#diff-857809b571a4129f6be9571f298a001cR89 and https://github.com/Adyen/adyen-magento/pull/856/files#diff-12fb05611e5c9a93cea6b697bc051775R23)
you can calculate it once and store it on additional data, here:
https://github.com/Adyen/adyen-magento/pull/856/files#diff-9f51100419c8e5c7060aa2eb0295c381R569
or expose the calculation on a helper
Please let me know if you can include it to the PR.
Best Regards,
Giorgos
Adyen

@lancergr yes, I will verify.

@lancergr can it be this way ?
https://github.com/Adyen/adyen-magento/pull/856/commits/fa832e3bf48eb6b149813bef844f38470bc214dc
https://github.com/Adyen/adyen-magento/pull/856/commits/cc2081cf32cca1dcaddc2fdb9b5af4f1640b39b0
https://github.com/Adyen/adyen-magento/pull/856/commits/207eb60aa2b280e0db2b0f8bb89c41630daf26a6

Hi @lfluvisotto
Exactly!
It's also good practice to avoid nested ifs when possible. Could you please also move the if statement of line 572 https://github.com/Adyen/adyen-magento/commit/fa832e3bf48eb6b149813bef844f38470bc214dc#diff-9f51100419c8e5c7060aa2eb0295c381R572
outside the if of line 569?
https://github.com/Adyen/adyen-magento/commit/fa832e3bf48eb6b149813bef844f38470bc214dc#diff-9f51100419c8e5c7060aa2eb0295c381R569
Best Regards,
Giorgos Adyen

@lancergr https://github.com/Adyen/adyen-magento/pull/856/commits/3beb2abe56bfc68c61957b90038a3faddac02649

Hi @lfluvisotto,
Many thanks for the PR and updates. I will merge it now.
Regards,
Rik
Adyen


Improve setting up clients to interact with DTR

When using self-signed certificates on DTR, it's not obvious that you need to configure your client to trust those certs.

Docker for Linux
Docker for Mac
Docker for Windows

Already addressed by https://docs.docker.com/datacenter/dtr/2.2/guides/user/access-dtr/


feat(Data/List/Chain): chain'_attachWith

Note that this is a PR to bump/v4.13.0, as it uses some new lemmas on attachWith.

Clearly I did something wrong, so I'll reopen this in a new PR.

#18429


Getting ValueError: Error calling check on server: Internal Server Error when checking server on an aws cluster

Hi! First off, I just wanna say runhouse is an awesome project! Really gonna revolutionize how people run machine learning workflows!
Describe the bug
I'm running into an issue where I can't run any remote functions on the cluster, but I can do a cluster.run_python(...)
Here's the code I'm running:
    import runhouse as rh
    cluster = rh.OnDemandCluster(
                name="cpu-cluster",
                instance_type="CPU:8",
                provider="aws",      # options: "AWS", "GCP", "Azure", "Lambda", or "cheapest"
            )

    cluster.up_if_not()
    cluster.run_python(['import numpy', 'print(numpy.__version__)'])
    print(cluster.check_server()) # ERRORS HERE

This runs fine until the cluster.check_server(), as you can see here:
INFO | 2023-07-10 21:51:56,953 | Loaded Runhouse config from /home/shyam/.rh/config.yaml
Refreshing status for 1 cluster ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--INFO | 2023-07-10 21:51:58,623 | Found credentials in shared credentials file: ~/.aws/credentials
INFO | 2023-07-10 21:52:05,743 | Running command on cpu-cluster: python3 -c "import numpy; print(numpy.__version__)"
1.25.1
INFO | 2023-07-10 21:52:07,304 | Connected (version 2.0, client OpenSSH_8.2p1)
INFO | 2023-07-10 21:52:07,855 | Authentication (publickey) successful!
INFO | 2023-07-10 21:52:08,095 | Checking server cpu-cluster
Traceback (most recent call last):
  File "/home/shyam/Code/trainyard/examples/test.py", line 54, in <module>
    print(cluster.check_server())
  File "/home/shyam/miniconda3/envs/py310/lib/python3.10/site-packages/runhouse/rns/hardware/cluster.py", line 363, in check_server
    self.client.check_server(cluster_config=cluster_config)
  File "/home/shyam/miniconda3/envs/py310/lib/python3.10/site-packages/runhouse/servers/http/http_client.py", line 48, in check_server
    self.request(
  File "/home/shyam/miniconda3/envs/py310/lib/python3.10/site-packages/runhouse/servers/http/http_client.py", line 41, in request
    raise ValueError(
ValueError: Error calling check on server: Internal Server Error

Not sure if I'm doing something wrong here, but I think my credentials work because I can see that the cluster is being created and I can ssh into it. My package versions can be seen below, let me know if you need more information! Thanks!
Versions
Python Platform: Linux-5.8.0-36-generic-x86_64-with-glibc2.31
Python Version: 3.10.0 (default, Mar  3 2022, 09:58:08) [GCC 7.5.0]

Relevant packages:
awscli==1.25.60
azure-cli==2.31.0
azure-cli-core==2.31.0
azure-cli-telemetry==1.0.6
azure-core==1.28.0
boto3==1.24.59
docker==6.1.3
fsspec==2023.1.0
gcsfs==2023.1.0
google-api-python-client==2.92.0
google-cloud-storage==2.10.0
pyarrow==12.0.1
pycryptodome==3.12.0
rich==13.4.2
runhouse==0.0.7
s3fs==2023.1.0
skypilot==0.3.1
sshfs==2023.4.1
sshtunnel==0.4.0
typer==0.9.0
wheel==0.38.4

Checking credentials to enable clouds for SkyPilot.
  AWS: enabled
  Azure: disabled
    Reason: Azure credential is not set. Run the following commands:
      $ az login
      $ az account set -s <subscription_id>
    For more info: https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli
  GCP: disabled
    Reason: GCP tools are not installed or credentials are not set. Run the following commands:
      $ pip install google-api-python-client
      $ conda install -c conda-forge google-cloud-sdk -y
      $ gcloud init
      $ gcloud auth application-default login
    For more info: https://skypilot.readthedocs.io/en/latest/getting-started/installation.html
  Lambda: enabled
  IBM: disabled
    Reason: Missing credential file at /home/shyam/.ibm/credentials.yaml.
    Store your API key and Resource Group id in ~/.ibm/credentials.yaml in the following format:
      iam_api_key: <IAM_API_KEY>
      resource_group_id: <RESOURCE_GROUP_ID>
  Cloudflare (for R2 object store): disabled
    Reason: [r2] profile is not set in ~/.cloudflare/r2.credentials. Additionally, Account ID from R2 dashboard is not set. Run the following commands:
      $ pip install boto3
      $ AWS_SHARED_CREDENTIALS_FILE=~/.cloudflare/r2.credentials aws configure --profile r2
      $ mkdir -p ~/.cloudflare
      $ echo <YOUR_ACCOUNT_ID_HERE> > ~/.cloudflare/accountid
    For more info: https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#cloudflare-r2

SkyPilot will use only the enabled clouds to run tasks. To change this, configure cloud credentials, and run sky check.
If any problems remain, please file an issue at https://github.com/skypilot-org/skypilot/issues/new
Clusters
NAME         LAUNCHED     RESOURCES            STATUS  AUTOSTOP  COMMAND
cpu-cluster  13 mins ago  1x AWS(m6i.2xlarge)  INIT    (down)    test.py

Managed spot jobs
No in progress jobs. (See: sky spot -h)

In addition, here's the end of the setup of the cluster:
--------------------
Ray runtime started.
--------------------

Next steps
  To add another node to this Ray cluster, run
    ray start --address='<IP_ADDRESS>:6380'

  To connect to this Ray cluster:
    import ray
    ray.init()
Shared connection to <IP_ADDRESS> closed.

  To submit a Ray job using the Ray Jobs CLI:
    RAY_ADDRESS='http://<IP_ADDRESS>:8266' ray job submit --working-dir . -- python my_script.py

  See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
  for more information on submitting Ray jobs to the Ray cluster.

  To terminate the Ray runtime, run
    ray stop

  To view the status of the cluster, use
    ray status

  To monitor and debug Ray, view the dashboard at
    <IP_ADDRESS>:8266

  If connection to the dashboard fails, check your firewall settings and network configuration.
/usr/bin/prlimit
2023-07-10 21:35:36,790	INFO log_timer.py:25 -- NodeUpdater: i-036c634eb67821936: Setup commands succeeded [LogTimer=92341ms]
2023-07-10 21:35:36,791	INFO updater.py:489 -- [7/7] Starting the Ray runtime
2023-07-10 21:35:36,792	VINFO command_runner.py:371 -- Running `export RAY_USAGE_STATS_ENABLED=0;export RAY_OVERRIDE_RESOURCES='{"CPU":8}';((ps aux | grep -v nohup | grep -v grep | grep -q -- "python3 -m sky.skylet.skylet") || nohup python3 -m sky.skylet.skylet >> ~/.sky/skylet.log 2>&1 &); ray stop; RAY_SCHEDULER_EVENTS=0 RAY_DEDUP_LOGS=0 ray start --disable-usage-stats --head --port=6380 --dashboard-port=8266 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml  --temp-dir /tmp/ray_skypilot || exit 1; which prlimit && for id in $(pgrep -f raylet/raylet); do sudo prlimit --nofile=1048576:1048576 --pid=$id || true; done; python -c 'import json, os; json.dump({"ray_port":6380, "ray_dashboard_port":8266}, open(os.path.expanduser("~/.sky/ray_port.json"), "w"))';`
2023-07-10 21:35:36,792	VVINFO command_runner.py:373 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_5a4cd850fc/7112f145b3/%C -o ControlPersist=10s -o ConnectTimeout=120s<EMAIL_ADDRESS>bash --login -c -i 'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (export RAY_USAGE_STATS_ENABLED=0;export RAY_OVERRIDE_RESOURCES='"'"'{"CPU":8}'"'"';((ps aux | grep -v nohup | grep -v grep | grep -q -- "python3 -m sky.skylet.skylet") || nohup python3 -m sky.skylet.skylet >> ~/.sky/skylet.log 2>&1 &); ray stop; RAY_SCHEDULER_EVENTS=0 RAY_DEDUP_LOGS=0 ray start --disable-usage-stats --head --port=6380 --dashboard-port=8266 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml  --temp-dir /tmp/ray_skypilot || exit 1; which prlimit && for id in $(pgrep -f raylet/raylet); do sudo prlimit --nofile=1048576:1048576 --pid=$id || true; done; python -c '"'"'import json, os; json.dump({"ray_port":6380, "ray_dashboard_port":8266}, open(os.path.expanduser("~/.sky/ray_port.json"), "w"))'"'"';)'`
2023-07-10 21:35:41,238	INFO log_timer.py:25 -- NodeUpdater: i-036c634eb67821936: Ray start commands succeeded [LogTimer=4447ms]
2023-07-10 21:35:41,238	INFO log_timer.py:25 -- NodeUpdater: i-036c634eb67821936: Applied config f62a597a450a8281871e7ace3caa155afb5dfe65  [LogTimer=183192ms]
2023-07-10 21:35:42,755	INFO log_timer.py:25 -- AWSNodeProvider: Set tag ray-node-status=up-to-date on ['i-036c634eb67821936']  [LogTimer=515ms]
2023-07-10 21:35:42,925	INFO log_timer.py:25 -- AWSNodeProvider: Set tag ray-runtime-config=f62a597a450a8281871e7ace3caa155afb5dfe65 on ['i-036c634eb67821936']  [LogTimer=170ms]
2023-07-10 21:35:43,090	INFO log_timer.py:25 -- AWSNodeProvider: Set tag ray-file-mounts-contents=24403a03b3acb79e10305dbf19904b00a057a0a1 on ['i-036c634eb67821936']  [LogTimer=165ms]
2023-07-10 21:35:43,091	INFO updater.py:188 -- New status: up-to-date
2023-07-10 21:35:43,273	INFO commands.py:836 -- Useful commands
2023-07-10 21:35:43,273	INFO commands.py:838 -- Monitor autoscaling with
2023-07-10 21:35:43,274	INFO commands.py:839 --   ray exec /home/shyam/.sky/generated/cpu-cluster.yml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*'
2023-07-10 21:35:43,274	INFO commands.py:846 -- Connect to a terminal on the cluster head:
2023-07-10 21:35:43,274	INFO commands.py:847 --   ray attach /home/shyam/.sky/generated/cpu-cluster.yml
2023-07-10 21:35:43,274	INFO commands.py:850 -- Get a remote shell to the cluster manually:
2023-07-10 21:35:43,274	INFO commands.py:851 --   ssh -o IdentitiesOnly=yes -i ~/.ssh/sky-key<EMAIL_ADDRESS>

After configuring gcp correctly, looks like this works when specifying "gcp" in provider:
INFO | 2023-07-10 22:37:35,284 | Loaded Runhouse config from /home/shyam/.rh/config.yaml
INFO | 2023-07-10 22:37:44,473 | Running command on cpu-cluster: python3 -c "import numpy; print(numpy.__version__)"
1.21.6
INFO | 2023-07-10 22:37:46,311 | Connected (version 2.0, client OpenSSH_7.9p1)
INFO | 2023-07-10 22:37:46,765 | Authentication (publickey) successful!
INFO | 2023-07-10 22:37:46,767 | Checking server cpu-cluster
INFO | 2023-07-10 22:37:48,945 | Server cpu-cluster is up.
None

Hi! Thank you for the detailed bug report. I actually noticed some new breakage this evening relating to the FastAPI 0.100.0 release, but given it's working on GCP I'm not sure that's it. Trying to reproduce on AWS now.

I was able to reproduce this and it is indeed an issue with the latest fastapi release (you can confirm by sshing into the server, viewing the server logs at ~/.rh/cluster_server_cpu-cluster.log, and confirming that the latest exception is a pydantic issue. The latest fastapi release was a massive update to Pydantic v2). I'm pushing a fix to main and releasing presently.

Just pushed https://github.com/run-house/runhouse/releases/tag/v0.0.8, which should fix this.

Awesome! Thanks so much!

@shyamsn97 confirming that the fix worked?

Yep seems like that worked and now I'm not getting an error on check_server. However the function is hanging for aws hangs indefinitely.
code from example:
import runhouse as rh

def num_cpus():
    import multiprocessing
    return f"Num cpus: {multiprocessing.cpu_count()}"

num_cpus()

# Using a Cloud provider
cluster = rh.ondemand_cluster(
              name="runhouse",
              instance_type="CPU:8",
              provider="aws",      # options: "AWS", "GCP", "Azure", "Lambda", or "cheapest"
          )

cluster.up_if_not()

num_cpus_cluster = rh.function(name="num_cpus_cluster", fn=num_cpus).to(system=cluster, reqs=["./"])

num_cpus_cluster()

Output:
(py310) shyam@shyam-ThinkPad-P53:~/Code/test-runhouse$ python test.py
INFO | 2023-07-11 11:50:06,755 | Loaded Runhouse config from /home/shyam/.rh/config.yaml
INFO | 2023-07-11 11:50:07,937 | Found credentials in shared credentials file: ~/.aws/credentials
I 07-11 11:50:08 optimizer.py:636] == Optimizer ==
I 07-11 11:50:08 optimizer.py:647] Target: minimizing cost
I 07-11 11:50:08 optimizer.py:659] Estimated cost: $0.4 / hour
I 07-11 11:50:08 optimizer.py:659]
I 07-11 11:50:08 optimizer.py:732] Considered resources (1 node):
I 07-11 11:50:08 optimizer.py:781] ------------------------------------------------------------------------------------------
I 07-11 11:50:08 optimizer.py:781]  CLOUD   INSTANCE      vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN
I 07-11 11:50:08 optimizer.py:781] ------------------------------------------------------------------------------------------
I 07-11 11:50:08 optimizer.py:781]  AWS     m6i.2xlarge   8       32        -              us-east-1     0.38          ✔
I 07-11 11:50:08 optimizer.py:781] ------------------------------------------------------------------------------------------
I 07-11 11:50:08 optimizer.py:781]
I 07-11 11:50:08 cloud_vm_ray_backend.py:3495] Creating a new cluster: "runhouse" [1x AWS(m6i.2xlarge)].
I 07-11 11:50:08 cloud_vm_ray_backend.py:3495] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 07-11 11:50:08 cloud_vm_ray_backend.py:1215] To view detailed progress: tail -n100 -f /home/shyam/sky_logs/sky-2023-07-11-11-50-07-852797/provision.log
I 07-11 11:50:09 cloud_vm_ray_backend.py:1539] Launching on AWS us-east-1 (us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1f)
I 07-11 11:51:42 log_utils.py:89] Head node is up.
I 07-11 11:53:34 cloud_vm_ray_backend.py:1352] Successfully provisioned or found existing VM.
I 07-11 11:53:41 cloud_vm_ray_backend.py:3544] Processing file mounts.
I 07-11 11:53:41 cloud_vm_ray_backend.py:3575] To view detailed progress: tail -n100 -f ~/sky_logs/sky-2023-07-11-11-50-07-852797/file_mounts.log
I 07-11 11:53:41 backend_utils.py:1254] Syncing (to 1 node): ~/.rh -> ~/.rh
I 07-11 11:53:44 cloud_vm_ray_backend.py:2808] Run commands not specified or empty.
Clusters
NAME         LAUNCHED        RESOURCES              STATUS  AUTOSTOP  COMMAND
runhouse     a few secs ago  1x AWS(m6i.2xlarge)    UP      (down)    test.py
cpu-cluster  20 mins ago     1x GCP(n2-standard-8)  UP      (down)    test.py

INFO | 2023-07-11 11:53:51,543 | Restarting HTTP server on runhouse.
INFO | 2023-07-11 11:53:51,543 | Running command on runhouse: pip install runhouse==0.0.8Collecting runhouse==0.0.8
  Downloading runhouse-0.0.8-py3-none-any.whl (142 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.7/142.7 kB 3.5 MB/s eta 0:00:00
Collecting sshfs
  Downloading sshfs-2023.4.1-py3-none-any.whl (15 kB)
Collecting typer
  Downloading typer-0.9.0-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.9/45.9 kB 2.6 MB/s eta 0:00:00
Collecting fsspec
  Downloading fsspec-2023.6.0-py3-none-any.whl (163 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 7.9 MB/s eta 0:00:00
Collecting uvicorn
  Downloading uvicorn-0.22.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 4.4 MB/s eta 0:00:00
Collecting sshtunnel>=0.3.0
  Downloading sshtunnel-0.4.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: pyOpenSSL>=21.1.0 in /opt/conda/lib/python3.10/site-packages (from runhouse==0.0.8) (22.1.0)
Collecting fastapi<=0.99.0
  Downloading fastapi-0.99.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.2/58.2 kB 321.7 kB/s eta 0:00:00
Collecting pyarrow
  Downloading pyarrow-12.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.9/38.9 MB 13.5 MB/s eta 0:00:00
Requirement already satisfied: rich in /opt/conda/lib/python3.10/site-packages (from runhouse==0.0.8) (13.4.2)
Requirement already satisfied: wheel in /opt/conda/lib/python3.10/site-packages (from runhouse==0.0.8) (0.38.4)
Requirement already satisfied: skypilot==0.3.1 in /opt/conda/lib/python3.10/site-packages (from runhouse==0.0.8) (0.3.1)
Requirement already satisfied: filelock>=3.6.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (3.12.2)
Requirement already satisfied: pulp in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (2.7.0)
Requirement already satisfied: click<=8.0.4,>=7.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (8.0.4)
Requirement already satisfied: pycryptodome==3.12.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (3.12.0)
Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (3.1)
Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (2.0.3)
Requirement already satisfied: grpcio<=1.51.3,>=1.42.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (1.51.3)
Requirement already satisfied: awscli in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (1.29.2)
Requirement already satisfied: oauth2client in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (4.1.3)
Requirement already satisfied: pendulum in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (2.1.2)
Requirement already satisfied: psutil in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (5.9.5)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (0.9.0)
Requirement already satisfied: cryptography in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (38.0.3)
Requirement already satisfied: colorama<0.4.5 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (0.4.4)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (1.28.2)
Requirement already satisfied: ray[default]<=2.4.0,>=2.2.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (2.4.0)
Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (4.23.4)
Requirement already satisfied: PrettyTable>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (3.8.0)
Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (23.1)
Requirement already satisfied: jsonschema in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (4.18.0)
Requirement already satisfied: jinja2>=3.0 in /opt/conda/lib/python3.10/site-packages (from skypilot==0.3.1->runhouse==0.0.8) (3.1.2)
Requirement already satisfied: typing-extensions>=4.5.0 in /opt/conda/lib/python3.10/site-packages (from fastapi<=0.99.0->runhouse==0.0.8) (4.7.1)
Collecting starlette<0.28.0,>=0.27.0
  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.0/67.0 kB 588.7 kB/s eta 0:00:00
Collecting pydantic!=1.8,!=1.8.1,<2.0.0,>=1.7.4
  Downloading pydantic-1.10.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 16.3 MB/s eta 0:00:00
Collecting paramiko>=2.7.2
  Downloading paramiko-3.2.0-py3-none-any.whl (224 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.2/224.2 kB 30.5 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.16.6 in /opt/conda/lib/python3.10/site-packages (from pyarrow->runhouse==0.0.8) (1.25.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/lib/python3.10/site-packages (from rich->runhouse==0.0.8) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/lib/python3.10/site-packages (from rich->runhouse==0.0.8) (2.15.1)
Collecting asyncssh<3,>=2.11.0
  Downloading asyncssh-2.13.2-py3-none-any.whl (349 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 349.3/349.3 kB 2.8 MB/s eta 0:00:00
Collecting h11>=0.8
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 11.9 MB/s eta 0:00:00
Requirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.10/site-packages (from cryptography->skypilot==0.3.1->runhouse==0.0.8) (1.15.1)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2>=3.0->skypilot==0.3.1->runhouse==0.0.8) (2.1.3)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->runhouse==0.0.8) (0.1.2)
Collecting pynacl>=1.5
  Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 856.7/856.7 kB 11.9 MB/s eta 0:00:00
Collecting bcrypt>=3.2
  Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 593.7/593.7 kB 67.6 MB/s eta 0:00:00
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.10/site-packages (from PrettyTable>=2.0.0->skypilot==0.3.1->runhouse==0.0.8) (0.2.6)
Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (2.28.1)
Requirement already satisfied: aiosignal in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.3.1)
Requirement already satisfied: virtualenv<20.21.1,>=20.0.24 in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (20.21.0)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.0.5)
Requirement already satisfied: frozenlist in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.3.3)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (5.4.1)
Requirement already satisfied: attrs in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (23.1.0)
Requirement already satisfied: aiohttp>=3.7 in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (3.8.4)
Requirement already satisfied: aiohttp-cors in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.7.0)
Requirement already satisfied: opencensus in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.11.2)
Requirement already satisfied: smart-open in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (6.3.0)
Requirement already satisfied: prometheus-client>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.17.1)
Requirement already satisfied: colorful in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.5.5)
Requirement already satisfied: gpustat>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.1)
Requirement already satisfied: py-spy>=0.2.0 in /opt/conda/lib/python3.10/site-packages (from ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.3.14)
Collecting anyio<5,>=3.4.0
  Downloading anyio-3.7.1-py3-none-any.whl (80 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.9/80.9 kB 16.7 MB/s eta 0:00:00
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from awscli->skypilot==0.3.1->runhouse==0.0.8) (0.6.1)
Requirement already satisfied: docutils<0.17,>=0.10 in /opt/conda/lib/python3.10/site-packages (from awscli->skypilot==0.3.1->runhouse==0.0.8) (0.16)
Requirement already satisfied: rsa<4.8,>=3.1.2 in /opt/conda/lib/python3.10/site-packages (from awscli->skypilot==0.3.1->runhouse==0.0.8) (4.7.2)
Requirement already satisfied: botocore==1.31.2 in /opt/conda/lib/python3.10/site-packages (from awscli->skypilot==0.3.1->runhouse==0.0.8) (1.31.2)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from botocore==1.31.2->awscli->skypilot==0.3.1->runhouse==0.0.8) (1.0.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.10/site-packages (from botocore==1.31.2->awscli->skypilot==0.3.1->runhouse==0.0.8) (1.26.11)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.10/site-packages (from botocore==1.31.2->awscli->skypilot==0.3.1->runhouse==0.0.8) (2.8.2)
Requirement already satisfied: referencing>=0.28.4 in /opt/conda/lib/python3.10/site-packages (from jsonschema->skypilot==0.3.1->runhouse==0.0.8) (0.29.1)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/conda/lib/python3.10/site-packages (from jsonschema->skypilot==0.3.1->runhouse==0.0.8) (2023.6.1)
Requirement already satisfied: rpds-py>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from jsonschema->skypilot==0.3.1->runhouse==0.0.8) (0.8.10)
Requirement already satisfied: six>=1.6.1 in /opt/conda/lib/python3.10/site-packages (from oauth2client->skypilot==0.3.1->runhouse==0.0.8) (1.16.0)
Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.10/site-packages (from oauth2client->skypilot==0.3.1->runhouse==0.0.8) (0.5.0)
Requirement already satisfied: httplib2>=0.9.1 in /opt/conda/lib/python3.10/site-packages (from oauth2client->skypilot==0.3.1->runhouse==0.0.8) (0.22.0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /opt/conda/lib/python3.10/site-packages (from oauth2client->skypilot==0.3.1->runhouse==0.0.8) (0.3.0)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->skypilot==0.3.1->runhouse==0.0.8) (2023.3)
Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas->skypilot==0.3.1->runhouse==0.0.8) (2023.3)
Requirement already satisfied: pytzdata>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pendulum->skypilot==0.3.1->runhouse==0.0.8) (2020.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp>=3.7->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.10/site-packages (from aiohttp>=3.7->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp>=3.7->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.9.2)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp>=3.7->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (2.1.1)
Collecting exceptiongroup
  Downloading exceptiongroup-1.1.2-py3-none-any.whl (14 kB)
Collecting sniffio>=1.1
  Downloading sniffio-1.3.0-py3-none-any.whl (10 kB)
Requirement already satisfied: idna>=2.8 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<=0.99.0->runhouse==0.0.8) (3.4)
Requirement already satisfied: pycparser in /opt/conda/lib/python3.10/site-packages (from cffi>=1.12->cryptography->skypilot==0.3.1->runhouse==0.0.8) (2.21)
Requirement already satisfied: nvidia-ml-py>=11.450.129 in /opt/conda/lib/python3.10/site-packages (from gpustat>=1.0.0->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (12.535.77)
Requirement already satisfied: blessed>=1.17.1 in /opt/conda/lib/python3.10/site-packages (from gpustat>=1.0.0->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.20.0)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /opt/conda/lib/python3.10/site-packages (from httplib2>=0.9.1->oauth2client->skypilot==0.3.1->runhouse==0.0.8) (3.1.0)
Requirement already satisfied: distlib<1,>=0.3.6 in /opt/conda/lib/python3.10/site-packages (from virtualenv<20.21.1,>=20.0.24->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.3.6)
Requirement already satisfied: platformdirs<4,>=2.4 in /opt/conda/lib/python3.10/site-packages (from virtualenv<20.21.1,>=20.0.24->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (3.8.1)
Requirement already satisfied: opencensus-context>=0.1.3 in /opt/conda/lib/python3.10/site-packages (from opencensus->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (0.1.3)
Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from opencensus->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (2.11.1)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (2022.9.24)
Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /opt/conda/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (2.22.0)
Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (1.59.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]<=2.4.0,>=2.2.0->skypilot==0.3.1->runhouse==0.0.8) (5.3.1)
Installing collected packages: typer, sniffio, pydantic, pyarrow, h11, fsspec, exceptiongroup, bcrypt, uvicorn, pynacl, anyio, starlette, paramiko, asyncssh, sshtunnel, sshfs, fastapi, runhouse
  Attempting uninstall: pydantic
    Found existing installation: pydantic 2.0.2
    Uninstalling pydantic-2.0.2:
      Successfully uninstalled pydantic-2.0.2
Successfully installed anyio-3.7.1 asyncssh-2.13.2 bcrypt-4.0.1 exceptiongroup-1.1.2 fastapi-0.99.0 fsspec-2023.6.0 h11-0.14.0 paramiko-3.2.0 pyarrow-12.0.1 pydantic-1.10.11 pynacl-1.5.0 runhouse-0.0.8 sniffio-1.3.0 sshfs-2023.4.1 sshtunnel-0.4.0 starlette-0.27.0 typer-0.9.0 uvicorn-0.22.0
INFO | 2023-07-11 11:53:58,889 | Running command on runhouse: pkill -f "python -m runhouse.servers.http.http_server"
INFO | 2023-07-11 11:54:00,015 | Running command on runhouse: pkill -f ".*ray.*6379.*"
INFO | 2023-07-11 11:54:01,244 | Running command on runhouse: ray start --head --port 6379 --autoscaling-config=~/ray_bootstrap_config.yaml
2023-07-11 18:54:03,331	INFO usage_lib.py:398 -- Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.
2023-07-11 18:54:03,331	INFO scripts.py:710 -- Local node IP: <IP_ADDRESS>
2023-07-11 18:54:05,396	SUCC scripts.py:747 -- --------------------
2023-07-11 18:54:05,397	SUCC scripts.py:748 -- Ray runtime started.
2023-07-11 18:54:05,397	SUCC scripts.py:749 -- --------------------
2023-07-11 18:54:05,397	INFO scripts.py:751 -- Next steps
2023-07-11 18:54:05,397	INFO scripts.py:754 -- To add another node to this Ray cluster, run
2023-07-11 18:54:05,397	INFO scripts.py:757 --   ray start --address='<IP_ADDRESS>:6379'
2023-07-11 18:54:05,397	INFO scripts.py:766 -- To connect to this Ray cluster:
2023-07-11 18:54:05,397	INFO scripts.py:768 -- import ray
2023-07-11 18:54:05,397	INFO scripts.py:769 -- ray.init()
2023-07-11 18:54:05,397	INFO scripts.py:781 -- To submit a Ray job using the Ray Jobs CLI:
2023-07-11 18:54:05,397	INFO scripts.py:782 --   RAY_ADDRESS='http://<IP_ADDRESS>:8265' ray job submit --working-dir . -- python my_script.py
2023-07-11 18:54:05,397	INFO scripts.py:791 -- See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
2023-07-11 18:54:05,397	INFO scripts.py:795 -- for more information on submitting Ray jobs to the Ray cluster.
2023-07-11 18:54:05,397	INFO scripts.py:800 -- To terminate the Ray runtime, run
2023-07-11 18:54:05,397	INFO scripts.py:801 --   ray stop
2023-07-11 18:54:05,397	INFO scripts.py:804 -- To view the status of the cluster, use
2023-07-11 18:54:05,397	INFO scripts.py:805 --   ray status
2023-07-11 18:54:05,397	INFO scripts.py:809 -- To monitor and debug Ray, view the dashboard at
2023-07-11 18:54:05,397	INFO scripts.py:810 --   <IP_ADDRESS>:8265
2023-07-11 18:54:05,397	INFO scripts.py:817 -- If connection to the dashboard fails, check your firewall settings and network configuration.
INFO | 2023-07-11 11:54:05,750 | Running command on runhouse: screen -dm bash -c 'python -m runhouse.servers.http.http_server |& tee -a ~/.rh/cluster_server_runhouse.log 2>&1'
/home/shyam/miniconda3/envs/py310/lib/python3.10/site-packages/runhouse/rns/function.py:113: UserWarning: ``reqs`` and ``setup_cmds`` arguments has been deprecated. Please use ``env`` instead.
  warnings.warn(
INFO | 2023-07-11 11:54:12,193 | Setting up Function on cluster.
INFO | 2023-07-11 11:54:12,564 | Connected (version 2.0, client OpenSSH_8.2p1)
INFO | 2023-07-11 11:54:13,099 | Authentication (publickey) successful!
INFO | 2023-07-11 11:54:13,108 | Checking server runhouse
INFO | 2023-07-11 11:54:14,768 | Server runhouse is up.
INFO | 2023-07-11 11:54:14,914 | Connected (version 2.0, client OpenSSH_8.2p1)
INFO | 2023-07-11 11:54:14,928 | Authentication (publickey) failed.
INFO | 2023-07-11 11:54:14,934 | Connected (version 2.0, client OpenSSH_8.2p1)
INFO | 2023-07-11 11:54:14,949 | Authentication (publickey) successful!
2023-07-11 11:54:14,949| ERROR   | Problem setting SSH Forwarder up: Couldn't open tunnel :50052 <> <IP_ADDRESS>:50052 might be in use or destination not reachable
ERROR | 2023-07-11 11:54:14,949 | Problem setting SSH Forwarder up: Couldn't open tunnel :50052 <> <IP_ADDRESS>:50052 might be in use or destination not reachable
INFO | 2023-07-11 11:54:15,083 | Connected (version 2.0, client OpenSSH_8.2p1)
INFO | 2023-07-11 11:54:15,097 | Authentication (publickey) failed.
INFO | 2023-07-11 11:54:15,103 | Connected (version 2.0, client OpenSSH_8.2p1)
INFO | 2023-07-11 11:54:15,118 | Authentication (publickey) successful!
INFO | 2023-07-11 11:54:15,119 | Checking server local-cluster
INFO | 2023-07-11 11:54:16,445 | Server local-cluster is up.
INFO | 2023-07-11 11:54:16,445 | Running command on local-cluster: ray start --head
INFO | 2023-07-11 11:54:17,903 | Copying folder from file:///home/shyam/Code/test-runhouse to: runhouse
INFO | 2023-07-11 11:54:19,653 | Installing packages on cluster runhouse: ['Package: test-runhouse']
INFO | 2023-07-11 11:54:20,726 | Function setup complete.
INFO | 2023-07-11 11:54:20,731 | Running num_cpus_cluster via HTTP
INFO | 2023-07-11 11:54:21,326 | Submitted remote call to cluster for num_cpus_cluster_20230711_115420
:job_id:01000000
:task_name:get_fn_from_pointers
:job_id:01000000
INFO | 2023-07-11 18:54:21,732 | Loaded Runhouse config from /home/ubuntu/.rh/config.yaml
:task_name:get_fn_from_pointers
INFO | 2023-07-11 18:54:22,423 | Appending /home/ubuntu/test-runhouse to sys.path
INFO | 2023-07-11 18:54:22,424 | Importing module test
SkyPilot collects usage data to improve its services. `setup` and `run` commands are not collected to ensure privacy.
Usage logging can be disabled by setting the environment variable SKYPILOT_DISABLE_USAGE_COLLECTION=1.
INFO | 2023-07-11 18:54:22,944 | Found credentials in shared credentials file: ~/.aws/credentials
I 07-11 18:54:23 aws_catalog.py:120] Fetching availability zones mapping for AWS...
I 07-11 18:54:25 optimizer.py:636] == Optimizer ==
I 07-11 18:54:25 optimizer.py:647] Target: minimizing cost
I 07-11 18:54:25 optimizer.py:659] Estimated cost: $0.4 / hour
I 07-11 18:54:25 optimizer.py:659]
I 07-11 18:54:25 optimizer.py:732] Considered resources (1 node):
I 07-11 18:54:25 optimizer.py:781] ------------------------------------------------------------------------------------------
I 07-11 18:54:25 optimizer.py:781]  CLOUD   INSTANCE      vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN
I 07-11 18:54:25 optimizer.py:781] ------------------------------------------------------------------------------------------
I 07-11 18:54:25 optimizer.py:781]  AWS     m6i.2xlarge   8       32        -              us-east-1     0.38          ✔
I 07-11 18:54:25 optimizer.py:781] ------------------------------------------------------------------------------------------
I 07-11 18:54:25 optimizer.py:781]
I 07-11 18:54:25 cloud_vm_ray_backend.py:3495] Creating a new cluster: "runhouse" [1x AWS(m6i.2xlarge)].
I 07-11 18:54:25 cloud_vm_ray_backend.py:3495] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 07-11 18:54:25 cloud_vm_ray_backend.py:1215] To view detailed progress: tail -n100 -f /home/ubuntu/sky_logs/sky-2023-07-11-18-54-22-466640/provision.log
I 07-11 18:54:26 cloud_vm_ray_backend.py:1539] Launching on AWS us-east-1 (us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1f)

weird because it tries to start the cluster twice? Unless I'm misunderstanding the process there

I see! Will try that now!

That worked! Thanks! This is good to close now :)

@dongreenberg i'm running into this issue but require fastAPI > 0.1 and Pydantic 2.x. How easy would it be to support these with the latest version of Runhouse (which seems to fix this issue)?

This is fixed. I've tested with a few of the new versions of fastapi and pydantic on main and something changed about our code, their code, or both such that they now work out of the box. I've relaxed the requirement on main, which should be in the release today.


Generating pgm/yaml map

Since the output of the 'finish_trajectory' service has changed, I'm unable to generate .pgm/.yaml map files for other nodes (non-Cartographer) to use.
I'm aware of the 'write_state' service which provides a .pbstream file, but is it still possible to write to a .pgm/.yaml file?

No, that is not yet supported by the asset writer pipeline. It should be resurrected though as it's useful for integration with other ROS nodes.

ROS already  has  map_server can do it.
See   http://wiki.ros.org/map_server   ,  and
rosrun map_server map_saver -f mymap

@damonkohler Thank you, i agree it can be useful. I used the previously integrated .pgm generation for this purpose.
@zyyfly2010 I'm aware of the ROS map_server and map_saver. Unfortunately, it doesn't properly recognize the map message on the /map topic from Cartographer. The output only contains free and unknown cells, occupied cells seem to be missing. Or i am doing something wrong?

I had to manually make changes to the map_saver source code in order to save values over the entire range of probabilities instead of just "fully occupied", "not occupied at all", and "unknown".

I also have troubled with the same problem.Map_saver will convert it to three values without permission.
I would like you to refer to my modification.
map_saver.cpp before
https://github.com/ros-planning/navigation/blob/kinetic-devel/map_server/src/map_saver.cpp
for(unsigned int y = 0; y < map->info.height; y++) {
  for(unsigned int x = 0; x < map->info.width; x++) {
    unsigned int i = x + (map->info.height - y - 1) * map->info.width;
    if (map->data[i] == 0) { //occ [0,0.1)
      fputc(254, out);
    } else if (map->data[i] == +100) { //occ (0.65,1]
      fputc(000, out);
    } else { //occ [0.1,0.65]
      fputc(205, out);
    }
  }
}

after
for(unsigned int y = 0; y < map->info.height; y++) {
  for(unsigned int x = 0; x < map->info.width; x++) {
    unsigned int i = x + (map->info.height - y - 1) * map->info.width;
    int pixel = (double)(100.0-map->data[i]) *2.54;
    fputc(pixel, out2);
  }
}

Yup, that's pretty much what I did - except additionally I mapped -1 to 128
I also changed https://github.com/ros-planning/navigation/blob/kinetic-devel/map_server/src/map_saver.cpp#L114 to print the same thresholds (0.49 and 0.51) that Cartographer used to use, instead of 0.19 and 0.65

Pretty much how i temporarily solved it for myself: https://github.com/rsmallegoor/navigation/blob/kinetic-devel/map_server/src/map_saver.cpp#L77#L88
But I'll probably look into resurrecting the code that previously did this in the asset writer if desired.

If anyone else needs it, #436 (which removed this) is still revertable on top of the current head without (much) pain and it works fine:
https://github.com/larics/cartographer_ros/tree/restore-assets-writer
Would the maintainers (@damonkohler @wohe) consider the possibility of retaining this functionality? I find the ability to directly generate a high-quality occupancy grid from the optimized trajectory very useful; I can then run some additional optimizations, and regenerate the occupancy grid to see the results.
If I remember correctly, this clashes with your desire to get rid of the scan data in the trajectory nodes, which is needed for building the occupancy grid? Perhaps we could find a middle ground, e.g. make that toggleable - if the user doesn't need the data, it is not kept around for increased performance.
Another idea would be to move this into the occupancy grid node and give it the ability to query the SLAM node for full trajectory data (and let the user choose whether to build the grid by fusing submaps or replaying the trajectory). This seems somewhat aligned with the direction you're taking, but involves lots of unnecessary moving the data around.

To be clear and as @damonkohler pointed out we are absolutely in favor of having this functionality. It was implemented in the wrong place though and moving it to the right place reduces memory requirements per time, i.e. longer SLAM using the same amount of memory.
Is there a requirement that the higher quality occupancy grid is available in real-time? If not, this functionality should be added to the assets writer. If someone can make the case for having this in real-time, we could discuss whether the occupancy grid node is the right place to add it.

To be clear and as @damonkohler pointed out we are absolutely in favor of having this functionality.

Of course. I was specifically referring to keeping the functionality of generating the higher quality occupancy grid in real time. My use case is that I am asynchronously pushing aditional data into the node and re-running optimization. The real-time asset writer functionality enables me to preview how this has affected the final high-quality map (rather than the approximation obtained by fusing the submaps, although that is useful too). I can then push additional data, rerun optimization, preview again, etc.

moving it to the right place reduces memory requirements per time, i.e. longer SLAM using the same amount of memory.

In order to generate the high quality map, you still need to somehow have available each trajectory node's point cloud data so you can replay it and do mapping with known poses, right? If you don't want to store it in memory, you either have to immediately serialize it, or somehow reference the right scan from the bag (although this is not an option with the online node).

We are also currently using a forked version that retains occupancy grid generation to simplify the process and to obtain a better map. Probability grid asset writer doesn't seem to generate a map as good as the old one.

@jihoonl, what "probability grid asset writer" are you referring to? AFAIK it doesn't exist.

I meant ProbabilityGridPointsProcessor. The output png file didn't seem as good as pgm file generated before.

I see. We're still missing pgm/yaml output from the asset writer. However, it seems rationale to start by ensuring that the output will be good quality1
The ProbabilityGridPointsProcessor does not do unwarping. Perhaps that's the issue?

Um.. what do you mean unwarping?

@jihoonl In the local trajectory builders, we call it scan accumulation: https://github.com/googlecartographer/cartographer/blob/ba6f782949eb7cdb96e43c178c7b6ef83bd57215/cartographer/mapping_2d/local_trajectory_builder.cc#L84
We chunk up the scans and then stitch them back together according to pose extrapolation (i.e. constant velocity, odometry, and IMU).
That said, unwarping is relatively new to 2D SLAM. How long have you seen a quality difference in the results of the asset writer as compared to the old-style occupancy grid generation?

I had a quick test with the version without scan accumulation(cartographer:3ef680e7914733a1fd15bff784f2217424cc75b5 and cartographer_ros:b12e32d6fc03876dd8f5c93acb10f060de75221d). And I was still able to observe the quality difference.
Removed walls

No dilusion on human footprint

This is our asset writer configuration
options = {
  tracking_frame = "base_footprint",
  pipeline = {
    {
      action = "min_max_range_filter",
      min_range = 0.0,
      max_range = 8.,
    },
    {
      action = "write_probability_grid",
      -- draw_trajectories = false,
      range_data_inserter = {
        insert_free_space = true,
        hit_probability = 0.55,
        miss_probability = 0.49,
      },
      filename = "output",
      resolution = 0.05,
      -- export_meta_data = true
    }
  }
}
return options

Am I misconfiguring something?

@jihoonl, @wohe and I discussed this and think it's likely due to the insertion of scans at interpolated positions. If you add an option to only insert the scan that is nearest to each trajectory node, it should solve your problem.
A more complicated approach would be to use an optimization problem to interpolate between trajectory nodes using IMU and odometry data.

There are at least three issues discussed in this bug. I'd like to refocus this back to the original issue in the title.
@cschuet and I will pair on a PGM and YAML writing points processor that will live in cartographer_ros and bring back the ability to generate ROS compatible maps from Cartographer runs using the assets_writer.
@ojura This will not solve your online use case, which has nothing to do with PGM and YAML and seems to require a bit more discussion and probably design work. If you feel this is worth investigation and further discussion, could you open a new bug against Cartographer to focus the discussion there?
The quality difference between rViz and the asset writer is pulled out into #521.


⚠ Bofrost.at has degraded performance

In d0ec8f9, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1863 ms

Resolved: Bofrost.at performance has improved in d009cd8 after 13 minutes.


Failed to disable authentication

Type of issue

[x] Bug
[ ] New feature
[ ] Improvement

Description
Failed to disable authentication following https://github.com/Azure/azure-iot-pcs-remote-monitoring-dotnet/wiki/Developer-Reference-Guide#disable-authentication. Seems the curl command _keyvault_secret_bundle=$(curl -X PUT -H "Content-Type: application/json" -H "Authorization: Bearer $AUTH_TOKEN" -d "{"value": $2}"  -L https://$PCS_KEYVAULT_NAME.vault.azure.net/secrets/corsWhitelist/?api-version=7.0) in the /app/auth.sh giving an error like

{"error":{"code":"BadParameter","message":"Property value has invalid value\r\nProperty value has invalid value\r\n"}}

Screenshot

Steps to reproduce

sudo /app/start.sh --unsafe
POST request using postman https://xxxxxxx.azurewebsites.net/telemetry/v1/alarms

Expected behavior
Current behavior
Known workarounds
Possible solution
Need to fix /app/auth.sh.
Context and Environment

Operating System: ...
GitHub branch: ...
.NET Runtime: ...

I am having the same issue here, @ritwikgiri did you solved it?

I am having the same issue here, @ritwikgiri did you solved it?

No still not resolved..

i found a problem with the script. in set_env_vars function, when is calling the _set_keyvault_secret, the second parameter must start and finish with a " becouse this parameter is used as a value in a JSON parameter
set_env_vars() {
    # parse through all variables (Every odd variable is env var name & even variables are secret key names in Key vault).
    if [ "$1" == "enable" ]; then
        _set_keyvault_secret "true" "\"\""
    else
        _set_keyvault_secret "false" "\"{ 'origins': ['*'], 'methods': ['*'], 'headers': ['*'] }\""
    fi
}

Submitted what I think is the same issue on the dotnet microservices repo https://github.com/Azure/remote-monitoring-services-dotnet/issues/250 but got no response. Tried playing around with the start.sh script but didn't manage to fix this.

Note: As of December 10th 2020, Remote Monitoring solution accelerator is no longer supported. All supporting repositories have been archived.


Don't require brkt-env when launching a GCE instance

Split BracketEnvironment parsing out from the rest of the values passed
when constructing InstanceConfig.  This allows callsites to determine
whether to specify BracketEnvironment or not.
Background: in commit bfe70c372f95730964a0f9c3366aa94a8806cf64 we
started always sending the Yeti environment to Metavisor.  This created
problems when launching a GCE image.  If the user didn't specify the
environment on the command line, we'd send the prod endpoints, which
would inadvertently override the Metavisor defaults.

in its current form this change doesn't fix the problem


Test minified build

After having problems with the minification configuration (the minification produced invalid code), I think we may add some form of testing at some level.

Now we are using the non-minified bundle to pass the tests. We could use the minified bundle for the tests or even pass the tests with both bundle files.

Hi. I have created a PR with this approach: https://github.com/CartoDB/carto-vl/pull/812

Closed by https://github.com/CartoDB/carto-vl/pull/812


Add NuGetizer to the ecosystem links

NuGetizer is a pretty popular alternative to SDK Pack (~200k downloads on nuget.org at this time). Recognize it by linking from the docs.

Thanks for the contribution, @kzu!


a_teammate/ao_fix

i'll merge that 2 master soon as well (today or if someone vetos tomorrow the latest)
its basicly just a fix (a crash caused when calclighting) + making ambient occlusion shadows antialiased

merge tmrw, no further changes planned

Plase squash your fix commits

Please also rebase your commits onto the latest master

merged


xref issues on acyl-CoA hydrolase activity child terms

id: GO:0052817
name: very long chain acyl-CoA hydrolase activity
def: "Catalysis of the reaction: H2O + a very long chain acyl-CoA = a very long chain carboxylate + CoA. A very long chain has chain length greater than C18." [EC:<IP_ADDRESS>]
synonym: "very long-chain acyl coenzyme A hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "very long-chain acyl-thioester hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "very long-chain hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "very long-chain-acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
is_a: GO:0047617 ! acyl-CoA hydrolase activity

[x] Add RHEA:67072
[x] change def and name to match RHEA:67072 (and update chain length def as per #26445)
[x] replace EC:<IP_ADDRESS> def xref with RHEA:67072 (see GO:0047380 below)
[x] Remove synonyms attribution to EC:<IP_ADDRESS> (see GO:0047380 below)

id: GO:0052816
name: long-chain acyl-CoA hydrolase activity
namespace: molecular_function
def: "Catalysis of the reaction: H2O + a long-chain acyl-CoA = a long-chain carboxylate + CoA. A long chain is a chain of greater than 12 carbons in length." [EC:<IP_ADDRESS>]
synonym: "long-chain acyl coenzyme A hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "long-chain acyl-thioester hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "long-chain hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "long-chain-acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
is_a: GO:0047617 ! acyl-CoA hydrolase activity
created_by: ai
creation_date: 2011-10-25T05:31:11Z

[x] Add RHEA:67680
[x] change def and name to match RHEA:67680 (and update chain length def as per #26445)
[x] replace EC:<IP_ADDRESS> def xref with RHEA:67680 (see GO:0047380 below)
[x] Remove synonyms attribution to EC:<IP_ADDRESS> (see GO:0047380 below)

id: GO:0052815
name: medium-chain acyl-CoA hydrolase activity
def: "Catalysis of the reaction: H2O + a medium-chain acyl-CoA = a medium-chain carboxylate + CoA. A medium chain is a chain of between eight and twelve carbons in length." [EC:<IP_ADDRESS>]
synonym: "medium-chain acyl coenzyme A hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "medium-chain acyl-thioester hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "medium-chain hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "medium-chain-acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
is_a: GO:0047617 ! acyl-CoA hydrolase activity

[x] Add RHEA:68184
[x] change def and name to match RHEA:68184 (and update chain length def as per #26445)
[x] replace EC:<IP_ADDRESS> def xref with RHEA:68184 (see GO:0047380 below)
[x] Remove synonyms attribution to EC:<IP_ADDRESS> (see GO:0047380 below)

id: GO:0047380
name: ADP-dependent medium-chain-acyl-CoA hydrolase activity
def: "Catalysis of the reaction: H2O + a medium-chain acyl-CoA = a medium-chain carboxylate + CoA. Requires ADP." [EC:<IP_ADDRESS>, MetaCyc:<IP_ADDRESS>-RXN]
synonym: "ADP-dependent medium-chain acyl coenzyme A hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent medium-chain acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent medium-chain acyl-thioester hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent medium-chain hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent myristoyl-CoA thioesterase activity" NARROW [EC:<IP_ADDRESS>]
synonym: "ADP-dependent-medium-chain-acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
xref: EC:<IP_ADDRESS>
xref: MetaCyc:<IP_ADDRESS>-RXN
is_a: GO:0052815 ! medium-chain acyl-CoA hydrolase activity

[ ] add RHEA:68184 here as a broad xref??  RHEA maps RHEA:68184 (no mention of ADP-dependency) to EC:<IP_ADDRESS> ("ADP-dependent medium-chain-acyl-CoA hydrolase")
(I guess the question is, are there ADP-independent medium-chain-acyl-CoA hydrolase activities?) Do you know @kaxelsen ?

id: GO:0047379
name: ADP-dependent short-chain-acyl-CoA hydrolase activity
def: "Catalysis of the reaction: H2O + a short-chain acyl-CoA = a short-chain carboxylate + CoA." [EC:<IP_ADDRESS>, MetaCyc:<IP_ADDRESS>-RXN]
synonym: "ADP-dependent propionyl coenzyme A hydrolase activity" NARROW [EC:<IP_ADDRESS>]
synonym: "ADP-dependent propionyl-CoA hydrolase activity" NARROW [EC:<IP_ADDRESS>]
synonym: "ADP-dependent propionyl-CoA thioesterase activity" NARROW [EC:<IP_ADDRESS>]
synonym: "ADP-dependent short-chain acyl coenzyme A hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent short-chain acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent short-chain acyl-CoA thioesterase activity" RELATED [EC:<IP_ADDRESS>]
synonym: "ADP-dependent-short-chain-acyl-CoA hydrolase activity" RELATED [EC:<IP_ADDRESS>]
xref: EC:<IP_ADDRESS>
xref: MetaCyc:<IP_ADDRESS>-RXN

[ ] add RHEA:68180 here as a broad xref??  RHEA maps RHEA:68180 (no mention of ADP-dependency) to EC:<IP_ADDRESS> ("ADP-dependent short-chain-acyl-CoA hydrolase")
[ ] add new parent to this term for 'short-chain acyl-CoA hydrolase activity' (exact xref RHEA:68180), to mirror how the two medium-chain terms are currently handled (see above)??
(I guess the question is, are there ADP-independent short-chain-acyl-CoA hydrolase activities?) Do you know @kaxelsen ?

Hi @sjm41
Should we obsolete GO:0047380 (no annotations) and GO:0047379 (1 annotation, paper does not mention ADP).
There are reference cited in the BRENDA records:
https://pubmed.ncbi.nlm.nih.gov/2901416/
https://pubmed.ncbi.nlm.nih.gov/2570608/
but I cannot tell which genes these paper describe.
I propose we obsolete these for now and restore if needed.
Unless @kaxelsen has more evidence for this reaction?

We could create 'short chain' - looks like many terms need to be reclassified:

Also, in seems that base on its children, 'GO:0016289 CoA hydrolase activity' means the same as 'GO:0047617 acyl-CoA hydrolase activity'.
What do you think ?

Right. 'GO:0016289 CoA hydrolase activity' says "Catalysis of the reaction: X-CoA + H2O = X + CoA; X may be any group.", but then 'X' is 'CoA' in all the child terms.
That said, I see there are a couple of things to consider/fix:

GO:0047617 acyl-CoA hydrolase activity has def "Catalysis of the reaction: acyl-CoA + H2O = CoA + a carboxylate." But based on the xrefs (EC:<IP_ADDRESS>, RHEA:16781) and children,  the name should be changed to "fatty acyl-CoA hydrolase activity" and the def should be "Catalysis of the reaction: a fatty acyl-CoA + H2O = a fatty acid + CoA + H+".

Then, I think 'GO:0016289 CoA hydrolase activity' could be renamed 'acyl-CoA hydrolase activity' with def "an acyl-CoA + H2O = a carboxylate + CoA + H+" with xref RHEA:70423

Talk later....

@sjm41

add RHEA:68184 here as a broad xref??  RHEA maps RHEA:68184 (no mention of ADP-dependency) to EC:<IP_ADDRESS> ("ADP-dependent medium-chain-acyl-CoA hydrolase")
(I guess the question is, are there ADP-independent medium-chain-acyl-CoA hydrolase activities?) Do you know @kaxelsen ?

No, I'm not very well into acyl-CoA hydrolases. Both EC <IP_ADDRESS> and EC <IP_ADDRESS> are defined based on pmid:2901416 describing "A novel type of short- and medium-chain acyl-CoA hydrolases in brown adipose tissue mitochondria", but only EC <IP_ADDRESS> has links to Swiss-Prot, and they are all 5 to enterobacteria

I only find
https://www.uniprot.org/uniprotkb/Q7NUH6/entry
which has an automatic annotation to EC <IP_ADDRESS> - but how did that get predicted?
Discussion with @sjm41 : we will obsolete and keep xrefs on the appropriate parent

Clarifying info from Lucila Aimo regarding acyl vs fatty acyl:

RHEA:70423, the R represents any acyl chain, including fatty acyl chains
RHEA:16781, the R represents a fatty acyl chain (a fatty acid is a carboxylic acid with an aliphatic chain). This reaction does not include phenyls, nor cholesteryl, nor acetyl, formyl or any acyl that is not a fatty acid.

In other words:

If the R group on the acyl-CoA is an aliphatic chain (ie. a hydrocarbon chain with no rings), then the molecule is classed as a fatty acyl-coA.
But if the R group contains a ring (e.g. phenylacetyl-CoA, RHEA:70427), or contains atoms other than H/C (e.g. malonyl-CoA, RHEA:40219), or has no carbons (e.g. formyl-CoA, RHEA:19741), or has only 1 carbon meaning there is no ‘chain' (acetyl-CoA, RHEA:20289) then it is not a fatty-acyl-CoA, though it is still an acyl-CoA.

Sounds good!

Are there 2 or 3 terms to obsolete? you have GO:0047380 twice in the list.

GO:0047379 ADP-dependent short-chain-acyl-CoA hydrolase activity >> replace by new term 'short-chain fatty acyl-CoA hydrolase activity'
GO:0047380 ADP-dependent medium-chain-acyl-CoA hydrolase activity >> replace by parent GO:0052815 medium-chain acyl-CoA hydrolase activity

+[Term]
+id: GO:0141126
+name: short-chain fatty acyl-CoA hydrolase activity
+namespace: molecular_function
+def: "Catalysis of the reaction: a small-chain fatty acyl-CoA + H2O = a medium-chain fatty acid + CoA + H+. A small-chain fatty acid has an aliphatic tail of less than 6 carbons." [RHEA:68180]
+comment: While there is not universal consensus on the lengths of short-, medium-, long- and very-long-chain fatty acids, the GO uses the definitions in ChEBI (see CHEBI:26666, CHEBI:59554, CHEBI:15904 and CHEBI:27283).
+xref: EC:<IP_ADDRESS> {source="skos:relatedMatch"}
+xref: RHEA:68180
+is_a: GO:0047617 ! fatty acyl-CoA hydrolase activity
+property_value: term_tracker_item "https://github.com/geneontology/go-ontology/issues/26441" xsd:anyURI
+created_by: pg
+creation_date: 2023-12-08T09:05:55Z

Thanks @pgaudet ! But can you change 'small' and 'medium' to 'short' in the def?
That is:
+def: "Catalysis of the reaction: a short-chain fatty acyl-CoA + H2O = a short-chain fatty acid + CoA + H+. A short-chain fatty acid has an aliphatic tail of less than 6 carbons." [RHEA:68180]

Also changed 'very-long-chain acyl-CoA dehydrogenase activity' to very-long-chain FATTY acyl-CoA dehydrogenase activity : isn't this right ?

Yep. I just starting to look at the acyl-CoA dehydrogenase branch systematically to see if other similar changes are needed there (as done for acyl-CoA hydrolase branch here). So might be another ticket coming....

ah yes thanks! I am spreading out...

Will do the last 2 obsoletions here: https://github.com/geneontology/go-ontology/issues/26597

Looks like I mistakingly made "3-hydroxyisobutyryl-CoA hydrolase activity (GO:0003860)" a child of "short-chain fatty acyl-CoA dehydrogenase activity (GO:0016937)" instead of the intended "short-chain fatty acyl-CoA hydrolase activity ( GO:0141126)".
I will fix this now.

Also just noticed this outstanding task mentioned above:

For some reason, I couldn't add RHEA:70423 xref to GO:0016289 "acyl-CoA hydrolase activity" - can you see if you can do that without getting an error?

I'll try to do this again now.

For some reason, I couldn't add RHEA:70423 xref to GO:0016289 "acyl-CoA hydrolase activity" - can you see if you can do that without getting an error?
I'll try to do this again now.

Nope, failed again.  Here's the error:
FAIL Rule ../sparql/one-to-one-xrefs-by-subject-violation.sparql: 1 violation(s)
c,one_to_one_prefix
http://purl.obolibrary.org/obo/GO_0016289,RHEA:

@pgaudet Do you see what the issue is here??

No, this is really weird. I am trying here: https://github.com/geneontology/go-ontology/pull/27911

The xrefs from the obsolete term GO:0047901 are being transferred to its replacement during the build. This causes the multiple xrefs error. This is a really confusing system, and I recently opened an issue to fix it: #27821

Aha, thanks @balhoff
@pgaudet All the term xrefs on this obsolete term should just be removed, right?
id: GO:0047901
name: obsolete formyl-CoA hydrolase activity
namespace: molecular_function
def: "OBSOLETE. Catalysis of the reaction: formyl-CoA + H2O = CoA + formate + H+." [EC:<IP_ADDRESS>, RHEA:19741]
comment: This term was obsoleted because the enzyme that catalyzes this reaction has not been identified.
synonym: "formyl coenzyme A hydrolase activity" RELATED [EC:<IP_ADDRESS>]
xref: EC:<IP_ADDRESS>
xref: KEGG_REACTION:R00521
xref: MetaCyc:FORMYL-COA-HYDROLASE-RXN
xref: RHEA:19741
property_value: term_tracker_item "https://github.com/geneontology/go-ontology/issues/26443" xsd:anyURI
is_obsolete: true
replaced_by: GO:0016289

Looked at the obsoletion doc.....

Cross-references
Check if there are cross-references to Reactome, RHEA, EC, MetaCyc xref to the term. If there are, notify the various groups in the obsoletion email.
These cross-references should be removed.
There is no need to edit Reactome xrefs are imported automatically from a file supplied by Reactome.
Definition cross-references to RHEA, EC, MetaCyc should be also be removed. Replace by GOC:curators if no reference is left.

So, I will go ahead and remove the term and def xrefs on GO:0047901, and then try re-adding RHEA:70423 xref to GO:0016289.

Success!
id: GO:0016289
name: acyl-CoA hydrolase activity
namespace: molecular_function
def: "Catalysis of the reaction: an acyl-CoA + H2O = a carboxylate + CoA + H+." [RHEA:70423]
synonym: "CoA hydrolase activity" BROAD []
xref: RHEA:70423
is_a: GO:0016790 ! thiolester hydrolase activity
property_value: term_tracker_item "https://github.com/geneontology/go-ontology/issues/26441" xsd:anyURI
id: GO:0047901
name: obsolete formyl-CoA hydrolase activity
namespace: molecular_function
def: "OBSOLETE. Catalysis of the reaction: formyl-CoA + H2O = CoA + formate + H+." [GOC:curators]
comment: This term was obsoleted because the enzyme that catalyzes this reaction has not been identified.
synonym: "formyl coenzyme A hydrolase activity" RELATED []
property_value: term_tracker_item "https://github.com/geneontology/go-ontology/issues/26443" xsd:anyURI
is_obsolete: true
replaced_by: GO:0016289


meanprop() returns implausible values for vectorial properties

Because the Base.sum() function calculates the sums of array lines crudely, the individual sums by far exceed the precision of Float32 numbers, causing significant errors in the return value of the Statistics.mean() and subsequently the _meanprop() function when entering vectorial particle data in Float32 precision with particle numbers exceeding 7 digits.

Amazing


Hide config icon

Hi there, I like this mod, but I dislike that the config item is shown on start, item screen and pause. Please add option to hide it. If I want to config, I go to the config files on my pc

Install ModMenu, the config icon is only shown if modmenu isn't present.

will you accept a PR with my solution, having it shown by default if no mod menu is present?

No. The config system is used in all of my mods, more than 10. It would be a huge hassle to do that in all mods, most of which have versions going back to 1.14, and break the expectations of all those mod users.

First of, adding a config option that defaults to the old value doesn't break anything.
Secondly, the icon is intrusive, this is just a boolean flag to change, and the answer to a boolean flag change should never be "Just install mod X to change the flag."

I apologize for my rudeness in the previous post. I don't know why your No has tilted me that much. It is your mod to do with as you wish. Have a nice week :-)


Am 2612 et wa add mapping rules ticket description

https://tools.hmcts.net/jira/browse/AM-2612
New logic added for Authorisation Validation to check for Employment Ticket Description
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

This PR is being closed as the additional change is no longer required after a call with the RD team: the ticket mapping to service code will be actioned up-stream in the RD Judicial API.


Hamburger Error

When I reduce my browser (Firefox, latest release) to the minimum width or display the site (https://openxr-tutorial.com/index.html) on a mobile device the left menu reduces to a hamburger. Unfortunately, the hamburger is non-functional. The cursor changes to a pointer (desktop), but there is no response when clicked. There is also no response on a mobile device.
Looking at the debugger, it appears that there is no event (either explicit or implicit [anchor-tag]) associated with the symbol and the cursor change is just through CSS.

Fixed again. Seems a bad merge may have deleted the code.


Remove emissary.suscribe('save') which is deprecated

[Following https://github.com/sindresorhus/atom-jshint/pull/78#issuecomment-91112912]

Thanks :)


Is it possible to detect syntax errors for spelling errors etc?

I'm loading settings from a toml file using Toml.ReadFile<SomeStructure> and I would like to detect when the settings file contains property names that are misspelled or in other words when a property in the TOML file doesn't map to any member on SomeStructure in this example.
Is this already supported? If not, would you be interested in adding that support or would you accept a PR for this?

This was not yet supported.
The linked commit shows my first idea to implement such a feature.
The callback approach with the various parameters, should allow some quite flexible error handling, as the user can decide what should happen if a property cannot be mapped (log message, throw exception etc.)
I'd love to hear some feedback on the API.

The suggested solution in the linked commit looks nice! If I understand the new tests correctly then keys on the root object as well as tables will trigger the missing property callback, perfect.

The callback approach with the various parameters, should allow some quite flexible error handling, as the user can decide what should happen if a property cannot be mapped (log message, throw exception etc.)

This is exactly what I had in mind as the preferred solution 👍

Can we have a 0.13 release soonish or do you want to think about this new feature some more?

I hope to release a new version this weekend.

Fixed with v0.13.0


Migrate Travis CI checks to GitHub workflows

https://app.travis-ci.com/github/Microsoft/tolerant-php-parser/builds is apparently a blank page and broken image at the time this issue was created.
This seems to be why https://github.com/microsoft/tolerant-php-parser/pull/405/files was created
The same functionality is possible (e.g. also run it in docker similar to how phpunit tests are run in GH Actions) if configuration is added (and tested)

You started this awhile back in https://github.com/microsoft/tolerant-php-parser/pull/386, should I merge that?

You started this awhile back in https://github.com/microsoft/tolerant-php-parser/pull/386, should I merge that?

Yes. PHPStan may have had some updates that would cause failures due to dead code (seen locally), which #406 would fix.


help needed...  poor performance on first load

Hi,
I created a website using this repo however the performance is really poor - I'm experienced developer but with limited knowledge in angular, can anyone take a look at the load screenshot and point me to the right direction please?....
Thanks.
The website is  http://www.bellabags.co.uk/
It showed the root document load took 9.36 seconds to load 74.8k which is rendered by the nodejs on server. the server spec is pretty high.
The load screenshot (Chrome)

Angular loads quite fast when I tested it.
But the API in the background is really slow, taking 5 seconds to just do one simple GET call. So you need to fix the back-end API first of all.

Yes, renders are extremely quick, but just keep in mind that the render waits until all API calls & setTimeouts (or anything else Async) finishes, before rendering.
The two options you have, are either using isPlatformServer() and simply -not- making that big API call during server-renders, and having them only run on the Browser, or fixing the backend code to improve that APIs performance (either splitting it into multiple smaller API calls run in unison, or seeing whatever other performance bottleneck you're experiencing in SQL/whatever DB you're using.
Hope that helps!


Added Someone Like You by Van Morrison

https://open.spotify.com/track/4beWEVoUbjSziAFPy8voaF?si=3C1OMr0lQ96J_We82RMgwg

I like it, could definitely slow dance to it


Add background color to all tech stack

Description
For now every tech stack must be defined style for background and text color, when there is a new tech stack it must be defined manually. I think it would be helpful when it automatically generates the background and text color for each tech stack.
If allowed I'm ready to contribute.

Please do!


Use type checking instead of casting in README

Changes in this pull request
Just a little improvement to a code example.
Pull request checklist

[x] All tests pass. Demo project builds and runs.
[x] I added tests, an experiment, or detailed why my change isn't tested.
[x] I have reviewed the contributing guide

Thanks @sharplet ! 🎉

TIL, this is awesome, and hella fast! We haven't even publicly announced this yet 😆
Merging since CLA signed too.

A little birdy told me about it! 😛


Build the style guide (Components: Pagination)

The design team has built the Style guide, which can be found here.
This issue is a chunk of #4, ensure to check it first.

Build the Pagination component similar to the design and document all classes used
Build with great focus on all states and responsiveness.

Assign me 🍾

Assign me
Also, what implementation are we using?
Examples are Google, Amazon.
What numbers are shown when for eg, the current page is 7 and there are 15 pages?

I just assigned you.
Kindly check the Style guide for the required implementation.

Assign me
Also, what implementation are we using?
Examples are Google, Amazon.
What numbers are shown when for eg, the current page is 7 and there are 15 pages?

I just assigned you.
Kindly check the Style guide for the required implementation.

It doesn't really say

Alright, we are doing something similar to Google's.
I'll cc @perriefidelis on this.

Alright

we can close this up ... once the PR is merged


switch to Sensor Settings for mode configuration for climate entity

Mode of operation is defined per zone based on Sensor Settings, not as currently implemented based on global Heating Mode.

@geduxas what do you think of this change?

@mrrudy thanks for you contribution and sorry for delayed feedback. I'm asking opinion of people more familiar with heishamon than myself.

@geduxas what do you think of this change?

Were well done :) nice to see that. Now based on that it would be nice to have thermostat levels.

@mrrudy thanks for you contribution and sorry for delayed feedback. I'm asking opinion of people more familiar with heishamon than myself.

no worries. I'm using the approach for 2 weeks and it works in case you have 2 zones without an issue. It would be good if someone with only one zone tested it beforehand but from the description of topics from heishamon that shouldn't be different.

@mrrudy ok that you write, i think we will have problems with that.
Now i am looking only in changes, don't know whole picture of that, maybe @kamaradclimber  will help.
As i see from code, you have removed "main/Heating_Mode" topic, but that topic is main information source to determinate if it's using compensative curve, or direct heating. From zone_sensor_settings you can only pick temperature sensing device.

So yes, triple checked service manual, you have 2 options in heating mode, Compensation curve/Direct.
and for direct you have ability to chose ether use direct water/[internal/external]_room/thermistor.

So yes, triple checked service manual, you have 2 options in heating mode, Compensation curve/Direct. and for direct you have ability to chose ether use direct water/[internal/external]_room/thermistor.

In my configuration the "main/Heating_Mode" is mqtt=0 so "COMPENSATION". In the same time I have Z1 set to Internal_room and Z2 set to direct water.  After my modification the mode (ZoneClimateMode.) is driven per zone. This allows to set the Z1 in range of 20 and Z2 -5 to 5, same as from the heat pump physical controller.
The "main/Heating_Mode" is, I assume, combination of Z1 and Z2 and should not drive the ranges as those can be mixed (it probably works if both of the Zones are set to the same mode as then this is set to the same value). The question is, when you have only one zone is the Z1 set properly. If yes we could ignore the "main/Heating_Mode" and use "main/Z1_Sensor_Settings".

Yes without expension board it will drive z1 only.
But! If i understand correctly direct in water mode you treated as working in compensation curve?

Yes without expension board it will drive z1 only.
But! If i understand correctly direct in water mode you treated as working in compensation curve?

Ok, so I went through all modes on the controller and it seems like so:
if(main/Z1_Sensor_Settings==INTERNAL OR main/Z1_Sensor_Settings==THERMISOR) ## ROOM temperature is the driver, you set it directly from 10:35 deg C
{ mode = ZoneClimateMode.ROOM }
else if(main/Z1_Sensor_Settings==EXTERNAL) ## in this mode you cannot set the temp at all
{ mode = ZoneClimateMode.NAN }
else if(main/Z1_Sensor_Settings==WATER) ## now you have to check if you are driving the temp of water directly (20:55 deg C) or by comp curve (-5:5 deg C)
{  if main/Heating_Mode==DIRECT
   { mode = ZoneClimateMode.DIRECT }
   else if main/Heating_Mode==COMPENSATION
   { mode = ZoneClimateMode.COMPENSATION}
 }

So it seems you are right, we still need the comp/dir distinguishing but only if we drive water temperature. For internal thermostat or thermistor, which give you room temperature directly you must set expected temp even if the curve is enabled, which makes sense as you measure expected results. The EXTERNAL is curious, as on the heat pump controller it just shows you icon of the thermostat and does not allow you to do anything. In that case HA would have to disable the entity i guess.
Please check if the above makes sense and let me know should I take a shot at writing it or you will do it. I'm more of a C guy as you might see from the metacode but should not be that hard for me to figure it out. I would probably have an issue with disabling the entity thow.

Sorry for late reply, indeed that's looks good enough :) thanks.

Awsamome job @mrrudy! Your doing right things :) nice to see how integration evolving and getting better and better with such heroes :)
If you're in mood, and if you find interested in that maybe you could also extend functionality for cooling? Don't know really if it's possible with one HA thermostat, but in Panasonic it's possible to manualy, or auto switch to cool mode.
I am not talking about this PR :) but i would be glad to see it in feature :) thanks

This is definitely going on the right direction. Let me know when you think I can make a final review and merge.

I think you can review now. I'm not sure how to approach harmonization, especially not knowing if there is difference in heat pipe models - this might need to wait for some feedback/investigation.

Thanks a lot for your contribution, it's in https://github.com/kamaradclimber/heishamon-homeassistant/releases/tag/1.7.0 if you want to test it (pre-release for now).
I'll let it bake for a few days and publish it if nobody reports an issue.


bump github.com/blang/semver

Bump github.com/blang/semver to v4 to match upstream usage.

I found this bumping kube version on openshift-controller-manager. The bump results in multiple github.com/blang/semver versions being used in such a way that results in compile errors. After fixing it in openshift-controller-manager, there was still an indirect reference to the older version due to a dependency on library-go, so here I am "cleaning it up".

/assign @sttts @deads2k
side note: we might want to update owners for this component?


Different hashes in js-ipfs-api compared to go-ipfs

I have this example project set up: https://github.com/VictorBjelkholm/ipfs-inconsistency
It contains a index.js that adds the test-data directory to ipfs.
If you run that file with node index.js, you get a list of different hashes compared to if you add the files with go-ipfs (by running ipfs add -r test-data) which is weird.
Example output from js-ipfs-api:
added QmPsoRskVhY7GowC75kBBncVHBkTyHAktiNxSmyhNgKxn3 test-data/readme.md
added QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH test-data/src
added Qmdn9zEotP3h7p7roppE3PzwtdiAwYPr6mQettwij1JJN4 test-data/src/lib.js

Output from ID (only versions):
{
  AgentVersion: 'go-ipfs/0.3.11',
  ProtocolVersion: 'ipfs/0.1.0' }

Example output from go-ipfs
added QmPsoRskVhY7GowC75kBBncVHBkTyHAktiNxSmyhNgKxn3 test-data/readme.md
added Qmdn9zEotP3h7p7roppE3PzwtdiAwYPr6mQettwij1JJN4 test-data/src/lib.js
added QmSZwVMTt2stxqs3MiAUCk9w6wX3n5Z8tiJoQUv4DDbmeT test-data/src
added QmSPvto83drjXTSLGUhYVzu1n1eKAu8o6Jd8JWrDQCCu2c test-data

Output from ID (only versions):
{
	"AgentVersion": "go-ipfs/0.3.11",
	"ProtocolVersion": "ipfs/0.1.0"
}

There are two problems with the output from js-ipfs-api. 1) The folder itself is not included 2) the hashes are not the same...
What could be wrong here?

This is very strange indeed. The first thing I noticed is that you are using go-ipfs 0.3.11, while you are using js-ipfs-api latest. go-ipfs 0.4.x introduced some breaking changes on the http-api and we do not recommend using latest js-ipfs-api with it (we do really recommend upgrading to the latest version of go-ipfs, currently 0.4.2)
Then, the second bug is that js-ipfs-api addFiles call is not sending the enclosing folder that is being added. This is a known bug. In fact, I believe we talked about this the other day, right?
The third bug, which is different hashes, seems like a new bug, it feels like js-ipfs-api is sending the file that lives in a subdir too late.
Thank you for detecting this! Could you convert it to a failing test? I'll try to look into it asap, but also, if you have time to go through the code, it will be much appreciated :)

@diasdavid I'll try to update and report back.
Secondly, yes, I have a PR opened about that ( https://github.com/ipfs/js-ipfs-api/issues/297 ) but was unsure since the hashes was different.
I'll see if I can take a look at a later point.
Thanks!

@diasdavid Yes, updating to ipfs 0.4.2 did indeed solve the issue.
New output:
➜  ipfs-inconsistency git:(master) node index.js
Added QmPsoRskVhY7GowC75kBBncVHBkTyHAktiNxSmyhNgKxn3 test-data/readme.md
Added Qmdn9zEotP3h7p7roppE3PzwtdiAwYPr6mQettwij1JJN4 test-data/src/lib.js
Added QmSZwVMTt2stxqs3MiAUCk9w6wX3n5Z8tiJoQUv4DDbmeT test-data/src
Added QmSPvto83drjXTSLGUhYVzu1n1eKAu8o6Jd8JWrDQCCu2c test-data
➜  ipfs-inconsistency git:(master) ipfs add -r test-data/
added QmPsoRskVhY7GowC75kBBncVHBkTyHAktiNxSmyhNgKxn3 test-data/readme.md
added Qmdn9zEotP3h7p7roppE3PzwtdiAwYPr6mQettwij1JJN4 test-data/src/lib.js
added QmSZwVMTt2stxqs3MiAUCk9w6wX3n5Z8tiJoQUv4DDbmeT test-data/src
added QmSPvto83drjXTSLGUhYVzu1n1eKAu8o6Jd8JWrDQCCu2c test-data

Thanks for that! Maybe it'll be useful to add a warning of some sorts if you're running js-ipfs-api with an incompatible version of go-ipfs?


[DOCS] Homepage Menu Button Does Not Work On IOS

Expected: menu button at https://qwik.builder.io/ works and shows “Docs, Qwik City, etc…”

Actual: menu button does nothing. It only works on non-homepage routes

Should be fixed now! thanks for the issue

@manucorporat TYVM!


Add DSA Roadmap covering all key issues

Outline all the topics that needs to be studied, order in which and some important questions to practice in each.

hi I am jai garg from Ipec , ghaziabad . I am new to open source but have knowledge related to this topic. want to contibute in this

Ok take help for syllabus from here. bit.ly/TathastuDSASeries