Error message in Company Communicator Author with the New Teams version desktop and web

When you access the new Teams and try to access the Company Communicator Author, an error is generated. This happens in the desktop and web version.

We are now working with this version: Version 5.1.1 Could you please help us solve this problem?
Thank you!

We encountered the same issue in "new teams" , please give some advice why the error occurs.

Hi @SANCOR1 ,
Thanks for raising the query. The app service sync failure issue is now resolved, and the changes have been pushed to Company Communicator official GitHub repository.
If current deployed version of the Company Communicator application is pointing to v5.5 (latest version), please follow the below procedure to resolve the issue -

Go to Azure Portal --> Go to Resource Group (where the CC deployment has been done) --> Select Web App (App Service) --> Go to Deployment Center --> Click on SYNC

Once the status shows successful within logs section of Deployment Center.

Note: Please make sure the Repo URL and Branch name is pointing to below values in Deployment Center Screen of app service

please go to the Overview Section and restart the app service.

Try accessing the CC application on Teams web client to avoid any cache related issue.

If current deployed version of the Company Communicator application is pointing to v5.4.1 or previous versions, please upgrade your app to latest version by following the migration guide as the fix is pushed to the main branch in Company Communicator's official GitHub repo.
If it’s a fresh deployment then proceed with the remaining steps as per the deployment guide.
Please let us know if you are still facing the issue.

Hello @gsv022 I do have the same issue and followed the steps to solve this and the issue still persists... is there any workaround?? Or can you help us to solve this

We also have this issue in new Teams. We are on version 5.0.1 but I'm assuming it's a Teams issue not a version issue. Would really like to know if there's a way to resolve it before MS removes classic Teams.

We are also getting the same issue with the NEW Teams. classic teams is okay

Still outstanding

Hello all, I wrote this in other ticket:
I found solutions which work in my company, I updated version from 5.2 to 5.5, and then on windows 11, go to Setting in windows - Application - Installed Applications - Microsoft Teams (work or school) NEW
and try FIX (in screen Napraw)
and Reset (on screen Resetuj)
and it start works on 2 different author accounts and PC's
image
image

Hi @SANCOR1 ,
Could you please let us know the status of the issue

Hello,
We got the following  advisory message on our M365 admin center.
We are thinking if this is the solution of this issue.
How about it?
**[TM718209] 2024/02/22 4:34 JST
We’ve released an improved security for Company Communicator app template authoring experience
If action isn’t taken, Microsoft Teams users may encounter errors when authoring Company Communicator app templates.
Due to a recent security change in the Teams client, users in your organization might see an error
while accessing the authoring experience in the Company Communicator app in Teams, with the error message
“Sorry, an error occurred while trying to access the service”.
Users will observe this issue on the Classic Teams web client, and also on the new Teams desktop and web Clients.
To correct this issue, the app developer or admin will need to perform the below changes on the Azure portal.
Step 1: In the Azure portal, locate and select Company Communicator app Resource group by using the search bar, or
Navigate to Resource groups from the Azure portal menu.
Step 2: Select the Azure Front Door from the list of resources.
Step 3: Under Settings, locate the Front Door designer section. Select “routingRule2” that was created during the
Company Communicator app deployment.
Step 4: Disable  the rule and save the changes.
Step 5: Restart your app service.
Step 6: Users may also need to clear the cache on the Teams desktop client(Classice Teams or new Teams).
Note: The Official Company Communicator App Template GitHub code and deployment instructions
will be updated accordingly in the upcoming weeks.**

Hello, We got the following advisory message on our M365 admin center, because we have used the Company Communicator. We are thinking if this is the solution of this issue. How about it?
**[TM718209] 2024/02/22 4:34 JST We’ve released an improved security for Company Communicator app template authoring experience
If action isn’t taken, Microsoft Teams users may encounter errors when authoring Company Communicator app templates.
Due to a recent security change in the Teams client, users in your organization might see an error while accessing the authoring experience in the Company Communicator app in Teams, with the error message “Sorry, an error occurred while trying to access the service”. Users will observe this issue on the Classic Teams web client, and also on the new Teams desktop and web Clients.
To correct this issue, the app developer or admin will need to perform the below changes on the Azure portal.
Step 1: In the Azure portal, locate and select Company Communicator app Resource group by using the search bar, or Navigate to Resource groups from the Azure portal menu.
Step 2: Select the Azure Front Door from the list of resources.
Step 3: Under Settings, locate the Front Door designer section. Select “routingRule2” that was created during the Company Communicator app deployment.
Step 4: Disable the rule and save the changes.
Step 5: Restart your app service.
Step 6: Users may also need to clear the cache on the Teams desktop client(Classice Teams or new Teams).
Note: The Official Company Communicator App Template GitHub code and deployment instructions will be updated accordingly in the upcoming weeks.**

I confirm that these steps work! thank you!!

Hi, @gsv022
Do you think  what  Microsoft Advisory tells and it is the workaround of this issue on all previous versions of Company Communicator ?
We are on version 4.1.0. Should I try them?
Thanks.

@ATshirai Legend, worked for me on V5.5. Thank you so much!

@ATshirai you should try it. We are on version 4 and did not need to upgrade. I followed what you posted and it worked.

@leoacnesi
Thank you so much for your advice.
We think your 4.1 is working fine means ours will be working fine.
I'll try it as you.

Hi @gsv022
My organisation is currently using Company Communicator version 5.5.
The end users have switched across using the new version of Teams in desktop and on web browser and was able to send message successfully and not having the issue error message when communicator loads up.
I also received a notification from Microsoft at the Admin portal service health  https://admin.microsoft.com/AdminPortal/home?#/servicehealth/:/alerts/TM718209

So does do I still need to follow Microsoft instruction or follow your instruction in this ticket to sync if we are not experiencing the error message "Sorry an error has occurred while trying access this service" in v5.5?
Please advise.


Homepage redesign

Please use the link below to access Figma file v2 version is the most updated design
https://www.figma.com/file/aQaIIMR7r8vMkQnkAk320a/AI?type=design&node-id=3365%3A2429&mode=design&t=LbKpHg8GarEXn8ZD-1

@laodao-zzw @danwang510  Above link does not work for me. if you could not open it. try this one https://www.figma.com/file/aQaIIMR7r8vMkQnkAk320a/AI?type=design&node-id=3365-2429&mode=design&t=MU38KnWDIoBL1nBU-4


Add sequences into report

Adds sequences into report. Link to example report below.
CI testing additions just need to be fixed (which is a check that the sequence.md files in validator/sequences are up to date , otherwise there is a risk of comparing to an out of date sequence.md file)
Sample report (click here) - the sequences.md don't actually match the results, just for illustration

@grafnu PTAL again


Updating dxi-toolbar-item options binding causes dx-scroll-view to scroll to top

To help us process the issue more efficiently, please provide the following information:

Specify the version of the devextreme-angular and devextreme packages you are using.
devexteme version: 17.1
devextreme-angular version: 17.1

Specify the type of the issue (check one with "x"):
[x] bug Report
[ ] feature request
[ ] support inquiry

See the plunk example.  Steps to reproduce:

Scroll to the bottom of the pop-up.
Click the "Enable submit" button.

Observed behavior: scroll view scrolls to top.
Expected behavior: scroll does not move.

I've managed to reproduce this behavior if I use the jQuery approach to create our widgets. So, I've created a corresponding ticket in our Support Center to research this behavior. See the Updating dxPopup toolbarItem options causes dxScrollView inside the popup content to scroll to top issue.


Add option to center menu relative to terminal width rather than root window width

I like my snipe window to show up in the absolute center of my terminal window, and that is what I would expect setting its position to 'center' to do.
Snipe appears to be determining the max width for centering by checking the width of the first window in the tabpage, which means that vertical splits interfere with it, for example:

Making the following change in Menu:get_window_opts centers the snipe window in the way that I would expect/like:

I created this issue rather than a pull request because I wasn't sure if you intended 'center' to behave like in my first screenshot, in which case I think this should be an additional position type, so I thought I should leave the implementation decision to you. Let me know if you need any more information. Cheers!

+1 I think at the very least this should open it in the center of the active split instead of the first one. Another setting for center of the window would be nice too

This is a very good catch ! I didn't really know that you could use vim.opt.columns so I did that tabpage thing to get the "root" window as you say. Thanks for pointing this out, Ill fix this when I get a chance !


vagrant don't up

Hello.
I can't up your vagrantfile on ubuntu server 17.10 artful
I'm get
dev@ubuntu:~/dso_repos/bootcamp$ vagrant destroy
    devsecops: Are you sure you want to destroy the 'devsecops' VM? [y/N] y
==> devsecops: Forcing shutdown of VM...
==> devsecops: Destroying VM and associated drives...
dev@ubuntu:~/dso_repos/bootcamp$ nano Vagrantfile
dev@ubuntu:~/dso_repos/bootcamp$ vagrant up
Bringing machine 'devsecops' up with 'virtualbox' provider...
==> devsecops: Importing base box 'centos/7'...
==> devsecops: Matching MAC address for NAT networking...
==> devsecops: Checking if box 'centos/7' is up to date...
==> devsecops: Setting the name of the VM: bootcamp_devsecops_1512232471751_23502
==> devsecops: Clearing any previously set network interfaces...
==> devsecops: Preparing network interfaces based on configuration...
    devsecops: Adapter 1: nat
==> devsecops: Forwarding ports...
    devsecops: 3000 (guest) => 3000 (host) (adapter 1)
    devsecops: 22 (guest) => 2222 (host) (adapter 1)
==> devsecops: Booting VM...
==> devsecops: Waiting for machine to boot. This may take a few minutes...
    devsecops: SSH address: <IP_ADDRESS>:2222
    devsecops: SSH username: vagrant
    devsecops: SSH auth method: private key
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

Docker didn't start, also and it don't  have logs.

Host is virtual machine, also.
I tried do vagrant up on my host (Windows 10) it did'n start also, but with other error.
Bringing machine 'devsecops' up with 'virtualbox' provider...
==> devsecops: Importing base box 'centos/7'...
C:/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.6.3/lib/childprocess/windows/process_builder.rb:43:in `join': incompatible character encodings: Windows-1251 and UTF-8 (Encoding::CompatibilityError)
        from C:/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.6.3/lib/childprocess/windows/process_builder.rb:43:in `create_command_pointer'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.6.3/lib/childprocess/windows/process_builder.rb:27:in `start'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.6.3/lib/childprocess/windows/process.rb:70:in `launch_process'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/childprocess-0.6.3/lib/childprocess/abstract_process.rb:82:in `start'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/subprocess.rb:141:in `block in execute'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/safe_chdir.rb:26:in `block (2 levels) in safe_chdir'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/safe_chdir.rb:25:in `chdir'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/safe_chdir.rb:25:in `block in safe_chdir'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/safe_chdir.rb:24:in `synchronize'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/safe_chdir.rb:24:in `safe_chdir'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/subprocess.rb:140:in `execute'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/subprocess.rb:22:in `execute'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/driver/base.rb:440:in `block in raw'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/busy.rb:19:in `busy'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/driver/base.rb:439:in `raw'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/driver/base.rb:377:in `block in execute'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/retryable.rb:17:in `retryable'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/driver/base.rb:372:in `execute'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/driver/version_5_0.rb:348:in `import'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/action/import.rb:53:in `import'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/action/import.rb:13:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/action/prepare_clone_snapshot.rb:17:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builtin/prepare_clone.rb:15:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/action/customize.rb:40:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/action/check_accessible.rb:18:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builder.rb:116:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/runner.rb:66:in `block in run'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/busy.rb:19:in `busy'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/runner.rb:66:in `run'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builtin/call.rb:53:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builder.rb:116:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/runner.rb:66:in `block in run'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/busy.rb:19:in `busy'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/runner.rb:66:in `run'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builtin/call.rb:53:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/warden.rb:34:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/builder.rb:116:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/runner.rb:66:in `block in run'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/util/busy.rb:19:in `busy'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/action/runner.rb:66:in `run'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/machine.rb:227:in `action_raw'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/machine.rb:202:in `block in action'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/environment.rb:631:in `lock'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/machine.rb:188:in `call'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/machine.rb:188:in `action'
        from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-2.0.1/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'

I encountered the same issue, can't up the vagrantfile because of timeout. I'm going through the setup from an Amazon windows instance.


Built-in timers/intervals?

Do we want??

Some resources that will be helpful here:
https://github.com/tokio-rs/tokio/tree/master/tokio-util/src/time
https://crates.io/crates/hierarchical_hash_wheel_timer

A simple, naive implementation has been implemented in #52 but I'd like to eventually replace that implementation with #53 after settling on a user-facing API.

#53 has been merged, but per comments therein we still want to introduce a helper actor + an ergonomic API on top of it. I'll open a PR to iterate that.


Bring back imgs so that forsk dont get broken

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

I think we just need to keep datasets_logo_name.jpg and course_banner.png because they appear in the README.md of the forks of datasets. The other images can be removed

Force pushed those two imgs only


CloudFormation template - Open in New Tab

Module 3: Remediation and Response CIS Benchmark and Custom Action

Deploy remediation playbooks for CIS Benchmarks
Deploy remediation playbooks via CloudFormation
Download CloudFormation template from GitHub.

Suggest that the CloudFormation template hyperlink is updated to open in new tab/window

added in commit a0e8502fff94bd0d80ce6c84add1ad153bbce28f


How to integrate Cordova to a project?

I want to add Cordova using Cordova-CLI.
So, what folder to put Cordova files?

+1 to this question.
I want to test this application using:

phonegap serve

But it can't find the phonegap application.


Patch 2

Adding a README to the bug-bounty-list directory.

Can we close this out?

@yesnet0 I had a number of small suggestions that hadn't been reviewed, but GitHub must have just added the ability to batch suggested changes and commit them (even though I was the person making the suggestions, whereas previously I had no way to apply them). If you're still happy with how it reads, we should be good to merge this.


AT Commands Firmware with AWS TLS/SSL Client connection

Hi, I have a post going over at ESP32.net as well, but I wanted to check in here since this forum is focused on ESP32 AT command firmware.
https://www.esp32.com/viewtopic.php?f=2&t=4901
Basically I am trying to connect to AWS IoT with the latest ESP32 AT cmd firmware, and I cannot get it to successfully complete the SSL hand shake. I am nearly positive that I have loaded the certs correctly, and tested them with the SSL windows command line utility, but it will not work on the ESP32. There are many more details on that other thread.
I would really appreciate some help on this and thought that there may be some AT Cmd experts watching this repo and could confirm that they have indeed connected an ESP32 running the AT firmware to AWS at some point? Thanks in advance!

Hi @schwac ,
Please refer to https://github.com/espressif/esp32-at/blob/master/tools/README.md about 2. PKI Bin

Ah ha! Thanks for referring me to that section. I just used that tool to generate the at_cert.bin, at_key.bin, and at_ca.bin and tried loading them to both the server and client locations - neither of these worked. Same error as before... I will keep trying various combinations. Have you got the AT cmd FW to connect to AWS IoT before?

I can confirm that this does not work if I load them to either the server locations (CERT = 0x24000, KEY 0x26000 and CA =0x28000) or the client locations (0x2A000, 0x2C000, 0x2E000). I have inspected the actual binaries created by the AtPKI python tool and they seem to be correct.

Hi,
We haven't connected to AWS IoT, but we have tested ssl server and client with cert and ca.
Could you give some log from UART0?

Have you had a chance to check out that UART log file yet?

Hi,
I (23906) openssl: ssl_tls.c:2441 ssl->f_send() returned -80 (-0x0050)

I (23912) openssl: ssl_tls.c:2846 mbedtls_ssl_flush_output() returned -80 (-0x0050)

I (23920) openssl: ssl_tls.c:4710 mbedtls_ssl_write_record() returned -80 (-0x0050)

I (23929) openssl: mbedtls_ssl_handshake() return -0x50
ssl connect fail

Maybe socket is reset by server in mbedtls_net_send. The error no is defined as
#define MBEDTLS_ERR_NET_CONN_RESET                        -0x0050  /**< Connection was reset by peer. */

thanks for taking a look... do you know if I should be loading the key, cert and ca bin files into the client positions or the server positions?

If you create as ssl client by "AT+CIPSTART" command , the key, cert and ca bin files should download into 0x2c000, 0x2a000 and 0x2e000.
The following is the partition information
# Name	       Type SubType	Offset	Size
client_cert	0x40	5	0x2a000	8K
client_key	0x40	6	0x2c000	8K
client_ca	0x40	7	0x2e000	8K

Please pay attention to the size of the file. And you can adjust offset and size if necessary.

I still cannot get this to work... I have loaded the binaries into the proper locations and it just won't connect to AWS via wifi. It programs fine and looks like it it starting the TLS handshake. Same sort of failure code as aIl the other  work on them.

esp32_at_SSL_AWS_Connect_Fail_03-20-18.txt
Any recommendations would be appreciated!.

@schwac ,
There is a lack of a command in the document, and sorry for that.
AT+CIPSSLCCONF — Config SSL client
Single connection: (+CIPMUX=0)
AT+CIPSSLCCONF=<auth_mode>,<pki_number>,<ca_number>
Multiple connections: (+CIPMUX=1)
AT+CIPSENDEX=<link ID>,<auth_mode>,<pki_number>,<ca_number>

Parameters:

<link ID>: ID of the connection (0~max), for multiple connections, if the value is max, it means all connections. By default, max is 5.

<auth_mode>:

0: no authorization.
1: load cert and private key for server authorization.
2: load CA for client authorize server cert and private key.
3: both authorization.

<pki_number>: the index of cert and private key, if only one cert and private key, the value should be 0.

<ca_number>: the index of CA, if only one CA, the value should be 0.

Notes:

Call this command before establish SSL connection if you want configuration take effect immediately.
The configuration changes will be saved in the NVS area. If you use AT+SAVETRANSLINK to set SSL passthrough mode, ESP32 SSL will be connected based on this configuration after power on.

Oh man! Yes, it sounds like that command is critical. I will test it ASAP.
Also, I think there is a typo in your example above: The command is AT+CIPSSLCCONF for both the single and multiple connections right? Not AT+CIPSENDEX?
Also, when will the next AT release be that contains SSL support for clients, etc?

Yes, The command is  AT+CIPSSLCCONF.
Now we are testing esp32-at, we will release it after testing Ok.

Awesome! That worked! I am connected successfully.
One more question: Can I make two or more client SSL connections simultaneously?

Yes, you can if ram is enough, otherwise, it will fail.

How do I know how much RAM is used? Can I conserve RAM by excluding certain features during build?

You can send AT+SYSRAM? to query remained RAM.

Cool. I consider this closed! Thank you so much for helping out!

works fine on NINA W102

Hi schwac
I am also trying to connect to aws iot via wroom32 with AT commands.
Do you have any docs / advice on how to build the packets?
i have connected to a local server with ssl... but i was in charge of data structure at both ends.  it is hard to find aws docs that don't use the freertos code or aws sdk command line.  Any help would be appreciated.

Did you create the 3 certificates and burn them ?

0x27000 client-cert.bin
0x28000 private_key.bin
0x29000  amazon-ca.bin

Port 8443
AT+CIPSTART="SSL","xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com",8443
cert_len=1224
prvtkey_len=1679
ca_len=1188
ctx ok
CONNECT
OK

And then
POST /things/lamp/shadow HTTP/1.1
Host: xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com
Content-Type: application/json
Content-Length: 89
{
"state": {
"reported": { "temp":"12","umidade":"88"
}
}
}

Not yet..
when you create a new certificate on AWS you can download
certificate.pem, public.pem, private.pem and a root CA for AWS IOT
so...
i keep the private.pem in a safe place
how do i convert the other 3 into *.bin files?

See item 2
https://esp32.com/viewtopic.php?t=9374

it´s a python utility! Works very nice!

Hi tcpipchip
Thank you for the detail above... I have successfully read data from the AWS iot shadow of my thing.
One small issue.. maybe you can help with.
I'm running the following version of AT command firmware on the esp-wroom-32

SDK version:v3.0.3
compile time:Sep  4 2018 08:53:40
Bin version(Wroom32):1.1.2

When i try to read my aws shadow thing i get the following message from the usb debug port.

cert_len=1220
prvtkey_len=1679
ca_len=1206
ctx ok
Task watchdog got triggered. The following tasks did not reset the watchdog in time:

IDLE (CPU 0)
Tasks currently running:
CPU 0: at_process_task

basically i looks like code on the ESP is stuck in a loop too long and causes a watchdog issue.
can this be fixed ... or is it a non issue.
issue 2
Also if i do a firmware upgrade
it upgrades to

SDK version:v3.2-127-gddbce78
compile time:Jun 27 2019 11:21:27
Bin version:1.2.0(WROOM-32)

but this message appears in the debug output

factory_parameter partition missed
1.2.0
factory_parameter partition missed

is this a problem?
Many thanks for all your help
William

well,
About the wdt, see if running 240Mhz !
About the second...
I found something
https://github.com/espressif/esp-at/blob/master/main/at_default_config.c
Are you using WROOM ?

Hi tcpipchip
Lots of progress this week.
Got esp working at 240MHz.. fixed the issue above.. fresh compile of latest git code.
I can connect via REST API to the AWS shadow and do a read & write...
Thanks for all the help.
I'm now trying to get the MQTT working....
so i have the options enabled for the AT command

AT_MQTT_USERCFG=0,3,"wtp","","",0,0,""
ie Link 0, MQTT over TLS(verify server cert), client ID= wtp, no username, no password, cert_key=0, CA_ID=0, no path.
i don't think you need username and password for AWS (is this correct??)

AT_MQTT_CONNCFG=0,30,0,"lwt_topic","lwt_msg",0,0
ie link 0, keepalive for 30s, clean session, lwt....etc

This is the debug output

[at-mqtt]warning code:24600
[at-mqtt]warning code:24603
all parameters are ready
mqtt_ca format error:ffff
the errors are letting me know the username and password are null.
I think there is a separate cert_key & CA_ID from the client cert,etc for https???
where do i find info on how to generate this cert, if it is different from before.. and where do i find the address table to know where to upload it to?

Many Thanks for your patience with my questions...

No necessary user pass to mqtt!
We have to know the difference between server-certifcate and client-certificates.
The address are

Look this curl server certificate

https://www.thingforward.io/techblog/2018-07-18-https-on-the-esp32-server-and-client-side.html

Sorry... is username and password necessary when connecting to AWS via MQTT?
if so, where are these found?
In the at_customise.csv (see attached) there are memory locations defined for mqtt_certificates, keys and  CA

Are these the same certificates as the client certificates, keys and CA?
Thank you

I never used user/pass to aws iot mqtt
I remember that are the same certificates for both, https and mqtt

De: William<EMAIL_ADDRESS>Enviado: segunda-feira, 7 de outubro de 2019 08:22
Para: espressif/esp-at<EMAIL_ADDRESS>Cc: tcpipchip<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Assunto: Re: [espressif/esp-at] AT Commands Firmware with AWS TLS/SSL Client connection (#75)
Sorry... is username and password necessary when connecting to AWS via MQTT?
if so, where are these found?
In the at_customise.csv (see attached) there are memory locations defined for mqtt_certificates, keys and CA
[at_customize_csv]https://user-images.githubusercontent.com/39193764/66295650-b8f82080-e8e3-11e9-8eaf-3229f192e783.JPG
Are these the same certificates as the client certificates, keys and CA?
Thank you
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-at/issues/75?email_source=notifications&email_token=ACXUZTHYV3IDM7NVO6XPFH3QNLWS5A5CNFSM4EUCFI62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAPNFIA#issuecomment-538890912, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACXUZTATGECHYGD3SIOBBIDQNLWS5ANCNFSM4EUCFI6Q.

I never used user/pass to aws iot mqtt
I remember that are the same certificates for both, https and mqtt.
But i never tested mqtt protocol on ESP32 AT COMMANDS

https://blog.csdn.net/espressif/article/details/101714951 is a demo to connect AWS using AT+MQTT. But it is written in Chinese.

Hi @xcguang
Interesting...but...where did they get user and pass ?

looks let you have to change the file contents and compile again, right ?
AmazonRootCA1.pem 替换 components/customized_partitions/raw_data/mqtt_ca/mqtt_ca.crt
xxx.cert.pem 替换 components/customized_partitions/raw_data/mqtt_cert/mqtt_client.crt
xxx.private.key 替换 components/customized_partitions/raw_data/mqtt_key/mqtt_client.key

Compiling is just to get cert, ca and key bin file for AT.
However, now you cannot get official AT bin, which is supporting AT+MQTT.
@ustccw Please comment something about user and password.

thank you!

@Wtpaul
I guess that you will need all informations soon!

@xcguang @tcpipchip username and password is optional for AWS connection, the most important is the device certificate and endpoint.

Nice!!!!! Thanks team!
@Wtpaul
Now you can solve the mqtt problem! Btw, did you compile to 4mb flash our 2mb flash ESP32 ?

Thanks to all... I have a successful connection with MQTT!! yea!!
I have not taken time to subscribe or publish yet.. will get to that very soon.
Thank you @xcguang for the Chinese link and to google translate :) .
@tcpipchip my ESP-wroom32 is a 4MB device.

Can you put instruction how to download and compile esp32-at to wroom32 and steps to make it work with mqtt ?

make menuconfig > Component config > AT > [*] AT MQTT command support.,
then make

The attached pdf has the AT commands i used to connect to AWS IOT with MQTT
ESP-wroom-32_at_commands.pdf

Is there some tip to compile ?
I followed
https://github.com/espressif/esp-at/blob/master/docs/ESP_AT_Get_Started.md (WROOM partition config)
But, i am always getting

no AT prompt

the at commands are via the UART (pins 16&17 )
not via the USB

hummm, nice! Where is in the make config is this parameter ?

@tcpipchip Maybe https://github.com/espressif/esp-at/blob/master/docs/How_To_Set_AT_Port_Pin.md can help you.

Hi @xcguang
Thank you so much! Then my app is really running, only not outputing to the right way

I cand see the prompt!
but no ECHO when i try to type something. No answer.
Some handshaking that is blocking  ? (RTS, CTS)....

Maybe you can try https://github.com/espressif/esp-at/issues/117

no solved, strange that there is not ECHO  when i type a AT command.
Strange! I am using the same serial of LOG

do you use handsacking ?

still tryng to make work the rcv to at command.
About
osboxes@osboxes:~/esp-at/module_config$ ls module_esp32_default  module_esp8266_default  module_wrover-32 osboxes@osboxes:~/esp-at/module_config$
where i download module_esp32_wroom ?

@Wtpaul
I will need you help! See that my problem! I compiled to a WROOM (esp32_default) and to NINA W102 (2Mb), in both i see the Ready message, but i cant type anything, no echo! Can you report step to step how to config the TXD/RXD log and TXD/RXD at console!
Thank you!

I have not encountered this as i have connected to the uart (pins 16&17) on esp32_DevKitc_v4.
Two things to try.

disable the log output - maybe the secondary stream(ie LOG) is lifting your data and dumping it before the AT command handler can see it.
ensure after each AT command a \r\n is sent. I used 'Real term' - 'Send' tab which allows a Send ascii with EndOfLine '\r' to be sent.

I have not needed to look at flow control.. if the processor is running at 240MHz it is well fast enough to pic uart data at 115200.

i will try
about
UART1 is for sending AT commands and receiving response, but its pins can be changed. The pins of UART1 is in the factory_param.bin, they can be changed in the esp-at/components/customized_partitions/raw_data/factory_param/factory_param_data.csv.
Wich of those must i modify and the compiler identify  to me to define the AT commands to UART 0 , pin 3 and 1 ?
PLATFORM_ESP32,WROOM-32,0xfcfc,1,1,78,1,13,CN,115200,17,16,15,14,-1,-1
PLATFORM_ESP32,WROVER-32,0xfcfc,1,2,78,1,13,CN,115200,22,19,15,14,-1,-1
PLATFORM_ESP32,PICO-D4,0xfcfc,1,3,78,1,13,CN,115200,22,19,15,14,-1,-1
PLATFORM_ESP32,SOLO-1,0xfcfc,1,4,78,1,13,CN,115200,17,16,15,14,-1,-1
PLATFORM_ESP8266,WROOM-02,0xfcfc,1,0,78,1,13,CN,115200,15,13,3,1,-1,-1
PLATFORM_ESP8266,WROOM-5V2L,0xfcfc,1,1,78,1,13,CN,115200,15,13,3,1,5,-1
I ask you because in
osboxes@osboxes:~/esp-at/module_config$ ls
module_esp32_default  module_esp8266_default  module_wrover-32
There is no WROOM

yes
I will do some more tests!
If i change anything, default txd and rxd will be GPIO 16 and GPIO17 to AT commands and GPIO 1 and GPIO3 to log! Right ?

after 33 hours tryng i sucessfully runs the AT COMMANDS version 2.0 on NINA W102, looks the problems was in a python lib xlrd. I observed a warning when was trying to open  the CSV table.
After runs
pip install xlrd --user
I solved my problem!
pip install xlrd --user

@wtpaul
Do your burn the 3 client certificates during the make flash process of after ?

works fine on NINA W102
PLEASE ,
Can you sent step how to connect via AT commands on AWS with Nina

or all commands you did :)

Havn't used this code for ages....
hope these few snips of info help..
`
//init chip
ESP_parms.wifi_mode		=  WIFI_STATION;
res	=  hal_ESP_WROOM_AT(AT_WIFI_MODE,ESP_parms.wifi_mode);
if(res != 1){	return(0);	}
res	=  hal_ESP_WROOM_AT(AT_AUTO_CONNECT,WIFI_AUTOCONNECT);
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_MUX,1);	//multiple connections at a time
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_CIPSSLCONF,WIFI_LINK0,3,0,0);	//SSL one direction only
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_CIPSSLCONF,WIFI_LINK1,3,0,0);	//SSL one direction only
if(res != 1){	return(0);	}

//get time for ssl
res	=  hal_ESP_WROOM_AT(AT_SNTP_SERVER_EN,1,0);	//1= SNTP enabled, 0 = time zone
if(res != 1){	return(0);	}
res	=  hal_ESP_WROOM_AT(AT_SNTP_SERVER_QRY);
if(res != 1){	return(0);	}

//connect to MQTT
res	=  hal_ESP_WROOM_AT(AT_MQTT_USERCFG,4,"wtp");
if(res != 1){	return(0);	}
res	=  hal_ESP_WROOM_AT(AT_MQTT_CONNCFG,30,0,"lwt","out",0,0);
if(res != 1){	return(0);	}
res	= hal_ESP_WROOM_AT(AT_MQTT_CONN_WR,server_tx.host_name,server_tx.host_port_mqtt,1);
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_MQTT_SUB_WR,server_tx.mqtt_subscribe_delta);
if(res != 1){	return(0);	}
res	=  hal_ESP_WROOM_AT(AT_MQTT_SUB_WR,server_tx.mqtt_subscribe_get);
if(res != 1){	return(0);	}

//send data
res	=  hal_ESP_WROOM_AT(AT_CIPSTATUS);
if(res != 1){	return(0);	}
if(	(ESP_parms.wifi_con_status == WIFI_STAT_GOT_IP) &&
	(ESP_parms.tcp_con_status == TCP_UDP_CREATED)	){
	//connected
}
else{
	res	=  hal_ESP_WROOM_AT(AT_CIPSTART,WIFI_LINK1,server_tx.type,server_tx.host_name,server_tx.host_port_https);
	if(res != 1){	return(0);	}
}
//don't request status update when talking to server
status_update_timer		=  200000;

res	=  hal_ESP_WROOM_AT(AT_CIPSEND,WIFI_LINK1,server_tx.data_len);
if(res != 1){	return(0);	}
hal_ESP_WROOM_tx((uint8_t*)server_tx.data,server_tx.data_len);

`

tcc formato livro - 14-11-2019 - NINA W102 - ALEXA.pdf
attached
It´s a piece of my ALEXA ESP32 BOOK

tcc formato livro - 14-11-2019 - NINA W102 - ALEXA.pdf
attached
It´s a piece of my ALEXA ESP32 BOOK

thanx a lot for info *but English only is what i can understand because it my foreign language - otherwise i speak Albanian :)

This is the code i used on an STM32 talking to ESP32 with AT command firmware
Havn't used this code for ages....
hope these few snips of info help..
`
//init chip
    ESP_parms.wifi_mode		=  WIFI_STATION;
res	=  hal_ESP_WROOM_AT(AT_WIFI_MODE,ESP_parms.wifi_mode);
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_AUTO_CONNECT,WIFI_AUTOCONNECT);
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_MUX,1);	//multiple connections at a time
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_CIPSSLCONF,WIFI_LINK0,3,0,0);	//SSL one direction only
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_CIPSSLCONF,WIFI_LINK1,3,0,0);	//SSL one direction only
if(res != 1){	return(0);	}

//get time for ssl
     res	=  hal_ESP_WROOM_AT(AT_SNTP_SERVER_EN,1,0);	//1= SNTP enabled, 0 = time zone
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_SNTP_SERVER_QRY);
if(res != 1){	return(0);	}

//connect to MQTT
    res	=  hal_ESP_WROOM_AT(AT_MQTT_USERCFG,4,"wtp");
if(res != 1){	return(0);	}
res	=  hal_ESP_WROOM_AT(AT_MQTT_CONNCFG,30,0,"lwt","out",0,0);
if(res != 1){	return(0);	}

res	= hal_ESP_WROOM_AT(AT_MQTT_CONN_WR,server_tx.host_name,server_tx.host_port_mqtt,1);
if(res != 1){	return(0);	}

res	=  hal_ESP_WROOM_AT(AT_MQTT_SUB_WR,server_tx.mqtt_subscribe_delta);
if(res != 1){	return(0);	}
res	=  hal_ESP_WROOM_AT(AT_MQTT_SUB_WR,server_tx.mqtt_subscribe_get);
if(res != 1){	return(0);	}

//send data
    res	=  hal_ESP_WROOM_AT(AT_CIPSTATUS);
if(res != 1){	return(0);	}

if(	(ESP_parms.wifi_con_status == WIFI_STAT_GOT_IP) &&
	(ESP_parms.tcp_con_status == TCP_UDP_CREATED)	){
	//connected
}
else{
	res	=  hal_ESP_WROOM_AT(AT_CIPSTART,WIFI_LINK1,server_tx.type,server_tx.host_name,server_tx.host_port_https);
	if(res != 1){	return(0);	}
}

//don't request status update when talking to server

status_update_timer		=  200000;

res	=  hal_ESP_WROOM_AT(AT_CIPSEND,WIFI_LINK1,server_tx.data_len);
if(res != 1){	return(0);	}
hal_ESP_WROOM_tx((uint8_t*)server_tx.data,server_tx.data_len);

`

exact i want STM32 to use with Wifinina

This is my SIM900 module. that I want to connect aws iot topic.

https://github.com/vshymanskyy/TinyGSM/issues/406#:~:text=TinyGSM is using all transport-layer commands with the,more command to open than the standard socket.)

@xcguang @tcpipchip Does ESP32S or ESP32 - WROOM - 32 support AT commands? I tried using QCOM_V1.1 as well.
No reply to AT commands. I have loaded all the tool chain using github clone as well.
My o/p of QCOM is as follows:

Same o/p is available on Arduino IDE as well.
Kindly let me know, where I'm going wrong.

https://nina-w102-at-commands.blogspot.com/2021/08/u-blox-nina-w106-e-comandos-at.html

When I use esp-idf, hello-world project runs comfortably. I have also
installed esp-at folder under esp. When i try to run it, it gives me cmake
error in line 9, 13 and 15.
what should I do so that esp-at runs smoothly.
Also when run from qcom, the console output doesn't show ready on last line.
Can any one help? I'm using Windows 10.
- Thanks & Regards,
Ashish R. Kulkarni
SMIEEE, MIE(India)
Asst. Professor in Electrical Engineering,
Delhi Technological University, Delhi(Formerly Delhi College of Engg.,)
Main Bawana Road, Shahbad Daulatpur
Delhi - 110042 (INDIA)Ph. +919013281975(M)<PHONE_NUMBER>,<PHONE_NUMBER>7 (O)
@., @.**
On Fri, Sep 23, 2022 at 4:38 PM tcpipchip @.***> wrote:

https://nina-w102-at-commands.blogspot.com/2021/08/u-blox-nina-w106-e-comandos-at.html
—
Reply to this email directly, view it on GitHub
https://github.com/espressif/esp-at/issues/75#issuecomment-1256077837,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZQBO6DBQRUMSTUSGQBAOPTV7WFRDANCNFSM4EUCFI6Q
.
You are receiving this because you commented.Message ID:
@.***>


Yandex SMTP don't work

Yandex smtp don't work. Can fix it?
https://img.aoli.ml/gvM84To.png

Use port 587

Your welcome


[docs] add a more complex example case

for now, I use mysql-connector-j deserialization, maybe I need to add a more complex example case.

0.3-beta version


iOS build error: warning: implicit conversion from 'long long' to 'NSInteger' (aka 'int')

/Users/christian/Downloads/flutter/.pub-cache/hosted/pub.dartlang.org/flutter_statusbarcolor-0.2.3/ios/Classes/FlutterStatusbarcolorPlugin.m:15:37: warning: implicit conversion from 'long long'     to 'NSInteger' (aka 'int') changes value from<PHONE_NUMBER>5 to -172247279 [-Wconstant-conversion]     static NSInteger statusBarViewTag =<PHONE_NUMBER>5;

How I use plugin in my code:
FlutterStatusbarcolor.setStatusBarColor(Theme.of(context).accentColor);

flutter doctor

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, v1.12.13+hotfix.5, on Mac OS X 10.15.2 19C57, locale it-IT)
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
[✓] Xcode - develop for iOS and macOS (Xcode 11.3)
[✓] Android Studio (version 3.5)
[!] VS Code (version 1.41.1)
✗ Flutter extension not installed; install from
https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter
[✓] Connected device (1 available)

ETA on this as it generates an XCode warning?


Cannot plot continuous var values to circles

Unfortunately it seems that also circles have a problem with var (probably fixing this would also solve the problems with labels).
I used this code to prepare and plot a variation of the blobs dataset so that the table annotates the circles.
import spatialdata as sd
from spatialdata.models import TableModel
from spatialdata.datasets import blobs
import spatialdata_plot
from napari_spatialdata import Interactive
import numpy as np

sdata = blobs()

# let's fetch the table metadata
region = sdata.table.uns[TableModel.ATTRS_KEY]['region']
region_key = sdata.table.uns[TableModel.ATTRS_KEY]['region_key']
instance_key = sdata.table.uns[TableModel.ATTRS_KEY]['instance_key']

# let's see what the table annotates
print(region)

# it annotates the labels; let's make the table annotate the circles instead
new_table = sdata.table.copy()
sdata.table.obs[region_key] = 'blobs_circles'

# there were 21 labels, but now there are only 5 circles, we need to subset
new_table = sdata.table[:5]
new_table.uns[TableModel.ATTRS_KEY]['region'] = 'blobs_circles'

# double checking that the indices match
print(new_table.obs[instance_key].to_numpy())
print(sdata['blobs_circles'].index.to_numpy())

# they don't, let's fix this
new_table.obs[instance_key] = np.array(range(5))

# the new table is ready, let's replace the old one
del sdata.table
sdata.table = new_table

# ready for viz
sdata.pl.render_shapes('blobs_circles', key='channel_0_sum').pl.show()

# Interactive(sdata)

I get this result

While the one produced by napari_spaitaldata is the following (note, the sizes of the circles are wrong in napari_spatialdata, we are waiting for a fix upstream in napari).

My mistake, I had to use color instead of key. The code correctly plots the data.


SizedWidget - For set all implementations automatically

Created SizedWidget to set SizeUntil.init() as following BoxConstrains and Layout.
Added Example to README.md

I decided to close this PR, because @TechnoUrmish didn't respond of.
Thanks...


How to login with a particular external identity provider bypassing Login.cshtml?

With regard to https://github.com/openiddict/openiddict-core/issues/143
How can I specify to ASP.NET infrastructure that I want to use particular identity provider and navigate directly to it's page but not a Login.cshtml first?
I want to send such a request. How to creates it?
Could you provide a sample code?

This issue is being closed because it has not been updated in 3 months.
We apologize if this causes any inconvenience. We ask that if you are still encountering this issue, please log a new issue with updated information and we will investigate.


UTF-8 bom causes incorrect indentation to be inserted

A UTF-8 bom at the beginning of the file causes the incorrect indentation to be inserted.
const file = project.createSourceFile("test2.ts", "\ufeffmodule Test {\n    export enum Test {\n    }\n}\n");
const enumDec = file.getNamespaceOrThrow("Test").getEnumOrThrow("Test");
enumDec.addMember({ name: "member", initializer: "5" });
console.log(JSON.stringify(file.getFullText()));

Outputs: module*Test*{\n****export*enum*Test*{\n*********member*=*5\n********}\n}\ -- too many spaces after the new enum member.

Ryan pointed me in the right direction to help fix this one: https://github.com/Microsoft/TypeScript/issues/27191#issuecomment-423621491


Fix buildenv that have --prefix hardcoded as they do not respect the ZOPEN_INSTALL_DIR

Projects affected: openssl, curl, xxhash, lz4, zstd, zlib

marked as a bug and enhancement - not sure which :)


Fix MM57562

Summary
Fix MM-57562
Ticket Link
Fix https://mattermost.atlassian.net/browse/MM-57562
Release Note
NONE

@yasserfaraazkhan Here is only needed some smoke testing around latex to make sure nothing has broken.
Both inline ($some latext$) and codeblock (triple back tick with language name latex).

Thank you @yasserfaraazkhan . Not sure if we want to document that as "known limitation", but looks good to me.


Use playwright for unit tests

This PR is an experiment of running our unit tests in webkit, chromium, and firefox using playwright. This can complement our current "run in electron renderer"-approach but only tests from/for the browser-layer can be run.
Open problems

Using Firefox doesn't work me...
Using **/test/**/{common,browser}/**/*.test.js pulls in tests that somehow depends on node modules. That is a sign of bad layering somewhere...

fyi @bpasero @Tyriar

@bpasero Looks like many of the workbench tests make false claims about the layer in which they can run. I don't yet see the full dependency graph but I get loader errors like this:
vs/workbench/test/common/editor/editorDiffModel.test{
  	"phase": "loading",
  	"moduleId": "crypto",
  	"neededBy": [
  		"vs/platform/environment/node/environmentService"
  	]

I am checking to split up workbenchTestServices into the layers...

Thanks @bpasero

@jrieken done via https://github.com/microsoft/vscode/commit/4f4e26aec864e324d48ba558de20d9029e93b405

@jrieken I did a quick pass over our existing tests and moved a few to browser or common where possibly. I left all of the extHost & mainThread tests which live in electron-browser but possibly could move a well to you.
PS: there was some mock stuff in electron-browser used by those that I moved into workbenchTestServices in common.

Thanks @bpasero. Much better now, I still see usage of Buffer and process. I will push changes for low hanging bits

merged so that we can go forward with this, I will create separate issues for failing tests and remaining work


client-id required

client-id required error even i have config.ini configured

exec ./"$BUILD"/zoomsdk  --config config.ini;
put this command in the run() in entry.sh

yes, it works!
thanks!

yes, it works!
thanks!

hey, can you get video recording ??

no, I can not.

@NandyRL @yuminglong318, if you're trying to build a meeting bot check out the Recall.ai API for your meeting bots instead.
It’s a simple 3rd party API that lets you use meeting bots to get raw audio/video from meetings without you needing to spend months to build, scale and maintain these bots.
Here are the API docs: https://docs.recall.ai/page/tutorials


fix(deps)!: drop support for Node 16

BREAKING CHANGE: Dropped support for Node 16

Bumped @octokit/core to 5.0, which requires Node 18
Bumped parse-diff to 0.11

Codecov Report

Merging #373 (88c3578) into trunk (f84dcdc) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 88c3578 differs from pull request most recent head 10aaf4c. Consider uploading reports for the commit 10aaf4c to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@           Coverage Diff           @@
##            trunk     #373   +/-   ##
=======================================
  Coverage   97.56%   97.56%
=======================================
  Files           5        5
  Lines         205      205
  Branches       64       64
=======================================
  Hits          200      200
  Misses          5        5


add eq3-8

same comment as I had on PR 90, sorry I read that one first and this one second
Is there a reason we're using a timestamps in these functions and not block heights? Can we use block heights instead without breaking anything? I'm reminded of the consensus bug we had due to differing timestamps among validators

copying comments from PR 90 here, as they apply here as well:
I think you should never refer to these equations by number because the number of an equation in the paper may change based on some edits to the paper over time. In my PRs I copy the actual equation math and put it directly in the comment. That way folks referencing the paper after the fact can look for the equation in the paper for the relevant information without any possibility that equation 9-11 may be totally different equations in the paper 3 months from now
It also makes my job easier as your PR reviewer, I don't have to go look at the paper to figure out wtf equations 9-11 are to check your implementation of them isn't broken

updated in branch off this PR. will merge in as soon as that passes. then will update branch to integrate merge confs


Feat 24 coords

Closes #24

generates schema to look like this for coords:

Other than the "describedBy" field, we want to resolve the jsonschema keys in snake case. This may require something like:
https://stackoverflow.com/questions/67995510/how-to-inflect-from-snake-case-to-camel-case-post-the-pydantic-schema-validation

(Maybe this can be opened as a separate issue) Other than the "describedBy" field, we want to resolve the jsonschema keys in snake case. This may require something like: https://stackoverflow.com/questions/67995510/how-to-inflect-from-snake-case-to-camel-case-post-the-pydantic-schema-validation

Ah yeah, this is default behavior for the pydantic BaseModel.schema_json method.  Let's investigate this in another thread.


Steps to run LipPostalNet console

I am new to .net! wonder how to run the console,  do I need install C library for libPostal first?  how to link it to this .net app?
Thanks

this is the error I got when run in debug:
←[31mERR←[39m   Error loading transliteration module, dir=(null)
←[90m at libpostal_setup_datadir (libpostal.c:1047) ←[94merrno: No such file o
r directory←[39m
←[31mERR←[39m   Could not find parser model file of known type
←[90m at address_parser_load (address_parser.c:208) ←[94merrno: No such file o
r directory←[39m
←[31mERR←[39m   Error loading address parser module, dir=(null)
←[90m at libpostal_setup_parser_datadir (libpostal.c:1111) ←[94merrno: No such
file or directory←[39m
←[31mERR←[39m   parser is not setup, call libpostal_setup_address_parser()
←[90m at address_parser_parse (address_parser.c:1657) ←[94merrno: No such file
or directory←[39m
←[31mERR←[39m   Parser returned NULL
←[90m at libpostal_parse_address (libpostal.c:1027) ←[94merrno: No such file o
r directory←[39m
how do I set up the datadir?
Thanks


Upgrade to socket.io 1.4

There seem to be no breaking changes (npm test is passing).
Operations returning large results would benefit from the automatic compression.

The ^1.3.7 version range matches version 1.4.x as well. If you reinstall your projects with a clean cache and run npm ls socketio you will see that you get the latest 1.4.
We use Greenkeeper for all our repositories to keep dependencies up to date and it will automatically submit a pull request if a new version that does not match the range or where the build fails got published.

Thank you!


Systemd: reload_signal does not work on CentOS 7

Bug report
System info:

Version used: 0.21.0
OS (cat /etc/*-release): CentOS 7
Kernel (uname -a): 3.10.0

System info:

Create Procfile with the reload_signal option defined.
Export given procfile to systemd.
Start systemd unit.
Run reload command using systemctl.

Expected behavior:
Command reload sends defined signal to the process.
Actual behavior:
systemd sends signal to parent process, whose helper does not handle signal and does not forward to its child process. So child process is unable to trap and process the signal.
Additional info:
Possible solutions:

Declare trap inside init-exporter helpers with given signal and sigHandler which sends signal to child process.
Provide mechanism to use custom ExecReload command (not only kill).

Hi! Thanks for your report and investigation.

Declare trap inside init-exporter helpers with given signal and sigHandler which sends signal to child process.

Signals can't be handled by the helper without sending service command to the background. In this case, we will have to remove exec which helps us with access restrictions (more info).

Provide mechanism to use custom ExecReload command (not only kill).

It is impossible due to security reasons. User must not be able to provide any command which will be executed under root user.
I think we should remove reload_signal support for both init systems.

reload_signal support removed in f147eec

You can try using the command pkill instead kill. It works for me.
ExecReload=/bin/pkill -SIGUSR2 -P $MAINPID

Reload signal feature was reimplemented in 0903f5a9d38ae52d603fd8bb649e7cc84f610a3b


Uppercase hex letters in ETH address fail on Offline Send Step 1

Trying to use this: https://www.myetherwallet.com/#offline-transaction
Input "From Address" for Step 1: Generate Information (Online Computer):
The address I used is: 0xAA7f9d16df7eb2ec96ca6318b0129423730a87ac
Expected: It generates the gas price and nonce.
Actual: It fails to recognize the address and doesn't do anything.
If I change the address to use lower-case "aa" at the beginning, it works as expected.

Thank you!


⚠ Bofrost.fr has degraded performance

In 9f7ccb2, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1269 ms

Resolved: Bofrost.fr performance has improved in 2ff135e.


It is creating only upper half  of the signature bitmap

When i am converting the bitmap into base64 and then i am trying to see it as image on online converter, But its only showing me the upper half part of the signature not, full.
Could you please help me out sir.

Can you try a different converter? It's not possible to diagnose if this is a problem with the SDK or the converter you are attempting to use.

Thanks for quick response. I am sorry i have created issue without investigation. ink-android library is working perfect. Actually logcat was not showing full log of the base64 encoded image, so i was trying to copy those  logs and converting into image. That is why it was showing half image.


gh-2476 Stop defaulting access roles to empty arrays.

Read and Write access role properties are mutually exclusive with read and write AccessPredicate properties.
Also added some @Test annotations (see #2452) and corrected the tests which were broken and now executing again.

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@1fb6823). Click here to learn what that means.
The diff coverage is 63.11%.

:exclamation: Current head 3109a71 differs from pull request most recent head 85074a3. Consider uploading reports for the commit 85074a3 to get more accurate results

@@            Coverage Diff             @@
##             develop    #2477   +/-   ##
==========================================
  Coverage           ?   63.47%
  Complexity         ?     2162
==========================================
  Files              ?      532
  Lines              ?    16892
  Branches           ?     2079
==========================================
  Hits               ?    10722
  Misses             ?     5194
  Partials           ?      976

Impacted Files
Coverage Δ

...v/gchq/gaffer/access/AccessControlledResource.java
0.00% <0.00%> (ø)

...n/java/uk/gov/gchq/gaffer/access/ResourceType.java
0.00% <0.00%> (ø)

...uk/gov/gchq/gaffer/cache/AbstractCacheService.java
0.00% <ø> (ø)

...affer/cache/exception/CacheOperationException.java
0.00% <0.00%> (ø)

...uk/gov/gchq/gaffer/cache/util/CacheProperties.java
0.00% <ø> (ø)

...v/gchq/gaffer/commonutil/ByteArrayEscapeUtils.java
0.00% <0.00%> (ø)

...n/java/uk/gov/gchq/gaffer/commonutil/ByteUtil.java
0.00% <0.00%> (ø)

...uk/gov/gchq/gaffer/commonutil/CommonConstants.java
0.00% <ø> (ø)

...uk/gov/gchq/gaffer/commonutil/ExecutorService.java
0.00% <0.00%> (ø)

...java/uk/gov/gchq/gaffer/commonutil/StreamUtil.java
20.63% <0.00%> (ø)

... and 127 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1fb6823...85074a3. Read the comment docs.


Wordpress example not working with BP_PHP_VERSION=8.0.x and paketobuildpacks/builder-jammy-full:0.3.290

Issue
UI tests detected a problem when binding MySQL service to an app:

This is also reproducible in CLI:
epinio service create mysql-dev mycustom-service
epinio push --name testapp-wordpress-80x --path /home/mmartin/suse/epinio-repos/example-wordpress -e BP_PHP_VERSION=8.0.x -e BP_PHP_SERVER=nginx -e BP_PHP_WEB_DIR=wordpress -e DB_HOST=x8e5ee833a0f2faebaf5c4171baca-mysql -e SERVICE_NAME=mycustom-service
sleep 2 # To avoid binding issues
epinio service bind mycustom-service testapp-wordpress-80x

Erro outputr:
 [stage-ns-wordpress-tes-9519204a1ca4beaac479298ecf45f328991cmc2v] buildpack failed to satisfy "php" dependency version constraint "8.0.x": no compatible versions on "io.buildpacks.stacks.jammy" stack. Supported versions are: [8.1.23, 8.1.24, 8.2.10, 8.2.11]

If trying any of the versions suggested (8.1.23, 8.1.24, 8.2.10, 8.2.11), then it seems it misses a component (ie with version 8.2.11)
epinio push --name testapp-wordpress-8211 --path /home/mmartin/suse/epinio-repos/example-wordpress -e BP_PHP_VERSION=8.2.11 -e BP_PHP_SERVER=nginx -e BP_PHP_WEB_DIR=wordpress -e DB_HOST=x8e5ee833a0f2faebaf5c4171baca-mysql -e SERVICE_NAME=mycustom-service
sleep 2 # To avoid binding issues
epinio service bind mycustom-service testapp-wordpress-8211

Output
rtestapp-wordpress-821-7cbe10c9bfea5ffa81f1c74431347087c62zb859 2023/10/30 14:50:05 [error] 27#0: *1 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught mysqli_sql_exception: Table 'my_database.wp_options' doesn't exist in /workspace/source/app/wordpress/wp-includes/wp-db.php:2056

Notes
Not sure if this may be related to the fact now we are using paketobuildpacks/builder-jammy-full:0.3.290 as default  when I try with paketobuildpacks/builder:0.2.443-full  as it was here  it works.

Possible solutions:

Change UI tests to point at old packet buildback and warn in docs
Try to see if bumping the application may help. Pastin here some possible link to explore courtesy of Tomas: https://core.trac.wordpress.org/browser/tags/6.3.2/src
Identifying exactly how to tackle the Uncaught mysqli_sql_exception

Ref. https://github.com/docker-library/wordpress/discussions/677 also includes patch for WP code.

Ref. https://github.com/docker-library/wordpress/discussions/677 also includes patch for WP code.
Thanks @thehejik . Your fix seems to have worked:

Closing issue after merge and successful UI test here:


updates to kubernetes deployment

create init container in coord to generate jobs
tune worker resource requests
general bug fixes
update app version

@ognots do we still want to merge this? what is the state of this?


What is the order in which props files (explicit & implicit) are imported?

Hi,
I have both explicit import of props file(Temp.props) and implicit import(Directory.Build.props) of props file in in my CSPROJ. I have OutputPath defined in both props file. What is the order in which the props file are imported? i.e. in the below CSPROJ for a particular configuration (x64 Debug), value from which props file will be considered?
Project file:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">
  <Import Project="..\..\..\PropertySheets\Temp.props" />
</Project>

Directory.Build.props file:
<?xml version="1.0" encoding="utf-8"?>
<Project>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x64'">
    <OutputPath>bin\x64\Debug\</OutputPath>
  </PropertyGroup>
</Project>

Temp.props file:
<?xml version="1.0" encoding="utf-8"?>
<Project>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x64'">
    <OutputPath>SomeOtherFolder\Debug\</OutputPath>
  </PropertyGroup>
</Project>

Actual behavior
I am not sure how to check which is in use.
Environment data
`Microsoft (R) Build Engine version 15.9.21+g9802d43bc3 for .NET Framework
<IP_ADDRESS>4
OS info:
OS: Windows 10
Visual Studio: VS2017
Regards
Basanth

Hi, if you want to know the details, you can run Msbuild with /bl to generate a binlog file that you can inspect and you should be able to inspect exact import ordering.
In your specific example though,I believe the directory.build.props will not be imported as I see no Microsoft.common.props being imported (this is done in normal project files either directly or because they use the dotnet sdk through MSBuild), see also https://docs.microsoft.com/en-us/visualstudio/msbuild/customize-your-build?view=vs-2019#directorybuildprops-and-directorybuildtargets

Thanks for sharing the command @rainersigwald @japj. It works


Bug: Adding an "admin.url" config option causes the API to return 404 errors.

Issue Summary
When applying the config option admin.url within the config, the api no longer outputs correct URI's for tags.
I wanted to create a quick docker example however admin.url config forced a https redirect on the domain, so this was not possible.
Server:
Ubuntu 20.04
Ghost latest
Browser:
Linux
Firefox 88
To Reproduce

Set admin url within config:

"admin": {
    "url": "https://admin.example.com"
}

Create a new integration
Copy the key
Curl the webpage

curl -L -H "Accept: application/json" "https://admin.example.com/ghost/api/v2/content/tags/?key=XXX"

Identify the response url for tags to contain 404 page as url

This seems to be a regression for #11043
Technical details:

Ghost Version: 4.6 (Latest, I think it's 4.6)
Node Version: v14
Browser/OS: Firefox
Database: MySQL 8

I'm not able to reproduce this on the v2, v3 or v4 API nor can I think of any reason why those two things would be related.
Can you restart Ghost and try again?


Does not support custom pins/markers

When I drop in my component for a custom pin I made, the generic pin is rendered. Any idea on a fix?

Thank you, you guys are awesome!


Global: colorspace workflow support Display and View inputs

Is your feature request related to a problem? Please describe.
A the moment we only support colorspace input from DCC, but we will need to support to make it more universal because Transcoding can be targeted with Colorspace or Display & View preference.
Describe the solution you'd like
We need to be able to store display and view keys in colorspaceData on instance's representation. Also need to consider to enhance  File Rules settings data model with Display and View.
[cuID:OP-4995]

This already happens in the PR, right? #4291 (files)

Yes and no. We are using the data in the mentioned transcoding exctractor, but it expects we do have them in colorspaceData on a representation. Although Maya is adding them in some cases Nuke/Hiero/Davinci/Flame/Fusion and some others are not.
There are two ways to add them. Once would be to set them in Settings for the project in imageio (like nuke does for baking) or equaier them from DCC active Viewer (similarly Maya does for playblast).

Yes and no. We are using the data in the mentioned transcoding exctractor, but it expects we do have them in colorspaceData on a representation. Although Maya is adding them in some cases Nuke/Hiero/Davinci/Flame/Fusion and some others are not.
There are two ways to add them. Once would be to set them in Settings for the project in imageio (like nuke does for baking) or equaier them from DCC active Viewer (similarly Maya does for playblast).

I still don't fully grasp the issue here. Could you break it down into baby steps for me? :)

I'll give an example from the Nuke point of view. Right now we have a baking process (ExctractReviewDataMov). Back in days, we didn't have the OIIO transcoding functionality in OP, so the only way to secure colorspace consistency for reviewable video files was by defining screen space (view & display) to bake into; from any of defined scene space (working space usually linear or acescg). For this process to be complete, we have to define what is the baking screen space we want to use so it is baked into intermediate mov file which is then used in ExctractReview for generating other representations regarding defined output presets.
So now we're almost able to do the same with the current OIIO transcoding workflow, except we're missing the part where we would collect the current display.
Therefore we would need to modify the current workflow so that we can get a consistent colorspace management with OIIO transcoding. We could do this by either providing a way to collect the actual viewer settings and assign them to the output file or by getting definitions from imageio settings where the production can be locked on baking display/view manually. Both methods should allow us to securely define the colorspace for our output representations and ensure colorspace consistency for the baking process.


gRPC does not build on armv7l because of a weird limitation in boringssl

Hello,
I create the issue here since it directly impacts gRPC and I didn't find a github repo for boringssl.
While trying to build gRPC on an NVidia Tegra K1 chip (ARM Cortex-A15), cmake quickly fails on boringssl with the following error:

CMake Error at third_party/boringssl/CMakeLists.txt:140 (message):
Unknown processor:armv7l

elseif (${CMAKE_SYSTEM_PROCESSOR} STREQUAL "armv7-a")

While this chip is an ARMv7-A, the Linux host reports it as armv7l and that's probably why it fails (I suppose there is the same issue with a Raspberry Pi).
I tried to force the value of CMAKE_SYSTEM_PROCESSOR when invoking cmake but it didn't help. So I ended up patching the CMakeFile:
sed -i -- 's/armv7-a/armv7l/g' $GRPCDIR/third_party/boringssl/CMakeLists.txt

and it works fine!
So I suggest to update the CMakeFile with something like that:
elseif ((${CMAKE_SYSTEM_PROCESSOR} STREQUAL "armv7l") OR ${CMAKE_SYSTEM_PROCESSOR} STREQUAL "armv7-a")

:-)

I confirm the same issue on armv6l (Raspberry Pi). I created a bash script to patch the boringssl makefile that way:
local CMFILE=$GRPCDIR/third_party/boringssl/CMakeLists.txt
sed -i -- 's/STREQUAL "armv6"/MATCHES "^armv6*"/g' $CMFILE || exit_failure
sed -i -- 's/STREQUAL "armv7-a"/MATCHES "^armv7*"/g' $CMFILE || exit_failure

before script:
elseif (${CMAKE_SYSTEM_PROCESSOR} STREQUAL "armv6")
  set(ARCH "arm")
elseif (${CMAKE_SYSTEM_PROCESSOR} STREQUAL "armv7-a")
  set(ARCH "arm")

after script:
elseif (${CMAKE_SYSTEM_PROCESSOR} MATCHES "^armv6*")
  set(ARCH "arm")
elseif (${CMAKE_SYSTEM_PROCESSOR} MATCHES "^armv7*")
  set(ARCH "arm")

And I could build gRPC for Raspberry Pi.

FYI, I submitted a patch to the boringssl team:
https://boringssl-review.googlesource.com/#/c/12240/


Remove requirejs and update packaging for ES6

Switch to rollup for combining/minifying JS code.
Switch to clean-css for combining/minifying CSS code.
Get eslint passing
Switch to esm for Node.JS ES6 support and make Node.js work again, which required re-fixing third-party code shims and upgrading AutoLinker and topojson
Remove generateStubs which is no longer being used.
Fix some merge issues with missing ES6 module imports
Deleted unneeded Source/main.js
Add back VERSION property on Cesium object.

Thanks for the pull request @mramato!

:heavy_check_mark: Signed CLA found.
:grey_question: Changes to third party files were made.

Looks like a file in one of our ThirdParty folders (ThirdParty/, Source/ThirdParty/) has been added or modified. Please verify that it has a section in LICENSE.md and that its license information is up to date with this new version.

Reviewers, don't forget to make sure that:

[ ] Cesium Viewer works.
[ ] Works in 2D/CV.
[ ] Works (or fails gracefully) in IE11.


Add support for Persistent Volume Claims

Add support for Persistent Volume Claims

I will work on this issue.

@durai89 do you need any help?

@durai89 could you plz let me know if you are working on this? else i can take it up

implemented


Nside=4096 is prohibitively expensive

Trying to run LAT LF simulation at Nside=4096. There are 222 detectors. I can only allocate two MPI processes on each KNL node not to exhaust the memory during PySM call. I split the communicator into 256 groups, each with one node and two processes. There are two rank communicators with 256 MPI processes, each trying to simulate 111 detectors. The job did not complete in two hours. The medium and high frequency tubes have 10X as many detectors.

I can run some tests to check if this is expected,
do you have a feeling on how this compares to my benchmark at https://github.com/healpy/pysm/issues/25#issuecomment-500657096?

how many threads are you using per MPI process?

That comment indicates 40 MPI tasks processing a channel every 60 seconds. Naively assuming perfect scaling, 256 MPI tasks would do the same in 10 seconds or 111 detectors in less than 20 minutes. Of course the KNL cores are slower but we have 32 OpenMP threads for each MPI task.

I started with a scaled down test on Popeye (48 core Intel skylake), basically 1/10 of the job:
13 nodes, 2 MPI processes per node, 24 OpenMP/Numba

Initialize sky: 143s
Bandpass integration per channel (10 points like TOAST now): 80s the first channel (loading CIB maps), 6s the others
Smoothing: 7s
Memory (with memreport) 9.5GB after initialization, 10 GB at the end

So just spreading maps over 26 MPI processes keeps memory consumption at ~10GB, is it possible that there are memory issues in other parts of the PySM operator? i.e. the map redistribution?

Thanks Andrea. This is really helpful. I cannot say if the memory consumption is driven by PySM or other parts of OpSimPysm. What happens if you try to run more than 2 processes per node?
memreport gives you the instantaneous memory usage but will not track the high water mark. Unless you are also calling it during PySM it won't be very helpful.

had the same question:
If I run 4 processes per node, same total number of nodes,
Initialize sky: 148s
Bandpass integration per channel (10 points like TOAST now): 80s the first channel (loading and caching CIB maps), 4s the others
Smoothing: 7s
Memory (with memreport) 9.35GB after initialization, 10.5 GB at the end
Total time for 10 channels: 6 minutes
memreport is per node, right?

If I run 6 processes per node, 8 threads, same total number of nodes,
Initialize sky: 137s
Bandpass integration per channel (10 points like TOAST now): 75s the first channel (loading and caching CIB maps), 2s the others
Smoothing: 6s
Memory (with memreport) 7.7GB after initialization, 11.6 GB at the end
Total time for 10 channels: 5 minutes

Correct: memreport is by node.  It is very impressive that you get the same walltime with 4 and 6 processes per node! Can you fit even more processes per node? Are those numbers the available memory on the node or the total allocated?

Wow. That is a lot of memory. Without calling memreport inside PySM those numbers do not mean much.

Comparing the 4x12 to 2x24 the memory consumption is 70% higher. 6x8 needs 130% more memory. Those are notable increases. If the maps were distributed evenly across the communicator, we would not expect any increase, would we?
Perhaps the map distribution in PySM is set up to broadcast the high resolution maps across the communicator before each process extracts its part of the map? If that were the case, it would drive up the memory high water mark and cause an unnecessary out-of-memory. Buffering the broadcast or using scatterv would avoid this.
This all seems compatible with my observation that at nside=4096 we cannot fit more than 2 MPI tasks on each KNL node. What happens when we go to nside=8192? TOAST includes a significant amount of Python glue that does not thread at all so limiting the number of MPI tasks like this is problematic.
The actual execution times you get are much better than my out-of-time test would have suggested. I wonder if OpSimPysm includes costly operations not included in your test. We probably need a test that runs the entire operator, not just PySM.

yes, correct, I broadcast the full map. I looked into using scatterv, but each process has rings which are not consecutive, and maps are often 2D, so it is going to be very complicated.
Also, in the worst case of IQU map at 4096, we are paying a temporary increase of 2.3 GB of memory per MPI process, so let's say you want to run 4 MPI processes per node, that would be ~10 GB. Or if we compare running 2 MPI tasks or 4 MPI tasks, in the second case you pay an extra 4.6 GB per node temporarily. It doesn't seem too bad. I am more worried about other parts of the PySM Operator.
I'll try to install TOAST on Popeye and see if I can run a test of the full OpSimPysm

So that is 40GB extra memory overhead for Nside=8192 and just 4 MPI processes per node? I don't think that is the best approach. If scatterv is not appropriate, then you can broadcast the map in 100MB pieces.

consider that this price is paid only:

at initialization for most components
during 1st channel bandpass integration for CIB (before smoothing)

so there are no timelines yet, so do you plan to have less than 40GB of timelines per node?

At this point the pointing has to be expanded already, so even if the signal doesn't yet exist, pixel numbers and pointing weights are in memory. OpSimPySM may also be adding to an existing signal so you cannot assume that memory to be available as workspace.
Furthermore, we need to have 8 or 16 MPI tasks per node, not 2 or 4.

could we use MPI shared memory?

That would be a step in the right direction. It still means that nside=8192 would require 10GB of workspace but at least it would not scale with the number of MPI tasks per node.

@keskitalo is the PySM call going out of memory in the initialization phase or during integration of the first channel or later on?

I don't think the log gave that information. The program termination under OOM is immediate. I'll check once NERSC is up again.

See mpi shared memory example in TOAST: https://github.com/hpc4cmb/toast/blob/master/src/toast/mpi.py#L532

I think the ideal solution would be to have input files in HDF5 and read them directly with a MPI-IO call into local buffers in each MPI process. @keskitalo would this be an option? Would this solution scale to the jobs you need for the current SO simulations?

So every process would read the same file? I suppose it works until it doesn't. You should make sure to open the files in read-only mode or otherwise the access is not parallel. We usually prefer read and broadcast because it will scale to very large concurrencies.
Are you planning on adding HDF5 support to Healpy? It is a lot of boilerplate to handle otherwise.

Debugging the memory issue with this configuration (after having implemented the mpi_shared feature in #32):
The low frequency LAT simulation only has 222 detectors. The nominal case runs with 32 nodes. I allocated 128 nodes for the realistic case, organized into groups of 16 nodes (8 MPI procsses per node). This job fails with out-of-memory during OpSimPySM.
Submitted equivalent job on Popeye to understand memory usage.

ran on Cori the equivalent of 1 group 16 nodes 8 MPI processes per node, see memreport below:
Memory usage                                                                                total :   94.250 GB  <   94.250 +-    0.000 GB  <   94.250 GB
   available :   71.759 GB  <   72.441 +-    0.188 GB  <   72.663 GB
   available :   72.415 GB  <   72.529 +-    0.058 GB  <   72.663 GB (after GC)
     percent :   22.900 %   <   23.100 +-    0.198 %   <   23.900 %
     percent :   22.900 %   <   23.000 +-    0.079 %   <   23.200 %  (after GC)
        used :    6.147 GB  <    6.356 +-    0.188 GB  <    7.027 GB
        used :    6.146 GB  <    6.270 +-    0.058 GB  <    6.378 GB (after GC)
        free :   72.016 GB  <   72.817 +-    0.198 GB  <   73.052 GB
        free :   72.672 GB  <   72.912 +-    0.079 GB  <   73.053 GB (after GC)
      active :    4.141 GB  <    4.233 +-    0.181 GB  <    4.889 GB                                                                             active :    4.141 GB  <    4.155 +-    0.023 GB  <    4.242 GB (after GC)
    inactive :   13.991 GB  <   14.005 +-    0.046 GB  <   14.154 GB
     buffers :    9.590 MB  <   10.043 +-    7.348 MB  <   40.336 MB                                                                             cached :   15.042 GB  <   15.051 +-    0.046 GB  <   15.200 GB
      shared :   14.621 GB  <   14.629 +-    0.006 GB  <   14.638 GB
        slab :    1.172 GB  <    1.181 +-    0.007 GB  <    1.198 GB

Are you calling memreport at the time of the highest memory requirement or after the operator finishes? Recall that memreport does not track the high watermark but only the current memory.

in my tests on Popeye memreport (inside the Operator) was pretty close with what I got with seff. @keskitalo what do you use to get high watermark on Cori?

I just call memreport where the memory consumption is expected to be the highest.

Asked NERSC, they don't have any suggestion on how to get the high watermark properly.
So I created 4GB timelines for each process,
so now even before PySM we are at:
***** Created fake timelines of 4.00 GB per process
Walltime elapsed since the beginning 6.9 s
Memory usage
       total :   94.250 GB  <   94.250 +-    0.000 GB  <   94.250 GB
   available :   57.072 GB  <   57.503 +-    0.166 GB  <   57.995 GB
   available :   57.072 GB  <   57.428 +-    0.119 GB  <   57.584 GB (after GC)
     percent :   38.500 %   <   39.000 +-    0.176 %   <   39.400 %
     percent :   38.900 %   <   39.100 +-    0.117 %   <   39.400 %  (after GC)
        used :   35.075 GB  <   35.567 +-    0.168 GB  <   36.011 GB
        used :   35.486 GB  <   35.643 +-    0.120 GB  <   36.011 GB (after GC)
        free :   57.483 GB  <   57.911 +-    0.166 GB  <   58.402 GB
        free :   57.483 GB  <   57.839 +-    0.119 GB  <   57.991 GB (after GC)
      active :   32.367 GB  <   32.796 +-    0.113 GB  <   32.941 GB
      active :   32.777 GB  <   32.867 +-    0.028 GB  <   32.941 GB (after GC)
    inactive :  530.578 MB  <  540.186 +-   14.302 MB  <  589.172 MB
     buffers :    8.234 MB  <    8.707 +-    0.320 MB  <    9.312 MB
      cached :  766.215 MB  <  774.553 +-   14.428 MB  <  825.289 MB
      shared :  379.680 MB  <  388.158 +-   14.145 MB  <  436.152 MB
        slab :    1.137 GB  <    1.156 +-    0.012 GB  <    1.187 GB

at the end:
***** Ran smoothing channel 9
Walltime elapsed since the beginning 1428.4 s
Memory usage
       total :   94.250 GB  <   94.250 +-    0.000 GB  <   94.250 GB
   available :   39.601 GB  <   40.327 +-    0.205 GB  <   40.539 GB
   available :   40.177 GB  <   40.445 +-    0.101 GB  <   40.544 GB (after GC)
     percent :   57.000 %   <   57.200 +-    0.219 %   <   58.000 %
     percent :   57.000 %   <   57.100 +-    0.116 %   <   57.400 %  (after GC)
        used :   38.268 GB  <   38.470 +-    0.206 GB  <   39.200 GB
        used :   38.263 GB  <   38.353 +-    0.103 GB  <   38.634 GB (after GC)
        free :   39.989 GB  <   40.716 +-    0.205 GB  <   40.927 GB
        free :   40.566 GB  <   40.832 +-    0.100 GB  <   40.931 GB (after GC)
      active :   36.141 GB  <   36.246 +-    0.180 GB  <   36.906 GB
      active :   36.141 GB  <   36.174 +-    0.025 GB  <   36.246 GB (after GC)
    inactive :   13.996 GB  <   14.005 +-    0.014 GB  <   14.054 GB
     buffers :    9.504 MB  <    9.965 +-    0.309 MB  <   10.516 MB
      cached :   15.040 GB  <   15.049 +-    0.014 GB  <   15.097 GB
      shared :   14.621 GB  <   14.629 +-    0.014 GB  <   14.676 GB
        slab :    1.172 GB  <    1.191 +-    0.012 GB  <    1.222 GB

Now trying with 8GB per MPI process.

works fine also with 8GB timelines per MPI process:
***** Created fake timelines of 8.00 GB per process
Walltime elapsed since the beginning 11.5 s
Memory usage
       total :   94.250 GB  <   94.250 +-    0.000 GB  <   94.250 GB
   available :   25.311 GB  <   25.609 +-    0.225 GB  <   26.557 GB
   available :   25.311 GB  <   25.472 +-    0.108 GB  <   26.098 GB (after GC)
     percent :   71.800 %   <   72.800 +-    0.244 %   <   73.100 %
     percent :   72.300 %   <   73.000 +-    0.115 %   <   73.100 %  (after GC)
        used :   66.527 GB  <   67.475 +-    0.225 GB  <   67.763 GB
        used :   66.985 GB  <   67.611 +-    0.107 GB  <   67.763 GB (after GC)
        free :   25.717 GB  <   26.020 +-    0.226 GB  <   26.970 GB
        free :   25.717 GB  <   25.883 +-    0.109 GB  <   26.512 GB (after GC)
      active :   63.788 GB  <   64.733 +-    0.224 GB  <   64.968 GB
      active :   64.243 GB  <   64.884 +-    0.106 GB  <   64.968 GB (after GC)
    inactive :  531.605 MB  <  533.812 +-    1.932 MB  <  539.012 MB
     buffers :    8.695 MB  <    8.869 +-    0.197 MB  <    9.523 MB
      cached :  762.945 MB  <  765.391 +-    3.761 MB  <  780.059 MB
      shared :  379.426 MB  <  379.451 +-    1.207 MB  <  384.445 MB
        slab :    1.175 GB  <    1.179 +-    0.002 GB  <    1.183 GB

at the end:
***** Ran smoothing channel 9
Walltime elapsed since the beginning 1419.1 s
Memory usage
       total :   94.250 GB  <   94.250 +-    0.000 GB  <   94.250 GB
   available :    7.697 GB  <    8.354 +-    0.186 GB  <    8.496 GB
   available :    8.340 GB  <    8.446 +-    0.038 GB  <    8.496 GB (after GC)
     percent :   91.000 %   <   91.100 +-    0.200 %   <   91.800 %
     percent :   91.000 %   <   91.000 +-    0.061 %   <   91.200 %  (after GC)
        used :   70.315 GB  <   70.456 +-    0.186 GB  <   71.112 GB
        used :   70.315 GB  <   70.366 +-    0.036 GB  <   70.459 GB (after GC)
        free :    8.086 GB  <    8.744 +-    0.186 GB  <    8.884 GB
        free :    8.723 GB  <    8.836 +-    0.039 GB  <    8.884 GB (after GC)
      active :   68.167 GB  <   68.290 +-    0.185 GB  <   68.952 GB
      active :   68.167 GB  <   68.210 +-    0.030 GB  <   68.271 GB (after GC)
    inactive :   13.991 GB  <   14.000 +-    0.006 GB  <   14.017 GB
     buffers :    9.957 MB  <   10.131 +-    0.197 MB  <   10.785 MB
      cached :   15.038 GB  <   15.041 +-    0.004 GB  <   15.057 GB
      shared :   14.621 GB  <   14.621 +-    0.001 GB  <   14.626 GB
        slab :    1.210 GB  <    1.214 +-    0.002 GB  <    1.218 GB

10GB failed with OOM

Looking again to this I realized that the MPI shared memory goes into shared and that is not properly cleared by Python. So I am now calling win.Free() explicitly and memory usage went down.
Now also 10 GB fake timelines per node work fine:
***** Ran smoothing channel 9
Walltime elapsed since the beginning 1449.1 s
Memory usage
       total :   94.250 GB  <   94.250 +-    0.000 GB  <   94.250 GB
   available :    5.609 GB  <    6.718 +-    0.273 GB  <    6.950 GB
   available :    6.005 GB  <    6.819 +-    0.214 GB  <    6.957 GB (after GC)
     percent :   92.600 %   <   92.900 +-    0.285 %   <   94.000 %
     percent :   92.600 %   <   92.800 +-    0.228 %   <   93.600 %  (after GC)
        used :   86.117 GB  <   86.348 +-    0.269 GB  <   87.435 GB
        used :   86.110 GB  <   86.245 +-    0.208 GB  <   87.039 GB (after GC)
        free :    6.010 GB  <    7.123 +-    0.274 GB  <    7.356 GB
        free :    6.406 GB  <    7.224 +-    0.215 GB  <    7.362 GB (after GC)
      active :   83.297 GB  <   83.411 +-    0.190 GB  <   84.059 GB
    inactive :  544.371 MB  <  555.404 +-    5.817 MB  <  571.734 MB
     buffers :    9.344 MB  <   10.277 +-    0.418 MB  <   10.945 MB
      cached :  781.066 MB  <  787.771 +-    7.591 MB  <  814.453 MB
      shared :  382.914 MB  <  388.900 +-    5.399 MB  <  406.613 MB
        slab :    1.123 GB  <    1.134 +-    0.012 GB  <    1.164 GB

See how shared memory is reduced now.
I'm closing this issue, reopen if needed.


Fatal error: Login error: "fail" status, message "CSRF token missing or incorrect".

instaloader --login=x --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"  y
provides
Session file does not exist yet - Logging in.
Enter Instagram password for x

Fatal error: Login error: "fail" status, message "CSRF token missing or incorrect".

same with user-agent and without

your bug report is incomplete. which version are you using?
Login was changed in the last release but tends to be buggy. Check alternatives like grabbing session from browser by script or cmd parameter --load-cookies


Extra cards added

Only 3 cards were suppose to be added but 6 were added. This happened after around the 15th level up. Even more cards started getting added after that. Sometimes it was 3, other times it was 6, 9, then 12, and so on.

This appears to happen when multiple xp orbs are picked up in the same frame.


cmd/go: build -buildmode=plugin -linkshared panic: slice bounds out of range

What version of Go are you using (go version)?
Tested on:

1.11.13 linux/amd64
1.12.9 linux/amd64
1.13 linux/amd64

Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
Inside TravisCI's "xenial" dist. It also reproduces on my local machine.

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/travis/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/travis/gopath"
GOPROXY=""
GORACE=""
GOROOT="/home/travis/.gimme/versions/go1.11.13.linux.amd64"
GOTMPDIR=""
GOTOOLDIR="/home/travis/.gimme/versions/go1.11.13.linux.amd64/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build631088280=/tmp/go-build -gno-record-gcc-switches"

What did you do?
git clone https://github.com/vmarkovtsev/hercules -b reproduce-panic
cd hercules && make
./hercules generate-plugin -n MyPlug -o myplug && cd myplug && make
# runs: go build -buildmode=plugin -linkshared my_plug.go my_plug.pb.go

What did you expect to see?
The built my_plug.so.
What did you see instead?
panic: runtime error: slice bounds out of range [:-2]
goroutine 1 [running]:
cmd/go/internal/load.(*Package).load(0xc0002da900, 0xc000299668, 0xc000453180, 0x0, 0x0)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:1491 +0x36db
cmd/go/internal/load.loadImport(0x0, 0xc00047d781, 0x1e, 0xc000554f00, 0x38, 0xc0000c3680, 0xc000299668, 0xc0000b80f0, 0x2, 0x2, ...)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:511 +0xdf2
cmd/go/internal/load.LoadImport(0xc00047d781, 0x1e, 0xc000554f00, 0x38, 0xc0000c3680, 0xc000299668, 0xc0000b80f0, 0x2, 0x2, 0x1, ...)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:464 +0xaa
cmd/go/internal/load.(*Package).load(0xc0000c3680, 0xc000299668, 0xc000088700, 0x0, 0x0)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:1605 +0x1993
cmd/go/internal/load.GoFilesPackage(0xc0000b6160, 0x2, 0x2, 0x4d3c01)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:2130 +0x543
cmd/go/internal/load.PackagesAndErrors(0xc0000b6160, 0x2, 0x2, 0x0, 0x0, 0x0)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:1956 +0x6ca
cmd/go/internal/load.PackagesForBuild(0xc0000b6160, 0x2, 0x2, 0xa3a450, 0x2, 0x1)
	/usr/local/go/src/cmd/go/internal/load/pkg.go:2023 +0x5d
cmd/go/internal/work.runBuild(0xea4280, 0xc0000b6160, 0x2, 0x2)
	/usr/local/go/src/cmd/go/internal/work/build.go:316 +0x88
main.main()
	/usr/local/go/src/cmd/go/main.go:189 +0x57f

Notes
This used to work fine without go modules.

Although the failure message is different, I feel like the following comment still applies: https://github.com/golang/go/issues/18671

I don't expect -buildmode=plugin -linkshared to work today. Somebody will have to investigate the problems and figure out whether they can be fixed. I don't know of anybody planning to do that.

I'm going to close this as a duplicate of #18671, but please comment if you disagree or if you feel that I misunderstood the issue.

It works if I build everything "old style" inside a GOPATH. This panic happens only when I turn the modules on.

@vmarkovtsev Thanks for the follow-up. I could be wrong about this for sure.
/cc @ianlancetaylor who was the author of the comment I quoted about buildmode=plugin -linkshared

Yes, I think this is a different issue, and we should leave this open separately.
But I also think that what I wrote earlier applies: I don't expect -buildmode=plugin -linkshared to work today. Somebody will have to investigate the problems and figure out whether they can be fixed. I don't know of anybody planning to do that.
I encourage you to investigate the problem and send a fix.

For what it's worth, this is not limited to -buildmode=plugin.  The same happens for me when I try to build a simple hello-world program with one go mod dependency (Go 1.13.1).

Reprioritizing for Go1.14. This is a panic in cmd/go that happens when cfg.BuildLinkshared is set. It doesn't actually seem related to the mechanics of linking and plugins. I think we're trying to trim a so suffix from a file name that happens to be empty.

I'm not able to reproduce this specific failure mode. However, I have a fix for #35759, which seems closely related.

@vmarkovtsev, could you try this using a go command built from head? It may be fixed.
If it is not fixed, please provide an exact command line (and, ideally, redacted source code) that reproduces the problem.

The panic was fixed, so at this point, I don't think this issue covers anything outside of #18671. Let's keep this closed as a duplicate.


new cnn_3dobj does not show up in cmake

and thus won't get built.
see e.g. here

Did you fill Caffe_INCLUDE_DIR, Caffe_LIBS entries correctly?

@rmekdma no, there is no caffe installed here.
but it should still show up as "unavailable" , or write a message like
Module opencv_sfm disabled because the following dependencies are not found: Eigen Glog/Gflags

it's kinda irrelevant now.


Outdated documentation for downloading datasets

On the consuming data page of the docs it says how a data repository can be created:

Run the sfaira download script (sfaira.data.utils.download_all).

I don't think this script or function exists. python -m sfaira.data.utils.download_all errors, and this page in the docs is the only result when searching the code for download_all.
Is there an updated way to do this, and could the docs get updated with this?

Thanks for pointing this out, I will update this on the docs. Right now, you can call https://sfaira.readthedocs.io/en/latest/api/sfaira.data.Universe.download.html#sfaira.data.Universe.download on an instance of Universe which is essentially what this script was wrapping before. I would recommend doing that because this gives you the most fine grained control over which data sets you actually want to download. We might add a CLI command for this too in the future, but defining your downloads in a python session will always result in maximal control.

Thanks for the quick response. That method does seem to work now that I can get a Universe instantiated.
On a related note, the current permissiveness of .download and .load is quite strange to me. If I've specified a set of datasets to load, I would expect them to load. Maybe this should be an error instead of a warning? I think filtering the datasets to load by availability seems like a reasonable step to include prior to loading, but only maybe getting the data I tried to load is unreasonable.

On a related note, the current permissiveness of .download and .load is quite strange to me. If I've specified a set of datasets to load, I would expect them to load. Maybe this should be an error instead of a warning? I think filtering the datasets to load by availability seems like a reasonable step to include prior to loading, but only maybe getting the data I tried to load is unreasonable.

I see your point, this does however create issues for people who do not want to download the entire data base. So far we have operated under the assumption that in a lot of cases, people would not want to download all "mouse" data sets to be able to able to subset what they have to mouse / human. This implies that a missing mouse data set must not result in an error if the Universe is subsetted for mouse. Happy to get feedback on this though if you disagree. We could:

Throw errors if missing data is encountered when data sets are specifically queried (ie no meta data query but only DOI query)


Updating help for task create/update commands

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[x] I adhere to the Command Guidelines.

conflict can not let CI continue. Please solve it first.

@rosanch can you take a look at the conflict?

@jaysterp  @shahzzzam