app_store_connect_api_key doesn't work with key_filepath as Environment Variable - Azure Bamboo

New Issue Checklist

[X] Updated fastlane to the latest version
[X] I read the Contribution Guidelines
[X] I read docs.fastlane.tools
[X] I searched for existing GitHub issues

Issue Description
When trying to create an app_store_connect_api_key using key_filepath, pilot will result in an error.
However, providing the key with key_content does work, but is not a solution for me.
It is expected to put the xxx.p8 file into a separate folder from the project to let my department use fastlane in a CI environment, Azure Bamboo in my case, with developers not allowed to have Apple authorization stuff at their disposal.
Command executed
desc "Push a new beta build to TestFlight"
  lane :beta do
    build_ios_app
    app_store_connect_api_key
    pilot
  end
end

The rest is set with ENV VARS:
LANG=UTF-8 LC_ALL=UTF-8
APP_STORE_CONNECT_API_KEY_KEY_ID=“xxxxxx”
APP_STORE_CONNECT_API_KEY_ISSUER_ID=“xxxxxx”
APP_STORE_CONNECT_API_KEY_KEY="-----BEGIN EC PRIVATE KEY-----\nxxxxxxxxx\n-----END EC PRIVATE KEY-----”
APP_STORE_CONNECT_API_KEY_DURATION=1200
APP_STORE_CONNECT_API_KEY_IN_HOUSE=false
APP_STORE_CONNECT_API_KEY_IS_KEY_CONTENT_BASE64=true

Complete output when running fastlane, including the stack trace and command used

 [REPLACE THIS WITH YOUR INFORMATION]

Environment

 I don't have this, it's running on Azure DevOps

May be, you need change to
LANG=UTF-8 LC_ALL=UTF-8
SPACESHIP_CONNECT_API_KEY_KEY_ID=“xxxxxx”
SPACESHIP_CONNECT_API_KEY_ISSUER_ID=“xxxxxx”
SPACESHIP_CONNECT_API_KEY_KEY="-----BEGIN EC PRIVATE KEY-----\nxxxxxxxxx\n-----END EC PRIVATE KEY-----”
SPACESHIP_CONNECT_API_KEY_DURATION=1200
SPACESHIP_CONNECT_API_KEY_IN_HOUSE=false
SPACESHIP_CONNECT_API_KEY_IS_KEY_CONTENT_BASE64=true

https://github.com/fastlane/fastlane/blob/5d9219e34f21e6e3bb87df87a5e621f1e7e42edc/spaceship/lib/spaceship/connect_api/token.rb#L43-L66

Unfortunately this not resolve the issue.
Using:
LANG=UTF-8 LC_ALL=UTF-8
SPACESHIP_CONNECT_API_KEY_ID=“69CDH97Z78”
SPACESHIP_CONNECT_API_ISSUER_ID="69a6de8a-27bf-47e3-e053-5b8c7c11a4d1"
SPACESHIP_CONNECT_API_KEY="MIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHkwdwIBAQQgcGqNNF4IR/PgTbeR\nwFgr7H6oq7fT5As4UVgdzLntOMugCgYIKoZIzj0DAQehRANCAARLpIF0PFSjLb2M\nYLOqUS52fbRChY6B+TMRb3Gj8rJXzjelB5jmGs32l4HemSKqz3IEZy03VwfNVkgT\n5wLBgV7u”
SPACESHIP_CONNECT_API_TOKEN_DURATION=1200
SPACESHIP_CONNECT_API_IN_HOUSE=false
SPACESHIP_CONNECT_API_KEY_IS_KEY_CONTENT_BASE64=true

I receive another kind of error:
[10:46:00]: ---------------------------------------
20-Oct-2020 10:46:00	[10:46:00]: --- Step: app_store_connect_api_key ---
20-Oct-2020 10:46:00	[10:46:00]: ---------------------------------------
20-Oct-2020 10:46:00	+--------------------+--------------------------------------+
20-Oct-2020 10:46:00	|                       Lane Context                        |
20-Oct-2020 10:46:00	+--------------------+--------------------------------------+
20-Oct-2020 10:46:00	| DEFAULT_PLATFORM   | ios                                  |
20-Oct-2020 10:46:00	| PLATFORM_NAME      | ios                                  |
20-Oct-2020 10:46:00	| LANE_NAME          | ios beta                             |
20-Oct-2020 10:46:00	| IPA_OUTPUT_PATH    | /Users/bamboo/atlassian-staging/rem  |
20-Oct-2020 10:46:00	|                    | oteAgent/AGENT_HOME/xml-data/build-  |
20-Oct-2020 10:46:00	|                    | dir/CICD-FAD-JOB1/application/build  |
20-Oct-2020 10:46:00	|                    | /ios/Mercury3.ipa                    |
20-Oct-2020 10:46:00	| XCODEBUILD_ARCHIVE | /Users/bamboo/Library/Developer/Xco  |
20-Oct-2020 10:46:00	|                    | de/Archives/2020-10-20/Mercury3      |
20-Oct-2020 10:46:00	|                    | 2020-10-20 10.45.55.xcarchive        |
20-Oct-2020 10:46:00	| DSYM_OUTPUT_PATH   | /Users/bamboo/atlassian-staging/rem  |
20-Oct-2020 10:46:00	|                    | oteAgent/AGENT_HOME/xml-data/build-  |
20-Oct-2020 10:46:00	|                    | dir/CICD-FAD-JOB1/application/build  |
20-Oct-2020 10:46:00	|                    | /ios/Mercury3.app.dSYM.zip           |
20-Oct-2020 10:46:00	+--------------------+--------------------------------------+
20-Oct-2020 10:46:00	[10:46:00]: No value found for 'key_id'
20-Oct-2020 10:46:00
20-Oct-2020 10:46:00	+------+------------------------+-------------+
20-Oct-2020 10:46:00	|              fastlane summary               |
20-Oct-2020 10:46:00	+------+------------------------+-------------+
20-Oct-2020 10:46:00	| Step | Action                 | Time (in s) |
20-Oct-2020 10:46:00	+------+------------------------+-------------+
20-Oct-2020 10:46:00	| 1    | default_platform       | 0           |
20-Oct-2020 10:46:00	| 2    | Switch to ios build    | 0           |
20-Oct-2020 10:46:00	|      | lane                   |             |
20-Oct-2020 10:46:00	| 3    | clear_derived_data     | 0           |
20-Oct-2020 10:46:00	| 4    | build_ios_app          | 6           |
20-Oct-2020 10:46:00	| 💥   | app_store_connect_api  | 0           |
20-Oct-2020 10:46:00	|      | _key                   |             |
20-Oct-2020 10:46:00	+------+------------------------+-------------+
20-Oct-2020 10:46:00
20-Oct-2020 10:46:00	[10:46:00]: fastlane finished with errors
20-Oct-2020 10:46:00	[!] No value found for 'key_id'

cc: @joshdholtz
After a lot of tests, it is important to say that "app_store_connect_api_key" is working correctly with the following set up:
    app_store_connect_api_key(
      key_id: "xxx",
      issuer_id: "xxx",
      key_content: "-----BEGIN EC PRIVATE KEY-----\nxxx\n-----END EC PRIVATE KEY-----"
    )

"Pilot" also works fine.
But, If for security reasons, the .p8 content file is assigned to a ENV variable (I'm using Azure Bamboo), Pilot is the anomaly, it is not possible to upload the .ipa. I receive this error:
20-Oct-2020 11:42:44 | [11:42:44]: --- Step: pilot ---
-- | --
20-Oct-2020 11:42:44 | [11:42:44]: -------------------
20-Oct-2020 11:42:44 | +--------------------+--------------------------------------+
20-Oct-2020 11:42:44 | \|                       Lane Context                        \|
20-Oct-2020 11:42:44 | +--------------------+--------------------------------------+
20-Oct-2020 11:42:44 | \| DEFAULT_PLATFORM   \| ios                                  \|
20-Oct-2020 11:42:44 | \| PLATFORM_NAME      \| ios                                  \|
20-Oct-2020 11:42:44 | \| LANE_NAME          \| ios beta                             \|
20-Oct-2020 11:42:44 | \| IPA_OUTPUT_PATH    \| /Users/bamboo/atlassian-staging/rem  \|
20-Oct-2020 11:42:44 | \|                    \| oteAgent/AGENT_HOME/xml-data/build-  \|
20-Oct-2020 11:42:44 | \|                    \| dir/CICD-FAD-JOB1/application/build  \|
20-Oct-2020 11:42:44 | \|                    \| /ios/Mercury3.ipa                    \|
20-Oct-2020 11:42:44 | \| XCODEBUILD_ARCHIVE \| /Users/bamboo/Library/Developer/Xco  \|
20-Oct-2020 11:42:44 | \|                    \| de/Archives/2020-10-20/Mercury3      \|
20-Oct-2020 11:42:44 | \|                    \| 2020-10-20 11.42.40.xcarchive        \|
20-Oct-2020 11:42:44 | \| DSYM_OUTPUT_PATH   \| /Users/bamboo/atlassian-staging/rem  \|
20-Oct-2020 11:42:44 | \|                    \| oteAgent/AGENT_HOME/xml-data/build-  \|
20-Oct-2020 11:42:44 | \|                    \| dir/CICD-FAD-JOB1/application/build  \|
20-Oct-2020 11:42:44 | \|                    \| /ios/Mercury3.app.dSYM.zip           \|
20-Oct-2020 11:42:44 | +--------------------+--------------------------------------+
20-Oct-2020 11:42:44 | [11:42:44]: invalid curve name
20-Oct-2020 11:42:44 |
20-Oct-2020 11:42:44 | +------+------------------------+-------------+
20-Oct-2020 11:42:44 | \|              fastlane summary               \|
20-Oct-2020 11:42:44 | +------+------------------------+-------------+
20-Oct-2020 11:42:44 | \| Step \| Action                 \| Time (in s) \|
20-Oct-2020 11:42:44 | +------+------------------------+-------------+
20-Oct-2020 11:42:44 | \| 1    \| default_platform       \| 0           \|
20-Oct-2020 11:42:44 | \| 2    \| Switch to ios build    \| 0           \|
20-Oct-2020 11:42:44 | \|      \| lane                   \|             \|
20-Oct-2020 11:42:44 | \| 3    \| clear_derived_data     \| 0           \|
20-Oct-2020 11:42:44 | \| 4    \| build_ios_app          \| 5           \|
20-Oct-2020 11:42:44 | \| 5    \| app_store_connect_api  \| 0           \|
20-Oct-2020 11:42:44 | \|      \| _key                   \|             \|
20-Oct-2020 11:42:44 | \| 💥   \| pilot                  \| 0           \|
20-Oct-2020 11:42:44 | +------+------------------------+-------------+
20-Oct-2020 11:42:44 |
20-Oct-2020 11:42:44 | [11:42:44]: fastlane finished with errors
20-Oct-2020 11:42:44 |
20-Oct-2020 11:42:44 | /usr/local/Cellar/ruby/2.7.2/lib/ruby/2.7.0/net/http.rb:1122: warning: The environment variable HTTP_PROXY is discouraged.  Use http_proxy.
20-Oct-2020 11:42:44 | Looking for related GitHub issues on fastlane/fastlane...
20-Oct-2020 11:42:44 |
20-Oct-2020 11:42:45 | ➡  app_store_connect_api_key doesn't work with key_content
20-Oct-2020 11:42:45 | https://github.com/fastlane/fastlane/issues/17340 [closed] 10 💬
20-Oct-2020 11:42:45 | a week ago
20-Oct-2020 11:42:45 |
20-Oct-2020 11:42:45 | ➡  [pilot][spaceship][action] set ASC API key in pilot, new app_store_connect_api_key action, and spaceship fixes for both of those
20-Oct-2020 11:42:45 | https://github.com/fastlane/fastlane/pull/17061 [closed] 25 💬

I'm working on other tests and I noticed that "Pilot" works fine also when the fastfile is configured as follows:
  app_store_connect_api_key(
    #key_id: ${bamboo.ASCAK_KEY_ID},
    #issuer_id: ${bamboo.ASCAK_ISSUER_ID},
    key_content: "-----BEGIN EC PRIVATE KEY-----\nxxx\n-----END EC PRIVATE KEY-----"
  )

${bamboo.ASCAK_KEY_ID} is one of the bamboo environment variables created ad hoc to hide they values, mainly.
So, the main point is if it is secure to set the .p8 content into the fastfile or into the .env file.
If it is possible, with this value, the "key_content", to let other developers perform operations on the Apple Panel without any kind of permission from the project owner.

I'm experiencing this problem as well.
When I register a private key in the environment variable, I get "invalid curve name" and it fails.
invalid curve name

CI: Screwdriver.cd
Action: get_provisioning_profile

I had a similar situation. I was using Environment Variables to populate app_store_connect_api_key and receiving "invalid curve name". It turned out that the formatting for the Environment Variables needed to be addressed. In my case there was an extra set of escaped quotes on the string. I was then able to fix this by removing the unwanted characters from the string.

I found the following workaround.
Replaces the secret key's newline code with "\n" and registers the string in an environment variable.
e.g.)
-----BEGIN PRIVATE KEY-----
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
EEEEEEEEEEEEE
-----END PRIVATE KEY-----

↓
-----BEGIN PRIVATE KEY-----\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\nCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC\nDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD\nEEEEEEEEEEEEE\n-----END PRIVATE KEY-----

After other test I can say that we can close this ticket.
This is my last configuration:
default_platform(:ios)

platform :ios do

desc "Build Binary"
  lane :build do
    	clear_derived_data
    	build_ios_app(
        output_directory: "build/ios"
      ) # gym
  end

desc "Push a new beta build to TestFlight"
  lane :beta do

    build

    app_store_connect_api_key(
      key_id: ENV["KEY_ID"],
      issuer_id: ENV["ISSUER_ID"],
      key_filepath: ENV["KEY_FILE_PATH"]
    )

    pilot

  end
end

As you can see, the App Store connect API parameters are now referencing to Environment variables.
In Bamboo it is possible to define them into the Job, so it let the app be signed and uploaded to TestFlight.
Thank you all.

I found the following workaround.
Replaces the secret key's newline code with "\n" and registers the string in an environment variable.
e.g.)
-----BEGIN PRIVATE KEY-----
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
EEEEEEEEEEEEE
-----END PRIVATE KEY-----

↓
-----BEGIN PRIVATE KEY-----\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\nCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC\nDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD\nEEEEEEEEEEEEE\n-----END PRIVATE KEY-----

I've had problems with secret "key" on JSON format today (a cryptic error message indicating invalid curve name), and fixed it after checking this comment.
So I think part of this comment would help a lot in here (all \n are important). --> http://docs.fastlane.tools/app-store-connect-api/#using-fastlane-api-key-json-file
How can I propose a change for web doc? I didn't find info on contributing guidelines


Support API requests from service accounts

Closes #24

@and-mora Would you be able to test this patch? You can find the plugin artifacts here.
Cheers!

@mahendrapaipuri
I tried the nightly plugin (listed as v1.1.0 in grafana UI).
i used the following cmd:
curl --location "<host>/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report?dashUid=<uid>&from=now-1M%2FM&to=now-1M%2FM" -v -H "Authorization: Bearer <token>"

I get the following error:
logger=context userId=2 orgId=1 uname=sa-grafana-report t=2024-04-18T09:58:13.050862749Z level=error msg="Failed to call resource" error="failed to receive call resource response: rpc error: code = Unknown desc = grafana app URL not configured in JSONData" traceID=
logger=context userId=2 orgId=1 uname=sa-grafana-report t=2024-04-18T09:58:13.05111735Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=<IP_ADDRESS> time_ms=11 duration=11.73287ms size=51 referer= handler=/api/plugins/:pluginId/resources/*

is there something i might have done wrong?
ps: the auth with service account is working correctly against che grafana server (I tried it against the dashboards API)

@and-mora Not the nightly version, I am sorry I was not clear enough. nightly version is current main branch.
You will need to download the artifacts from CI.
Regarding the error you are getting

logger=context userId=2 orgId=1 uname=sa-grafana-report t=2024-04-18T09:58:13.050862749Z level=error msg="Failed to call resource" error="failed to receive call resource response: rpc error: code = Unknown desc = grafana app URL not configured in JSONData" traceID=
logger=context userId=2 orgId=1 uname=sa-grafana-report t=2024-04-18T09:58:13.05111735Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=<IP_ADDRESS> time_ms=11 duration=11.73287ms size=51 referer= handler=/api/plugins/:pluginId/resources/*

You will need to provision the plugin with proper settings. Grafana related config parameters cannot be configured from UI. The error is saying Grafana's App URL is not found which is normally http://localhost:3000. Here is a sample provisioning config file that you will need to install at /etc/grafana/provisioning/plugins/reporter.yml assuming that you are using default Grafana paths.

@and-mora Awesome !! Cheers for testing. Appreciate it!

I just mentioned the name inside the plugin-artifact.zip that I downloaded from the ci action.

Oh yes, that is true. Probably we should use a branch specific name in CI to better communicate what we are testing

Sorry @mahendrapaipuri, maybe I miss-formulated the previous message. The plugin is still not working and returns 401 when I call it with curl.
What I pointed out as working were the other tests I made (plugin with session cookie browser and dashboard API) before testing the plugin itself.

Sorry, I havent looked at the logs keenly. How are you generating the token? Does token has viewer role?

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-04-18T10:35:49.640822156Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard d0a30598-95a5-4098-89cc-435d965df315: error obtaining dashboard from /api/dashboards/uid/d0a30598-95a5-4098-89cc-435d965df315. Got Status 401 Unauthorized, message: {"extra":null,"message":"Unauthorized","messageId":"auth.unauthorized","statusCode":401,"traceID":""}\n " pluginID=mahendrapaipuri-dashboardreporter-app

This line indicates that token does not have permissions to view that dashboard. So you need to verify the permissions on the token you are using to make the request
I tested it using the docker-compose included in the repo and it worked. What I did is:

Created a service account
Generated a token with viewer role
Made API request with that token in header Authorization


Format: Fix indent in arguments

Fixed #8197
Also remove @multiline_call_indent trick introduced by #5234.
It was originally introduced for dot alignment as @dot_column.
However it is removed and this trick can be implemented without any trick.
Thank you.

Thank you @MakeNowJust :heart:


Naming of Key Pairs

Hi,
I have used the Create Key pair(s) option from the Deploy menu. This resulted in the creation of a private key Matrixuseast1-1 in ~/Keys where the filename has suffix -1.
https://github.com/cryptobiu/MATRIX/blob/77a331626027cc4c7c4855aff8fe640ecde15e69/Deployment/aws_deploy.py#L29-L30
During deployment, the Matrix client tries to use the key name from regions.json where this suffix is omitted (it is just Matrixuseast1).
https://github.com/cryptobiu/MATRIX/blob/77a331626027cc4c7c4855aff8fe640ecde15e69/Deployment/aws_deploy.py#L116-L132
Choose deployment task
1. Deploy Instance(s)
2. Create Key pair(s)
3. Create security group(s)
4. Get instances network data
5. Terminate machines
6. Change machines types
7. Start instances
8.

I can change the entry of the regions.json, but I am not sure what happens when there are multiple regions involved.

 Finish integration with Lenneart

Hi,
At your config file, you defined your requested regions:
https://github.com/cryptobiu/MATRIX/blob/77a331626027cc4c7c4855aff8fe640ecde15e69/ProtocolsConfigurations/Config_ABY_InnerProduct.json#L30-L33
The b at the end specify the placement group inside the region.
Since the regions.json file is common to all the placement groups inside a region, the code snippet omits the placemant group specifier and not the suffix of the key name.
Hence you should change the regions.json file according to your key file name.


helper method names quite confusing when using other user_model

If you have both User and Account models in your application, it can be quite confusing to understand the scope of the helper methods. I think user_model option should define the helper method names suffix. Would you accept a pull request for that?

Which helper methods were you thinking, specifically? current_user? I can't say I'm likely to accept making that name dynamic.
If you want current_account then it can be implemented as a helper method:
def current_account
  current_user
end

helper_method :current_account

I can see how having the extra methods would be annoying but I don't think it's worth moving to define_method for.


Add app manifest for better mobile experience

https://developer.mozilla.org/en-US/docs/Web/Manifest
Example file: https://github.com/SteamDatabase/steamstat.us/blob/gh-pages/manifest.json

:+1:


Developers want to build Flutter apps on Windows (not deployment target)

Rough TODO list:

[x] Make //flutter/snapshotter work on Windows.
[x] Make the flutter command line tool work on Windows.

[X] Teach the flutter.ps1 where to get the Dart SDK and how to bootstrap into Dart.
[x] Teach the tool about Windows as a host platform (e.g., where to find the snapshotter)
[x] Teach the tool how to control Android devices via adb on Windows.

[ ] Infrastructure for continuously building and testing the Windows toolchain.

 Issue by Scorpiion
Sunday Aug 23, 2015 at 18:18 GMT
Originally opened as https://github.com/flutter/engine/issues/771

I understand that Windows support is not prio 1 atm, but in the future, as a platform for developing Android apps (and maybe iOS apps), would it be possible to add Windows support? I understand that it would be some work, but basically what I'm asking is if it's part of the long term plan or simply ignored on purpose for some reason (other than maybe no devs on the team are using Windows..).
Even if I myself only use Linux (close to 10 years Linux only) I think it can be good for the project to support Windows in the long run, at Stackoverflow's latest developer survey Windows developers was still a little bit more that 50% (Mac and Linux at ~21% each):
http://stackoverflow.com/research/developer-survey-2015#tech-os
The only reference of Windows I could find was this old comment in an old readme:
https://github.com/domokit/sky_engine/pull/165/files#r35055666
And just to be extra clear and avoid any confusion, this issue is about developing Android and iOS apps on Windows, not deploying Sky to Windows Phone.

 Comment by sethladd
Sunday Aug 23, 2015 at 20:43 GMT

To be clear, are you asking to deploy to windows phone, windows desktop, or
simply develop on windows?
On Sun, Aug 23, 2015 at 11:18 Robert Åkerblom-Andersson <
<EMAIL_ADDRESS>wrote:

I understand that Windows support is not prio 1 atm, but in the future, as
a platform for developing Android apps (and maybe iOS apps), would it be
possible to add Windows support? I understand that it would be some work,
but basically what I'm asking is if it's part of the long term plan or
simply ignored on purpose for some reason (other than maybe no devs on the
team are using Windows..).
Even if I myself only use Linux (close to 10 years Linux only) I think it
can be good for the project to support Windows in the long run, at
Stackoverflow's latest developer survey Windows developers was still a
little bit more that 50% (Mac and Linux at ~21% each):
http://stackoverflow.com/research/developer-survey-2015#tech-os
The only reference of Windows I could find was this old comment in an old
readme:
https://github.com/domokit/sky_engine/pull/165/files#r35055666
And just to be extra clear and avoid any confusion, this issue is about
developing Android and iOS apps on Windows, not deploying Sky to Windows
Phone.
—
Reply to this email directly or view it on GitHub
https://github.com/domokit/sky_engine/issues/771.

 Comment by Scorpiion
Sunday Aug 23, 2015 at 21:39 GMT

Hi @sethladd, "simply develop on windows", yes.
If anyone have a suggestion for a more clear issue name then be my guest and change it or tell me and I change it.
Edit: Added "(development host only, not deployment target)" to the topic for now.

 Comment by eseidel
Sunday Aug 23, 2015 at 22:12 GMT

Yup.  It is definitely be possible to support Windows development of Sky apps (for running on mobile OSes for which there exist Windows SDKs) at some point in the future.  (Not clear one could develop Sky apps for iOS on windows ever for instance.)  Right now we're focused on polishing the actual deployed product first.  If the product isn't first awesome, it doesn't matter what development environments it supports. :)

 Comment by sethladd
Monday Aug 24, 2015 at 03:16 GMT

Thanks! Agreed... there are a ton of Windows developers.
On Sun, Aug 23, 2015 at 2:39 PM Robert Åkerblom-Andersson <
<EMAIL_ADDRESS>wrote:

Hi @sethladd https://github.com/sethladd, "simply develop on windows",
yes.
—
Reply to this email directly or view it on GitHub
https://github.com/domokit/sky_engine/issues/771#issuecomment-133941620.

 Comment by Scorpiion
Monday Aug 24, 2015 at 08:45 GMT

Thanks for the reply @eseidel, sounds great, I think the team have the right focus, looking forward to that initial awesome product! :+1:

Now that Dart SDK 1.13.0 is released, are we getting close to adding support for developing on Windows?  The additional feedback from Windows devs will help you make flutter awesome.

@cooler-king what does Dart 1.13.0 has that could help Flutter and Windows development ? (curious).
I'm currently using Windows, and I'm very interested in testing Flutter.

@adaojunior Maybe nothing.  Just hoping for some movement on this issue.

Any chance we could get an update on this one?  If the answer is still we'll get to it when we get to it, that's fine.  But if you have an estimated month or quarter (without any firm commitment) that would be better.

@cooler-king Have you tried using it though a virtual box VM ?
I haven't yet but I may try in the future.

@adaojunior I tried a couple of months ago to no avail.  Not to say it's impossible, but I couldn't get it fully working at that time.

Sorry, no updates. Thanks for checking in!

I'm curious about this too... There are some pull requests updating instructions on how to run this on Windows (using Python 2) so I'm a little confused about whether this works or not?

I'm a little confused about whether this works or not?

Does not currently work.  The sticking point is that we need to run a binary on the host to snapshot the Dart code before sending it to the target.  We don't have a build of this binary for Windows yet.

See also https://github.com/flutter/flutter/issues/1437

Does the Windows Subsystem for Linux help alleviate any of the problems in bringing Flutter to Windows?

Very likely.

Today I was able to manage to run everything in Oracle VirtualBox hosting Ubuntu/Debian. The only interesting and off-path part was to add USB filter for the phone to VirtualBox and to ensure you connect the phone after VM start but not earlier. I may clarify the experience if someone is interested :-)

Why don't you use PowerShell instead of BAT files? I saw in comments into flutter.bat that you are struggling with a lot of things that are natevily implemented in PS or .net

PowerShell is part of Windows 7.
It is used by chocolatey the package manager recommended by Dart Project for installing tools on Windows.
It is far more easy for people behind a company proxy (see the examples with chocolatey).
No need for external tools such as curl, etc.

Why don't you use PowerShell instead of BAT files?

That's probably a good idea.  We'd certainly welcome a patch that replaced our existing BAT file with PowerShell.

Ive pushed a PR with a PowerShell script that fix the BAT script in addition to provide an alternative (#5484). What exactly should be done to make it works on Windows? It looks that it is almost done, but they are some details failing. For example, flutter doctor tells me:
Android SDK file not found: C:\Users\benjamin\AppData\Local\Android\sdk\platform-tools\adb

But this location is valid, moreover flutter devices works fine... Is it possible to list the tasks to do so as to complete this port?

@bbalet no one (to my knowledge) has looked at building a list of TODOs for making flutter development possible from Windows.  One big step will be that we'll need to start building/publishing sky_snapshot binaries (the binary run on the host to compile Dart code into ARM for the device) for Windows.  That would involve building https://github.com/flutter/engine on Windows, which although I'm sure is possible to make work hasn't been attempted to my knowledge.

I don't know the chromium building tool, but I've made some attempts.
In the source of the engine, windows platform is not defined in flutter/sky/shell/platform
In the contributing guide of flutter/engine, you should add the command gclient runhooks in order to get gn.exe (if i'm not mistaken).
In buildroot -> tools/dart/update.py
+WINDOWS_64_SDK = 'dartsdk-windows-x64-release.zip'

+  elif sys.platform.startswith('win'):
 +    sdk_url = SDK_URL_BASE + WINDOWS_64_SDK
 +    output_file = os.path.join(DART_SDK_DIR, WINDOWS_64_SDK)

In engine -> sky/tools/roll/roll.py (files_not_to_roll):
 + 'tools/android/VERSION_WINDOWS_NDK',
 + 'tools/android/VERSION_WINDOWS_SDK',

I don't know what is the process in charge of filling these files with what looks like a git revision or the hash of a distributed package (the content is not the same that what we can get from the git repo or from the binary release). Anyway, the two new files must be added but I don't know how and with what content.
As I said, I don't know the building tool, so I'm stuck with these errors:
PS C:\tools\engine\src> python ./flutter/tools/gn --android --unoptimized
gn gen --check in out\android_debug_unopt
ERROR at //build/config/BUILDCONFIG.gn:589:5: Assertion failed.
    assert(false, "Unknown host for android cross compile")

And this one due to the fact that sky/shell is not yet implemented for Windows:
PS C:\tools\engine\src> python ./flutter/tools/gn --unoptimized
gn gen --check in out\host_debug_unopt
ERROR at //flutter/sky/shell/BUILD.gn:487:3: Assertion failed.
  assert(false, "Unsupported platform")

In the source of the engine, windows platform is not defined in flutter/sky/shell/platform

We should try to get //flutter/snapshotter working first.  It has a lot less code in it and having the snapshotter will unblock developing on Windows.  (We only need sky_shell if we want to run tests on Windows hosts.)

In buildroot -> tools/dart/update.py

That looks like a good patch.  Would you be willing to contribute it?

I don't know what is the process in charge of filling these files with what looks like a git revision or the hash of a distributed package (the content is not the same that what we can get from the git repo or from the binary release)

For the change to buildroot, just send a pull request to https://github.com/flutter/buildroot and at-message me.  After the change lands, we'll need to update the DEPS file in engine.git, but I can take care of that mechanical step for you.

python ./flutter/tools/gn --android --unoptimized

We don't need to cross-compile the Android bits from Windows.  We can build the Android bits from Linux or Mac, and that's sufficient.  What we need is to be able to build the sky_snapshot host binary for Windows.  That means we'll want to invoke:
python ./flutter/tools/gn
and use is_windows checks in the BUILD.gn file to limit the build to just //flutter/snapshot.  That should simplify the problem dramatically because there's much less code to build.

And this one due to the fact that sky/shell is not yet implemented for Windows

Yes, the good news is that we don't need an implementation of sky_shell for Windows, at least not to get developers on Windows going.

Actualy you can reopen my patch https://github.com/flutter/buildroot/pull/3/files
Don't forget sky/tools/roll/roll.py once you've updated the DEPS file

I've update src/flutter/BUILD.gn:
  } else {
     if (current_os == "win") {
         deps = [
           "//flutter/snapshotter($host_toolchain)",
         ]
       } else {
         deps = [
           "//flutter/sky",
         ]
     }
   }

I've install Visual Studio 2015 update 3 with the recommended options. But now I get this error:
gn gen --check in out\host_debug
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:/tools/engine/src/build/vs_toolchain.py", line 21, in <module>
  File "C:/tools/engine/src/build/vs_toolchain.py", line 21, in <module>
        import gyp
import gyp
ImportErrorImportError: : NNo module named gypo module named gyp
Traceback (most recent call last):

  File "C:/tools/engine/src/build/vs_toolchain.py", line 21, in <module>
    import gyp
ImportError: No module named gyp
ERROR at //build/config/win/visual_studio_version.gni:27:7: Script returned non-zero exit code.
      exec_script("../../vs_toolchain.py", [ "get_toolchain_dir" ], "scope")
      ^----------
Current dir: C:/tools/engine/src/out/host_debug/
Command: C:/Python27/python.exe C:/tools/engine/src/build/vs_toolchain.py get_toolchain_dir
Returned 1.

I've set DEPOT_TOOLS_WIN_TOOLCHAIN to 0.

In order to solve the issue about Python module gyp, I had to install gyp (I don't know if it is the normal process, because I thought that gclient would install all dependancies):
git clone https://chromium.googlesource.com/external/gyp
cd gyp
python ./setup.py install

But another error appeared and I've noticed that into {build folder}/src/tools/gyp/pylib/gyp there is only one file related to mac (again I don't know the building system, but I guess that a variable indicating that win is a supported platform is missing somewhere).
So I ended up to copy the content of {gyp install}\gyp\build\lib\gyp into {build folder}/src/tools/gyp/pylib/gyp
Now, when I run python ./flutter/tools/gn, I get this error:
gn gen --check in out\host_debug
ERROR at the command-line "--args":1:180: Build argument has no effect.
use_glfw=false is_clang=true use_goma=false dart_experimental_interpreter=false
use_aura=false is_debug=false dart_runtime_mode="develop" flutter_runtime_mode="
debug" flutter_aot=false goma_dir="None" use_system_harfbuzz=false
               ^----
The variable "flutter_aot" was set as a build argument
but never appeared in a declare_args() block in any buildfile.

I've solved the issue with the build file and I got one version that produces the ninja files:
    if (current_os == "win") {
        import("//flutter/common/config.gni")
        import("//flutter/lib/snapshot/toolchain.gni")
        deps = [
          "//flutter/assets",
          "//flutter/snapshotter",
          "//flutter/snapshotter($host_toolchain)",
        ]
      } else {
        deps = [
          "//flutter/sky",
        ]
    }
  }

But it seems to require some mojo bindings and in DEPS file there is nothing for Windows (only Mac and Linux) for mojom_tool and mojom_generators sections. I don't know what to put into the parameters and if it exists on the server side (I mean on Google storage) because after some search, I'm unable to find an example of a DEPS file that uses win platform.

This is relevant to my interests.  Is there any work in progress on this or where should I start?  I've read the thread and I'd like to know where help is needed right now. @abarth / @bbalet / @eseidelGoogle
Is there some documentation I can dig into?

@binary132 the current state is basically what it says in the first comment in this bug.

FYI, I attempted to setup flutter very briefly just now with bash on windows 10 (and failed):

Installed linux (http://www.windowscentral.com/how-install-bash-shell-command-line-windows-10)
Installed git and unzip (sudo apt-get install git unzip)
Cloned flutter git clone https://github.com/flutter/flutter.git
Ran flutter doctor
Hangs indefinitely at "Building Flutter tool..."
Doctor with -v didn’t show anything.

Further investigation, turns out that it's pub get hanging inside flutter_tools.  Even when I ran it with -v it still just seemed to freeze w/o any error.  This may be folly attempting to get pub working inside Windows bash, may be better to use the normal cmd.exe.

I suspect on the medium term we will want to run in native cmd.exe windows anyway, just to limit our dependencies.

+1 to getting things working in the standard windows Command Prompt. We current fail with a bunch of path issues:
C:\Users\Michael Thomsen>set PATH=%PATH%;"C:\Users\Michael Thomsen\flutter\bin"
C:\Users\Michael Thomsen>flutter doctor
The system cannot find the path specified.
The system cannot find the path specified.
The system cannot find the file C:\Users\Michael.
Updating flutter tool...
'pub.bat' is not recognized as an internal or external command,
operable program or batch file.
'pub.bat' is not recognized as an internal or external command,
operable program or batch file.
'dart' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the path specified.
'dart' is not recognized as an internal or external command,
operable program or batch file.

We current fail with a bunch of path issues

This is not related to this issue. You are missing some prequisites such as Dart (maybe not in the path or not installed). Even if your setup is complete, Flutter doesn't support Windows yet as the snapshotter has been ported to Windows.

flutter doctor is expected to download Dart (like it does on Mac and Linux). But we need changes to the \bin\flutter script to make that work on Windows.
And, yes, we need snapshotter.

We should try to get //flutter/snapshotter working first. It has a lot less code in it and having the snapshotter will unblock developing on Windows. (We only need sky_shell if we want to run tests on Windows hosts.)

I've generated vs files with gn and major issue is the dependencies on fuschia/ftl. The Fuchsia Template Library is highly platform dependent and only supports poisix systems.

I am making good progress on this issue: I am currently able to compile a Flutter app on Windows and deploy it from Windows on Android (in other words: flutter run --release is pretty much working on Windows). Most of the bits and pieces are already checked in for that and I am working on resolving the remaining smaller issues. I also just started work on enabling the fast development refresh cycle ("hot reload" with flutter run). Once those two pieces are in, I'll provide some instructions for using Flutter on Windows for early adopters (sharp edges included). Watch this space for updates!

Hi goderbauer,
Are there some instructions on how the Flutter/Windows is to work with DART?  Please advise.  Appears to be possible, but I don't see a link to the solution.

Hi @davisnrjr,
while many pieces are already in place, Windows support for Flutter is not 100% there yet. I am currently working on resolving the last few issues and once that is done, I will provide step-by-step instructions for using Flutter on Windows. Stay tuned!

@goderbauer will this also make it possible to help in the development of the framework itself on windows?

Awesome

@goderbauer   Awesome, thx!!

@b-strauss You will be able to make contributions to anything in the flutter/flutter repository from Windows. If you want to contribute to flutter/engine, you are most likely limited to changes that don't effect the Flutter Android or iOS runtime as we currently don't have plans to make those runtimes compilable on Windows (they ship as a pre-built binary for users of the Flutter framework).

@goderbauer is progress far enough that we should close this and track with other (smaller) bugs?

Today is the day! If you want, you can try out Flutter on Windows. I posted some setup instructions to get started on Windows. Please be aware that Windows-support for Flutter is brand-new. Do expect some sharp edges and please file bugs when you encounter them. There are some known issues, the two main ones are:

Hot Reload is slower on Windows than it is on other platforms (#8513)
The flutter test command is not supported on Windows (#8516)

I hope you follow these instructions to give Flutter on Windows a try.
Feel free to report back on this bug if that worked for you and you got your first Flutter app deployed from Windows.

I've been building my app for days on Windows and most things work great!
Sometimes relaunching an app from Android Studio complains about needing to replace the APK and bad things happen if you let it... but if you just ignore it and retry until it stops complaining that seems to work.

@rbishop-bah Thanks for confirming that things work. Can you please file a new bug for the Android Studio issue you are seeing with the exact error? Thanks.

@goderbauer Thank you very much for paying so much effort of making Flutter available for developers on Windows like me. I can't wait until I have some time to try it out. One question: is it really necessary to install Android Studio, or do I only need the android_sdk? (forgive my English)

@CoenB95 I believe, Android Studio is the only official way to get the Android SDK these days. In addition to using the Android SDK from Android Studio, Flutter also uses Gradle from Android Studio and possibly other components. In other words: in theory you probably don't need the full Android Studio installation, but you'd currently be on your own to figure out what components you would need to install instead and from where to get them. This is true for all platforms (Windows, Mac, Linux) that Flutter supports.

I see. I thought I only needed the sdk, which I could get using the standalone ADB tool, but if you say Grade is needed too, than I indeed wouldn't know where to get it from. Tomorrow I'll try it out using the IntelliJ plugin. If I don't forget it, I'll try to give feedback.

@goderbauer You can also download the SDK separately. Do I need Android Studio, or is IntelliJ IDEA + Android Plugin enough?

Cool, it works! Finally! Installed and configured everything as back-burner to other activities and it took only half an hour! Just hot-reloaded sample app on my cellphone. Thank you very much!

@b-strauss: For the time being we require Android Studio to be installed if you want to target Android as Flutter uses some of its components to deploy on Android (this is true on all supported host platforms, not only Windows). You don't have to use Android Studio to write your code, though. For that we recommend IntelliJ + Dart Plugin + Flutter Plugin (see https://flutter.io/intellij-setup/).
Having said that, if you absolutely want, you could download and install the components Flutter requires from Android Studio separately. But that's not officially supported nor documented and might break at any time. You'd be on your own there.

Bravo @goderbauer!

Demo app works for me on Windows 10. \o/

Good morning. I've just succeeded (yeey) to run flutter doctor. Some feedback so far:

I could not use PowerShell right-away; it said "running scripts is disabled on this system" (tips welcome).
Cmd did work. But first I needed to add a PATH so flutter could find Git (I used git bash, therefore it wasn't required earlier).
The first time I tried, I had to interrupt the internet connection (Wifi on the bus was to slow) while "transferring BITS". This resulted in an endless loop instead of just failing where the doctor constantly retries to get a connection. Is this loop expected behavior, or should I open a issue (screenshot available)?
Second try with better Wifi resulted in a success. All marks were checked (including already installed InteliJ-plugin), except the android_sdk which is to old according to flutter (shall update). I believe it is recommended to check regularly in order to keep that sdk up-to-date?

Hello CoenB95,

I could not use PowerShell right-away; it said "running scripts is disabled on this system" (tips welcome)
For safty reasons it's not possible to run unsigned scripts by default. A Power Shell Script could do something unwanted on your system.

You can activate the execution with the following command:
Set-ExecutionPolicy Unrestricted

Having some trouble running now after a flutter upgrade this morning: https://github.com/flutter/flutter/issues/8607

@CoenB95

I could not use PowerShell right-away; it said "running scripts is disabled on this system" (tips welcome).

In a PowerShell, execute Set-ExecutionPolicy RemoteSigned to allow running scripts. I am investigating if there is a better way, otherwise I will add this step to the setup instructions (see #8615).

Cmd did work. But first I needed to add a PATH so flutter could find Git (I used git bash, therefore it wasn't required earlier).

The setup instructions say that you should make sure to have Git added to the PATH by the Git installer. I will add a note for people installing Git from other sources.

The first time I tried, I had to interrupt the internet connection (Wifi on the bus was to slow) while "transferring BITS". This resulted in an endless loop instead of just failing where the doctor constantly retries to get a connection. Is this loop expected behavior, or should I open a issue (screenshot available)?

Please file a separate bug.

Second try with better Wifi resulted in a success. All marks were checked (including already installed InteliJ-plugin), except the android_sdk which is to old according to flutter (shall update). I believe it is recommended to check regularly in order to keep that sdk up-to-date?

There shouldn't be a need to manually check this. The tool should tell you if it can't perform a certain action because the SDK is too old. You can update then.

@rbishop-bah @jsiedentop I replied to your bugs on the issues you filed. Thanks for the reports.

Thanks everybody for trying out Flutter on Windows and for reporting back. I am now going to close this issue as the main goal is achieved. Going forward, we will track the remaining work for Windows in other (smaller) issues.
If you encounter any new bugs, please open new issues and don't reply to this one. Feel free to mention me in the issue, if it is related to Windows.

Windows support still refer to this pr in flutter website https://flutter.io/setup/


Green wire

What would happen if you connected the green wire that gets left loose in the same spot as the other end of the green wire on the chip?

Ha I've thought the same thing myself using a diode inline

The green wire is the other half of the communication from the BMS to the Board. By cutting and leaving disconnected the Owie can pretend to be the BMS. The Owie can read the data from the BMS off the white wire, spoof what it needs then modulate the white and green out to the Board. Reconnecting it is not needed. Adding a diode inline just blocks the flow of electricity, rendering the line useless, the same as not connecting it. Connecting it to the Owie in the same spot the line to the Board is connected would cause both the BMS and Owie to try sending data on the line to the Board. most likely it would all come through as gibberish and an error 16.
Just leave it disconnected and wrapped in tape or heat shrink.

Mr HaleYa is spot on.


Implement a log folder command-line parameter, else date-time in current folder

For trace start.
Add a command-line parameter to accept an absolute or relative log folder name.
If the folder exists, check whether it's empty.
If not, prompt the user to delete the contents or use a datetimestamp-named folder.
Store the folder name in a system environment variable.
For trace stop, read the folder name from the system environment variable, if not supplied in the command-line.

Done


Error: 302823254 is not callable.

When downloading a song I get this error
Error: 302823254 is not callable.       While Error : Calvin Harris, Dua Lipa -  One Kiss (with Dua Lipa)  Traceback (most recent call last):   File "/usr/local/lib/python3.9/site-packages/spotdl/download/downloader.py",  line 270, in download_song     youtubeHandler.streams.filter(only_audio=True)   File "/usr/local/lib/python3.9/site-packages/pytube/__main__.py", line 311, in streams     return StreamQuery(self.fmt_streams)   File "/usr/local/lib/python3.9/site-packages/pytube/__main__.py", line 223, in fmt_streams     extract.apply_signature(self.player_config_args, fmt, self.js)   File "/usr/local/lib/python3.9/site-packages/pytube/extract.py", line 476, in  apply_signature     new_n = cipher.calculate_n(initial_n)   File "/usr/local/lib/python3.9/site-packages/pytube/cipher.py", line 63, in  calculate_n     raise TypeError(f'{curr_func} is not callable.') TypeError: 302823254 is not callable.

I have the same error. I hope to have a solution.

https://github.com/pytube/pytube/issues/1053

pytube/pytube#1053

It seems we have no solution yet :(

Hi folks, I believe this is a transient issue. I cannot reproduce the issue any more, nor can one of the people who reported the issue in pytube. Can you confirm if this still affects you?

Hi folks, I believe this is a transient issue. I cannot reproduce the issue any more, nor can one of the people who reported the issue in pytube. Can you confirm if this still affects you?

Can confirm it now works, doesn't affect me anymore

Hi folks, I believe this is a transient issue. I cannot reproduce the issue any more, nor can one of the people who reported the issue in pytube. Can you confirm if this still affects you?

Now it works! Thanks!


ddpm is disabled?

Hi, I find this code disables the instantialization of GaussianDiffusion since it does not allow self to be GaussianDiffusion.
Why? Is this just a base class that can not be directly used?
https://github.com/lucidrains/denoising-diffusion-pytorch/blob/23fd887a5fcd48468bb79c744acb1e8bb8b0dc3b/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L431

the logic reads: the base class cannot be used if the unet does not produce an output that is equal to the number of input channels

the logic reads: the base class cannot be used if the unet does not produce an output that is equal to the number of input channels

Thank you sooo much~


Build failure: stalwart-mail.webadmin

Steps To Reproduce
Steps to reproduce the behavior:

nix build .#stalwart-mail.webadmin

Build log
error: builder for '/nix/store/snfda1ymaya1m3lygfd602iqlwnm7njf-webadmin-0.1.17.drv' failed with exit code 1;
       last 10 log lines:
       > error from build pipeline
       >
       > Caused by:
       >     0: error from asset pipeline
       >     1: couldn't find application wasm-bindgen
       > Error: error from build pipeline
       >
       > Caused by:
       >     0: error from asset pipeline
       >     1: couldn't find application wasm-bindgen

Full log: https://gist.github.com/pacien/c1c93a36f6af51496ae75e53217b9ed7
Additional context
/
Notify maintainers
@onny @Shawn8901
Metadata
Failing on master at 'github:NixOS/nixpkgs/807e9154dcb16384b1b765ebe9cd2bba2ac287fd'.

Add a :+1: reaction to issues you find important.

Currently bisecting this

The cause is the version bump of wasm-bindgen-cli in #349729.

@onny why are we using trunk-ng and not trunk? This seems to be a fork of trunk but at least in nixpkgs it hasn't been update.

Fyi: Not sure why I am pinged/listed here as maintener, I am not listed as maintener neither for stalwart, the web admin nor the module, tho I am a stalwart user.

@onny why are we using trunk-ng and not trunk? This seems to be a fork of trunk but at least in nixpkgs it hasn't been update.

back then i had more success with trunk-ng, but not sure if this is still the case. we might want to switch back to trunk if it is the tooling used upstream

Switching this trunk gives us a better error message :D
couldn't find the required version (0.2.93) of the application wasm-bindgen (found: 0.2.95), unable to download in offline mode

+1
building the system configuration...
error: builder for '/nix/store/3p6brxrhjc23praqq959km0wvmj2bh0q-webadmin-0.1.17.drv' failed with exit code 1;
       last 10 log lines:
       > error from build pipeline
       >
       > Caused by:
       >     0: error from asset pipeline
       >     1: couldn't find application wasm-bindgen
       > Error: error from build pipeline
       >
       > Caused by:
       >     0: error from asset pipeline
       >     1: couldn't find application wasm-bindgen
       For full logs, run 'nix log /nix/store/3p6brxrhjc23praqq959km0wvmj2bh0q-webadmin-0.1.17.drv'.
error: 1 dependencies of derivation '/nix/store/3plag6xpz00xqd6yg88zpyhpdd4md4dl-stalwart-mail.toml.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qldpljhcs6ixbkz9hb8ggp6091vj56j4-unit-stalwart-mail.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/fz0jkkv7s4naka94bdirfy8hhpyf4nx5-system-units.drv' failed to build
error: 1 dependencies of derivation '/nix/store/2py6jl87z6a6nqmzvdz9caqx0kv6bxk7-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/b90fipwdfhvd3q5rf51w15cvs1xp877l-nixos-system-Kolyma-1-24.05.20241030.080166c.drv' failed to build

Ok so the problem is that wasm-bindgen crate, which the application uses and the wasm-bindgen-cli are required to be the exact same version. I'll create a PR to pin this for the webadmin derivation and will switch to trunk

Use this as a workaround until the PR this the relevant branches.
    nixpkgs.config.packageOverrides = pkgs: {
      stalwart-mail = pkgs.stalwart-mail.overrideAttrs (old: {
        passthru.webadmin = pkgs.stalwart-mail.webadmin.override {
          wasm-bindgen-cli = pkgs.wasm-bindgen-cli.override {
            version = "0.2.93";
            hash = "sha256-DDdu5mM3gneraM85pAepBXWn3TMofarVR4NbjMdz3r0=";
            cargoHash = "sha256-birrg+XABBHHKJxfTKAMSlmTVYLmnmqMDfRnmG6g/YQ=";
          };
        };
      });
    };

@xanderio I'm very sorry for disturbing you, I'm still experiencing the issue. If you have some spare time, can you check my configuration please?
https://github.com/kolyma-labs/instances/blob/main/nixos/kolyma-1/services/mail.nix

@orzklv You need to override the correct package. https://github.com/kolyma-labs/instances/blob/main/nixos/kolyma-1/services/mail.nix#L39

@xanderio Thank you very much!


Ambari 25265. upgrade AngularJS to v1.7.5 in ambari-admin ui

What changes were proposed in this pull request?
upgrade AngularJS to v1.7.5 in ambari-admin ui
How was this patch tested?
manually
Please review Ambari Contributing Guide before opening a pull request.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5212/
Test PASSed.


CUDA OOM Error during Compression of Meta-Llama/Llama-2-13B-Chat-HF to W8A8 on 80GB A800 GPU

Describe the bug
When attempting to compress the Meta-Llama/Llama-2-13b-chat-hf model to W8A8 using a combination of GPTQ and SmoothQuant algorithms on an NVIDIA A800 GPU with 80GB of VRAM, I encountered a CUDA OOM (Out of Memory) error. The issue occurs specifically when compressing model.layers.40
Expected behavior
The model’s original weight file size is 26GB in FP16 format, so I expected that 80GB of GPU memory should be sufficient to complete the quantization. However, the OOM occurs before compression completes, suggesting that memory usage during the compression process significantly exceeds the weight file size.
Environment
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.27.7
Libc version: glibc-2.35
Python version: 3.8.18 (default, Sep 11 2023, 13:40:15)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.119-19-0009.11-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A800-SXM4-80GB
Nvidia driver version: 470.182.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          232
On-line CPU(s) list:             0-231
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7K83 64-Core Processor
CPU family:                      25
Model:                           1
Thread(s) per core:              2
Core(s) per socket:              58
Socket(s):                       2
Stepping:                        1
BogoMIPS:                        4890.80
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat umip pku ospke vaes vpclmulqdq rdpid fsrm
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       3.6 MiB (116 instances)
L1i cache:                       3.6 MiB (116 instances)
L2 cache:                        58 MiB (116 instances)
L3 cache:                        512 MiB (16 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-115
NUMA node1 CPU(s):               116-231
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Versions of relevant libraries:
[pip3] numpy==1.24.4
[pip3] nvidia-cublas-cu11==<IP_ADDRESS>
[pip3] nvidia-cublas-cu12==<IP_ADDRESS>
[pip3] nvidia-cuda-cupti-cu11==11.7.101
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu11==11.7.99
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu11==11.7.99
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu11==<IP_ADDRESS>
[pip3] nvidia-cudnn-cu12==<IP_ADDRESS>
[pip3] nvidia-cufft-cu11==<IP_ADDRESS>
[pip3] nvidia-cufft-cu12==<IP_ADDRESS>
[pip3] nvidia-curand-cu11==<IP_ADDRESS>
[pip3] nvidia-curand-cu12==<IP_ADDRESS>
[pip3] nvidia-cusolver-cu11==<IP_ADDRESS>
[pip3] nvidia-cusolver-cu12==<IP_ADDRESS>
[pip3] nvidia-cusparse-cu11==<IP_ADDRESS>
[pip3] nvidia-cusparse-cu12==<IP_ADDRESS>
[pip3] nvidia-dali-cuda120==1.36.0
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu11==2.14.3
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvimgcodec-cu12==<IP_ADDRESS>
[pip3] nvidia-nvjitlink-cu12==12.4.99
[pip3] nvidia-nvtx-cu11==11.7.91
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pynvml==11.5.0
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.2
[pip3] transformers-stream-generator==0.0.4
[pip3] triton==3.0.0
[pip3] tritonclient==2.43.0
[conda] numpy                     1.24.4                   pypi_0    pypi
[conda] nvidia-cublas-cu11        <IP_ADDRESS>               pypi_0    pypi
[conda] nvidia-cublas-cu12        <IP_ADDRESS>                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu11    11.7.101                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu11    11.7.99                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu11  11.7.99                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu11         <IP_ADDRESS>                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         <IP_ADDRESS>                 pypi_0    pypi
[conda] nvidia-cufft-cu11         <IP_ADDRESS>                pypi_0    pypi
[conda] nvidia-cufft-cu12         <IP_ADDRESS>                pypi_0    pypi
[conda] nvidia-curand-cu11        <IP_ADDRESS>               pypi_0    pypi
[conda] nvidia-curand-cu12        <IP_ADDRESS>               pypi_0    pypi
[conda] nvidia-cusolver-cu11      <IP_ADDRESS>                 pypi_0    pypi
[conda] nvidia-cusolver-cu12      <IP_ADDRESS>               pypi_0    pypi
[conda] nvidia-cusparse-cu11      <IP_ADDRESS>                pypi_0    pypi
[conda] nvidia-cusparse-cu12      <IP_ADDRESS>               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu11          2.14.3                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.99                  pypi_0    pypi
[conda] nvidia-nvtx-cu11          11.7.91                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] pynvml                    11.5.0                   pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.44.2                   pypi_0    pypi
[conda] transformers-stream-generator 0.0.4                    pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
[conda] tritonclient              2.43.0                   pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.0@32e7db25365415841ebc7c4215851743fbb1bad1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity
GPU0     X      116-231 1
Legend:
X    = Self
SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX  = Connection traversing at most a single PCIe bridge
NV#  = Connection traversing a bonded set of # NVLinks
To Reproduce
Exact steps to reproduce the behavior:
Modify the MODEL_ID in examples/quantization_w8a8_int8/llama3_example.py to meta-llama/Llama-2-13b-chat-hf, run the compress on one 80GB A800 GPU.
Errors
The issue occurs specifically when compressing model.layers.40 using GPTQ, but SmoothQuant compression passes successfully.
 python compress.py
Loading checkpoint shards: 100%|██████████████████████| 10/10 [00:14<00:00,  1.48s/it]
2024-09-06T18:25:30.560482+0800 | main | WARNING - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: False
2024-09-06T18:25:30.561738+0800 | main | INFO - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
...
2024-09-06T18:25:30.607266+0800 | _check_create_state | INFO - State created for compression lifecycle
2024-09-06T18:25:30.609257+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-09-06T18:25:30.609495+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
Detected kernel version 5.4.119, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-09-06T18:25:30.661723+0800 | one_shot | INFO - *** One Shot ***
2024-09-06T18:25:30.666487+0800 | from_modifiers | INFO - Creating recipe from modifiers
2024-09-06T18:25:30.667419+0800 | create_instance | WARNING - Could not process input as a file path or zoo stub, attempting to process it as a string.
2024-09-06T18:25:30.786620+0800 | _check_compile_recipe | INFO - Recipe compiled and 1 modifiers created
2024-09-06T18:25:32.158512+0800 | _calibrate | INFO - Running SmoothQuantModifier calibration with 512 samples...
100%|█████████████████████████████| 512/512 [03:41<00:00,  2.31it/s]
2024-09-06T18:29:13.938457+0800 | _apply_smoothing | INFO - Smoothing activation scales...
2024-09-06T18:29:14.139377+0800 | on_initialize_structure | WARNING - GPTQ quantization is set to True without an active quantization modifier.
2024-09-06T18:29:14.139649+0800 | _build_quant_modifier | INFO - Building quantization modifier with args: {'targets': 'Linear', 'scheme': 'W8A8', 'ignore': ['lm_head']}
2024-09-06T18:29:14.176179+0800 | _check_calibration_data | INFO - Skipping QuantizationModifier calibration, it is not required for the provided quantization config.
2024-09-06T18:29:14.332806+0800 | initialize_compression | INFO - Preparing model.layers.0 for compression
2024-09-06T18:29:14.338628+0800 | initialize_compression | INFO - Preparing model.layers.1 for compression
...
2024-09-06T18:29:14.670908+0800 | initialize_compression | INFO - Preparing model.layers.39 for compression
2024-09-06T18:29:14.679910+0800 | **apply_compression | INFO - Running GPTQModifier calibration with 512 samples**...
  0%|                                                                                | 0/512
 File "/root/.cache/huggingface/modules/transformers_modules/XVERSE-13B-Chat-latest/modeling_xverse.py", line 309, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/mnt/llm_dataset/willhe/miniconda3/envs/vllm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/llm_dataset/willhe/miniconda3/envs/vllm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/XVERSE-13B-Chat-latest/modeling_xverse.py", line 248, in forward
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
  File "/mnt/llm_dataset/willhe/miniconda3/envs/vllm/lib/python3.8/site-packages/torch/nn/functional.py", line 1890, in softmax
    ret = input.softmax(dim, dtype=dtype)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 616.00 MiB. GPU 0 has a total capacity of 79.35 GiB of which 398.19 MiB is free. Process 21946 has 78.96 GiB memory in use. Of the allocated memory 78.36 GiB is allocated by PyTorch, and 97.93 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Additional context
Model: Meta-Llama/Llama-2-13b-chat-hf
Compression Tool: llm-compressor
Quantization Algorithms: GPTQ + SmoothQuant
Hardware: NVIDIA A800 GPU (80GB VRAM)
Quantization Scheme: W8A8 (8-bit weights and activations)
Questions:
Algorithm Clarification:
Why is it necessary to mix both GPTQ and SmoothQuant algorithms for quantizing weights and activations? Can SmoothQuant alone be used to quantize both weights and activations, or does GPTQ offer benefits that SmoothQuant alone cannot provide?
Memory Usage Issue:
Given that the model weight file is 26GB in FP16, why is 80GB of GPU memory insufficient to complete quantization, particularly when the OOM happens at layer 40? What factors during the compression process could cause memory usage to increase so significantly (e.g., intermediate activations, temporary data structures, etc.)?

GPTQ requires inverting the Hessian matrix, which has overhead relative to the size of the model. If you set sequential_update=True in your GPTQModifier, the quantization experiment will take a longer time, but there will be significantly reduced memory usage (since we will compute the hessian for one layer at a time, rather than all at once)

GPTQ requires inverting the Hessian matrix, which has overhead relative to the size of the model. If you set sequential_update=True in your GPTQModifier, the quantization experiment will take a longer time, but there will be significantly reduced memory usage (since we will compute the hessian for one layer at a time, rather than all at once)

Thank you for the detailed response. I have one more question regarding the algorithm clarification:
Could you explain why both GPTQ and SmoothQuant algorithms are used together for quantizing weights and activations? Is it possible to use SmoothQuant alone for both weights and activations, or does GPTQ provide specific benefits or advantages that SmoothQuant alone does not?
Given that our business model currently only utilizes SmoothQuant, we are uncertain whether incorporating GPTQ quantization will impact the model’s performance. As a more direct approach, would it be feasible to include only the SmoothQuantModifier in the recipe and exclude the GPTQModifier? If so, can this approach quantify whether the model’s performance is affected?


Support provided/compile only configurations

Resolves #62
These dependencies are not exported and are only used to compile the target similar to gradle behavior

Looks good!


Sync code with example-voting-app repo

The stale code has been removed to sync the tutorial with the [docker/example-voting-app repository] https://github.com/docker/example-voting-app

We're actually going to revert the Example Voting App so it we're going to keep this as is.
https://github.com/docker/example-voting-app/pull/40


getResource() in a nested for loop and props not updated with fetched data

Hi
Please I have this issue when fetching Layout Builder, I have the following code.
export default function LB(props){

    const LB: Array<any> = props.LBPage.layout_builder__layout;

    /*const pageStructure = {
        sectionsNum: LB.length,
        sectionsSetting: [],
        contentTypes: [],
    };

    // Add each section setting to the pageStructure

    LB.forEach(section => {
        const sectionSetting = {
            columns: null,
            column_widths: section.section.layout_settings.column_widths,
            context_mapping: section.section.layout_settings.context_mapping,
            label: section.section.layout_settings.label,
        };

        sectionSetting.columns = sectionSetting.column_widths ? sectionSetting.column_widths.split('-').length : null;

        pageStructure.sectionsSetting.push(sectionSetting);
    });*/

    console.log(LB, props.resource, props.resource[1], props);

    return (
        <Fragment>
            {
                LB.map(section => {
                    console.log(props.resource);
                    return <section className={
                                section.section.layout_settings.column_widths
                                    ?
                                section.section.layout_settings.column_widths.split('-').length + '-col' + ' ' + 'col-' +section.section.layout_settings.column_widths
                                    :
                                '1-col'
                                }
                            >
                            </section>;
                })
            }
        </Fragment>
    );
}

export async function getStaticProps(){

    let resource = [];
    let resources = [];

    const  LBPage = await getResource('node--page', '6acb560b-f999-4011-9bab-160856d21f9f')

    LBPage.layout_builder__layout.forEach((section, i) => {

        resource.push({section: []});

        //console.log('sections: ', section.section.components)

        // console.log('section.section.components: ', section.section.components, section.section);

        section.section.components.forEach(async (comp, j) => {
            //console.log('resource[i]: ', i, j);
            //console.log('component: ', comp.provider);
            //console.log('params: ', comp.configuration.provider + "--basic", comp.configuration.id, comp.configuration.id.replace(comp.configuration.provider + ':', ''));

            if(comp.configuration.provider === 'block_content'){
                const nodeType = comp.configuration.provider + "--basic";
                const nodeUid = comp.configuration.id.replace(comp.configuration.provider + ':', '');

                resources.push('response');
                const blocks = await getResource(nodeType, nodeUid);
                console.log(i, 'Response: ', resource[i].section, blocks);
                //resource[i].section.push(blocks);
                setTimeout(() => {
                    resource[i].section.push(blocks);
                }, 15000);
                resource[i].section.push('blocks');
                resources.push(blocks);
            }
        });

    });

    return {
        props: {
            LBPage,
            resource,
            resources,
        },
    };
}

LBPage fetched normally but resource has empty arrays here is it hen logged on the devtool
resource: [
    {
        "section": []
    },
    {
        "section": []
    },
    {
        "section": []
    },
    {
        "section": []
    }
]

every section there should has it's own blocks inside the array, by the way, the blocks fetched has been logged on the server nodejs but not passed to props
resources is just to know whether the block inside the if condition work or not
Thanks in advance.

Hello, each interation call in:
section.section.components.forEach(async (comp, j) => {...}
returns a Promise that is not resolved, try to wrap this loop with
with "await Promise.all(...)".

"async" returned from forEach returns an unresolved Promise, that receives data back after execution getStaticProps function body.
await Promise.all(section.section.components.forEach(async (comp, j) => {...});
Anyway do you need to call
const blocks = await getResource(nodeType, nodeUid);
inside loop? This is not a good practise, because you spin every request for each iteration. Can't you get those data before loop, and avoid await/async within loop?

You're probably using the layout builder patch.
@soft4net is right. You should wrap the top forEach in a Promise.all.
await Promise.all(LBPage.layout_builder__layout.forEach())

You could also use a for ... of ...
const  page = await getResource('node--page', '6acb560b-f999-4011-9bab-160856d21f9f')

for (const section of page.layout_builder__layout) {
  for (const component of section.components) {
    // .....

  }
}

Hi
@shadcn Yes I use that layout builder patch
Okay thanks a lot, I will test what you say and come back to you with the result
Thanks a lot for both of you.

Hi @shadcn
Thanks a lot I've got the block using for loop .
I've tried Promise.all it has to work but maybe I did a mistake, anyway.
Please I have a question why I'm not able to push blocks fetched using the first approach with .forEach method and i can push using the for loop?
Thanks a lot.


EFCore Nullable Owned entity type Obsolete documentation

In the documentation : https://docs.microsoft.com/en-us/ef/core/modeling/owned-entities#current-shortcomings

Since EFCore 3.0, it is possible.
https://github.com/dotnet/EntityFramework.Docs/issues/1646
Also, you should put an example with the nullable constraints activated (https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/nullable-reference-types). A good example/caveat is when you have an entity that all his property are null, the value return by EFCore is a null value, not an empty object so the type of the property (that use an owned entity type) should be nullable, but when all or one of the property are required then the type of the property should not be nullable.

Duplicate of https://github.com/dotnet/EntityFramework.Docs/issues/2320

@AndriySvyryd
Should i open a new issue or post in the referenced issue for this part

Also, you should put an example with the nullable constraints activated (https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/nullable-reference-types). A good example/caveat is when you have an entity that all his property are null, the value return by EFCore is a null value, not an empty object so the type of the property (that use an owned entity type) should be nullable, but when all or one of the property are required then the type of the property should not be nullable.

@Davilink You can comment on #2320


fix(android): add invalidate method for rn74+, onCatalystInstanceDestroy delegates

leaving the old method here so that old react-native versions will still work correctly, but it just delegates to the new method, which has the old implementation
should have no effect on anyone except for fixing compile issues
Supercedes / Closes #723

:tada: This PR is included in version 11.3.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


vagrant plugin post install conflict with rackspace provider

In the example below  the managed-servers plugin is not compatible with other provisioning providers.
not sure if this is just vagrant not allowing you to specify the provider using --provider with a provision request.
My only workaround so far is to uninstall the managed-servers plugin everytime I need to provision to rackspace.  Is there a better workaround available?
$ vagrant plugin install vagrant-managed-servers
Installing the 'vagrant-managed-servers' plugin. This can take a few minutes...
Installed the plugin 'vagrant-managed-servers (0.7.1)'!
$ vagrant plugin list
vagrant-cachier (1.2.1)
vagrant-host-shell (0.0.4)
vagrant-hosts (2.6.1)
vagrant-managed-servers (0.7.1)
vagrant-rackspace (0.1.10)
vagrant-share (1.1.4, system)
$ vagrant provision
No host IP was given to the Vagrant core NFS helper. This is
an internal error that should be reported as a bug
$ vagrant plugin uninstall vagrant-managed-servers
Uninstalling the 'vagrant-managed-servers' plugin...
$vagrant provision
==> pinas01: Rsyncing folder: /Users/PLACES/projects/platforms/rs_hosting/ => /vagrant
I have just tested this out using libvirt and dont see the same issues?  is this a bug with vagrant-managed-servers or rackspace. plugin?

Hmm, that's weird. Mind sharing your whole Vagrantfile?

not sharing the whole file as it has about 10 hosts in the all the same as this one.
config.vm.define :GUESTNAME do |GUESTNAME|
GUESTNAME.vm.hostname = "GUESTNAME..GUESTDOMAIN"
GUESTNAME.vm.provider :managed do |managed, override|
managed.server = "GUESTNAME..GUESTDOMAIN"
override.vm.box = "tknerr/managed-server-dummy"
override.ssh.username = "root"
override.ssh.private_key_path = "~/.ssh/PRIVATE_id_rsa"
end
GUESTNAME.vm.provision "shell", inline: "ip a | grep inet | grep eth"
GUESTNAME.vm.provision :shell do |shell|
	shell.inline = $script
end

GUESTNAME.vm.provision "puppet" do |puppet|
	puppet.hiera_config_path = "hiera.yaml"
	puppet.module_path = "modules"
	puppet.options = "--parser future"
end

end
Dont see the problem with libvirt and this plugin.  i was going to the say it was a RS plugin issue.

This is this vagrant 1.7.4, right?
And does it make a difference if you add a GUESTNAME.vm.provider :rackspace line inside the VM definition in addition to GUESTNAME.vm.provider :managed?
(i.e. so that the managed provider is not the only one that is "mentioned" for that VM)

Looks like the symptoms you are seeing (i.e. "No host IP was given to the Vagrant core NFS helper") happen with other plugins as well, eg. with vagrant-aws.
Maybe this helps as a workaround?
https://github.com/mitchellh/vagrant/issues/5401#issuecomment-115240904

Some more references to this issue from vagrant-rackspace and vagrant-google plugin here:
https://github.com/mitchellh/vagrant-rackspace/issues/113
https://github.com/mitchellh/vagrant-google/issues/94
Maybe we have to use the prepare_nfs_settings action in order to fix this, as suggested here:
https://github.com/mitchellh/vagrant/issues/4192#issuecomment-120047652

yes thanks I will give that a a try.
GUESTNAME,vm.provider :rackspace do |rs, override|
override.nfs.functional = false
...
Yes that has done the trick.   adding this to the Rackspace  Vagrantfile seems to have sorted it.  thanks


Use absolute paths for imports

Currently using relative paths, refactor them into absolute paths

Blocking this issue as not relevant as we are migrating to NextJS


belindas-closet-nextjs_6_372_data-cleaning-research

This is a pull request!
Please fill out this description with the following:
Resolves <issue #> 372
Description of what this pull request is for.
As a developer/researcher I completed my research on data cleaning.
I tried to put the codes on a black screen, but not all codes were possible.

Hi @el-riber , I saw that you did well on the research and the file you have been added


Improve performance of find_Node functions:

PERFORMANCE IMPROVEMENT
We have few functions to search for a particular node.
The most used is findNodeAtPosition which is used for every diagnostic to find the related node.
The current approach the function is implemented is to walk through a whole file from top to bottom (all AST tree nodes), but we can just technically skip all nodes that don't include the target node inside (getStart and getEnd).
The same happens with findTheFirstParentNodeInTheLine and findNodeEndsAtPosition.

fixed in #912


Using wantMessagesSigned with encrypted assertions causes "invalid response"

I have a onelogin application setup with the following configuration:
"security":

"nameIdEncrypted": false,

"authnRequestsSigned": true,

"logoutRequestSigned": true,

"logoutResponseSigned": true,

"signMetadata": true,

"wantMessagesSigned": true,

"wantAssertionsSigned": true,

"wantNameId": true,

"wantNameIdEncrypted": false,

"wantAssertionsEncrypted": false,

"allowSingleLabelDomains": false,

"wantAttributeStatement": false,

"signatureAlgorithm": "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256",

"digestAlgorithm": "http://www.w3.org/2001/04/xmlenc#sha256"

That works fine.
However, enabling encryption in my onelogin portal results in an error:
"Invalid_response The message of the response is not signed and the SP require it"
This doesn't make sense, as my application can clearly decrypt the request, because disabling "wantMessagesSigned" allows me to log in successfully with encryption enabled.
Has anyone had or solved similar issues? Is signing the message useful when we're encrypted using the x.509 anyways?

The flag 'wantAssertionsSigned' will verify that the Assertion inside the SAMLResponse is signed.
If you provide an encrypted Assertion, the toolkit will decrypt it and then confirm that the decrypted assertion is signed.
It seems OneLogin does not sign the assertion when you encrypt it, so you are getting this error.


Neither SpotCrime nor CrimeReports sensors work

Make sure you are running the latest version of Home Assistant before reporting an issue.
You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:
Home Assistant release (hass --version):
0.65.4
Python release (python3 --version):
3.5.3
Component/platform:
RaspberryPi 3 - Stretch
Description of problem:
Neither SpotCrime nor CrimeReports sensors work. Always show 0. I tried to increase the radius from 0.01 to 50000 and nothing changes. When I added sensors, I saw results for a few hours, but it switched to 0 and would not change.
Expected:
Problem-relevant configuration.yaml entries and steps to reproduce:
#CrimeReports
  - platform: crimereports
    name: 'Crime Reports'
    radius: 50000

#SpotCrime
  - platform: spotcrime
    name: 'Crime in the Area'
    radius: 50000
    days: 30

Traceback (if applicable):

Additional info:
I do not see any errors in the log.

I see the same thing here and when I search on the websites I see lots of results, maybe we are  not understanding this component properly?

It's possible that Crime Reports does not collect data in your area. Check their website to see if you can find more information: https://www.crimereports.com/
Spot Crime was recently updated to require an API key. I created a PR to update the docs, but it doesn't seem to have gone live yet. Not sure why. Here is some more information about the Spot Crime sensor: https://github.com/home-assistant/home-assistant/pull/12926
For more information on obtaining a Spot Crime API key, you can check the readme here: https://github.com/jcconnell/python-spotcrime

@jcconnell I get 0 crimereports in the sensor and taking a look online I see almost 500 incidents in my area, so something is not right here.
I will provide an update on spot crime after the merged code is released

My installation says REQUIREMENTS = ['spotcrime==1.0.2'].  I use latest 0.65.6 version of Home Assistant. How can I upgrade it?
Also, is the only way to get that API key by calling or emailing them? I do not like the idea of giving them my information. Should I say it is for development?

I'll ask about why it's not in the latest release yet. The PR has been accepted but maybe because it's a breaking change it has not been included.
You'll have to contact them for more info on access to the API. They had an API key that was publicly available and it was used in the initial python-spotcrime package but I was asked not to include it any longer and it's no longer valid.

A few days ago I emailed them and asked about API key. I did not receive any reply from them. Is there anything I can do about this? Should I email them again and again until they reply? Please explain how will it work with requesting API key.

I did not receive reply from them, but I found that someone commented on this page: https://community.home-assistant.io/t/0-66-vesync-switches-homematicip-and-a-group-cover/48776/11
This is what they said: "So I requested an API key from Spotcrime, and was told they only give out API keys to commercial contracts. Is that the only option now - pay for a license?"
What to do? Is there any way around it? Will we be able to use this component?

Still no response on my end too.

Anybody know why crime reports stopped working?
hass: 0.69.1
Python: 3.6
sensor.yaml
`# Crime Reports

platform: crimereports
name: Crime Alerts
radius: 3219`

This is not resolved. Crime reports was updated some time ago but Hass is not compatible with it. During and update, there's an error about this and if course crime reports does not work because of it.


[FRAGE] I'm Search the Funktion to Test is the Player Power On or offline

Hello,
I'm search a methode to check is a player Power ON bzw. connectet to the IP-Network
Gruß
Lothar

Hello.
Universal node provides the command household.test.player - use as a config file any player, which is "always on" and as payload the name of the player you would like to test - on/off.
.... but there are some limitations.
First of all, SONOS players are always "on" - there is no power switch off. If you remove power (with plug, or pull the line) it will take some time (maybe 5 minutes or more) until that is shown in the SONOS system and restarting also takes 1/2 to 1 minute.
So the above command works fine in the standard situation but not if you hard stop the power supply.
What would you like to achieve? Do you manually remove the player from power?

Du kannst diesen Flow probieren:
[{"id":"3cedf51f.6bc98a","type":"inject","z":"62cdb70e.b8cc78","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"<IP_ADDRESS>","payloadType":"str","x":140,"y":1797,"wires":[["7016e2fb.bf8a7c"]]},{"id":"8d947f7e.94d8e","type":"http request","z":"62cdb70e.b8cc78","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","authType":"","x":530,"y":1797,"wires":[["61f44fa1.b9466"]]},{"id":"61f44fa1.b9466","type":"switch","z":"62cdb70e.b8cc78","name":"","property":"headers.server","propertyType":"msg","rules":[{"t":"cont","v":"Linux UPnP/1.0 Sonos/","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":670,"y":1797,"wires":[["75075933.593828","9e3b8c2a.de538"]]},{"id":"75075933.593828","type":"change","z":"62cdb70e.b8cc78","name":"true","rules":[{"t":"set","p":"payload","pt":"msg","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":809,"y":1797,"wires":[["bf626131.b2648"]]},{"id":"52ddc88d.0acb28","type":"change","z":"62cdb70e.b8cc78","name":"false","rules":[{"t":"set","p":"payload","pt":"msg","to":"false","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":809,"y":1897,"wires":[["bf626131.b2648"]]},{"id":"bf626131.b2648","type":"debug","z":"62cdb70e.b8cc78","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":978,"y":1796,"wires":[]},{"id":"7de6f721.3469d8","type":"delay","z":"62cdb70e.b8cc78","name":"delay","pauseType":"delayv","timeout":"50","timeoutUnits":"milliseconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":666,"y":1897,"wires":[["52ddc88d.0acb28"]]},{"id":"9e3b8c2a.de538","type":"change","z":"62cdb70e.b8cc78","name":"reset","rules":[{"t":"set","p":"reset","pt":"msg","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":667,"y":1858,"wires":[["7de6f721.3469d8"]]},{"id":"7016e2fb.bf8a7c","type":"change","z":"62cdb70e.b8cc78","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"'http://' & $.payload & ':1400/info'","tot":"jsonata"},{"t":"set","p":"delay","pt":"msg","to":"10","tot":"num"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":1797,"wires":[["8d947f7e.94d8e","7de6f721.3469d8"]]},{"id":"d5547acb.0c58a8","type":"inject","z":"62cdb70e.b8cc78","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"<IP_ADDRESS>","payloadType":"str","x":140,"y":1857,"wires":[["7016e2fb.bf8a7c"]]},{"id":"1ebdcaff.449b95","type":"inject","z":"62cdb70e.b8cc78","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"<IP_ADDRESS>","payloadType":"str","x":140,"y":1917,"wires":[["7016e2fb.bf8a7c"]]},{"id":"90b9692.7e68098","type":"catch","z":"62cdb70e.b8cc78","name":"","scope":["8d947f7e.94d8e"],"uncaught":false,"x":510,"y":1757,"wires":[[]]},{"id":"42232be.d3a09d4","type":"comment","z":"62cdb70e.b8cc78","name":"adjust msg.delay","info":"The delay should be more then the time needed to process the http request plus reset delay.","x":340,"y":1760,"wires":[]}]

Werde es resten
--
Lothar Schweikle-Droll
25.09.2020 09:52:05 H. Klages<EMAIL_ADDRESS>
Du kannst diesen Flow probieren:
[{"id":"3cedf51f.6bc98a","type":"inject","z":"62cdb70e.b8cc78","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"<IP_ADDRESS>","payloadType":"str","x":140,"y":1797,"wires":[["7016e2fb.bf8a7c"]]},{"id":"8d947f7e.94d8e","type":"http request","z":"62cdb70e.b8cc78","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","authType":"","x":530,"y":1797,"wires":[["61f44fa1.b9466"]]},{"id":"61f44fa1.b9466","type":"switch","z":"62cdb70e.b8cc78","name":"","property":"headers.server","propertyType":"msg","rules":[{"t":"cont","v":"Linux UPnP/1.0 Sonos/","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":670,"y":1797,"wires":[["75075933.593828","9e3b8c2a.de538"]]},{"id":"75075933.593828","type":"change","z":"62cdb70e.b8cc78","name":"true","rules":[{"t":"set","p":"payload","pt":"msg","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":809,"y":1797,"wires":[["bf626131.b2648"]]},{"id":"52ddc88d.0acb28","type":"change","z":"62cdb70e.b8cc78","name":"false","rules":[{"t":"set","p":"payload","pt":"msg","to":"false","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":809,"y":1897,"wires":[["bf626131.b2648"]]},{"id":"bf626131.b2648","type":"debug","z":"62cdb70e.b8cc78","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":978,"y":1796,"wires":[]},{"id":"7de6f721.3469d8","type":"delay","z":"62cdb70e.b8cc78","name":"delay","pauseType":"delayv","timeout":"50","timeoutUnits":"milliseconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":666,"y":1897,"wires":[["52ddc88d.0acb28"]]},{"id":"9e3b8c2a.de538","type":"change","z":"62cdb70e.b8cc78","name":"reset","rules":[{"t":"set","p":"reset","pt":"msg","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":667,"y":1858,"wires":[["7de6f721.3469d8"]]},{"id":"7016e2fb.bf8a7c","type":"change","z":"62cdb70e.b8cc78","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"'http://' & $.payload & ':1400/info'","tot":"jsonata"},{"t":"set","p":"delay","pt":"msg","to":"10","tot":"num"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":1797,"wires":[["8d947f7e.94d8e","7de6f721.3469d8"]]},{"id":"d5547acb.0c58a8","type":"inject","z":"62cdb70e.b8cc78","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"<IP_ADDRESS>","payloadType":"str","x":140,"y":1857,"wires":[["7016e2fb.bf8a7c"]]},{"id":"1ebdcaff.449b95","type":"inject","z":"62cdb70e.b8cc78","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"<IP_ADDRESS>","payloadType":"str","x":140,"y":1917,"wires":[["7016e2fb.bf8a7c"]]},{"id":"90b9692.7e68098","type":"catch","z":"62cdb70e.b8cc78","name":"","scope":["8d947f7e.94d8e"],"uncaught":false,"x":510,"y":1757,"wires":[[]]},{"id":"42232be.d3a09d4","type":"comment","z":"62cdb70e.b8cc78","name":"adjust msg.delay","info":"The delay should be more then the time needed to process the http request plus reset delay.","x":340,"y":1760,"wires":[]}]
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub[https://github.com/hklages/node-red-contrib-sonos-plus/issues/97#issuecomment-698781182], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AHK66UAVMVRS3WAPVRMBGITSHRDZJANCNFSM4RWICINQ]. [https://github.com/notifications/beacon/AHK66UDMYMYNGNERZ7J653TSHRDZJA5CNFSM4RWICIN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFGTI37Q.gif]


Supporting symbol keys or auto-completion of string keys

Hi! I'm new using your package and have only read part of the code. I just wonder is it okay to support Symbol keys like replacing
TNewCoordinates[coordinatesID_String, coordinates_List?VectorQ] :=

by
keytype=(_String|_Symbol)
TNewCoordinates[coordinatesID:keytype, coordinates_List?VectorQ] :=

Will this replacement of patterns break codes in other place?
Or if supporting Symbol keys is hard, can String keys be auto-completed by
FE`Evaluate@FEPrivate`AddSpecialArgCompletion[_];

in this answer. Where are the right places to add this function?

Hi @yuriever, thanks for opening this issue. Sorry it took me so long to reply. I'm currently on hiatus from developing this package, since I'm too busy teaching. However, I plan to release a new version with many fixed and new features in the summer.
I have considered replacing the strings with symbol names, but I'm not sure this is a good idea, since it will make it easier to break encapsulation. However, I do plan on adding autocomplete for the object names. It won't be in the way that you suggest, since that is an undocumented and possibly broken feature, but I might use AddCodeCompletion or something similar.
Closing this issue since this is already planned to be implemented in the next release.