Unable to deploy Custom Container Images to an API with Amplify

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?
npm install @aws-amplify/cli
If applicable, what version of Node.js are you using?
16.7.0
Amplify CLI Version
5.3.1
What operating system are you using?
Mac
Amplify Categories
api
Amplify Commands
push
Describe the bug
Hello I have made countless attempts to deploy a custom docker image to an api with Amplify with no luck. At this point it is either an issue in Amplify, unsupported or I am missing something small. I keep trying to guess what it could be but my deployment keeps failing at the same spot. I've only been able to successfully deploy a container to the api when using the single express template or the express and python docker compose template on fully managed only.  Both of these templates fail on Self Managed with an error you can find below.
With my application I am taking a multi-repo approach instead of a monorepo approach. I have a .net 5.0 api in a separate CodeCommit repo connected and deploying directly to a private Amazon ECR with CodePipeline. Following this thread from a previously reported issue that Josef pointed out in the discord group (https://github.com/aws-amplify/amplify-cli/issues/7751#issuecomment-898809424). I setup my docker file to look this:
      FROM AccountNumber.dkr.ecr.us-west-2.amazonaws.com/stat-api:latest

      ENV PORT=80
      EXPOSE 80

And I setup a docker-compose file to look like this:
  version: "3.8"

  services:

      statapi:
          build:
              context: ./tally-stat-api
              dockerfile: Dockerfile
          container_name: statapi
          restart: unless-stopped
          ports:
              - "8080:80"

I can docker-compose up, I can docker build just the image locally completely fine and hit my api and receive real time results from a DB. But deployment always fails.
If I try to deploy with Fully Managed Selected I receive this:
CREATE_FAILED      DeploymentAwaiter AWS::CloudFormation::CustomResource Wed Aug 25 2021 03:28:06 GMT-0600 (Mountain Daylight Time) Received response status [FAILED] from custom resource. Message returned: The execution didn't succeed  Logs: /aws/lambda/amplify-statapp-dev-17582-AwaiterCustomCompleteHan-W5yVcSHXZoFQ      at Runtime.exports.handler (/var/task/index.js:102:13)     at processTicksAndRejections (internal/process/task_queues.js:97:5) (RequestId: c5cb4558-eedb-498e-a8a2-6c3052cd6db4)

Resource Name: pipelineawaiter-amplify-statapp-175824-containercdd4945e-service-statapi-8080 (AWS::CloudFormation::CustomResource)
Event Type: create
Reason: Received response status [FAILED] from custom resource. Message returned: The execution didn't succeed

Logs: /aws/lambda/amplify-statapp-dev-17582-AwaiterCustomCompleteHan-W5yVcSHXZoFQ

    at Runtime.exports.handler (/var/task/index.js:102:13)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) (RequestId: c5cb4558-eedb-498e-a8a2-6c3052cd6db4)

I can provide these logs if needed but there did not appear to be anything useful.
If I try to deploy with "Self Managed" (my preference) I get this different error:
CREATE_FAILED      ApiPipelinePipeline68596884                                             AWS::CodePipeline::Pipeline Wed Aug 25 2021 08:37:35 GMT-0600 (Mountain Daylight Time) 1 validation error detected: Value at 'pipeline.stages.1.member.actions.1.member.configuration' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 50000, Member must have length greater than or equal to 1] (Service: AWSCodePipeline; Status Code: 400; Error Code: ValidationException; Request ID: 9ed19d6d-1ca4-418e-be69-e896bd488cab; Proxy: null)

Resource Name: ApiPipelinePipeline68596884 (AWS::CodePipeline::Pipeline)
Event Type: create
Reason: 1 validation error detected: Value at 'pipeline.stages.1.member.actions.1.member.configuration' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 50000, Member must have length greater than or equal to 1] (Service: AWSCodePipeline; Status Code: 400; Error Code: ValidationException; Request ID: 9ed19d6d-1ca4-418e-be69-e896bd488cab; Proxy: null)

I have tried deploying really simple custom images from the public space. I also tried deploying the exact file from the issue reported link above. Same issues. And my custom image successfully deploys using Elastic Beanstalk. I love amplify and the power it brings and want to use it for my web app release. I either need a way for container deployments to work with custom images or I need to look at deploying and managing an API Gateway out side of amplify but using it in my Frontend (not preferred).
FYI here is what my custom image Dockerfile looks like. The final command was ENTRYPOINT but I switched to CMD to see if that could've been the cause since all example Dockerfiles that were deploying used CMD.
FROM mcr.microsoft.com/dotnet/aspnet:5.0.9 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/sdk:5.0.202 AS build
WORKDIR /src
COPY ["stat-api.csproj", "./"]
RUN dotnet restore "stat-api.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "stat-api.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "stat-api.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
CMD "dotnet" "stat-api.dll"

Expected behavior
I expect to be able to deploy custom docker images hosted on ECR to an Amplify managed API.
Reproduction steps

Create a custom docker image with a reachable exposed endpoint. (In my case a .net 5 api)
Deploy this custom image to Amazon ECR
In a separate project run amplify init
Turn container deployment on with AWS Configure project
Run amplify add api
Select Rest, Containers & Fully Managed or Self Managed
Paste in a dockerFile or dockercompose with Dockerfile all in the src (these should look like what I described in the bug description)
Run docker build or compose locally to test
After successful local test run - amplify push
Observer error

GraphQL schema(s)
Not Using GraphQL
Log output

# Put your logs below this line

Additional information
No response

I just run into this problem. There are couple of containers for PHP app: nginx and the app itself
App container is multi-stage build
FROM public.ecr.aws/docker/library/node:15.11.0-alpine3.13 as node
FROM public.ecr.aws/composer/composer:2.4.4-bin AS composer
FROM public.ecr.aws/docker/library/php:7.4-fpm-alpine
LABEL Description="Application container"

amplify push --profile ... produces this very helpful message
🛑 The following resources failed to deploy:
Resource Name: ApiPipelinePipeline68596884 (AWS::CodePipeline::Pipeline)
Event Type: create
Reason: 1 validation error detected: Value at 'pipeline.stages.1.member.actions.1.member.configuration' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 50000, Member must have length greater than or equal to 1] (Service: AWSCodePipeline; Status Code: 400; Error Code: ValidationException; Request ID: 20dab75a-9c13-4e2d-a58c-8f513fea3eb2; Proxy: null)

🛑 Resource is not in the state stackUpdateComplete
Name: ApiPipelinePipeline68596884 (AWS::CodePipeline::Pipeline), Event Type: create, Reason: 1 validation error detected: Value at 'pipeline.stages.1.member.actions.1.member.configuration' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 50000, Member must have length greater than or equal to 1] (Service: AWSCodePipeline; Status Code: 400; Error Code: ValidationException; Request ID: 20dab75a-9c13-4e2d-a58c-8f513fea3eb2; Proxy: null), IsCustomResource: false

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

Session Identifier: 86181475-8fde-44b2-9953-4a63b268769f

I don't even pretend to understand what it's trying to do. I see it's creating ECS resources, networking resource and... pipeline? So pipeline stack fails and it roll back everything. How do you even start troubleshooting this if cli designed to roll back all cloud formation on failure?
I have a question though, maybe somebody can help
Is it worth pursuing building ECS-based API with Amplify?
We're deploying this app with front-end using CDK and it's working perfectly fine: a CDK Pipelines, with triggered builds, and so on. I wanted to separated and switch the API to Amplify, hoping that amplify with abstract nicely all the infra. Plus easy way to manage auth.
Yet that's exactly where it is not working with this 2.5 thousand lines cloud formation template.


HTTP/2 support

Hi, I am looking at some greenfield development and would like to make use of HTTP/2 PUSH_PROMISE.  Any idea if/when http/2 will be supported in Kitura or do I need to make a mess and try to write it?  I am seeing support in WKWebView on the IOS client side lots of server-side but would like to see it somewhere via Swift on the server side as in Kitura. https://github.com/http2/http2-spec/wiki/Implementations

FYI, we have not evaluated this PR yet, but you may find it of interest: https://github.com/IBM-Swift/CCurl/pull/6

@buckstephenh Am I correct in that you are looking for HTTP/2 support on the server side of things? If so, the PR IBM-Swift/CCurl#6 is not relevant. That one is for the client side of things.

Yes server-side.  I think http/2 will be huge in terms of responsiveness and increased scalability of servers.  Please let me know how I can help.

On Aug 30, 2016, at 1:09 PM, Shmuel Kallner<EMAIL_ADDRESS>wrote:
@buckstephenh Am I correct in that you are looking for HTTP/2 support on the server side of things? If so, the PR IBM-Swift/CCurl#6 is not relevant. That one is for the client side of things.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I'd point out that in the interim Kitura supports FastCGI so you can use something like Nginx ahead of Kitura to handle HTTP2/SSL/static content etc.

How do you propose a server push_promise via fastcgi?  I didn’t realize fastcgi was two-way.  Is it?  What normally is the slave of inbound requests can initiate outbound calls via fastcgi to the web server?

On Sep 4, 2016, at 1:30 PM, Dylan Neild<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
I'd point out that in the interim Kitura supports FastCGI so you can use something like Nginx ahead of Kitura to handle HTTP2/SSL/static content etc.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/IBM-Swift/Kitura-net/issues/86#issuecomment-244616292, or mute the thread https://github.com/notifications/unsubscribe-auth/ATg8EYdVlTmIfhVCPmZZupH9H8UA5mkNks5qmwBEgaJpZM4Jv75c.

Issue moved to IBM-Swift/Kitura #895 via ZenHub


Use maven publish plugin for central publishing

Re-publish of #256
This simplifies setup using the gradle maven publish plugin for publishing to central. This removes jcenter publishing, but publishing just to central covers jcenter since it proxies it anyway.
https://github.com/vanniktech/gradle-maven-publish-plugin
This requires added local gradle properties (recommended to put them in your ~/.gradle/gradle.properties) for signing and auth, described here: https://github.com/vanniktech/gradle-maven-publish-plugin#where-to-upload-to
Most significantly, this automatically handles publishing the gradle plugin marker artifact as well so that it can be used directly in plugins {...}
This also makes snapshots development (./gradlew uploadArchives when the version ends in -SNAPSHOT) and local development ./gradlew install) easier
Resolves #267

Hi Zac, I tried this out after resolving the conflicts against the next branch. Installing the plugin locally does seem to work and I can see javadoc is attached to the jars, but I'm running into trouble when running the uploadArchives or bintrayUpload task, namely:

./gradlew uploadArchives                  `
Configuration(s) specified but the install task does not exist in project :.
Publication Publication not found in project :.

This seems to result in only the bugsnag-android-gradle-plugin-5.X.X.jar being uploaded to Sonatype, and similar behaviour for Bintray. I'm assuming that there's some difference in the vanniktech plugin that it doesn't pick up the rest of these files by default. I think this would be the only obstacle to merging this PR.

Let me take a look and test this out locally

@fractalwrench can you test again? I think the install task issue was due to a previous configuration being leftover that I removed in f6e7a85.
I see all the requisite files when I install locally

I don't really understand these errors from the bintray plugin
Configuration(s) specified but the install task does not exist in project :.
Publication BintrayPublication not found in project :.

The bintray plugin is... not good. We use jfrog internally in combination with this plugin for publishing, could you try the latest change with ./gradlew uploadArchivesBintray? Specify the user and key as you did before

This seems to work nicely when installing locally, after setting the sourceCompatibility to JDK 8 explicitly. It also appears to upload the correct artefacts to Sonatype although I've not tried resolving the dependency from the staging repository as of yet. The bintray publish task seems to have advanced onto a new error message:

Execution failed for task ':publishPluginMavenPublicationToBintrayRepository'.
Failed to publish publication 'pluginMaven' to repository 'bintray'
Could not PUT 'https://api.bintray.com/maven/fractalwrench/artifact-sandbox/;publish=1/com/bugsnag/bugsnag-android-gradle-plugin/5.3.0/bugsnag-android-gradle-plugin-5.3.0.jar'. Received status code 404 from server: Not Found

I tried altering the URL to use bugsnag rather than $bintray_user which advanced things a little further, but I got caught up on a similar error message:

Execution failed for task ':publishPluginMavenPublicationToBintrayRepository'.
Failed to publish publication 'pluginMaven' to repository 'bintray'
Could not PUT 'https://api.bintray.com/maven/bugsnag/artifact-sandbox/;publish=1/com/bugsnag/bugsnag-android-gradle-plugin/5.3.0/bugsnag-android-gradle-plugin-5.3.0.jar'. Received status code 404 from server: Not Found

bintray_user and bintray_api_key are both present in ~/.gradle/gradle.properties, so I'm not too sure what's going on here.
Also I noticed that publish=1 - I've altered this to publish=0 as my understanding is that the first would publish the package automatically.

Hey @ZacSweers,
Just to let you know this has now been released in V5.7.7.
Thanks again for the PR.


Updated variable handling in Arcweave.Project

Modification of SaveVariables and LoadVariables methods in Arcweave.Project class for changing the format of saving and loading variables

It's will be fixed in next version?


feat(obstacle_cruise_planner): add new rss distance

Description
Change rss distance

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

This change did not improve adaptive cruise planner, so I close this PR.


CVE-ID for recent XSS issues ?

I was wondering if a CVE ID had been assigned to the XSS issues fixed in #495 / 1.7.0.
If not, it might be a good idea to request one, and reference it in the various discussion threads.

I've now requested one though the DWF since they appear to deal specifically with CVEs for open-source.
You can view the submission here: https://pending-requests-v5.distributedweaknessfiling.org/

Cheers !

I just checked the status and CVE-2018-1000162 has been assigned by DWF, but MITRE have not yet accepted it (CVEProject/cvelist#411)

A-ha, I received an email last week with the CVE and have been checking the MITRE site for when it goes though. I hadn't seen that PR though! The batch seems to be held up by another CVE, so I'd expect that our CVE passes through okay. I was planning on holding off referencing the assigned CVE in #495 until it's accepted by MITRE

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1000162


https://github.com/users/sntgopl/projects/2

Note: There are two students in this group. Student A and Student B.

Required Changes ♻ ⚠ 🚦
Hi @sntgopl,
Good job so far!
You still need to work on some issues to go to the next project, but you are almost there!

Aspects to improve ♻

[ ] Kindly provide the link to your GitHub project in the issue's description since this is a project requirement and we need to evaluate it. 👍

Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think the code review was unfair, you can request a second opinion using this form.

https://github.com/users/sntgopl/projects/2

Hi Team,👋🏻👋🏻
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉🔥🔥💯💯

Highlights ✨✨
📌 Descriptive pull request✅
📌 Great working on the set up✅
📌 Good commit messages ✅
Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


elm-reactor 0.15.1 Build from source fails.

Get the following for elm-reactor (all other items build ok)
Resolving dependencies...
In order, the following would be installed:
SHA-<IP_ADDRESS> (new package)
bytestring-mmap-0.2.2 (new package)
entropy-0.3.7 (new package)
enumerator-0.4.20 (new package)
attoparsec-enumerator-0.3.4 (new package)
blaze-builder-enumerator-<IP_ADDRESS> -newbuilder (new package)
hinotify-0.3.7 (new package)
monads-tf-<IP_ADDRESS> (reinstall) changes: transformers-<IP_ADDRESS> -> <IP_ADDRESS>
MonadCatchIO-transformers-<IP_ADDRESS> (new version)
regex-base-0.93.2 (reinstall) changes: mtl-2.1.2 -> 2.2.1
regex-posix-0.95.2 (reinstall)
system-filepath-<IP_ADDRESS> (new package)
system-fileio-<IP_ADDRESS> (new package)
fsnotify-<IP_ADDRESS> (new package)
unix-compat-<IP_ADDRESS> (new package)
websockets-<IP_ADDRESS> (new package)
zlib-bindings-<IP_ADDRESS> (new package)
zlib-enum-<IP_ADDRESS> (new package)
snap-core-<IP_ADDRESS> (new package)
snap-server-<IP_ADDRESS> (new package)
websockets-snap-<IP_ADDRESS> (new package)
elm-reactor-0.3.2 (new package)
cabal: The following packages are likely to be broken by the reinstalls:
MonadCatchIO-transformers-<IP_ADDRESS>
MonadCatchIO-mtl-<IP_ADDRESS>
cgi-3<IP_ADDRESS>
regex-compat-0.95.1
Use --force-reinstalls if you want to install anyway

Yes, I see this too on certain platforms. What OS are you using?
I have not been able to figure out the root cause though, I suspect we need to add constraints to something. Can you try to figure out which projects are requiring MonadCachIO?

Ubuntu 15.04
I'll take a look and see what I can work out,

Looks to be the versions of transformers and mtl installed on ubuntu
The following enables a successful build:
sudo ghc-pkg --force unregister transformers
sudo ghc-pkg --force unregister mtl
But I'm not sure what harm that does to the underlying ghc install and ubuntu packages.

It looks like it's an issue with the haskell-platform package.  Un-installing it and installing just the ghc package fixes the problem, with the added advantage of having no broken packages.
Let me know if I can do anything more

Hmm, it may make sense to add the fubar.sh script to the build from source process. I find that the best way to get haskell running is to get ghc and cabal and then trash everything else.

I not sure that it should be included in the process by default.  My reservation is that while fixing the elm build it could trash other Haskell builds.  Or am I worrying about nothing.
So maybe as an option to run if there are broken packages.

Oh, definitely! If you have other stuff done in Haskell it could be bad. Good point!


Hello, splash filter can not work on this case?

I want block a url.
like this: http://t12.baidu.com/it/u=1853176499,2178064445&fm=76
I put following rule in easylist.txt, but now work.
||t12.baidu.com^
how to solved it? thank you:)

@genius001 how have you solved it?

I'm sorry, I forgot explain it.
splash filters work fine.
I just restart splash.sever --filters-path=/path/to/filters/
Does must reload it to take effect?

Thanks for explanation!
Yes, large filters (e.g. easylist) load very slowly - it take seconds, so they are loaded only once, at startup.


[storage-queue] Modernized strings

Description
Resolves https://github.com/Azure/azure-sdk-for-python/issues/22057
Hello, @jalauzon-msft! I removed unicode strings and replaced format methods with f-strings, may you please review them? Thank you. I hope you have a wonderful day!
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

@wonhyeongseo, not sure if you have permission to see ADO pipeline results but there is one PyLint error caught by ci. Posting here as an FYI and hopefully you can address as part of feedback. Thanks!


Spring Boot 3, Sanitizer compatibililty

Please see https://github.com/esanchezros/quickfixj-spring-boot-starter-examples/issues/25

@david-gibbs-ig I just release 2.17.1, which should fix the issue you described in the examples repo. Please let me know if it works for you.
Thanks

Thank you. I can see it working in the examples project, I still have problem with my implementation. Most likely error my side, investigating.

@david-gibbs-ig I just release 2.17.1, which should fix the issue you described in the examples repo. Please let me know if it works for you.
Thanks

Quickfixj Actuator endpoint is working for me, thank you.

Fixed in 2.17.1


Define a Common Command Line Interface to run any container

Instead of dealing with specific command line interfaces (CLIs), it would be relevant to define a standard CLI to run benchmark containers.
Such a CLI should include:

Introspection commands to list the benchmarks, workloads and other parameters (e.g., threads) available and their value.
Execution commands to run the benchmark with a specific workload and set of parameters.

This CLI should be exposed by all the available containers.

Hello,
Must i copy the  run of NPB, and adapt it to all done benchmarks?
Or make a new run ?
Currently the run of NPB contains :
  *a generic help.
  *The help of each benchmark.
  *And if the first param is not -help', so we suppsed that the user want to run the benchmark.

So should we add a "-bench", a  "-run" , or just continue with the same idea as NPB?
You are the final users, so you can choose or propose your favorite way. =)

You can take inspiration from NPB, but it is expected to have a new script with the same syntax (parameters, options) for all the containers of this project that will delegate to the specific script provided by the benchmark suite (whenever it exists and parameters/options can be mapped).


evdev / libinput: use multiple devices

The current evdev and libinput drivers store state in static variables. That means they can only be used with one device at a time but I cannot use, say, two trackpads or a trackpad and a keyboard at the same time (or at least my weak C-fu doesn't let me see how).
It would be great if the driver state could instead be stored in an external structure, to be managed by the caller. This way I could connect an arbitrary number of input devices via either of these drivers.
In order to not break the interface of the existing evdev / libinput drivers maybe a new driver with support for multiple devices could be created and the existing drivers could be changed to use the new one under the hood.

Actually we wouldn't even need the state to be managed externally. We could instead store it in the driver's user_data field. I made a hacky proof-of-concept here.

I agree to support various type of indevs (mouse+touch+keypad) but I can't imagine a use case for e.g. 2 keyboard, or 2 mouses.
If we target only 1 indev per type we can simply hardcode state_mouse, state_touch, etc.
What do you think?

Ah, that's an interesting thought. I suppose that would get me half of the way at least.
There are two cases I can think of where being able to connect multiple devices of the same type might be helpful:
1.A device with built-in keyboard (for instance, a Pinebook) that is also docked to an external USB keyboard
2. Any device docked to an external trackpad and a keyboard with mouse emulation. For example, I have an Ergodox EZ and I occasionally like to control the mouse with it but most of the time I use a trackpad. The keyboard's mouse emulation shows up as a separate device though.
/dev/input/event8:      Apple Inc. Magic Trackpad 2
/dev/input/event9:      ZSA Ergodox EZ
/dev/input/event10:     ZSA Ergodox EZ Mouse
/dev/input/event11:     ZSA Ergodox EZ System Control
/dev/input/event12:     ZSA Ergodox EZ Consumer Control
/dev/input/event13:     ZSA Ergodox EZ Keyboard

Both of these scenarios are probably not overly common though so if you think it'd add too much complexity to allow for multiple devices of the same type, just permitting one device per type might be enough. On the other hand, the step from that solution to allowing an arbitrary number of devices might also not be too big anymore. :thinking:

I see, these are really possible scenarios. The problem is that now lvgl's libinput driver has an init and read_cb API. It means the user needs to create the indev driver assign the  read_cb. If we need to use the user_data to  store e.g. the name of the dev in the driver instance we need to pass it to the user too.
It'd be more flexible if an lv_libput_init() created as many indev driver's as required. E.g. 2 mouse, 1 keyboard, 1 touchpad.
But such change would be API breaking. Therefore (for now) I vote for supporting one indev per type as it fits into current API schema.
We have plans to rework the driver and maybe marge into the lvgl repo. We need think about this case when designing a new unified driver API.

If we need to use the user_data to store e.g. the name of the dev in the driver instance we need to pass it to the user too.

Yeah, that's true. In my current hacky workaround I had to introduce void libinput_multi_init_driver(lv_indev_drv_t * indev_drv) to be called instead of void libinput_init(void) for initializing the driver state and bool libinput_multi_set_file(lv_indev_drv_t * indev_drv, char* dev_name) to be called instead of bool libinput_set_file(char* dev_name) to store the device file into the driver state.
One thing I wonder is if we could maybe encapsulate all the currently global state into a struct and add a default struct variable that is used whenever the user calls the current libinput_init / libinput_set_file functions. That way we could keep the current public API that supports only one device at a time but also offer the two additional functions to handle multiple standalone drivers. :thinking:
Let me maybe play around with this a little and I may send a draft PR to better demonstrate what I mean.
Otherwise, supporting one indev per type would be ok-ish for my use case as well in the short term.
PS: Thanks a ton for considering all my requests! :slightly_smiling_face:

PS: Thanks a ton for considering all my requests!

I also thank you for sending these PRs and issues. This is what moves LVGL forward! :metal:
As you suggest we can have an other init function that creates all the required indevs. Maybe it's not the cleanest solution  but very useful.


Change $LISTEN_PORT variable to an integer

Reverting the deployment template LISTEN_PORT variable back to an
integer rather than a string.
Signed-off-by: Chris Collins<EMAIL_ADDRESS>

/lgtm


Extensions Documentation: AttributePanel "You can add or overwrite"

Sorry for my ignorance but I'm having a very difficult time understanding how to "add" as it states in the readme for extensions package.
Example: Would like to add a dimension switch to the section block for being able to add "full-width" attribute.
The documentation example is a bit misleading as it shows you how to replace the entire panel with text, rather than adding to it.
If there is something I could reference that points me in the right direction of how to achieve this, that would be great and in return I do PR to add more detailed documentation to the read.me file in the extensions package.
Thank you for the lovely app, I plan on sticking around and will help out as much as I can.

I think it is very complicated to add Field alone in certain places, and different people have different needs.
It is much simpler to completely overwrite and rewrite. You only need to copy the original Panel code and change it.

Thanks for the response. Perhaps you could help me identify how to overwrite the panel code and in return I will add the documentation. I took a look around and see there have been a number of other users who have also asked about full width attribute being added to section block. I'm not a power react user so it's easy for me to figure this out without a little push in the right direction. I think in order to grow the community we will want to work on adding documentation for attributes panels with examples that are baked into MJML framework, such as the full width attribute.
I do not mind creating PR and  documenting my process for adding this to the panel.
Thanks for the response, Ryan.

https://github.com/zalify/easy-email/blob/master/packages/easy-email-extensions/src/AttributePanel/components/blocks/Section/index.tsx
<TextField name={`${focusIdx}.attributes.full-width`} />

Fill  full-width .
The easiest way is to copy this file overlay and add a TextField to control the full-width value.
Of course, the best way should be to use SwitchField to control Boolean values


Backend: Suche in kundenspezifische Preise funktioniert nicht / Customer-specific pricing does not working

Description:
We have several customer-specific pricing für most customers. In the backend we see that customer prices and the product name.
In backend > Customer > Customer-specific pricing
there are a search bar. This search bar does not working. Neither sku nor productname was found.
Environment:
Steps to reproduce:
Expected result:
Actual result:

[public] Automated response: This issue is linked to the internal issue https://shopware.atlassian.net/browse/NEXT-31991.
[created from NEXT-31991, comment 485454]

[public] Hello,
Within our company roadmap and work capacity, we try to address each bug or improvement request but admit that not each one will be resolved. To continue our culture of honesty and openness, we are closing this issue to focus on our roadmap on behalf of all Shopware users.
If you feel like this issue is still impacting you, please create a new issue and let us know. Thank you again for your valuable feedback.
[created from NEXT-31991, comment 492283]


[SPARK-34261][SQL] Avoid side effect if create exists temporary function

What changes were proposed in this pull request?

Add function exists check before load resource.
Why are the changes needed?

We should not add jar into classpath if the create temporary function is already exists.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Add test.

Test build #134543 has started for PR 31358 at commit da3e7b3.

Test build #134543 has started for PR 31358 at commit da3e7b3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39129/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39129/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39129/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39129/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39129/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39129/

Test build #134543 has finished for PR 31358 at commit da3e7b3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134543 has finished for PR 31358 at commit da3e7b3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134543/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134543/

Test build #134555 has started for PR 31358 at commit 590b0b3.

Test build #134555 has started for PR 31358 at commit 590b0b3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39141/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39141/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39141/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39141/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39141/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39141/

Test build #134555 has finished for PR 31358 at commit 590b0b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134555 has finished for PR 31358 at commit 590b0b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134555/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134555/

thanks @srowen , also cc @maropu @cloud-fan

thanks @srowen , also cc @maropu @cloud-fan

Thanks! Merged to master.

Thanks! Merged to master.

thanks for merging !

thanks for merging !


proposal: runtime: an http.Handler that implements 'go tool trace' in process

For applications that already include a web server and generate event traces (e.g. gopls), it is convenient for the developer to be able to collect and view a trace of events trace from within the application itself. Unfortunately, the event trace viewer is currently only available through the 'go tool trace' command, which opens its own web server, complicating the integration and management of processes and lifetimes.
This proposal requests a package, maintained by the runtime team, that provides an http.Handler that can be added to the application's ServeMux to allow the develop to view event traces within the existing server.
Related:

https://github.com/golang/go/issues/63185

@mknyszek

I like this idea. At it's core, it is about moving most of cmd/trace to an importable package to make it usable by other servers.
IMO, this would make the most sense as part of x/debug/trace (#62627). Perhaps x/debug/trace/http?
The current trace viewer needs multiple endpoints. /trace serves the HTML/JS UI and then the JS fetches the actual trace content from /jsontrace. I'm not sure the best way to avoid exposing that via the API. Can the package provide an http.ServeMux with all of the endpoints? I'm not sure if there is an easy way to use a ServeMux as "subdirectory" within another ServeMux?


Internal blacklist

Is your feature request related to a problem? Please describe.
We have a brawland blacklist on us, but it isn't interactive. So we want an interactive blacklist that won't allow blacklisted people to enter our servers and will check whether they are in our clubs.
Describe the solution you'd like
Make a list of blacklisted tags in the database. Add a check to all three welcoming processes on whether the tag is blacklisted, disallow server entry if the tag is. Add a command to search through the saved clubs and check whether there's a user with a blacklisted tag in one of the clubs.
Describe alternatives you've considered
Maybe we should make a command timed. Most likely no, but it's up to further discussion.

Let's just forget about it.


If tornado backend returns 599 status, we should raise an exception

This comment makes it look like an exception is raised: https://github.com/pusher/pusher-http-python/issues/118#issue-360409787


check_locations function: insufficient road layer dimensions to reach chosen nearest town.

Currently the road layer is downloaded with the standard buffer dimensions applied early on in PEMprepr
It would be better to download a new road layer for the Samplr portion that uses the AOI plus the chosen towns to create a proper sized bounding box for road layer download.
Also here

Road layers are extracted at two extends. Modelling extent and expanded landscape extent.  A landscape scale AOI is used on line 247 of the vignette. Does this cover this issue?

moved to here: https://github.com/ninoxconsulting/PEMsamplr/issues/12


Build fails with one custom package source

Symptom
A solution that has a nuget.config with one custom key under <packagesources> will fail to deploy. Upon inspecting the Deployment Details > Activity Logs the user will discover output like the following:
Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling .NET Web Application deployment.
Unable to find version '5.6.4' of package 'Microsoft.Data.Edm'.
Failed exitCode=1, command=nuget restore "D:\home\site\repository\UrlShortening.sln"
Unable to find version '5.6.4' of package 'Microsoft.Data.OData'.
An error has occurred during web site deployment.
Unable to find version '5.6.4' of package 'Microsoft.Data.Services.Client'.
Unable to find version '3.1.0' of package 'Microsoft.WindowsAzure.ConfigurationManager'.
Unable to find version '7.0.1' of package 'Newtonsoft.Json'.
...

The build passes locally in Visual Studio (probably because the NuGet package source is added to the environment by default) as well as on AppVeyor.
Repro: https://github.com/Durwella/UrlShortening/tree/0f85db19f033995f325b14cf9f6fb758961c39f9
Root Cause
The package(s) located at the custom package source will be found. However packages located at the "default" NuGet package source will not be found.
Mitigation
Add the NuGet package source to the nuget.config
<packageSources>
    <add key="NuGet official package source" value="https://nuget.org/api/v2/" />
    <add key="My custom source" value="http://example.packagesources" />
</packageSources>

Suggestion
It's not obvious that this should be "fixed". But, given that the build works in other places it is confusing.  Either one of the following two options would be helpful in this case:

After a build fails, if it fails due to missing packages and their exists a nuget.config with <packageSources> and the nuget.org value is missing, then print a warning message to the log such as

Your build failed due to missing packages. If you use standard packages then consider adding the following to the <packageSources> section of your nuget.config: <add key="NuGet official package source" value="https://nuget.org/api/v2/" />

Include a nuget.config in a parent directory that includes the standard NuGet package source. It seems that it will be chained together with the solution-level nuget.config.
It could be that some users actually want to exclude the nuget.org package source. They could still achieve this using  <clear /> as the first line inside of <packageSources>. I'd expect this to be the less-than-20% case.

Thanks!

@jfoshee thanks for reporting. As you suggest, the reason it works on VS machines is that VS installs a %APPDATA%\nuget\NuGet.Config which has the default package source. And it might be that AppVeyor build machines also have VS. But if you run the same nuget.exe command on some random machine, I think you'd see the same error. In that sense, I'm leaning toward thinking it is not a bug, but is the NuGet.exe design.
As for improving the error reporting, I think it's something that should live in NuGet.exe. All we do in the Azure build is run NuGet.exe, and we have zero knowledge of nuget.config. As such, it would feel dirty having to look for it (I think it can be anywhere in the parent folder chain), parsing and and making error handling decision based on what it contains.

I'm having this problem right now... The mitigation does not work however. I have to remove the custom package source... let the microsoft packages install and let the build fail then add back my custom package location and rebuild and it will then use the cached versions it downloaded already of microsoft packages and then find mine...  something is definitely not right with this.

https://stackoverflow.com/questions/46531036/nuget-package-restore-fails-for-webapp-with-custom-nuget-config

@willthiswork89 having you reproed this from straight Kudu console, to isolate from any other part of Kudu deployment? Is that with nuget restore, or dotnet restore?

Hi David I am not positive it's whatever runs for a webapp and .netcore . I am not super familiar with kudu but can try this tonight

@willthiswork89 yes, please do isolate down to a minimal repro, per wiki.

The problem occurs inside the kudu console as well with a dotnet restore

I'm not sure what's going on. In the end, this is using the standard 'dotnet restore', so it should work the same on Azure as anywhere else. Maybe it is not finding your customer feed. Are you using nuget.config to specify it? And are you sure that the same works on a machine that doesn't the feed registered machine level?

So The workaround i have found is to remove my nuget config, do a dotnet restore. It will pull all microsoft packages into cache then fail to find my custom packages. From there i reinstall my nuget.config with my custom source, and run the dot net restore again.... this will then use the microsoft packages from cache and then pull the custom packages down just fine from my custom source and the build will succeed.
It seems to be something with multiple package locations causing it. The nuget.config i used to do this was copied from my development machines %appdata%\nuget\nuget.config

And yes the custom package URL works fine on a dev machine

`

<add key="NuGet official package source" value="http://nuget.org/api/v2/" />

<add key="OTL" value="http://www.myget.org/F/**{Redacted}/**api/v3/index.json" />

<add key="enabled" value="True" />

<add key="automatic" value="True" />

<add key="skip" value="False" />

<add key="format" value="0" />

<add key="disabled" value="False" />

`

I have tried http, https, nuget.org/v3 of the api tons of things... The only way I get it to work is to remove the config, let kudu get the packages from nuget, then fail to find custom packages. Then add my nuget.config back with the custom package location and kick the deployment off again

If you can share a small repro repo of this, I'll investigate what might be going on. You can set up a dummy myget feed if you don't want to show your real one.

Great ill get one put together tonight after work for you. Appreciate any help and insights.

Hey @davidebbo finally got this replicated sorry it took so long. I trimmed out as much as possible to still have a functioning build. Its public at https://github.com/willthiswork89/testnugetfailure

So to replicate, just attempt to deploy that app into a WebApp on Azure and kudu will kick off and fail because of a strange error about failing to connect to pull down microsoft nuget packages.
Once that happens, delete the nuget config and run it again. It will fail again but this time it will succeed with getting nuget references for microsoft, then fail getting them for the custom packages.
Once that happens, add the nuget config BACK untouched. Now it will restore the microsoft packages from the local location and then use the custom location and pull down my custom packages fine and the build will succeed and deploy.

@willthiswork89 perfect, I'll take a look, thanks!

Ok, I can indeed repro this, and I'm going to need help from NuGet experts to figure out what's causing it.
@emgarten maybe you can help?
Situation is:

We run within the App Service sandbox
We set DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true
nuget.config points to both nuget.org and another source (see it here)
We run dotnet restore within the App Service sandbox

And for some reason, it fails to restore nuget package, with error above. But if nuget.config just has nuget.org, then it works fine.
Why would the second feed break restore?

I'm glad you were able to repro. Whats more interesting is the fact that if you take JUST nugget config, restore then swap it for the custom, they both work individually but not together

I haven't seen an additional feed cause problems like that. The error message around missing packages has been improved in NuGet 4.3.0/dotnet CLI 2.0.2, is there any chance you could restore using that?
In the latest version it will tell you which feeds it tried, and what package versions were found.

@emgarten are you sure that has been released? The latest I see on https://www.microsoft.com/net/download/core#/sdk is 2.0. How would one get 2.0.2?

Ok, so I figured out the root of the mystery. The App Service sandbox limits the socket connections to 300, and it seems that having two feeds causes NuGet to blow that limit, while having just one stays under the limit. That explains the puzzling behavior.
Now we need to figure out how to make NuGet less socket greedy...

One more thing...
@willthiswork89 what App Service Plan were you using? I believe the 300 limit only exists in Free/Shared. So in theory this should not happen in Basic/Standard.

Aha! This makes sense. We are indeed running on the free version. So no work around for now then other than upgrading to basic tier then downgrading.

Right, unless NuGet has some secret mechanism for reducing concurrency.

Oh, @emgarten just told me that you can run dotnet restore --disable-parallel which limits things to a single http connection. So give that a try! You'll need a custom deployment script to tweak how this is called.
Though it'll probably run a low slower.
Closing here since there is really nothing to be done in Kudu itself. See also https://github.com/NuGet/Home/issues/4538.

You could also try setting the property <RestoreDisableParallel>true</RestoreDisableParallel> in all csproj files (or including a props file that does this to workaround this if you aren't calling dotnet restore yourself.
The command line sets this property in MSBuild and is picked up restore, setting it in your projects would have a similar effect.

@emgarten nice! That'll avoid the need for a custom deployment script. On the downside, that'll make the restore slow on your local machine.


[Bug Report] el-checkbox-group not working with Vuex

Element UI version
1.3.3
OS/Browsers version
Windows / Chrome 57.0.2987.133
Vue version
2.3.3
Reproduction Link
https://jsfiddle.net/5u6gb339/
Steps to reproduce
Use :value and @change on checkbox-group instead of v-model.
See jsfiddle.
You can't use v-model with Vuex, you need to use :value and @change, so in conclusion the el-checkbox-group does not work with Vuex.
What is Expected?
The @change should be the new values, even if :value is already set.
@change should trump :value like it does on the el-input for example.
I have provided in the link to reproduce, how the el-input works in contrast.
What is actually happening?
The el-checkbox-group.
:value overwrites @input
Thank you!

U should use the 'input' event insteadof 'change'


Create ReadRows() examples showing how to use multiple row keys.

Thanks to Bastien Duclaux for suggesting this.  We want an example that show how RowSet can have multiple keys, e.g.:
namespace cbt = google::cloud::bigtable;
auto rows = cbt::RowSet();
rows.Append("abc");
rows.Append("xyz");
...
for (auto const& row : table->ReadRows(std::move(row_set) , cbt::Filter::Latest(1))) {
   auto const& cell = row.cells().front();
 // ...
}

We should also consider an example that shows how to use multiple ranges

@coryan, I think that the example should also include RowRange created from a Prefix

Created #1926 to track that.  Two separate examples are probably easier to read.

Here is my approach. I have created a new file bigtable_rowset_examples.cc similar to bigtable_hello_world.cc.
create a table
add some rows
create RowSet
append a few keys
read those rows with those keys  from the table
delete table
Any thoughts?

It would be nice if the example showed that you can read discontinuous rows keys, e.g., if row1, row2, and row3 are in the table show that you can read row1 and row3 without reading row2.

This is exactly the plan.
Put in keys from 10-20 and read only 10, 14 and 18
On Fri, Feb 1, 2019 at 4:43 PM coryan<EMAIL_ADDRESS>wrote:

It would be nice if the example showed that you can read discontinuous
rows keys, e.g., if row1, row2, and row3 are in the table show that you
can read row1 and row3 without reading row2.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-cpp/issues/1564#issuecomment-459878062,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ArwWzd9Z5UJL62GN5S0OVfnglbH-Nyw1ks5vJLUagaJpZM4Y8K7K
.

--
Robert P. Simion


feat(bayesian_optimization.py): Allow registering of temporary observations

Potential fix for #347

Codecov Report
Base: 99.25% // Head: 95.36% // Decreases project coverage by -3.88% :warning:

Coverage data is based on head (94cd1a2) compared to base (8e5c89f).
Patch coverage: 18.51% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #365      +/-   ##
==========================================
- Coverage   99.25%   95.36%   -3.89%
==========================================
  Files           8        8
  Lines         535      561      +26
  Branches       81       85       +4
==========================================
+ Hits          531      535       +4
- Misses          2       23      +21
- Partials        2        3       +1

Impacted Files
Coverage Δ

bayes_opt/target_space.py
91.72% <14.28%> (-8.28%)
:arrow_down:

bayes_opt/bayesian_optimization.py
91.45% <23.07%> (-8.55%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Hey @Rizhiy, thanks for contributing!
I (superficially) follwed the discussion in the original issue and I agree with the notions that a) this package should be compatible with parallel optimization and b) in it's current state, that can't really be considered the state (as you have shown). Nonetheless, I'm always somewhat sceptical of messing with internal data and even moreso here, since it involves providing "wrong" values. Is there some way that this could go wrong? Off the top of my head:

What happens if there are duplicate points (i.e. points with the same parameters), would it be possible that the "real" point is updated instead of the dummy?
What happens if the optimization process terminates early (maybe some stopping criterion is reached), will dummies persist?

If I understand correctly, the idea behind this technique of registering dummies is to reduce the covariance of the GPR at a point that is being sampled so that other workers are suggested other points. Is there a way to perform this covariance reduction without registering dummies? Alternatively, is there a way to force different points to be returned without using dummies, e.g.

...by changing the pbounds?
...messing with x_seeds in acq_max so that only seeds are used which are a certain distance away from the last suggestion?
...penalizing points in the objective to_minimize of acq_max that are close to points other workers are considering? (this is probably what I would try)

In particular, I think that modifying acq_max seems to be a more stable/more elegant way of achieving what you're going for: To my understanding, what you're trying to accomplish is for the optimizer to suggest points that are away from points currently investigated by other workers. The acq_max function is the place where suggestions are made, so I would also expect it to be the appropriate place for your modifications.

Hi @till-m, regarding your concerns:

What happens if there are duplicate points (i.e. points with the same parameters), would it be possible that the "real" point is updated instead of the dummy?

Not sure what you mean, dummy points are stored in _dummies variable:
https://github.com/Rizhiy/BayesianOptimization/blob/94cd1a29ab12e334f1f60528a03cea1642f82786/bayes_opt/bayesian_optimization.py#L135
Interactions for other variables should be the same, can you perhaps try it out and I can correct the wrong behaviour.

What happens if the optimization process terminates early (maybe some stopping criterion is reached), will dummies persist?

They persist until you overwrite them.
I would guess to make it more user-friendly, the relevant functions (e.g. max & res) should include the filter for dummies.

Is there a way to perform this covariance reduction without registering dummies?

There very well might be, but I haven't dived deep into how the optimiser works on the inside yet.
If you know a better way, please let me know and I will update this Pull Request.
For my approach I only need the functionality to let the optimiser know that a point is being evaluated.

In particular, I think that modifying acq_max seems to be a more stable/more elegant way of achieving what you're going for

I will look into that.

Hi @Rizhiy,
I misread the code and after thinking about it again, I think you've covered the scenario that I was picturing. My apologies.

@till-m I have looked into other ways you suggested achieving similar effect and currently it seems that they won't be as effective or too difficult to implement.
Basically, it is very difficult to properly define which area exactly we don't want to sample.
There very well might be a better approach for parallel evaluation, but I haven't been able to find any yet.

Looking more closely at the code, perhaps it would make more sense to put dummy logic into TargetSpace class, since it has kind of similar logic with constraints.

Hi @Rizhiy,

I would guess to make it more user-friendly, the relevant functions (e.g. max & res) should include the filter for dummies.

I agree. I think logging should also be considered -- dummies probably shouldn't show up in logs, especially when the JSONLogger is used, so that loading from logs doesn't register the dummies normally.

Looking more closely at the code, perhaps it would make more sense to put dummy logic into TargetSpace class, since it has kind of similar logic with constraints.

Agreed.

I have looked into other ways you suggested achieving similar effect and currently it seems that they won't be as effective or too difficult to implement.

Thanks for looking into this. I also had a look around the literature and I found a few potentially interesting things. Your approach seems to be related to "Constant Liars", see Ginsbourger et al; 2008. Consider also the "Kriging Believer" which may be a more natural way to pick the values of the dummies.
I would also like to highlight the approach outlined in Batch Bayesian Optimization via Local Penalization (see also here), that works by penalizing the acquisition function and would be close to what I was thinking of. Finally, Kandasamy et al might also be worth considering, since it seems to work very well while not being too hard to implement, at least at first glance.
There's a lot more literature on this problem than I thought there would be, two others that I found potentially interesting would be:

Wang et al; Parallel Bayesian Global Optimization of Expensive Functions, 2016
Snoek et al; Practical Bayesian Optimization of Machine Learning Algorithms, 2012

Hi @Rizhiy, I decided to write a quick implementation of Kandasamy to see it work. The preliminary results are pretty disappointing, will post later.

Okay, so first of all it looks like sampling from the multivariate normal is very computationally expensive. I had to adapt n_warmup in acq_max to make this feasible, so I'm now using 1e4 instead of 1e5. I've adapted your code from the issue.
Here are the results:
Output:
Checking num_workers: 100%|██████████████████████| 4/4 [12:11<00:00, 182.88s/it]
num_workers=1, mean_min_distance=0.406, best_result=-0.171
num_workers=2, mean_min_distance=0.468, best_result=-0.276
num_workers=4, mean_min_distance=0.359, best_result=-0.707
num_workers=8, mean_min_distance=0.419, best_result=-0.112

Plot:

I'm somewhat disappointed with these results. It doesn't really seem to hone in on the max very well. Additionally, sampling from the multivariate normal also seems to be quite computationally intensive -- I takes around 4x as long to run the script with async_thompson=True. Admittedly my optimization loop is very naive, but I currently don't see a way of speeding this up.
Output (async_thompson=False)
Checking num_workers: 100%|███████████████████████| 4/4 [03:18<00:00, 49.56s/it]
num_workers=1, mean_min_distance=0.634, best_result=-3.440
num_workers=2, mean_min_distance=0.320, best_result=-14.352
num_workers=4, mean_min_distance=0.192, best_result=-3.855
num_workers=8, mean_min_distance=0.209, best_result=-1.974

from __future__ import annotations

from typing import Callable

import matplotlib.pyplot as plt
import numpy as np
from bayes_opt.bayesian_optimization import BayesianOptimization
from bayes_opt.util import UtilityFunction
from pandas.core.generic import pickle
from scipy.optimize import rosen
from scipy.stats import multivariate_normal
from tqdm import tqdm
from sklearn.gaussian_process import GaussianProcessRegressor
import warnings

class asyTS():
    def __init__(self) -> None:
        self.random_state_generator = lambda: np.random.randint(0, 100000)

    @property # I know that this is giga-hacky monkey patching, please bear with me.
    def utility(self):
        random_state = self.random_state_generator()
        def func(x, gp: GaussianProcessRegressor, y_max):
            with warnings.catch_warnings():
                warnings.simplefilter("ignore")
                # fixing random state causes it to return the same function
                return gp.sample_y(x, random_state=random_state)
        return func

def _closest_distance(point, points):
    return min(np.linalg.norm(point - p) for p in points if p is not point)

def optimize(
    func: Callable[..., float], num_iter: int, bounds: dict[str, tuple[float, float]], num_workers=0, async_thompson=True
):
    init_samples = int(np.sqrt(num_iter))

    optimizer = BayesianOptimization(f=None, pbounds=bounds, verbose=0)
    init_kappa = 10
    kappa_decay = (0.1 / init_kappa) ** (1 / (num_iter - init_samples))
    if async_thompson:
        utility = asyTS()
    else:
        utility = UtilityFunction(
            kind="ucb", kappa=init_kappa, xi=0.0, kappa_decay=kappa_decay, kappa_decay_delay=init_samples
        )

    init_queue = [optimizer.suggest(utility) for _ in range(init_samples)]
    result_queue = []
    tbar = tqdm(total=num_iter, leave=False)
    while len(optimizer.res) < num_iter:
        sample = init_queue.pop(0) if init_queue else optimizer.suggest(utility)

        loss = func(list(sample.values())) * -1
        result_queue.append((sample, loss))
        if len(result_queue) >= num_workers:
            try:
                optimizer.register(*result_queue.pop(0))
                tbar.update()
            except KeyError:
                pass
    return optimizer.res

bounds = {"x": [-5, 5], "y": [-5, 5]}

all_results = {}
for num_workers in tqdm([1, 2, 4, 8], desc="Checking num_workers"):
    results = []
    results = optimize(rosen, 150, bounds, num_workers, async_thompson=True)
    all_results[num_workers] = results

with open("results.pkl", "wb") as f:
    pickle.dump(all_results, f)
with open("results.pkl", "rb") as f:
    all_results = pickle.load(f)

fig, axs = plt.subplots(2, 2)
fig.set_figheight(8)
fig.set_figwidth(8)
axs = [item for sublist in axs for item in sublist]
for idx, (num_workers, results) in enumerate(all_results.items()):
    if num_workers > 8:
        continue
    samples = [np.array(list(res["params"].values())) for res in results]
    axs[idx].scatter(*zip(*samples), s=1)
    axs[idx].set_title(f"{num_workers=}")
    avg_min_distance = np.mean([_closest_distance(sample, samples) for sample in samples])
    best_result = max([res["target"] for res in results])
    print(f"{num_workers=}, mean_min_distance={avg_min_distance:.3f}, {best_result=:.3f}")
fig.tight_layout()
fig.savefig("results.png")

Is there something similar to kappa decay in your code?
For the algorithm to hone in on the maximum better you need to progressively reduce kappa to 0.
In my code there is utility.update_params() which is called after every register.

My apologies for the delay.

Is there something similar to kappa decay in your code?

In theory this shouldn't be necessary, but I could add it. I will try to get around to doing that. Overall, I'm not sure if the TS approach is worth exploring due to the sampling speed.

Hi @Rizhiy,
apologies for letting this sit for a while. I've been thinking and we should definitely have functionality for this, but it would be better to first redesign the acquisition function API. I've implemented some ideas and will push them shortly, including an implementation of the Kriging believer algorithm. Would you be interested in reviewing and/or testing that code?

Hi, I can go over it briefly, but TBH I haven't been looking in this area since last year, since my approach works fine for me.

Closing as this is handled in #447.
@Rizhiy I adapted your benchmarking code as an example incl. attribution. I hope that's okay.


Implements certificate credential authentication using JWT assertions

Since Microsoft is no longer issuing client secrets with indefinite validity for O365 application registrations, I have decided to implement certificate credential authentication using JWT assertions. At this time, there seem to be no restrictions on the validity of self-signed certificates (I managed to upload and use a 10yr certificate to Entra just fine).
Microsoft even recommends certificate authentication:

For best security, we recommend using certificate credentials.

The workflow and technicals for certificate authentication are described here.
This PR adds two account configuration options jwt_certificate_path and jwt_key_path. These options are to be used in place of client_secret (i.e., that value should be removed). jwt_certificate_path is expected to be a path to a x509 pem-encoded certificate; jwt_key_path is expected to be a path to an unencrypted pem-encoded private key.
To generate the certificate and private key, the following command can be used:
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out certificate.pem -sha256 -days 3650 -nodes -subj "/CN=email-oauth2-proxy"

The certificate.pem that is generated can then be uploaded to the application registration in the Entra admin console.

Thanks for the contribution!
In order to test before merging this I'll need access to an account that can be authenticated via JWT, which I don't have at the moment. Is this something you could provide for this purpose?

I could provide you with a temporary test account. How can I contact you privately?

That would be perfect. You can reach me via my GitHub username at Gmail.

I have sent you an email.

Thanks for providing the test account – I've just taken a look. I made a few minor edits:

Remove the edit to pyproject.toml as this is not needed (but clarify the message here for future reference)
Lower the minimum pyjwt version to 2.4 so that Python 3.6 is still supported
Variable renaming (including in the configuration file) for consistency with the rest of the proxy
Lint reformatting to match the project style
Remove some of the JWT parameters that RFC 7519 specifies as optional and O365 does not require
Add basic documentation

Just so there is a record, here's a sample configuration file entry using this method:
<EMAIL_ADDRESS>permission_url = https://login.microsoftonline.com/*** your tenant id here ***/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/*** your tenant id here ***/oauth2/v2.0/token
oauth2_scope = https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access
redirect_uri = http://localhost
client_id = *** your client id here ***
jwt_certificate_path = /path/to/certificate.pem
jwt_key_path = /path/to/key.pem

Unless there are any other features to be added, I think this is probably ready to be merged. Let me know what you think?

Thanks for taking your time to polish this - it's looking really good now.
Re:

Remove some of the JWT parameters that RFC 7519 specifies as optional and O365 does not require

They might not be required by the spec, but at least for jti Microsoft's documentation is worded a little more strict:

The identifier value must be assigned in a manner that ensures that there's a negligible probability that the same value will be accidentally assigned to a different data object [...]

I also found this page which states:

The claims expected by Microsoft Entra ID in the signed assertion are: [...]

and

If you want to provide your own claims, including the mandatory claims expected by Microsoft Entra ID, pass in false for the mergeWithDefaultClaims parameter.

Interesting – thanks for following up. The original page isn't completely clear – it essentially just lists the properties defined in the RFC. Because of this, I determined through experimentation which ones O365 does and does not actually require for OAuth 2.0 login. I hadn't seen the other link you've found, but in its Crafting the assertion section it gives this example:
// no need to add exp, nbf as JsonWebTokenHandler will add them by default.
var claims = new Dictionary<string, object>()
{
    { "aud", tokenEndpoint },
    { "iss", clientId },
    { "jti", Guid.NewGuid().ToString() },
    { "sub", clientId }
};

As explained in the comment, the exp property is added automatically by JsonWebTokenHandler in this example. The proxy currently sets a 5-minute expiry time here. Interestingly, Microsoft's default expiry time is 60 minutes, though their documentation suggests using a value of 5-10 minutes at most. It's safe to stick to the proxy's current approach for this property, I think.
From testing, I found that the nbf property is not actually required by O365, though I agree that we should probably add it back since it seems like it may be expected, and the default that O365 sets this value to (the Unix epoch) does not seem sensible. The iat property is not mentioned on this page, and not currently added in JsonWebTokenHandler either (though that hasn't always been the case).
So: long story short, I think nbf and jti should be added, but we can leave iat out. What do you think?

So: long story short, I think nbf and jti should be added, but we can leave iat out. What do you think?

Sounds reasonable to me.


GPIO

Added GPIO feature

@ArthurSonzogni There's a minor issue in the UI.
In the GPIO menu, whenever I select the lower GPIO menu it just takes a lot of time before I can select from the page.
Example:


Fail to compile on Win7 SP1 with VS 2010

What steps will reproduce the problem?
1. Download cudpp 2.0
2. Standard cmake config procedure
3. Build solution

2>C:\Program Files\Microsoft SDKs\Windows\v7.0A\include\objbase.h(239): error :
identifier "IUnknown" is undefined

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 5 Apr 2012 at 5:56

Strange that none of us have ever seen this.  I develop on Win7.

Original comment by<EMAIL_ADDRESS>on 10 Sep 2012 at 10:09

There seems to be a simple temporary fix:
On the top of the file cuda_util.h I added the lines
#ifdef _WIN32
#define WIN32_LEAN_AND_MEAN
#endif
This resolved the problem for me now. However it should be addressed in a clean
way.
A related post is http://forums.nvidia.com/index.php?showtopic=67822

Original comment by<EMAIL_ADDRESS>on 5 Apr 2012 at 7:55


Paper segment: Speeding up MTEB (English) + co2 impact

This segment is intended to aggregate the performance gains over our changes in a reasonable format.
Related to: #836, #835
@imenelydiaker could imagine that you might be interested in this?

This segment is intended to aggregate the performance gains over our changes in a reasonable format.
Related to: #836, #835
@imenelydiaker could imagine that you might be interested in this?

Yes I'm interested in this, happy to collaborate with any person who's interested!

Perfect @imenelydiaker I will assign you to feel free to recruit collaborators. I am more than @mrshu might already have done some of the work needed for this task.

Hey @mrshu! Did you finish running the English benchmark?

@imenelydiaker unfortunately, it's still running. With sentence-t5-xxl some of the datsets take days to evaluate (e.g. FEVER or MSMARCO). I am running this at a single H100 and didn't really look at running it in parallel but I am afraid I am running out of ideas otherwise -- the model is pretty big and encoding a single batch of 391 samples takes roughly 55 minutes.

Might be worth running it with a smaller model?

Yeah should definetly use a smaller models, I'd go for one of the baseline models we used: multilingual-e5-small to be consistent with the rest of evaluations and the paper in general.
@KennethEnevoldsen @mrshu wdyt?

Yea - a larger model like e5-large would produce a larger effect so I would probably go for that, but a 7b or larger model might be too frustrating to run.

ok then let's go for multilingual-e5-large.
For the fast version of MTEB, I wonder if it's not worth waiting for this to complete #836 ? All Clustering tasks have been converted to Fast for the moment, but it would be better to have a faster version of retrieval also.
@mrshu would you be able to run the experiments again on the English benchmark with multilingual-e5-large? We should start with the old English benchmark only, and we'll wait a little bit to start running the fast version (just wait few weeks to see if #836 completes or ignore retrieval and focus on clustering + classification downsampling only).

@imenelydiaker I am on it :)

@mrshu you will have to make sure that you use the correct implementation of e5 - I implement e5 large in #876.

@KennethEnevoldsen ugh, is there a chance just running the code in https://github.com/embeddings-benchmark/mteb/commit/4e1bab4ef964291aadacf808465d54e93d3db4cc will save wrong data?

The e5 model will not produce the same results as the model expect a slightly formatted input so it should perform slightly better with the PR (especially on retrieval)

Thanks @KennethEnevoldsen.
The processing has failed with the following anyhow:
INFO:mteb.evaluation.evaluators.RetrievalEvaluator:For evaluation, we ignore identical query and document ids (default), please explicitly set ``ignore_identical_ids=False`` to ignore this.
ERROR:mteb.evaluation.MTEB:Error while evaluating ArguAna: Expected object_relevance_per_qid dictionary and measures set.
Traceback (most recent call last):
  File "/home/mrshu/mteb/scripts/run_mteb_english.py", line 115, in <module>
    evaluation.run(model, output_folder=f"carbon_results/{model_name}", eval_splits=eval_splits, co2_tracker=True)
  File "/home/mrshu/mteb/mteb/evaluation/MTEB.py", line 319, in run
    raise e
  File "/home/mrshu/mteb/mteb/evaluation/MTEB.py", line 297, in run
    results, tick, tock = self._run_eval(task, model, split, output_folder, **kwargs)
  File "/home/mrshu/mteb/mteb/evaluation/MTEB.py", line 216, in _run_eval
    results = task.evaluate(model, split, output_folder=output_folder, **kwargs)
  File "/home/mrshu/mteb/mteb/abstasks/AbsTaskRetrieval.py", line 165, in evaluate
    scores = self._evaluate_monolingual(retriever, corpus, queries, relevant_docs, None, **kwargs)
  File "/home/mrshu/mteb/mteb/abstasks/AbsTaskRetrieval.py", line 191, in _evaluate_monolingual
    ndcg, _map, recall, precision = retriever.evaluate(relevant_docs, results, retriever.k_values, ignore_identical_ids=kwargs.get("ignore_identical_ids", True))
  File "/home/mrshu/mteb/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 207, in evaluate
    evaluator = pytrec_eval.RelevanceEvaluator(qrels, {map_string, ndcg_string, recall_string, precision_string})
  File "/home/mrshu/mteb/.v/lib/python3.10/site-packages/pytrec_eval/__init__.py", line 59, in __init__
    super().__init__(query_relevance=query_relevance, measures=measures, relevance_level=relevance_level, judged_docs_only_flag=judged_docs_only_flag)
TypeError: Expected object_relevance_per_qid dictionary and measures set.

Would it make sense to merge your changes in and re-run the whole pipeline again?

@imenelydiaker are you familiar with this error? (probably related to #833)

@imenelydiaker are you familiar with this error? (probably related to #833)

Nope haven't seen this error before. #833 has not been merged yet so it's not related imo, it may be related to MIRACL Reranking #830 and Abstention #854, these are the latest PRs that updated the Retrieval evaluator

Hmm couldn't reproduce the error on the latest branch:
from sentence_transformers import SentenceTransformer

import mteb

tasks = mteb.get_tasks(tasks=["ArguAna"])

eval = mteb.MTEB(tasks=tasks)

model = SentenceTransformer("all-MiniLM-L6-v2")

eval.run(model, output_folder="temp")  # type: ignore

@KennethEnevoldsen that's the tough part -- this is running on a very old branch intentionally (the code in https://github.com/embeddings-benchmark/mteb/commit/4e1bab4ef964291aadacf808465d54e93d3db4cc builds off of the 1.2.0 release), to show the differences between the time when the benchmark was optimized to run faster and at the same time use less CO2.
This didn't cause an issue for me previously with other models, so I wonder whether it might be multilingual-e5-large that might cause trouble here?
In any case, any suggestions would be greatly appreciated!

Ahh right! sorry yea. You should still implement the encode_corpus, encode_query for the e5 (or potentially we can choose a model that does not require a custom implementation - something within bert-base to large which is somewhat standard). There might be a bug in the specific version can you run using a small model on the arguana just to check?

@Muennighoff you might be more aware of this bug than me

Also havn't seen that bug before - maybe an issue with the pytrec_eval version you have installed? Maybe make sure to install whatever was in the requirements.txt file at that point in time, not what's there now? 🤔

Thanks @Muennighoff and @KennethEnevoldsen. Unfortunately, pytrec_eval hasn't been updated in the past 3 years (https://pypi.org/project/pytrec-eval/) so it doesn't seem like it will be that.
I'll try the smaller model just to check.

The sentence-t5-xxl processing has finally finished -- the result can be seen in https://github.com/embeddings-benchmark/mteb/compare/main...mrshu:mteb:mrshu/port-carbon-emissions-estimation

Closing this for now since the segment is completed.


Terms of Access Language

(TOA will be reviewed with IQSS/Berkman/etc)
Data passed from Dataverse to the OpenDP App:
Exact Data being passed

See queryParams in the manifest

What it Means

User tokens allow access to private data including:

Raw private data (tabular file)
DDI XML with summary statistics

In addition, tokens will allow access to the current Dataverse user's data:

Dataverse user's: persistent identifier, name, and email
The email address may be used to send messages from the OpenDP App, but only after user consent

Likely, the user will have the option to the change which email address receives notice

(!)Additional systems

The User token with access to the private data will also be sent to an Execution Engine, a system running in Azure cloud services

At the end of the full (successful) process, DP Release files be deposited in the user's Dataverse Dataset.

3/26:Last round of edits sent to the cyberlaw clinic


window.getConfiguration() returns values that are not defined/visible in any setting files

I had some users report a while back that the arguments passed into debug configs were being duplicated that I never got to the bottom of. Today I'm seeing what I suspect it the same issue, but I can't explain it. I'm using workspace.getConfiguration() with a URI to get resource-specific configuration for a folder, but it's returning settings (which inspect says come from globalValue) but the setting is not defined in my VS Code user settings (nor in the folder itself) - I've included a screenshot of the user settings - although I've checked the JSON too:
I don't know how to reproduce, but here's some code showing the issue (run as the first thing in my activate method):

I've checked all the settings files I can find, and I can't find this set anywhere. I suspect at some point in the past I read this array out of settings and then mutated it and that it has persisted somewhere, but it feels like a bug that I could get into this state (and it's not clear how to get out of it).

As is always the way - it occurred to me exactly what this was immediately after posting. I run my tests with their own --user-data-dir and guess what I found in that data dirs settings.json? 🙈


Upgrade OpenFOAM to v2206

Still on Ubuntu 20.04.
After the build succeeds, we still need to replace the swak4Foam compilation step with pre-built binaries, to save space and time.

Mainly for future reference:
this job succeeded, but with swak4Foam again falsely continuing as if it was successful, with the following error:
2022-09-19T08:46:42.8514130Z     default: wmake funkyPythonPostproc
2022-09-19T08:46:43.0270540Z     default: Making dependency list for source file funkyPythonPostproc.C
2022-09-19T08:46:43.0921460Z     default: wmkdepend: could not open 'pythonInterpreterWrapper.H' for source file 'funkyPythonPostproc.C': No such file or directory
2022-09-19T08:46:43.1253340Z     default: g++ -std=c++14 -m64 -pthread -DOPENFOAM=2206 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-attributes -Wno-unknown-pragmas -O3  -DNoRepository -ftemplate-depth-100  -IMake/linux64GccDPInt32Opt  	  -I/usr/lib/openfoam/openfoam2206/src/finiteVolume/lnInclude -I/usr/lib/openfoam/openfoam2206/src/sampling/lnInclude -I/usr/lib/openfoam/openfoam2206/src/meshTools/lnInclude -Wno-old-style-cast -Wno-deprecated-register -I../../Libraries/languageIntegration/swakPythonIntegration/lnInclude/ -I../../Libraries/languageIntegration/swakGeneralLanguageIntegration/lnInclude/ -I../../Libraries/simpleFunctionObjects/lnInclude/ -I../../Libraries/swak4FoamParsers/lnInclude/ -iquote. -IlnInclude -I/usr/lib/openfoam/openfoam2206/src/OpenFOAM/lnInclude -I/usr/lib/openfoam/openfoam2206/src/OSspecific/POSIX/lnInclude   -fPIC -c funkyPythonPostproc.C -o Make/linux64GccDPInt32Opt/funkyPythonPostproc.o
2022-09-19T08:46:43.4673450Z     default: funkyPythonPostproc.C:41:10: fatal error: pythonInterpreterWrapper.H: No such file or directory
2022-09-19T08:46:43.4678570Z     default:    41 | #include "pythonInterpreterWrapper.H"
2022-09-19T08:46:43.4682970Z     default:       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-09-19T08:46:43.4683480Z     default: compilation terminated.
2022-09-19T08:46:43.4704170Z     default: make: *** [/usr/lib/openfoam/openfoam2206/wmake/rules/General/transform:35: Make/linux64GccDPInt32Opt/funkyPythonPostproc.o] Error 1

Here are the full logs (GitHub Actions auto-cleans after some time): logs_25.zip
I have not yet checked if all the files we need are still compiled.

@fsimonis Can I get a code review? I have already tested that the changes work. The most cumbersome part was producing the new binaries for swak4Foam and then switching to them.


Spendable Balance on Account Page is Total

On the account page, the listed total balance is the same as the spendable balance. However, I have most of balance locked in tickets at the moment.
These are tickets for the wallet itself, not via a stakepool, so it might not have been tested that way in Paymetheus. Still seems odd that it's reporting the total rather than spendable.

Wanted to bang this out for the 1.0 release but it's a bit more complex of a change that i expected.  Paymetheus needs a way to determine the type of a transaction, and while that info is now available over RPC, it will be a big refactor to plumb that info through all the types.  It is also not as easy to implement in Paymetheus itself as the coinbase check currently done.


ERROR: _reactNative.NetInfo.getConnectionInfo is not a function

Description:
https://github.com/geektimecoil/react-native-onesignal/blob/master/index.js#L59
This line is causing this error because NetInfo is not available in the current RN version.
Can you add conditions to this extra functionality?
If I comment this code the issue is fixed: https://github.com/geektimecoil/react-native-onesignal/blob/master/index.js#L47-L67
Environment
"react": "16.0.0-alpha.12",
"react-native": "0.45.1",
"react-native-onesignal": "github:geektimecoil/react-native-onesignal#master",
Steps to Reproduce Issue:
Instal RN 0.45.1 version and "react-native-onesignal": "github:geektimecoil/react-native-onesignal#master" and the app will display that error.

@facuescobar  React-Native 0.45.1 was released over a year ago. Does this issue occur if you update to a more recent version of react-native?

@facuescobar I have done some more investigating, and it appears like usage of the NetInfo API is no longer needed in our SDK, so I went ahead and removed it. Please let me know if you encounter any other issues.

@Nightsd01 a RN upgrade wasn't an option for us because it required a lot of things right now.
Thanks for this update!


Replace GzDecoder with MultiGzDecoder in io.rs

A gzip member consists of a header, compressed data and a trailer. The gzip specification, however, allows multiple gzip members to be joined in a single stream. MultiGzDecoder will decode all consecutive members while GzDecoder will only decompress the first gzip member.
ref: https://github.com/wdecoster/chopper/commit/a4409fcda38ebffa32a87451da8139079c856b92#r142191434

Hey Tao, this is fantastic. Thanks for your contribution. If I understand you correctly, you want to analyze multiple GFA files at once (that are packed into the same gz archive)?

Hey Tao, this is fantastic. Thanks for your contribution. If I understand you correctly, you want to analyze multiple GFA files at once (that are packed into the same gz archive)?

Yes, this is in case the file contains multiple gzip compressed blocks.

Hey Tao, this is fantastic. Thanks for your contribution. If I understand you correctly, you want to analyze multiple GFA files at once (that are packed into the same gz archive)?

Yes, this is in case the file contains multiple gzip compressed blocks.

At the moment, panacus cannot process multiple GFA files at once. The limitation is not MultiGZ, but the fact that GFA files can have overlapping node IDs. Panacus is not able to distinguish between them and will raise errors if multiple nodes with the same ID are read in.

Hey Tao, this is fantastic. Thanks for your contribution. If I understand you correctly, you want to analyze multiple GFA files at once (that are packed into the same gz archive)?

Yes, this is in case the file contains multiple gzip compressed blocks.

At the moment, panacus cannot process multiple GFA files at once. The limitation is not MultiGZ, but the fact that GFA files can have overlapping node IDs. Panacus is not able to distinguish between them and will raise errors if multiple nodes with the same ID are read in.

Perhaps my description was incorrect; I didn't mean to process multiple GFA files at once, but rather a single GFA.gz file that is composed of multiple gzip compressed blocks.

Ok, I understand now. Thanks for the contribution, this is useful.

gzip part1.gfa part2.gfa -c > all.gfa.gz
in this case GzDecoder will lost  all content  in part2.gfa, but MultiGzDecoder will not


Make orchestrator.docker properties optional

Support building base image from alpine-based image (resulting tag is the same, -alpine suffix is removed)
Make orchestrator.docker properties optional and not used in Kubernetes
Attempt to enable lock file maintenance in renovate.json

Closes #1031

I've reverted changes with alpine-based images, because we are currently building executables of save-agent and save-cli linked with glibc on Ubuntu. So they are unable to start on alpine.


Import issue with auto schema creation with dataclass

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
Hi,
I boiled down 2 issues (that look like just 1 really) to these 2 files and steps.
When running file 1 as is, I get the following error:
pydantic.errors.PydanticUndefinedAnnotation: name 'Optional' is not defined

if I import manually Optional in file 1, then it "works".
In a seemingly, related issue, at this point, I get this error:
pydantic.errors.PydanticUndefinedAnnotation: name 'datetime' is not defined

which can be corrected by changing the import to
import datetime

and updating the usages to
Optional[datetime.datetime]

All in all, it looks like Pydantic does not use/execute the imports for file 2 when running file 1 even though it uses file 2
Having 2 files is required for the issue to reproduce (all in 1 file does not crash).
Example Code
file 1:
from __future__ import annotations

from my_class import MyClass
from pydantic import ConfigDict, validate_call

@validate_call(
    config=ConfigDict(strict=True, arbitrary_types_allowed=True),
    validate_return=True,
)
def test(my_class: MyClass):
    pass

file 2:
from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime
from typing import Optional

@dataclass
class MyClass:
    member: Optional[str] = None
    created: Optional[datetime] = None
    updated: Optional[datetime] = None
    deleted: Optional[datetime] = None

Python, Pydantic & OS Version
$ python -c "import pydantic.version; print(pydantic.version.version_info())"

             pydantic version: 2.5.3
        pydantic-core version: 2.14.6
          pydantic-core build: profile=release pgo=false
                 install path: .virtualenvs/env3/lib/python3.9/site-packages/pydantic
               python version: 3.9.12 (main, May 12 2023, 09:21:14)  [Clang 14.0.3 (clang-14<IP_ADDRESS>.1)]
                     platform: macOS-14.1.2-x86_64-i386-64bit
             related packages: email-validator-2.0.0.post2 typing_extensions-4.7.1

@lpgros,
If you remove from __future__ import annotations from file2, things work as expected. This doesn't seem to be an issue with pydantic, but rather your import patterns.

@sydney-runkle annotations is needed when using forward references with type hinting and an integral part of the Python standard library.
It is listed as "working" in the Pydantic doc: https://docs.pydantic.dev/latest/concepts/postponed_annotations/
How could I have a fully typehinted project without them?
Please advise

@lpgros,
Taking a closer look at this, I think there is something we can do on our end to fix this issue. We'll open a PR shortly. Thanks for following up!

cheers! That would be great 😄

@lpgros,
Sure thing. When investigating more, we discovered that this is related to how we deal with namespaces for dataclasses. We have more intentional handling of namespaces for BaseModel types, so it's not surprising that this issue popped up for dataclasses. I suspect it might be an issue for TypedDict types as well.
Additionally, this problem crops up because of limitations with processing annotations that have been resolved with later Python versions. Stay tuned for our PR :).

thx!!
In the meantime, is there a way (I did not find one) to turn off Pydantic's need to generate a schema for Python dataclasses rather than treating them as "yet another class"? that would buy me time. Thank you! 😸

🙌


feat: RPC : use auto-generated types for the StarknetRpcApi trait

Feature Request
Describe the Feature Request
The rpc-core currently defines manually a StarknetRpcApi trait that stands for the Starknet RPC spec.
However, this type can (and should) be generated automatically from the RPC spec itself.
Describe Preferred Solution
The spec update PR has been merge so we can now use @open-rpc/typings to generate these types.
Additional Context
There might still be a discussion to have with other rust-based projects of the ecosystem to unify starknet types.
If the feature request is approved, would you be willing to submit a PR?

[x] Yes
[ ] No

Can we close in favor of https://github.com/xJonathanLEI/starknet-rs/tree/dev/jsonrpc_0_3_0 ?

I thought I was closed already actually, I'm working on integrating these types rn


Update lerna, and let it know that we are using Yarn workspaces

Related: https://github.com/liferay/liferay-themes-sdk/issues/149

Rebased to resolve merge conflict with my other PR.


Update references from Java to Java/Kotlin in the body of the article

Just noticed that we had references to Java and .java files in the body of the text. I'm updating them.

but it is not necessary to do this in this PR. I can prepare another PR soon, and check if there are more instance of using external links for the internal pages. 🙂

I've included it regardless as it was a valid fix 👍


Stalebot seems too aggresive

Because we have limited contributors at the moment, we are unable to fix the issues or merge the PRs in progress.
I think we should increase the values in the stalebot configurations to something like -
days-before-stale: 90
days-before-close: 30

IMHO, 120 days of time is going to be too long and one might not be able to keep track of the changes they did last time, if they worked on something ~90 days ago.

Good reading: https://www.hanselman.com/blog/always-be-closingpull-requests

IMHO, 120 days of time is going to be too long and one might not be able to keep track of the changes they did last time, if they worked on something ~90 days ago.

I agree with the above scenario where if someone starts working on a PR, then 120 days is too long of an inactivity duration.
My concern is issues not getting picked due to less no. of contributors.
Would it be possible to have separate values for issues and PRs. If yes, then maybe keeping days-before-stale to 30 on PRs and 90 on issues.

While working on this, I noticed that we are exempting issues/PRs with labels awaiting-approval, work-in-progress, help-wanted, but we haven't added these labels yet. And I don't think help wanted label (which we already have) is equivalent to help-wanted.
I think that we should exempt issues/PRs with labels awaiting-inputs, work-in-progress and help-wanted. Contributors cannot add any labels currently, so I don't see the use for awaiting-approval label. Instead the maintainers can add awaiting-inputs label on issues/PRs when more clarification is needed.
If we want to allow contributors to add awaiting-approval label, let's create a separate issue for that.


Mobile Wallet stops working if password on a startup was not re-entered

Bug Description
This bug appears in possibly rare use case. It appears if the initial flow of creating password was interrupted in a way that password was never re-entered. Mobile Wallet can't recover, it has to be reinstalled completely.
Steps to Reproduce

Install Mobile Wallet.
In "Select Passcode" dialogue, type in 6 digits.
New dialogue "Re-enter Passcode" opens.
Stop application (Android more menu to list the apps and close it).
Start application again.
"Enter Password" dialogue opens.
Type in 6 digits and select Continue.
Select Continue.

Expected Result
I am not sure what is expected, I assume that flow of creating a password should be restarted in this case.
Actual Result
Application stops.
Attached video:
https://user-images.githubusercontent.com/98821384/152296587-5f110b4e-30a1-4447-9034-7dd41fdf9141.mp4
Versions

Software Version: 1.2.6 (69)
OS: Android 10
Browser
Mobile device, if applicable: Samsung Galaxy J6

Software Version: 3.0.6 (90)
OS: Android 12
Mobile device: Samsung S21 Ultra
It is working also after the force stop.


Can't see annotation while too zoomed out

I try to refresh my view when my zoom level changes too much, cause zooming out I should show more annotations with a different radius.
However, zooming out too much makes all my annotations disappear, I print my count of self.mapClousterController.annotations.count and they return 17.

Apparently it only happens when I add after remove. I tried commenting out the line with removeannotations things are as expected

Can you reproduce the same issue in the example app? You'll probably need to implement the completion handler in addAnnotations:annotations withCompletionHandler: and remove in there to avoid messing up the annotations

Hmm, I tried replicating that in the example app but I failed apparently. But to continue with your advice.

You'll probably need to implement the completion handler in addAnnotations:annotations withCompletionHandler: and remove in there to avoid messing up the annotations

Why am I removing my annotations in the addAnnotations completion block?
Currently this is how I do it.
`                self.mapClusterController.removeAnnotations(Array(self.mapClusterController.annotations), withCompletionHandler: {
self.mapClusterController.addAnnotations(self.locations, withCompletionHandler: nil)
            })`

I haven't heard back from this issue in a while, so I'm closing it for now. Let me know if you have more issues – thanks


Less spammy mode when running as relaybot

Don't complain about everything, use more strict addressing

Strict addressing (with localpart) added, edit complains disabled.
Probably need a mode to suppress all IRC events.


Removing the explicit createdirectory call in release extension, this…

… is already taken care in CleanupDirectory method.

What if agent/worker crash or Power-Off that cause the previous cleanup didn't run?

@TingluoHuang we are calling releaseFileSystemManager.cleanupDirectory which makes sure the directory exists and it does not have any content. If there is a crash or poweroff, this will run again and cleanup the directory. Don't you think?

@kasubram
here is what i mean:

you queue a release, it runs on the agent and download the artifact.
the worker crash, so the artifact folder hasn't been clean up.
you queue another release, is there any chance 2 releases use same artifact download folder? will the previous failed release affect following release?

If release from the same release definition, it will use the same download folder.
Previously uncleaned directory won't cause any issue, as we always call cleanupdirectory before downloading the artifact.

@kasubram
ok.


Phoenix Query Server failed to start in HBase

@lucasbak
ambari version：<IP_ADDRESS>.0-26
odp version：<IP_ADDRESS>-105
Cause analysis: The actual script is under /usr/odp/<IP_ADDRESS>-105/phoenix-queryserver/bin/queryserver.py, but the queryserver.py script executed during the installation process is in the /usr/odp/<IP_ADDRESS>-105/phoenix/bin/ directory Search  , which results in the python script being executed not being found.
The error message is as follows：

The soft connection of the node where the non-Phoenix Query Server is located is as follows：

The node where Phoenix Query Server is located is softly connected as follows：

Hi @SGITLOGIN
Indeed /usr/odp/current/phoenix-server should point to /usr/odp/current/phoenix-server
We are working on reproducing and fixing the issue.

@SGITLOGIN
We have succesfully reproduced the issue.
We are fixing and will let you know when it is fixed.
Best regards,
Clemlab Team

@lucasbak Will this issue with HBase affect the dependent component Altas?

@SGITLOGIN ,
We have identified the solution.
We are currently testing it and the whole impacts.
It will not affect HBase nor Atlas, as it is a separate package

OK，Should a new version be released after the problem is fixed？

Hi @SGITLOGIN ,
Yes the new version release will contains fixes for Phoenix and will come with new ambari version also.

OK

@lucasbak
When is the new version expected to be released?

@SGITLOGIN ,
this week, we are also bringing some fixes on other component.