KAFKA-9298: reuse mapped stream error in joins

When performing a join with a stream that needs repartitioning, Kafka Streams automatically creates a repartition topic.  If the user does not use StreamJoined to name to repartition topic, Kafka Streams uses the generated name of the KStream instance for the repartition topic name.
If the KStream instance requiring the repartition participates in another join, the second repartition topic is created using the name of the operator. This name reuse is what causes the InvalidTopologyException.  The error occurs because the InternalTopologyBuilder has already registered the repartition source name previously.
For example, this topology will cause an error because Kafka Streams will attempt to create two repartition topics (which is correct behavior) but using the same name each time which causes the error.
KStream<String, String> newStream = stream1.map((k, v) -> new KeyValue<>(v, k));
newStream.join(stream2, (value1, value2) -> value1 + value2, JoinWindows.of(ofMillis(100))).to("out-one");
// using newStream in another join here causes the error
newStream.join(stream3, (value1, value2) -> value1 + value2, JoinWindows.of(ofMillis(100))).to("out-to");

However this topology, which is the same except the user has provided repartition topic names, is fine.  Note the use of StreamJoined.withName here
KStream<String, String> newStream = stream1.map((k, v) -> new KeyValue<>(v, k));
final StreamJoined<String, String, String> streamJoined = StreamJoined.with(Serdes.String(), Serdes.String(), Serdes.String());
newStream.join(stream2, (value1, value2) -> value1 + value2, JoinWindows.of(ofMillis(100)), streamJoined.withName("first-join")).to("out-one");
// using newStream in another join here is fine because the name of the repartition topic is unique
newStream.join(stream3, (value1, value2) -> value1 + value2, JoinWindows.of(ofMillis(100)), streamJoined.withName("second-join")).to("out-two");

This bug has been present for some time as I tested this out on 2.0 before we added the optimization layer.
Ideally, the fix should be to generate a repartition topic name each time to avoid such issues.  But IMHO that ship has already sailed because by introducing a new name generation will cause compatibility issues for existing topologies.  So generating new names is out for now, at least.
The proposed fix is:

For KStream objects needing repartitioning and  using generated names, reuse the repartition topic node in any additional joins.
For KStream instances needing repartitioning using user-provided names always create a new repartition topic node for each join as each one will have a unique name

I've added tests confirming the expected behavior.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Java 8 passed
Java 11 failed with kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup updated existing Jira ticket
retest this please.

ping @mjsax @vvcephei for review

ping @mjsax and @vvcephei, I know you guys are busy, but do you mind having a look?

Ideally, the fix should be to generate a repartition topic name each time to avoid such issues. But IMHO that ship has already sailed because by introducing a new name generation will cause compatibility issues for existing topologies.

Why that? Because such a topology would hit the bug, it could never be deployed, and thus nobody can actually run such a topology? In fact, shouldn't be "burn" and index even if a name is provided (IIRC, we do this for some cases)?
I agree thought, that merging repartition topics (as proposed in (1)) should be done if possible (it's a historic artifact that we did not merge them in the past and IMHO we should not make the same mistake again?).
For (2), it's a tricky question because the different names are used for different stores and changelog topics  (ie, main purpose?) -- it seems to be a "nasty side effect" if we would end up with two repartition topics for this case? Of course, given the new repartition() operator, a user can work around it by using it after map() and before calling join(). Just brainstorming here what the impact could be and what tradeoff we want to pick.

Why that? Because such a topology would hit the bug, it could never be deployed, and thus nobody can actually run such a topology? In fact, shouldn't be "burn" and index even if a name is provided (IIRC, we do this for some cases)?

Yes in some cases we increment the index when users provide names.  But right now we don't increment the counter at all when creating repartition topics as we reuse the name as is.  My main point is that if we started to generate new names for repartition topics we'd create topology compatibility issues as the newly generated name would bump the count for all downstream nodes.  Right now I'm leaning towards going with the solution you presented in point one in https://github.com/apache/kafka/pull/8504#discussion_r413380852

I agree thought, that merging repartition topics (as proposed in (1)) should be done if possible (it's a historic artifact that we did not merge them in the past and IMHO we should not make the same mistake again?).

But by doing so we are "leaking" optimization logic as you pointed out above.  I'm leaning towards building the topology "as is", meaning create two repartition topics if that's what is required.  But I don't have a strong opinion and I would be fine with keeping the current solution in this PR.

For (2), it's a tricky question because the different names are used for different stores and changelog topics (ie, main purpose?) -- it seems to be a "nasty side effect" if we would end up with two repartition topics for this case? Of course, given the new repartition() operator, a user can work around it by using it after map() and before calling join(). Just brainstorming here what the impact could be and what tradeoff we want to pick.

I'm not sure I follow here the "nasty side effect" comment.
If a user does streamA.join(streamB, ..., StreamJoined.name("foo") and streamA.join(streamC, ..., StreamJoined.name("bar")  then we should create two repartiton topics as that's what the user is expecting.  If they elect to use optimization then removing redundant repartition topics is expected behavior.  I think this also goes back to your original comment about the leaking of optimization details.

@mjsax thanks for the review!

I'm on the fence about whether this is an "optimization" or "reasonable behavior". It sort of feels like the latter, and the only reason we needed to introduce the "repartition-collapsing" optimization is that we failed to introduce reasonable behavior from the beginning.

Thanks for the comments @vvcephei.  That is a good point you raise here.
@mjsax WDYT about I merge this PR as is and we re-visit repartition topics and optimizations in general later on possibly when looking at topology evolution?

we should create two repartiton topics as that's what the user is expecting.

Is she? I guess it's clearly document that the name is use for repartition topics, too, but it might still be an unexpected "side effect"? But I see the point that it's not too easy to resolve: Maybe we could log a warning about creating an unnecessary duplicated repartition topics (and update the docs how to resolve this, ie, via calling repartition() upfront before the join to specify one name)? Just picking one of both names seems wrong. Throwing an exception and refuse to build the topology seems too harsh?
Overall, it seems it might be best to merge the PR using the current approach, and merge the repartition topics directly in StreamsBuilder instead of waiting for the optimization (this also avoid the issue with index bumping and is better default behavior anyway).
Can you add the test for streamA-tableB, streamA-tableC join?

Is she? I guess it's clearly document that the name is use for repartition topics, too, but it might still be an unexpected "side effect"? But I see the point that it's not too easy to resolve: Maybe we could log a warning about creating an unnecessary duplicated repartition topics (and update the docs how to resolve this, ie, via calling repartition() upfront before the join to specify one name)? Just picking one of both names seems wrong. Throwing an exception and refuse to build the topology seems too harsh?

Thinking about this again, I'm now leaning towards reusing a repartition node regardless if user provides a name or not.  We can update the docs to say repartition topics are only built when required and once one is created the same repartition topic is used again.
WDYT?

Can you add the test for streamA-tableB, streamA-tableC join?

Will do.

@mjsax updated this

Thinking about this again, I'm now leaning towards reusing a repartition node regardless if user provides a name or not. We can update the docs to say repartition topics are only built when required and once one is created the same repartition topic is used again.
WDYT?

Works for me. Uses can also use TopologyDescription go get more details. We just need to ensure that we pick the same name each time... \cc @vvcephei

Works for me

Ack, I'll make the updates soon

@mjsax- Updated this to always reuse repartition graph node.
Also, Aggregates follow a similar pattern to what joins used to do.  IMHO we need to make the behavior consistent, so I've created https://issues.apache.org/jira/browse/KAFKA-9976 and will provide a PR soon.

One failure for Java 8, 11 and 14 test results cleanup already.
retest this please.

Java 8 failed with 11:46:32 Caused by: java.lang.OutOfMemoryError: unable to create new native thread

@bbejeck Are the any backward compatibility concerns for KAFKA-9976 ?

@bbejeck Are the any backward compatibility concerns for KAFKA-9976 ?

Good point, maybe we should just leave this one alone for now.  If you agree I'll close that PR.

@mjsax @vvcephei updated (rebased as well) this to original PR state, only reuse repartition node when the name is generated.

Java 8, 11, and 14 failed.  Test results cleaned out already.
Retest this please.

Java 8 passed.
Java 11 failed with  org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[false]
Java 14 failed with  org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[false] and  org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[true]
I've updated the existing Jira tickets.
Retest this please.

Java 8 passed,
Java 11 failed with

org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication
org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[false]
org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[true]

Java 14 failed with

org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[false]
org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[true]

Retest this please.

Retest this please

retest this please

Java 14 passed,
Java 11 failed with
21:42:58 > Process 'Gradle Test Executor 48' finished with non-zero exit value 1

Java 8 failed with org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest > shouldUpgradeFromEosAlphaToEosBeta[true] FAILED
retest this please

Java 14:
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest.testCreateInternalTopicsWithFewerReplicasThanBrokers
org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[true]

Java 11: org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[true]
Java 8: org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration

(Sorry for the forth-and-back discussion on the PR -- it's just a delicate subject to make sure we don't break compatibility.)

No worries @mjsax! it was a great discussion, and it's always good to go over these type of issues thoroughly

cherry-picked to 2.5


contrib/docker doesnt support corporate environment/proxy

A clear and concise description of what the bug is.
Expected results
If you run
incubator-superset/contrib/docker]$ docker-compose run -e SUPERSET_LOAD_EXAMPLES=yes --rm superset ./docker-init.sh
what you expected to happen.
I would expect the container to run
Actual results
what actually happens.
Connection errors
Screenshots

How to reproduce the bug
docker-compose run -e SUPERSET_LOAD_EXAMPLES=yes --rm superset ./docker-init.sh
Environment
Any environment with a proxy

Hi @dprophet,
Can you please try: https://docs.docker.com/network/proxy/


[5.8] Don't call redirectTo() when you don't need it

Sometimes We Write some logic in the redirectTo() method, as well as creating an API based workflow on the same codebase, there are some cases where the logic we write get executed even in the API because by default, laravel calls the \Illuminate\Auth\Middleware\Authenticate::redirectTo() to pass it as the 3rd param to the AuthenticationException, so we need to be able to ignore the useless call of that method if the request excepts json wantsJson().
Sorry if I didn't explain it in words perfectly.

Will leave as-is for now.


Exiting fullscreen is breaking the layout.

I have this issue on Chrome. Basically, I request fullscreen on a angular component and after exiting full screen it basically breaks the layout and do not return to original height/width

This is just a polyfill for the native API. This module doesn't have anything do to with the actual fullscreening.


Debian one command install script broken with invalid certificate

Describe the bug
Following the Debian one command install, the script exits without download or installed Azure CLI.
Related command
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
Errors
Script exits (code 60) without downloading and installed CLI tool.
With curl in verbose mode, errors shows the domain has an invalid certificate.
Issue script & Debug output
* Connection #0 to host aka.ms left intact
* Issue another request to this URL: 'https://azurecliprod.blob.core.windows.net/$root/deb_install.sh'
*   Trying <IP_ADDRESS>:443...
* Connected to azurecliprod.blob.core.windows.net (<IP_ADDRESS>) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
...
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.blob.core.windows.net
*  start date: Mar 20 00:57:24 2023 GMT
*  expire date: Mar 20 00:57:24 2024 GMT
*  subjectAltName does not match azurecliprod.blob.core.windows.net
* SSL: no alternative certificate subject name matches target host name 'azurecliprod.blob.core.windows.net'
* Closing connection 1
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, close notify (256):

Expected behavior
script downloads and installed Azure CLI.
Environment Summary
n/a
Additional context
No response

Thank you for opening this issue, we will look into it.

linked to MicrosoftDocs/azure-docs-cli/issues/3747

I can't repro.
You log is pretty weird: subject: CN=*.blob.core.windows.net should match the domain.
subjectAltName does not match azurecliprod.blob.core.windows.net
Here is my log:
* Server certificate:
*  subject: CN=*.blob.core.windows.net
*  start date: Mar 20 00:57:24 2023 GMT
*  expire date: Mar 20 00:57:24 2024 GMT
*  subjectAltName: host "azurecliprod.blob.core.windows.net" matched cert's "*.blob.core.windows.net"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft RSA TLS CA 01

Are you using a proxy, are you able to open https://aka.ms/InstallAzureCLIDeb in browser?

The certificatet was updated and script is now working.
Below is the full log message
curl -vsL https://aka.ms/InstallAzureCLIDeb
*   Trying <IP_ADDRESS>:443...
* Connected to aka.ms (<IP_ADDRESS>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=go.microsoft.com
*  start date: Oct 13 21:29:19 2022 GMT
*  expire date: Oct  8 21:29:19 2023 GMT
*  subjectAltName: host "aka.ms" matched cert's "aka.ms"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 06
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /InstallAzureCLIDeb HTTP/1.1
> Host: aka.ms
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Server: Kestrel
< Location: https://azurecliprod.blob.core.windows.net/$root/deb_install.sh
< Request-Context: appId=cid-v1:7d63747b-487e-492a-872d-762362f77974
< X-Response-Cache-Status: True
< Expires: Wed, 19 Jul 2023 18:16:46 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 19 Jul 2023 18:16:46 GMT
< Connection: keep-alive
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
<
* Connection #0 to host aka.ms left intact
* Issue another request to this URL: 'https://azurecliprod.blob.core.windows.net/$root/deb_install.sh'
*   Trying <IP_ADDRESS>:443...
* Connected to azurecliprod.blob.core.windows.net (<IP_ADDRESS>) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.blob.core.windows.net
*  start date: Mar 20 00:57:24 2023 GMT
*  expire date: Mar 20 00:57:24 2024 GMT
*  subjectAltName does not match azurecliprod.blob.core.windows.net
* SSL: no alternative certificate subject name matches target host name 'azurecliprod.blob.core.windows.net'
* Closing connection 1
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, close notify (256):

@chanster is your issue resolved?

oh yes, it issue is resolved.


First version of GLSL cross-compilation

Using SPIRV-Cross. The goal is to prototype loading GLSL shaders in the Vulkan fork of the demo and to use SPIRV-Cross as an inspiration for future GLSL codegen.

See https://github.com/Microsoft/DirectXShaderCompiler/wiki/SPIR‐V-CodeGen and https://github.com/Microsoft/DirectXShaderCompiler/blob/main/docs/SPIR-V.rst#optimization for future work.


sms_autofill: ^2.2.0 when press any key into textbox

Steps to Reproduce

Execute flutter run on the code sample
...
...

Expected results:
Actual results:

Code sample

Logs

I'm pretty sure issues with sms_autofill  should be opened here

Issues with third-party packages need to be reported to their authors.


"onRouteChanged" is sometimes not called

Probably a bug in Logseq. As a temporary solution, there is a way to use another "onPageHeadActionsSlotted" together.

Thank you very much for your contribution. This is a great commit.


Implement removal of layers

We should implement a layer.remove() method that removes layers from the visualization.

+1 APLpy has this, and I’ve found it awkward (last time I’ve used it) to remove a layer with multiple plotted. Especially for notebook and lab users, if the layer isn’t saved as a variable, I couldn’t remove it.
I’m sorry to not have had time to dig into the code more deeply before suggesting this, but do layers get added to a list that can always be accessed?

@philrosenfield - not yet, see https://github.com/WorldWideTelescope/pywwt/issues/132 :)

It looks like layer removal doesn't work at the moment with the JS API:
wwtlib.LayerManager.deleteLayerByID(layer.id)

doesn't seem to do anything - it removes the layer from the manager, but the layer is still visible. I also tried layer.cleanUp but no luck.

The functionality is already there, you just have to use all three parameters, if you skip the last two parameters it defaults as false and does nothing. Since the WWT layer universe is a multiverse (one for ambient/editing tours, one for playing tours) the delete can have different effects on the current state.
DeleteLayerByID(Guid ID, bool removeFromParent, bool updateTree)
Remove from parent will remove a tour from being referenced.. Must be true or the parental reference will still keep it alive.
Update tree will cause the UI to refresh to show the changes.
In short when you use JavaScript to access c# projected methods, make sure you are not short-changing the parameter list. JavaScript is duck-typed and will let you call methods with too few too many or wrongly type parameters and hilarity ensues, so make sure you are checking the parameter list.

Just confirming that fixes it - thanks!


Missing to free pThreadInfo leads to memory leak in iot_threads_freertos.c

Describe the bug
DEMO_RUNNER_RunDemos call Iot_CreateDetachedThread in which pThreadInfo is malloced. And it is freed if returns from threadRoutine. When executing the task, at the end of runDemoTask, vTaskDelete( NULL ) is called as a result of which a pendsv interrupt is triggered and task reschedules to another task, then it will not return from pThreadInfo->threadRoutine( pThreadInfo->pArgument ). So the malloced pThreadInfo will not be freed. This causes memory leak.
System information

Which hardware board or part numbers?    All hardware as long as I run the demo.
IDE used      DS5
Operating System [Windows|Linux|MacOS]    All
Version of Amazon FreeRTOS (run git describe --tags to find it)   201906.00_Major-428-g026fc88e
Project [Custom Application|MQTT|Shadow|Greengrass|OTA|Device Defender|TCP Echo|Sub pub Demo|Simple TCP Server]     MQTT Demo
If your project is a Custom Application, please add the relevant code snippet in the section Code to reproduce the bug.

Expected behavior
A clear and concise description of what you expected to happen.
No memory leak when I run the demo.
Screenshots or console output
If appropriate, please cut-and-paste the console output to explain the issue. If possible, include the call stack.
To reproduce
Steps to reproduce the behavior:
Just run the MQTT demo on any target board.
Code to reproduce the bug
The code should be wrapped in the cpp tag in order to be displayed clearly. For example:
printf("Hello World")

If the code is longer than 30 lines, GIST is preferred.
Additional context
Add any other context about the problem here.
Thank you!

Thank you taking time and reporting this issue. We have fixed this in this PR: https://github.com/aws/amazon-freertos/pull/991
Thanks.

Thanks for fixing it.


make event.context.params usable within server middleware

Describe the feature
Currently any usage of event.context.params, getRouterParams or any equivalent within server middlewares (~/server/middleware) will return undefined.
We should be able to access event.context.params within server middlewares as it can open a lot of possible usage.
For example, directly loading users' permissions within middlewares and deciding whether they can use the API endpoint.
Additional information

[ ] Would you be willing to help implement this feature?

Hi. This is not technically possible to access matched params in middleware because they run before the router handler.
You can use custom pattern matcher using event.path in middleware to apply logic to a group of matched routes.


wrong error notification on @click @submit

wrong error notification on @click @submit

Duplicate of #1985


"ModuleNotFoundError: No module named 'scipy'" CI Test Failures (in master)?

Reproducing code example:
See CI of gh-13441, gh-13463
Error message:
Four (separate) failures occurring in both of these PRs. I'm not sure if the failures are related, but I noticed them at the same time, so I'm putting them all here for now.
In Windows builds on Azure:
==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : 2.180.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\ca35efdd-7955-4298-a881-94018e1affc9.ps1'"
Traceback (most recent call last):
  File "runtests.py", line 565, in <module>
    main(argv=sys.argv[1:])
  File "runtests.py", line 291, in main
    __import__(PROJECT_MODULE)
ModuleNotFoundError: No module named 'scipy'
##[error]PowerShell exited with code '1'.
Finishing: Run SciPy Test Suite

In refguide_asv_check and Linux Tests we're seeing directed_hausdorff failures:

     _________________________ TestHausdorff.test_symmetry __________________________
[gw0] linux -- Python 3.8.7 /opt/hostedtoolcache/Python/3.8.7/x64/bin/python
/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/tests/test_hausdorff.py:37: in test_symmetry
    forward = directed_hausdorff(self.path_1, self.path_2)[0]
        self       = <scipy.spatial.tests.test_hausdorff.TestHausdorff object at 0x7f28fbea4970>
/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/distance.py:461: in directed_hausdorff
    result = _hausdorff.directed_hausdorff(u, v, seed)
        seed       = 0
        u          = array([[ 0.35923135,  0.93324854,  0.        ],
       [-0.71983479, -0.69414542,  0.        ],
       [-0.92442618,  ...0695034,  0.        ],
       [ 0.93629495,  0.3512147 ,  0.        ],
       [-0.48493437, -0.87455055,  0.        ]])
        v          = array([[ 1.18546344,  3.07972019,  0.        ],
       [-1.43966959, -1.38829085,  0.        ],
       [-1.84885236,  ...1390068,  0.        ],
       [ 1.8725899 ,  0.7024294 ,  0.        ],
       [-0.96986874, -1.74910109,  0.        ]])
_hausdorff.pyx:37: in scipy.spatial._hausdorff.directed_hausdorff
    ???
        __all__    = ['directed_hausdorff']
        __builtins__ = <builtins>
        __doc__    = '\nDirected Hausdorff Code\n\n.. versionadded:: 0.19.0\n\n'
        __file__   = '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/_hausdorff.cpython-38-x86_64-linux-gnu.so'
        __loader__ = <_frozen_importlib_external.ExtensionFileLoader object at 0x7f2939a17490>
        __name__   = 'scipy.spatial._hausdorff'
        __package__ = 'scipy.spatial'
        __pyx_unpickle_Enum = <built-in function __pyx_unpickle_Enum>
        __spec__   = ModuleSpec(name='scipy.spatial._hausdorff', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f2939a.../hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/scipy/spatial/_hausdorff.cpython-38-x86_64-linux-gnu.so')
        __test__   = {}
        directed_hausdorff = <built-in function directed_hausdorff>
        np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/__init__.py'>
mtrand.pyx:4478: in numpy.random.mtrand.RandomState.shuffle
 E   UserWarning: `x` isn't a recognized object; `shuffle` is not guaranteed to behave correctly. E.g., non-numpy array/tensor objects with view semantics may contain duplicates after shuffling.
        MutableSequence = <class 'collections.abc.MutableSequence'>
        RandomState = <class 'numpy.random.mtrand.RandomState'>
        _MT19937   = <class 'numpy.random._mt19937.MT19937'>
        __all__    = ['beta', 'binomial', 'bytes', 'chisquare', 'choice', 'dirichlet', ...]
        __builtins__ = <builtins>
        __doc__    = None
        __file__   = '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so'
        __loader__ = <_frozen_importlib_external.ExtensionFileLoader object at 0x7f2942eb7d60>
        __name__   = 'numpy.random.mtrand'
        __package__ = 'numpy.random'
        __spec__   = ModuleSpec(name='numpy.random.mtrand', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f2942eb7d60...'/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so')
        __test__   = {'RandomState.binomial (line 3262)': '\n        binomial(n, p, size=None)\n\n        Draw samples from a binomial dist...      array(['pooh', 'pooh', 'pooh', 'Christopher', 'piglet'], # random\n              dtype='<U11')\n\n        ", ...}
        _rand      = RandomState(MT19937) at 0x7F2943136E40
        beta       = <built-in method beta of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        binomial   = <built-in method binomial of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        bytes      = <built-in method bytes of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        chisquare  = <built-in method chisquare of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        choice     = <built-in method choice of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        dirichlet  = <built-in method dirichlet of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        exponential = <built-in method exponential of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        f          = <built-in method f of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        gamma      = <built-in method gamma of numpy.random.mtrand.RandomState object at 0x7f2943136e40>

        geometric  = <built-in method geometric of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        get_state  = <built-in method get_state of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        gumbel     = <built-in method gumbel of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        hypergeometric = <built-in method hypergeometric of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        laplace    = <built-in method laplace of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        logistic   = <built-in method logistic of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        lognormal  = <built-in method lognormal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        logseries  = <built-in method logseries of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        multinomial = <built-in method multinomial of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        multivariate_normal = <built-in method multivariate_normal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        negative_binomial = <built-in method negative_binomial of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        noncentral_chisquare = <built-in method noncentral_chisquare of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        noncentral_f = <built-in method noncentral_f of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        normal     = <built-in method normal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages/numpy/__init__.py'>
        operator   = <module 'operator' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/operator.py'>
        pareto     = <built-in method pareto of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        permutation = <built-in method permutation of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        poisson    = <built-in method poisson of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        power      = <built-in method power of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        rand       = <built-in method rand of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        randint    = <built-in method randint of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        randn      = <built-in method randn of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        random     = <built-in method random of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        random_integers = <built-in method random_integers of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        random_sample = <built-in method random_sample of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        ranf       = <built-in function ranf>
        rayleigh   = <built-in method rayleigh of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        seed       = <built-in method seed of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        set_state  = <built-in method set_state of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        shuffle    = <built-in method shuffle of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_cauchy = <built-in method standard_cauchy of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_exponential = <built-in method standard_exponential of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_gamma = <built-in method standard_gamma of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_normal = <built-in method standard_normal of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        standard_t = <built-in method standard_t of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        triangular = <built-in method triangular of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        uniform    = <built-in method uniform of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        vonmises   = <built-in method vonmises of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        wald       = <built-in method wald of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        warnings   = <module 'warnings' from '/opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/warnings.py'>
        weibull    = <built-in method weibull of numpy.random.mtrand.RandomState object at 0x7f2943136e40>
        zipf       = <built-in method zipf of numpy.random.mtrand.RandomState object at 0x7f2943136e40>

On AppVeyor:
C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgtc.o C:\Use
rs\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgtf.o
C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgts.o -LC:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\lib\gcc\x86_64-w64-mingw32\6.3.0
-Lc:\python37-x64\libs -Lc:\python37-x64\PCbuild\amd64 -Lbuild\temp.win-amd64-3.7 -o build\temp.win-amd64-3.7\Release\.libs\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.dll
build\temp.win-amd64-3.7\Release\.libs\libopenblas.3HBPCJB5BPQGKWVZAVEBXNNJ2Q2G3TUP.gfortran-win_amd64.dll -Wl,--allow-multiple-definition
-Wl,--output-def,build\temp.win-amd64-3.7\Release\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64" failed with exit status 1
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy
Failed to build scipy
ERROR: Failed to build one or more wheels
Exception information:
Traceback (most recent call last):
  File "c:\python37-x64\lib\site-packages\pip\_internal\cli\base_command.py", line 189, in _main
    status = self.run(options, args)
  File "c:\python37-x64\lib\site-packages\pip\_internal\cli\req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "c:\python37-x64\lib\site-packages\pip\_internal\commands\wheel.py", line 191, in run
    "Failed to build one or more wheels"
pip._internal.exceptions.CommandError: Failed to build one or more wheels

I am also seeing this in macOS tests, but that might be unrelated, as I've seen it intermittently before today:
gcc: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/source.c
gcc /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/source.o -L/usr/local/lib -lopenblas -o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp76pay1ja/a.out
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /usr/local/include/python3.9 is invalid.
  return self.get_paths(self.section, key)
blas_opt_info:
blas_mkl_info:
  libraries mkl_rt not found in ['/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_info:
C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/opt/sqlite/include -I/usr/local/opt/sqlite/include

creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T
creating /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b
compile options: '-c'
gcc: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/source.c
gcc /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/source.o -L/usr/local/lib -lopenblas -o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpayy_ek6b/a.out
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']

blas_info:
Traceback (most recent call last):
  File "/Users/runner/work/scipy/scipy/setup.py", line 605, in <module>
    setup_package()
  File "/Users/runner/work/scipy/scipy/setup.py", line 601, in setup_package
    setup(**metadata)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/core.py", line 135, in setup
    config = configuration()
  File "/Users/runner/work/scipy/scipy/setup.py", line 524, in configuration
    config.add_subpackage('scipy')
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/setup.py", line 18, in configuration
    config.add_subpackage('optimize')
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/optimize/setup.py", line 102, in configuration
    ext = pythran.dist.PythranExtension(
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/pythran/dist.py", line 131, in __init__
    cfg_ext = cfg.make_extension(python=True, **kwargs)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/pythran/config.py", line 220, in make_extension
    numpy_blas = numpy_sys.get_info(user_blas)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 584, in get_info
    return cl().get_info(notfound_action)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 844, in get_info
    self.calc_info()
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 2040, in calc_info
    info = self.check_libs(lib_dirs, blas_libs, [])
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 986, in check_libs
    info = self._check_libs(lib_dirs, libs, opt_libs, [ext])
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 1057, in _check_libs
    found_dirs, found_libs = self._find_libs(lib_dirs, libs, exts)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 1041, in _find_libs
    found_lib = self._find_lib(lib_dir, lib, exts)
  File "/Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/python3.9/site-packages/numpy/distutils/system_info.py", line 1022, in _find_lib
    raise RuntimeError(_accel_msg.format(filename=p[0]))
RuntimeError: Found /usr/lib/libblas.dylib, but that file is a symbolic link to the MacOS Accelerate framework, which is not supported by NumPy. You must configure the build to use a different optimized library, or disable the use of optimized BLAS and LAPACK by setting the environment variables NPY_BLAS_ORDER="" and NPY_LAPACK_ORDER="" before building NumPy.

Build failed! (0:01:47.617718 elapsed)
Error: Process completed with exit code 1.

The Windows build failures on Azure are due to the Build step failing with similar errors to AppVeyor.  However, the Build step shows as passing where it should be reporting failure, not sure why it would be doing this

The Windows build failures on Azure are due to the Build step failing with similar errors to AppVeyor.  However, the Build step shows as passing where it should be reporting failure, not sure why it would be doing this

Eeew, this something I got some time ago #13072. Probably picked up the wrong compilers or something. I never understood why but renewed my whole toolchain relating to mingw and it disappeared.

Eeew, this something I got some time ago #13072. Probably picked up the wrong compilers or something. I never understood why but renewed my whole toolchain relating to mingw and it disappeared.

NumPy 1.20.0 was released 7 hours ago, so seems like some connection with that may be possible.
For the other point, we need special Azure stage settings to report failure at the exact point of failure. I think I set that up a bit more robustly for NumPy CI ages ago, but there are various ways to set the condition() field so that failures don't skip through to the next step or whatever (conversely, sometimes you do want to ignore an non-zero exit code).

NumPy 1.20.0 was released 7 hours ago, so seems like some connection with that may be possible.
For the other point, we need special Azure stage settings to report failure at the exact point of failure. I think I set that up a bit more robustly for NumPy CI ages ago, but there are various ways to set the condition() field so that failures don't skip through to the next step or whatever (conversely, sometimes you do want to ignore an non-zero exit code).

directed_hausdorff depends on specific details of NumPy random behavior at the Cython level -- I believe I pinged @mattip and @bashtage about that ages ago, but maybe something has changed a bit there.

directed_hausdorff depends on specific details of NumPy random behavior at the Cython level -- I believe I pinged @mattip and @bashtage about that ages ago, but maybe something has changed a bit there.

Ah yes, I remember indeed fiddling with my numpy version too. That's the rambling I have in the linked issue.

Ah yes, I remember indeed fiddling with my numpy version too. That's the rambling I have in the linked issue.

There's also (see here):
RuntimeError: Found /usr/lib/libblas.dylib, but that file is a symbolic link to the MacOS Accelerate framework,
which is not supported by NumPy. You must configure the build to use a different optimized library,
or disable the use of optimized BLAS and LAPACK by setting the environment variables
NPY_BLAS_ORDER="" and NPY_LAPACK_ORDER="" before building NumPy.

There's also (see here):
RuntimeError: Found /usr/lib/libblas.dylib, but that file is a symbolic link to the MacOS Accelerate framework,
which is not supported by NumPy. You must configure the build to use a different optimized library,
or disable the use of optimized BLAS and LAPACK by setting the environment variables
NPY_BLAS_ORDER="" and NPY_LAPACK_ORDER="" before building NumPy.

directed_hausdorff depends on specific details of NumPy random behavior at the Cython level -- I believe I pinged @mattip and @bashtage about that ages ago, but maybe something has changed a bit there.

I opened gh-13468 for this one. Looking into it now.

directed_hausdorff depends on specific details of NumPy random behavior at the Cython level -- I believe I pinged @mattip and @bashtage about that ages ago, but maybe something has changed a bit there.

I opened gh-13468 for this one. Looking into it now.

I removed the last part of the AppVeyor failure in the issue description about a duplicate numpy, that was misleading. The relevant bit is:
scipy/integrate/quadpack/dqag.f:180: undefined reference to `xerror_'
...
C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgts.o -LC:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\lib\gcc\x86_64-w64-mingw32\6.3.0
-Lc:\python37-x64\libs -Lc:\python37-x64\PCbuild\amd64 -Lbuild\temp.win-amd64-3.7 -o build\temp.win-amd64-3.7\Release\.libs\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.dll
build\temp.win-amd64-3.7\Release\.libs\libopenblas.3HBPCJB5BPQGKWVZAVEBXNNJ2Q2G3TUP.gfortran-win_amd64.dll -Wl,--allow-multiple-definition
-Wl,--output-def,build\temp.win-amd64-3.7\Release\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64" failed with exit status 1

  Building wheel for scipy (PEP 517): finished with status 'error'

No idea what the problem is. It's for pip 21.0.1, numpy 1.20.0. Unsure what the deal is with numpy master being pointed at on line 14 of appveyor.yml, that doesn't seem used.
Next step for that: confirm it's a problem with numpy 1.20.0 by checking if it goes away if we install numpy `1.19.5 instead.

I removed the last part of the AppVeyor failure in the issue description about a duplicate numpy, that was misleading. The relevant bit is:
scipy/integrate/quadpack/dqag.f:180: undefined reference to `xerror_'
...
C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-ev3j_zdy\build\temp.win-amd64-3.7\scipy\integrate\quadpack\dqwgts.o -LC:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\lib\gcc\x86_64-w64-mingw32\6.3.0
-Lc:\python37-x64\libs -Lc:\python37-x64\PCbuild\amd64 -Lbuild\temp.win-amd64-3.7 -o build\temp.win-amd64-3.7\Release\.libs\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.dll
build\temp.win-amd64-3.7\Release\.libs\libopenblas.3HBPCJB5BPQGKWVZAVEBXNNJ2Q2G3TUP.gfortran-win_amd64.dll -Wl,--allow-multiple-definition
-Wl,--output-def,build\temp.win-amd64-3.7\Release\libdqag.QD7OX7SDRWGIZYWWAF2SE2SG6JDJXW3B.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64" failed with exit status 1

  Building wheel for scipy (PEP 517): finished with status 'error'

No idea what the problem is. It's for pip 21.0.1, numpy 1.20.0. Unsure what the deal is with numpy master being pointed at on line 14 of appveyor.yml, that doesn't seem used.
Next step for that: confirm it's a problem with numpy 1.20.0 by checking if it goes away if we install numpy `1.19.5 instead.

Eeew, this something I got some time ago #13072 (expand the details section). Probably picked up the wrong compilers or something. I never understood why but renewed my whole toolchain relating to mingw and it disappeared.

Ah yes, that's the issue that explains the Azure failures.

Eeew, this something I got some time ago #13072 (expand the details section). Probably picked up the wrong compilers or something. I never understood why but renewed my whole toolchain relating to mingw and it disappeared.

Ah yes, that's the issue that explains the Azure failures.

I suspect the last one about Accelerate being detected is caused by https://numpy.org/devdocs/release/1.20.0-notes.html#negation-of-user-defined-blas-lapack-detection-order

I suspect the last one about Accelerate being detected is caused by https://numpy.org/devdocs/release/1.20.0-notes.html#negation-of-user-defined-blas-lapack-detection-order

We have two issues left:

The Fortran quadpack issue, which was failing with numpy 1.20.0 on both Appveyor and Azure. This is gh-13072
The Accelerate issue on macOS. For that there's only this issue open.

I'll have a look at the Accelerate issue. If anyone wants to dig into the Windows issue, that'd be very helpful.

We have two issues left:

The Fortran quadpack issue, which was failing with numpy 1.20.0 on both Appveyor and Azure. This is gh-13072
The Accelerate issue on macOS. For that there's only this issue open.

I'll have a look at the Accelerate issue. If anyone wants to dig into the Windows issue, that'd be very helpful.

And after that we should carefully reshuffle our CI jobs to better spread our testing against numpy master and pre-releases.

And after that we should carefully reshuffle our CI jobs to better spread our testing against numpy master and pre-releases.

I'm going to try to fire up windows and reproduce the issue, please anyone
else feel free to pick this up as well
On Sun, Jan 31, 2021, 13:14 Ralf Gommers<EMAIL_ADDRESS>wrote:

And after that we should carefully reshuffle our CI jobs to better spread
our testing against numpy master and pre-releases.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/scipy/scipy/issues/13464#issuecomment-770443255, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AG3HG352MGRI54ONPN27LQDS4W2YRANCNFSM4W22RPEQ
.

I'm going to try to fire up windows and reproduce the issue, please anyone
else feel free to pick this up as well
On Sun, Jan 31, 2021, 13:14 Ralf Gommers<EMAIL_ADDRESS>wrote:

And after that we should carefully reshuffle our CI jobs to better spread
our testing against numpy master and pre-releases.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/scipy/scipy/issues/13464#issuecomment-770443255, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AG3HG352MGRI54ONPN27LQDS4W2YRANCNFSM4W22RPEQ
.

Okay, getting somewhere:

NumPy removed Accelerate support in May '20 in https://github.com/numpy/numpy/pull/15759, but that only landed in 1.20.0.
SciPy has had a patched system_info.py in scipy/_build_utils/ since 2018. It is a copy of the NumPy version of blas_opt_info and lapack_opt_info with sections for Accelerate removed.
SciPy is careful to only import system_info from scipy._build_utils.
However, when we merged experimental Pythran support, that relied on another piece of build machinery, PythranExtension
We did not have a CI config using Pythran on macOS and testing against NumPy master or pre-releases.
The actual problem is at https://github.com/serge-sans-paille/pythran/blob/master/pythran/config.py#L220, where a numpy_sys.get_info(user_blas) gets called. We need to get that to use our patched version in scipy._build_utils somehow, or bypass it.

There is no way to pass user_blas = 'none' to PythranExtension. The make_extension function only looks in its own .cfg file and if that doesn't exist it always tries to use the BLAS that NumPy detects. That call is not only the wrong one for us, it also fails to pick up on the site.cfg we create in CI: https://github.com/scipy/scipy/blob/master/.github/workflows/macos.yml#L64
I'm glad it's not a new bug in numpy.distutils though, this should be less painful to deal with. Although it involves build machinery spread over five projects (distutils, setuptools, numpy.distutils, pythran.dist and scipy._build_utils) - it's a little horrifying.
My plan for now is:

undo the CI pin to numpy 1.19.5, disable Pythran in macOS CI
change the title of this issue to just be about the Accelerate issue, mark it for 1.7.0
find a decent solution after that.

@serge-sans-paille FYI.

Okay, getting somewhere:

NumPy removed Accelerate support in May '20 in https://github.com/numpy/numpy/pull/15759, but that only landed in 1.20.0.
SciPy has had a patched system_info.py in scipy/_build_utils/ since 2018. It is a copy of the NumPy version of blas_opt_info and lapack_opt_info with sections for Accelerate removed.
SciPy is careful to only import system_info from scipy._build_utils.
However, when we merged experimental Pythran support, that relied on another piece of build machinery, PythranExtension
We did not have a CI config using Pythran on macOS and testing against NumPy master or pre-releases.
The actual problem is at https://github.com/serge-sans-paille/pythran/blob/master/pythran/config.py#L220, where a numpy_sys.get_info(user_blas) gets called. We need to get that to use our patched version in scipy._build_utils somehow, or bypass it.

There is no way to pass user_blas = 'none' to PythranExtension. The make_extension function only looks in its own .cfg file and if that doesn't exist it always tries to use the BLAS that NumPy detects. That call is not only the wrong one for us, it also fails to pick up on the site.cfg we create in CI: https://github.com/scipy/scipy/blob/master/.github/workflows/macos.yml#L64
I'm glad it's not a new bug in numpy.distutils though, this should be less painful to deal with. Although it involves build machinery spread over five projects (distutils, setuptools, numpy.distutils, pythran.dist and scipy._build_utils) - it's a little horrifying.
My plan for now is:

undo the CI pin to numpy 1.19.5, disable Pythran in macOS CI
change the title of this issue to just be about the Accelerate issue, mark it for 1.7.0
find a decent solution after that.

@serge-sans-paille FYI.

Wow sweet surgery right there Ralf!

It's all fixed now except for the Windows issue, which is gh-13072. So I'll close this.

Thanks everyone!

If anyone else comes looking here because of the "No module named scipy" error in Azure, you're likely in the wrong place. The issue is simply that here:

The build job didn't actually succeed. There's an error propagation missing (Cc @tylerjereddy it may be obvious to you how to fix that).
To see the actual issue, you have to scroll up in the build job till:
  Building wheel for scipy (PEP 517): finished with status 'error'
  ERROR: Failed building wheel for scipy

the build failure will precede that. The Azure setup is wrong, so that is followed by a ton of:
Failed to build scipy
ERROR: Failed to build one or more wheels
Exception information:
Traceback (most recent call last):
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pip\_internal\cli\base_command.py", line 189, in _main
    status = self.run(options, args)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pip\_internal\cli\req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pip\_internal\commands\wheel.py", line 191, in run
    "Failed to build one or more wheels"
pip._internal.exceptions.CommandError: Failed to build one or more wheels
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0
  Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2
.... <many more "Found link" lines ...


Add VM Tags from Config Script

Is there a way to add the tags when adding a VM to an environment? I know that you can do it through the interactive prompts from the command line, but I want to script an unattended install just like you can do with a deployment groups and include the tags as an argument to the script.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 91d0d31f-81ee-c024-db7e-daddbf525f71
Version Independent ID: 330f1649-386c-d0aa-5f96-b8343a1480d3
Content: Environment - Virtual machine resource - Azure Pipelines
Content Source: docs/pipelines/process/environments-virtual-machines.md
Product: devops
Technology: devops-cicd-process
GitHub Login: @juliakm
Microsoft Alias: jukullam

@isaiah778 -- thank you for your question.  You may find answers here:

Azure DevOps Support Bot
Azure DevOps on Stack Overflow

@juliakm -- please look into this issue.

Really curious for this one as well, any updates ?

Any update on this? Still seems to be something you can't do.

figured it out, you can use --addvirtualmachineresourcetags --virtualmachineresourcetags "<tag>"


Inconsistent Google Calendar behavior - calendar ID

The API inconsistently returns the calendar ID. Sometimes it's the calendar name, other times it's the calendar ID. Pretty maddening and I'm not really sure if it's an implementation issue in the gem or a Google API issue. This makes it impossible to consistently store a calendar ID and retrieve information about it. Any ideas? Here's some sample code:
service = client.discovered_api('calendar', 'v3')
result = client.execute(:api_method => service.calendar_list.list)

@calendars = result.data

<%=form_tag "/calendars/save" do %>
            <% @calendars.items.each do |calendar| %>
                <%=label_tag do %>
                    <%=check_box_tag 'calendar_ids[]', calendar["id"], false %>
                    <%= calendar["summary"] %>
                <% end %>
                <br>
            <% end %>
            <%=submit_tag %>
        <% end %>

Haha ok, close my issue without looking at it...it's irrelevant anyway at this point I suppose!

It's based on an old version of the client and I'm not able to reproduce it. If you can reproduce with 0.9.x, please re-open and make sure that https://developers.google.com/apis-explorer/#search/calendar/m/calendar/v3/calendar.calendarList.list works as expected for your account.


Fetch all pipeline descriptors

As a Pipeline Runner I want to be able to fetch all the pipeline descriptors which exists in the pipeline's Repository, so I can know which pipeline descriptors are supported.

TID 340

Fetch all pipeline descriptorsTest
File: tid_310.js
describe : "pipeline actions"

Test
description

TID-340- get all pipeline (git 70 41)
validate  Fetch all pipeline


setup.py improvements

Description

check python version before installing
use context manager for opening files
read version from __init__.py
add an option to install hiredis as an extra option and promote this in documentation (this will allow controlling supported hiredis versions)
update old extension building code

Thanks a lot


Show NFTs by most emoted

Expand sort by most emoted NFTs
Test on

gallery feed
profile
collection detail
collections

kodadot/ideas#7 could be added to Flip as well

Status:

✅ gallery feed
✅ profile
✅ collection detail
❌ collections (CollectionList, collectionSortOption sortBy CollectionEntityOrderByInput need to be replaced by NFTEntityOrderByInput)

Emotes are not longer present on AHP/AHK
Happy to respin on EVM

https://github.com/kodadot/gibanica/issues/3


Install failed: Value does not fall within the expected range.

Describe the bug
After renewing the certificate, everything goes to plan, but the installation of the certificate in the IIS server fails with the following message:
"Install failed: Value does not fall within the expected range."
To Reproduce

open win-acme
go to manage renewals
run the renewal.

Expected behavior
I expected for the renewal to succeed.
Log
2020-04-21 13:24:45.904 +02:00 [INF] Installing with IIS... 2020-04-21 13:24:45.928 +02:00 [ERR] Error installing System.ArgumentException: Value does not fall within the expected range.    at Microsoft.Web.Administration.Interop.IAppHostProperty.get_Value()    at Microsoft.Web.Administration.ConfigurationElement.GetPropertyValue(IAppHostProperty property)    at Microsoft.Web.Administration.ConfigurationElement.GetAttributeValue(String attributeName)    at Microsoft.Web.Administration.Binding.GetBindingAttributeValue(String attributeName)    at Microsoft.Web.Administration.Binding.get_CertificateHash()    at PKISharp.WACS.Clients.IIS.IISBindingWrapper.get_CertificateHash()    at PKISharp.WACS.Clients.IIS.IISHttpBindingUpdater2.<>c__DisplayClass3_0.b__3(ValueTuple2 sb)    at System.Linq.Enumerable.WhereListIterator1.ToList()
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)    at PKISharp.WACS.Clients.IIS.IISHttpBindingUpdater2.AddOrUpdateBindings(IEnumerable1 identifiers, BindingOptions bindingOptions, Byte[] oldThumbprint) 2020-04-21 13:24:45.929 +02:00 [ERR] (ArgumentException) Unable to install certificate: Value does not fall within the expected range.
Platform:

Win 10 Pro x64
Version <IP_ADDRESS>3 (RELEASE, PLUGGABLE)

Additional context
It worked great before, and nothing was changed in the meantime.

It looks like something funky is going on with your current binding, you should check in IIS if there is a certificate linked to it.

Correct!
One of my bindings was messed up.
Maybe in future add a message that informs the user that he/she should check the bindings for the websites if auto install fails?

This is reasonably common with Certify as well, the trick is to catch the exception when Binding.CertificateHash read fails and treat it as a null, as whatever it's bound to isn't a valid cert anymore (so the hash value is no longer important).

Agreed, this will be fixed in the next release!

Hi,
We've recently started experiencing the same issue on a Server 2016 machine that was previously working fine. We've tried using version 2.1.7 and 2.1.8 and both produce the error below.
I can see in the stack trace it's failing when calling the setter on the CertificateStoreName property so tried running WACS with the --certificatestore parameter and manually setting it to 'Personal' to no avail.
No clarify, our certificates generate correctly however the IIS bindings do not update/create, this is the case for both renewals and fresh certs.
Happy to provide any additional logs that might help.
[DBUG] Scanning IIS site bindings for hosts
[VERB] 149 named bindings found in IIS
[DBUG] Filtering by site(s) [25]
[VERB] 1 bindings remaining after site filter
[VERB] No host filter applied
[VERB] 1 matching binding found
[DBUG] Scanning IIS sites
[VERB] Targeted convert into 1 order(s)
[VERB] Handle order 1/1: Main
[DBUG] Refreshing order...
[VERB] Loading ACME account signer...
[DBUG] Loading signer from C:\ProgramData\win-acme\acme-v02.api.letsencrypt.org\Signer_v2
[VERB] Constructing ACME protocol client...
[DBUG] Send GET request to https://acme-v02.api.letsencrypt.org/directory
[VERB] Request completed with status OK
[DBUG] Send HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce
[VERB] Request completed with status OK
[DBUG] Loading account information from C:\ProgramData\win-acme\acme-v02.api.letsencrypt.org\Registration_v2
[DBUG] Send POST request to https://acme-v02.api.letsencrypt.org/acme/order/1111111111/111111111
[VERB] Request completed with status OK
[WARN] Using cached order. To force issue of a new certificate within 1 days, run with the --force switch. Be ware that you might run into rate limits doing so.
[VERB] Handle authorization 1/1
[DBUG] Send POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/1111111111
[VERB] Request completed with status OK
[INFO] Cached authorization result for my-domain.com: valid
[WARN] Using cached certificate for [IIS] My Domain, (any host). To force a new request of the certificate within 1 days, run with the --force switch.
[DBUG] Certificate store: My
[INFO] Store with CertificateStore...
[INFO] Installing certificate in the certificate store
[DBUG] Opened certificate store My
[INFO] Adding certificate [IIS] My Domain, (any host) @ 2020/6/2 19:04:53 to store My
[VERB] CN=my-domain.com - CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US (7E8D847F22770860E3001ACF0A12E449E69678CD)
[DBUG] Closing certificate store
[VERB] CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US - CN=DST Root CA X3, O=Digital Signature Trust Co. (E6A3B45B062D509B3382282D196EFE97D5956CCB) to store CA
[DBUG] Closing store CA
[INFO] Installing with IIS...
[INFO] Adding new https binding *:443:my-domain.com
[EROR] Error creating binding my-domain.com: Value does not fall within the expected range.
System.ArgumentException: Value does not fall within the expected range.
at Microsoft.Web.Administration.Interop.IAppHostProperty.get_Value()
at Microsoft.Web.Administration.ConfigurationElement.GetPropertyValue(IAppHostProperty property)
at Microsoft.Web.Administration.ConfigurationElement.GetAttributeValue(String attributeName)
at Microsoft.Web.Administration.Binding.GetBindingAttributeValue(String attributeName)
at Microsoft.Web.Administration.Binding.get_CertificateHash()
at Microsoft.Web.Administration.Binding.set_CertificateStoreName(String value)
at PKISharp.WACS.Clients.IIS.IISClient.AddBinding(IISSiteWrapper site, BindingOptions options)
at PKISharp.WACS.Clients.IIS.IISHttpBindingUpdater2.AddBinding(TSite site, BindingOptions options)    at PKISharp.WACS.Clients.IIS.IISHttpBindingUpdater2.AddOrUpdateBindings(TBinding[] allBindings, TSite site, BindingOptions bindingOptions)
at PKISharp.WACS.Clients.IIS.IISHttpBindingUpdater2.AddOrUpdateBindings(IEnumerable1 identifiers, BindingOptions bindingOptions, Byte[] oldThumbprint)
[WARN] No bindings have been changed
[INFO] Scheduled task looks healthy
[INFO] Next renewal scheduled at 2020/7/27 19:14:38
[INFO] Certificate [IIS] My Domain, (any host) created

What happens when you try to create the binding manually?

When we try creating the binding manually via the IIS GUI, selecting the certificate that was generated as part of the failed automatic binding, it adds it correctly.

I just read through the code of Microsoft.Web.Administration, which turned out to be pretty horrifying. But I think I've found a way to make this more reliable, by assigning values to the properties in a different order. Would you mind to test this build?
https://ci.appveyor.com/project/WouterTinus/win-acme-s8t9q/builds/33276174/artifacts

@WouterTinus Thanks a lot for that, I've just tried the 'win-acme.v<IP_ADDRESS>3.x64.pluggable.zip' version on our server and it's worked perfectly, the bindings are now being correctly assigned to the container again!

Released in <IP_ADDRESS>


output holomap='scrubber' looping bug

%%output holomap='scrubber'
def show_a_holomap():
    def ith():
        x_vals = np.linspace(0,1,10)
        h1 = hv.Curve( (x_vals,np.random.uniform(-2,2,size=10)))
        h2 = hv.Curve( (x_vals,np.random.uniform(-2,2,size=10)))
        # must wrap in an overlay before adding to a holomap
        return hv.Overlay([h1,h2])

    # precompute 5 sets of curves
    return hv.HoloMap({ i: ith() for i in range(5)})
show_a_holomap()

loops backwards, but does not loop forwards

I think I noticed this issue the last time I used scrubber (which is a while back!). Thanks for filing the issue.

Not a high priority but assigning to 1.8.1 in case it is an easy fix.

The scrubber should now be fixed. Please reopen if you encounter any more issues.


Disable context propagation on kprobes

The black-box context propagation seems to create wrong traces on NodeJS. I'm going to disable this for now until we figure out a safe way to re-enabled it, perhaps selectively, perhaps we'll find a better way to match server and client calls.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (06a0246) 79.19% compared to head (d2fc918) 45.11%.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #556       +/-   ##
===========================================
- Coverage   79.19%   45.11%   -34.08%
===========================================
  Files          69       67        -2
  Lines        5830     5652      -178
===========================================
- Hits         4617     2550     -2067
- Misses        990     2952     +1962
+ Partials      223      150       -73

Flag
Coverage Δ

integration-test
?

unittests
45.11% <ø> (+0.17%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Dr/styling

add loading of fonts
add svg loader
load custom theme json file colors
add user flows for password reset and other pages (logic still needs to be implement)
implement account page and updating user profile
add styling to various existing pages.

Btw I added the swift file so my xcode would build -- not sure if will break yours. I can clean it up a little if necessary.

Btw I added the swift file so my xcode would build -- not sure if will break yours. I can clean it up a little if necessary.


Camera's incorrect fade zone when static tile layer is in front

Version

Phaser Version:

<= 3.21

Operating system:

Linux
Description

If there exist 2 tilemap layers and static is in front of dynamic one (setDepth), run camera's fade effect - fade zone is static and does not move when camera moves. In the example below (a modified version of https://labs.phaser.io/view.html?src=src\game%20objects\tilemap\collision\tile%20callbacks.js) - I've added some camera shift to the right to see the bug better.
Use arrows to collect coins - fade effect will run and you can see that it's position is not where camera's is. If you continue to move to the right while collecting coins - fade zone won't move with camera and you'll see fade zone's right border in the middle of the map.
If you set both layers depth to 10 or static is below a dynamic one - no issue.
Happens with WEBGL only.
Expected result - fade zone follows camera and takes all visible tilemap.
Example Test Code

var config = {
    type: Phaser.WEBGL,
    // type: Phaser.CANVAS, // <- This forks fine
    width:  800,
    height: 576,
    backgroundColor: '#555',
    physics: {
        default: 'arcade'
    },
    scene: {
        preload: preload,
        create:  create,
        update:  update
    }
};

var game = new Phaser.Game(config);
var map;
var cursors;
var debugGraphics;
var text;
var player;
var showDebug = false;
var groundLayer;
var coinLayer;
var coinsCollected = 0;

function preload ()
{
    this.load.image('ground_1x1',     'assets/tilemaps/tiles/ground_1x1.png');
    this.load.spritesheet('coin',     'assets/sprites/coin.png', { frameWidth: 32, frameHeight: 32 });
    this.load.tilemapTiledJSON('map', 'assets/tilemaps/maps/tile-collision-test.json');
    this.load.image('player',         'assets/sprites/phaser-dude.png');
}

function create ()
{
    map = this.make.tilemap({ key: 'map' });
    var groundTiles = map.addTilesetImage('ground_1x1');
    var coinTiles = map.addTilesetImage('coin');

    // Static layer above dynamic here
    groundLayer = map.createStaticLayer('Ground Layer', groundTiles, 0, 0).setDepth(10);
    coinLayer =   map.createDynamicLayer('Coin Layer',  coinTiles,   0, 0).setDepth(0);

    groundLayer.setCollisionBetween(1, 25);
    coinLayer.setTileIndexCallback(26, hitCoin, this);

    player = this.physics.add.sprite(80, 70, 'player');

    this.physics.add.collider(player, groundLayer);
    this.physics.add.overlap(player, coinLayer);

    this.cameras.main.setBounds(0, 0, map.widthInPixels, map.heightInPixels);
    this.cameras.main.startFollow(player);
    // Small shift to see bug better
    this.cameras.main.x += 200;

    cursors = this.input.keyboard.createCursorKeys();
}

function hitCoin (sprite, tile)
{
    coinLayer.removeTileAt(tile.x, tile.y);

    this.cameras.main.fadeOut(250);

    setTimeout(() => {
        this.cameras.main.fadeIn(250);
    }, 500);

    return false;
}

function update (time, delta)
{
    player.body.setVelocity(0);

    if (cursors.left.isDown)
        player.body.setVelocityX(-200);
    else if (cursors.right.isDown)
        player.body.setVelocityX(200);

    if (cursors.up.isDown)
        player.body.setVelocityY(-200);
    else if (cursors.down.isDown)
        player.body.setVelocityY(200);
}

Im also getting this issue only in WebGL. The cameras overlay looks to be stuck at 0,0 of the level and not attached to the camera. Did you resolve the issue?

@c-andrews , nope. Still waiting for a fix or a workaround advice

@inmylo, I've not been able to fix it but I've just added a Graphics object and set its depth to the top and it seems to be working. Not the best solution but it works for me.
this._overlay = this.add.graphics(); this._overlay.fillStyle(0x000000, 0); this._overlay.fillRect(0, 0, this.game.scale.width, this.game.scale.height); this._overlay.setScrollFactor(0); this._overlay.setDepth(OVERLAY_DEPTH);

I can confirm this was fixed in the 3.23 release. Sorry for not updating this issue with that news.


Coveralls badge not being updated

It seems that the coveralls badge is not reflecting current coverage. Can you please take a look at it?

It is working... 🤣


Feature: Support filetype related to frontend development

Context
Some filetypes are not supported by hlchunk.nvim
Screenshot

filetype
before
after

*.tsx

*.jsx

*.html

okay, thanks for your contribution


Flex frag + retries

Resolves #108 and #76
/cc @Shaggs and @112bzo as FLEX users
This moves the FLEX fragmented messages into an object with the address as key - fragmented messages that arrive with other messages in between should now properly join. The only real edge case I can think of is if in poor reception and you receive an F message, but never its corresponding C message, then you might one day receive a randomly joined message. Would be rare enough not to worry about, IMO.
The other change in this is the retries that @s3m1s0n1c requested - messages will retry sending on failure up to 10 times, with the time between retries increasing exponentially (Math.pow(2, retries)). This isn't the best way to handle this, especially for machines that will be offline for some time. The alternative would be to use a proper queuing system like Redis, but that would greatly increase the barrier to entry and reduce compatibility/portability. Probably something to look at if there was ever a demand for a "production ready" pagermon.

@davidmckenzie I'll test tonight.  In regards to reception even doing this in python or c++ has the same potential. Nothing is perfect in life minor issues like that can't be helped.
Our old FELX system didn't have this issue but it's all about streamlining and getting pages out faster.
Thanks for all your time and effort


docs(config): Replace UglifyJSPlugin references with TerserPlugin

Since as of webpack 5, the default options.minimizer is now terser-webpack-plugin:
https://github.com/webpack/webpack/pull/8036
I've also:

updated the custom minimizer example to include all of the default options, to reduce the chance people forget to enable them.
removed the optimize/UglifyJsPlugin reference, since it's no longer included in the webpack repository.
removed the "Here are a few more popular ones" mention, since it's not accurate (they aren't more popular).
removed manual plugin configuration from configuration-languages.md since the defaults for minimizer make it unnecessary.

Fixes #2520.

Thanks!

Thank you for the reviews/merge :-)


fix(clustering): add missing aws-lambda plugin new option in removed fields

Summary
This PR adds the missing new config field disable_https in the aws-lambda plugin, to the removed field so that clustering compat can work normally.

Related feature PR: https://github.com/Kong/kong/pull/9799
Checklist

[ ] The Pull Request has tests
[ ] There's an entry in the CHANGELOG
[ ] There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

[Implement ...]

Issue reference

Fix #[issue number]

Could you please link the feature PR and add some tests?

@vm-001 Feature PR linked. I think we do not have removed_field test(old tests were deleted for a long time), so I'm going to ignore that for now :)


RHTAPWATCH-244: Configure ClusterLogForwarder to Splunk for Production env

This commit contains configuration for forwarding OCP application, infrastructure and audit logs to the internal RH Splunk. It also refactors some staging related configuration so it can be used more efficiently by Kustomize, so the staging environment might be affected as well by this change.

Hey @ifireball @Omeramsc @amisstea
Can you please review this PR?
It will enable us to also retain production logs in Splunk.

/retest

/lgtm

/lgtm

/retest

/retest

/retest

/retest-required

/retest

/retest

/retest

/test appstudio-e2e-tests


Terminal tasks cannot be processed by problemMatcher

Build tasks are using the builtin terminal, but he problemMatcher is not able to cope with terminal width.
See for instance the following terminal output:

Because the error has more than (terminal width) characters, the problem matcher is only able to process the first line and not the others, resulting in the following cutout error in Problems:

It's not practical to have compiler error messages limited by the Terminal Window Width. It's even more problematic since unless you navigate to the Terminal tab it will keep its original 80 width, resulting in even more cutout error messages.

Which problem matcher are you using? Is this an issue with the Haxe extension?

Yes this is the problem matcher used:
{
	"name": "haxe",
	"regexp": "^(.+):(\\d+): (?:lines \\d+-(\\d+)|character(?:s (\\d+)-| )(\\d+)) : (?:(Warning) : )?(.*)$",
	"file": 1,
	"line": 2,
	"endLine": 3,
	"column": 4,
	"endColumn": 5,
	"severity": 6,
	"message": 7
}

(you can see the whole package definition here https://github.com/vshaxe/vshaxe/blob/master/package.json)
The problem is that tasks are using a terminal with a specific width, so errors get newlines inserted (at least on Windows) when the max width is reached.

Please note that the problem is not specific to Haxe extension. All problemMatchers should be affected the same.

Actually this is a dup of https://github.com/Microsoft/vscode/issues/43664. There are only some cases where some of the info on the next line is not covered. For all the details see 43664.


Show line number on column errors

When an invalid column is specified, show the requested line number as well.

We do that now, the change will be available in the upcoming<EMAIL_ADDRESS>It was added in https://github.com/steelbrain/atom-linter/pull/145


UnboundLocalError: local variable 'experiment_data' referenced before assignment

Hello,
When i evaluate the model, i'm facing this issue. Can you please help me  to resolve this issue??
(decathlon) sivajyothi@cougar-U:~/unet/3D$ python evaluate_model.py --data_path /home/sivajyothi/data/Task01_BrainTumour/ --saved_model /home/sivajyothi/unet/3D/saved_model/3d_unet_decathlon.hdf5
Using TensorFlow backend.
Started script on 2019-11-11 00:26:31.874186
OMP: Warning #181: OMP_PROC_BIND: ignored because KMP_AFFINITY has been defined
Loading images and masks from test set
File /home/sivajyothi/data/Task01_BrainTumour/dataset.json doesn't exist. It should be part of the Decathlon directory
Traceback (most recent call last):
File "evaluate_model.py", line 77, in
**validation_data_params)
File "/home/sivajyothi/unet/3D/dataloader.py", line 83, in init
self.list_IDs = self.create_file_list()
File "/home/sivajyothi/unet/3D/dataloader.py", line 146, in create_file_list
self.output_channels = experiment_data["labels"]
UnboundLocalError: local variable 'experiment_data' referenced before assignment
Thank you.

In case someone else still faces this issue, it is because of the wrong path to --data_path


Missing United Arab Emirates

Can this add, for example, Dubai and Abu Dhabi?

Yes, they can be added. Please open a PR.


Change 'externals' and 'global' condition in magic-sdk > CDN build

Fixes an issue where OAuthExtension was incompatible with Magic SDK via CDN.

:rocket: PR was released in<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>:rocket:


refactor: [migration] convert iframe chart into dashboard markdown component

SUMMARY
Dashboard v2 support native markdown component, and user can also create iframe inside markdown. So that it's not necessary to have iframe/markup/separator viz_type in slices (or add them into dashboard).
This PR will do:

Find iframe type chart that used in dashboard.
Convert iframe slices into dashboard Markdown component. Slices in markup and separator viz_type already converted into dashboard component when dashboard v2 rollout.
Delete iframe/markup/separator slices, and remove viz_type support (python and js)

TEST PLAN
Before migration, create a iframe chart and add into dashboard.
After migration, check the dashboard, iframe chart should be converted into Markdown component with a iframe.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[x] Requires DB Migration.
[x] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

@mistercrunch @etr2460 @john-bodley @serenajiang

Codecov Report

Merging #10590 into master will decrease coverage by 5.00%.
The diff coverage is 1.21%.

@@            Coverage Diff             @@
##           master   #10590      +/-   ##
==========================================
- Coverage   64.37%   59.36%   -5.01%
==========================================
  Files         775      360     -415
  Lines       36555    23024   -13531
  Branches     3459        0    -3459
==========================================
- Hits        23531    13668    -9863
+ Misses      12912     9356    -3556
+ Partials      112        0     -112

Flag
Coverage Δ

#cypress
?

#javascript
?

#python
59.36% <1.21%> (-0.58%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ns/978245563a02_migrate_iframe_to_dash_markdown.py
0.00% <0.00%> (ø)

superset/migrations/versions/f80a3b88324b_.py
0.00% <0.00%> (ø)

superset/viz.py
57.55% <ø> (-0.10%)
:arrow_down:

superset/tasks/slack_util.py
89.47% <100.00%> (ø)

superset/db_engine_specs/mysql.py
79.16% <0.00%> (-12.50%)
:arrow_down:

superset/db_engine_specs/presto.py
70.56% <0.00%> (-12.14%)
:arrow_down:

superset/db_engine_specs/sqlite.py
65.62% <0.00%> (-9.38%)
:arrow_down:

superset/examples/world_bank.py
97.10% <0.00%> (-2.90%)
:arrow_down:

superset/examples/birth_names.py
97.36% <0.00%> (-2.64%)
:arrow_down:

superset/views/database/mixins.py
80.70% <0.00%> (-1.76%)
:arrow_down:

... and 423 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acb00f5...9de1d87. Read the comment docs.

@graceguo-supercat
I am attempting to create a KPI card as shown above. In version 0.36 I could do this by using the markdown chart in HTML mode  where I would use 3 iFrames. Each iFrame would point to a big_number_total chart
With 0.37 this is no long possible as I understand.
ERROR: Separator
ERROR:superset.views.base:'separator'
Traceback (most recent call last):
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/base.py", line 173, in wraps
    return f(self, *args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/utils/decorators.py", line 67, in wrapper
    check_perms(*args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 426, in check_datasource_perms
    force=False,
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 111, in get_viz
    viz_obj = viz.viz_types[viz_type](datasource, form_data=form_data, force=force)
KeyError: 'separator'

ERROR: markup
KeyError: 'markup'
ERROR:superset.views.base:'markup'
Traceback (most recent call last):
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/base.py", line 173, in wraps
    return f(self, *args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/utils/decorators.py", line 67, in wrapper
    check_perms(*args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 426, in check_datasource_perms
    force=False,
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 111, in get_viz
    viz_obj = viz.viz_types[viz_type](datasource, form_data=form_data, force=force)

@stevensuting Superset will not support markup and separator charts any more. you should use Dashboard Markdown components.


system: replace log_error() with log_msg()

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Is your feature request related to a problem? Please describe.
Since we support logging levels log_error() is bad because it logs error even for diagnostics and such.
Describe the solution you like
It would be best to avoid log_error() in core code altogether and leave it available for a bit of compatibility before scrubbing plugins as well. I don't think we want to end up with a mix of log_error() and log_msg() so the real errors should also end up as log_msg($msg, LOG_ERR).
core.git # git grep '[^a-z]log_error(' | wc -l
      1
plugins.git # git grep '[^a-z]log_error(' | wc -l
     139

Describe alternatives you considered
Duality, leading people to use either or just copying suboptimal code leading to more cleanups.
Additional context
#5498 #5575

I'm leaving most of plugins.git as is for the time being. There is not much to gain from it: emulated log_error() MVC variant, dyndns, rfc2136 as most common users.


add mips and mipsel dependencies

#8

Thanks for the PR @psych0d0g. The Windows build seems to be broken. I'll check why and get back to you shortly.

Looking good.
Would you know of a way to add automated builds for the mips and mipsel platforms ?

I will push a new version of the library to crates.io in a few seconds so you can install it via cargo.

i dont know how travisci handles mips and mipsel builds but for rust you would need the stdlib for mips and mipsel from http://static.rust-lang.org/dist/rust-std-nightly-mipsel-unknown-linux-gnu.tar.gz and http://static.rust-lang.org/dist/rust-std-nightly-mips-unknown-linux-gnu.tar.gz.
also you would need the gcc crosscompiler toolchains for those 2 architectures

Alright, thanks for the explanation. I'll try to set this up someday if I find the time.


Add commands to run playground without opening Android Studio

The same way react-native run-android works.

We can use appFolder and variant.


Rename the test file into e2e-upgrade-tests.sh for upgrade

Issue to be fixed
Fixes #94
Proposed Changes

We rename this script, since we would like to use it as a base to do upgrade tests.
Customize the function integration_tests to make sure the integration tests only run the script e2e-tests.sh.

Release Note

/lgtm