Native HA is unavailable - AMQ5708E - Error 93 for v7.0.1

Dear all,
using the helm chart v7.0.1 we are getting the following error:
2023-07-19T10:47:17.950Z Using queue manager name: P01M01G
2023-07-19T10:47:17.950Z CPU architecture: amd64
2023-07-19T10:47:17.950Z Linux kernel version: 5.15.0-1035-azure
2023-07-19T10:47:17.950Z Base image: Red Hat Enterprise Linux 8.8 (Ootpa)
2023-07-19T10:47:17.950Z Running as user ID 1001 with primary group 0, and supplementary groups 0,2000
2023-07-19T10:47:17.950Z Capabilities: none
2023-07-19T10:47:17.950Z seccomp enforcing mode: disabled
2023-07-19T10:47:17.950Z Process security attributes: cri-containerd.apparmor.d (enforce)
2023-07-19T10:47:17.951Z Detected 'ext4' volume mounted to /mnt/mqm-data
2023-07-19T10:47:17.951Z Detected 'ext4' volume mounted to /mnt/mqm-log
2023-07-19T10:47:17.951Z Detected 'ext4' volume mounted to /mnt/mqm
2023-07-19T10:47:17.958Z Created directory structure under /var/mqm
2023-07-19T10:47:17.958Z Image created: 2023-05-25T14:16:50+00:00
2023-07-19T10:47:17.958Z Image tag: ibm-mqadvanced-server:<IP_ADDRESS>-r3.20230525140420.7d14e60-amd64
2023-07-19T10:47:17.977Z MQ version: <IP_ADDRESS>
2023-07-19T10:47:17.977Z MQ level: p930-005-230413
2023-07-19T10:47:17.977Z MQ license: Production
2023-07-19T10:47:23.215Z Creating queue manager P01M01G
2023-07-19T10:47:23.241Z Error 93 creating queue manager: AMQ5708E: Native HA is unavailable.
2023-07-19T10:47:23.241Z /opt/mqm/bin/crtmqm: exit status 93
I just reverted back to v7.0.0. With this version the pods do come up.
Thank you,
Uwe

Hi Uwe - sorry to see that you are having difficulties. Was this during a helm upgrade or for a new deployment. Also could you provide the parameters you are using so I can reproduce locally.
Thanks
Callum

Understood - could you provide the parameters you are using so I can recreate. If they contain any sensitive data I'm happy to jump on a webex session and observe.

Hi Callum,
attached the charts, one qmgr chart that references the base chart … is this all you need?
Thank you,
Uwe
From: callumpjackson @.>
Sent: Wednesday, July 19, 2023 4:06 PM
To: ibm-messaging/mq-helm @.>
Cc: Schmiedel Uwe, FG-232 @.>; Author @.>
Subject: Re: [ibm-messaging/mq-helm] Native HA is unavailable - AMQ5708E - Error 93 for v7.0.1 (Issue #51)
Sent from outside the BMW organization - be CAUTIOUS, particularly with links and attachments.
Absender außerhalb der BMW Organisation - Bitte VORSICHT beim Öffnen von Links und Anhängen.

Understood - could you provide the parameters you are using so I can recreate. If they contain any sensitive data I'm happy to jump on a webex session and observe.
—
Reply to this email directly, view it on GitHubhttps://github.com/ibm-messaging/mq-helm/issues/51#issuecomment-1642155544, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A6UGXEI74HAKU7UT7DCNHMTXQ7SU7ANCNFSM6AAAAAA2PZJOR4.
You are receiving this because you authored the thread.Message ID: @.***>

Hi Uwe,
I’ve good news and bad news. I tried to reproduce the issue and tried the latest MQ CD release and this works as expected and originally tested.
2023-07-19T15:00:43.097Z CPU architecture: amd64
2023-07-19T15:00:43.097Z Linux kernel version: 4.18.0-372.49.1.el8_6.x86_64
2023-07-19T15:00:43.098Z Container runtime: kube
2023-07-19T15:00:43.098Z Base image: Red Hat Enterprise Linux 8.8 (Ootpa)
2023-07-19T15:00:43.098Z Running as user ID<PHONE_NUMBER> with primary group 0, and supplementary groups 0,1000680000
2023-07-19T15:00:43.098Z Capabilities: none
2023-07-19T15:00:43.098Z seccomp enforcing mode: filtering
2023-07-19T15:00:43.098Z Process security attributes: system_u:system_r:container_t:s0:c15,c26
2023-07-19T15:00:43.098Z Detected 'ext4' volume mounted to /mnt/mqm
2023-07-19T15:00:43.098Z Environment variable LOG_FORMAT is deprecated. Use MQ_LOGGING_CONSOLE_FORMAT instead.
2023-07-19T15:00:43.343Z Using queue manager name: secureapphelm
2023-07-19T15:00:43.494Z Created directory structure under /var/mqm
2023-07-19T15:00:43.494Z Image created: 2023-06-05T08:54:47+00:00
2023-07-19T15:00:43.494Z Image tag: ibm-mqadvanced-server-dev:<IP_ADDRESS>-r1.20230605085013.3599852-amd64
2023-07-19T15:00:43.541Z MQ version: <IP_ADDRESS>
2023-07-19T15:00:43.541Z MQ level: p933-L230531
2023-07-19T15:00:43.541Z MQ license: Developer
2023-07-19T15:00:46.628Z Starting web server
2023-07-19T15:00:49.622Z FIPS cryptography is not enabled.
2023-07-19T15:00:49.622Z Creating queue manager secureapphelm
2023-07-19T15:01:03.812Z Created queue manager
2023-07-19T15:01:03.913Z Removing existing ServiceComponent configuration
2023-07-19T15:00:51.610Z AMQ6287I: IBM MQ V<IP_ADDRESS> (p933-L230531). [CommentInsert1(Linux 4.18.0-372.49.1.el8_6.x86_64 (MQ Linux (x86-64 platform) 64-bit)), CommentInsert2(/opt/mqm (Installation1)), CommentInsert3(<IP_ADDRESS> (p933-L230531))]
2023-07-19T15:00:51.727Z AMQ7467I: The oldest log file required to start queue manager secureapphelm is S0000000.LOG. [CommentInsert1(secureapphelm), CommentInsert2(S0000000.LOG)]
2023-07-19T15:00:51.729Z AMQ7468I: The oldest log file required to perform media recovery of queue manager secureapphelm is S0000000.LOG. [CommentInsert1(secureapphelm), CommentInsert2(S0000000.LOG)]
2023-07-19T15:00:53.307Z AMQ7229I: 5 log records accessed on queue manager 'secureapphelm' during the log replay phase. [ArithInsert1(5), CommentInsert1(secureapphelm)]
2023-07-19T15:00:53.309Z AMQ7230I: Log replay for queue manager 'secureapphelm' complete. [ArithInsert1(5), CommentInsert1(secureapphelm)]

This made me review your original output and I realize that you are using <IP_ADDRESS>, the LTS instead of CD stream. When I change the container image to 9.3.0 I am able to reproduce:
2023-07-19T15:07:19.190Z CPU architecture: amd64
2023-07-19T15:07:19.191Z Linux kernel version: 4.18.0-372.49.1.el8_6.x86_64
2023-07-19T15:07:19.191Z Container runtime: kube
2023-07-19T15:07:19.191Z Base image: Red Hat Enterprise Linux 8.6 (Ootpa)
2023-07-19T15:07:19.191Z Running as user ID<PHONE_NUMBER> with primary group 0, and supplementary groups 0,1000680000
2023-07-19T15:07:19.191Z Capabilities: none
2023-07-19T15:07:19.191Z seccomp enforcing mode: filtering
2023-07-19T15:07:19.191Z Process security attributes: system_u:system_r:container_t:s0:c15,c26�
2023-07-19T15:07:19.191Z Detected 'ext4' volume mounted to /mnt/mqm
2023-07-19T15:07:19.346Z Using queue manager name: secureapphelm
2023-07-19T15:07:19.420Z Created directory structure under /var/mqm
2023-07-19T15:07:19.420Z Image created: 2022-06-15T13:12:21+00:00
2023-07-19T15:07:19.420Z Image tag: ibm-mqadvanced-server-dev:<IP_ADDRESS>-r1.20220615130653.3111d48-amd64
2023-07-19T15:07:19.517Z MQ version: <IP_ADDRESS>
2023-07-19T15:07:19.518Z MQ level: p930-L220606
2023-07-19T15:07:19.518Z MQ license: Developer
2023-07-19T15:07:23.026Z Starting web server
2023-07-19T15:07:26.207Z Creating queue manager secureapphelm
2023-07-19T15:07:26.507Z Error 93 creating queue manager: AMQ5708E: Native HA is unavailable.
2023-07-19T15:07:26.507Z /opt/mqm/bin/crtmqm: exit status 93

The reason for the difference is that we made a licensing change in 9.3.2 where Native HA is available within an MQ Advanced license. This change involved the removal of an environment variable called AMQ_CLOUD_PAK. The reason why we updated between 7.0.0 and 7.0.1 is that we considered it a bug, as it should have been within Helm Charts supporting 9.3.2 onwards.
The general strategy is that a single Helm Chart release corresponds to an IBM MQ release. Its fair to say that we have seen good backward and forward compatibility, so this hasn’t been a major issue to date.
In the case of the LTS this corresponds to MQ Helm Chart 3.0.0. Unless you need new features of the Helm Chart in the later releases, I would suggest it would be easier to remain on an earlier version than 7.0.1.

Hi Callum,
I understand, ok fine for me as long as I know it. So we will stay with 7.0.0 for now.
Thank you,
Uwe
From: callumpjackson @.>
Sent: Wednesday, July 19, 2023 6:32 PM
To: ibm-messaging/mq-helm @.>
Cc: Schmiedel Uwe, FG-232 @.>; Author @.>
Subject: Re: [ibm-messaging/mq-helm] Native HA is unavailable - AMQ5708E - Error 93 for v7.0.1 (Issue #51)
Sent from outside the BMW organization - be CAUTIOUS, particularly with links and attachments.
Absender außerhalb der BMW Organisation - Bitte VORSICHT beim Öffnen von Links und Anhängen.

Hi Uwe,
I’ve good news and bad news. I tried to reproduce the issue and tried the latest MQ CD release and this works as expected and originally tested.
2023-07-19T15:00:43.097Z CPU architecture: amd64
2023-07-19T15:00:43.097Z Linux kernel version: 4.18.0-372.49.1.el8_6.x86_64
2023-07-19T15:00:43.098Z Container runtime: kube
2023-07-19T15:00:43.098Z Base image: Red Hat Enterprise Linux 8.8 (Ootpa)
2023-07-19T15:00:43.098Z Running as user ID<PHONE_NUMBER> with primary group 0, and supplementary groups 0,1000680000
2023-07-19T15:00:43.098Z Capabilities: none
2023-07-19T15:00:43.098Z seccomp enforcing mode: filtering
2023-07-19T15:00:43.098Z Process security attributes: system_u:system_r:container_t:s0:c15,c26
2023-07-19T15:00:43.098Z Detected 'ext4' volume mounted to /mnt/mqm
2023-07-19T15:00:43.098Z Environment variable LOG_FORMAT is deprecated. Use MQ_LOGGING_CONSOLE_FORMAT instead.
2023-07-19T15:00:43.343Z Using queue manager name: secureapphelm
2023-07-19T15:00:43.494Z Created directory structure under /var/mqm
2023-07-19T15:00:43.494Z Image created: 2023-06-05T08:54:47+00:00
2023-07-19T15:00:43.494Z Image tag: ibm-mqadvanced-server-dev:<IP_ADDRESS>-r1.20230605085013.3599852-amd64
2023-07-19T15:00:43.541Z MQ version: <IP_ADDRESS>
2023-07-19T15:00:43.541Z MQ level: p933-L230531
2023-07-19T15:00:43.541Z MQ license: Developer
2023-07-19T15:00:46.628Z Starting web server
2023-07-19T15:00:49.622Z FIPS cryptography is not enabled.
2023-07-19T15:00:49.622Z Creating queue manager secureapphelm
2023-07-19T15:01:03.812Z Created queue manager
2023-07-19T15:01:03.913Z Removing existing ServiceComponent configuration
2023-07-19T15:00:51.610Z AMQ6287I: IBM MQ V<IP_ADDRESS> (p933-L230531). [CommentInsert1(Linux 4.18.0-372.49.1.el8_6.x86_64 (MQ Linux (x86-64 platform) 64-bit)), CommentInsert2(/opt/mqm (Installation1)), CommentInsert3(<IP_ADDRESS> (p933-L230531))]
2023-07-19T15:00:51.727Z AMQ7467I: The oldest log file required to start queue manager secureapphelm is S0000000.LOG. [CommentInsert1(secureapphelm), CommentInsert2(S0000000.LOG)]
2023-07-19T15:00:51.729Z AMQ7468I: The oldest log file required to perform media recovery of queue manager secureapphelm is S0000000.LOG. [CommentInsert1(secureapphelm), CommentInsert2(S0000000.LOG)]
2023-07-19T15:00:53.307Z AMQ7229I: 5 log records accessed on queue manager 'secureapphelm' during the log replay phase. [ArithInsert1(5), CommentInsert1(secureapphelm)]
2023-07-19T15:00:53.309Z AMQ7230I: Log replay for queue manager 'secureapphelm' complete. [ArithInsert1(5), CommentInsert1(secureapphelm)]
This made me review your original output and I realize that you are using <IP_ADDRESS>, the LTS instead of CD stream. When I change the container image to 9.3.0 I am able to reproduce:
2023-07-19T15:07:19.190Z CPU architecture: amd64
2023-07-19T15:07:19.191Z Linux kernel version: 4.18.0-372.49.1.el8_6.x86_64
2023-07-19T15:07:19.191Z Container runtime: kube
2023-07-19T15:07:19.191Z Base image: Red Hat Enterprise Linux 8.6 (Ootpa)
2023-07-19T15:07:19.191Z Running as user ID<PHONE_NUMBER> with primary group 0, and supplementary groups 0,1000680000
2023-07-19T15:07:19.191Z Capabilities: none
2023-07-19T15:07:19.191Z seccomp enforcing mode: filtering
2023-07-19T15:07:19.191Z Process security attributes: system_u:system_r:container_t:s0:c15,c26�
2023-07-19T15:07:19.191Z Detected 'ext4' volume mounted to /mnt/mqm
2023-07-19T15:07:19.346Z Using queue manager name: secureapphelm
2023-07-19T15:07:19.420Z Created directory structure under /var/mqm
2023-07-19T15:07:19.420Z Image created: 2022-06-15T13:12:21+00:00
2023-07-19T15:07:19.420Z Image tag: ibm-mqadvanced-server-dev:<IP_ADDRESS>-r1.20220615130653.3111d48-amd64
2023-07-19T15:07:19.517Z MQ version: <IP_ADDRESS>
2023-07-19T15:07:19.518Z MQ level: p930-L220606
2023-07-19T15:07:19.518Z MQ license: Developer
2023-07-19T15:07:23.026Z Starting web server
2023-07-19T15:07:26.207Z Creating queue manager secureapphelm
2023-07-19T15:07:26.507Z Error 93 creating queue manager: AMQ5708E: Native HA is unavailable.
2023-07-19T15:07:26.507Z /opt/mqm/bin/crtmqm: exit status 93
The reason for the difference is that we made a licensing change in 9.3.2 where Native HA is available within an MQ Advanced license. This change involved the removal of an environment variable called AMQ_CLOUD_PAK. The reason why we updated between 7.0.0 and 7.0.1 is that we considered it a bug, as it should have been within Helm Charts supporting 9.3.2 onwards.
The general strategy is that a single Helm Chart release corresponds to an IBM MQ release. Its fair to say that we have seen good backward and forward compatibility, so this hasn’t been a major issue to date.
In the case of the LTS this corresponds to MQ Helm Chart 3.0.0. Unless you need new features of the Helm Chart in the later releases, I would suggest it would be easier to remain on an earlier version than 7.0.1.
—
Reply to this email directly, view it on GitHubhttps://github.com/ibm-messaging/mq-helm/issues/51#issuecomment-1642403475, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A6UGXEPOWYYDRVPXID5EEU3XRADWXANCNFSM6AAAAAA2PZJOR4.
You are receiving this because you authored the thread.Message ID: @.***>

Thanks - closing issue.


CFE: optional super-initializer parameters inherit default values incorrectly

This is a part of https://github.com/dart-lang/sdk/issues/47525.
co19/LanguageFeatures/Super-parameters/semantics_A06_t10 test fails with RuntimeError on the VM because default values of super-parameters are incorrect.
class S {
  int s1;
  int s2;
  S(this.s1, [this.s2 = 42]);
}

class C extends S {
  int i1;
  int i2;
  C(this.i1, super.s1, int x, [super.s2]) : this.i2 = x;
}

Kernel:
  class S extends core::Object {
    field core::int s1;
    field core::int s2;
    constructor •(core::int s1, [core::int s2 = #C1]) → sem::S
      : sem::S::s1 = s1, sem::S::s2 = s2, super core::Object::•()
      ;
  }
  class C extends sem::S {
    field core::int i1;
    field core::int i2;
    constructor •(core::int i1 = #C1, core::int s1, core::int x, [core::int s2 = #C2]) → sem::C
      : sem::C::i1 = i1, sem::C::i2 = x, super sem::S::•(s1, s2)
      ;
  }

  #C1 = 42
  #C2 = null

Here s2 parameter in C should have a default value #C1 (42), but it has a default value #C2 (null). Parameter i1 is not optional, so it shouldn't have a default value at all.
/cc @johnniwinther @chloestefantsova

co19/LanguageFeatures/Super-parameters/summary_A04_t01 and co19/LanguageFeatures/Super-parameters/summary_A04_t02 also fail with RuntimeError due to this bug.

Thanks for spotting that! I'm working on a fix at https://dart-review.googlesource.com/c/sdk/+/228204.


forceCommit, and maybe segment commit in general, might pick an unhealthy/lagging winner

We were working through an incident today where we lost a number of realtime servers. When new servers came up, they were waiting up to 45 minutes to catch up before /health would return OK. Because they were taking up to 30 minutes to consume up to realtime, we decided to force commit consuming segments so they could just download the data from S3.
For the most part, this worked great. But in one case, the server that was behind remained behind, and the other 2 healthy servers went from being 10 seconds of lag to reporting they were equally behind.
 I'm confident we saw the lag spike after the force commit, so there is definitely an issue here. But I will spend some time validating the actual mechanism by which this happened tomorrow.

@mcvsubbu @sajjad-moradi @npawar @navina In the consuming segment commit protocol, do we pick the candidate server that has the highest consuming offset to commit? If so, does the same logic apply to the force commit scenario?

If a forced commit is in play, then we pick the first replica that the controller heard from, for a given segment. All replicas are given the opportunity to do so.
The idea to pick the first one is to NOT cause any additional delay (in terms of waiting for other replicas).
It may have happened that the one with largest lag was just too quick to respond :)

@mcvsubbu Thanks for the clarification! IMO we should still run the regular protocol to pick the server which consumes the most data. Force commit is just a trigger for the commit process. It might take longer to reach consensus, but will be more close to the regular commit behavior.

I believe we had a discussion on that front, and decided to add force commit  the same as the criteria where the completion is due to numrows consumed. I don't recollect the reason here, but @sajjad-moradi  may do so.

I think one of the use-cases that forcecommit was meant to solve the case where the ingestion is seems stuck. In such cases, force committing based on the first responsive replica rather than winning replica seems appropriate.
@jadami10 were you able to validate that forcecommit was the issue?

@jadami10 were you able to validate that forcecommit was the issue?

yes, after looking through the logs of the winners, the server that was lagging was chosen as the winner for 5 partitions.

I think one of the use-cases that forcecommit was meant to solve the case where the ingestion is seems stuck. In such cases, force committing based on the first responsive replica rather than winning replica seems appropriate.

Assuming my previous comment/reading of the code above is correct, when your ingestion is stuck and you call this, you will at most wait 3.3 seconds more seconds for a winner. In practice, it's likely minutes before you've noticed ingestion is stuck.
But in the case where 1 server is significantly behind (in our case 2 hours behind), accidentally picking it has pretty big ramifications. Now every server is 2 hours behind for those partitions.
There's also very few cases where the wait period is skipped (only row limit and end of partition reached). Even for time based or size based segment completion, the protocol still waits, and that doesn't seem to be a problem. I'll wait to hear from @sajjad-moradi, but I would be strongly in favor of undoing this part of the force commit feature.

I agree with @jadami10 that there is a timeout of 3.3s  to pick the winner with the best offset, so it may be useful to wait out that time.
I just don't recall the full discussion, so I will let @sajjad-moradi  chime in here (he will be back Monday)

The stopReason seems to be REASON_FORCE_COMMIT_MESSAGE_RECEIVED, so it is identifiable. I wonder if the issue is this line,
https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java#L1174
.
The quickest fix (to this issue) would be to remove that line and let it wait for the rest of the servers. I know you mentioned we prefer to commit immediately with forceCommit, but the wait time isn't that long. MAX_TIME_TO_PICK_WINNER_MS seems to be hard coded to 3300ms.

This is indeed the case. I am less worried about the extent of the change, than the reason why we put it there in the first place. And yes, that was the exact reason we added another value to "stopReason" -- so that the completion manager can choose to take whatever action. All logic (when I look at it now) seems to suggest that we should move back to choosing the best offset behavior for this value of stopReason. But like I said, let us think a little bit.

Subbu is correct. The reason was to immediately kick off the force commit. Having said that, I agree that since the wait time to find out the server with the highest ingested offset is short - around 3s - we can modify the force commit to function like time-based commit scenarios and select the winner with the highest offset.

To be exact, there is no timer that goes off in 3.3s. Whenever the next segmentConsuming call comes in, we check the time, and if it is more than 3.2s then we just pick a winner amongst the replicas that indicated intent to complete the segment (instead of waiting for all replicas).

To be exact, there is no timer that goes off in 3.3s. Whenever the next segmentConsuming call comes in, we check the time, and if it is more than 3.2s then we just pick a winner amongst the replicas that indicated intent to complete the segment (instead of waiting for all replicas).

That's true, but the server itself will only hold 3 seconds between commit attempts per https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java#L62. It seems to be timed such that the server will try at most 3 times before a winner is picked.


Error Opening Wallet: java.lang.IllegalStateException: Expected BEGIN_ARRAY but was BEGIN_OBJECT at line 1 column 2 path $

Hi, I can't open Trezor Wallet when I enter my passphrase, that throws the following text:
java.lang.IllegalStateException: Expected BEGIN_ARRAY but was BEGIN_OBJECT at line 1 column 2 path $
Do you know what's the problem? Perhaps a character not allowed?
Thanks!!

Can you provide more details on your setup? Did you successfully import the Trezor into Sparrow, and are now trying to reopen the wallet file? Please also copy the full error from the log file (Help menu).

Hi, it happens the first time that I try to open the wallet of Trezor, I go to: file --> new wallet:
The error not appear in the last registries of the log file, you can see more info link of below:
https://drive.google.com/file/d/1PmudX9GY1Jy-zMpVQXrtEmRfLg2pVfwn/view?usp=sharing
https://drive.google.com/file/d/1PrZz4l3dMCWu1KZ3YYN_O7niSF-fbu8k/view?usp=sharing
Last registries of log file:
2021-11-17 13:45:54,428 WARN [Thread-17] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 8 secs 2021-11-17 13:51:47,248 WARN [Thread-31] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 16 secs 2021-11-17 13:52:03,566 WARN [Thread-31] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 14:02:55,252 WARN [Thread-55] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 14:03:34,436 WARN [Thread-55] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 14:05:07,693 WARN [Thread-60] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 15:19:07,724 WARN [Thread-223] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 15:19:46,690 WARN [Thread-223] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 15:21:11,889 WARN [Thread-227] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 15:21:44,284 WARN [Thread-227] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 15:22:11,890 WARN [Thread-229] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 34 secs 2021-11-17 15:35:59,763 WARN [Thread-33] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 8 secs 2021-11-17 15:36:13,460 WARN [Thread-33] c.s.s.n.TcpTransport [null:-1] No response from server, setting read timeout to 16 secs
Thanks!!

Yeah, I use these caracteres in my passphrase and the rest usual numbers and letters --> ,#*:"|
Are these characters allowed? --> ,#*:"|
Sorry, I don't stay prepared to run from source. I don't understand how do it. I'm running Windows Installer (7+) - Sparrow-1.5.2.exe in Windows 10.
Thanks!

Hi,
Finally, I created a new wallet with a new passphrase including only simple characters and it worked correctly without to appear anything error. I confirm that the issue was the specials characters that I used.
I hope you support it soon.
Thank you!!

I have reopened the bug as the special characters support is not resolved.

This turned out to be a Windows-only issue, and only when the quote " character was present in the passphrase. Fixed in d76aecb.


Make embed QR green + bottom left + helper text

New:
Old:

I think the green's an improvement over black but it's not a perfect match

I think the team voted against this idea, but thanks for the taking a shot at it!


Huge Latency for Window Events

Window Key Presses currently need to go through a lot of steps to reach the end callback:

Sent to keyQueue from GLFW callback
keyQueue checked by TickEvent in main thread
Fire Window Key Press Event, which goes into the main event queue
WindowKeyPressEvent is processed and it's callback(s) are ran in either the main or worker threads.

This is a large amount of steps for critical input from the player, which could result in damages to gameplay.
Proposed Fix:

Cut out the WindowKeyPressEvent and instead have games register input callbacks directly to the window
If the above option is not enough, then simply remove both WindowKeyPressEvent and the keyQueue and have the GLFW callback call the user callback (LAST RESORT)

I think the best solution is to create a dedicated InputManager with it's own dedicated input thread like Renderer. This will allow more than one input event to be processed per tick, and independent from the main thread.

Current issues include that multiple keys pressed will only register one key. This is probably because the callback is flooding the queue with GLFW_REPEAT. Solution is to detect presses and releases, and then use those to modify an object's velocity. This will require work in the ScreenObject class and wrapping.

Above comment has been implemented. Issue closed for now.


Build js in prod mode for the release

Follows #19

Check release fails related to https://github.com/jupyter-server/jupyter_releaser/issues/131

Thank you soo much @jtpio!
Merging and making a release to check that everything it's okay!


feature - create alert add to cart ui

Github Issue URL/Tags (required)
What's this PR do? Please give a technical summary (required)

[x] create alert add to cart

Any references (stackoverflow …) (optional)

Total linter violations in pull request: 2
Serious: 0
Warnings: 2

:+1: There are no linter violations.

Please check my pull request. Thank you!

:+1: There are no linter violations.


Error when running the "python main.py --mode=train" command. What am I doing wrong?

Hello. Would you like to know how to solve this problem? I'm a beginner, I performed the installation as indicated in the tutorial, but when running the command "python main.py --mode=train" occurs the following error to which I could not understand it. The error:
Traceback (most recent call last):
File "/home/lspgaucho/FinRL/finrl/main.py line 126, in
main()
File "/home/lspgaucho/FinRL/finrl/main.py", line 60, in main
train(
TypeError: 'module' object is not callable.
python --version is Python 3.9.7
What am I doing wrong?what do I need to do to fix this problem? Could you post an example? Thank you.

The error is due to a collision in the file name named main.py and the function called is main(). So try to change the main() to something like finrl_main().

This is a bug of main.py. You can correct it :
from finrl import train
fix to:
from finrl.train import train
so
from finrl import test
fix to
from finrl.test import test
from finrl import trade
fix to
from finrl.trade import trade


How to unregister from an instance_id

We have changed the instance, and after uploading the new version of the mobile app, it crashes on the user's phone and we recognized that the users are getting this crash due to trying to register to the new instance id and it requires them to uninstall and reinstall the app.
We deleted the old instance from our pusher account and we thought this is going to unregister the devices from the old instance, but unfortunately we still get the same error.
How to unregister a device (from an instance_id) when the user updates the app or how to solve this issue.??

Hi! I believe you have contacted our support team about this issue who have provided some code that can be used for this. I will close this issue as this is not a bug with the library.

Yes, someone already answered the email and the following worked for me. I'll leave it here for others who may need it later
I believe you will need to issue an update to your app that includes the code below immediately before you try and register/start the PushNotifications service with the new instance ID.
DeviceStateStore deviceStateStore = new DeviceStateStore(applicationContext);
if (deviceStateStore.getInstanceId().equals("OLD_INSTANCE_ID")) {
Log.e("TESTING", "clearing state");
deviceStateStore.clear();
}
// Start beams SDK with new Instance ID


V10 Urgent Warning!

"version" : 100000,
"protocolversion" : 70002,
"walletversion" : 60000,
"balance" : 0.00000000,
"blocks" : 30552,
"timeoffset" : -1,
"connections" : 5,
"proxy" : "",
"difficulty" : 30412271.47290681,
"testnet" : false,
"keypoololdest" :<PHONE_NUMBER>,
"keypoolsize" : 101,
"paytxfee" : 0.00000000,
"relayfee" : 0.00001000,
"errors" : "URGENT: Alert key compromised, upgrade required"

Use the version 0.14.1

I was, but the chain was stuck. I see you have now changed the releases again now though. What branch actually works?

Or....?

All branches work


KAFKA-2360: Extract producer-specific configs out of the common PerfConfig

Separate batch.size, message-size and compression-code from PerfConfig to a newly-created ProducerPerfConfig in order to hide them in ConsumerPerf tool.

@ijuma A minor change. Please review. Thanks.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6511/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6497/
Test PASSed (JDK 8 and Scala 2.12).

@ijuma Please take some time to review this minor change. Thanks

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6558/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6543/
Test PASSed (JDK 8 and Scala 2.12).

@ijuma Thanks for the comments. How about this change?

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6629/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6614/
Test PASSed (JDK 8 and Scala 2.12).

Merging to trunk.


Rendering parts of page based on given criteria

I'm looking into using this pjax library. https://github.com/MoOx/pjax A pjax request will have a header X-PJAX-Selectors that contains the parts of the page that need to be rendered. I'd like to create pages as normal and when a pjax request happens, only render the html elements that match the selectors.
What are some options for accomplishing this?
I was thinking that maybe I could create a custom tag helper where I specify the selector or maybe I can take the generated HTML and strip out anything that doesn't match that filter. Both of these would require me to intercept the output of the razor page to manipulate it. Where is the best place to do that?
I'm not sure where the best place is to begin.

Hi. It looks like this is a question about how to use ASP.NET Core. While we do our best to look through all the issues filed here, to get a faster response we suggest posting your questions to StackOverflow using the asp.net-core-mvc tag.

@NTaylorMullen, can you please advice on available options? Thanks!

@JoshClose there isn't a great way to accomplish what you're looking for sadly. My suggestion would be to create a middleware that wraps the entire response of your server => parse it with angle sharp or some other Html parsing library => apply your selector logic.

Is angle sharp fast enough? That is my ideal scenario.
What I'm currently doing is using some custom tag helpers. <pjax-show></pjax-show <pjax-hide></pjax-hide>

 This means I need to duplicate code and duplicate properties.

I hadn't seen AngleSharp before. I think this will do exactly what I need.

@NTaylorMullen How would I do this? If I register middleware after app.UseMvc, it never gets called.

I think I found something that will work. Let me know if this is wrong. https://stackoverflow.com/questions/27452734/modifying-the-raw-html-output-from-mvc-6

@JoshClose your middleware has to happen before the app.UseMvc and you need to capture the output of the Mvc middleware to properly parse it.
The stackoverflow link you provided will work.

Thanks. This seems to work pretty well.
public async Task InvokeAsync(HttpContext context)
{
	// Intercept the content written to the body.
	var originalBodyStream = context.Response.Body;
	var captureBodyStream = new MemoryStream();
	context.Response.Body = captureBodyStream;

	// Run the MVC middleware.
	await next(context);

	captureBodyStream.Position = 0;
	var reader = new StreamReader(captureBodyStream);
	var body = await reader.ReadToEndAsync();
	using (captureBodyStream) { }

	// Manipulate body here.

	using (var newBodyStream = new MemoryStream())
	using (var writer = new StreamWriter(newBodyStream))
	{
		writer.Write(body);
		writer.Flush();
		newBodyStream.Position = 0;
		await newBodyStream.CopyToAsync(originalBodyStream);
	}
}


Add Operator for last "x" emitted values

Feature Request
Is your feature request related to a problem? Please describe.
Need to do some data filtering based on previous values from a stream.  These values need to be consistently the last x values from the stream.
Describe the solution you'd like
An operator that returns the current value as well as the last x values emitted in an array.  Like PairWise, but for as many values as the user wants.
Describe alternatives you've considered
Currently adding the latest value to an array outside of the observables and using the last x values of the array as data for the stream
(If this is new operator request) describe reason it should be core operator
Accessing previously emitted values is key to doing mathematical functions related to data streams.

Isn't bufferCount what you are looking for? Or, at least, very close to what you are looking for?

bufferCount is not exactly what I am looking for, pretty close though.  BufferCount collects x-amount of values and then emits them as 1 array.  But it does not emit every time a value comes through the stream, bufferCount(10) would emit an array of 10 values every 10 ticks.
What I need is a continuous stream of the last 10 values.  F.ex. to calculate an exponential moving average for a data stream.

Have a closer look at its second argument.

Boom, thank you, can't believe I didn't notice that before.  No amount of googling mentioned this.
Guess that closes this issue.


release_vcs/svn: added get_current_revision, create_release_tag

just added some missing func to the svn release plugin, to make it minimally functional

Closing this now that master is the target branch - see:
https://github.com/nerdvegas/rez/pull/258


Can't upload profile picture

Subject: Can't upload profile picture
Description:
I'm trying to update my profile picture but I'm unable to confirm the upload. The error message "Couldn't connect to the server. Please check your Internet connection and try again." appears when clicking on the checkmark after selecting a photo, but my internet connection is fine.
I've tried on Chrome, Firefox and Safari, including incognito modes.
Results:
Backend version: develop-aeb1abd8
Frontend version: develop-aeb1abd8
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0
Page: https://couchers.org/profile/edit/about
User: @caminodeplata (49343)

I can't replicate on prod?

The user has given me more details, the issue appears to be with CORS.


Sleep Pin (on/off control) RockBLOCK 9603 v3.F

Due to supply-chain shortage, some components in RockBLOCK 9603 3.F have been replaced with alternatives. This does not adversely affect the normal operation of the device.
Because of these component changes, 3.F now includes a transistor circuit on the On/Off control pin to ensure correct power up at all supply voltages. The On/Off control pin should not be clamped below the RockBLOCK supply voltage and should be disconnected/left floating to turn on.
Potential issue:
This change has introduced an issue when interfacing directly to a 3V3 microcontroller pin while supplying the RB with 5V. The new On/Off circuitry now provides a high impedance (1Meg resistor) to the super cap voltage, equal to the supplied voltage. This may exceed the maximum IO voltage of the micro.
If the RB was supplied by 5V, the connecting device should also be 5V tolerant. If not, it is recommended to use an interfacing circuit, eg MOSFET, transistor, relay, etc. to connect to a microcontroller.
More info:
https://docs.rockblock.rock7.com/docs/power-supply#onoff-control

Is there a specific code change that will facilitate this, or would you like to add information about this change to the README?

This looks like an information thing that affects the interfacing hardware. Nothing requiring a code change.
I've request clarification from Ground Control if that issue applies only to the SLP pin, or affects all the other MCU interface pins as well. Will update the Learn Guide based on their response.
https://learn.adafruit.com/using-the-rockblock-iridium-modem/overview

Received confirmation from Ground Control that this only applies to the SLP pin.
Updated the guide with this info:
https://learn.adafruit.com/using-the-rockblock-iridium-modem/hardware#mixed-power-slash-logic-warning-on-slp-pin-3173236
This is a hardware issue, not a code issue. I don't think there are any necessary changes needed to this library.

Going to close this since it's not a library issue.


Fix broken LinkedIn link

The LinkedIn link on the blog site gives a '404 Page not found' error. I believe replacing this line with 'cole-s-smith' i.e. just your username should do the trick.

Thanks for your help!! Fixed in b592718994657b87e01765f0057b3ef3b95ac592 :)


feat: gatefi onramp

Description

Addition of GateFi Onramp which allows purchases of Assets supported by GateFi as well as additional localized payment methods in Latin America.
Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[x] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #
Risk

Testing

Engineering

Operations

Screenshots (if applicable)

closing in lieu of https://github.com/shapeshift/web/pull/4370


Sentiment Analysis With Naive Bayes Algorithms

Proposed Title: Sentiment Analysis With Naive Bayes Algorithms
Introduction paragraph(s):
In the era of hate speech on social media, how do we easily analyze the true intent of people's conversations among themselves on the social web, where people can be anonymous and throw slander at others from the corners of their rooms. This project is targeted at getting the sentiments in people's tweets whether it is positive or negative.
First and foremost, we built a simple sentiment analysis model using Naive Bayes and discover which of its algorithms is best. Also, it is not a tutorial about which algorithm is best for sentiment analysis but rather, which of the Naive Bayes Algorithms is best.
An IMDB dataset is used, it’s somewhat complicated data, a function was written to be able to read it into our notebooks. You can check out the function in the utils.py of our Github repo.
Code Base on Github
Key Takeaways:
By the end of this tutorial, the reader will be able to:

understand fully the concept of sentiment analysis.
understand Bayes Algorithms and the types.
carry out effective exploratory data analytics and feature engineering.
build a functioning model of several Naive Bayes Algorithms.
understand the concept of Natural Language Processing.
evaluate effectively classification models.

References:

What is Sentiment Analysis? A Complete Guide for Beginners
Sentiment Analysis: What is it and how does it work?
Word Vectorization (TFIDF/Word2Vec)

@wizardcalidad
to help avoid a backlog queue we are only accepting one article topic/pr approved at a time 👍


make roots_register_sidebars() less redundant

I tweaked this for all of my personal projects so I didn't have so much redundant code for having so many dynamic sidebars. one of my projects has about 12 sidebars (individual ones per page). edit if necessary.

so, I haven't experienced anything weird with this. technically if the issue is for easy unregistering of the sidebar in a child theme, then a simple priority number override when hooking to widgets_init would suffice and shouldn't be an issue. technically my pull request is practically just condensing a2d33c7 to a foreach loop and maintaining their unique id numbers.

thx a lot!

thank you for an awesome framework.


validate slp txs

Pull Request Test Coverage Report for Build 1495

0 of 0   changed or added relevant lines in 0 files are covered.
126 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.4%) to 74.746%

Files with Coverage Reduction
New Missed Lines
%

dist/routes/v2/slp.js
126
42.25%

Totals

Change from base Build 1431:
-0.4%

Covered Lines:
2920

Relevant Lines:
3710

💛  - Coveralls

:tada: This PR is included in version 1.14.5 :tada:
Your semantic-release bot :package::rocket:


Dynamically refresh device list

Currently one has to close and reopen the app to see the new input/outputs. It would be nice to have the program refresh the device list when a new one is detected by the system - or just allow a user to refresh the list manually with a button or Ctrl+R.

Will add manual refresh, there's no automatic notification currently, the GUI would have to constantly poll the devices and this just feels like a waste of cycles...

Makes sense. I wasn't sure if there was a way of receiving a trigger to poll when Windows detects a change in devices.

@flipswitchingmonkey I've been testing the refresh feature with various devices and it doesn't seem to work with any devices I've connected.


Fix AsyncOpenAI "RuntimeError: Event loop is closed bug" when instances of AsyncOpenAI are rapidly created & destroyed

… avoid https://github.com/openai/openai-python/pull/1262
Description
When AsyncOpenAI classes are rapidly created and destroyed, the underlying httpx.AsyncClient is not properly opened and closed unless requests are made from a context manager.  AsyncOpenAI clients underlie LlamaIndex OpenAI and OpenAI-Like classes, so if we rapidly create and destroy these (e.g. in multithread/multiprocess context) this error will arise. This PR just wraps the completions.create() calls within a client context manager and ensures that the underlying httpx.AsyncClient used by the OpenAI python client is properly opened and closed.
For completeness, completion.create() requests from the Sync OpenAI client are also wrapped in a client context manager. This PR also aligns the _achat() method to return logprobs similar to _chat() method.
Fixes issues raised in https://github.com/openai/openai-python/pull/1262, https://github.com/openai/openai-python/issues/1254
New Package?
Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

[ ] Yes
[x] No

Version Bump?
Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

[ ] Yes
[x] No

Type of Change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Added new unit/integration tests
[ ] Added new notebook (that tests end-to-end)
[x] I stared at the code and made sure it makes sense
[x] I implemented it in my project code and it resolves the RuntimeError

Suggested Checklist:

[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I ran make format; make lint to appease the lint gods

@philipchung This seems to have caused more issues when I run your branch?
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage

llm = OpenAI()

async def arun():
  await llm.acomplete("Hello World!")
  gen = await llm.astream_complete("Hello world!")
  async for token in gen:
    pass
  await llm.achat([ChatMessage(role="user", content="Hello World!")])
  gen = await llm.astream_chat([ChatMessage(role="user", content="Hello World!")])
  async for token in gen:
    pass

def run():
  llm.complete("Hello World!")
  gen = llm.stream_complete("Hello world!")
  for token in gen:
    pass
  llm.chat([ChatMessage(role="user", content="Hello World!")])
  gen = llm.stream_chat([ChatMessage(role="user", content="Hello World!")])
  for token in gen:
    pass

async def run_many():
  tasks = [arun(), arun(), arun()]
  await asyncio.gather(*tasks)

# test async
import asyncio
tasks = [arun(), arun(), arun()]
asyncio.run(run_many())

# test sync
run()

I get this error
RuntimeError: Cannot send a request, as the client has been closed.

If I install an older version of the openai package, it works fine though

I know using a global LLM is probably not advisable, but I know many users who do this

I believe I resolved the issues, and running your test code works.  Also passes pytest.
Changes are:

Using a client context manager closes the connection pool after leaving the context block, so if reuse_client=True, then this would cause an error for future completion.Create() calls. I changed the code to only use only client context manager if reuse_client=False.
I removed client context manager from streaming methods so they are now unchanged. It was unclear to me how to implement the client context manager with a generator.  Also, use cases with streaming completion.Create() calls are unlikely to use multithread/multiprocessing.

Yea streaming is kind of hard in this case -- lgtm otherwise now


Add PartitionsTable for partition metadata

LGTM except the missing test that @xabriel pointed out.

I've also created #659 to create a common class for metadata tables tests.

Looks like there is a conflict after merging #654.

Tests are passing so I'll merge this. Thanks for reviewing, @aokolnychyi and @xabriel!


.Net native compilation error

Hi, I'm using this lib on my UWP project but when I run the app on Release mode I get following exception due to creating a new object but I don't know why
Method 'AssemblyBuilder.DefineDynamicAssembly(AssemblyName, AssemblyBuilderAccess)' was not included in compilation, but was referenced in ImplementationBuilder..ctor(). There may have been a missing assembly.

You won't be able to use RestEase with .NET Native.
RestEase creates new types at runtime. NET Native won't support this.

So what should I do?!
You mean I can't package my app?!

Hmm, a quick google suggests it can be worked around. What is the exception type, exactly?

Try adding something like this to your default.rd.xml?
<Namespace Name="System.Reflection.Emit" Dynamic="Required All"/>

The exception type is System.Exception & exception message is what I posted in first post with nothing special more.
I tried the code, added it to the file under <Application> in <Directives> but again same error.

Are you sure? That would be very unusual. Can you post the entire exception details, including the stack trace, etc?

System.Exception: Method 'AssemblyBuilder.DefineDynamicAssembly(AssemblyName, AssemblyBuilderAccess)' was not included in compilation, but was referenced in ImplementationBuilder..ctor(). There may have been a missing assembly.
   at RestEase.RestClient.For[T]()
   at MyProject.ViewModel.ViewModelBase..ctor() in C:\fakepath\ViewModel\ViewModelBase.cs:line 17
   at MyProject.ViewModel.MainPageVM..ctor() in C:\fakepath\ViewModel\MainPageVM.cs:line 120
   at MyProject.MyProject_XamlTypeInfo.XamlTypeInfoProvider.Activate_132_MainPageVM() in C:\fakepath\obj\x64\Release\XamlTypeInfo.g.cs:line 606
   at Template10.Template10__Library__XamlTypeInfo.XamlUserType.ActivateInstance()
   at __Interop.ReverseComStubs.Stub_7(Object __this, Void** unsafe___value__retval, IntPtr __methodPtr) in C:\fakepath\obj\x64\Release\ilc\intermediate\MyProject.McgInterop\SharedStubs.g.cs:line 26428

For creating dynamic classes, you can also take a look at:
https://github.com/StefH/System.Linq.Dynamic.Core/blob/master/src/System.Linq.Dynamic.Core/DynamicClassFactory.cs
or
https://github.com/StefH/System.Linq.Dynamic.Core/blob/master/src/System.Linq.Dynamic.Core/ClassFactory.cs

@StefH How does that help? That still uses an AssemblyBuilder.

From some googling around, it looks like System.Reflection.Emit is simply not available in UWP. Despite UWP claiming to implement the .NET Standard. Yay.
Until/unless I get something together with Fody / source generators, I'm afraid you're out of luck here. Refit is good :)

Thanks man <3
Actually I was Refit user there also I've got another problem that they suggest me to use RestEase 😆

Haha! I've been recommending them for a while, interesting to see they're recommending me now...

@canton7
Hi,
The new VS 2017 (V15.4) & UWP fall creators SDK seems got some good news, could you please checkout here? (In the middle of page, .NET Standard 2.0 Support section)

So far as I can see, that only adds support for .NET Standard 2.0? There's nothing about supporting things like TypeBuilder in .NET Native?


Hugo fails if "+" is in MediaType

Hugo does like it if you have "+" sign in mediaType
this config.toml is OK:
baseurl = "/"
languageCode = "en-us"
title = "Homepage"

[mediaTypes]
  [mediaTypes."application/xml"]
    suffix = "abc"
  [mediaTypes."image/svg"]
    suffix = "svg"
  [mediaTypes."image/bpm"]
    suffix = "bpm"

[outputFormats]
  [outputFormats.raw]
    mediaType = "application/xml"
    isPlainText = false
  [outputFormats.svg]
    mediaType = "image/svg"
    isPlainText = true

[outputs]
  home = ["html"]
  page = ["html"]
  post = ["html"]

But this one leads to failure:
baseurl = "/"
languageCode = "en-us"
title = "Homepage"

[mediaTypes]
  [mediaTypes."application/xml"]
    suffix = "abc"
  [mediaTypes."image/svg+xml"]
    suffix = "svg"
  [mediaTypes."image/bpm"]
    suffix = "bpm"

[outputFormats]
  [outputFormats.raw]
    mediaType = "application/xml"
    isPlainText = false
  [outputFormats.svg]
    mediaType = "image/svg+xml"
    isPlainText = true

[outputs]
  home = ["html"]
  page = ["html"]
  post = ["html"]

The error message is
[20:30:55] Error: Hugo build failed
    at formatError (D:\YM\Documents\GitHub\cms1\node_modules\gulp\bin\gulp.js:169:10)
    at Gulp.<anonymous> (D:\YM\Documents\GitHub\cms1\node_modules\gulp\bin\gulp.js:195:15)
    at emitOne (events.js:116:13)
    at Gulp.emit (events.js:211:7)
    at Gulp.Orchestrator._emitTaskDone (D:\YM\Documents\GitHub\cms1\node_modules\orchestrator\index.js:264:8)
    at D:\YM\Documents\GitHub\cms1\node_modules\orchestrator\index.js:275:23
    at finish (D:\YM\Documents\GitHub\cms1\node_modules\orchestrator\lib\runTask.js:21:8)
    at cb (D:\YM\Documents\GitHub\cms1\node_modules\orchestrator\lib\runTask.js:29:3)
    at ChildProcess.<anonymous> (D:/YM/Documents/GitHub/cms1/gulpfile.babel.js:95:7)
    at emitTwo (events.js:126:13)

if you run Hugo from outside Gulp, you will get a bette error message, surely telling you that "+" is not supported in the media type name.

hugo/media/mediaType.go has the following code. I am not a programming expert, but the code looks like "+" should be supported?
// FromString creates a new Type given a type sring on the form MainType/SubType and
// an optional suffix, e.g. "text/html" or "text/html+html".
func FromString(t string) (Type, error) {
	t = strings.ToLower(t)
	parts := strings.Split(t, "/")
	if len(parts) != 2 {
		return Type{}, fmt.Errorf("cannot parse %q as a media type", t)
	}
	mainType := parts[0]
	subParts := strings.Split(parts[1], "+")

	subType := subParts[0]
	var suffix string

	if len(subParts) == 1 {
		suffix = subType
	} else {
		suffix = subParts[1]
	}

	return Type{MainType: mainType, SubType: subType, Suffix: suffix, Delimiter: defaultDelimiter}, nil
}

But what does the error message you get from Hugo say?

I don't know how to use hugo command. I use Netlify's CMS starter kit, and the only command I know is "npm start"

@dan1wang You can look into the package.json file to see which command is run. The section in the package.json is named scripts.
Try to run hugo standalone with the following command in a terminal.
$ hugo serve -v

This command starts the hugo server (which reloads your page on change) and -v enables verbose output.

I got this error message

Error: Error building site: media type keys cannot contain any '+' chars. Valid example is "text/css"

You are correct "+" is not supported
https://github.com/gohugoio/hugo/blob/47fdfd5196cd24a23b30afe1d88969ffb413ab59/media/mediaType.go#L155-L158


3 ➡ 4

CI is very unhappy. :(

CI is very unhappy. :(

Ok, there are some conflicts with PR 213 which will need some love.

Closing in favor of #104


Feat: software upgrade via chain manager [NTRN-400]

task
https://hadronlabs.atlassian.net/browse/NTRN-400
this PR

updates chain manager contract binary from this PR: https://github.com/neutron-org/neutron-dao/pull/115.

related PRs:

https://github.com/neutron-org/neutron-dao/pull/115
https://github.com/neutron-org/neutron-integration-tests/pull/352

https://github.com/neutron-org/neutron-tests/actions/runs/10911240382


Fix flaky integration tests on CI

ERROR 2020/10/21 07:25:56 Failed to get JSON resource "https://api.instagram.com/oembed/?url=https://instagram.com/p/BWNjjyYFxVx/&hidecaption=1": Failed to retrieve remote file: Bad Request

Also similar for the Twitter.

I'm assuming this is a rate limit issue?  I wonder if it would it help to add backoff+retry options to the getJSON func.

We're not adding stuff to getJSON -- that will soon (hopefully)  be replaced by a (hopefully) more robust resources.GetRemote (or something).

Finally!!
cc: @kaushalmodi

@onedrawingperday Thanks for tagging me here (and even remembering that I very much was troubled by that getJSON error).

@kaushalmodi You're welcome. I've also been beaten by this error myself and I followed the discussion back then between you and Bjørn Erik closely. So of course, I remembered.

Due to failed HTTP requests the instagram short code has been removed from https://github.com/gohugoio/hugoBasicExample. However, approximately 20 whitelisted themes on www.themes.gohugo.io use a (customized) version of the https://github.com/gohugoio/hugoBasicExample that still include the instagram shortcode. Thus the demos for this themes can't be build.
As mentioned by Alexandros here the error can be suppressed for the whitelisted themes with an config option once Hugo 0.77.0 is released.

The actual reason why getJSON fails seems to be that the instagram shortcode is generating urls that point to a deprecated Instragram API endpoint.
When using the instagram shortcode with id BWNjjyYFxVx the shortcodes uses this template to construct the url https://api.instagram.com/oembed/?url=https://instagram.com/p/BWNjjyYFxVx. When opening it a notice informs you about the deprecation of the linked API endpoint.

Hello @digitalcraftsman

instagram shortcode is generating urls that point to a deprecated Instragram API endpoint.

The above merits its own issue in the main issues tracker.
Apparently Instagram is ending the last remaining public API endpoint.
The Instagram shortcodes will either need to be refactored or removed.

Hello Alexandros,
I've added the information for context but created a new issue as suggested by you.


BAC-Control: Set "tag" property

Set the "tag" property of the newly inserted edge browser control to distinguish between "simple" edge browser control and Better Access Control.

Released with version <IP_ADDRESS>


Fix warnings and optimize constructor calls

Fixed the warnings related to docs and some other minor issues.
Added missing using statements in PasswordlessClient.
Tried to optimize constructor calls in PasswordlessClient to avoid repetition.

I also want to note that the current wiring of HttpClient in PasswordlessClient is not very good. It mutates the state of the client (i.e. sets the BaseUrl property and adds default headers). This is suboptimal because clients can be reused and changing those properties may cause issues in other parts of the program.
There are two recommended approaches.

(Basic) Take the HttpClient as is and instead set all the required items on HttpRequestMessage every time it's created. This can be encapsulated in a helper method.
(Complex) Create a PasswordlessHttpHandler that will wire the correct headers and resolve proper URLs. It can then take the upstream HttpClient or another HttpMessageHandler in its delegation chain. This is too complex for this situation but might be reasonable if we want to add things like custom retry handling (for example on 429 responses).

This is an example of what the complex approach may look like:

https://github.com/Tyrrrz/YoutubeExplode/blob/master/YoutubeExplode/YoutubeHttpHandler.cs
https://github.com/Tyrrrz/YoutubeExplode/blob/master/YoutubeExplode/Utils/ClientDelegatingHandler.cs
https://github.com/Tyrrrz/YoutubeExplode/blob/master/YoutubeExplode/YoutubeClient.cs#L43

@abergs @justindbaur I reverted the constructor changes because they warrant their own, larger PR. Please review the remaining changes so I can merge.


MINOR: [C++][Parquet] Fix GzipMembers tests when built w/o zlib

Rationale for this change
[----------] 1 test from TestGzipMembersRead
[ RUN      ] TestGzipMembersRead.TwoConcatenatedMembers
unknown file: Failure
C++ exception with description "NotImplemented: Support for codec 'gzip' not built" thrown in the test body.
[  FAILED  ] TestGzipMembersRead.TwoConcatenatedMembers (0 ms)
[----------] 1 test from TestGzipMembersRead (0 ms total)
[----------] 1 test from CodecTest
[ RUN      ] CodecTest.CodecRoundtripGzipMembers
/Users/gangwu/Projects/arrow/cpp/src/arrow/util/compression_test.cc:373: Failure
Failed
'_error_or_value43.status()' failed with NotImplemented: Support for codec 'gzip' not built
[  FAILED  ] CodecTest.CodecRoundtripGzipMembers (0 ms)
[----------] 1 test from CodecTest (0 ms total)
What changes are included in this PR?
Skip failed tests with zlib is not built.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.

A minor fix. PTAL. Thanks! @pitrou @mapleFU

Will rerun failed job and merge after all job success


Add nodeSelector, tolerations and affinity to Pod

Motivation
Currently there is no option to assign switchboard pod to a specific node.

Explanation
Using these options you can assign switchboard pod to a specific node

Assigning Pods to Nodes (nodeSelector and affinity)
Taints and Tolerations

Changes

Checklist

[ ] I added relevant tests - tests not needed
[x] I have updated the documentation if necessary - updated values.yaml

I'm not sure why the CI fails, I'll have to investigate this (seems to be unrelated to this PR).

@borchero any luck in fixing the CI?

Working on it, already found the issue. Will finish this tomorrow or Friday ... sorry for the delay 😅


PyPI release workflow failed on 0.13.4

For some reason, the workflow for PyPI release did not succeed on 0.13.4, see here. Therefore 0.13.4 was published manually, but we should probably understand why that was the case and fix it.

Fixed in #2999


Quick onboarding development dependencies: python, virtualenv and go

Description
Improving the onboarding documentation.
Related Issues
None.
Testing
Documentation only. Ambassador build was performed using the dependencies highlighted in the README.md file.
Todos

[x] Tests
[x] Documentation

Other

If this is a documentation change, please open the pull request at https://github.com/datawire/ambassador-docs instead.
Code changes should occur against master.

Thanks!!

(Since this is doc-only, I'm not going to wait for CI before landing. :wink:)


Could not run @clerk/remix on CloudFlare Workers/ Miniflare. Cannot use 'import.meta' outside a module Syntax Error

Browser/OS
macOS Big Sur
Wrangler Version
2.12.3
NodeJs Version
v18.15.0
Description
I want to @clerk/workers it in my next application. We are building a job board in Remix and hosting on Cloudflare Workers. After following the documents(https://clerk.com/docs/quickstarts/get-started-with-remix) I made sure that Clerk/Remix supports all the edge env including CF workers but when I tried to build the app using Miniflare(Wrangler 2.12.3) locally by following the steps in the official documentation, I got the following error at the 2nd step of Wrapping the App with ClearkApp.
SyntaxError: Cannot use 'import.meta' outside a module

interestingly the failure occurs only after the Build succeeded not sure if this is the problem of Miniflare/Wrangler or Remix or Clerk/Remix packages. I kindly request you to help me debug this. Thanks a lot.
### Steps to reproduce:
Please clone the repo to local machine into a local folder using
git clone https://github.com/aerojeyenth/demo-for-cf-error
run
npm run install
after npm install is compete
run npm run dev
Actual Behaviour:
npm run command fails with the following error:
SyntaxError: Cannot use 'import.meta' outside a module
Expected Behaviour:
npm run command should run without any errors and the remix application should run locally.

Closing this issue because it is related to HMR changes made in Remix


Determine list of packages that will migrate from jhuntwork/merelinux to merelinux/core

As a first iteration, I think this is probably a good list to start with:
acl
asignify
attr
bash
base-layout
binutils
busybox
bzip2
cmake
execline
dropbear
e2fsprogs
fedup
flex
gettext
gcc
gmp
git
libarchive
libffi
linux
llvm
make
meson
mold
mksh
mpc
mpfr
musl
ncurses
ninja
openssl
pacman
patch
perl
pkgconf
python
readline
s6
skalibs
sqlite
sudo
syslinux
utmps
xz
zlib-ng


🐛 Fix Makefile target: kind-create problem

What this PR does / why we need it:
The target "make kind-create" is broken due to incorrect file location.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #145

/test-ubuntu-integration-main

/retitle 🐛 Fix make target kind-create problem

/cherry-pick release-1.2

/lgtm

/cherry-pick release-1.2

/cherry-pick release-1.1


CARLA Render on Multiple GPUs?

Does the default coach system render CARLA on multiple GPUs? Or does it just set it to the default one? I couldn't get it running to check, so I figured this may be faster to get an answer.

Coach currently only supports 1 GPU for CARLA
Carla has a solution that uses vnc and virtualgl to implement it, but it is not implemented in Coach.
Have a look at this thread for more info

Gotcha; I've been using the instructions from there to set up a multi-gpu setup, but was having some problems with communication so I thought I'd use the coach implementation if it was available. Thanks again!


Add support for Gatsby 4.1

It looks like things moved around quite a bit for Gatsby v4.1 to support the jsxRuntime options.
Documented here: https://www.gatsbyjs.com/docs/reference/config-files/gatsby-config/#jsxruntime
PR here: https://github.com/gatsbyjs/gatsby/pull/33050/files
Trying to make this change backwards compatible, and the best place I could see to introspect the option now is off the webpack config's js loader options but perhaps there is a better alternative?

Nice work!


Resource loading

This implements a "magic" module '@jkcfg/std/resource'. It's magic
because it's generated each time it's imported, so that it can take
into account the path of the importing module (known at import time,
and not thereafter).
The implementation is this:

give the read procedure an option module for specifying a module to use as the base path (a missing value means use the input directory);
when @jkcfg/std/resource is imported, hash the current import base path (i.e., the directory of the importing module), then load module code exporting a procedure that reads using the hash.

~This has some flaws:~

~at present, jk (v8worker2, really) gives imports using the same
specifier (in this case @jkcfg/std/resource) the same module,
which is not much use for magic modules.~

~it relies on the module base path being readable (if you change the
input path when you run jk, it might not be)~

(EDIT: strike out flaws, which have been excised, and explain how module base paths are identified.)

One thing I'm not sure of is the resource API using import to load a specific resource. That means the std.read hidden in the magic importer cannot receive arguments to specify how the resource should be read.


Add initial implementation

The issue came up in the discussion of a PR to the Yireo_WebP compat module.
There currently is no way to request additional fields in GraphQL queries, that are usually embedded in .phtml templates.
Accessing additional related data is a common task in modules.
My first idea for a solution would be a service as a view model, that takes a GraphQL query string, parses it into a PHP array, dispatches an event and also provides a public method to plug into, and then returns the query as a string again.
In a template, instead of:
$query = "
products(filter: {} pageSize: 20) {
  items {
    {$type}_products {
        sku
        id
        small_image {
          url
        }
    }
  }
}";

the code would look something like:
$query = $gqlViewModel->query("product_list_query", "
products(filter: {} pageSize: 20) {
  items {
    {$type}_products {
        sku
        id
        small_image {
          url
        }
    }
  }
}");

The goal here would be to introduce as little complexity as possible from a developer perspective and keep backward compatibility.
A query customization is done with an event observer:
if ($event->getData('name') !== 'product_list_query') {
    return;
}
$query = $event->getData('gql_container')->getData('query');

$gqlEditor = new GraphqlQueryEditor();

// add a single field to a result object
$query = $gqlEditor->setFieldIn($query, ['products', 'items', 'small_image'], 'url_webp');

// add multiple fields to a result object
$query = $gqlEditor->setFieldIn($query, ['products', 'items', 'image'], 'url label url_webp');

// add a query argument
$query = $gqlEditor->setArgumentIn($query, ['products', 'filter', 'name'], 'match', 'Tank');
$query = $gqlEditor->setArgumentIn($query, ['products'], 'pageSize', 2);

$event->getData('gql_container')->setData('query', $query);

I've refactored the PR so the internal parsed AST is no longer part of the API.
It is now an implementation detail of the GraphqlEditor.
The benefits are:

No overhead of parsing and reserializing of the query if no customizations are present.
This is my main reason for the change.
Future changes of the parsing library can easily be integrated

The downsides of this change are:

Customizing queries now requires getting the query string from a container object in the event, and setting the updated query on the container after the desired changes where made.
If multiple customizations need to be done the query is parsed and serialized once for each call to $editor->setFieldIn() and $editor->setArgumentIn().

I've updated the examples in the PR description accordingly.


Break utils up into true and project utils

note to self: remember to change gruntfile

@gsvr please review.  Have a look at the functions in both utils files and let me know if you think certain functions should be moved back or forth between the two files.  Like briefly discussed earlier, we could consider making some of the bigger project-specific functions more generic and move them over to our shared utils...

@gsvr review plz


Seismic model cleanup

Small clean-up change that trims the Model in our seismic examples. Most of this is cosmetics to improve readability:

Rename IGrid to Model and move to seismic sub-directory
Let the Model provide the symbols m and damp and manage the associated data
Initialise Model with nbpml and provide shape as well as shape_pml - This makes operator construction cleaner, since we now provide all the grid-related meta-data through the model.
Let set_vp perform the automated m update
Make things more Pythonic :snake: by turning various get/set methods into properties

I opened an issue regarding my comments since they don't need to be addressed in this PR. I am all for merging this ASAP since I need it for other stuff that I am working on too.

Ok, I think all comments have been addressed now. @mloubout can you please check the last commit on the branch and make sure TTI still works as you'd expect?

Looks good to me.

Ok, travis is happy, so merging...


feat: Add statuspage exporter to list of exporters

Hello! I want to add my StatusPage exporter to the list of community exporters.
This exporter allows the monitoring multiple external status pages using a multi-target exporter pattern. It already supports the 2 most used status page engines and more are in progress.
For now it supported by me but also adopted by SRE team in my organization

:+1: Thanks!


Sometimes present methods

Thermocyclers has a block of code with
if thermocycler.respond_to?(:set_dye)
this is a type check.  Just like with inheritance, it is better to have a method that executes this block in the classes that have the set_dye method but does nothing in those that don’t

@bjkeller Would this new method be in the Thermocyclers module?

I'd guess AbstractThermocyclers

The reason I am not pushing it there is that show methods don't work from inside classes. The ThermocyclerHelper module is intended to be a layer between the protocol and the thermocycler that handles this logic.

in general, you should avoid code that says "if the thing has this type, then do behavior" and use polymorphism be it through inheritance or duck typing to ensure this behavior is somehow bound to the definition of the type.
I don't have a definitive answer for you when the behavior is presentation, but given the prevalence of MVC I'm sure it is out there.  I can look later if needed.
(BTW It is possible to pass the protocol and use the show block on that object)

OK, I understand. What I think I've realized here is that I am subtly overloading the thermocycler object. Whether the thermocycler is qPCR-capable is not the same as whether I want to do a qPCR experiment.

If I want to do a qPCR experiment, but the thermocycler is not qPCR capable, then it should raise an exception. This is terrible code!

Could you define a pattern for passing the protocol? IIRC, the show stealer thing that Eric came up with wasn't very reliable.


Rework kernel.rs

Closes https://github.com/hermitcore/rusty-loader/issues/90.

Parses the kernel from the ELF file without any allocations. (I will completely remove the allocator in a future PR)
Parses the kernel without any use of unsafe.
Heavily reduces the amount of transitive dependencies.
The loader can now be built with opt-level = 0 on x86. (I will perform this change in a future PR)

bors r+


Fresh, no-options install, can't register new user.

Hi, I followed the instructions on the Github page to add the latest gem to my project and then ran 'rails g authentification', then migrated the database. So just a generic install without options, under Rails 7.0.3.
Tried to add myself as a new user but the registrations controller keeps rejecting the signup, with the logs reporting unpermitted parameters for :authenticity_token and :commit.
Looked at my Users model but don't see anything that jumps out.  If I add those missing params to the Registrations controllers' permitted params, then the next error is "Unknown Attribute Error" for :authenticity_token on the User model.
Any idea how to fix? Thanks!

Why is your request a turbo stream ?

I can turn off Turbo on the submit like so:
<%= form_with(url: sign_up_path, data: { turbo: false }) do |form| %>
....it's no longer submitted as a turbo stream but I'm still having the same problem.

Unpermitted parameters are not an error, just a warning, it's not even shown in production... The password must be 12 characters, 1 lowercase, 1 uppercase, and a number.

Well, that explains it! The prompt said the password needed to be 12 characters….so when I satisfied that, I thought it would accept the password. Didn’t realize it had other requirements (1 lower case, etc). When I looked at the User model, I just didn’t notice the logic for that. It was end-of-day Friday! ;-)
So everything’s fine, thank you. But you might consider changing the prompt to also mention those other password requirements, since they all need to be satisfied.

Then that information about 12 characters should write also about lowercase and uppercase req.

Noo... Let's keep it like hey.com, I would recommend also setting an attribute passwordrules="required: lower; required: upper; required: digit; minlength: 12;" to password inputs. I don't want to put it on the template to keep it simple.

I removed the format validation from the password... I realized that hey.com validates though pwned only.


Change _TimeViewer slider style

It occurs to me that I never asked for feedbacks for the slider style, I just went with what I found practical but apparently the current design has some limitations. For example, since the slider tube is very thin it's easy to miss it and rotate the brain instead (reported in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-583536539). This PR tries to address that by changing the style of the slider (and by doing so, modifyint it's hitbox).
I suggest 2 styles of sliders. In both styles I removed the end cap. In style 1, I increased the tube width but the overall style stays similar. The style 2 is a bit extreme but fixes the original issue quite easily.

import os
import mne
from mne.datasets import sample
data_path = sample.data_path()
sample_dir = os.path.join(data_path, 'MEG', 'sample')
subjects_dir = os.path.join(data_path, 'subjects')
fname_stc = os.path.join(sample_dir, 'sample_audvis-meg')
stc = mne.read_source_estimate(fname_stc, subject='sample')
initial_time = 0.1
mne.viz.set_3d_backend('pyvista')
brain = stc.plot(subjects_dir=subjects_dir, initial_time=initial_time,
                 clim=dict(kind='value', pos_lims=[3, 6, 9]),
                 hemi='lh',
                 )

master
Style 1
Style 2

Except the size, it's also possible to change the colors (c.f. https://github.com/pyvista/pyvista/pull/511):

Sadly, changing the shape or replacing the handle by a texture would require way more work on the VTK part.

What do you think @agramfort, @hoechenberger, @larsoner, @drammock ?

+1 for style 2 with slightly less thick tube & slider. This then also mimics the sensor-level plots (e.g. Raw.plot()) and it's also the kind of slider used in all (?) modern operating systems / GUIs: tube has the same width as the slider.

Style 3 is just a rework of 2) but with the same size used in master.

Also feel free to share if you have any suggestions for the colors.

Style 3 is just a rework of 2) but with the same size used in master.

That could work! What do the others think? Sure, I think it's more aesthetically pleasant to look at those sliders where the tube is narrower than the slider element, but I think this here is an important tradeoff to make between eye candy and usability.
Regarding the colors, can you adjust the tube color to be slightly less salient? Very-light-gray-ish or something?

Regarding the colors, can you adjust the tube color to be slightly less salient? Very-light-gray-ish or something?

Good idea!

It's not by any chance possibly to specify a color for the outline (border) of the tube, is it? Bc then you could make the tube black (as the background) and still indicate that this is a slider widget by having a fine outline drawn, inside which the slider can move.

AFAIK not just with sliders but maybe with vtkBorderWidgets...
Reference: https://vtk.org/Wiki/VTK/Examples/Cxx/Widgets/BorderWidget

AFAIK not just with sliders but maybe with vtkBorderWidgets...

Seems too complicated… let's see if a less prominent tube color does the trick for now :)

Trying with my firefox slider on KDE:

I obtain:

@GuillaumeFavelier looks good to me!!

@GuillaumeFavelier I just pulled and tried this out and I noticed another (minor) oddity:
When I click a slider somewhere left or right of its horizontal center point and then start dragging, as soon as dragging begins, the slider jumps such that it's centered horizontally below the cursor.

If you click the tube it's normal that the slider jumps but when I drag, the slider moves only at the end of the interaction for me. I don't think I can reproduce this

@GuillaumeFavelier I've recorded a video:
https://drive.google.com/open?id=1EkPfPqRT47euD87_E8Idrr4P98PQX2hp

I think we can live with that :)

I've recorded a video

Thank you, now I understand what you mean.

Looks great to me, I do like the slider filling up the entire height of the tube

This is ready for merge then

Thanks @GuillaumeFavelier


ARROW-3325: [Python][Parquet] Add "read_dictionary" argument to parquet.read_table, ParquetDataset to enable direct-to-DictionaryArray reads

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.
Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks
https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7
TODO

[ ] Add documentation to parquet.rst to explain how to use this

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:
Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

1000 unique strings of length 50
Total number of rows: 10 million
Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

I just pushed a docstring only fix. Merging this

Oops, my doc fix broke the Python 2.7 build. I will fix


RQ backend viable ?

I was wondering if those who know enough Luigi's internals if an RQ backend would be viable.
We would love to have a simple batch workflow atop RQ and Luigi looks quite neat but Hadoop is way too overkill for our current needs.

What's RQ?

Sorry, I forgot to put a link : http://python-rq.org/

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the
background with workers. It is backed by Redis and it is designed to have a low barrier to entry.
It can be integrated in your web stack easily.

sure, would make sense

I'm not sure I understand what you ask for. Are you asking if a RQ backend would viable as a replacement for something??

Hi,
Looks like we have the same kind of issue: we have to run batch of jobs with complex dependencies that luigi seems to address very well. But the jobs are not well-suited for map-reduce and we would like to use RQ instead. Any chance that luigi can handle RQ jobs instead of hadoop ones?
I don't know enough of luigi to look at the code myself but if you have any indication on where to start, I can have a look.
Thanks!

Luigi is really in no way tied to map-reduce. It has it's root there though and many of design decisions still today consider luigi as predominantly used in big data settings (few tasks, but usually big).
I don't know enough RQ to really understand where to point you. Sorry. :)

all you need to do is subclass luigi.Task and implement a run method – should be easy :)

Comparing to M/R, luigi submits a single luigi.Task per hadoop job of course but that job has many parts parallelized. Are you trying to have a 1-to-1 with luigi task in the flow or would you expect to have many little RQ-task's for a given luigi task step?

Thanks for the comments.
Subclassing luigi.Task and implementing run would make sense only if the run method itself wait for the rq job to end before returning, otherwise it would just enqueue the job and return, and job dependencies would be broken. But this could perhaps be a solution, I have to think more about it.

Are you trying to have a 1-to-1 with luigi task in the flow or would you expect to have many little RQ-task's for a given luigi task step?

1-to-1.

closing this issue.

[x] It has been inactive for +4 months.
[ ] It's not about luigi core, so not as many users are affected about this.
[ ] The change seems quite big, it's unlikely to be sporadically picked up.
[x] The owner haven't responded or disappeared.
[ ] I don't understand what this issue is about.
[x] There exists a reasonable workaround for this.
[ ] We need to check if this hasn't been fixed by now (for old issues).
[x] This is kind of by design and not a bug.
[ ] Resolving this would probably add a lot of complexity.

Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.
Feel free to reopen this issue at any point if you have the intent to continue to work this. :)


Mandatory claim mapping for primary userstore

Claim mapping for primary user store should be mandatory. We need to add a validation in UI level and in back-end service level to restrict adding claim mapping without having a claim mapping for primary userstore.

Currently the UI do validate the Mapped Attribute as mandatory.
But there's an issue in making claim mapping for primary user store deletable.
Shall we remove the selection from the very first attribute mapping and make it a mandatory field and remove "Delete" for that domain name?
Before

After

WDYT?

+1

Thank you for your contribution!
We are closing this issue since it has not been prioritized for a long time. Chances are that it has already been solved in more recent versions. If not, we will be re-evaluating this when it becomes a priority.


v0.3.10 is empty

This issue is for a:
- [X] bug report

Minimal steps to reproduce
I've found 2 ways to reproduce the issue :
On a fresh App

create an app with create-react-app and cd into it
yarn install react-aad-msal, that installs v0.3.10 at the time of writing
in any .js file, add import { AzureAD } from 'react-aad-msal'; and use <AzureAD>
yarn start

Using the sample App

Clone the git repo
cd react-aad-msal/sample
rm -rf node_modules
in package.json, replace  "react-aad-msal": "file:../", by "react-aad-msal": "0.3.10",
yarn start

Any log messages given by the failure
In both cases, the console where App is started says
Module not found: Can't resolve 'react-aad-msal'
And I'm unable to use the components of the package.
Expected/desired behavior

react-aad-msal actually contains something

OS and Version?
macOS High Sierra 10.13.4
Versions

node v8.11.2
yarn v0.19.1

Mention any other details that might be useful
Here is the content of node_modules/react-aad-msal, that seems a bit empty to me

Hope this is clear enough, talk to you soon! 👋

Until we fix this, 0.3.7 does contain the /dist folder and should work fine; there hasn't actually been any changes between 0.3.7 and 0.3.10.

Thanks for the quick reply and the hint on 0.3.7.
Good luck on the Travis CI battle :p

A build was triggered that creates a version 3.17 and that isn't empty. Sorry about this @cadichris -- still transitioning to Travis version and publish
Thanks, and no worries.

this should be addressed now with 3.18 closes #40
Perfect, I'll use that one!


Can we update the project NET Framework to version 4.8?

Yesh.

For development purposes, you can set it to .NET 4.8.
But I choose .NET 4.7.2 for wider app distribution.


ios core : ios / JS exchanges encoding issue with non UTF8 chars

When sending an Emoji from native side to the web, the string encoding of the JSON is crashing.
Because Emoji are not in the scope of UTF8 and the ios isValidJSON method is not warning about any problem.

@ggendre Fixed or not fixed? That is the question...

Seems to work


Stream write callback doesn't allow handling errors

foreign import writeStringImpl
  :: forall r
   . Writable r
  -> String
  -> String
  -> Effect Unit -- callback should allow error handling
  -> Effect Boolean

The callback should allow error handling, for example:
(Nullable Error) -> Effect Unit though the error value passed to the callback may be undefined or js Error.

See https://github.com/purescript/purescript/pull/4299#issuecomment-1112528546=

@wclr Can you provide an example that demonstrates a time where the arg will be undefined? The docs don't really clarify this, so I assumed it was always Error and forgot about the null possibility.
@thomashoneyman I think we'll need to fix this before 0.15.0 is released. Our FFI bindings cannot have wrong type signatures.

Can you provide a Node.js example that demonstrates a time where the arg will be undefined?

It is undefined in v16.13.2:
const fs = require("fs")
var writeStream = fs.createWriteStream("./file.txt")
writeStream.write("some", null, (err) => {
  console.log("err", err) // prints out `undefined`
  writeStream.end((endErr) => {
       console.log("endErr", endErr) // prints out `undefined`
   })
})

So to use Nullable that means that implementation:
export function write(w) {
  return chunk => done => () => w.write(chunk, null, done);
}

Should be changed to:
export function write(w) {
  return chunk => done => () => w.write(chunk, null, (err) => done(err || null);
}

The same for writeStringImpl and end (this too may give the error, for example if try to end already finished stream) foreing implementations.

Thanks. I can confirm that. Idiomatically, it would be better if the API exposed to the end-user was Maybe Error -> Effect Unit.

It would be better if the API exposed to the end-user was Maybe Error -> Effect Unit.

I think this is right, I just usually try to make FFI code as simple as possible.

My thought was that if we used Nullable Error, all end users would have to write something like this:
-- We want to handle both cases
\err -> do
  case toMaybe err of
    Nothing -> ...
    Just e -> ...

-- We only care if it's an actual error:
\err -> for_ (toMaybe err) \actualErr -> ...

-- We only care if it's not an error:
\err -> when (null == err) $ ...

Since 2/3 use toMaybe and the third could be written in terms of Maybe (e.g. when (isNothing err) $ ...), we could implement the FFI like this:
foreign import writeStringImpl
  :: forall r. Writable r -> String -> String -> (Nullable Error -> Effect Unit) -> Effect Boolean

writeString
  :: forall r. Writable r -> String -> String -> (Maybe Error -> Effect Unit) -> Effect Boolean
writeString w s1 s2 cb = writeStringImpl w s1 s2 (cb <<< toMaybe)

Should be changed to
export function write(w) {
 return chunk => done => () => w.write(chunk, null, (err) => done(err || null));
}

Not quite. The err || null part isn't needed. toMaybe is defined as:
toMaybe :: forall a. Nullable a -> Maybe a
toMaybe n = runFn3 nullable n Nothing Just

nullable is implemented as:
export function nullable(a, r, f) {
  return a == null ? r : f(a);
}

and running nullable(undefined, true, () => false) returns true.
If nullable was implemented as a === null rather than a == null, then we'd need the err || null part.

We do tend to use Maybes rather than Nullable in the web libraries, as it is nicer to deal with for consumers.

The same for ... end (this too may give the error, for example if try to end already finished stream) foreign implementations.

Yeah, I can confirm this too with this example:
import fs from "fs";

var file = fs.createWriteStream("./file.txt")
file.on('error', () => {});
file.write("something", () => {
  file.destroy(new Error("Muahahah!"));
  file.end((err) => {
    console.log("Example where 'err' is Error");
    console.log(err)
  });

  var file2 = fs.createWriteStream("./file2.txt")
  file2.end((err) => {
    console.log("Example where 'err' is undefined");
    console.log(err);
  });
});

which outputs:
Example where 'err' is Error
Error [ERR_STREAM_DESTROYED]: Cannot call end after a stream was destroyed
    at new NodeError (node:internal/errors:371:5)
    at WriteStream.Writable.end (node:internal/streams/writable:636:11)
    at file:///home/user/z.mjs:7:8
    at afterWrite (node:internal/streams/writable:497:5)
    at onwrite (node:internal/streams/writable:477:7)
    at node:internal/fs/streams:410:5
    at FSReqCallback.wrapper [as oncomplete] (node:fs:801:5) {
  code: 'ERR_STREAM_DESTROYED'
}
Example where 'err' is undefined
undefined


[MB-6905, MB-6945] Add service to read files from Syncada via SFTP

Description
We created a service in pkg/services/invoice/syncada_sftp_reader.go to read files from an SFTP directory, process them, and delete files that were successfully processed.
This service will attempt to process every file in the remote directory that it can. It does not walk the directory tree, so subdirectories will be ignored. Failures will be logged, and will not stop processing of other files.
Processing is done by passing something that implements SyncadaFileProcessor to syncadareaderSFTPSession.FetchAndProcessFiles.
The processor should take a file's contents as a string and do whatever needs to be done with it, returning an error if something failed. For testing purposes, we have a simple processor that prints the file path and contents to stdout
The processing function should be able to deal with duplicates across different runs because it will be run in a scheduled task, and one execution could overlap with another if it takes longer to run than the time between runs.
FetchAndProcessFiles takes a time argument and returns the modified time of the newest file it processed. This can be used to skip over files that we were unable to read or process in previous runs, so we don't log the same error a million times. This has been useful for testing, but I'm not sure it's a feature we will actually want in production, since errors will hopefully be rare and we delete files after processing them.
Currently on the testing server, we have all response files going to the same directory. We're hoping that we will get different kinds of responses (997s and 824s) in different directories.
We also added a command line utility for manual testing:
# Fetch and process files from syncada:
go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/

# Fetch, process and delete files from syncada:
go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/ --delete-after-processing

# Omit files created before a particular time with `--last-read-time`:
go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/ --last-read-time 2021-03-19T18:25:36Z

Reviewer Notes
Setup
Sending EDIs to Syncada
This has been a bit finicky for me. Sometimes it works and sometimes it doesn't, and i haven't figured out yet whether there's just a long delay, or the files aren't being uploaded. Since sending the files isn't part of this PR, you may want to skip over this and try downloading files that have already been uploaded.
The devseed data has some payment requests that aren't set up quite right, so the most foolproof way to go about this is to start with a fresh DB and make a move from scratch.
If you do want to use make db_e2e_populate, you'll have to go into the payment_requests table and update the statuses of any moves that you don't want to send to SENT_TO_GEX.
make server_run

# in another terminal
make client_run

Create a move with a payment request, and approve the payment request as a TIO.
Send EDIs for all reviewed payment requests to Syncada with:
{
  "body": {
    "sendToSyncada": true
  }
}

prime-api-client --insecure support-reviewed-payment-requests --filename sendtosyncada.json

Download files from Syncada
go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/

You should see some EDI files logged.
If you delete the files with:
go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/ --delete-after-processing

and then run again, you shouldn't see any more files logged.
References
We ended up doing both of these stories together:

jira story for reading files
jira story for deletion

Messages

:book:
:link: MB-6905

  Generated by :no_entry_sign: dangerJS against edbd5e6ed333f95c6c92fded1f0c98167522f981

@mboren @reggieriser I ran into this issue when running the command to read files.
FATAL   fetch-from-syncada-via-sftp/main.go:109 couldn't initialize SSH client {"error": "failed to parse host key ssh: no key found"
I'm guessing, I might be missing some environment variables locally?

I'm guessing, I might be missing some environment variables locally?

Yes, sounds like it. You may need to try resetting your .envrc.local from chamber so you pick up the needed SFTP variables. I'm not sure how you have yours set up, but see if these instructions help. If you're still having problems, we can try pairing.

Thanks! I was able to download/read the files with go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/. I then ran go run ./cmd/fetch-from-syncada-via-sftp  --directory /CSFTP866/Pickup/ --delete-files-after-processing to delete them and I think that worked but when I check by re-running the first command, I got back:
 INFO    invoice/syncada_sftp_reader.go:50       Error while downloading Syncada file    {"path": "/CSFTP866/Pickup/.working", "error": "failed to open file over SFTP: permission denied"}

Thanks! I was able to download/read the files with go run ./cmd/fetch-from-syncada-via-sftp --directory /CSFTP866/Pickup/. I then ran go run ./cmd/fetch-from-syncada-via-sftp --directory /CSFTP866/Pickup/ --delete-files-after-processing to delete them and I think that worked but when I check by re-running the first command, I got back:
 INFO    invoice/syncada_sftp_reader.go:50       Error while downloading Syncada file    {"path": "/CSFTP866/Pickup/.working", "error": "failed to open file over SFTP: permission denied"}

There are some files out there like this .working file that are in the directory on the SFTP server but we don't have access to open/read them. Since we don't know the pattern yet for the filenames we should be reading, it's tough to know what to automatically skip. This is just an informational log message. @mboren might have more context.

Yep, Reggie's got it. I think I'll update the log message to be a bit less alarming and add a comment with some context.


Admins not able to view all workflows

When admins go to view a workflow from the Administer > Users or Administer > Teams views they get a 403 PERMISSION DENIED response

Verified as working in Flowabl environment


encryption: tikv panic when encryption is enabled after restore a backup

Bug Report

What version of TiKV are you using?
tikv master
What operating system and CPU are you using?

Steps to reproduce
turn on encryption on an empty tikv, restore a backup. restore again (or run some write workload), tikv panic with error:
[2020/04/16 12:22:54.418 +08:00] [FATAL] [lib.rs:480] ["rocksdb background error. db: kv, reason: compaction, error: Corruption: bad checksum type"]

What did you expect?
tikv should restore backup successfully
What did happened?
tikv panic

Cause: tikv makes a copy of the SST before ingesting it to rocksdb. The copy is not tracked by DataKeyManager and when rocksdb read the ingested file, DataKeyManager tell rocksdb the file is not encrypted but actually it is. Will send a fix tomorrow.


Replace map by foreach

Replace calls to .map by .forEach when nothing uses the result
Also fix suspicious code probably due to a refactor that was filtering on a array that is not used anyway

extensions.contribution.ts - ✅

searchModel.ts ✅