Parser errors/warnings for new syntax are confusing

Summary:
When playing around with the new syntax I found that the parser errors and warnings were not very informative. Well, they were informative enough for me because I already know the correct syntax and was intentionally trying to trigger the error messages, but I imagine they would pretty opaque to users. Example in Reproducible Steps, below.
Description:
See below in Reproducible Steps

[x] fix error message for missing vertical bar
[ ] fix duplicate warning message

Reproducible Steps:
First put the following in logistic.stan
data {
  int<lower=0> N;
  int<lower=0,upper=1> y[N];
}
parameters {
  real theta;
}
model {
  theta ~ normal(0, 1);
  y ~ bernoulli_logit(theta);
}
generated quantities {
  vector[N] log_lik;
  for (n in 1:N)
    log_lik[n] = bernoulli_logit_log(y[n], theta);
}

then run
library("rstan")
stanc(file = "logistic.stan")

This gives the warning:
DIAGNOSTIC(S) FROM PARSER:
Warning: Deprecated function 'bernoulli_logit_log'; please replace suffix '_log' with '_lpdf' for density functions or '_lpmf' for mass functions
Warning: Deprecated function 'bernoulli_logit_log'; please replace suffix '_log' with '_lpdf' for density functions or '_lpmf' for mass functions

which is good (not one of the ones I found confusing), but why does it appear twice if
bernoulli_logit_log is only used once in the Stan program? Is it because the sampling
statement y ~ bernoulli_logit(theta) is also used? Either way it's kind of weird.
The stuff I found more confusing comes next, when trying to follow the suggestion in the warning.  First I modify the bernoulli_logit_log line in generated quantities:
log_lik[n] = bernoulli_logit_lpdf(y[n], theta); \\ I know this is incorrect

and the error is
SYNTAX ERROR, MESSAGE(S) FROM PARSER:
ERROR at line 15

 13:      vector[N] log_lik;
 14:      for (n in 1:N)
 15:        log_lik[n] = bernoulli_logit_lpdf(y[n], theta);
                                             ^
 16:    }

PARSER EXPECTED: <probability function arguments>
Error in stanc(model_code = paste(program, collapse = "\n"), model_name = model_cppname,  :
  failed to parse Stan model 'logistic' due to the above error.

which just says that it expected <probability function arguments>. I assume that's trying to tell me that I forgot the vertical bar but that's not what the message says. Would it be possible to mention the vertical bar in the message? In addition, I used a function that doesn't exist (bernoulli_logit_lpdf even though it's a mass function) but didn't get any error about that. So it seems to be giving me an error about using the wrong arguments to a function that doesn't exist.
So next I use the vertical bar:
log_lik[n] = bernoulli_logit_lpdf(y[n] | theta);

and that gets rid of the mysterious <probability function arguments> and now I get the error about bernoulli_logit_lpdf not existing:
SYNTAX ERROR, MESSAGE(S) FROM PARSER:

No matches for:

  bernoulli_logit_lpdf(int, real)

Function bernoulli_logit_lpdf not found.

ERROR at line 15

 13:      vector[N] log_lik;
 14:      for (n in 1:N)
 15:        log_lik[n] = bernoulli_logit_lpdf(y[n] | theta);
                                                           ^
 16:    }

Error in stanc(model_code = paste(program, collapse = "\n"), model_name = model_cppname,  :
  failed to parse Stan model 'logistic' due to the above error.

except it doesn't tell me that I just need to use lpmf instead of lpdf. Would it be possible to detect that that someone is using a function that just has an incorrect suffix?
Current Output:
The current output is given above in Reproducible Steps
Expected Output:
I think it would be helpful if:

Errors about functions not existing were thrown before errors about incorrect arguments
Errors about missing vertical bars said something about missing vertical bars instead of
just <probability function arguments>
Only one warning is issued if deprecated function is only once in a Stan program
Using lpdf instead of lpmf or vice-versa resulted in a more informative message
indicating that the correct function just has a different suffix

Are there reasons I'm overlook why any of these would be difficult?
Additional Information:
Current Version:
v2.10.0

Here are some possible alternatives for a few of the error/warning messages mentioned above:
Current:

Function bernoulli_logit_lpdf not found.

Suggestions:

Function bernoulli_logit_lpdf not found. Did you mean bernoulli_logit_lpmf?
Function bernoulli_logit_lpdf not found. For mass functions use _lpmf instead of _lpdf.

Current:
* `PARSER EXPECTED: `
I saw this in the context of a missing vertical bar, but I'm not sure if that's the only time this error is thrown. If it only pertains to the vertical bar then maybe replace with something like one of these
Suggestions:

PARSER EXPECTED: vertical bar instead of comma after first argument of bernoulli_logit_lpmf
PARSER EXPECTED: '|' instead of ',' after first argument of  bernoulli_logit_lpmf

Regarding PARSER EXPECTED: <probability function arguments>, between that error message and the manual, it's not clear to me when I'm supposed to use |.

After the first argument whenever there's an lpdf, lpmf, lcdf, or lccdf.
I could relax the syntax to make the '|' optionally a ',', but I've been
reluctant to have two ways of doing the exact same thing.

On Jul 3, 2016, at 3:18 PM, Jeffrey Arnold<EMAIL_ADDRESS>wrote:
Regarding PARSER EXPECTED: , between that error message and the manual, it's not clear to me when I'm supposed to use |.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

But can the parser identify when a | is missing and suggest it instead
of just giving the “probability function arguments” warning?
On Jul 4, 2016, at 8:54 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:

After the first argument whenever there's an lpdf, lpmf, lcdf, or lccdf.
I could relax the syntax to make the '|' optionally a ',', but I've been
reluctant to have two ways of doing the exact same thing.

On Jul 3, 2016, at 3:18 PM, Jeffrey Arnold<EMAIL_ADDRESS>wrote:
Regarding PARSER EXPECTED: , between that error message and the manual, it's not clear to me when I'm supposed to use |.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Yes, that's what I plan to do.  It's already recognizing
it, just a poor choice of error message---I should've provided
an example, as in variate | parameters.  I'm still not exactly
sure how to say that.  Should I just say "expected |" and be done
with it?

On Jul 4, 2016, at 12:11 PM, Michael Betancourt<EMAIL_ADDRESS>wrote:
But can the parser identify when a | is missing and suggest it instead
of just giving the “probability function arguments” warning?
On Jul 4, 2016, at 8:54 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:

After the first argument whenever there's an lpdf, lpmf, lcdf, or lccdf.
I could relax the syntax to make the '|' optionally a ',', but I've been
reluctant to have two ways of doing the exact same thing.

On Jul 3, 2016, at 3:18 PM, Jeffrey Arnold<EMAIL_ADDRESS>wrote:
Regarding PARSER EXPECTED: , between that error message and the manual, it's not clear to me when I'm supposed to use |.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

How about “Expected a | separating the random variables from the parameters in ”?
On Jul 4, 2016, at 10:38 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:

Yes, that's what I plan to do. It's already recognizing
it, just a poor choice of error message---I should've provided
an example, as in variate | parameters. I'm still not exactly
sure how to say that. Should I just say "expected |" and be done
with it?

On Jul 4, 2016, at 12:11 PM, Michael Betancourt<EMAIL_ADDRESS>wrote:
But can the parser identify when a | is missing and suggest it instead
of just giving the “probability function arguments” warning?
On Jul 4, 2016, at 8:54 PM, Bob Carpenter<EMAIL_ADDRESS>wrote:

After the first argument whenever there's an lpdf, lpmf, lcdf, or lccdf.
I could relax the syntax to make the '|' optionally a ',', but I've been
reluctant to have two ways of doing the exact same thing.

On Jul 3, 2016, at 3:18 PM, Jeffrey Arnold<EMAIL_ADDRESS>wrote:
Regarding PARSER EXPECTED: , between that error message and the manual, it's not clear to me when I'm supposed to use |.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I'll +1 this issue as well if possible.
I was going through the deprecation appendix in the latest stan doco and updating all _log functions to _lpdf calls, and had no idea that the | syntax was even used, especially because the API documentation explicitly uses a comma and not a separator (eg chapter 44.3 for skew_normal - the first distribution I went to update and got stuck on, not a special example).

Agreed.  Those will both be fixed for the next release. The issue's flagged
here for the parser warning and in the manual issue for the manual.

@betanalpha  That would be confusing because any of the arguments can be random variables and the first one isn't required to be a random variable.  It's much more work to call out the specific function (I'd have to write a new rule for the case where someone forgets the vertical bar, then create a custom message for it, then have it fail after it's matched and reported an error.)  What I went with produces this:
SYNTAX ERROR, MESSAGE(S) FROM PARSER:

ERROR at line 5

  3:    }
  4:    model {
  5:      target += normal_lpdf(y, 0, 1);
                               ^
  6:    }

PARSER EXPECTED: <probability function arguments with first
 two arguments separated by a vertical bar;
 e.g., foo_lpdf(y | alpha, beta).
 This syntax required for functions with suffixes
 _lpdf, _lpmf, _lcdf, _lccdf>

To be technical, variables on the right hand side of the bar aren’t
actually random anymore.  When you condition you condition on
an explicit value.  In other words, the conditional distribution is not
defining a distribution for those variables, hence they take on a
different interpretation.
On Aug 22, 2016, at 9:07 AM, Bob Carpenter<EMAIL_ADDRESS>wrote:

@betanalpha That would be confusing because any of the arguments can be random variables and the first one isn't required to be a random variable. It's much more work to call out the specific function (I'd have to write a new rule for the case where someone forgets the vertical bar, then create a custom message for it, then have it fail after it's matched and reported an error.) What I went with produces this:
SYNTAX ERROR, MESSAGE(S) FROM PARSER:
ERROR at line 5
3:    }
4:    model {
5:      target += normal_lpdf(y, 0, 1);
^
6:    }
PARSER EXPECTED: <probability function arguments with first
two arguments separated by a vertical bar;
e.g., foo_lpdf(y | alpha, beta).
This syntax required for functions with suffixes
_lpdf, _lpmf, _lcdf, _lccdf>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Understood, but it's the "anymore" that's problematic.  They are
random variables from the Stan program's perspective.  And if you
talk to Andrew (or read BDA), then everything's a random variable
in Bayesian stats, even the predictors in a regression that come in
as data and get no distribution.  That's why he refuses to let us use
the semicolon notation (the frequentists use that precisely to distinguish
the random variables from other variables). Going against BDA, even with
technical correctness on our side, seems like a mug's game --- it'll just
confuse users and annoy Andrew without much upside.

That’s not quite the argument.
The semicolon is purely frequentist and is meant to avoid
interpreting the likelihood as a conditional probability distribution,
and I agree that this has no place in Bayesian inference (in some
sense the difference between | and ; distills the fundamental
differences between frequentist and Bayesian inference).
Instead the difference is in the definition and use of a random
variable.  When using a conditional probability distribution, any
variables to the left of the | are having their distribution defined
by the conditional distribution.  Any variables to the right of the
|, however, are just be queried by value independent of their
distribution.  Sequential generative modeling with conditional
probability distributions is just a way to isolate how a random
variable is defined and how it is used to influence other random
variables.
In other words, the interpretation of what is a random variable
and what isn’t is being locally scoped just to the definition of
that conditional probability distribution.  I understand how this
would be confusing to users not familiar with probability, and
even to those who are as the error messages would be
referring to this very local scope and not the global scope of
the program where everything is random variable.  It was just
a suggestion.  Ultimately anything we choose is either going to
be ambiguous at some level, inconsistent with the actual math,uses
or confusing to most users.
On Aug 22, 2016, at 11:03 AM, Bob Carpenter<EMAIL_ADDRESS>wrote:

Understood, but it's the "anymore" that's problematic. They are
random variables from the Stan program's perspective. And if you
talk to Andrew (or read BDA), then everything's a random variable
in Bayesian stats, even the predictors in a regression that come in
as data and get no distribution. That's why he refuses to let us use
the semicolon notation (the frequentists use that precisely to distinguish
the random variables from other variables). Going against BDA, even with
technical correctness on our side, seems like a mug's game --- it'll just
confuse users and annoy Andrew without much upside.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


distcheck is broken

@liuqun already pointed out over on the tabrmd that the make dist / distcheck target was broken. No surprise that it's broken here as well.

resolved in 7c25ce729dc8c850f04511bbbb761bcf9fd0e4fb


Visualizations: do not infer dataShape for numeric & date variables to constrain at viz level not at data loading

We need it to be possible to choose continuous variables that have <10 values (or whatever the cutoff for inferring categorical) as the x or y axis variable in scatterplots (and also line plots). For example, we would like to select "Study timepoint" (which has values of 1, 2, 3, 4, &5) for the x-axis in scatterplots in the SCORE S. mansoni Cluster Randomized Trial.

I tested again with GEMS1A, on qa.clinepidb.org
Household wealth index, categorical has 5 unique number values (0, 1, 2, 3, 4, 5)
Traditional healer transport expenses amount has 9 unique number values (0, 45, 70, 100, 200, 250, 300, 2000, 3000)
In the Browse / subset tab, these variables are both showing up in table format (not as a histogram). This is the behavior we wanted.
In the Visualize tab, I tried using Household wealth index, categorical as the x-axis variable.
*Plots that worked:
Scatterplot --> got drawn properly, but there is a lot of overplotting. This is the behavior we expected & want.

Barplot --> looks good

Boxplot --> looks good

Overlay --> looks good

Plots that didn't work:
Histogram --> the loading spinner has been going for 5+ minutes with no progress & no error message (same issue as 2 weeks ago)

@d-callan @jaycolin @dmfalke seems like numeric variables with <10 values are only not working for histograms

This is a web-eda issue, covered by
https://github.com/VEuPathDB/web-eda/issues/491
On Fri, Oct 1, 2021 at 3:30 PM Danica Helb @.***> wrote:

@d-callan https://github.com/d-callan @jaycolin
https://github.com/jaycolin @dmfalke https://github.com/dmfalke seems
like numeric variables with <10 values are only not working for histograms
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/VEuPathDB/web-eda/issues/498#issuecomment-932496968,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACZEUZSRSW755J5TQOGSSDUEYD5DANCNFSM5FE6F7EQ
.

Checking date variables now, using PRISM2:
Death date has 3 unique values 2018-05-13, 2018-10-30, and 2019-01-24
Histogram --> looks good

Bar plot --> looks good

Scatter --> looks good (though overplotted)

Box plot --> looks good

This had an issue
Overlay --> legend doesn't make sense

I'm leaving this ticket in "Failed QA" until https://github.com/VEuPathDB/web-eda/issues/491 is finalized & I can retest. adding paused label now

Dates with <10 values work in all plots
These plots work with numerics with <10 values:

Bar plot (as x-axis var)
scatter plot (as either x-axis var or y-axis var)
as overlay in histogram, bar plot, scatter, and box plot

@ryanrdoherty @d-callan @dmfalke I'm getting 500 errors when i try to plot numerics with <10 unique values:

in the histogram (as x-axis var)

in the boxplot (as y-axis var) --> should we even allow this???


Documentation - Product requirements

This is duplicate with https://github.com/CDCgov/prime-central/issues/65


Version 0.7.2

Landed in https://github.com/metarhia/globalstorage/commit/5ecbd87703a9c9268bfb4cdb09cd602d7d888fbf.


Update version: PicGo.PicGo version 2.3.1

Pull request has been created with Komac v1.8.1 :cherries:
Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/111371&drop=dogfoodAlpha

/azp run

/azp run


Harden QUICHE stream buffer size

StreamEncoder::encodeData() assumes transport layer takes all the data. In QUICHE we achieve this by setting FLAGS_quic_buffered_data_threshold to be large enough (512MB) to buffer all the data under the high watermark plus a flow control window DEFAULT_INITIAL_STREAM_WINDOW_SIZE (256MB). This is a really large buffer for each stream, but we don't expect the upstream will negotiate such large flow control window for all backend services.
Ideally, we should ask the router filter about the negotiated receive flow control window and adjust the down stream send buffer size.
Another thing to watch out for is FLAGS_quic_buffered_data_threshold doesn't apply to QUIC crypto stream and header stream. So their send buffer in theory can grow infinitely.
In IETF QUIC, we don't have header stream, but send headers on data stream. The total stream count doesn't decrement till all the sent data is ACK'ed.  So the total buffer used by the connection is capped by the max concurrent stream limit. In Google QUIC, after connection buffers headers in headers stream with FIN the data stream is considered as closed and decrement the stream count. And the fix should be as easy as add a uppper limit to header stream send buffer and close the connection if it buffers too much data. The IETF implementation is under testing now, so by the time Envoy support QUIC in production, we very likely already switch to IETF one.
Crypto stream doesn't honor FLAGS_quic_buffered_data_threshold either, but unless our handshake implementation has problem, it shouldn't transfer more than 16KB data. But currently QUICHE doesn't have this check on the send side, though we check this limit on the receive buffer. These enforcement needs to be added to QUICHE to make sure the QUICHE buffers are always capped.

To enhance buffer limit on headers stream, it is easier to count bytes buffered in header stream send buffer into connection level buffer in Envoy instead of having its own buffer limit in upstream QUICHE.

Tagging as quic alpha just to make sure we are happy with buffer limits as configured before we un-hide config.
I think some of the work can wait for mvp but will let dan document what goes where :-)

Regarding QUIC crypto stream buffer, we could trust QUICHE handshake implementation wouldn't OOM itself even without send buffer enforcement.
Regarding QUIC header stream buffer, it's send buffer contributes to connection level watermark which is configured via perConnectionBufferLimitBytes.

We are removing support for gQUIC and thus no need to worry about header and crypto stream. But we need to make sure all the H3 streams' send buffers are bounded, i.e. the peer can't make control stream and Qpack send/receive streams to buffer infinitely.

I have constructed tests that I wanted to run and also looked deeper into the integration between Envoy and QUIC. All of my test shave passed, meaning that there I do not currently see a way to abuse QUIC to consume more memory than flow control would allow. It even works for local responses. However I think some missing coverage needs to be improved, without blocking GA. Specifically:

There are no tests that verify that downstream stream limit is working (we do not even know if plumbing from listener config to QUIC codec is working).
There are no integration tests that validate that Envoy's flow control with QUIC protocol works.

It would good to provide a way to override some behavior for the QUIC client, such as allowing it to ignore peer specified stream limit and also set fixed size for the congestion window, such that high watermark levels can be known.
Apart from that this issue can be resolved.


cmd: add support for private repositories in serve restic - fixes #3247

This is WIP to make it easier to discuss changes.

That is looking very good.
What more do you want to add to it?

Tests I guess, just not sure how to write them yet 😃
On Sun, Jun 9, 2019, at 16:02, Nick Craig-Wood wrote:

That is looking very good.
What more do you want to add to it?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/pull/3249?email_source=notifications&email_token=AAAT5C5XFPOWQG3TT3IYL4DPZUEOFA5CNFSM4HVZDAZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXIKVYA#issuecomment-500214496, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAT5CZ5IXTMPS22YFS4SN3PZUEOFANCNFSM4HVZDAZA.

Check out the these tests for some ideas https://github.com/ncw/rclone/blob/master/cmd/serve/restic/restic_appendonly_test.go

@ncw I have added tests as good as I could. One thing that I noticed though: Passing the username ala r.Header.Set("Remote-User", username) is pretty much dangerous since a user could probably override by manually sending this header. I do not see how the current code would allow for this, but it seems massively dangerous! http headers are not to be trusted after all :)

I mean technically we are only setting it after we verified the user and only then ever call the old handler, but it still feels wrong. Is there nothing else we can use to pass a "context" on or so?

I have pushed a follow up commit which uses go contexts to pass the information along, this should be 100% safe

One thing to note: My tests only test the restic handler, I couldn't figure out how to get access to the httplib handler to test the whole basic auth flow.

And the following change:
diff --git a/cmd/serve/restic/restic.go b/cmd/serve/restic/restic.go
index c5543fcc2..28c560947 100644
--- a/cmd/serve/restic/restic.go
+++ b/cmd/serve/restic/restic.go
@@ -215,10 +215,20 @@ func (s *server) handler(w http.ResponseWriter, r *http.Request) {
        remote := makeRemote(path)
        fs.Debugf(s.f, "%s %s", r.Method, path)

-       v := r.Context().Value(httplib.ContextUserKey)
-       if privateRepos && (v == nil || !strings.HasPrefix(path, "/"+v.(string)+"/")) {
-               http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
-               return
+       if privateRepos {
+               // Check against the cert CN if client cert auth is enabled
+               if s.Opt.ClientCA != "" {
+                       if !strings.HasPrefix(path, "/"+r.TLS.PeerCertificates[0].Subject.CommonName+"/") {
+                               http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
+                               return
+                       }
+               } else { // Otherwise check the username from basic auth
+                       v := r.Context().Value(httplib.ContextUserKey)
+                       if v == nil || !strings.HasPrefix(path, "/"+v.(string)+"/") {
+                               http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
+                               return
+                       }
+               }
        }

        // Dispatch on path then method

would most likely allow validation against the client cert's CN -- but I did not test that.

@ncw I have added tests as good as I could. One thing that I noticed though: Passing the username ala r.Header.Set("Remote-User", username) is pretty much dangerous since a user could probably override by manually sending this header. I do not see how the current code would allow for this, but it seems massively dangerous! http headers are not to be trusted after all :)

We could make sure the header is "" when auth fails... Point taken however...

I have pushed a follow up commit which uses go contexts to pass the information along, this should be 100% safe

That is probably the proper way of doing it.  I did consider that when writing my original advice to set "Remote-User" however I thought it would be a lot more complicated - I didn't know about the Request.WithContext method!

One thing to note: My tests only test the restic handler, I couldn't figure out how to get access to the httplib handler to test the whole basic auth flow.

That should probably be in the (non existent ;-) httplib tests anyway!

If you are okay with my latest changes I'll squash the commits.

Yes I think they look great.  If you are happy with them then I'll merge.

And the following change:
...
would most likely allow validation against the client cert's CN -- but I did not test that.

I wonder whether it might be better to set the Context key in httplib than check it here?  That would save code duplication.
Shall we leave that to a separate PR?  Or do you want to get it in this one?

I didn't know about the Request.WithContext method!

me neither, everything is new for me there :)

Shall we leave that to a separate PR? Or do you want to get it in this one?

 I squashed and the commit should be ready to go.

Indeed, possible but unlikely? I think as long as we document which one takes precedence then it would be fine.

I squashed and the commit should be ready to go.

Thank you for this excellent piece of work - much appreciated :-)
Will merge now.


(Bug) Minted-Multitier: allow owner to finalize before end of crowdsale if all tokens were sold in latest tier

Closes #1026

Added new validation related to latest tier

Black Duck Security Report
Merging #1054 into 2.0 will not change security risk.
Click here to see full report


Add resource quota on critical priority class

Description
A good recommendation for critical pods is to use system-cluster-critical and in the case of dapr-share system-node-critical. But this can collide with admin limitations on the cluster. For example in GKE by default one cannot create pods with these priorityClassName if not in kube-system (this was the default before ResourceQuota for priority class was added).
This PR creates a ResourceQuota when one of these two default high priority classes are added.
These classes should also be recommended as best practice when using Dapr (a future PR for docs after this PR is done).
Issue reference
https://github.com/dapr/dapr/issues/7783
Please reference the issue this PR will close: #[issue number]
Checklist
Please make sure you've  completed the relevant tasks for this PR, out of the following list:

[ ] Code compiles correctly
[ ] Created/updated tests
[ ] Unit tests passing
[ ] End-to-end tests passing
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

@holopin-bot @filintod Enjoy your Dapr badge!


Add inspirational call-to-action

with the appropriate styling
This needs to be tested; I haven't gotten website running locally yet.
but I might forget about this if I don't make the PR

@jkwok91 Great call to action!

actually it's @jonleung's call to action :smile:

lgtm


[feature] Add gentoo

https://github.com/ryanoasis/vim-devicons/blob/master/plugin/webdevicons.vim#L105

PR is welcome 👍

I know expand('$OSTYPE') ==# 'linux-android' will return truein nvim on android. I think it can be utilized to add an android icon . but I don't sure if it is a good idea (don't use lsb_release).

$PREFIX !=# '/data/data/com.termux/files/usr' will be better because $OSTYPE is an environment variable which only is know by shell and vim must use expand to know it.


Remove version: Jackett.Jackett version 0.20.3004

404 - release deleted
Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/Jackett/Jackett/releases/download/v0.20.3004/Jackett.Installer.Windows.exe
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 918)


[QUESTION] Confusion with BsonCtorAttribute

The documentation in respect of the BsonCtorAttribute says..

Please note that all the parameters in the constructor annotated with BsonCtorAttribute must be of a simple type, BsonDocument or BsonArray.

So I was expecting the following code to work
[BsonCtor]
public Foo(BsonDocument doc)
{
        //Instantiate class from doc
}

But doc is always null
I can get simple types to work, just not the former.
What am I missing ?

@ordinaryorange This feature works by trying to match fields in the document with parameters in the constructor annotated by BsonCtor. For example, if your constructor has the signature public Foo(int id, string name) and your document has fields _id and Name, the value of these fields will be matched with the corresponding parameter in the constructor call (name matching is case-insensitive).
If you want to control how your object is deserialized from a document, you should use a custom deserializer.

So the documentation appears to be incorrect then ?
In the quoted sentence I provided, "BsonDocument and BsonArray" should be deleted therefore ?

@ordinaryorange You can use BsonDocument and BsonArray to populate lists and sub-objects. Example:
public class SubEntry
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class SuperEntry
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class Entry
{
    public int Id { get; private set; }
    public string Name { get; private set; }
    public SuperEntry SuperEntry { get; private set; }
    public List<SubEntry> SubEntries { get; private set; }

    [BsonCtor]
    public Entry(int id, string name, BsonDocument superEntry, BsonArray subEntries)
    {
        var mapper = BsonMapper.Global;
        Id = id;
        Name = name;
        SuperEntry = mapper.Deserialize<SuperEntry>(superEntry);
        SubEntries = subEntries.Select(x => mapper.Deserialize<SubEntry>(x)).ToList();
    }
}

It would be really nice if the documentation had some examples like this for the BsonCtor.


Monte Carlo forecasts report

GitHub Issue: #2647
This pull requests adds a toolkit report that generates Monte Carlo forecasts based on the user's historical inflows and outflows.
The chart displays five forecasts, each corresponding with confidence levels 10%, 25%, 50%, 75%, and 90% respectively.
The algorithm generates 100 scenarios, each 10 years in length, by randomly selecting one-week periods from the user's transaction history, thereby creating 100 unique potential futures for the user based on their past behavior.
It then sorts these scenarios based on the final net worth descending. In this way, by taking the 10th returned scenario, only 10% of generated scenarios resulted in that net worth or higher, corresponding to a 10% confidence level.

This looks good! Folks have been looking to these sorts of reports so let's get it in!

OMG! I love this.  Just signed into my YNAB this morning to see the new Monte Carlo feature.  For a single scenario: does it order the random weeks chronologically or is it completely random ordering of the weeks within a scenario?

This is an awesome feature! My only issue with the current implementation is that it's basing you starting net worth on the filterableTransactions instead of allReportableTransactions. I've used YNAB for several years, but my income and spending habits have drastically changed over those years based on numerous life events. I'd like to pick the range I want to use for the simulation (probably the last year or so), but use my actual current net worth. But since it uses filterableTransactions to calculate the net worth, the only way I can get my starting net worth in the graph is by using my entire history of transactions. That's why I propose using allReportableTransactions instead for calculating net worth and then using filterableTransctions for the simulation.

Hey, for folks like me that have no clue what a Monte Carlo forecast is, this report is bad from a user experience perspective. All other reports are somewhat intuitive, this one is not. I don't know what any of these plots mean and I have no way to look what they are - they have no meaningful labels, they have no help, the name of the report doesn't help either, the on-hover has no clear clue of what it means. I can't tell if the forecast is good or bad, I can't tell anything looking at it. And I had to go out of my way just to find this pull request and that's the only reason I know it's a Monte Carlo forecast.
So, I believe that once I learn what a Monte Carlo forecast is, I might come to understand how to appreciate this report, but most folk won't even know what a git something something is. Or have the curiosity to get this far just to get the name of the report.
Here are some ideas that might improve user experience

Change the report name to "Monte Carlo Forecast" or something that would lead users to a proper google search
Explain a little about what this report means (maybe hijack the ? button), even if it's somewhere external to YNAB
Add some sort of explanation for the graph labels

@kceb1995 It's completely random within a scenario.
@ssartell Hmm, yes. I'm not sure what the right answer is there. I was hoping by implementing it the way I did you could forecast a single budget category. But I see your point, too, so unsure.
@mobius3 All good points! Unfortunately I'm no longer using YNAB for my own budgeting. Does anyone know if there might be a dummy account I could use for further development?

@narthur I actually put in a PR to address the issue I called out.

@ssartell Ok! I'll repeat myself over there. Thank you for that!

Thanks for this awesome feature. I'm not sure if there is an additional setting that I need to adjust but the currency for the forecast is displayed in dollars even though I am using a different currency. Is there any way to fix this on my side? Thank you very much. :)

@iron59 PR #2703 should fix that issue.

I'm not even clear on what exactly this report is forecasting. My current guesses are 1) net worth and 2) change in net worth, but neither of those match up with the values I'm seeing for start points (far left values) which don't correspond to the current net worth value nor seem to represent a reasonable values for change in net worth.

The current implementation takes all transactions within the date range and categories specified, sums the total to get a starting point, and then randomly samples previous weeks within the date range to create 100 projections. Then it displays the percentiles. I made a pull request that starts with your current total net worth, but there was some conversation about the starting point being the total allocated to selected categories. I was using it to project total net worth, but the creator intended it to project categories.

@ssartell Thank you for elucidating that for me. (At first blush, I think your pull request would have been sensible. As it is, I don't have a clear idea of how I would use the report. That said,I haven't heard the countervailing arguments.)


fix: specify snowflake-connector-python version

After conversation with @czgu in slack, I found out that snowflake-connector-python should be specified in order to prevent an error.
Previously, snowflake-connector-python was version 2.4.5 in Querybook, and it created this error:
(certifi 2021.5.30 (/usr/local/lib/python3.7/site-packages), Requirement.parse('certifi<2021.0.0'), {'snowflake-connector-python'})
By specifying version to 2.6.1, everything worked fine in my production environment. This version was selected after reviewing the version that was used by Apache Superset.
Currently we only use snowflake-sqlalchemy for our requirements, but since snowflake-connector-python is included as a dependency, we should also specify the version to prevent any issues.
Related Issue: https://github.com/snowflakedb/snowflake-connector-python/issues/732

have you tried to bump snowflake-sqlalchemy version instead? if that doesn't work can you add a comment above
snowflake-connector-python==2.6.1

to explain that it is a transitive dependency and link this github issue

I doubt that bumping up would fix the issue since it would install any version less than 3.0.0
(https://github.com/snowflakedb/snowflake-sqlalchemy/blob/6db57555263147abb7d80cda81b1fc05ad634d9a/setup.py#L38)
I can add a comment for explanation. Thank you.


Notebook Markdown/HTML  tag broken when nested

Environment data

VS Code version: 1.51.0
Extension version: v2020.10.332292344
OS and version: Win10 1909
Python version: Python 3.8.3 (64bit) (w/ anaconda)
Anaconda Version:  conda 4.8.5
Type of virtual environment used: conda
Relevant/affected Python packages and their versions: N/A
Relevant/affected Python-related VS Code extensions and their versions: N/A
Value of the python.languageServer setting: "Pylance"

Constant Info
Due to issue/feature-request #13871 I am having to use HTML tags in certain places in order to accomplish certain formatting that is normally possible in Jupiter notebook/lab & Google colab.
In this case I am trying to make a web link clickable while being formatted inside of some  tags.
Since while URL's not surrounded by HTML tags are clickable in markdown syntax by default
(i.e. without having to use the [...](...) markdown syntax)
they are not clickable when surrounded by HTML tags.
Therefore I used an  tag to make it clickable, resulting in the code below:

    The DTM for w#x where w is a substring of x.

    (See the definition of substring and subsequence at
    https://en.wikipedia.org/wiki/Substring)

Expected behavior
First of all I would prefer to be able to just do something like this:

  The DTM for w#x where w is a substring of x.

  _(See the definition of substring and subsequence at https://en.wikipedia.org/wiki/Substring )_

But regardless it should end up looking something like this:

Except the font size should respect what is in the HTML  tags.

Actual behavior
It does this weird thing:

As you can see it somehow replaces the contents of the  tag with the corresponding markdown syntax, which is just plain strange.

Steps to reproduce
Just nest an  tag inside some other html code,
(I gave my code before in the Constant Info section above).
Logs

N/A

Thanks for reporting this @osterhoutan, I'm able to replicate this behavior based on your code sample. We'll discuss prioritization of a fix at our weekly triage meeting.


Doubt topologies

I have this topology:
(ns pmy-FixEvent
  (:use     [streamparse.specs])
  (:gen-class))

(defn pmy-topology [options]
   [
    ;; spout configuration
    {"KafkaReaderSpout" (python-spout-spec
          options
          "spouts.KafkaSpoutFix.KafkaReaderSpout"
          ["values"]
          :p 1
          )
    }
    ;; bolt configuration
    {"FixEvent" (python-bolt-spec
          options
          {"KafkaReaderSpout" :shuffle}
          "bolts.FixEventBolt.FixEvent"
          ["data"]
          :p 2
          )

      "FixHadoop" (python-bolt-spec
          options
          {"FixEvent" :shuffle}
          "bolts.FixHadoopBolt.FixHadoop"
          ["data"]
          :p 2
          )
    }
  ]
)

what am I looking this workflow:
KafkaReaderSpout(read from kafka) => FixEvent(Proces this events)=>FixHadoop('write this event in hadoop').
Not working :-1: . If i take a bolt, work, but not the necessary flow.
Any idea?
Thank!

Could you please provide a little more info? How is it "not working"? Are you getting an exception? If so, could you please provide it?


Bumpversion behaves incorrectly when files have been formatted using black

Bumpversion incorrectly updates version="0.4.0" to version="version='0.5.0'" when files are autoformatted using black

Ok, I just changed the search string in setup.cfg to contain double-quotes. It works now!


Problem of group_keypoints

@Daniil-Osokin Thanks for your great job! I need some help for training custom dataset.I have trained a horse dataset with 22 keypoints and I set 10 pairs of paf.I show the heatmap of each paf and the keypoints which is predicted by iter_19500.pth.I set the min_paf_score is 0 and the success_ratio is 0.1 but some limbs still failed.

Hi! I did not get which pafs you are using exactly (between which keypoints). However, I believe that it might be hard for the network to learn paf over background, e.g. in the last image, the horse ankle is connected with horse hoof, and so on (in this example paf goes over background). This might be a reason that grouping of some parts fail.

Hope, it is helped.

I'm so sorry to reply you so late,I have solved this problem,I found the setting of pafs are error by debug. I only set the horse ankle connect to the hoof before for reducing the limbs connection.

Great to hear it, good luck!

@yoletPig can you share the code you used to display the heatmaps and paf, would really appreciate that, thanks!

my test code
`import cv2
import numpy as np
import torch
from matplotlib import pyplot as plt
from torch.utils.data import DataLoader
from torchvision import transforms
from openpose.datasets.coco import CocoTrainDataset
from openpose.datasets.transformations import Scale, Rotate, CropPad
from openpose.models.with_mobilenet import PoseEstimationWithMobileNet
from openpose.modules.load_state import load_state
from openpose.val import infer, heatmap2d
def heatmap2d(arr: np.ndarray, image_constant, id_of_paf):
plt.imshow(image_constant)
plt.imshow(arr, cmap='viridis', alpha=0.6)
plt.title(str(id_of_paf))
plt.colorbar()
plt.show()
net = PoseEstimationWithMobileNet(num_refinement_stages=3, num_heatmaps=23, num_pafs=20)
checkpoint = torch.load("default_checkpoints/checkpoint_iter_11000.pth")
load_state(net, checkpoint)
net.cuda()
prepared_train_labels = "prepared_train_annotation.pkl"
train_images_folder = "../../mmpose_data/Horse-10/"
stride = 8
sigma = 7
path_thickness = 1
dataset = CocoTrainDataset(prepared_train_labels, train_images_folder,
stride, sigma, path_thickness,
transform=transforms.Compose([
Scale(),
CropPad(pad=(128, 128, 128))
])
)
train_loader = DataLoader(dataset, batch_size=32, shuffle=False, num_workers=8)
plot_index = 0
for batch_data in train_loader:
plot_index += 1
if plot_index < 20:
continue
images = batch_data['image'].cuda()
keypoint_masks = batch_data['keypoint_mask'].cuda()
paf_masks = batch_data['paf_mask'].cuda()
keypoint_maps = batch_data['keypoint_maps'].cuda()
paf_maps = batch_data['paf_maps'].cuda()
index = 31
image = images[index].detach().cpu().numpy()
image = image.transpose((1, 2, 0))
image = image * 256 + 128
# avg_heatmaps, avg_pafs, pafs = infer(net, image, scales=[1], base_height=368, stride=stride)
# image = images[index].detach().cpu().numpy()
# image = image.transpose((1, 2, 0))
# image = image * 256 + 128
paf_map = paf_maps[index].detach().cpu().numpy()
keypoint_map = keypoint_maps[index].detach().cpu().numpy()
# paf_map = cv2.resize(paf_map[19], dsize=None, fx=stride, fy=stride, interpolation=0)
# image[:, :, 0] = image[:, :, 0] * (paf_map5 + 1)
# image[:, :, 1] = image[:, :, 1] * (paf_map5 + 1)
# image[:, :, 2] = image[:, :, 2] * (paf_map*5 + 1)
# cv2.imwrite("Figure.jpg", image)
# paf_map = avg_pafs[:, :, 1]
# paf_map = np.logical_and(paf_map, paf_map > 0).astype(np.int8)
# paf_map = avg_pafs[:, :, 19] * paf_map
# image[:, :, 0] = image[:, :, 0] * (paf_map * 10 + 1)
# image[:, :, 1] = image[:, :, 1] * (paf_map * 10 + 1)
# image[:, :, 2] = image[:, :, 2] * (paf_map * 10 + 1)
# cv2.imwrite("Figure.jpg", image)
image = image.astype(int)
for idx_paf in range(20):
paf_map_i = cv2.resize(paf_map[idx_paf], dsize=None, fx=stride, fy=stride, interpolation=0)
heatmap2d(paf_map_i, image, idx_paf)
htmp = np.sum(keypoint_map[:22,:,:], axis=0)
pafm = np.sum(paf_map, axis=0)
htmp = cv2.resize(htmp, dsize=image.shape[:2],  interpolation=0)
pafm = cv2.resize(pafm, dsize=image.shape[:2],  interpolation=0)
fig = plt.figure(figsize=(8, 6))
ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
ax1.title.set_text('Heatmap ' + str(1))
ax1.imshow(htmp, cmap='RdYlBu')
im1 = ax1.imshow(image, alpha=0.4)

fig.colorbar(im1, ax=ax1, orientation='vertical')

ax2.title.set_text('PAF' + str(1))
ax2.imshow(pafm, cmap='RdYlBu')
im2 = ax2.imshow(image, alpha=0.4)

fig.colorbar(im2, ax=ax2, orientation='vertical')
plt.show()
print()
break

`

Hello, I have some questions about training custom datasets, can you price contact information?
Or answer online

I'm currently doing pose tests on pigs, using 20 key points. The following problems exist:

The mapping between BODY_PARTS_KPT_IDS and BODY_PARTS_PAF_IDS is not clear. I'd like to ask you something.
I modified part of the code according to my own data set in accordance with the training process. Then run
python train.py --train-images-folder /train2017/ -- preparation-train-labels prepared_train_annotation.pkl --val-labels val_subset.json --val-images-folder /val2017/ --checkpoint-path /mobilenet_sgd_68.848.pth.tar --from-mobilenet

When    (train.py  132#)
evaluating (val_labels, val_output_name, val_images_folder, net), avg_heatmaps, avg_pafs = infer(net, img, scales, base_height,
The value in avg_heatmaps is very small, close to 0.0001
In total_keypoints_num += extract_keypoints(avg_heatmaps[:, :, kpt_idx], all_keypoints_by_type, total_keypoints_num)
After heatmap[heatmap < 0.1] = 0, the values of avg_heatmaps are all zeros.
Eventually, the detections.json file is empty, resulting in an error like #304!
求大佬指教啊

你要修改coco eval方法里面的sigma值，默认只有17个关键点

你可以加我微信13025811310


Finite field arithmetic

Currently tinygrad mainly is used for floating-point arithmetic. I would like to use tinygrad for fast polynomial arithmetic with coefficients in a finite field, but this is very hard to do right now. Are there any plans to support this?

Not an issue, and not a concrete feature request


rails_best_practices (1.15.7) is not compatible with psych 2.0.13

Needs details and Ruby version

I'm also seeing this in Ruby 2.1.5
/Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/core/checks_loader.rb:22:in `checks_from_config': undefined method `load_file' for Psych:Module (NoMethodError)
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/core/checks_loader.rb:27:in `load_checks_from_config'
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/core/checks_loader.rb:11:in `load_lexicals'
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/core/runner.rb:48:in `initialize'
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/analyzer.rb:54:in `new'
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/analyzer.rb:54:in `analyze'
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/lib/rails_best_practices/command.rb:136:in `<top (required)>'
	from /Users/prokizzle/.rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /Users/prokizzle/.rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /Users/prokizzle/.gem/ruby/2.1.5/gems/rails_best_practices-1.15.7/bin/rails_best_practices:6:in `<top (required)>'
	from /Users/prokizzle/.gem/ruby/2.1.5/bin/rails_best_practices:23:in `load'
	from /Users/prokizzle/.gem/ruby/2.1.5/bin/rails_best_practices:23:in `<main>'

@prokizzle I tried ruby 2.2.2p95 and psych 2.0.13, didn't see such error


Scheduled artifact transform build operations for execution and their nodes in the execution plan

Summary of the changes:

Task graph build operation result is extended with a more generic execution plan of PlannedNodes that for now can be either a task or a transform.
Scheduled transform execution build operation is extended with additional info that eventually is shown to the users.

Historically, the task graph computed from the calculated execution plan contained only PlannedTask nodes. However, the execution plan also contains other nodes that may be of interest, such as nodes related to artifact transforms. More specifically, TransformationNode is a node in the execution plan that corresponds to a scheduled transform. Such a node can take considerable build time, previously it will not show up in the execution plan exposed in the build operations.
In order to make artifact transforms better analyzable, their detail information is now included into the relevant build operations.
Similarly to task nodes, transformation nodes cannot be identified only by their properties. As such, transformation nodes now also contain a unique opaque identifier that serves as a tiebreaker in terms of the transformation identity.
Unlike the tasks, transformations require much more properties to get identified reasonably:

(consumer) buildPath

We've settled on the Build Operations API. The rest of the implementation is Gradle-internal, so we'll do follow-up PRs to make quality of life improvements and add more test coverage then.

@bot-gradle test and merge

I've triggered a build for you.


Remove user's names from drain info window

Problem
We realized tonight that showing the name of the person who adopted a drain might be a security risk because generally the person lives within a couple of houses.
Resolution
Remove the "By: FirstName LastName" line from both info windows - the "adopted thank you" window and the "adopted" window.

This may have been a drive-by requirement that never made it to an issue.  This is definitely an LGROW decision. I would like the name to be kept as part of the account data. I believe we can use it to help out with deterring profanity in the drain names.

@Wilfongjt This issue might be resolved by #253

same as #255


Remove refined from circe/circe

This has been moved to circe/circe-refined
Fixes #2276

@zarthross please review again, I updated the readme.


extended collections : Add a materialize async method

Should trigger the materialization process using collectStream and equivalent (new) reactive fold operator.
This will allow collections to be populated asynchronously before 1st access.

Released in 2.1.0


GCP Bucket images previews do not appear in v1.7.2 but work fine in v1.7.1

Describe the bug
We've synced with a GCP Source storage successfully but image preview do not appear in v1.7.2 - downgrading back to v1.7.1 and image previews work just fine.
To Reproduce
Steps to reproduce the behavior:

Add a Cloud Storage source
Sync with Source
Go back to main screen

Expected behavior
We expect image previews to appear in v1.7.2 instead a missing image and console shows many errors reading gs://xxxx/image.jpg
Environment (please complete the following information):

OS: Ubuntu
Label Studio Version 1.7.2

Additional context

Same problem

@jvence @plannaAlain Could you make a screenshot with an opened network panel where we can see request statuses?
E.g.:

If you find any red lines there, you can open a specific request and send its body screenshot too, .e.g:

When using v1.7.2dev, the photo urls are:
gs://test-external-asset/0199544B-CC2B-4B7A-957F-DB02600BB61D.jpeg net::ERR_UNKNOWN_URL_SCHEME
When downgrading to v1.7.1, the urls is:
https://storage.googleapis.com/test-external-asset/0199544B-CC2B-4B7A-957F-DB02600BB61D.jpeg?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxxxt&X-Goog-Date=20230309T133017Z&X-Goog-Expires=60&X-Goog-SignedHeaders=host&X-Goog-Signature=xxxx
Somehow v1.7.2 is NOT translating the URLs appropriately even though the setup/configuration is the same on both version.

@jvence Could you show a screenshot with page Project => Settings => Cloud storages?
Also I need a screenshot with storage settings.

Have been debugging this this as well. The issue is in https://github.com/heartexlabs/label-studio/blob/3de2ace9b53cf3ab213054a9106079aaf61796a7/label_studio/io_storages/gcs/models.py:
    def _get_signing_credentials(self):
        cache_key = f'{self.google_application_credentials}'
        signing_credentials = credentials_cache.get(cache_key)
        if signing_credentials is None or signing_credentials.expired:
            signing_credentials, _ = google.auth.default(['https://www.googleapis.com/auth/cloud-platform'])
            auth_req = google.auth.transport.requests.Request()
            signing_credentials.refresh(auth_req)
            credentials_cache[cache_key] = signing_credentials
        return signing_credentials

Whatever is in your google_application_credentials, Label Studio always takes the ADC. This is the only place where the cache is set.

Try running LS this way:
fflag_feat_back_dev_4166_google_project_id_11012023_long=0 label-studio

it seems this happened here:
https://github.com/heartexlabs/label-studio/pull/3453/files

Hi @plannaAlain, this is an env variable you need to set. You can, for example, run that command before running label-studio if you are in local:
export fflag_feat_back_dev_4166_google_project_id_11012023_long=0
Or set it in your cloud instance by finding how to set an env variable.

Working on this issue in the ticket LSDV-4902.

Seems this issue still exists in 1.7.5.dev - using export fflag_feat_back_dev_4166_google_project_id_11012023_long=0 does not seem to work.

@plannaAlain @nlassaux @GBPeters @jvence could you please check how GCS works in this PR
https://github.com/heartexlabs/label-studio/pull/4189
?

@plannaAlain @nlassaux @GBPeters @jvence One more question: do you have any Application Default Credentials (ADC)? E.g. environment variable GOOGLE_APPLICATION_CREDENTIALS is set OR gcloud auth application-default login OR your code is running on GAE or GCE instances where credentials provided by the Google App Engine or Google Compute Engine metadata server?
This might happen because ADC settings overrides your credentials from LS Project => Cloud Storages.

We have GOOGLE_APPLICATION_CREDENTIALS set.

We tested PR #4189 with no success -
Steps:
1- Removed the value of placeholder in label_studio/io_storages/gcs/form_layout.yml
2- export GOOGLE_APPLICATION_CREDENTIALS=keys.json
3- Tested that authentication works in project Settings | Cloud Storage | Edit (it can connect just fine)
but images urls are still: gs://...

@plannaAlain If you reset it to empty, LS GCS should work. But also you can try my PR. It fixes this problem.

Are you sure these credentials keys.json are correct? If you use my PR and set google credentials inside of UI instead of environment variable, doesn't it work? Is it possible to have a quick call with you, seems it will be much faster to debug it.

@plannaAlain Please check my PR again, now it should work.

PR is merged.

I updated to the latest develop image and the issue is still there. I can't load images on 1.8.1, but I can on 1.7.1.
{
  "release": "1.8.1dev",
  "label-studio-os-package": {
    "version": "1.8.1dev",
    "short_version": "1.8",
    "latest_version_from_pypi": null,
    "latest_version_upload_time": null,
    "current_version_is_outdated": false
  },

  "label-studio-os-backend": {
    "message": "fix: LSDV-5131: Fix Submit button disabled state (#4233)",
    "commit": "28b62696517b35b8870e742ce0f448491637e2bc",
    "date": "2023/05/24 10:40:53",
    "branch": "develop",
    "version": "20230524.104326-develop-28b62696"
  },

  "label-studio-frontend": {
    "message": "fix: LSDV-5131: Fix Submit button disabled state",
    "commit": "4c163b42a8695e1e55f1bec4617d45d11c50ae15",
    "branch": "master",
    "date": "2023/05/24 11:32:20"
  },

nevermind, it's working now. Something must have changed with CORS between 1.7.1 and later versions since I wasn't getting CORS errors before. After following the guide https://cloud.google.com/storage/docs/using-cors?cloudshell=false#gsutil to apply a CORS configuration to both source and target buckets things are working again. Thanks!
this was the CORS file I used
cors.json
[
    {
        "origin": [
            "*"
        ],
        "responseHeader": [
            "Content-Type",
            "x-goog-resumable"
        ],
        "method": [
            "GET",
            "HEAD",
            "DELETE",
            "POST",
            "PUT"
        ],
        "maxAgeSeconds": 3600
    }
]

nevermind, it's working now. Something must have changed with CORS between 1.7.1 and later versions since I wasn't getting CORS errors before. After following the guide https://cloud.google.com/storage/docs/using-cors?cloudshell=false#gsutil to apply a CORS configuration to both source and target buckets things are working again. Thanks!
this was the CORS file I used
cors.json
[
    {
        "origin": [
            "*"
        ],
        "responseHeader": [
            "Content-Type",
            "x-goog-resumable"
        ],
        "method": [
            "GET",
            "HEAD",
            "DELETE",
            "POST",
            "PUT"
        ],
        "maxAgeSeconds": 3600
    }
]

Tried the cors hack with no success. Testing connection works fine but nothing is synced. Could it have to do with the type of annotation files. There seems to be both .XML and JSON files in the annotation bucket. Could this be causing the issue?

Hey! This still happens when using default credentials, but we were able to avoid it by generating a json key and loading it directly.


[pubsub/indexer] Support for bigInt and bigFloat

Closes #769

PR checklist

[ ] Tests written/updated
[ ] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments

Superseded by #797 .


Strange results after computing lcmv in volume source space

Dear MNE users,
I'm attempting to compute lcmv beamformer on evoked data from a simple motor (finger tapping) task. I can compute and plot surface-based estimates with no problem, which gives me nice, sensible-looking activation maps. However, when I try to compute the stc in volume source space I get very bizarre results with a clear, circumscribed border of activation (see attached picture). The attached picture was generated using nilearn plot_stat_map, however I don't think it's a plotting issue, because when I export the stc as a .nii file I see the same thing in FSL.

Has anyone encountered this issue before? any advice would be appreciated!

@britta-wstnr any ideas?

Hi @lbailey25
Did you contrast conditions in your pipeline and/or use some kind of weight normalization (weight_norm='unit-noise-gain' or 'nai')?
Without knowing more about it your results look like a center-of-head bias.

Hi @britta-wstnr
Thanks for the reply. No, I didn't contrast conditions or use weight normalization. Below is the code that I used - perhaps it will shed some light on the problem
# Compute volume source space:
vol_src = setup_volume_source_space(subject=subject, bem=fname_bem, surface=None,
                                    mri=fname_aseg, mindist=5.0, exclude=0.0)

# Compute forward solution:
vol_fwd = fwd = make_forward_solution(fname_evoked, fname_trans, vol_src, fname_bem, mindist=0, meg=True, eeg=False)

# stc in volume source space:
filters = make_lcmv(evoked.info, vol_fwd, data_cov, noise_cov=noise_cov, pick_ori=None)
vol_stc = apply_lcmv(evoked, filters)
vol_stc = vol_stc.as_volume(src = fwd['src'], dest='mri')

Hope that provides some useful information!

@britta-wstnr the default of make_lcmv is weight_norm='unit-noise-gain' and it is not set above, so it should be normalized.
We should look into this at the sprint.

@lbailey25 does this problem still exist? If so, can you share the data so we can have a look?
CC @wmvanvliet @ckiefer0

@britta-wstnr pushed some fixes that probably fix this, I'll close but @lbailey25 please check and let us know if it's better (we'll reopen if it's not!)


ROS2 Dashing Support

Is there any chance that this package could be used with Dashing as well as Humble? Right now when I try to build the package, I get a lot of error messages related to RCLCPP, which seems to have changed significantly between ROS2 versions.

Hi @sofiakwok
There is no plan to have a Dashing version, but the errors seem trivial to fix. Use RCLCPP_INFO macro and change the stream style to printf style. For example:
RCLCPP_INFO_STREAM(get_logger(), "System [" << get_name() << "] started with topics:");

to
RCLCPP_INFO(get_logger(), "System %s started with topics:", get_name().c_str());

and so on

I've managed to get a version of this successfully building, thanks!


call SSL_CTX_new in diffenent thread, received signal SIGSEGV

I use my own compiled OpenSSL 3.0.12, and run the the following C++ code
int main() {
    bool flag = false;

    std::thread thd1([&flag] {
        while (true) {
            while (!flag) continue;
            SSL_CTX_new(TLS_client_method());
            sleep(1);
        }
    });
    thd1.detach();

    std::thread thd2([&flag] {
        while (true) {
            while (!flag) continue;
            SSL_CTX_new(TLS_client_method());
            sleep(1);
        }
    });
    thd2.detach();

     sleep(2);
    flag = true;

    while (true) sleep(1);
}

I got a error Segmentation fault.
the callstack is
received signal SIGSEGV, Segmentation fault.
3368    ./malloc/malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ffff751f3fe in __GI___libc_free (mem=0x395de6a1517c3c1e) at ./malloc/malloc.c:3368
#1  0x00007ffff7aed350 in EVP_KEYMGMT_free () from libcrypto.so.3
#2  0x00007ffff7afa5fd in EVP_PKEY_CTX_free () from libcrypto.so.3
#3  0x00007ffff7db41b4 in ?? () from libssl.so.3
#4  0x00007ffff7da9f89 in SSL_CTX_new_ex () from libssl.so.3
#5  0x00005555555b0e00 in operator() (__closure=0x555555c8f548) at test.cpp:275

If you could build your library with debugging enabled, it would provide more information as to what might have gone wrong, though by the looks of it, it would seem that an invalid memory address was passed to EVP_KEYMGMT_free.
That said, the reproducer above isn't really a good use case.  It leaks memory on every call to SSL_CTX_new, and so it may be possible that you just hit a memory exhaustion case.
FWIW, however, I built your test case on the latest master, and it seems to run indifinately, so it may be worth trying a bisect on master to see what fixed it.

I also built 3.0.12 this morning, but seem to be unable to reproduce your error with the above test case.  My build/exec commands are:
g++ -I/home/nhorman/git/worktrees/openssl/include -L/home/nhorman/git/worktrees/openssl -g -pthread -std=c++11 ./test.cc -lcrypto -lssl

LD_LIBRARY_PATH=$PWD ./a.out

Thanks! @nhorman
To avoid some unnecessary misunderstandings, I have made some modifications to the code:
#include <unistd.h>
#include <thread>
#include <openssl/ssl.h>

void Test1() {
    bool flag = false;

    std::thread thd1([&flag] {
        while (!flag) continue;
        SSL_CTX_new(TLS_client_method());
    });
    thd1.detach();

    std::thread thd2([&flag] {
        while (!flag) continue;
        SSL_CTX_new(TLS_client_method());
        sleep(1);
    });
    thd2.detach();

    sleep(2);
    flag = true;

    while (true) sleep(1);
}

int main(int argc, char const* argv[]) {
    Test1();
    return 0;
}

I have compiled a debug version of OpenSSL so that more detailed call stack information can be displayed during debugging.
I ran the above code and the result is as follows:
Starting program: test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7465640 (LWP 4102669)]
[New Thread 0x7ffff6c64640 (LWP 4102670)]

Thread 2 "test_openssl-1." received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7465640 (LWP 4102669)]
0x00007ffff77b1b3e in getrn (lh=0x7ffff0000bd0, data=0x7fffe8000ff0, rhash=0x7ffff7464bb8) at ../src/crypto/lhash/lhash.c:307
307             if (n1->hash != hash) {
(gdb) bt
#0  0x00007ffff77b1b3e in getrn (lh=0x7ffff0000bd0, data=0x7fffe8000ff0, rhash=0x7ffff7464bb8) at ../src/crypto/lhash/lhash.c:307
#1  0x00007ffff77b13a5 in OPENSSL_LH_insert (lh=0x7ffff0000bd0, data=0x7fffe8000ff0) at ../src/crypto/lhash/lhash.c:117
#2  0x00007ffff77d936f in lh_OBJ_NAME_insert (lh=0x7ffff0000bd0, d=0x7fffe8000ff0) at ../src/crypto/objects/obj_local.h:12
#3  0x00007ffff77d9a94 in OBJ_NAME_add (name=0x7ffff7951a54 "DES3", type=2, data=0x7ffff7951a47 "DES-EDE3-CBC") at ../src/crypto/objects/o_names.c:214
#4  0x00007ffff7759254 in openssl_add_all_ciphers_int () at ../src/crypto/evp/c_allc.c:42
#5  0x00007ffff77b7c91 in ossl_init_add_all_ciphers () at ../src/crypto/init.c:211
#6  0x00007ffff77b7c7c in ossl_init_add_all_ciphers_ossl_ () at ../src/crypto/init.c:203
#7  0x00007ffff77cabdb in CRYPTO_THREAD_run_once (once=0x7ffff7a63cb0 <add_all_ciphers>, init=0x7ffff77b7c6f <ossl_init_add_all_ciphers_ossl_>) at ../src/crypto/threads_none.c:70
#8  0x00007ffff77b8281 in OPENSSL_init_crypto (opts=2097228, settings=0x0) at ../src/crypto/init.c:558
#9  0x00007ffff7f1cfdb in OPENSSL_init_ssl (opts=2097228, settings=0x0) at ../src/ssl/ssl_init.c:115
#10 0x00007ffff7f24c8c in SSL_CTX_new_ex (libctx=0x0, propq=0x0, meth=0x7ffff7fb42c0 <TLS_client_method_data.14>) at ../src/ssl/ssl_lib.c:3211
#11 0x00007ffff7f25334 in SSL_CTX_new (meth=0x7ffff7fb42c0 <TLS_client_method_data.14>) at ../src/ssl/ssl_lib.c:3414
#12 0x00005555555552b7 in operator() (__closure=0x55555556beb8) at test_openssl.cpp:10
#13 0x0000555555555ade in std::__invoke_impl<void, Test1()::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#14 0x0000555555555a64 in std::__invoke<Test1()::<lambda()> >(struct {...} &&) (__fn=...) at /usr/include/c++/11/bits/invoke.h:96
#15 0x00005555555559be in std::thread::_Invoker<std::tuple<Test1()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55555556beb8) at /usr/include/c++/11/bits/std_thread.h:259
#16 0x0000555555555966 in std::thread::_Invoker<std::tuple<Test1()::<lambda()> > >::operator()(void) (this=0x55555556beb8) at /usr/include/c++/11/bits/std_thread.h:266
#17 0x000055555555592e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<Test1()::<lambda()> > > >::_M_run(void) (this=0x55555556beb0) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x00007ffff7d8d253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007ffff7afdac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x00007ffff7b8f660 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

My development environment and runtime environment are as follows:
OS: ubuntu22.04
kernel version: 5.15.0-91-generic
openssl: 3.0.12
gcc: 11.4.0
gdb: 12.1
glibc: 2.35

I just tried this on my ubuntu 22.04 system with OpenSSL 3.0.12 and your test case and I cannot seem to reproduce it.
Please can you provide the output from the following command (run from the top of the OpenSSL build tree):
perl configdata.pm --dump

a.tar.gz
I use conan build openssl.
The file “output” contains the output of the "perl configdata.pm --dump" command
The file "20-conan.conf" will be placed in the "Configurations" directory.

The cause of your problem is immediately apparent from the configdata.pm output. This tells me what compile time options OpenSSL was built with:
    /usr/bin/perl ../src/Configure dp_conan shared no-unit-test no-threads no-tests no-md2 enable-fips --prefix=/home/ubuntu/.conan2/p/b/opens2737afeb59b50/p --libdir=lib --debug

Note that OpenSSL was built with the "no-threads" option. That option is described as following in our documentation:
### no-threads

Don't build with support for multi-threaded applications.

Importantly this build won't do any kind of locking to protect shared/global data. This is why it is failing in your test. Clearly shared/global data is becoming corrupted hence the crash. The solution is not to build OpenSSL with the no-threads option.

Ahh! I didn't notice this option, this is my foolish oversight, thank you for your help!


Neurodesk Docker Container Missing Software

Hi I followed the instructions in https://www.neurodesk.org/docs/neurodesktop/getting-started/linux/ to setting up neurodesk and I can get it running through docker. I have also ensured that neurodesktop-storage has all read and write permissions.
When I load into neurodesktop, using pip list indicates that nipype is not installed.
In addition, I have tried pip install nipype[all], run spm12 r7771 once and in a separate terminal "ml spm12/r7771". After which I ran
from nipype.interfaces import spm
matlab_cmd = '/opt/spm12/run_spm12.sh /opt/mcr/v97/ script'
spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True)
print(spm.SPMCommand().version)
which returns none .
I have also repeated the same steps with older neurodesk images but I run into the same problem. Would appreciate some advice in what I might be doing wrong in getting started. Thank you!
OS:
Ubuntu 20.04 LTS
Host Terminal output:
sudo docker run
--shm-size=1gb -it --privileged --name neurodesktop
-v ~/neurodesktop-storage:/neurodesktop-storage
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)"
-p 8080:8080
-h neurodesktop-20220813 vnmd/neurodesktop:20220813
==================================================================
Starting SSH server

Restarting OpenBSD Secure Shell server sshd                           [ OK ]
==================================================================
Starting RDP server
Restarting Remote Desktop Protocol server                                    xrdp-sesman[56]: (56)(140051190646336)[DEBUG] libscp initialized

xrdp-sesman[56]: (56)(140051190646336)[DEBUG] Testing if xrdp-sesman can listen on <IP_ADDRESS> port 3350.
xrdp-sesman[56]: (56)(140051190646336)[DEBUG] Closed socket 4 (AF_INET6 ::ffff:<IP_ADDRESS> port 3350)
xrdp-sesman[57]: (57)(140051190646336)[INFO ] starting xrdp-sesman with pid 57
xrdp-sesman[57]: (57)(140051190646336)[INFO ] listening to port 3350 on <IP_ADDRESS>
                                                                     [ OK ]

Setting UID to 1000
usermod: no changes
Setting GID to 1000
==================================================================
Mounting CVMFS
CernVM-FS: running with credentials 109:115
CernVM-FS: loading Fuse module... done
CernVM-FS: mounted cvmfs on /cvmfs/neurodesk.ardc.edu.au
==================================================================
Testing which CVMFS server is fastest
OK
[0] http://cvmfs-ashburn.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (50 ms)
[1] http://cvmfs.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (55 ms)
[2] http://cvmfs-zurich.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (154 ms)
[3] http://cvmfs-sydney.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (205 ms)
[4] http://cvmfs-perth.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (351 ms)
Active host 0: http://cvmfs-ashburn.neurodesk.org/cvmfs/neurodesk.ardc.edu.au
Starting tomcat server
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:   -Xms512M -Xmx1024M
Tomcat started.
Starting Guacamole Daemon
Use this link for direct Neurodesktop:

!!! http://localhost:8080/#/?username=user&password=password !!!
Once connected to the session, your user info is:
Username: "user"
Password: "password"
guacd[199]: INFO:	Guacamole proxy daemon (guacd) version 1.4.0 started
guacd[199]: INFO:	Listening on host <IP_ADDRESS>, port 4822

Dear @JMmedphys,
You did everything right and the problem is that we haven't finished the integration of nipype. However, if you are happy to provide feedback there isn't much missing and we hopefully get it to work nicely in a few days :)
what you ran is correct, but it only works when you open spm through the application menu and since we had not yet finished the nipype integration Python is not on the path:
Can you let me know if this works for you?

open spm12 from the application menu

PATH=$PATH:/opt/miniconda-latest/bin/
python
from nipype.interfaces import spm
matlab_cmd = '/opt/spm12/run_spm12.sh /opt/mcr/v97/ script'
spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True)

print(spm.SPMCommand().version)

if that works for you, I can extend this to enable other toolkits as well. Which tools are you aiming to combine in a nipype workflow?

Dear @stebo85
Thank you for the reply! I see, glad to know I did not mess up somewhere.
I can confirm that what you suggested works when I run spm through the application menu.
print(spm.SPMCommand().version) returns 12.7771 .
I intend to combine SPM, AFNI and FSL within a nipype workflow.
Thank you so much for your time and attention.

Dear @JMmedphys,
I finished a first version of Nipype in Neurodesk. Could you test neurodesktop version "20220825" and let me know if everything works as expected?
You find nipype under Applications -> workflows
You can find the instruction here and it's also output inside the container:
https://github.com/NeuroDesk/neurocontainers/tree/master/recipes/nipype
I also noticed that you had a quite high latency to our cvmfs servers, with the fastest server being in US east and 50ms latency:
[0] http://cvmfs-ashburn.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (50 ms)
I now added another server in US west - hoping that this is where you are roughly located and that applications startup a lot faster for you now :)
Thank you
Steffen

Dear @stebo85
Thank you for the update and for setting up another server! I followed the instruction in the link you provided but I think the python package is missing.
Thank you for your attention to this issue.

Dear @JMmedphys,
Thank you for trying it so quickly :)
You need to start the nipype container from the application menu -> workflows -> nipype
you could also try to use module load in the general terminal ('ml nipype'), but I haven't tested that yet :p
Can you let me know if that works?

Dear @stebo85
So I have been trying to run the code but I keep coming across permission denied, which is weird because I am running the neurodesktop container with sudo and --privileged flag. What I tried this time was to start the nipype container from the application menu and follow the instructions in the link.
For the setup steps: I switched to another terminal and used sudo to run the wget and gunzip lines. Otherwise I would have encountered permission denied errors.
Trying to run the spm part of the python code gives the error in the image below.
I realised I forgot to mention in my earlier reply than I am now using WSL2 with ubuntu 20.04 and running docker (not docker desktop) through WSL2.
===
sudo docker run
--shm-size=1gb -it --privileged --name neurodesktop
-v ~/neurodesktop-storage:/neurodesktop-storage
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)"
-e NVIDIA_VISIBLE_DEVICES=all
-e NVIDIA_DISABLE_REQUIRE=1
-p 8080:8080 -h neurodesktop-20220825
vnmd/neurodesktop:20220825

Dear @JMmedphys
You need to make the neurodesktop-storage folder writable before starting the docker container as described in the Linux instructions:
chmod a+rwx ~/neurodesktop-storage

Dear @stebo85
Thank you for your patience.
I did do chmod command originally. I fear this an issue with using docker through wsl2 instead of docker desktop on windows. I will be setting up an ubuntu20.04 virtual machine to attempt again. Will update.

Dear @JMmedphys,
It shouldn't be a problem of WSL - I run it the same way.
Can you check the permissions of the neurodesktop-storage folder in WSL?

Dear @stebo85
Please see attached.

@JMmedphys,
ok, great. Now open neurodesktop in the browser and check if you can create files inside /neurodesktop-storage

Dear @stebo85
This is the result:

hm, that's strange. Did you stop the container before changing permissions?
If you like to quickly debug this interactively: https://uqz.zoom.us/j/84115698774

ok, problem solved - and for completeness the solution here: There were two different neurodesktop-storage folders and one had the wrong permission set.

@stebo85 Thank you so much for your patience and attention on this matter. Just wanted also follow up and add that I had no issues with https://github.com/NeuroDesk/neurocontainers/tree/master/recipes/nipype
Thank you so much once again!

Wonderful to hear:)


README.md: ASP.NET 5 -> ASP.NET Core 1.0

Simple search/replace.

Hi @MaximRouiller, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, DNFBOT;


Smoothing Functions For Actors

Summary of changes
Changes introduced in this pull request:

Adds packages forest_math and forest_smooth under the utils/ directory
Adds structs FilterEstimate and  AlphaBetaFilter
Adds tests

Reference issue to close (if applicable)

Closes
590
Other information and links

I dont think you needed to create 2 new crates for these. Can just throw them into vm/actor/src/util. I don't think these will be used anywhere else.


nix-shell breaks if bash is in scope

Issue description
Terminal handling within nix-shell appears to break if the bash derivation is in scope.
Steps to reproduce
First verify that nix-shell works if bash isn't in scope,
[ben@nixos-unstable:~]$ nix-env -e bash
uninstalling ‘bash-4.4-p12’
[ben@nixos-unstable:~]$ nix-shell '<nixpkgs>' -p rr

[nix-shell:~]$

Try pressing up arrow to go back in shell history, note that it works as expected. Send mouse event to TTY (e.g. click in a mouse-capable terminal emulator), note that nothing happens.
Now try bringing bash into scope,
[ben@nixos-unstable:~]$ nix-env -i bash
installing ‘bash-4.4-p12’
[ben@nixos-unstable:~]$ nix-shell '<nixpkgs>' -p rr

\[\][nix-shell:~]$\[\]

Note how prompt is mangled, pressing up arrow key results in ^[[A being echoed to terminal instead of moving within shell history. Things are generally broken.
Technical details

System: 18.03pre116572.9824ca6975 (Impala), also confirmed on 17.03.
Nix version: nix-env (Nix) 1.11.15
Nixpkgs version: "18.03pre116572.9824ca6975"
Sandboxing enabled: build-use-sandbox = false
nix-shell version: nix-shell (Nix) 1.11.15

@cleverca22 points out on #nixos that bash lacks interactive support. I should really be using bash-interactive.

I observe same "picture" when run nix-shell inside nix-shell, btw
[danbst@LX:~]$ nix-shell -p stdenv

[nix-shell:~]$ nix-shell -p stdenv

\[\][nix-shell:~]$\[\]

This is a Nix issue.

And I can't reproduce it with nixUnstable:
[chronos@yegatool:/tmp]$ nix-env -iA nixos.nixUnstable
installing ‘nix-unstable-1.12pre5732_fd10f6f2’
building path(s) ‘/nix/store/wkbfp2anpzcv6b16mr7h97vmr7nqjvs5-user-environment’
created 7 symlinks in user environment

[chronos@yegatool:/tmp]$ nix-shell --version
nix-shell (Nix) 1.12pre5732_fd10f6f2

[chronos@yegatool:/tmp]$ nix-env -i bash
installing 'bash-4.4-p12'

[chronos@yegatool:/tmp]$ nix-shell '<nixpkgs> -p rr
>

[chronos@yegatool:/tmp]$ nix-env -iA nixos.nixUnstable
installing ‘nix-unstable-1.12pre5732_fd10f6f2’
building path(s) ‘/nix/store/wkbfp2anpzcv6b16mr7h97vmr7nqjvs5-user-environment’
created 7 symlinks in user environment

[chronos@yegatool:/tmp]$ which nix-shell
/home/chronos/.nix-profile/bin/nix-shell

[chronos@yegatool:/tmp]$ nix-shell --version
nix-shell (Nix) 1.12pre5732_fd10f6f2

[chronos@yegatool:/tmp]$ nix-shell -p stdenv

[nix-shell:/tmp]$ nix-shell -p stdenv

[nix-shell:/tmp]$ nix-shell -p stdenv

[nix-shell:/tmp]$

Yeah, looks like https://github.com/NixOS/nix/commit/c94f3d5575d7af5403274d1e9e2f3c9d72989751 fixed it

Actually, this is quite a duplicate
https://github.com/NixOS/nixpkgs/issues/27126
https://github.com/NixOS/nix/issues/756
https://github.com/NixOS/nix/issues/730
And pending fix for 1.11.x
https://github.com/NixOS/nix/pull/1483

The PR pointed to above https://github.com/NixOS/nix/pull/1483 says that it's a backport...
On Darwin, I still get a broken shell if I run a pure shell (arrow buttons do not work):

Passing -p bashInteractive fixes the shell, but best if it's unbroken by default:

But on my NixOS 18.03 Thinkpad, not having the same problem.


Logging isn't working perfect on Linux

For some reason, the Linux build is writing log.txt to /log.txt (ignoring the variable ${gdc:StoragePath}). At some point the variable starts working and it begins to write to /config/Logs/log.txt
==> /log.txt <==
2024-07-24 23:59:09.4981|DEBUG|LANCommander.Program|Loading settings
2024-07-24 23:59:09.5116|DEBUG|LANCommander.Program|Loaded!
2024-07-24 23:59:09.5116|DEBUG|LANCommander.Program|Validating settings
2024-07-24 23:59:09.5116|DEBUG|LANCommander.Program|Done validating settings
2024-07-24 23:59:09.5116|DEBUG|LANCommander.Program|Configuring MVC and Blazor

==> /config/Logs/log.txt <==
2024-07-24 23:59:09.5643|DEBUG|LANCommander.Program|Building Application
2024-07-24 23:59:09.6586|DEBUG|LANCommander.Program|Registering Endpoints
2024-07-24 23:59:09.7510|DEBUG|LANCommander.Program|Ensuring required directories exist
2024-07-24 23:59:09.7510|DEBUG|LANCommander.Program|Migrating database if required

==> /log.txt <==
2024-07-24 23:59:09.5286|DEBUG|LANCommander.Program|Starting web server on port 1337
2024-07-24 23:59:09.5343|DEBUG|LANCommander.Program|Initializing DatabaseContext with connection string "Data Source=LANCommander.db;Cache=Shared"
2024-07-24 23:59:09.5368|DEBUG|LANCommander.Program|Initializing Identity
2024-07-24 23:59:09.5523|DEBUG|LANCommander.Program|Initializing Controllers
2024-07-24 23:59:09.5523|DEBUG|LANCommander.Program|Initializing Hangfire
2024-07-24 23:59:09.5560|DEBUG|LANCommander.Program|Registering Swashbuckle
2024-07-24 23:59:09.5574|DEBUG|LANCommander.Program|Registering AntDesign Blazor
2024-07-24 23:59:09.5595|DEBUG|LANCommander.Program|Registering Services
2024-07-24 23:59:09.5595|DEBUG|LANCommander.Program|The beacons have been lit! LANCommander calls for players!

==> /config/Logs/log.txt <==
2024-07-24 23:59:10.5263|DEBUG|LANCommander.Program|No pending migrations are available. Skipping database migration.
2024-07-24 23:59:10.5263|DEBUG|LANCommander.Program|Autostarting Servers

Current Settings.yml:
# cat /config/Settings.yml | grep Logs
Logs:
  StoragePath: /config/Logs

Revamping has been reworked completely in the v1.0.0 branch, so it will be fixed on the next release. Marking this as complete.


TestCitadelBootstrapKubernetes test failures

Last 4 test runs are failing at https://testgrid.k8s.io/istio-presubmits-release-1.2#integ-security-k8s-presubmit-tests&width=20
Examples:
https://prow.k8s.io/view/gcs/istio-prow/pr-logs/directory/integ-security-k8s-presubmit-tests-release-1.2/104
https://prow.k8s.io/view/gcs/istio-prow/pr-logs/directory/integ-security-k8s-presubmit-tests-release-1.2/105
https://prow.k8s.io/view/gcs/istio-prow/pr-logs/directory/integ-security-k8s-presubmit-tests-release-1.2/106

Another example is here: https://github.com/istio/istio/pull/14591#issuecomment-501407971

The weird thing is we get these clumps of successes: https://testgrid.k8s.io/istio-presubmits-release-1.2#integ-security-k8s-presubmit-tests
Normally the flakes are like every other test fails, but this we get 10 fails then 5 success.

So the logs are missing "Failed to create a self-signed Citadel" but have "Failed to write secret to CA"
Looking at the logs I saw 		2019-06-12T19:35:49.528237Z	error	Failed to write secret to CA (error: secrets "istio-ca-secret" already exists). This CA will not persist when restart. which is what the error said until ~1 day ago when this test was added: https://github.com/istio/istio/commit/6efd5850fa52e94326907dcb3b1cdb37d65c2966
I looked at the git tree for one where it always fails and see this PR is NOT in the tree. See https://github.com/istio/istio/commits/93e831745e013fc0e7ef6e9f9e14203f3e6ec3de
So somehow, I have a PR that is created before this other change that includes a code change + test, and prow is running the new test but not the new code? The fact that it is picking up the new test isn't that weird, since prow does rebase before testing -- what is weird is it picks up the test but not the code.
My theory:

Due to https://github.com/istio/istio/pull/14351, the image is built with the changes in the original PR
THEN prow rebases the test (and the code!)
The tests run with the image running old code, leading to this problem

I am not sure if that theory is correct but I think we can say this is not a bug in real code, but an issue with our testing.
cc @fpesce
It seems we build images like SHA-test.integration.framework.kube.presubmit but then are running images in the form of SHA. Could be related? See this build, we pass --istio.test.tag=e58133f2df179cd03680ede17e38a9167df34641.
@icygalz any ideas

release-1.2 shouldn't be affected by https://github.com/istio/istio/pull/14351 because that PR is only for master. We didn't cherry pick it to release-1.2 so the images will be built only with istio.test.tag=sha and the test will also use that tag.
Only master branch has BUILD and test for each test. Release-1.2 is still using the old logic; only 1 BUILD and test are re-using the created images.

I see. Does the image get built before the rebase logic then?

For release-1.2 branch, the image gets built during istio-presubmit.sh test. That test will be run after the rebase. So the image should be built after the rebase and all other tests which depends on these images will only get started AFTER istio-presubmit.sh succeed.

I am not sure whether this is the same issue or not but AFAIK, that test on master branch was failing when we first created the branch. If you remember this issue https://github.com/istio/istio/issues/14364. Seems like the same issue?

@icygalz the tests are grouped into one target but (should) be completely distinct from each other.
My two new theories: when PR is submitted, image is created and all is good. Then when the new test is merged:

a new image is NOT made because it already exists (same tag as the first test run)
or a new image is made, but prow caches the image and uses the old one (if imagePullPolicy is set to IfNotPresent on the cluster then this is possible)

if theory 1 is true then we are really in a bad shape. It means we are not testing anything during merge. We can check istio-presubmit.sh logs to see if it pushed new images or not.
theory2 is highly unlikely because each prow/*.sh test rent out a new boskos resource. The cluster(s) inside newly rented boskos resource are new/unused so it is unlikely for those clusters to have any istio images cached. After the test is done, the test will return their rented boskos resources and the clusters inside that rental will be destroyed.
in one of you comment above:
"So somehow, I have a PR that is created before this other change that includes a code change + test, and prow is running the new test but not the new code? "
Can we track down this "code change + test" PR and make sure the "code change" exists in release-1.2?

Here is the PR https://github.com/istio/istio/pull/14704. If we browse the tree at this point we see: https://github.com/istio/istio/commits/93e831745e013fc0e7ef6e9f9e14203f3e6ec3de.
A commit immediately after this PR was created was merged, https://github.com/istio/istio/commit/6efd5850fa52e94326907dcb3b1cdb37d65c2966. Note that the tree I posted above does NOT have this change in it. This change adds a test, and changes a log statement.
Now when the tests run for PR #14704, it runs the test (which does not exist in the PR, so it must be getting it from rebasing). Yet if you look in the logs, it does not pick up the changes to the log statement which are made in the same PR. Therefor, the images that it is testing and the test are out of sync.
Does that make sense? It is a bit convoluted

So i talked with Prow/tide experts and the culprit is highly likely to be: "run_after_success" mechanism.
We build the images as part of istio-presubmit test and all other tests will only be starter after istio-presubmit test pass. Each test will create a git clone/checkout with base_branch and the changes on that PR.
so let's call:
PR #14704 --> PRa
https://github.com/istio/istio/commit/6efd5850fa52e94326907dcb3b1cdb37d65c2966   --> PRb
This is the failing test on PRa:
https://storage.googleapis.com/istio-prow/pr-logs/pull/istio_istio/14697/integ-security-k8s-presubmit-tests-release-1.2/141/build-log.txt
this is the corresponding checkout/clone log for failing test:
https://storage.googleapis.com/istio-prow/pr-logs/pull/istio_istio/14704/integ-security-k8s-presubmit-tests-release-1.2/137/clone-log.txt
From that clone log, we see that the base_branch is https://github.com/istio/istio/commit/b3ab248bbe9b46527321576e4ee887c377c0bef2  which includes PRb. and  https://github.com/istio/istio/pull/14704/commits/93e831745e013fc0e7ef6e9f9e14203f3e6ec3de is the current head commit of PRa and is what is merged into the base branch.
so it made sense that integ-security-k8s-presubmit-tests-release-1.2 has the PRb code BUT the one that is building and pushing the image is not integ-security-k8s-presubmit-tests-release-1.2 .
It is entirely possible for the base_branch to be different for istio-presubmit and integ-security-k8s-presubmit-tests-release-1.2 and it will be resulting in this issue...
re-triggering istio-presubmit.sh test will solve this issue.
This is also one of the reason why prow got rid of "run_after_success" in the head binary...

The findings here makes sense to me (though I don't know the details of how the testing system picks up the versions).
Seeing the failing test on PRa:
https://storage.googleapis.com/istio-prow/pr-logs/pull/istio_istio/14697/integ-security-k8s-presubmit-tests-release-1.2/141/build-log.txt
The log This CA will not persist when restart exists, but it's actually removed by https://github.com/istio/istio/commit/6efd5850fa52e94326907dcb3b1cdb37d65c2966.
I see now the test isn't flaky since yesterday. I guess it's because the Citadel version being tested caught up.
Guys from testing team, please feel free to assign and let me know if I can help here.

Rescoped the issue to general prow issue, the citadel one was just a symptom of the problem as you mentioned

We no longer have run after success - images are built in each test, so I don't think this is an issue. The only problem could be if we don't have image pull policy Always and it pulls the old version, but I don't think this is a problem. Closing this now as it should be impossible to happen now


Any alert: not getting all alerts matching query sent through to slack

It seems like only some alerts matching my query get sent to slack whilst many are missed out.
My elastalert rule:
    name: service_logs_err

    # Type of alert.
    # The any rule will match everything. Every hit that the query returns will generate an alert
    type: any

    # Index to search, wildcard supported
    index: fluentd-*

    filter:
    - query:
        query_string:
          query: "level:>30 AND _exists_:err.message"

    # The alert is used when a match is found
    alert:
    - "slack"

    alert_text: |
      Issue on {0} service occurred at {1} with level {2}
      {3}
      -------------
    alert_text_args:
    - "hostname"
    - "@timestamp"
    - "level"
    - "err.message"

    slack_emoji_override: ":rotating_light:"
    slack_webhook_url:
    - "_replace_slack_webhook_"

My Elastalert config:

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: _replace_elasticsearch_host_

# The Elasticsearch port
es_port: _replace_elasticsearch_port_

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: _replace_elastalert_index_

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

Please help.

Anyone here able to assist? Or is there a better channel to find this kind of support?


Missing icon for Disconnect button in context menu on 1809

Description
In Home screen, the Disconnect option in the drive context menu (see screenshot) has 'tofu' instead of an icon.
Steps To Reproduce
No response
Expected behavior
--
Files Version
<IP_ADDRESS> from MS Store
Windows Version
Windows 10 1809 LTSC [Version 10.0.17763.2452]
Relevant Assets/Logs

@GargDivanshu using Character Map, you can browse the Segoe Fluent Icon font to find a good icon to use. Once you found one, you'll need to find the file where the current icon is being used and update it to use the new icon.

@yaichenbaum sure, i will do it

@yaichenbaum i am on windows 7, and it can't be downloaded on this, can there be other way to do it or should i move with other issue?

@yaichenbaum hello, i need to know, since it's my first time, will installing Linux, WSL under windows will help me to work on these projects??

@GargDivanshu You can also try browsing the assets directly here

@yaichenbaum

@esibruti would you be intrested in openig a pull request?

This issue should be resolved with #8854


Remove useless cli run parameter --prompt

--prompt arg is not used, if ARGS exists it will be used as a prompt.

@arouene needs a rebase and a re-push to run through CI properly... Would like to see a green build here... For example we may have tests that break because of removal of this, which may need altering

You also need to remove --prompt definition in docs/ramalama-run.1.md man page.


Prerequisites for mono repo emulator tests

Add test category for query tests so test can be split to adhere to 30min max test duration.
Add properties needed to run emulator in cloud test.

/azp run


feat: OS-14993 Make redirects from one file to another transparent

Description of Change
This change will trigger a 'transparent' redirect if we have a file url and onBeforeRequest is
passed a redirection url that is also a file. A transparent redirection means simply changing
the source url.
Method RestartInternal makes the call to OnBeforeRequest and passes a callback, which
is ContinueToBeforeSendHeaders for the non header client case. We add a check
to ContinueToBeforeSendHeaders to see if redirect_url_ is set and it and the source url
are using file schema.  If this occurs the source url is set to redirect_url_ and redirect_url_
is cleared to prevent a normal redirect from occurring.
Checklist

[ ] PR description included and stakeholders cc'd
[ ] npm test passes
[ ] tests are changed or added
[ ] relevant documentation, tutorials, templates and examples are changed or added
[ ] PR release notes describe the change in a way relevant to app developers, and are capitalized, punctuated, and past tense.

Release Notes
Notes:

After approving your change, I've noticed you don't have a test case. Change looks ok and it is basic, but just to make sure we don't lose the functionality for one reason or another(ex: when we do electron updates), can you please add a basic test case into api-net-spec.ts to test this.
Your test can assert if callback is run more than once after redirect...

Done, test case added to api-web-request-spec.ts as that was a more appropriate place to add it.


docs: fixed docs website publication

For htmltest:

Continue building in the docs/build/crc directory for htmltest.
Ignore production website (for cases like today when it has published with mistakes).
Removed leftover comment.

Publish the ./docs/build/crc directory

fixes #3844

Thank you @praveenkumar ! I apologize again for my mistake.