Improve the security of the kubernetes/helm charts

Is your feature request related to a problem? Please describe.
We are trying to use helm charts in contrib module. First, deployment in dev failed due to some hardcodings. In the meantime, we also discovered it having some security vulnerabilities like passwords being passed in "verbatim", etc
Describe the solution you'd like
Secure and automated deployment of helm charts
Describe alternatives you've considered

move the password parameters from verbatim mode to kube-secrets
use configmaps to discover service configurations for easy deployment
initcontainer for initializing the mysql schema
remove hard-coded references like "datahub-gms-deployment-configmap"
refactorings for ex, removal of obsolete code like contrib/kubernetes/yaml

@bharatak could you help fix these issues?

Sorry some confusions here. @shakti-garg-saxo will be working on a PR for this.

@shakti-garg-saxo Are you planning on including support for creating RBAC resources too? For example:
rbac:
  # Specifies whether RBAC resources should be created
  create: true

We are currently not trying to create service account as part of helm chart. We are going with assumption that serviceaccount (roles, clusterRole, etc) is pre-configured and is a parameter to LDH charts.

@shakti-garg-saxo do you plan to re-use this issue for further security improvements? If not please feel free to close it.


Module tags

Looking for a way a can tag the modules.
Say I have may modules themes, plugins etc. A module can only be one of these and be able to call say
Module::byType('themes')->all()

The module class implements the Macroable trait, so you could in theory add a method like this on it.

Yes, each module has the Macroable trait.


ICA AROMA: interacting with the other confounds

The standard way to use ICA-AROMA with non-aggressive denoising in conjunction with other confounds in the tsv (specifically acompcor, tcompcor, and other measures derived from the bold), is to pull out those signals after denoising of the data, as to not re-introduce noise removed by ICA-AROMA (if we calculated acompcor, tcompcor before ICA-denoising). So my question is: does this mean we should re-organize the code to also pass in the non-aggressively denoised bold to calculate these other measures? or would it be equivalent to do full regression removing the variance explained by the "noise components" on the other confounds in the tsv as I did in the final part of this jupyter notebook. The latter would be substantially easier, but I don't know if we are sacrificing something, or doing something not recommended.

As it is right now, ICA-AROMA is run in the original BOLD time-series after: 1) Slice timing correction (if available); 2) Head motion compensation; 3) Susceptibility-derived distortion correction (if available); 4) resampling in MNI space.
Your point sounds very reasonable. I have two questions at this point:

Is it possible to run ICA-AROMA in native BOLD space? If so, that would be the best option. Then, the components maps could be moved into MNI for visualization purposes.
Should we move some of the processing after ICA-AROMA (other than a/tCompCor)?

Good questions,

Is it possible to run ICA-AROMA in native BOLD space? If so, that would be the best option. Then, the components maps could be moved into MNI for visualization purposes.

The obstacle to this is that ICA-AROMA depends on having an FSL warp file and affine to align the edge fraction measures with individual subjects, we can submit a pull request to ICA-AROMA to include the possibility of using the ANTs transforms, or cannibalize the ICA-AROMA code to work in FMRIPREP.

Should we move some of the processing after ICA-AROMA (other than a/tCompCor)?

Yes, I should have thought about it more, we should also move all DVARs measures, the compcors, the region average signals, and maybe the cosine basis set, perhaps I need a primer on what the cosine basis set is.
Then another question is if someone specifies --use_aroma, should we include tsvs for before and after ICA-AROMA is applied?

Good questions,

Is it possible to run ICA-AROMA in native BOLD space? If so, that would be the best option. Then, the components maps could be moved into MNI for visualization purposes.

The obstacle to this is that ICA-AROMA depends on having an FSL warp file and affine to align the edge fraction measures with individual subjects, we can submit a pull request to ICA-AROMA to include the possibility of using the ANTs transforms, or cannibalize the ICA-AROMA code to work in FMRIPREP.

Should we move some of the processing after ICA-AROMA (other than a/tCompCor)?

Yes, I should have thought about it more, we should also move all DVARs measures, the compcors, the region average signals, and maybe the cosine basis set, perhaps I need a primer on what the cosine basis set is.
Then another question is if someone specifies --use_aroma, should we include tsvs for before and after ICA-AROMA is applied?

Going back to the main issue I am not convinced that there is a risk of "reintroducing noise" via regression and thus a need for orthogonalization. @jdkent could you provide some evidence via simulations that this is indeed a problem?
@nicholst it would be great to hear what do you think!
PS This paper seems relevant: http://sci-hub.la/10.1016/j.neuroimage.2013.05.116

In the paper you've referenced, I'm suggesting the current approach is  most akin to:

bandpass filtering followed by nuisance regression
(BpReg)

where we swap out bandpass filtering with ICA-AROMA, since one of the noise component selection criteria for ICA-AROMA is frequency (high frequency components are removed). If we keep nuisance regressors that may still include these high frequency components (since they were derived from the non-denoised bold) then we run the risk of re-introducing noise that ICA-AROMA originally removed.
Or at least that's how I see it... the denoising process for ICA-AROMA is different from band-pass filtering so it may not have an impact.

I understand your reasoning, but the assumption is that regression and bandpass behave the same way.
Here's a though experiment that makes me skeptical. Lets say I have a variable A from which I regress out variable B (fit a model to get residuals). This gives me variable C which has no shared variance with variable B (by definition). Now I will do the same thing again - regress variable B from variable C. Would this "reintroduce" variance from variable B that was previously removed? I doubt it, but we should simulate this.

Chris & co,
I’m just finalising a blog post to flesh out my response to our conversation over the summer stimulated by your “What is aggressive/non-agressive?” python notebook.  Here is what I would say is the relevant take away:
Consider response (BOLD data) Y, some signal of interest X and confound C.
In an ideal world we’d fit everything together, at once:
    Y ~ X + C

in R-style linear modeling notation, giving inference on effect of X while controlling for C.  Let’s call this the ‘gold standard’, the result we want to replicate.
In practice, we want to provide adjusted data for diverse uses, so we can instead do it in stages in one of two ways and get the very same inference.
Approach 1.
Fit the only the nuisance
    Y ~ C

making what we can call Ya1, adjusted data.  With this, if we want to get the same inferences as the gold standard then we must run the following regression:
    Ya1 ~ Xa

where Xa comes from the adjustment of X, residuals from X ~ C.  Note, an equivalent approach is to just leave X as is and include C in the design...
    Ya1 ~ X + C

… Ya1 doesn’t need the C, but adding C to the regression automatically implements the C-adjustment-of-X.
Approach 2.
Alternatively, if X is available when we are doing preprocessing, we could fit the gold standard model but only subtract off the estimated nuisance effect of C (this corresponds to the “non-agressive” setting, where C is all bad ICA, components, X all non-bad ICA components, and only the C*beta portion of the fitted model is subtracted off). Call the adjusted data Ya2.
In this case, perhaps surprisingly, you can do subsequent analyses ignorant of C
    Ya2  ~ X

and you’ll get the gold standard inference.
So… if we are intent in obtaining what would be the gold standard analysis we have two choices:  (1) Have the final actual regressors of interest available at the time of preprocessing, or (2) Keep the nuisance regressors around, available at the time of final inference.
Make sense?
PS: In all of these settings, to get the DF right you’ll have to keep track of the the previous regressions.

@chrisfilo, here is the simulation code. Just used the code you provided from your ICA_AROMA investigations. In the simple case you provided, your intuition was correct, the same regressor applied twice does not result in different outcomes. My intuition wasn't correct, I'll read more about the differences between bandpass filtering and regression.
I don't know if I need to model any additional complexities to make the simulation more valid.
Thanks @nicholst! I'll need to digest that for a bit. The only comment I'll add is that we are additionally interested in including confounds that were derived from other methods (e.g. framewise displacement, DVARS) and if it's necessary to "denoise" the other (non ica-aroma) confounds so they don't add variance that ICA_AROMA removed (as in the paper @chrisfilo referenced), but I'm starting to think this was an ill-formed question.

@chrisfilo I took another stab at testing ica_aroma between extracting confounds before or after denoising, and found different results depending on whether the confound was extracted before or after denoising, please see and check my work
Thank you!

Since global signal is a spatial average I don't think it would work the way you expect. In non-agressive denoising each voxel is denoised in different way so averaging all of them (when calculating global signal) will yield a different regressor. I don't think it is that surprising that you are getting a different result depending on the order.
@nicholst thanks for the explanation. Indeed we are dealing with a slightly different question here: if you do non-aggressive denoising to the data and you later want to regress other sources of noise (say motion) would that additional regression reintroduce noise previously removed by AROMA? I don't think it would.

thanks @chrisfilo. Then if it's not surprising the results are different, should it be necessary to keep the global signal as close to the data as possible (as well as other measures that use multiple voxels to inform their values, such as acompcor & tcompcor)? That is, calculate the confounds from the denoised data as is proposed in #959? (assuming people are interested in the non-aggressive denoising).

I don't have a strong opinion on this. I don't think either approach is wrong. Might be worth searching the literature to see if someone did compare those strategies.

I couldn't find a comparison between the two approaches, but the Pruim (2015) paper does the nuisance regression after denoising of the bold series.

Do they say which stage are nuisance regressors estimated at?
Best,
Chris
On Mon, Feb 5, 2018 at 4:54 PM, James Kent<EMAIL_ADDRESS>wrote:

I couldn't find a comparison between the two approaches, but the Pruim
(2015) paper does the nuisance regression after denoising of the bold
series.
[image: image]
https://camo.githubusercontent.com/2d4625ac43953c1654ac305da117806a09b4f331/68747470733a2f2f6172732e656c732d63646e2e636f6d2f636f6e74656e742f696d6167652f312d73322e302d53313035333831313931353030313832322d6772312e6a7067
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/817#issuecomment-363273112,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp4gjdVfJW9Uk72df9FX7PhrHWk3Tks5tR6LHgaJpZM4QVx-A
.

page 269

Within the typical fMRI participant-level preprocessing
stream ICA-AROMA is applied after spatial smoothing but
prior to high-pass filtering and further nuisance regression.

Probably correct. I see your point.
The only problem now is that if we estimate global signal, CSF, WM etc. after applying non-aggressive denoising those regressors will only be usable with the denoised data. Should we even save non denoised data then ICA AROMA is requested? Such situation may lead to a risk of users using wrong regressors.
Alternative is to include a variant of confounds for the ICA-AROMA output.
WDYT @oesteban?

@nicholst thanks for the explanation. Indeed we are dealing with a slightly different question here: if you do non-aggressive denoising to the data and you later want to regress other sources of noise (say motion) would that additional regression reintroduce noise previously removed by AROMA? I don't think it would.

No, that isn't possible.  Non-agressive denoising isn't the GLM, and so there certainly can be variance left in the data that can be later be removed by other methods (even the very temporal ICA components used in non-agressive denoising).
Each successive "agressive" correction and only reduce the total variance in the data.

Thank you for your insight @nicholst. Perhaps I am missing something here, I've made another toy example to try to demonstrate the issue.
Let's say we derived nuisance confounds from the bold data before denoising with ICA-AROMA, and then do non-aggressive denoising. If I then wanted to regress the nuisance confound from the denoised bold data (e.g. bold ~ nuisance), the output I'm interested in is the residual (e.g. what remains after accounting for the variance described by the nuisance confound bold - bold_predicted). In the toy example I linked, imagine the straight line is the denoised data, and the random number line is a nuisance confound that was calculated before denoising. If I remove the variance explained by the "nuisance confound" in the denoised bold series then the residual bold series is worse off (i.e. is more noisy) than it was before. I can imagine this scenario would occur if non-aggressive denoising removed a spike in the bold series, but then regressing a nuisance confound that retained this spike (i.e. was calculated before denoising the bold series) would adversely impact the residual bold series. Or at least this is the current thought process.
P.S. Thank you all for this stimulating conversation!

Hi @jdkent,
Nice example, but my point was solely about total sum-of-squares:  Residuals will always be smaller with any additional regressor.  I'm not yet a Python head... in Matlabese
Y=[1:100]';
X=[ones(100,1) rand(100,1)];
betahat=pinv(X)*Y;
Yhat=X*betahat;
plot(Y,Y-mean(Y),Y,Y-Yhat)
[mean((Y-mean(Y)).^2) mean((Y-Yhat).^2)]
% I got : 833.2500  811.2734

But, certainly, there can be variation in the regressor that doesn't help (as is visually seen with the 'jaggies' in your example) but to the extent that there is any overlap between the nuisance and the data it will remove something.
Does this help?

Yes, that helps tremendously @nicholst, thank you! Now I see what you are referring to. With that in mind, I see my concern is not really with increasing variance (which is impossible), it's about adding structured noise to the residual bold series (This is like one of those epiphany moments :smile:). The structured noise could come from breathing/heart rate/other physiological factors, movement, and anything else ICA-AROMA happens to classify as noise. If the nuisance confounds were derived from the non-denoised bold, the worry is that this structured noise may be added back in to the denoised bold residual (represented by the jaggies in the example). This would be bad for any functional connectivity analyses as this may introduce non-neurally based correlations between brain regions. Thank you both @nicholst @chrisfilo for putting up with my incorrect use of terms; I think I've finally caught up.

Just to clarify, the 'jaggies' are reflections of living in the 'orthogonal subspace' of the nuisance… i.e. residuals must be orthogonal to the regressor space, and hence will reflect their 'complimentary' structure. In general if the nuisance is completely irrelevant (unrelated to anything) such adjustment shouldn’t worsen any subsequent analysis.  But if the nuisance is related to the signal of interest at all then it will be important to equivalently adjust the signal of interest.

Thank you. Your follow-up inspired me to run some more tests in this notebook to get an intuition of what would happen if I took two timeseries (y1 and y2; or voxel1 and voxel2), and regressed the same nuisance confound (representing the confound pulled from non-denoised BOLD data) from each of them.
For the outcome measure, I compared the correlations between the residuals and between the original y1 and y2. Perhaps unsurprisingly, on average the correlation between the residuals of y1 and y2 decreased (relative to the original correlation between y1 and y2) because some of the variance shared between y1 and y2 was also shared with the nuisance confound, so when I removed the influence of the nuisance confound, y1 and y2 were more independent. However, this is only true "on average", and not necessarily true.
Sometimes the correlation increases, and I think this is due to how the pearson's correlation coefficient is calculated since it includes the variance of y1 and y2 in the denominator. So in the scenario where the variance decreases in y1 and y2  more than the decrease of covariance between y1 and y2, the pearson's correlation would be inflated. So while I think the covariance between y1 and y2 necessarily decreases or stays the same and the variance of y1 and y2 necessarily decreases (or stays the same) as the result of regression, putting hard rules on ratio of covariance/variance (i.e. covariance(y1,y2)/sqrt(variance(y1)*variance(y2)) ) is less clear.
But this is kind of a divergence from my original concern, and if anything it makes it harder for me predict whether a regressor derived from the non-denoised bold signal is worse than a regressor derived from the denoised bold signal. I will think further upon this, and thanks again!
I've now effectively buried @chrisfilo's question to @oesteban, sorry:

Probably correct. I see your point.

The only problem now is that if we estimate global signal, CSF, WM etc. after applying non-aggressive denoising those regressors will only be usable with the denoised data. Should we even save non denoised data then ICA AROMA is requested? Such situation may lead to a risk of users using wrong regressors.

Alternative is to include a variant of confounds for the ICA-AROMA output.

WDYT @oesteban?

We should probably move this conversation back to the pull request, now that we are dealing with the code implementation of the solution #959.

Hi, I am pretty swamped right now to process all of this but will do soon. Thanks so very much you all for this great conversation!
I just wanted to make sure we don't miss a bit of detail here. For that, I would suggest both @chrisfilo and @jdkent to add their notebooks into the fmriprep repo. We could have a /notebooks folder and below we could have /notebooks/aroma where your precious notebooks would sit.
It could be done in separate PRs, as it is just documentation. Alternatively/additionally, we could have them rendered to rst and added in the documentation.
WDYT?
As regards where to keep the conversation, it is probably best to keep it here since all the details are given above.

@jdkent I know it's not as appealing, but just to confirm -- If Y_1 and Y_2 are independent, no transformation done to each individually can induce dependence:
If we have two time series Y_1 and Y_2  that are independent (but possibly have their own autocorrelation)

then applying a deconfounding transformation

to each changes the dependence within each but cannot induce dependence between:

Math mode is definitely appealing, I'm just not well versed, but working on it, albeit slowly.  :smile:
@oesteban: I'll work on making a more sensible story with the code bits worked up, most of it is me not clearly understanding the problem, and then using incorrect terms to describe the problem. Furthermore,  I am no longer convinced that it is definitely a problem, but maybe a problem. Hopefully will make more sense in the write-up.

Hi all,
Thank you for all your posts here. I learned a lot!

Regarding native space denoising, I have a question. Since i am now working on infants brain, using adult-based 2mm MNI template would not be optimal for the final processing. However, if I run ICA-AROMA anyway with affine matrix and warping field map first, and get noise list using Melodic_IC_thr_MNI2mm.nii.gz and regress them out (fsl_regfilt) on the native images. Although it is still not optimal, I hope it is somewhat acceptable. What do you think?

Thanks!

I'm not sure how you can do that with FMRIPREP. One thing is certain ICA AROMA depends on your data being in the MNI space (or being able to put it in the MNI space) because it uses ROIs defined in MNI.
Out of curiosity which template are you using as coregistration target for your infant study?

oh! I thought that this thread is in the ICA-AROMA git. It is my bad; i did massive google search, so I lost my way. ;p I am using ICA-AROMA out of FMRIPREP.
You are right that ICA-AROMA plays on the MNI space by converting the native image to MNI template. However my understanding is that ICA-AROMA run Melodic on the native space first, and then normalize only melodic_IC.nii.gz to MNI space (melodic_ic_thr_MNI2mm), using reg matrix I put in the command line, to match noise property they have based on 2mm MNI space. Therefore I guess it would be okay to regress out classified-noise ICs for the native image since melodic_mix file, ICA-AROMA uses for denoising, still is from the native image. it is not optimal for spatial properties though. But not sure.
For baby template, I am using UNC's IBIS baby template (https://www.med.unc.edu/bric/ideagroup/free-softwares/unc-infant-0-1-2-atlases)

Hi, I'll try to close this discussion with permission from @chrisfilo, @jdkent and @nicholst.
After re-reading the whole thread, my impression is that we are safe to keep FMRIPREP working as it is implemented now, and recommend the following:
If you are to fit the "aggressively" denoised time-series, you can still use other regressors (with the exception of global signals?) if you orthogonalize them against the "non-aggressive" confounds.

Would that be correct? @jdkent what was the outcome of that meeting with your lab back in May. Did you arrive to this conclusion?

What about the particular case of the global signals?

Thanks!

@oesteban

If you are to fit the "aggressively" denoised time-series, you can still use other regressors (with the exception of global signals?) if you orthogonalize them against the "non-aggressive" confounds.

non-aggressive case: Based on the notebook, my current opinion is that one should be able to the use other confounds (such as global signal) without modification even if they were derived from the non-denoised data.
aggressive case: also should be okay to use the global signal without modification as well since you will just be including the noise columns in the confounds tsv along side the global signal column.

Going back to the main issue I am not convinced that there is a risk of "reintroducing noise" via regression and thus a need for orthogonalization. @jdkent could you provide some evidence via simulations that this is indeed a problem?
@nicholst it would be great to hear what do you think!
PS This paper seems relevant: http://sci-hub.la/10.1016/j.neuroimage.2013.05.116

I have proform preprocessing by fmriprep. And there are some files created, like
task-rest_run-001_desc-confounds_regressors.tsv;
task-rest_run-001_AROMAnoiseICs.csv;
task-rest_run-001_desc-MELODIC_mixing.tsv;
task-rest_run-001_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz;
task-rest_run-001_space-MNI152NLin2009cAsym_desc-smoothAROMAnonaggr_bold.nii.gz;
I have some questions as follow:
1. The file of 'task-rest_run-001_space-MNI152NLin2009cAsym_desc-smoothAROMAnonaggr_bold.nii.gz', I am not sure whether it has been regressed out the confounds, like motion,csf,wm…? If so, can I use it to perform groupICA using MELODIC? Or other analysis, like alff, functional connectivity?
2. And I still don't know whether this file 'task-rest_run-001_space-MNI152NLin2009cAsym_desc-smoothAROMAnonaggr_bold.nii.gz' have been performed high-pass or low-pass filter for resting-state data.

If I want to calculate functional connectivity, which file should I use, and when I perform group comparison using GLM, which confound variables should I use? (if I want to regress motion-related, WM, CSF confounds)
;

I have proform preprocessing by fmriprep. And there are some files created, like
task-rest_run-001_desc-confounds_regressors.tsv;
task-rest_run-001_AROMAnoiseICs.csv;
task-rest_run-001_desc-MELODIC_mixing.tsv;
task-rest_run-001_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz;
task-rest_run-001_space-MNI152NLin2009cAsym_desc-smoothAROMAnonaggr_bold.nii.gz;
I have some questions as follow:
1. The file of 'task-rest_run-001_space-MNI152NLin2009cAsym_desc-smoothAROMAnonaggr_bold.nii.gz', I am not sure whether it has been regressed out the confounds, like motion,csf,wm…? If so, can I use it perform groupICA using MELODIC? Or other analysis, like alff, functional connectivity?
2. And I still don't know whether this file 'task-rest_run-001_space-MNI152NLin2009cAsym_desc-smoothAROMAnonaggr_bold.nii.gz' have been performed high-pass or low-pass filter for resting-state data.
3. If I want to calculate functional connectivity, which file should I use, and when I perform group comparison using GLM, which confound variables should I use? (if I want to regress motion-related, WM, CSF confounds);

Hi @yuan0821. Closed issues are easy to skip over, so in general, it would be better to open a new issue, and link to this one, for example:
<issue description>

Related: #817

But while we're here:

No. No regression has happened except for the non-aggressive denoising. @jdkent We don't run high pass filtering on data before running MELODIC, do we?
I don't think so, but see Q1. If James doesn't have a chance to weigh in, I can go look at the code. :-)
This is an area of active research and debate, and fMRIPrep is intentionally agnostic. We provide the confounds that can be used as the researcher chooses, and it's up to you to justify this. There have been a number of discussions on Neurostars, including a pretty systematic examination. There are also a number of papers. I hesitate to link them, because I haven't surveyed lately and any I might suggest could well be out of date.

Thanks for pinging me so I could see this. No, we do not do high pass filtering before running MELODIC, all we do is spatial smoothing and motion correction on the bold image in MNI space. We follow the guidelines set in Pruim (2015).

Hey @jdkent, may I ask you to distill your latest comment and @effigies' into a PR to the documentation? It would be very helpful.

Got it! just opened an issue (#1804) so I can keep track of it.


dream.set_text("a quiet pond underneath the midnight moon") = object has no attribute 'set_text'

Looks like def set_text is now def set_clip_encoding?
Also, various settings aren't honoured using dream.set_clip_encoding then dream (such as save_best = True). Would it be possible / make sense to retain settings from the previous dream?

Hi! True, it is renamed now and not updated in the README.
But what do you mean when you talk about retaining settings from a previous dream? Was this working before and now it does not work anymore? How does save_best influence all of this?

It's back :)

For "save_best", I was thinking for an example like below:
dream = Imagine(
    text = "An apple on fire",
    epochs = 6,
    append_seed = True,
    save_progress = False,
    save_best = True
)
dream()

dream.set_text("A pear on fire")
dream()

epochs, append seed and save_progress,etc all remain set when dreaming the pear, but save_best doesn't and so seems to be the "odd one out"?

Ah, I see. The issue is that self.current_best_score is not reset to 0 when set_clip_encoding is called, that only happens in the __init__ at the moment. Maybe @lucidrains can quickly edit that?

@NotNANtoN fixed :)

Awesome! :)


Feat/removing memo

Hi @bonfim996, thank you for implementing a FlashList version of this module, can you please explain the reason behind removing memoization? Thanks


There's no linux arm64 wheel

Looking at the available builds on pypi, there is no linux arm64 build: https://pypi.org/project/ai-edge-model-explorer-adapter/#files
Would it be possible to provide such a build? Or provide build instructions for an installation from source? Thanks!

Hi @yijie-yang, can you please take a look? Thanks.

Sorry for the late reply, I missed this issue in my backlog. Since there is no public github runners that guarantee to be linux arm64, we'll request for a self-hosted runner for this architecture. Will update it here once supported.

consider using qemu? https://github.com/onnx/onnx/blob/main/.github/workflows/release_linux_aarch64.yml#L26C39-L26C46

I just went for a native linux machine in the end


Suggestion: Use github.com/natefinch/pie for plugins

I made a plugin toolkit package - https://github.com/natefinch/pie - which uses subprocesses that call RPC over stdin/stdout.  I've seen some calls for plugins in Hugo in the issues in the past, and I think this could be an efficient way to make it work.  The nice thing about this is that it's a persistent connection between Hugo and the plugin, so you don't have to re-run the plugin every time you want to call it (for example, if you had a highlight plugin, you could call it over and over with different content without the overhead of restarting the plugin application each time).
I see a lot of places where this could be useful in hugo:

content type support external to Hugo, so you don't have to modify Hugo itself to add support for asciidoc or latex (for example).
shortcode support external to Hugo
probably 1000 other things I can't think of right now.

The nice thing about Pie is that it is language agnostic - you can write plugins in whatever language you want, as long as they can serve RPC over stdin and stdout.  This works well for getting people who don't write Go to write plugins for your application.
I may hack on this some and see what I can come up with, if anyone thinks this is worthwhile.   I'd like to put Pie through its paces in a real application, to make sure I'm not missing anything.

actually, I should post this on discuss.gohugo.io... so I will :)

Yea, discussions works best at discuss.

Note, here's the discussion: http://discuss.gohugo.io/t/using-for-plugins-in-hugo/1239


Pagination suffix configuration didn't always work

First Link, Page 1, and Prev Link (when Page 1 or 2 is active) didn't respect suffix configuration.
$config['suffix'] = "?f=data";

CI_VERSION 3.1.6

Looks like you're not using it correctly, but it is also unclear what the problem is. Provide examples please.


Update Rust gen to use .wit spec

This commit updates the regeneration script for the Rust bindings to use wasi-nn.wit types instead of wasi-nn.witx.
The scripts has also been run and the bindings (src/generated.rs) have been updated.
This example currently compiles but fails at runtime because the version of the wasi-nn spec that wasmtime is using is not the latest. Once it has been updated this example should work (I've tested it with the version that wasmtime is currently on and it works but that requires changes to the examples in this repository).

@danbev, thanks for this PR and apologies for not looking at it sooner. The reason is that I'm actually working on this at the moment (e.g., https://github.com/bytecodealliance/wasmtime/pull/8519) and have been thinking: once users fully switch to using WIT for the interface, is there a need to keep the wasi-nn crate around? Once Wasmtime implements the latest spec, users should just be able to use wit-bindgen to create a component (or something like that).

thanks for this PR and apologies for not looking at it sooner.

@abrown No worries at all!

once users fully switch to using WIT for the interface, is there a need to keep the wasi-nn crate around?

I see your point and that is an option. I just found that having this crate simplified the code in the project I was working on. But it is not a big deal to handle and if it means not having to maintain this repo I understand the value in that. So both ways work for me :+1:


[Reflection] Try to read only the reflection sections.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

@apple/swift5-branch-managers no ABI impact, fixes a substantial performance regression in swift-lldb.

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - 04cbdfae731c1913b90425a95165cf6cc71337df

Is this already on master? Is it associated with a radar number?

@swift-ci test linux

I'm not really authorized to merge this, but I would appreciate if somebody can do it on my behalf.


Create Preprocessing Documentation for Labels (np_utils.to_categorical)

Includes useful documentation that was only found through an example in the sequential tutorial.
The use of keras.utils.np_utils.to_categorical(labels, nb_classes is hard to find in current documentation and is very useful for many categorical classification tasks that many students/beginners start out with.
to_categorical belongs to the np_utils file and not a preprocessing one so this addition could be misleading in terms of code structure but I believe it is useful to include in the current documentation.
Link to method documented: https://github.com/fchollet/keras/blob/master/keras/utils/np_utils.py
To test follow documentation: https://github.com/fchollet/keras/blob/master/docs/README.md

I believe this should be a note inserted in the objectives page: "categorical_crossentropy expects categorical binary labels, here's a way to convert your integer labels to categorical".
The documentation follows the structure of the codebase and as such this page wouldn't fit.

I have updated the docs to reflect this.


Webshim causes style issues with date/month inputs when using bootstrap

Example: http://jsfiddle.net/9n118znw/4/
This is only visible in Firefox, as Chrome has it's own built-in picker controls.  Tested on Firefox 34.0 on Ubuntu 14.04 x64.
It looks like some of the styles are not getting copied across properly when the original input element is cloned/enhanced to create the webshims element.  I was expecting the bold labels to appear above the date/month input elements, as it does for the text/password/number input elements.
This might occur with other input types but I have only tested on date and month.

Yes and no. Has we don't have shadowdom we need to influence the styles. And I can't fix this, you can simply fix this your own by adding:
.control-label {
    display: block;
}

Which is also better from UX standpoint (larger label to click). See http://jsfiddle.net/9n118znw/5/

That works great!  Thanks.


Fix bug with order input.

Coverage decreased (-0.02%) to 55.147% when pulling d28eaab01ae6461047d5c57c016f27bf097b4b71 on order-input into d8cf5cd314b5791b44765d3efdddf4c9a7dc9dbd on master.

The PR was closed without merging! Reopened and merged.


Allow to specify the Culture used in log-values

The culture is fixed to the invariant culture. I'd like to configure the culture used in https://github.com/aspnet/Logging/blob/adac42c183a525813ea9759717620bb0aad14fb3/src/Microsoft.Extensions.Logging.Abstractions/Internal/LogValuesFormatter.cs#L145
Default should remain invariant, but via config a different culture should be allowed.
For Europeans like me especially date-formats are odd, and like today (06/01/2018) I'll get easily trapped by 6th Jan 2018 :wink:
Should we go for this or are there any reasons to stick to invariant culture?

Thanks for the report, @gfoidl.
@davidfowl was there any good (or historical) reason why this wasn't configurable? Or is it just a miss?

We don't see a good way to achieve this at the abstraction level. One option would be using a sink that supports Culture.


Publish documentation on gh-pages using circleci

@axic I need write access to this repo to set up gh-pages.

@jakelang you have write access

It is published as a crate here: https://docs.rs/ewasm_api/0.4.0/ewasm_api/

And here: https://docs.rs/crate/ewasm_api


feat(fsengagement): add gestures and animations

@bweissbart
for RNN V2:

add animations for full screen card (animate content card to story) with react-native-animatable
add gesture handler so we're able to swipe up on full screen card to go into the story
fix typing for getInboxMessages method

@nathan-sankbeil I've moved the panhandler to the constructor and marked the methods private.


Feature Request: Make firebase_messaging support creating channels from Dart

In Android Oreo, notifications are encouraged to have "channels" so users can control types of notifications more easily. Can this feature be natively supported in the firebase_messaging plugin?
For now, I've managed to solve this problem with a semi-hacky platform channel, as below:
    new MethodChannel(getFlutterView(), "notifications").setMethodCallHandler(
        new MethodCallHandler() {
            @Override
            public void onMethodCall(MethodCall call, Result result) {
                if(call.method.equals("registerChannel")) {
                    if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
                        String id = call.argument("id");
                        String name = call.argument("name");
                        String description = call.argument("description");

                        NotificationChannel mChannel =
                            new NotificationChannel(id, name,
                                                    NotificationManager.IMPORTANCE_DEFAULT);
                        mChannel.setDescription(description);

                        NotificationManager notificationManager =
                            (NotificationManager) getSystemService(NOTIFICATION_SERVICE);

                        notificationManager.createNotificationChannel(mChannel);

                        result.success("Notification channel " + name + " created");
                    } else {
                        // No-op
                        result.success("Android version is less than Oreo");
                    }
                }
            }
        });
  }

This works, but doesn't allow us to set notification priorities without significant modifications, and would be better as part of the plugin anyway.
NB: Am willing to make a PR for this later, with the priority functionality as soon as I figure out how to do it (and if others say this is a good idea).

It would be wonderful if anybody could implement the functionality to support Notification Channels (let it be in firebase_messaging, another package, or out-of-the-box) as this is a requirement for Android API 28 and above 🤓

Also looking forward to have this feature! :)

Please implement this feature ASAP.

Implement oreo channels ! please. One year old request ! 😯

Has anyone done this? Perhaps, OP should provide a gist as to how to handle it.

@zoechi : As you guided, we added reaction to this, but this feature is still stuck, & there is no workaround for the same.
We tried to create FlutterService also, but it doesn't exists, like FlutterActivity.
Thus I request everybody to comment maximum on this issue, such that Flutter Team can help us here.

Still cannot do this...

Really sad.

Dear flutter team..  we stopped working on flutter.... because this feature  is very much important. Our project is stuck due to unavailability of channelID..

In order to send custom notifications we need channel ids and it sucks.. over a year ..!

@ghost
Does the workaround work in the sense it let's you send heads up notifications?
I can only get notifications to appear subtlety in the sys tray... Can't even set default channel in manifest

What is the progress? Still the same?

@ghost
This issue has been moved to https://github.com/FirebaseExtended/flutterfire/issues/873. Any further collaboration will be done there.

AndroidNotificationDetails() this can be used for managing notification channels in flutter

Getting Notification sound in Samsung S5 but no notification sound in Samsung S9. The app uses the guidelines of Flutter firebase_messaging. Any workaround?


PipeFusion multi-node Error

torchrun --nproc_per_node=8 --nnodes=2 --node_rank 1 --master_addr <IP_ADDRESS> --master_port 6000 scripts/./pixart_example.py --model_id /data/fjr/PixArt-XL-2-1024-MS/ -p pipefusion --height 2048 --width 2048 --no_use_resolution_binning --pp_num_patch 16 --pipefusion_warmup_step 4
[2024-06-27 09:15:30,809] torch.distributed.run: [WARNING]
[2024-06-27 09:15:30,809] torch.distributed.run: [WARNING] *****************************************
[2024-06-27 09:15:30,809] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-06-27 09:15:30,809] torch.distributed.run: [WARNING] *****************************************
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: Transformer2DModelOutput is deprecated and will be removed in version 1.0.0. Importing Transformer2DModelOutput from diffusers.models.transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.modeling_outputs import Transformer2DModelOutput, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']
Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']
Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']
INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']
Loading pipeline components...:   0%|          | 0/5 [00:00<?, ?it/s]INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
Loading pipeline components...:   0%|          | 0/5 [00:00<?, ?it/s]Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']
INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB<?, ?it/s]
INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
Loading pipeline components...:   0%|          | 0/5 [00:00<?, ?it/s]Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...:  20%|██        | 1/5 [00:00<00:0Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']       | 0/2 [00:00<?, ?it/s]
INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']       | 0/2 [00:00<?, ?it/s]
Some weights of the model checkpoint were not used when initializing PixArtTransformer2DModel:
['caption_projection.y_embedding']
Loading pipeline components...:   0%|          | 0/5 [00:00<?, ?it/s]INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
INFO 06-27 09:15:43 [distri_dit_pipefusion.py:51] Using pipeline parallelism, world_size: 16 and n_device_per_batch: 16
DistriPixArtAlphaPipeline from pretrain stage 1 0.0 GB
Loading pipeline components...:  20%|██        | 1/5 [00:00<00:00,  6.28it/s]Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...:  60%|██████    | 3/5 [00:00<00:0Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|██████████| 2/2 [00:06<00:00,  3.48s/it]it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:06<00:00,  3.49s/it]s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:06<00:00,  3.40s/it]s/it]
Loading pipeline components...:  60%|██████    | 3/5 [00:07<00:03,  1.89s/it]Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.47s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.67s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.64s/it]s/it]
Loading pipeline components...:  20%|██        | 1/5 [00:07<00:29,  7.34s/it]Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.47s/it]
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.43s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:06<00:00,  3.41s/it]
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.44s/it]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...:  40%|████      | 2/5 [00:07<00:09,  3.13s/it]Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.53s/it]
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.55s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.65s/it]
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.54s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.70s/it]
Loading pipeline components...: 100%|██████████| 5/5 [00:07<00:00,  1.55s/it]
DistriPixArtAlphaPipeline from pretrain stage 2 11.83976448 GB
DistriPixArtAlphaPipeline from pretrain stage 2 11.841435648 GB
DistriPixArtAlphaPipeline from pretrain stage 2 11.841435648 GB
0%|          | 0/6 [00:00<?, ?it/s]DistriPixArtAlphaPipeline from pretrain stage 2 11.841435648 GB
0%|          | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 226, in
main()
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 140, in main
pipeline = DistriPixArtAlphaPipeline.from_pretrained(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 126, in from_pretrained
ret = DistriPixArtAlphaPipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 57, in init
self.prepare()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 179, in prepare
self.pipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pip/distri_pixartalpha.py", line 362, in call
self.comm_manager.irecv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 314, in irecv_from_prev
self.first_recv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 272, in first_recv_from_prev
self.recv_shape = self.recv_shape_comm()
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 233, in recv_shape_comm
dist.recv(dim, src=self.prev_rank, group=self.extra_group if is_extra else None)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1804, in recv
pg.recv([tensor], src, tag).wait()
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1961, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.19.4
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:
socketStartConnect: Connect to <IP_ADDRESS><44589> failed : Software caused connection abort
0%|          | 0/6 [00:00<?, ?it/s]DistriPixArtAlphaPipeline from pretrain stage 2 11.83976448 GB
0%|          | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 226, in
main()
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 140, in main
pipeline = DistriPixArtAlphaPipeline.from_pretrained(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 126, in from_pretrained
ret = DistriPixArtAlphaPipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 57, in init
self.prepare()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 179, in prepare
self.pipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pip/distri_pixartalpha.py", line 362, in call
self.comm_manager.irecv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 314, in irecv_from_prev
self.first_recv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 272, in first_recv_from_prev
self.recv_shape = self.recv_shape_comm()
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 233, in recv_shape_comm
dist.recv(dim, src=self.prev_rank, group=self.extra_group if is_extra else None)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1804, in recv
pg.recv([tensor], src, tag).wait()
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1961, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.19.4
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:
socketStartConnect: Connect to <IP_ADDRESS><59809> failed : Software caused connection abort
0%|          | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 226, in
main()
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 140, in main
pipeline = DistriPixArtAlphaPipeline.from_pretrained(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 126, in from_pretrained
ret = DistriPixArtAlphaPipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 57, in init
self.prepare()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 179, in prepare
self.pipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pip/distri_pixartalpha.py", line 362, in call
self.comm_manager.irecv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 314, in irecv_from_prev
self.first_recv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 272, in first_recv_from_prev
self.recv_shape = self.recv_shape_comm()
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 233, in recv_shape_comm
dist.recv(dim, src=self.prev_rank, group=self.extra_group if is_extra else None)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1804, in recv
pg.recv([tensor], src, tag).wait()
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1961, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.19.4
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:
socketStartConnect: Connect to <IP_ADDRESS><59431> failed : Software caused connection abort
DistriPixArtAlphaPipeline from pretrain stage 2 11.83976448 GB
DistriPixArtAlphaPipeline from pretrain stage 2 11.743849472 GB
0%|          | 0/6 [00:00<?, ?it/s]DistriPixArtAlphaPipeline from pretrain stage 2 11.83976448 GB
0%|          | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 226, in
main()
File "/data/fjr/PipeFusion/scripts/./pixart_example.py", line 140, in main
pipeline = DistriPixArtAlphaPipeline.from_pretrained(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 126, in from_pretrained
ret = DistriPixArtAlphaPipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 57, in init
self.prepare()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/fjr/PipeFusion/pipefuser/pipelines/pixartalpha.py", line 179, in prepare
self.pipeline(
File "/data/fjr/PipeFusion/pipefuser/pipelines/pip/distri_pixartalpha.py", line 362, in call
self.comm_manager.irecv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 314, in irecv_from_prev
self.first_recv_from_prev(dtype)
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 272, in first_recv_from_prev
self.recv_shape = self.recv_shape_comm()
File "/data/fjr/PipeFusion/pipefuser/utils.py", line 233, in recv_shape_comm
dist.recv(dim, src=self.prev_rank, group=self.extra_group if is_extra else None)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1804, in recv
pg.recv([tensor], src, tag).wait()
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1961, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.19.4
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:
socketStartConnect: Connect to <IP_ADDRESS><33359> failed : Software caused connection abort
0%|          | 0/6 [00:00<?, ?it/s][2024-06-27 09:15:55,841] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62833 closing signal SIGTERM
[2024-06-27 09:15:55,843] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62834 closing signal SIGTERM
[2024-06-27 09:15:55,843] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62835 closing signal SIGTERM
[2024-06-27 09:15:55,844] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62836 closing signal SIGTERM
[2024-06-27 09:15:55,845] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62838 closing signal SIGTERM
[2024-06-27 09:15:55,863] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62839 closing signal SIGTERM
[2024-06-27 09:15:55,864] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 62840 closing signal SIGTERM
[2024-06-27 09:15:56,301] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 4 (pid: 62837) of binary: /usr/bin/python
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 347, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 812, in main
run(args)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 135, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
scripts/./pixart_example.py FAILED

Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2024-06-27_09:15:55
host      : localhost.localdomain
rank      : 12 (local_rank: 4)
exitcode  : 1 (pid: 62837)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

fix this bug with dist.barrier()

fix this bug with dist.barrier()

how to use?

fix this bug with dist.barrier()

how to use?

Please refer to readme. The script to run example is here.
examples/run.sh


Fixes F# extraction(#1185 , #1210, #1236, #1242, #1277), Updates ulib/fs

Few of us use non-bootstrapped F# version but I'm willing to review some extensions to the F# realization of the F* standard library

The issues regarding F# extraction affect both the bootstrapped and non-bootstrapped F* versions.

@ad-l  can you flag this as needs-review? Something about the fix for unary negation is causing the OCaml build to always return failiure: Option.get called on None.

Thanks, this sequence of patches looks pretty good to me. We need to run the test suite on them to make sure they don't break anything, but of course our %#!$@ CI doesn't do that for external PRs. Ugh.
I'll create a local branch to track yours, and forward the build logs… sorry for the trouble!

Done :) And the checks pass! Woohoo.
Let's wait for a second review, and then wait. It's a shame to have good PRs lingering like this.
Thanks for the cool work!

https://github.com/FStarLang/FStar/pull/1279

Merged! Awesome work :)


Convert to Reason

What would this look like in reason-urql?
  const mutate = React.useCallback(() => {
    executeMutation({ email, password, name })
      .then(({ data }) => {
        const token = data && data[isLogin ? 'login' : 'signup'].token
        if (token) {
          setToken(token)
          props.history.push('/')
        }
      });
  }, [executeMutation, props.history, email, password, name, isLogin]);

https://github.com/howtographql/react-urql/blob/662b62f4768e5e2147d95af5dbf8ab8565b63f72/src/components/Login.js#L33-L42
I can't seem to get the token back and setting it.
 Here is where I am.

  let handleToken = token => {
    Js.log2("TOKEN_DATA", token);
    Token.setToken(token);
    ReasonReactRouter.push("/");
  };

 let submitLogin = () => {
    executeLoginMutation() |> ignore;
    Js.Promise.then_(loginState => {
        Js.log2("TOKEN_loginState", loginState);
        let token = loginState##data##signup##token->Belt.Option.getExn;
        handleToken(token)
        Js.Promise.resolve();
      })
      |> ignore;
    // ReasonReactRouter.push("/");
  };

Here I try to log the response but get nothing:
let submitLogin = () => {
    executeLoginMutation()
    |> response => {
      Js.log2("submitLogin",response)
      Js.log(response)
    };
    ReasonReactRouter.push("/");
  };

I'm probably just not familiar enough with the library. Any guidance would be appreciated. Thank you.

Example: https://gist.github.com/gugahoa/b959b7d0b98b87cc1822b6754b85fa34
That can be seen in action at https://app.kinase.med.br

Obrigado, @gugahoa! So you do you have the SignUp query in its own component? I was trying both in same component like this: https://gist.github.com/idkjs/50e4ce47836f63a53ebcf98119a77628

Yeah, the intent is to have a SignUp query on it's own component, but we haven't implemented it yet. It should be quite similar to the Login query and component, though.

@gugahoa if you still feel like teaching, what is going on here:
  let {user: _, login}: Auth.t = React.useContext(Auth.context);

You have an Auth.re that defines an Auth.context value which has a user property that takes two arguments, a first function that is ignored and a login function. Later, see you call login. Is this the login you are referring to?
Thank you, sir. No worries if you can't respond.

@idkjs Auth.t has type:
type t = {
  user: auth_user,
  login: (Dom.Storage.t, user, token) => unit,
  logout: unit => unit
};

At line 56 in Login.re, I declare a new login:
  let login =
    state.remember ?
      login(Dom.Storage.localStorage) : login(Dom.Storage.sessionStorage);

Later on I use that login declared on line 56


module 'keras_frcnn.config' has no attribute 'pickle'

NOTE: this repo is now deprecated. Instead, I recommend keras users use https://github.com/delftrobotics/keras-retinanet which runs faster and is more accurate.
when I run the following code:
from keras_frcnn import config
config.pickle
the error has occured!
who can help me solve the problem.thanks!

Refer to the resource: https://github.com/softberries/keras-frcnn/tree/master/keras_frcnn


feat: vuepress updated

:tada: This PR is included in version 1.2.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix #4276: shouldn't drop all series when regex doesn't match

+1, tests look fine.
Of course, this was a pretty simple issue, I commented about the switch in case there is something we can do there.


Increase transactions processing speed with blocks vector

& Testing signature generation time
Draft test vectors for processing (with test_network rules changes): https://github.com/SergiySW/raiblocks/tree/debug_processing

Additional fix: blocks from local accounts should be processed & only then returned to RPC/GUI
https://github.com/clemahieu/raiblocks/issues/204


Set Preset Selection Variable

I would like to set the Preset Selection Variable to a raw value not just modify (increase/decrease), it would be easier to display which Preset is actual.
In the moment I solve it with a custom variable, but so it would be simpler.
Or could you tell me the visca command therefore, so I could solve it with the custom command action

Hi @paule26 , I would love to see this added too!
Very interrested in getting your workaround solution!
Would you ming explaining the steps to have a set a raw value to the preset selector variable?

Hi Moulaythami, I create a Custumer variable (e.g. PTZ1_Preset) and add the command "internal: Custom Variable: Set raw value" to the Recall of an Preset (e.g. 9) and as feedback "internal: Variable: Check value" with PTZ_Preset = 9


[python] added class_weight parameter in sklearn wrapper

Fixed #1107.
Should we somehow take into account scale_pos_weight and is_unbalance parameters?

it looks good to me. ping @wxchan  for review.

BTW, I think the mulit-class should only use the class_wight.
And scale_pos_weight and is_unbalance is only used in binary classification.
We can address this in document.

@StrikerRUS please merge if this is ready.


usability improvement/suggestion: prevent naming projects "."

I created an ionic project with ionic start . meaning: start an ionic project in the current (already created) directory.  Figuring I'd name the thing later, I didn't specify the command-line flag for setting the name.  Ionic used '.' as the name for everything, which caused a hard-to-debug mayhem of error messages from within Cordova.  Easy to fix, but also easy to just prevent in the ionic-start code.

This issue was moved to driftyco/ionic-cli#246


Asserts with unmatched XPath nodes

When no node matches an XPath assert, the (string) expected value is parsed as JSON and this crashes httpyac instead of cleaning failing the assert.
I looked at the code and I think I know what happens.

provideAssertValueXPath uses xpath lib to select the matching nodes. If no node in response matches, this would be an empty array.

provideAssertValueXPath maps the array to strings, but the final provided value remains an empty array.
https://github.com/AnWeber/httpyac/blob/main/src/plugins/xml/provideAssertValueXPath.ts#L40-L44
I suggest this is where the fix might be: add a case for empty array, and return undefined instead?

parseAssertLine attempts an auto-conversion of expected value to actual value type.
https://github.com/AnWeber/httpyac/blob/main/src/plugins/assert/assertHttpRegionParser.ts#L57
When XPath matched a node, the target value is a string and there's no conversion: it works great.
But when there's no match, target value is an empty array, and the auto-conversion attempts to parse the expected value as JSON.
https://github.com/AnWeber/httpyac/blob/main/src/plugins/assert/assertHttpRegionParser.ts#L98-L100

Maybe another possible fix, that might be more general:
I guess the fundamental issue is that trying to convert an expected string value into the current target value is inherently fragile, even when matching JSON or other sources. If the target value is not a string as expected that process is likely to crash.
So a better fix could be to wrap that JSON parse in a try..catch and cleanly fail the assert when the target and expected value types are incompatible.
(might still be nice to convert a non-matching xpath selector into undefined).

@jods4 Yeah, what can I say. I agree with you on all points. You can tell that I don't use the XPath myself. For objects, it currently works rather mediocre. My asserts are either very simple or I use the javascript notation right away.

Thank you!

update is released with v6.6.1


Curl issues

A new inspection was created.

A new inspection was created.

A new inspection was created.

A new inspection was created.

The inspection completed: 14 updated code elements

The inspection completed: 14 updated code elements


MSDKUI-1683: Rename GuidanceEstimatedArrivalViewFragment into Guidanc…

…eEstimatedArrivalFragment
Class and unit tests class renamed.
Signed-off-by: Tadeusz Staszak<EMAIL_ADDRESS>

Codecov Report

Merging #42 into master will increase coverage by 2.93%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #42      +/-   ##
============================================
+ Coverage     70.86%   73.79%   +2.93%
+ Complexity     1556      971     -585
============================================
  Files           128       69      -59
  Lines          5443     3782    -1661
  Branches        820      398     -422
============================================
- Hits           3857     2791    -1066
+ Misses          976      763     -213
+ Partials        610      228     -382

Impacted Files
Coverage Δ
Complexity Δ

...e/msdkuiapp/routing/RoutingInstructionsFragment.kt

...java/com/here/msdkuiapp/common/PermissionsUtils.kt

.../com/here/msdkuiapp/routing/OptionPanelFragment.kt

...ere/msdkuiapp/routing/RouteManeuverListFragment.kt

.../here/msdkuiapp/routing/SubOptionPanelPresenter.kt

...m/here/msdkuiapp/routing/SubOptionPanelFragment.kt

...msdkuiapp/guidance/GuidanceNextManeuverFragment.kt

...om/here/msdkuiapp/position/LocationServiceUtils.kt

...ain/java/com/here/msdkuiapp/about/AboutActivity.kt

...e/msdkuiapp/position/LocationPermissionFragment.kt

... and 46 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e631c0...9eb6bf3. Read the comment docs.


Replace Passenger with another server for OATS

Description
ARM64 machines are not compatible with Passenger Phusion. As we're moving to M1 devices in the not-so-distant future, we should prepare for this by changing our OATS server to something that is both reliable (the reason why we moved to passenger/nginx) and compatible with ARM64.
Acceptance Criteria

[ ] ARM64-compatible server used to run OATS server

Design Consideration/Limitations

We stopped using the AGENT_ENGINE environment variable that allowed puma, thin, or webrick to be alternated between for the server and replaced it with passenger/nginx during Summer 2021 to increase reliability of the results

Dependencies

None

Additional context

Passenger is running fine on our GitHub Enterprise SSH machines right now, idk if/when those will be updated to ARM64 devices

Estimates
M

In planing the team decided to shelve anything OATS related for now until there is an APM wide strategy in place for moving forward with it.


page.open and page.onLoadFinished callbacks are invoked before window.onload

I couldn't find an existing ticket, so hopefully this isn't a duplicate...
According to the docs page.onLoadFinished and page.open should invoke their callbacks when the page is finished loading.  However these callbacks are invoked before document.readyState === 'complete' and before window.onload.  In 2.5.0, these callbacks also happen before async scripts are downloaded and run.
1. Which version of PhantomJS are you using? Tip: run phantomjs --version.
$ phantomjs --version
2.1.1
$ phantomjs25 --version
2.5.0-development
$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.10.5
BuildVersion:	14F2109

2. What steps will reproduce the problem?
Do page.evaluate within page.open or page.onLoadFinished and check document.readyState is not "complete".
Load progress timing results follow.  Test code is at the bottom.
phantomjs 2.1.1: page.onLoadFinished happens after DOMContentLoaded and after async scripts are downloaded and run.
$ phantomjs test.phantom.readystate.js
0.18ms	First inline script running.
2.6ms	readystatechange	interactive	[Object]
3.5ms	DOMContentLoaded	[Object]
37.9ms	async script downloaded and ran.
38.44ms	phantomjs page.onLoadFinished found readystate	interactive
38.72ms	phantomjs page.open found readystate	interactive
38.84ms	readystatechange	complete	[Object]
39.14ms	window.onload	[Object]

phantomjs 2.5.0: page.onLoadFinished happens after DOMContentLoaded and before async scripts are downloaded and run.
$ phantomjs25 test.phantom.readystate.js
0.1ms	First inline script running.
0.83ms	readystatechange	interactive	{"isTrusted":true}
0.88ms	DOMContentLoaded	{"isTrusted":true}
36.66ms	phantomjs page.onLoadFinished found readystate	interactive
36.96ms	phantomjs page.open found readystate	interactive
37.13ms	async script downloaded and ran.
37.22ms	readystatechange	complete	{"isTrusted":true}
37.29ms	window.onload	{"isTrusted":true}

Both versions appear to have the page.onLoadFinished callback happen before the page has the onload event fires.
This leads to hacks where people use a long setTimeout (sub-optimal) or poll document.readyState to ensure the document is ready before doing whatever they are going to do.  Would be nice if it worked as expected: page.onLoadFinished callback happens after window.onload or document.readyState === 'complete'.
3. Which operating system are you using?
See above.  OSX Yosemite.
4. Did you use binary PhantomJS or did you compile it from source?
binaries.
5. Please provide any additional information below.
test.phantom.readystate.js
var page = require('webpage').create(),
    url = 'http://somedomain.com/test_onload_onready.html';

function makeString () {
    var str='', i, arg, sep='\t';
    for (i=0; i<arguments.length; i++) {
        arg = arguments[i];
        if (typeof arg === 'object') {
            str += JSON.stringify(arg);
        }
        else {
            str += arg;
        }
        str += sep;
    }
    return str;
}

phantom.onError = function(msg, trace) {
    console.log('phantom.onError:' + makeString.apply(this, arguments));
};

page.onError = function(msg, trace) {
    console.log('page.onError:' + makeString.apply(this, arguments));
};

page.onResourceError = function(resourceError) {
    console.log('page.resourceError:' + makeString.apply(this, arguments));
};

page.onConsoleMessage = function(msg) {
    console.log('page.onConsoleMessage:' + makeString.apply(this, arguments));
};

page.onLoadFinished = function(status) {
    if (status !== 'success') {
        console.log('Error: Unable to access network ('+status+')');
    } else {
        page.evaluate(function() {
            log('phantomjs page.onLoadFinished found readystate', document.readyState);
        });
    }
};

page.open(url, function(status) {
    var readyState;
    if (status !== 'success') {
        console.log('Error: Unable to access network ('+status+')');
    } else {
        page.evaluate(function() {
            log('phantomjs page.open found readystate', document.readyState);
        });

        // delay to give events a chance to fire before we check test log
        setTimeout(function () {
            var logText = page.evaluate(function() {
                return logText;
            });
            console.log(logText);
        }, 5000);
    }

    // use setTimeout to reduce spurious resourceErrors
    //   ie: "Operation canceled" for async scripts
    //  happens when phantom exits before pending requests get their http responses.
    setTimeout(function () {
        phantom.exit();
    }, 9000);

});

test_onload_onready.html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title></title>
<script type="text/javascript">
var logText = '',
    t0 = performance.now();
function log () {
    var i, arg,
        argSep='\t',
        time = performance.now() - t0;
    time = Math.round(time*100)/100;
    logText += time + 'ms' + argSep;
    for (i=0; i<arguments.length; i++) {
        arg = arguments[i];
        if (typeof arg === 'object') {
            try {
                logText += JSON.stringify(arg);
            } catch (e) {
                logText += '[Object]';
            }
        }
        else {
            logText += arg;
        }
        logText += argSep;
    }
    logText += '\n';
}
log('First inline script running.');
document.onreadystatechange = function (e) {
    log('readystatechange', document.readyState, e);
};
window.document.onload = function (e) {
    log('window.document.onload', e);
};
window.onload = function (e) {
    log('window.onload', e);
};
window.addEventListener('DOMContentLoaded', function (e) {
    log('DOMContentLoaded', e);
}, false);
</script>
</head>
<body>
<script async src="async_script.js"></script>
</body>
</html>

async_script.js
log('async script downloaded and ran.')

+1 .. I am using a long timeout after phantomjs says the page is loaded as well because of this.

Upgrading to version 2.1.1 fixed my issue that onLoadFinished didn't evaluated my window object:
    page.onLoadFinished = function(status) {
        console.log(status);
        var fyndle = page.evaluate(function() {
            return window.fyndle;
        });

        console.log(JSON.stringify(fyndle,  undefined, 4));
        phantom.exit();

    };


[BUG] passing partition key for reading conflict is not consistent with other APIs

see the existing sample
https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/main/src/main/java/com/azure/cosmos/examples/conflictfeed/async/SampleConflictFeedAsync.java#L324-L325
for Document read apis we explicitly ask PartitionKey to be passed along id for read container.readItem(id, pk, ..) we should have similar API for conflict read and deprecate the apis for setting partition key in the ConflictOptions.
relevant CRI:
https://portal.microsofticm.com/imp/v3/incidents/details/226232248/home

We no longer plan to do this.


[#448] Move JS related files to Bots/JavaScript folder

Fixes # 448
Summary
This PR moves all the lint and yarn files used for JS to the Bots/JavaScript folder.
Specific changes

Moved the following files into the Bots/JavaScript folder:

.yarn folder
package.json
yarn.lock

Updated .gitignore with current files location
Updated jsBotsBuildCI.yml to run lint and yarn command in the new location

Promoted to MS PR 456


Please create a USER FRIENDLY user interface to set powershell execution policies

Summary of the new feature / enhancement
At the time of writing i own a new microsoft pc for about 1 day.
Powershell restricts all execution of powershell scripts by default.
I need to enable it through a command that is far away hidden.
Please have a setting to allow or configure execution policies of powershell through a user interface or revise the security restrictions. What is the use of security if it blocks completely the funcitonality of the system?
Proposed technical implementation details (optional)
No response

Thanks for submitting this issue @FlightControl-User! In Settings>Privacy & Security > For Developer> PowerShell there is a setting to allow local scripts to run without signing changing the execution policy. This currently only works for Windows PowerShell version 5.1, the version of PowerShell built shipped inbox with Windows. Are you interested in having a similar setting for PowerShell 7?

Thanks for your constructive reply. I would recommend a yes and a no. It's double.
Allow me pls to speak freely. As a simple mind and observer from a distance ...
Powershell is a scripting language and enables to do more things with your computer than it provides out of the box. What good is it to buy a new laptop (from Microsoft), then wanting to use the new laptop using powershell scripts (which worked on my previous laptop and synchronized through one drive on my new local machine), to get blocked by default ???
Also, does bash (linux subsystem) prevent out of the box to run shell scripts ??? I mean, i just feel historically this all has grown as a kind of a mess and needs a revision.
I would suggest to enable remotesigned scripts by default on the machines, and for those who want extra security to prevent or configure extra settings.
And yes, such a UI feature for powershell 7 is needed, or again a lot of complaints will come!
I would recommend Microsoft to think over carefully what the end to end solution and security model is still a fit with modern times ...

@StevenBucher98, while that would be nice, the challenge with respect to PowerShell Core is twofold:

multiple versions may be installed side by side, which would complicate the UI
versions may be installed in custom locations that may not be discoverable via $env:PATH

@FlightControl-User, I agree that RemoteSigned would make for a more sensible default - and in fact it is the default, but only on (recent) server editions of Windows.
(On a more fundamental note, you could argue that simply having no execution policies may be preferable - seems to be no problem for PowerShell Core on Unix).
As for:

I need to enable it through a command that is far away hidden.

It's not hard to discover the relevant command, but it's harder than it needs to be:
Currently, the error message you get when a script is blocked is:
'...' cannot be loaded because running scripts is disabled on this system. For more information,
see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170

The link leads to about_Execution_Policies, and if you're running in Windows Terminal you can even Ctrl-click it directly and it will open in your browser.
That said, given the conceptual nature of the topic, the relevant commands do not appear until much further down the page.
I think it would help if the current error message were amended to include something like the following:
For instance, run `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned -Force`
to enable script execution for the current user for all scripts except those downloaded from the web.

Hello there,
Thanks for replying. I read your feedback with great interest and thought this over a bit ...
Please consider the below considerations:

multiple versions may be installed side by side, which would complicate the UI

@StevenBucher98 , @mklement0 Afraid this is a developer answer. Please think from an end-user perspective. Also, when you buy a car, is a Toyota RAV4 the same as a Toyota Prius? No, they have different dashboards. So yes, the UI may be different depending on the powershell you have on your machine! I'm sorry but complexity comes with design and the battle of backward compatibility needs to be dealt with, considering that this backward compatibility is required.

It's not hard to discover the relevant command, but it's harder than it needs to be:

I would like to emphasize that it is hard to discover the relevant command! An end user needs to jump into a terminal, (imagine, on a user interface driven operating system!). Then, he needs need to magically know the command to set user policies, which indeed appear in the error, but, you need to "see" the error. Sometimes this error appears hidden in an other component and the end user may not even know what is going on. Does Microsoft really want an end user to type in a command like Set-ExecutionPolicy -Scope CurrentUser RemoteSigned -Force ???
I spent about half an hour searchin through the quite technical explanations of Microsoft on the Set-ExecutionPolicy which is simply incomprehensible for end users!!! So please think from an end user perspective how the Powershell scripting subsystem can be improved/serviced. What functions would you want to embed in Windows to make it more easy for end users to use the system? Is it only "execution policies" or are there more functions to consider?

(On a more fundamental note, you could argue that simply having no execution policies may be preferable - seems to be no problem for PowerShell Core on Unix).

@mklement0 As stated by myself earlier, I confirm this also, linux, bash has no issue with execution policies at all. So imagine how confusing this all is for end users ...

I think it would help if the current error message were amended to include something like the following:

@mklement0 Thank you! That is a good solution, but not the solution as an end user. But it will definitely help!
So to summarize:

Consider from an end user perspective how Powershell configuration can be enhanced within the UI.
Is execution policy the only thing?
Consider the -RemoteSigned as the default. Consider even patching this in existing windows systems.
Consider, as @mklement0 correctly has advised, to correct the error message.

Don't expect me to respond any further to this feedback. I think the message has been written clear enough.

@FlightControl-User until (if ever) PowerShell 7 is included in Windows, then I don't think you will get a UI baked into Windows for it,
However IMO the right place for this kinda feedback would be via the Windows Insider program and it's Feedback hub as this is a Windows UI request (IMO) and not so much PowerShell specific (although the lines are blurry here), so I'd recommend that you raise it there unless others have already done so as I'm not going to.
Also most end users will never use PowerShell 7 (or even PowerShell 5.1) so this is perhaps this ask isn't the right one.
re this

Does Microsoft really want an end user to type in a command like Set-ExecutionPolicy -Scope CurrentUser RemoteSigned -Force ???

Yes, if they want to learn how to end up being a developer and not just an end user. Because by showing them that and the info in the error message it gives them the choice to go further or not. Which is the right way forward to enabling everyone to reach their true potential. As they say "Curiosity is only a good thing" we should give as many chances as possible to enable people to become more curious.

I've given up on the insider program. It is a time wasting resource hog that brings no direct benefits.
The project manager of the PowerShell team has responded in this request so I assume that he will pick this idea up. I don't think it requires an insider program issue beside the other thousands of issues, and then this request slipping through because nobody reads it.

@FlightControl-User IMO you are very wrong in your assumption about the insider program and perhaps are expecting changes to occur far too quickly than is ever realistically going to happen. Also I was suggesting to you that route as one that you could undertake based on my many years of being very heavily invested in the mass majority of the Microsoft ecosystem, including Windows as an insider since 2014 and until recently the MVP program since 2016.
Whilst its fair to feel like its a waste of time, I think given enough time they will correct some of the things that aren't currently ideal, just like they have always done.
re this

I don't think it requires an insider program issue beside the other thousands of issues, and then this request slipping through because nobody reads it.

I think it also needs one as different teams read different feedback mechanisms so being able to link them to get gives much more weight, in my experience 🙂


Remove hashes as we don't have hashes for all platforms

Having single hashes means the plugin can fail to build if someone is using the plugin on a platform that we haven't got a hash for.

Shouldn't these be consistent across all platforms? Pure Python wheels should be

Hmm yeah I guess we only need to do this for the bootstrap deps that aren't pure python
Interestingly portalocker is: https://pypi.org/project/portalocker/#files
Not sure why this hash is breaking.


Error state positioned incorrectly

When filtering details tables, I noticed that the error state is positioned incorrectly. The component should not appear at the bottom of the page.
This regression seems to have occurred after updating to the PatternFly "breaking changes release".
https://issues.redhat.com/browse/COST-306

Codecov Report

Merging #1597 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1597   +/-   ##
=======================================
  Coverage   78.74%   78.74%
=======================================
  Files         237      237
  Lines        3886     3886
  Branches      713      713
=======================================
  Hits         3060     3060
  Misses        727      727
  Partials       99       99

Impacted Files
Coverage Δ

.../state/maintenanceState/maintenanceState.styles.ts
100.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ec5f3b...ac0af8d. Read the comment docs.


block device  is in not in active state

Description
i have a hdd device on my node ,called /dev/sdc .it swich on active and unactive in the bd

where can i find the log and fix this
for: "cstor-cspc.yaml": admission webhook "admission-webhook.cstor.openebs.io" denied the request: invalid cspc specification: invalid pool spec: block device "blockdevice-1ac102847bc0adf6a765400b7104879f" is in not in active state
Your Environment

kubectl get nodes:
NAME      STATUS   ROLES                  AGE    VERSION
eb60246   Ready    node                   540d   v1.23.12
eb60247   Ready    node                   540d   v1.23.12
eb60248   Ready    node                   540d   v1.23.12
eb60249   Ready    node                   540d   v1.23.12
eb60250   Ready    node                   540d   v1.23.12
eb63218   Ready                     553d   v1.23.12
eb63221   Ready    control-plane,master   553d   v1.23.12
kubectl get pods --all-namespaces:
NAME                                                              READY   STATUS    RESTARTS      AGE
cstor-disk-pool-hq68-544674d668-77rvs                             3/3     Running   0             63m
openebs-cstor-admission-server-57c895b649-6kf9r                   1/1     Running   0             3h37m
openebs-cstor-csi-controller-0                                    6/6     Running   0             3h41m
openebs-cstor-csi-node-294fs                                      2/2     Running   0             3h44m
openebs-cstor-csi-node-7cw4d                                      2/2     Running   0             3h44m
openebs-cstor-csi-node-fqmnf                                      2/2     Running   0             3h43m
openebs-cstor-csi-node-km2tz                                      2/2     Running   0             3h44m
openebs-cstor-csi-node-pbfzz                                      2/2     Running   0             3h44m
openebs-cstor-csi-node-pnxmx                                      2/2     Running   0             3h42m
openebs-cstor-cspc-operator-5bc489cbd7-29j68                      1/1     Running   0             3h46m
openebs-cstor-cvc-operator-78b57f5f88-5cf22                       1/1     Running   0             3h37m
openebs-localpv-provisioner-6dfd6d789d-kcp8h                      1/1     Running   0             3h46m
openebs-ndm-hs95c                                                 1/1     Running   0             12m
openebs-ndm-ncdzt                                                 1/1     Running   0             3h46m
openebs-ndm-operator-6fb8987ff7-95996                             1/1     Running   0             3h46m
openebs-ndm-pq6cr                                                 1/1     Running   0             12m
openebs-ndm-q2lsg                                                 1/1     Running   0             12m
openebs-ndm-vsnw5                                                 1/1     Running   0             12m
openebs-ndm-zwq7h                                                 1/1     Running   0             12m
OS (from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (from uname -a):
Linux eb60247 3.10.0-1160.el7.x86_64 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Install tools:
helm

Please note:  cStor is a Legacy project and has been archived.
This was done in May/April 2024 on guidance with CNCF TOC and CNCF leadership.
cStor is no longer actively managed as a part of the openEBS project, and all cStor repo's have been migrated to the CCNF  openEBS Archive org
You can get community support for cStor on the openEBS slack channel. The openEBS eng/dev/QA team are very active in that CNCF Slack support channel and may provide best effort to support you (with no guarantee).
We advise all cStor users to migrate to openEBS MayaStor.

This issue will be closed in 7 days.

Closing this based on the above comments.


Remove ... to allow flog to work

So looks like code climate doesn't include the specs for the flog run, eliminating a lot of false positives. Lots of issues being picked up here, and a quick oggle of the top ones looks like its grabbing some pretty crunch methods.


Problem with creation subfolder in Android Device

Hi,
I use solar2d v. 3674,
every month I release an app containing a small website with its subfolders. This month I compiled the app, as always, but the site view goes into error no longer viewing the contents of the folders (images / logo.png example): debugging the code I saw that the subfolders in the zip file are not created in the documentsDirectory of the device.
1 - I have a zip file containing the miniSite, with the subfolders
2 - In buildSetting there is the plugin declaration
3 - It always worked great up to 15 days ago
On iOS version all ok
On Android make this error
Some idea?
All the best
Pasquale

Hi,
is there a way to go back to a previous version of the plugin?
Kind Regards
Pasquale

Hello! I just updated Android part of the plugin building it with old toolchain. Please, verify if it helped your. Update should be live now.

Yes! Thanks for your quick fix. All the Best, Pasquale


[PRESET] Create/Update Compiler Preset PSP Castlevania: Symphony of the Night

I'd like for -sym on to be added to the preset to enable line numbers.
Current preset:
        {
            "id": 132,
            "name": "Castlevania: Symphony of the Night",
            "platform": "psp",
            "compiler": "mwccpsp_3.0.1_219",
            "assembler_flags": "",
            "compiler_flags": "-Op -lang c -sdatathreshold 0",
            "diff_flags": [],
            "decompiler_flags": "",
            "libraries": [],
            "num_scratches": 77,
            "owner": null
        },

Preset Name: Castlevania: Symphony of the Night
Preset ID (when updating): 132
Platform: e.g. psp
Compiler ID: e.g. mwccpsp_3.0.1_219
Compiler Flags: e.g. -Op -lang c -sdatathreshold 0 -sym on
Libraries (if applicable):  N/A

dun :)