[ROCm] Adding scratch_size field to dnn::AlgorithmDesc

Description:
Currently the scratch_size (workspace memory size), required to execute
a given convolution using a given algorithm, is determined via a call to
the CUDNN API. This call happens each time a convolution kernel is
called, because the scratch_size is not stored on neither the
dnn::AlgorithmDesc nor the dnn::AlgorithmConfig datastructures. Adding
scratch_size to dnn::AlgorithmDesc will remove the need to call the
CUDNN API to retrieve this information everytime we need to do
convolution.
There are also some ROCm platform specific reasons for making this change.

on the ROCm platform, there is no equivalent of the CUDNN API to
determine the workspace memory size corresponding to a given
(conv_params, algo) tuple. There is a ROCm API to determine the
workspace memory size for a given conv_params, but that gives a
worst case answer since that API does not take a specific algo as an
input. As a consequence, on the ROCm fork of TF, we maintain a
scratch_size field on the dnn::AlgorithmConfig data-structure. This
change will remove the need for this discrepancy in the ROCm TF fork.

A side-effect of this commit is that the GetConvolve*Algorithms
routines will now take the conv_params (input, filter, conv, output)
as input. This will allow TF code to leverage underlying CUDNN /
ROCm MIOpen APIs that return a list of algos that are applicable to
the given conv_params. Currently the list of algorithms list is
compiled in the TF code itself, without querying the CUDNN / MIOpen
APIs.

@tatianashp : just FYI
@timshen91 : I am changing code that you recently refactored and hence including you on the cc-list
@chsigg @whchung

I am unable to locally reproduce the failures listed in the "GPU Python3" details. All 15 of the failing tests listed there, pass for me locally (I am using CUDA 9.0 + CUDNN 7.3.1).
Same applies to the 2 failures int he "GPU CC" details

+@chsigg as well.
For the context, internally we are discussing improving the API surface of StreamExecutor. Specifically, we don't think that it makes sense to make a consistent layer of API for both cuDNN and MIOpen. Today, they are already different in a rather non-trivial way. MIOpen may also want to reserve the right to diverge further in the future.
In the end, we should remove the unified API "DnnSupport", and instead use thin wrappers over MIOpen, cuDNN, and XBlas libraries, etc. The wrappers do two things:

C++-ize the API
decouple the build-time dependency of cudnn/miopen using dynamic loading.

And users should feel comfortable directly using MIOpen-specific API in their code.
@chsigg is working on that, but that doesn't happen in fortnight. For now, we want to make whatever changes that solves the problem this PR solves, but also be align with the goal above.
What I'm thinking is

add specific MIOpen APIs we need into DnnSupport, and mark them as MIOpen-only in the comments. They should be dead-thin wrappers.
In all callers, whenever necessary, branch on whether the target is MIOpen, and use the API differently. I'd prefer a runtime boolean branching, instead of "#ifdef".

This approach certainly puts changes on all user code e.g. tf kernels and XLA, but you already change them anyway.
What do you think?

@timshen91

In the end, we should remove the unified API "DnnSupport", and instead use thin wrappers over MIOpen, cuDNN, and XBlas libraries, etc

It would be nice if we can get there. Let us know if we can help in anyway.
Will the above change be limited to the DNN Support layer (cuDNN/MIOpen) or will it also extend to the other parts of the Stream Executor (blas, fft, rng, etc).

What I'm thinking is
...
What do you think?

I think I follow what you mean, but a concrete example will help ensure we are on the same page.

add specific MIOpen APIs we need into DnnSupport, and mark them as MIOpen-only in the comments. They should be dead-thin wrappers.

example for this  would be the dnn::DoFusedBatchNormActivationInference routine added by this PR which currently only has a MIOpen implementation. It is missing an explicit comment indicating that this routine is only functional in ROCm / MIOpen, but I think this is what you had in mind. Is my understanding correct?
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/google_upstream_add_miopen/tensorflow/stream_executor/dnn.h#L2392
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/google_upstream_add_miopen/tensorflow/stream_executor/rocm/rocm_dnn.cc#L4203

In all callers, whenever necessary, branch on whether the target is MIOpen, and use the API differently. I'd prefer a runtime boolean branching, instead of "#ifdef".

We actually go the #ifdef route in our fork route now.
I am guessing we still need to #ifdef the `REGISTER_KERNEL* macros for GPU_DEVICE.  Currently they are all #ifdef CUDA and we have changed them to be #ifdef CUDA || TENSORFLOW_USE_ROCM in our code...see example below:
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/core/kernels/conv_ops.cc#L549
We then reuse the same approach when the manner in which we use the DNN Support API diverges between CUDA and ROCM...see example below
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/core/kernels/conv_ops.cc#L899
My assumption is that the runtime boolean suggestion only applies to the later scenario. Is the correct? If so, where should this boolean be defined, and what would be the correct way to set it?
Thanks
deven

@timshen91

In the end, we should remove the unified API "DnnSupport", and instead use thin wrappers over MIOpen, cuDNN, and XBlas libraries, etc

It would be nice if we can get there. Let us know if we can help in anyway.
Will the above change be limited to the DNN Support layer (cuDNN/MIOpen) or will it also extend to the other parts of the Stream Executor (blas, fft, rng, etc).

What I'm thinking is
...
What do you think?

I think I follow what you mean, but a concrete example will help ensure we are on the same page.

add specific MIOpen APIs we need into DnnSupport, and mark them as MIOpen-only in the comments. They should be dead-thin wrappers.

example for this  would be the dnn::DoFusedBatchNormActivationInference routine added by this PR which currently only has a MIOpen implementation. It is missing an explicit comment indicating that this routine is only functional in ROCm / MIOpen, but I think this is what you had in mind. Is my understanding correct?
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/google_upstream_add_miopen/tensorflow/stream_executor/dnn.h#L2392
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/google_upstream_add_miopen/tensorflow/stream_executor/rocm/rocm_dnn.cc#L4203

In all callers, whenever necessary, branch on whether the target is MIOpen, and use the API differently. I'd prefer a runtime boolean branching, instead of "#ifdef".

We actually go the #ifdef route in our fork route now.
I am guessing we still need to #ifdef the `REGISTER_KERNEL* macros for GPU_DEVICE.  Currently they are all #ifdef CUDA and we have changed them to be #ifdef CUDA || TENSORFLOW_USE_ROCM in our code...see example below:
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/core/kernels/conv_ops.cc#L549
We then reuse the same approach when the manner in which we use the DNN Support API diverges between CUDA and ROCM...see example below
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/core/kernels/conv_ops.cc#L899
My assumption is that the runtime boolean suggestion only applies to the later scenario. Is the correct? If so, where should this boolean be defined, and what would be the correct way to set it?
Thanks
deven

@timshen91

In the end, we should remove the unified API "DnnSupport", and instead use thin wrappers over MIOpen, cuDNN, and XBlas libraries, etc

It would be nice if we can get there. Let us know if we can help in anyway.
Will the above change be limited to the DNN Support layer (cuDNN/MIOpen) or will it also extend to the other parts of the Stream Executor (blas, fft, rng, etc).

Yes, that's what I meant. It's not limited to DNN, but any domain-specific vendor logic.

What I'm thinking is
...
What do you think?

I think I follow what you mean, but a concrete example will help ensure we are on the same page.

add specific MIOpen APIs we need into DnnSupport, and mark them as MIOpen-only in the comments. They should be dead-thin wrappers.

example for this would be the dnn::DoFusedBatchNormActivationInference routine added by PR #25436 which currently only has a MIOpen implementation. It is missing an explicit comment indicating that this routine is only functional in ROCm / MIOpen, but I think this is what you had in mind. Is my understanding correct?

Yes, I think DoFusedBatchNormActivationInference is a good example to follow for now. Maybe DoMIOpenFusedBatchNormActivationInference is a better name.

https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/google_upstream_add_miopen/tensorflow/stream_executor/dnn.h#L2392
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/google_upstream_add_miopen/tensorflow/stream_executor/rocm/rocm_dnn.cc#L4203

Oh this is even better. You already have the separate API service MIOpenSupport. Maybe you can just upstream the whole class?

In all callers, whenever necessary, branch on whether the target is MIOpen, and use the API differently. I'd prefer a runtime boolean branching, instead of "#ifdef".

We actually go the #ifdef route in our fork route now.
I am guessing we still need to #ifdef the `REGISTER_KERNEL* macros for GPU_DEVICE. Currently they are all #ifdef CUDA and we have changed them to be #ifdef CUDA || TENSORFLOW_USE_ROCM in our code...see example below:
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/core/kernels/conv_ops.cc#L549

Correct, we are not touching those.

We then reuse the same approach when the manner in which we use the DNN Support API diverges between CUDA and ROCM...see example below
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/core/kernels/conv_ops.cc#L899
My assumption is that the runtime boolean suggestion only applies to the later scenario. Is the correct? If so, where should this boolean be defined, and what would be the correct way to set it?

Correct. I was only suggesting all "vendor domain-specific libraries" callers (dnn, blas, etc) to use runtime booleans, maybe like the following:
if (stream_executor->SupportsMiOpen()) {
  /* MIOpen-specific functions, with dynamic loading */
}

As for SupportsMiOpen(), it can be as simple as return TENSORFLOW_USE_ROCM;. My suggestion on runtime boolean should introduce no functional change, but makes building the tests faster.
To illustrate the motivation, imagine that we do a full build of CUDA, run all unit tests, then switch to ROCM, do a full build, and run all unit tests.
If the TF ops use #ifdef, they have to be rebuilt when switching to ROCM; if the ops use a runtime boolean function SupportsMiOpen(), and SupportsMiOpen() is defined in a single .cc file, switching to ROCM effectively requires to only re-compile that file.
BTW, my guess is that, we also want to use gcc/clang to compile such op, not nvcc or hipcc; in long term, nvcc only compiles CUDA kernels, and hipcc only compiles HIP kernels. The goal is to reuse the build results across the bazel configurations.
Does "reduce build time" scenario make sense to you? From what I heard, people complain about TF's long-build time. Out of curiosity, do you think such problem exists, and deserves a solution? If so, we may just start with reducing #ifs...

rebased twice

once to remove the merge conflicts
once to add MIOpen part of the changes for this PR (now that PR #25436 has been merged)

In today's sync this PR was mentioned. Are you still working on this, or it's my turn?

Also, to reiterate: I suggested not to use "#ifdef" under the assumption that it's easy to implement. If it turns out not to be easy, describe so and we can use "#ifdef" for now.

@timshen91
Thank you for input during the meeting yesterday, it was very informative.
Regarding this PR, I think the ball is in your court. I do not believe there are any further changes that need to be done on my end, please let me know if that is not the case.
Regarding the #ifdef discussion, I think it is somewhat orhtogonal to this PR. It is more applicable to the future PRs we will be filing, and hence my interest in the outcome for it. Once the current batch of ROCm PRs is merged, we will have all the pre-requisites in place to start enabling ROCm support for TF operators. Most of that code that does GPU support for TF operators is currently within
#ifdef GOOGLE_CUDA

and the PRs that we file will either change that to either
#if defined(GOOGLE_CUDA) || defined(TENSORFLOW_USE_ROCM)

or
#ifdef GOOGLE_CUDA
...
#elif TENSORFLOW_USE_ROCM
...

depending on whether or not the ROCm implementation for the TF operator needs to diverge from that of CUDA.
Changing from  #ifdef to a runtime bool requires, at the very least, changing all CUDA/ROCm libs to be dynamically loaded, and thorough testing on both platforms.  I expect this to require a lot of testing effort, and what was not clear was whether this was something you were already in the process of doing, or something you were thinking of doing. Knowing that we were going to have a online meeting,  we took the opportunity to discuss this live.
I guess the conclusion on it for now is that

ROCm team should continue with the #ifdef route for the PRs that we will file in the near future.
Google team will evaluate the feasibility of making the switch from #ifdef to runtime bool
If it is deemed a viable approach,

Google team will communicate the details to the ROCm team
ROCm team will evaluate/test whether it works on the ROCm TF

Let me know if above summary is inaccurate / needs changes.
Thanks
deven

No problem, let's forget about the runtime-bool for a moment.
Looking at the commit, it doesn't seem to deliver what I commented before (quoting my previous comment):

add specific MIOpen APIs we need into DnnSupport, and mark them as MIOpen-only in the comments. They should be dead-thin wrappers.
In all callers, whenever necessary, branch on whether the target is MIOpen, and use the API differently. I'd prefer a runtime boolean branching, instead of "#ifdef".

But I still see GetConvolveAlgorithms() is changed to a union for both cuDNN and MIOpen; I imagine it should be "MIOpenGetConvolveAlgorithms()as a pure addition. I also didn't see any dispatch incudnn_conv_runner.cc; I imagine it should call GetConvolveAlgorithms()orMIOpenGetConvolveAlgorithms()`, depending on which target is built for.
I know that the ROCm fork is already implemented in one way, and I'm sorry that it's probably harder for you to rebase your rest of the patches onto the suggested design, but I think we should go for the right solution.
In addition, let's talk about the runtime-bool in XLA. I knew from @jlebar that XLA doesn't contain any #if GOOGLE_CUDA, and you already landed some of the XLA patches. Supposedly we already have some runtime-bool for ROCm in XLA?

@timshen91
I think we have a misunderstanding.
The signature of GetConvolve*Algorithms() routines is being changed, because on the CUDA side, I need the extra params to determine the scratch_size for any given algo, and I need access to the scratch_size to pass to the AlgorithmDesc constructor.
We will eventually need the same extra args on the MIOpen side as well (in a couple of months), and hence this update to GetConvolve*Algorithms() will be applicable to MIOpen too.
I took your comment regarding "specifc MIOpen APIs" to mean to apply towards something future work where we might need to introduce some MIOpen specific funtionality.
In this PR, I guess the MIOpen specific requirement is being added to the AlgorithmDesc data structure (i.e. the addition of the scratch_size field).  This results in the GetConvolve*Algorithms() API being updated, but those updates are not cuDNN/MIOpen specifc.

@timshen91
just wanted to confirm with you that you are not expecting any further changes on my end.
thanks
deven

@timshen91 : rebased this PR to remove the merge conflicts.

Reading back the PR description, I see that for CUDA (and maybe also ROCM), the PR intends to reduce the number of calls to GetConvolve*Algorithms

This PR should not have any impact on the order/number of times the GetConvolve*Algorithms() / Convolve() routines are called.
This PR simply adds extra arguments to pass to the GetConvolve*Algorithms(), because those are needed to call the CUDA (and eventually MIOpen) APIs that determine the scratch_size needed for a given (conv_params, algo) tuple.

Determining the scratch_size needs to now be done in GetConvolve*Algorithms(), because it needs to be passed to the constructor of AlgorithmDesc, which is instantaited by the GetConvolve*Algorithms()

Now that we will have scratch_size stored in AlgorithmDesc, we will no longer need to call the CUDA/MIOpen API to determine the scratch_size during each call to Convolve()...we can grab that value directly from AlgorithmDesc
Hoping this clears things up.
Let me know if you need anything else from my side.

@timshen91 : rebased this PR to remove the merge conflicts.

Reading back the PR description, I see that for CUDA (and maybe also ROCM), the PR intends to reduce the number of calls to GetConvolve*Algorithms

This PR should not have any impact on the order/number of times the GetConvolve*Algorithms() / Convolve() routines are called.
This PR simply adds extra arguments to pass to the GetConvolve*Algorithms(), because those are needed to call the CUDA (and eventually MIOpen) APIs that determine the scratch_size needed for a given (conv_params, algo) tuple.

It increasingly sounds like you want to look at PrepareForConvolution(). It takes (conv_params, AlgorithmConfig) and returns (AlgorithmDesc, scratch_size). As a special case, it takes (conv_paras, AlgorithmConfig(specific_algo)) and returns (specific_algo, scratch_size), which looks like what you described.

PrepareForConvolution() might be too late for what is needed here. AlgorightmDesc is instantiated in the GetConvolve*Algorithms routines, and hence the need to determine the scratch_size within that routine, (so that we can pass it to the constuctor of AlgorithmDesc).  We can get around the need to specify the scratch size value at construction by having a "set" routine for it, but there is no compelling reason to create such a routine, because there should be no reason to change the value of the scratch_size field once initialized.
Another reason not go the PrepareForConvolution() route would be that it is called everytime we do convolution. The GetConvolve*Algorithms routine is called only once per cache miss in the AutotuneConv map.

PrepareForConvolution() might be too late for what is needed here. AlgorightmDesc is instantiated in the GetConvolve*Algorithms routines, and hence the need to determine the scratch_size within that routine, (so that we can pass it to the constuctor of AlgorithmDesc). We can get around the need to specify the scratch size value at construction by having a "set" routine for it, but there is no compelling reason to create such a routine, because there should be no reason to change the value of the scratch_size field once initialized.

GetConvolve*Algorithms is called in the kernels (kernels/conv_ops*). I think you want to change the ops in order to reflect what's going on there. The result is pretty readable, since everything happens here. For example, this is roughly what we have today:
#ifdef GOOGLE_CUDA
if (/* not autotuned */) {
  for (auto algorithm : GetConvolve*Algorithms(...)) {
    stream->ThenConvolve(...)
  }
  /* populate autotuning cache */
}
stream->ThenConvolve(...); // the actual convolution
#endif

I think you should feel free to add #ifdef TENSORFLOW_USE_ROCM here, e.g.
#ifdef TENSORFLOW_USE_ROCM
if (/* not autotued */) {
  RocmSpecificAutotuning(...);
  /* populate autotuning cache */
}
stream->ThenConvolve(...); // the actual convolution
#endif

...given that the cuDNN vs MIOpen API is sufficiently different. Depending on the actual content of RocmSpecificAutotuning(), we may or may not be able to share the same use of PrepareForConvolution() between CUDA and ROCM. If they aren't able to share PrepareForConvolution(), that's totally fine too.

Another reason not go the PrepareForConvolution() route would be that it is called everytime we do convolution. The GetConvolve*Algorithms routine is called only once per cache miss in the AutotuneConv map.

Autotuning is also in kernels/conv_ops*. I suggest to change those "top-level" implementations to reflect the underlying cuDNN and MIOpen difference.

@deven-amd is this PR still WIP ?

@rthadur .... it is on the back-burner for now, as we focus on the PRs for upstreaming ROCm support for various ops.
I will get back to this in a week or two...will ping @timshen91 when I do. I suspect that this PR will require more discussion/changes before we reach an agreement on it :)
deven

It has been 14 days with no activity and the awaiting response label was assigned. Is this PR still valid? Assigning the stalled label. Please comment to reassure me that this is still being worked on.

Closing it out this PR, since there have been changes on the TF side and there are forthcoming changes in the ROCm implementation that impact the change being discussed here. We anticipate that we will not be needing the changes in this PR.
deven


build_and_deploy: update to v0.0.9

This update will deploy as final when automation tests have been passed
for the previous merge commit.
Changelog-entry: Update build and deploy workflow to v0.0.9
Signed-off-by: Alex Gonzalez<EMAIL_ADDRESS>

@resin-jenkins retest this please

@balena-ci retest

@balena-ci retest this please

@resin-jenkins retest this please

@resin-jenkins retest this please

@resin-jenkins retest this please

This DT is now discontinued.


min/max limit

in the current implementation you can limit the dates by using:
    $scope.$watch 'model.filters.endTime', (value) ->
        if value > new Date()
            $scope.model.filters.endTime = new Date()

but this still allows the user to navigate to unwanted future days
i think there should be a min/max limit that blocks the user from navigating outside the wanted range, i recommand using the min and max attribute and integrate them like they work for the input[date] element

That would be very nice indeed. Any ideas on how to implement that?

I worked on the max and min validation. I "borrowed" some code from angular.js source and wrote the following inside directive date-time's link function:
                function isValidDate(value) {
                    // Invalid Date: getTime() returns NaN
                    return value && !(value.getTime && value.getTime() !== value.getTime());
                }

                if (angular.isDefined(attrs.min) || attrs.ngMin) {
                    var minVal;
                    ngModel.$validators.min = function (value) {
                        return !isValidDate(value) || angular.isUndefined(minVal) || value >= minVal;
                    };
                    attrs.$observe('min', function (val) {
                        minVal = new Date(val);
                        ngModel.$validate();
                    });
                }

                if (angular.isDefined(attrs.max) || attrs.ngMax) {
                    var maxVal;
                    ngModel.$validators.max = function (value) {
                        return !isValidDate(value) || angular.isUndefined(maxVal) || value <= maxVal;
                    };
                    attrs.$observe('max', function (val) {
                        maxVal = new Date(val);
                        ngModel.$validate();
                    });
                }

I'll work now on having the calendar limit these dates, and not just validate them.
PS.: @g00fy- Would you be kind enough to review and add my code to your source?

@SuricateCan can you make PR compiling this changes? thanks

@eralha PR #102

Is min/max limit working for you at all?
I'm trying to use it, but as soon as use, let say, min-date, no matter which date I pick (before or after min-date) the date-time input is marked is ng-invalid ng-invalid-min.
Here is the Plunker example with min-date set to the current date.
Am I doning something wrong?

@evdoks You can't use "now" in max-date attribute. You have to use a valid, ISO format, date string.

Yes its true, the readme is wrong! only date string not expressions.

@evdoks You could extend this functionality by testing the variable before the new Date stuff.


how can i close all opened swipe、only can open one swipe at the same time?

i wish can close an opened swipe,the way ?
if i swipe a row's button, when i swipe at another row, i hope close other swipe, how ?
@dancormier

+1 I'm looking to close the open swipeout item when scrolling the Scrollview

I'm also begineer in react native but I think it will work with EventEmitter module and Subscribable Mixin.
I'm using this on my ListView Component, when I click 'Edit' button on Navigation Bar, EventEmitter emit 'true' value and every ListView Items are listening this value with "addListenerOn" function within Subscribable Mixin. So when this value is true, all the ListView Items close swipeout with "this.refs['swipe']._close();". I think you can apply it to your case.

+1, now i am directly use a ref to call _close

+1. using _close for now.

Check out the example
https://github.com/dancormier/react-native-swipeout/blob/master/example/index.ios.js

@hufeng i am directly call _close too, but it can not work, can you show your code?

@xuanpyco Hi, can you provide me with your solution? Becuase I use ref, call _close and it does not work. Thanks in advance

@olejka91  I ended up a solution as following.
class ListItem extends Component {

_onSwipeOpen = () => {
this.props.onSwipeOpen(this.swipeInstance);
}

render() {
return (
<SwipeOut
ref={instance => this.swipeInstance = instance}
onOpen={this._onSwipeOpen}
/>
);
}

}

class MyList extends Component {

_onSwipeOpen = (swipeInstance) => {
if(this.currentlyOpenSwipe && this.currentlyOpenSwipe !== swipeInstance) {
this.currentlyOpenSwipe._close();
}

this.currentlyOpenSwipe = swipeInstance;
}

_renderRow  = () => {
 return (
<ListItem
 onSwipeOpen={this._onSwipeOpen}
/>
);
}

render(){
return (
<ListView
renderRow={this._renderRow}
 />
}
}

@xuanpyco I just replaced this library with a react-native-swipe-list-view. Thanks for reply

thanks @xuanpyco sir, your solution worked for me.


Angle Helper UI refresh

Before:

After:
Chrome

Safari

Firefox

Works great in Edge and IE as well


Links to admin centre

All the links to "Microsoft 365 admin center" currently 404 - looks like https:// needs adding to make them not relative to the page.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2eebb06d-174a-4f38-6e9e-c6176e97b3ba
Version Independent ID: b3feb7ce-6756-29af-f18b-2b9fd4e29f3e
Content: Purchase Power Apps for your organization - Power Platform Admin center
Content Source: power-platform/admin/signup-for-powerapps-admin.md
Service: power-platform
GitHub Login: @jimholtz
Microsoft Alias: jimholtz

@sionide0278
Thanks for bringing this to our attention. Links updated.


DRILL-7189: Revert DRILL-7105 Error while building the Drill native c…

…lient
This reverts commit a72f1700a8d260a2f92ba8955c77df6428feb919.

Closing this PR and will cherry-pick these changes to 1.16. branch since it's not intended for master


Path issue on windows using NVM to manage node, "ERROR: Cannot find npm binary"

Describe The Bug:
When using nvm on windows logs show "ERROR: Cannot find npm binary"
I think this is due to this function here https://github.com/oznu/homebridge-config-ui-x/blob/45a49188e2cadf52857314bb8620d594e1bb6cc7/src/modules/plugins/plugins.service.ts#L639
It checks a couple set paths on windows but does not check the $PATH environment variable. When using NVM the node/npm installation is on the path but not at any of the fixed locations checked by that function. I think this could be quickly resolved by also just checking if npm is on the path when using windows.
To Reproduce:
Install node via nvm windows, install homebridge-config-ui-x and service. Run it and see this error in the logs.
Expected behavior:
homebridge-config-ui-x should always check the path for binaries, even on windows.
Logs:
https://github.com/oznu/homebridge-config-ui-x/blob/45a49188e2cadf52857314bb8620d594e1bb6cc7/src/modules/plugins/plugins.service.ts#L639

Homebridge Config:
N/A

Screenshots:
N/A
Environment:

Node.js Version: 12.18.3
NPM Version: 6.14.6
Homebridge Version: 1.2.2
Homebridge Config UI X Version: 4.27.1
Operating System: Windows
Process Supervisor: none

I tried to build locally to make a fix for a PR but I couldn't get config-ui-x to build on windows with the instructions in CONTRIBUTING.MD
There are two ways to fix this that I see, one is
npm install --save command-exists
plugins.service.ts
import { sync as commandExistsSync } from 'command-exists';

....
 /**
   * Helper function to work out where npm is
   */
  private getNpmPath() {
    // Check if npm is on the path, regardless of system
    if(commandExistsSync('npm')){
      const npmPath = ['npm'];
      if (os.platform() === 'win32'){
        // Add global if on Windows
        npmPath.push('-g');
      }
      return npmPath;
    }

    if (os.platform() === 'win32') {

      // if npm not on path and running on windows find the full path to npm in some common locations
      const windowsNpmPath = [
        path.join(process.env.APPDATA, 'npm/npm.cmd'),
        path.join(process.env.ProgramFiles, 'nodejs/npm.cmd'),
      ]
        .filter(fs.existsSync);

      if (windowsNpmPath.length) {
        return [windowsNpmPath[0], '-g'];
      } else {
        this.logger.error(`ERROR: Cannot find npm binary. You will not be able to manage plugins or update homebridge.`);
        this.logger.error(`ERROR: You might be able to fix this problem by running: npm install -g npm`);
      }

    }
    // Fallback if commandExistsSync somehow didn't find it, Linux and macOS don't require the full path to npm
    return ['npm'];
  }

OR
add NVM_SYMLINK to the windows paths searched
plugins.service.ts
 // if npm not on path and running on windows find the full path to npm in some common locations
      const windowsNpmPath = [
        path.join(process.env.APPDATA, 'npm/npm.cmd'),
        path.join(process.env.ProgramFiles, 'nodejs/npm.cmd'),
        path.join(process.env.NVM_SYMLINK, 'npm.cmd'),
      ]
        .filter(fs.existsSync);

I worked around the issue for now with a symlink, from an administrator cmd prompt mklink /J "C:\Program Files\nodejs" C:\path\to\nvm\nodejs

Thanks, fixed with the NVM_SYMLINK solution.


micropython

I found that you are working on ztachip.I got issues with the micropython
I have done the ztachip build process , standalone image is working fine.
While going with micro-python ,I followed these steps
git clone https://github.com/micropython/micropython.git
cd micropython/ports
cp -avr /micropython/ztachip_port .
cd ztachip_port
export PATH=/opt/riscv/bin:$PATH
export ZTACHIP=<given the path , where zta-chip is built>.
make clean
make
But the error i got is same again after "make"

Even while opening minicom using " sudo minicom -w -D /dev/ttyUSB1" i could not open any prompt to paste the example codes.
Resolving this issue will be very grateful to me

@bittu-digital
There's something (ztachip installation folder) missing in the list of commands you provided
Please follow this as it is
git clone https://github.com/micropython/micropython.git
cd micropython/ports
cp -avr <ztachip installation folder>/micropython/ztachip_port .
cd ztachip_port
export PATH=/opt/riscv/bin:$PATH
export ZTACHIP=<ztachip installation folder>
make clean
make

Also, I think you're using you host machine's GCC to compile. Make sure this line points to the riscv compile that supports 32-bit RISC-V ARCH and ABI in your host machine. if you're unsure, you may check it with the following command.
$ riscv64-unknown-elf-gcc -print-multi-lib
.;
rv32e/ilp32e;@march=rv32e@mabi=ilp32e
rv32ea/ilp32e;@march=rv32ea@mabi=ilp32e
rv32em/ilp32e;@march=rv32em@mabi=ilp32e
rv32eac/ilp32e;@march=rv32eac@mabi=ilp32e
rv32emac/ilp32e;@march=rv32emac@mabi=ilp32e
rv32i/ilp32;@march=rv32i@mabi=ilp32
rv32ia/ilp32;@march=rv32ia@mabi=ilp32
rv32im/ilp32;@march=rv32im@mabi=ilp32
rv32if/ilp32f;@march=rv32if@mabi=ilp32f
rv32ifd/ilp32d;@march=rv32ifd@mabi=ilp32d
rv32iaf/ilp32f;@march=rv32iaf@mabi=ilp32f
rv32iafd/ilp32d;@march=rv32iafd@mabi=ilp32d
rv32imf/ilp32f;@march=rv32imf@mabi=ilp32f
rv32imfd/ilp32d;@march=rv32imfd@mabi=ilp32d
rv32iac/ilp32;@march=rv32iac@mabi=ilp32
rv32imac/ilp32;@march=rv32imac@mabi=ilp32
rv32imafc/ilp32f;@march=rv32imafc@mabi=ilp32f
rv32imafdc/ilp32d;@march=rv32imafdc@mabi=ilp32d
rv64i/lp64;@march=rv64i@mabi=lp64
rv64ia/lp64;@march=rv64ia@mabi=lp64
rv64im/lp64;@march=rv64im@mabi=lp64
rv64if/lp64f;@march=rv64if@mabi=lp64f
rv64ifd/lp64d;@march=rv64ifd@mabi=lp64d
rv64iaf/lp64f;@march=rv64iaf@mabi=lp64f
rv64iafd/lp64d;@march=rv64iafd@mabi=lp64d
rv64imf/lp64f;@march=rv64imf@mabi=lp64f
rv64iac/lp64;@march=rv64iac@mabi=lp64
rv64imac/lp64;@march=rv64imac@mabi=lp64
rv64imafc/lp64f;@march=rv64imafc@mabi=lp64f
rv64imafdc/lp64d;@march=rv64imafdc@mabi=lp64d

Lastly, please open issues in the main project so that anyone with similar interest/error can contribute in the conversation or find solutions. Feel free to @ mention me there, I will reply. This repo is experimental and will endup as a subfolder whenever it is ready.


p4lang/p4app:stable is not existed, change to latest

p4lang/p4app:stable is not existed, change to latest

I got error message from docker with p4lang/p4app:stable,
and I have checked p4lang/p4app:latest works well.

could you please merge this request?

Is there a reason this was not merged? As it is now, p4app update is broken because it can't find "p4lang/p4app:stable". This PR would fix this.

Hey Theo, I fixed this in a separate PR.


Exception in Settings/SettingsFieldset.

While removing ServiceLocatorAwareInterface in #217, a constructor dependency was added to SettingsFieldset.
But this class is meant to be extended without the need to create a factory to inject this dependency.
Because it is planned to refactor the settings module in the near future, we will move the dependecy injection to a setter method called in the build logic of the SettingInterface.

It was simpler to just create a factory for SettingsFieldset, which should be used by any module providing custom Settings fieldsets.


Update howto-mfa-nps-extension.md

The current documentation is outdated and does not reflect MGGraph PowerShell commands.

@Justinha
Can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


Copy output console

It would be great (for submitting errors) to be able to copy the omnisharp output and paste into a github issue

Yes!

:+1:

Resolved in #163


Create authentication for frontend

Using HTTPS is sure thing but we have to prevent unauthorized people from using the FE.

Here are couple of options.

htaccess the whole application and its endpoint. Good enough for a week or two
create username/password login with session cookie - registration is probably not yet needed.
integrate with Google SSO

We will run on Azure env and I've been told it's not that easy to set basic auth there, so the authentication should be solved by the app itself. So I guess the middle option is quite fine for now - login screen, hardcoded credentials (for MVP).

Pro MVP by to mohlo stacit staticky:
if header['Authorization'] == 'dGVzdHVzZXI6dmVyeW11Y2h0ZXN0aW5n':
    return true

(login testuser heslo verymuchtesting)

for 2.
There will be an endpoint, where I will post username and sha-256 of a password. The endpoint will authenticate it and set a cookie that will autheticate every request

for 2.
There will be an endpoint, where I will post username and sha-256 of a password. The endpoint will authenticate it and set a cookie that will authenticate every request

why sha/digest of the password is sent to the server ? doesn't it mean that it actually makes the digest to be a plain-text password effectively, hence the protocol would be susceptible to pass-the-hash attack ?

Let's go with basic auth in MVP, ok?
We'll implement it on the backend.

@Kobzol Please take this over.

Basic auth is there, so with HTTPS we can just create credentials and send them to (backend) users.


Handling colums names containing spaces

Below is an example of the SQL I'm trying to get via the API
psql -d  -c "select "cHostname","Server Model" from "Server Today" where "cHostname" ilike '%nim%';"
cHostname  |      Server Model
------------+-------------------------
argnim1    | IBM,9113-550 (P5-550)
argnim2    | IBM,9113-550 (P5-550)
daaa2nim71 | IBM,9131-52A (P5-52A)
daah3nim71 | IBM,8406-71Y (P7-PS701)
hbnim1     | IBM,9133-55A (P5-55A)
(5 rows)
Some examples of what I get when I've tried using curl, as per the chat room the table name syntax seems to work fine with %20, but not for columns.

Example of the table and cHostname column, which works fine.

curl 'http://:3000/Server%20Today?select=cHostname&cHostname=ilike.nim'; echo ""
[{"cHostname":"argnim1"},{"cHostname":"argnim2"},{"cHostname":"daaa2nim71"},{"cHostname":"daah3nim71"},{"cHostname":"hbnim1"}]

Example of adding the column with the space and using %20, as per the last part of the error seems to ignore the %20 and truncate the column to Server

curl 'http://:3000/Server%20Today?select=cHostname,Server%20Model&cHostname=ilike.nim'; echo ""
{"hint":"Perhaps you meant to reference the column "Server Today.MQ Server".","details":null,"code":"42703","message":"column Server Today.Server does not exist"}

Example when trying to quote it for SQL.

curl 'http://:3000/Server%20Today?select=cHostname,"Server Model"&cHostname=ilike.nim'; echo ""
Bad Request

Same problem with "." in column name
http://<IP_ADDRESS>:3000/vulns_details?pluginattributes.vuln_information.cpe=like.*microsoft*
pluginattributes.vuln_information.cpe is the column name, condition is never triggered..

This may not be suitable for others, but in my use case I've got a workaround for now.

Create a separate schema for Postgrest use. (obviously need to update the config file to use it)
Using CREATE TABLE newschema.newtable (LIKE public."old table" INCLUDING ALL); To newtable has the space removed to fix that bit.
Simple use INSERT to take a copy of the source table into the new schema with new table.
That run a query which looks at the system tables to create suitable ALTER TABLE commands to rename the columns in the new schema.

The query I'm using is as follows:-
SELECT 'ALTER TABLE ' || quote_ident(n.nspname) || '.' || quote_ident(c.relname) || ' RENAME column ' || quote_ident(a.attname) || ' TO ' || REGEXP_REPLACE(REGEXP_REPLACE(quote_ident(a.attname), ' ', '', 'g'), '.', '', 'g')
FROM pg_catalog.pg_attribute a
JOIN pg_catalog.pg_class c ON c.oid = a.attrelid
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE a.attnum > 1
AND reltype > 0
AND a.attname <> 'geom'
AND NOT a.attisdropped
AND n.nspname !~~ 'pg_%'  -- exclude catalog & temp tables, to be sure
AND n.nspname = 'api' -- specific new API schema name
AND REGEXP_REPLACE(REGEXP_REPLACE(quote_ident(a.attname), ' ', '', 'g'), '.', '', 'g') <> quote_ident(a.attname) -- Only include ALTER TABLE lines if value changes, can not rename columns to the same name.

✔ reproduced:
create table "fun times" as select generate_series(1,10) as "fun column";

# this one works
curl "localhost:3000/fun%20times"

# this one does not
curl "localhost:3000/fun%20times?select=fun%20column"

{
  "hint":null,
  "details":null,
  "code":"42703",
  "message":"column fun times.fun does not exist"
}


[FEA] Added bindings for head and tail

Using gather seemed like the logical choice to me.
Changes

Added series head & tail
Added dataframe head & tail
Added tests for both

Merging, since it seems like all PR feedback is addressed, let me know if anything else needs to be changed.


Fast launch MacOS PiP

Hey! Awesome project, i like it 😎
What do you think of using https://github.com/albinekb/open-pip-cli with this? What's the best way I could support streaming from watch?
$ watch "Rick and morty" | open-pip
$ watch "Rick and morty" | open VLC.app
The cli would just spit out the streaming URL to stdout, making it possible to use different apps to view the video.
maybe a flag to make it spit out the URL or use a different player? Just thinking.. 👍  any ideas?

It's already possible to stream to a variety of apps, you can see the full list here under the "Options (streaming):" section. My fork of webtorrent-cli also supports IINA, which can open PIP already.
I guess there's no easy way to integrate watch with open-pip at the moment.
I like the idea of a flag for outputting the url, but in that case I think we would need to spawn WebTorrent in a detached manner, basically you'll have to kill it manually afterwards.
Maybe a --pip flag would be more pratical, allowing for:
$ watch "Rick and morty" -- --pip

I'll see what I can do, thanks for the suggestion.

I've used IINA, it's cool, maybe I'll PR there to create an option to launch it straight into PiP mode, thats probably a way better solution since it has lots more features than my simple CLI.

That would be great.

@albinekb any news on this?

@fabiospampinato nope, been super busy. Not watched anything since last being here 😄
I think it would be cool if we had an option to just get a http-stream or something to the video file? Then you could play it anywhere 👍


Issue created from linear

Created this issue in linear :-)

Automatically created from Linear (#12)
via LinearSync

Comment from Linear (#12)
Hey can you check this out please!

via LinearSync

Comment from Linear (#12)
test from linear

via LinearSync

Checking this out! @samuelOsborne


unlicense is not good use mit license

unlicense is not good use mit license

Didn't notice the issue. Can you explain why it is not good and why MIT is better?

Not sure, ask a lawyer. Often times saying nothing leave things open to debate. BSD and MIT licenses state all your rights and leave less room for debate on any issue wluld a lawsuit occur. Google has lawyers and says it's forbidden: https://opensource.google/documentation/reference/patching
Some licenses are copyleft (less open) such as Apache 2, which limits how you can distribute. To my knowledge, BSD2 and MIT licenses have the benefit of just distribute your software with the license link available and you can do whatever you want.
I'm not a lawyer and can't give legal advice. But that's what I concluded as a layman. If you do MIT License, people can do whatever they want, and that's the best locense imnho. If you want to control the software then do apache 2 and you can copy the people who use your stuff, but I'd rarely do business with any apache 2. Other licenses probably are too bad to even check

Got it. Updated to MIT license.


Corrected a typo in bio

Changes proposed

Check List (Check all the applicable boxes)

[X] My code follows the code style of this project.
[X] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[X] All new and existing tests passed.
[ ] This PR does not contain plagiarized content.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

My profile is yet to be displayed


DPUService stuck in deletion

A DPUService may be stuck in its Deleting phase when a pod created as part of the DPUService can not be deleted.
When in this case the DPUService will hang in deletion forever. The ArgoCD Application will also be stuck in deletion and may not have sufficient information to debug.
More information is available on the DPUCluster where the look at the pods on the DPUCluster. To do so you first need to retrieve the DPUCluster kubeconfig and use it to query the DPUCluster. You can do so using:
kubectl get secrets -n $DPUCLUSTER_NAMESPACE $DPUCLUSTER_ADMIN_KUBECONFIG_SECRET_NAME -o jsonpath='{.data.admin\.conf}' | base64 -d > $HOME/tenant1-kubeconf

export KUBECONFIG=$HOME/tenant1-kubeconf

kubectl get pods -A

You should be able to see the pods linked to your DPUService as part of the DPUCluster. When looking at the description of the pod it may be stuck in ContainerCreating. This means there is some configuration issue - likely related to the DPUServiceIPAM, DPUServiceChain or DPUServiceInterface linked to the DPUService.
Workaround:
To force remove the pod run kubectl delete pod --grace-period=0 --force against the pod which is stuck in ContainerCreating. This should unblock the deletion of the ArgoCD Application and the DPUService.

Ref#4213229


signal-desktop-beta: only build on x86_64-linux

Description of changes
Build for aarch64-linux is currently failing since forever.
Upstream does not provide packages for aarch64-linux:

Note that these instructions only work for 64 bit Debian-based Linux
distributions

Source: https://support.signal.org/hc/en-us/articles/360007318471-Signal-Beta

Formatted using nixfmt-rfc-style

ZHF: #309482

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Relevant: #286188
Currently that PR is also modifying the pkgs/applications/networking/instant-messengers/signal-desktop/default.nix

I think we should wait for #286188 to be merged before doing anything here as we're still forming a consensus on the pkgs/applications/networking/instant-messengers/signal-desktop/default.nix

@ofborg eval

@teutat3s you should be using nixfmt-rfc-style for formatting

@kirillrdy Done.

In my opinion, this change does not interfere with the other mentioned PR and can be reviewed and merged independently.

How so? Both PRs modify the same file. Since #286188 was created first and there is already a discussion about how to deal with the file this PR is trying to modify, I think it will be better to wait and see what consensus will be reached, so later this PR can be rebased and follow that consensus

That PR seems to be stalled and needs to be rebased anyway...


Request: change the default sha256 hash file seperator for blobs to - (minus) instead of :

Heya,
I have a setup where I'd like to use two systems, one testing on Windows and one active on Linux use the same path for model and manifest to save storage.
The issue I have is: Linux deployment creates blob files with the name sha256:SHATHATHASBEENGENERATED and on Windows  sha256-SHATHATHASBEENGENERATED this makes their filenames incompatible with each other and troublesome on Windows since any file with a : inside it is considered "non-existent"
Desired resolution:
A Patch where it'll be backwards-compatible with files that have : but the new default to be - so it will even work when the storage is on a NTFS or exFAT file system.
Thank you!

Let's track this separator cleanup via #2032


The docs for update-strategies were missing one word

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Choose segment features

This PR implements a popup multi-selector to choose which segment features to calculate (instead of calculating all by default). Currently the user can choose mean, median, or both.
This triggers when "calculate features for project" is selected from the menu bar. Currently the picker does not run when "calculate features" is selected during the project import (can be added).

@rj3d Latest commit should fix the active vs. all image feature calculation


gRPC responses for equivalent HTTP 4xx or 5xx responses should be returned as an error instead of response object in OK status

Version information
v0.2.1
Describe the bug
Different response types for HTTP API maps to different protobuf messages with error code in the name of the message.
All those responses are returned only during gRPC OK.
What this implies is, even after doing a check for err != nil post gRPC API call, users have to further check for errors inside response messages. This also complicates the retrieval of actual usable response.
Solution is to return all error responses as err from gRPC API call. More details TBD.
To Reproduce
	request := otg.SetConfigRequest{Config: config.Msg()}
	ctx, cancelFunc := context.WithTimeout(context.Background(), api.grpc.requestTimeout)
	defer cancelFunc()
	resp, err := api.grpcClient.SetConfig(ctx, &request)
	if err != nil {
		return nil, err
	}
	if resp.GetStatusCode_200() != nil {
		return NewResponseWarning().SetMsg(resp.GetStatusCode_200()), nil
	}
	if resp.GetStatusCode_400() != nil {
		data, _ := yaml.Marshal(resp.GetStatusCode_400())
		return nil, fmt.Errorf(string(data))
	}
	if resp.GetStatusCode_500() != nil {
		data, _ := yaml.Marshal(resp.GetStatusCode_500())
		return nil, fmt.Errorf(string(data))
	}

Expected behavior
	request := otg.SetConfigRequest{Config: config.Msg()}
	ctx, cancelFunc := context.WithTimeout(context.Background(), api.grpc.requestTimeout)
	defer cancelFunc()
	resp, err := api.grpcClient.SetConfig(ctx, &request)

        // should include error responses defined for http 4xx or 5xx codes
	if err != nil {
		return nil, err
	}

Originally reported by @robshakir @marcushines

For reference: https://cloud.google.com/apis/design/errors

Notes from high level discussion about the possible solution:

Constraining API response schema

Today, open-api allows users to specify different response schemas for different HTTP status codes.
We might look into possibility of constraining responses to two kinds, Ok response and Error response
Ok response may have different schema based on URL + method and must be provided by author. Only 200 status code shall be supported - TBD ?
Error response MUST be same for all API calls (URL + method)
Error response MUST be provided by author and contain certain minimum attributes (e.g. StatusCode, Messages, etc.)
These constraints shall be validated by openapiart

Changes in generated SDK for gRPC

The generated struct for Error response will automatically be provided with Error() method in order to implement error interface in Go
The server will essentially work with Error struct to fill in all the possible details. Upon returning the error, Error() method would serialize the message into readable and parsable string.
Upon reception of error on client side, it can be easily passed along. If further inspection is needed, it can be de-serialized into Error struct
The status code used would be FAILED_PRECONDITION for validation errors and INTERNAL for application errors.

From user's POV, following would be the workflow for API calls:res, err := api.SetConfig(config)
if err != nil {
    // simply pass it along to parent func
    return err
    // or inspect the error further
    errObj := api.NewError(err)
    fmt.Println(errObj.StatusCode())
    fmt.Println(errObj.Messages())
}

// use response - it won't be nested anymore
fmt.Println(res.Warnings())

@ajbalogh @alakendu FYI

Proposal


project: standardise builders

Standardise the builders across the crate. The builders crate is handled separately in PR #361.
The command parser is now a normal, referencing builder. Methods take &mut self and return &mut Self. This is because it can be mutated at runtime, and the alternative would make other applications difficult.
The gateway's cluster and shard builders are both consuming as they were before. The only difference is that the ShardConfigBuilder::intents method has been corrected from being referencing to being consuming like the rest of the methods.
The HTTP client builder is now consuming to be consistent both with the request builders and the gateway's configuration builders.
In total, this makes every builder except for the command parser consuming.
Part of #239.

Whether to make this consuming or referencing is up to debate but I wanted to get a visual representation of what this would look like to get the issue going, I'm personally in favour of generally using consuming builders (so as implemented in this PR)

I am generally also a fan of consuming builders


Binary for RHEL based systems

Please, could you release a binary for RHEL, Fedora, Rocky, Alma and CentOS based systems?

Hi - Apologies for the delay in response on this one. At the moment, Procmon for Linux does not support Fedora based systems. We are working to change that. Please stay tuned.

I see the same package you ask to install for procmol ubuntu here : https://packages.microsoft.com/config/centos/7/, does Procmon work with Centos now ?

Hi - RHEL package is now available with the latest changes. Please give it a try and let us know!


THRIFT-3821

Going over 2 GB is quite possible when using Thrift for serialization. We can detect the condition and throw an exception.

Please rebase (and squash to a single commit, if you can) and push here, so the build system can build the change and we can verify it before merging it.  Thanks.

Thanks for the review, I'll address both. It's my preference to squash the commits as well, sounds like I've misinterpreted https://thrift.apache.org/docs/HowToContribute (it says "When bugfixing: add test that will isolate bug before applying change that fixes it").

I've disabled the test for Windows since allocating 1GB on 32-bit is likely to fail with bad_alloc.
Test failure looks unrelated, also got a clean run before: https://travis-ci.org/apache/thrift/jobs/263298365.

Are there any additional changes I should make to this pull request? As I've said in the inline comment, I believe comparing to std::numeric_limits<uint32_t>::max() is actually consistent with what the error message says.

I'd prefer if we refactored it to make the limit configurable with a default matching what it is now; that way tests won't need multiple gigabytes of memory to function properly.  I'll approve and merge this for now as it is an improvement.


Label studio crashing again and again. Runtime error

Describe the bug
I deploy the label studio using the Run on Google Cloud button provided in the readme section of this repository. It successfully installs in the google cloud run. When I try to create a project or add data to a project it gives me the error that I attach as a screenshot. And this error gave me a random way. Sometimes It also automatically logout. Sometimes it's deleting the created project that I created.
By clicking the Run on Google Cloud it installs label-studio version 1.6.0rc5
Screenshots

Environment (please complete the following information):

Google Cloud Run
Label Studio Version 1.6.0rc5

Is there anything that I'm missing
Thanks

Running into the same issue running heartexlabs/label-studio:1.6.0 on k8s with Postgres running in AWS.

I was unable to reproduce this issue.
Without logs it's really hard to troubleshoot something.

@farioas, In  my case, click on Run on Google Cloud reproduce this issue

After clicking on the link I have working instance


[Bug]: App crashes on clicking Save (EditActivity)

Summary
App crashes on rotating image and clicking on save on few images.
Steps to reproduce

Select a image from gallery
Click on Edit Image
Rotate Image
Click on save Image

Expected behaviour
App should not crash and rotated image should be shown
Actual behaviour
App crashes
Device name
Samsung S21
Android version
Android 14
Commons app version
4.2.1-debug-issues5439
Device logs
java.lang.NullPointerException
at fr.free.nrw.commons.edit.EditActivity.copyExifData(EditActivity.kt:260)
at fr.free.nrw.commons.edit.EditActivity.getRotatedImage(EditActivity.kt:241)
at fr.free.nrw.commons.edit.EditActivity.init$lambda$3(EditActivity.kt:125)
at fr.free.nrw.commons.edit.EditActivity.$r8$lambda$fIjJEeuPIZcPAnhPtUzdHmWgXF0(Unknown Source:0)
at fr.free.nrw.commons.edit.EditActivity$$ExternalSyntheticLambda2.onClick(Unknown Source:2)
at android.view.View.performClick(View.java:8043)
at android.widget.TextView.performClick(TextView.java:17816)
at android.view.View.performClickInternal(View.java:8020)
at android.view.View.-$$Nest$mperformClickInternal(Unknown Source:0)
at android.view.View$PerformClick.run(View.java:31850)
at android.os.Handler.handleCallback(Handler.java:958)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:230)
at android.os.Looper.loop(Looper.java:319)
at android.app.ActivityThread.main(ActivityThread.java:8893)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:608)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Screen-shots
https://github.com/commons-app/apps-android-commons/assets/114223204/f48ad2c7-82ad-4220-85f0-42713f432c3b
Would you like to work on the issue?
Yes

Hey @neeldoshii, can you please reproduce this issue first (Since it happens only for specific images). Maybe for those without exif data.

Reproduced. For me the bug happened to me at first image.
Not sure what its causing, I am investigating it. This is giving crash to me on all.

I feel this issue is caused by images with no exif data (as the logs suggest)

Thanks a lot for this, it helped me exactly for a pinned point for the cause of issue and fix. Below is the code patch that solves the issue.
    fun getRotatedImage() {

        val filePath = imageUri.toUri().path
        val file = filePath?.let { File(it) }

        val rotatedImage = file?.let { vm.rotateImage(imageRotation, it) }
        if (rotatedImage == null) {
            Toast.makeText(this, "Failed to rotate to image", Toast.LENGTH_LONG).show()
        }
-        val editedImageExif = rotatedImage?.path?.let { ExifInterface(it) }
-        copyExifData(editedImageExif)
+        val editedImageExif: ExifInterface?
+        if (rotatedImage?.path != null) {
+            editedImageExif = ExifInterface(rotatedImage.path)
+            copyExifData(editedImageExif)
+        }
        val resultIntent = Intent()
        resultIntent.putExtra("editedImageFilePath", rotatedImage?.toUri()?.path ?: "Error");
        setResult(RESULT_OK, resultIntent);
        finish();
    }

If you would like, I can submit a pull request with this fix, or if you prefers to work on it, use this patch and this will fix the app crash.
CC : @nicolas-raoul when we have no exif data of image do we want to show snackbar/toast to user that this image doesn't contain exif information?

Assigning to Kanahia due to "Would you like to work on the issue? Yes"

Sure @nicolas-raoul, I have fixed the issue can I provide the next updated patch for @kanahia1 for reference?

Ah yes of course, and I am very sorry for the delay in assigning this!

Did a quite more digging this PR #5569 solves this issue once it gets merged.

Hey @neeldoshii, Thank you for your work. If possible, can you please share the updated patch?

Hey @neeldoshii, Thank you for your work. If possible, can you please share the updated patch?

As said, this issue is already solved in PR #5569

Strangely I am not able to reproduce this crash.
Is it happening always for you two? Or in special conditions?
Thanks! :-)

Strangely I am not able to reproduce this crash.
Is it happening always for you two? Or in special conditions?
Thanks! :-)

Here are the steps

Download any image from google, check its exif data of the image. Make sure the image you taking has no exif data.
Now use that image.

Indeed I am now able to reproduce, thanks!


Update ISL 0.24 and avr-libc

See also #33 (for isl in GCC-bootstrap). See and also [avrdudes/avr-libc/issues/889](https://github.com/avrdudes/avr-libc/issues/889} for renaming reserved file in Windows aux.c renamed to xau.c.


syntax error, unexpected '?' in SecureHeaders.php

Hello, I Had installed this package and configured it well on my local machine in laravel 5.4 and it ran without any problems but when i uploaded to the server, I encountered this error .
FatalErrorException in SecureHeaders.php line 121: syntax error, unexpected '?'
It was complaining about this line of code in vendor/bepsvpt/secure-headers/src/SecureHeader.php
$csp = new CSPBuilder($this->config['csp']);
if (! ($this->config['csp']['https-transform-on-https-connections'] ?? true)) {
/* @todo wait for upstream package releasing new version */
// $csp = $csp->disableHttpsTransformOnHttpsConnections();
}
Can anyone help figure out how to fix this, please?

Hi @jeanjoe,
Please check the php version on your server, the minimum requirement version is 7.0.

Thank you very much, I was using 5.6 and i just changed to 7.0 and it worked perfectly. Thanks for the great package and great team for your support


Issue with k8s.io/docs/tutorials/kubernetes-basics/expose/expose-interactive/

Interactive tutorial Katacoda doesn't work since 4 of March 2021

https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-interactive/

@BenHall There have been a few similar reports
The page works fine for me.

@sftim It looks like our security script had a false positive on the last couple of days. I've cleared the list and updated a script so everyone should be able to access the platform again.
@nkulinichev  Sorry for the problem. It should now be fixed but if you are still having issues please let me know.

/label team/katacoda


Switch to using pre-commit for managing our linting setup

Switch to using https://pre-commit.com/ to manage our linting setup

Some implementation questions (a reminder for me to ask in the sprints tomorrow):

I see black, flake8, and isort in the pyproject.toml, which have hooks built-in, along with mypy, which requires the use of a mirror. Which libraries are you looking to include as hooks?
In order for contributors to use the pre-commit hooks, there is some setup required. Is there a specific spot to place this information in CONTRIBUTING.md or is under the Pull requests section fine?
This issue is tagged with "github_actions" – should an action be added to use the pre-commit hooks to lint a PR?
There are some additional pre-commit hooks that I like to use to check for valid YAML, TOML, rST formatting, and more. Are you interested in adding these or is there a specific list?
While testing the current implementation of the pre-commit hooks, I noticed that there were a few files with issues (trailing spaces, black reformatting, isort import order). Should I fix this in this PR or a separate one?


[Bug] Crash when authenticating player

Please fill in the following fields:

Unity editor version: 2021.3.19f1
Firebase Unity SDK version: 11.0.0
Source you installed the SDK: .unitypackage
Problematic Firebase Component: Auth
Other Firebase Components in use: Auth, Database, Analytics, Crashlytics
Additional SDKs you are using: Facebook, AdMob, IronSource (LevelPlay), Unity ads, AppLovin, Vungle
Platform you are using the Unity editor on: Windows
Platform you are targeting: Android
Scripting Runtime: IL2CPP
Pre-built SDK from the website or open-source from this repo: website

Please describe the issue here:
We have noticed a new crash in our app, caused by the Firebase Authentication library. We don't know exactly which line of code is causing this error.

No, the issue appears sometimes, it's not a regular error.
What's the issue repro rate?
70% anytime, during playing the game, it crashes
What happened? How can we make the problem occur?
Don't know really which line of code is causing the error
If you have a downloadable sample project that reproduces the bug you're reporting, you will
likely receive a faster response on your issue.
Relevant Code:
PlayGamesPlatform.Instance.Authenticate(success =>
            {
                if (success == SignInStatus.Success)
                {
                    PlayGamesPlatform.Instance.RequestServerSideAccess(true, code =>
                    {
                        authCode = code;
                        Firebase.Auth.FirebaseAuth auth = Firebase.Auth.FirebaseAuth.DefaultInstance;
                        Firebase.Auth.Credential credential = Firebase.Auth.PlayGamesAuthProvider.GetCredential(authCode);

                        auth.SignInAndRetrieveDataWithCredentialAsync(credential).ContinueWithOnMainThread(task =>
                        {
                            if (task.IsCanceled)
                            {
                                Debug.Log("SignInAndRetrieveDataWithCredentialAsync was canceled.");
                                return;
                            }
                            if (task.IsFaulted)
                            {
                                Debug.Log("SignInAndRetrieveDataWithCredentialAsync encountered an error: " + task.Exception);
                                return;
                            }

                            Firebase.Auth.AuthResult result = task.Result;
                            id = "0";
                            uid = result.User.UserId;
                            player = result.User.DisplayName;

                            Debug.LogFormat("User signed in successfully: {0} ({1})", result.User.DisplayName, result.User.UserId);
                        });
                    });
                }
                else
                {
                    // Not Successful
                    Debug.Log("Not successfully authenticated");
                }
            });

Hi @bayonmbayo,
Version 11.1.0 of the SDK has been released and it should contain the fix for this behavior.
That said, I'll be closing this for now. Let me know in case the issue persists or a new issue arises with this SDK version so that this ticket could be reopened.


Update @shopify dependencies to latest

Shopify CLI prior to v 3.59.1 has a bug that causes the wrong theme to be displayed when using the dev command: https://github.com/Shopify/cli/issues/3766
I'm not totally sure if these changes are complete, as I attempted to run the dev and build commands locally, but encountered a number of Typescript errors. I thought it was worth putting this up as a start either way, and would be interested in understanding how to get things running locally.
The newer version of the CLI also does not appear to have an oclif plugins array, which seems to be causing an issue with the patch-cli script:

I added a fix, but not sure if more changes are required.

Thanks for raising this issue and opening the PR.
I saw the v3.59.x releases, but haven't had a moment to make the updates yet.
I need to investigate what the next steps are and how we should integrate the plugin. We may need to inject the plugins entry in addition to adding our package to the list. Or, we might need a different tact altogether. It's possible the changes are meant to prevent what we've been doing.

@jeffreyguenther Sounds good, I'm happy to assist in any way if it would be helpful.
Somewhat related: there is a React version mismatch with cli-kit that I've been running into. Here's the ticket in case you encounter it also: https://github.com/Shopify/cli/issues/3800#issuecomment-2084289153

Some additional related info here: https://github.com/Shopify/cli/issues/3843

@jeffreyguenther Going to close this as it looks like v3.60.0 solves the shopify theme dev issue, but there's a number of other changes that seem to be required based on the updates to the CLI.

I will try to get this update done this week.

I will try to get this update done this week.

Hey @jeffreyguenther Any update here? Recognizing this is open source, but also keen to use a single Shopify CLI dependency in our projects. Thanks!

Thanks for the quick reply! I'll be at Editions too 👍

I look like my avatar. Come say hi!


Convert more icons to symbolic icons.

Converts most of the window chrome icons to symbolic icons.
Additionally contains fixes to the history panel to make it render correctly in dark themes.

Looks great, thanks!


How to read process.env in app?

I only got these variable, when I console log process.env
NODE_ENV: "development"
PUBLIC_URL: ""

how to access process environment variable (defined via .env) in app?

Core Create React App outlines that here https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/template/README.md#adding-custom-environment-variables

apparently, we can add env variable by adding it in .env file with REACT_APP_ prefix. e.g
REACT_APP_MY_VARIABLE=value

thanks.


feat: gmacario solutions to days 04, 05, 06, 07, 08

📑 Description

The Title says it all.
Solution for Day 06, Part 2 is still WIP at the moment.

✅ Checks

[x] My pull request adheres to the code style of this project
[x] My code requires changes to the documentation
[x] I have updated the documentation as required
[x] All the tests have passed

ℹ Additional Information

May I ask some kind fellow team member to please review and merge this PR?

Thanks for your review @davmacario.
Merging PR
Re:

This PR is one of the reasons I hate Jupyter Notebooks

Could you please elaborate?

Could you please elaborate?

Git diff: +4880 -0
Lines of useful code ~300

Could you please elaborate?

Git diff: +4880 -0
Lines of useful code ~300

Have you ever tried this
jupyter nbconvert --to script 'my-notebook.ipynb'


Fundamentals of Algorithms with Applications

Hacktoberfest notes:

due to volume of submissions, we may not be able to review PRs that do not pass tests and do not have informative titles.
please read our contributing guidelines
be sure to check the output of Travis-CI for linter errors
if this is your first open source contribution, make sure it's not your last!

What does this PR do?
Add Resource(s) | Remove Resource(s) | Add info | Improve Repo
For resources
Description
Why is this valuable (or not)?
How do we know it's really free?
For book lists, is it a book? For course lists, is it a course? etc.
Checklist:

[ ] Not a duplicate
[ ] Included author(s) if appropriate
[ ] Lists are in alphabetical order
[ ] Needed indications added (PDF, access notes, under construction)

You may have forgotten to save your file before stagings and committing your changes to the repository.
The commit doesn't add any new lines.


WC-1070 Add Swagger support for endpoint documentation.

One question: do we really need 'username' in MissingConfirmRegistrationException?
Yes, this field is used to resend activation link
Why not 'userName'?
username this is one word


fix WhichKeySeparator hl group

That is intentional. There used to be a typo in the HL group. This part of the code checks if a HL group with the typo is defined and uses that.


Local View hides after Remote view is rendered

After call connection, local view hides when remote view renders. this happens only with Android version greater than 7.0

@HaseemUlHaq  could you pls help check if the issue persists in https://github.com/AgoraIO-Community/Agora-RN-Quickstart


Invalid relative path concatenation in get_chunk_url

Context:
I am using django-webpack-loader in a multi-pages Django app that runs VueJS components in Django templates. I set the publicPath option in webpack to an empty string so that I can control the static url from the Django settings and properly use a CDN server, as described in #160
Issue:
JS and CSS assets are saved to css and js subfolders. Here is what webpack-stats.json looks like:
    "status": "done",
    "chunks": {
        "compA": [{
            "name": "css/compA.css",
            "path": "D:\\home\\site\\wwwroot\\common-static\\assets\\frontend\\css\\compA.css"
        }, {
            "name": "css/compA.3564e3ce75ddc57936fd.css",
            "path": "D:\\home\\site\\wwwroot\\common-static\\assets\\frontend\\css\\compA.3564e3ce75ddc57936fd.css"
        },
             ...
        }],
             ...
}

Note that the chunk name uses a UNIX-like path "css/compA.css".
In settings.py I have the following configuration:
WEBPACK_LOADER = {
    'DEFAULT': {
        # when true, webpack will read stats file only once and cache results
        'CACHE': not DEBUG,
        # dir in which webpack outputs the bundles - it should not be full path
        'BUNDLE_DIR_NAME': os.path.join("assets", "frontend", ""),  # add a trailing slash
        # location of the bundle tracker file
        'STATS_FILE': STATS_FILE,
        'POLL_INTERVAL': 0.3,
        'TIMEOUT': None,
        'IGNORE': [r'.+\.hot-update.js', r'.+\.map']
    }
}

The BUNDLE_DIR_NAME value ends up being 'assets\frontend\' on Windows.
With these settings, django-webpack-loader prints an invalid path in the Django template by concatenating BUNDLE_DIR_NAME with the chunk name: \static\assets\frontend\css/compA.css
The issue is in the get_chunk_url of webpack_loader/loader.py:
def get_chunk_url(self, chunk):
        public_path = chunk.get('publicPath')
        if public_path:
            return public_path

        relpath = '{0}{1}'.format(
            self.config['BUNDLE_DIR_NAME'], chunk['name']
        )
        return staticfiles_storage.url(relpath)

Concatenation does not take into account that the two relative paths may be formatted differently. In addition it requires the BUNDLE_DIR_NAME settings to end with a slash.
Solution
This issue is easily fixed by replacing the path concatenation with:
def get_chunk_url(self, chunk):
        public_path = chunk.get('publicPath')
        if public_path:
            return public_path

        relpath = os.path.normpath(
            os.path.join(self.config['BUNDLE_DIR_NAME'], chunk['name']
        )
        return staticfiles_storage.url(relpath)

This makes the path concatenation OS-independent, and allow greater flexibility to define output paths in the webpack config.

Edit:
As get_chunk_url is supposed to return a url, it actually makes sense to return a POSIX-like path with forward slash, even on Windows. I think my confusion came from the fact that the webpack loader settings is called "BUNDLE_DIR_NAME", which implies that this is a directory path. In fact, BUNDLE_URL would be more appropriate !

Makes sense. Perhaps we should improve the naming in 1.0.0. Adding to milestone.


Confirmations

Now, trona execute command immediately after user type it into terminal. We need confirmation for any command.

https://github.com/solid-soda/evolutions/pull/57


Bots with mouse are working...

The bots with mouse do work check this video out uploaded today...  On raga.pw   I think they're using this, come on guys implement it:
https://camo.githubusercontent.com/6eb2a5c0272942aa574c8ed017e459953290e8c8/687474703a2f2f692e696d6775722e636f6d2f345139527862452e706e67

https://www.youtube.com/watch?v=zLWKvI3CIdQ

Yeah I saw that too u might want to look into the code or it might just be inside of there own server


Use inside Next.js app router?

Is there any way to use this within Next.js 13's app router? I've been looking at multiple issues here in GitHub, but they all use the old pages method and I'm not sure how to best update them for the new app router.
Thanks in advance!

I'm not an extremely skilled developer, but it seems to me that the Express version would be the likely candidate to adapt for Next.js's new route handler in the app router.

Not so simple with app route.
The only "good" option that I've know of, is to use custom express server, integrate the board with it, and use next.js app for handling rest of the routes. (This is what I'm doing in my app)


[Feature Request] Multiple mongo clients. Oplog read user and db read user.

With Mlabs, you can create an Oplog user but that user is very limited. I recently forked the repo about 2 months ago and added a feature to use multiple mongo clients.
Most of the required changes you will see here https://github.com/ballerabdude/mongo-connector/commit/4f8b837fc5f169981db6575142699df13906afc9
Here is my original issue https://github.com/mongodb-labs/mongo-connector/issues/626
Here is a pull request that fixed the original issue https://github.com/mongodb-labs/mongo-connector/pull/654
This new feature will have an oplog user client and another client for db reads if provided.

I don't have an Mlabs account at the moment. Can you connect to your Mlabs instance, log in as the Oplog user, and run db.runCommand({"connectionStatus": 1, "showPrivileges": 1});? Please post the output of that command here, for example:
> db.runCommand({"connectionStatus": 1, "showPrivileges": 1});
{
	"authInfo" : {
		"authenticatedUsers" : [
			{
				"user" : "user",
				"db" : "admin"
			}
		],
		"authenticatedUserRoles" : [
			{
				"role" : "readWriteAnyDatabase",
				"db" : "admin"
			},
			{
				"role" : "dbAdminAnyDatabase",
				"db" : "admin"
			},
			{
				"role" : "clusterAdmin",
				"db" : "admin"
			},
			{
				"role" : "userAdminAnyDatabase",
				"db" : "admin"
			}
		],
		"authenticatedUserPrivileges" : [
			{
				"resource" : {
					"cluster" : true
				},
				"actions" : [
					"addShard",
					"appendOplogNote",
					"applicationMessage",
					"authSchemaUpgrade",
					"cleanupOrphaned",
					"connPoolStats",
					"connPoolSync",
					"cpuProfiler",
					"diagLogging",
					"flushRouterConfig",
					"fsync",
					"getCmdLineOpts",
					"getLog",
					"getParameter",
					"getShardMap",
					"hostInfo",
					"inprog",
					"invalidateUserCache",
					"killop",
					"listDatabases",
					"listShards",
					"logRotate",
					"netstat",
					"removeShard",
					"replSetConfigure",
					"replSetGetConfig",
					"replSetGetStatus",
					"replSetStateChange",
					"resync",
					"serverStatus",
					"setParameter",
					"shardingState",
					"shutdown",
					"top",
					"touch",
					"unlock"
				]
			},
			{
				"resource" : {
					"db" : "admin",
					"collection" : "system.users"
				},
				"actions" : [
					"collStats",
					"createIndex",
					"dbHash",
					"dbStats",
					"dropIndex",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "config",
					"collection" : ""
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"enableSharding",
					"find",
					"getShardVersion",
					"indexStats",
					"insert",
					"killCursors",
					"listCollections",
					"listIndexes",
					"moveChunk",
					"planCacheRead",
					"remove",
					"splitChunk",
					"splitVector",
					"update"
				]
			},
			{
				"resource" : {
					"db" : "",
					"collection" : "system.namespaces"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "admin",
					"collection" : "system.roles"
				},
				"actions" : [
					"collStats",
					"createIndex",
					"dbHash",
					"dbStats",
					"dropIndex",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "local",
					"collection" : "system.namespaces"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "",
					"collection" : ""
				},
				"actions" : [
					"bypassDocumentValidation",
					"changeCustomData",
					"changePassword",
					"collMod",
					"collStats",
					"compact",
					"convertToCapped",
					"createCollection",
					"createIndex",
					"createRole",
					"createUser",
					"dbHash",
					"dbStats",
					"dropCollection",
					"dropDatabase",
					"dropIndex",
					"dropRole",
					"dropUser",
					"emptycapped",
					"enableProfiler",
					"enableSharding",
					"find",
					"getShardVersion",
					"grantRole",
					"indexStats",
					"insert",
					"killCursors",
					"listCollections",
					"listIndexes",
					"moveChunk",
					"planCacheIndexFilter",
					"planCacheRead",
					"planCacheWrite",
					"reIndex",
					"remove",
					"renameCollectionSameDB",
					"repairDatabase",
					"revokeRole",
					"splitChunk",
					"splitVector",
					"storageDetails",
					"update",
					"validate",
					"viewRole",
					"viewUser"
				]
			},
			{
				"resource" : {
					"db" : "admin",
					"collection" : "system.version"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "admin",
					"collection" : "system.new_users"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "",
					"collection" : "system.js"
				},
				"actions" : [
					"collStats",
					"convertToCapped",
					"createCollection",
					"createIndex",
					"dbHash",
					"dbStats",
					"dropCollection",
					"dropIndex",
					"emptycapped",
					"find",
					"insert",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead",
					"remove",
					"renameCollectionSameDB",
					"update"
				]
			},
			{
				"resource" : {
					"db" : "",
					"collection" : "system.users"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "local",
					"collection" : ""
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"enableSharding",
					"find",
					"getShardVersion",
					"indexStats",
					"insert",
					"killCursors",
					"listCollections",
					"listIndexes",
					"moveChunk",
					"planCacheRead",
					"remove",
					"splitChunk",
					"splitVector",
					"update"
				]
			},
			{
				"resource" : {
					"db" : "config",
					"collection" : "system.namespaces"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "local",
					"collection" : "system.indexes"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "config",
					"collection" : "system.js"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "admin",
					"collection" : "system.backup_users"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "local",
					"collection" : "system.js"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "local",
					"collection" : "system.replset"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "config",
					"collection" : "system.indexes"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "$setFeatureCompatibilityVersion",
					"collection" : "version"
				},
				"actions" : [
					"insert",
					"remove",
					"update"
				]
			},
			{
				"resource" : {
					"db" : "",
					"collection" : "system.profile"
				},
				"actions" : [
					"collStats",
					"convertToCapped",
					"createCollection",
					"dbHash",
					"dbStats",
					"dropCollection",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			},
			{
				"resource" : {
					"db" : "",
					"collection" : "system.indexes"
				},
				"actions" : [
					"collStats",
					"dbHash",
					"dbStats",
					"find",
					"killCursors",
					"listCollections",
					"listIndexes",
					"planCacheRead"
				]
			}
		]
	},
	"ok" : 1
}

Can you confirm that you cannot create a user manually (see https://github.com/mongodb-labs/mongo-connector/wiki/Usage-with-Authentication#required-permissions) on your Mlabs cluster? The mlabs documentation is not very clear on this.

Here is goes mlabs lets you create an oplog user and a db user separately. The instance is shard which would explain why I can't do much with the oplog user they provide me
MongoDB shell version: 3.2.9
connecting to: ds135828-a0.mlab.com:35828/admin
rs-ds135828:PRIMARY> db.runCommand({"connectionStatus": 1, "showPrivileges": 1});
{
        "authInfo" : {
                "authenticatedUsers" : [
                        {
                                "user" : "oplog-reader",
                                "db" : "admin"
                        }
                ],
                "authenticatedUserRoles" : [
                        {
                                "role" : "readOplog",
                                "db" : "admin"
                        }
                ],
                "authenticatedUserPrivileges" : [
                        {
                                "resource" : {
                                        "db" : "local",
                                        "collection" : "oplog.rs"
                                },
                                "actions" : [
                                        "find"
                                ]
                        },
                        {
                                "resource" : {
                                        "db" : "local",
                                        "collection" : "system.replset"
                                },
                                "actions" : [
                                        "find"
                                ]
                        }
                ]
        },
        "ok" : 1
}
rs-ds135828:PRIMARY>

So mlabs responded back to my email and they have granted my oplog user read access. Thanks so much @ShaneHarvey for you assistance! We can close this. since emailing mlab fixes the issue.


provide example for list return value in sort callable

my %h = :one(1), :two(2), :three(3); say %h.keys.sort({$^a.chars, $^a.Str})
# OUTPUT: «(one two three)␤»

It's not really a special case in sort but still not obvious (and darn usefull) so a nice example and maybe an explaination would be nice.

So you can use the comma for separating sorting criterium and return value?


metrics

want to track:

page views
total trucks
clicks to donate
Facebook shares
Tweets

For Facebook, we can use a version of the button that includes a count.  For Twitter, we can add a hashtag (#tacotruckify)
Make a super simple metrics page that shows trucks and clicks.

a6e2d9571c621bce86f


[WFCORE-5441] Logic in try/finally blocks hides exceptions

Fixes https://issues.redhat.com/browse/WFCORE-5441

Hello @jfdenise @darranl please review again and approve. I think all issues should have been fixed now, checks have passed.

@jfdenise I am happy with the security changes, I will let you add final approval as this touches more CLI classes.

@boris-unckel thanks


without

https://lodash.com/docs/4.17.15#without

duplicated with https://github.com/shogo82148/hi/issues/107


simple_usertype with newindex / index

It's not in the documentation, but perhaps it was just an oversight - I can't seem to bind new_index or index when using a simple usertype. For my particular application, I can't create the entire usertype at once, so I was hopeful to see I can use the set function with simple usertypes... however, the getter / setter are never called using this test code:
class CTestType {
public:
	CTestType() { ; }
	static std::shared_ptr<CTestType> NewTest() {
		return std::make_shared<CTestType>();
	}

	sol::object getter(const std::string& _name, sol::this_state _s) {
		if (_name == "a") {
			return sol::make_object(_s, 2);
		}

		sol::state_view L(_s);
		return sol::object(_s, sol::in_place, sol::nil);
	}

	void setter(const char* _name, sol::object _val, sol::this_state _s) {

	}
};

void print(const std::string& _msg) {
	std::string msg(_msg);
	msg = _msg;
}

void main() {
	sol::state lua;

	{
		lua["print"] = &print;
		lua.new_simple_usertype<CTestType>("test",
			sol::call_constructor, sol::factories(&CTestType::NewTest),
			sol::meta_function::index, &CTestType::getter,
			sol::meta_function::new_index, &CTestType::setter
		);
	}

	lua.script(R"(
		local t = test()
		print(t.a)
	)");
}

but if I use a regular usertype<T> then it's fine.
Are the metamethods just not supported for the simple_usertype?

Since my main issue is that I wanted to dynamically add functions as well, I'm just returning functions in the index method for now. This works just dandy, but it still seemed a little weird they didn't work with a simple usertype.

Thanks for reporting. This was fixed with a major overhaul of the simple_usertype implementation underneath. Performance investigation is still pending, but correctness >>> performance, and it should not really degrade at all given we're still sticking to lifting as much computation to compile-time as possible.
Good luck with sol2

Pull from the latest or download from the latest single/sol/sol.hpp to get going.

Thanks a ton for the update!


Specify custom CSS for the login UI

We now have a default login page from AWS Cognito. For example:
https://nens.auth.eu-west-1.amazoncognito.com/login?response_type=code&client_id=1f1rf3n93dnsvb6jinske1ccrl&redirect_uri=http%3A%2F%2Flocalhost%3A8000%2Fauthorize%2F&scope=openid+email+profile&state=CDcWBrIFmZ6mgxrOGyr53tCIut1YSR&nonce=77jaooEz6veIRaiBB1sv
The logo and CSS can be customized per client (application like Lizard / 3Di), and we can also set a N&S-wide default one. See:
See https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-app-ui-customization.html
Tagging   @Derryrover @LirryPinter  - we should decide how to approach this. The design options are limited.

Solved in https://github.com/nens/tf-cognito/pull/32

Solved in https://github.com/nens/tf-cognito/pull/32


ZERO Stage 3 with NVME Offload very slow

When running ZERO Stage 3 with NVME offload on a 10B parameter model, I am observing roughly 2.3 TFLOPS/GPU (whereas we expect to see closer to 30-40 TFLOPS/GPU for v100 GPUs, which we do when using ZERO Stage 2).
Potentially related to pre-fetching problems seen in https://github.com/microsoft/DeepSpeed/issues/1069
Method to reproduce is: https://gist.github.com/stephenrawls/1d396fe3f3c63ec1dc7edf85747503e5

we havent been updating this issue, but we were seeing the same issues even after optimizations to stage3 discussed in https://github.com/microsoft/DeepSpeed/issues/1069 and trying to figure out why (optimizations: https://github.com/jfc4050/DeepSpeed/tree/stage3)
i think i found the biggest (there's other smaller factors) culprit
with the blocking HtoD copy. note the cudaStreamSynchronize calls. These copies do not need to be blocking since the tensors are in pinned memory

with non-blocking HtoD copy (commit for the fix https://github.com/jfc4050/DeepSpeed/commit/0d1ac060a26cf155b55be82305caf0f76b5f2cd4)

i havent fully verified correctness yet, but based on behavior when there is no prefetching, seems like it is fine to make the copy non-blocking

getting 21.75 TFLOPS/GPU (single p3dn) after the fix

@jfc4050, thanks for sharing this update. Are you doing NVMe offload as well similar to the original post?

@tjruwase yes results were with NVMe offload (optimizer states and params). here's the full config
    ds_config = {
        "wall_clock_breakdown": False,
        "train_micro_batch_size_per_gpu": 8,
        "gradient_accumulation_steps": 1,
        "gradient_clipping": 1.0,
        "fp16": {"enabled": True},
        "zero_optimization": {
            "stage": args.zero_stage,
            "overlap_comm": True,
            "contiguous_gradients": True,
            "reduce_bucket_size": 1.5e8,
            "allgather_bucket_size": 1.5e8,
            "stage3_max_live_parameters": 1e9,
            "stage3_param_persistence_threshold": 1e3,
            "offload_optimizer": {
                "device": "nvme",
                "nvme_path": "/ssd1",
                "pin_memory": False,
            },
            "offload_param": {
                "device": "nvme",
                "nvme_path": "/ssd1",
                "pin_memory": False,
            },
        },
        "activation_checkpointing": {
            "partition_activations": False,
            "contiguous_memory_optimization": False,
            "number_checkpoints": 50,
            "cpu_checkpointing": False,
            "profile": True,
        },
        "optimizer": {
            "type": "Adam",
            "params": {
                "lr": 1e-4,
                "betas": [0.9, 0.95],
                "eps": 1e-8,
                "weight_decay": 0.1,
            },
        },
    }

Any update for this issue? Have it been solved?

@butterluo, unfortunately this thread went cold almost 2 years ago. The code has changed substantially.  Can you please open a new issue and share your experience? Thanks!


Multiple related links treated as one

Here's another thing I've noticed in converting my comment based help to an md file. The help looks like this:
.Link
http://jdhitsolutions.com/blog/2013/04/get-ciminstance-from-powershell-2-0

.Link
Get-CimInstance
New-CimSession
New-CimsessionOption

But when converted to markdown looks like this:
## RELATED LINKS

[http://jdhitsolutions.com/blog/2013/04/get-ciminstance-from-powershell-2-0](http://jdhitsolutions.com/blog/2013/04/get-ciminstance-from-powershell-2-0)

[Get-CimInstance
New-CimSession
New-CimsessionOption]()

[Online Version:]()

Which should appear as:
RELATED LINKS
http://jdhitsolutions.com/blog/2013/04/get-ciminstance-from-powershell-2-0
Get-CimInstance
New-CimSession
New-CimsessionOption
Online Version:
I think the markdown should look like this:
[Get-CimInstance]()
[New-CimSession]()
[New-CimsessionOption]()
[Online Version:]()

And I added 2 spaces so that each command is its own line.

I see here that the original markdown shows as 3 lines but it is really being treated as a single command. When I use other online markdown viewers like dillinger this is very clear.

Does comment based-help supports
.Link
Get-CimInstance
New-CimSession
New-CimsessionOption

to define 3 links? cc @jongeller

We create one markdown link from each related link that help engine produce in object from Get-Help.

Comment based help supports multiple entries under one .Link heading. But an online URL needs to be under a separate link. But I think I see the problem. PowerShell commands have a each link under a separate .Link entry in a maml file. But comment based help is more forgiving. Although that seems to be a problem here. I may have to adjust my comment help first.

Modified by comment help to use .Link heading for each entry and markdown now is as expected.

I added wontfix, since it's more "help engine" quirk, not platyPS.

Agreed. You can close this issue if you want. One thing that comes to mind is to have some sort of checklist for existing comment based help. These would be things to check before you attempted to convert. Maybe such a list already exists but if so I didn't come across it.  Or have a model comment-based example that converts perfectly to markdown.

ok, so I created a separate context in pester tests for comment-based help case.
https://github.com/PowerShell/platyPS/blob/843f322e9016e6525255259e35042546eb1c98ed/test/Pester/PlatyPs.Tests.ps1#L89

Closing old issues. If it comes up again, feel free to open a new one.


Overload all method signatures with an URL string type as an URL class

Small pull request who overload all method signatures with an URL as string type as an URL class.

I'm ok with this but it needs to be rebased with the new master after merging with OpenUnirest, which includes adding these to the new interfaces

Closing for now. Can be reopened if there is still interest but there is quite a bit of work to rebase this with master


Update the validator with spec modifications in Q3 2022

Describe the problem
The spec has advanced in Q3 2022 and the validator needs to be updated accordingly. The only PR that got merged in Q3 2022 on the GTFS Schedule Reference is:

Add trip-to-trip transfers with in-seat option #303

Describe the changes

New validation rules for the following changes:

from_stop_id and to_stop_id goes from Required to Conditionally required
from_trip_id and to_trip_id goes from Optional to Conditionally required

Adding 4 and 5 as enum in transfers.transfer_type

Sample GTFS datasets
No response
Severity
ERROR
Additional context
No response

Closing, duplicate of #1264