Is there a way to use images with no bounding boxes to train a FasterRCNN?

Hi, I am currently using version 1.7.1 of pytorch with the corresponding version of torchvision. Everything is going perfect while training a FasterRCNN FPN 50 using your reference training project. That is until I added images with no bounding boxes to the training data, then the whole thing  broke down.
I need background images with no bounding boxes because the model needs to learn that certain types of images will not have objects in them, and if I only feed it images with objects in them, it will never see these types of images and predict nonsense when it gets them during inference.
Is this feature supported? The error I currently get when trying to train is:
/home/joaqo/.local/lib/python3.6/site-packages/torchvision/models/detection/generalized_rcnn.py 69 forward
boxes.shape))

ValueError:
Expected target boxes to be a tensorof shape [N, 4], got torch.Size([0]).

Which makes perfect sense if you want to force your model to only train on images with targets, regrettably thats not my use case. If this is not currently supported? Would anyone provide some pointers for me to add support for this?
Thanks.

Hi @joaqo !
Yes it is possible to use images with no bounding boxes in training data with torchvision.
This feature was implemented from release 0.6
Here is a note on how to use it.
If you have background image with No bounding boxes at all. The boxes are assumed to be a [0, 0, 0, 0], hence the area is 0. Also the label is assumed as background label which is 0 (predefined by torchvision models).
For e.g if such image has image_id =  3. So for such images you need to pass.
{
"image_id" : 3,
"labels": torch.zeros(0, dtype=torch.int64),
"boxes": torch.zeros((0, 4), dtype=torch.float32,
"area": torch.zeros(0, dtype=torch.float32),
}
I think the reference script will work out of box, if you make changes to your dataset.
Feel free to post problems you face in this issue.

Hi @oke-aditya thanks for your prompt reply!
I've followed your instructions, and they work great when training on a simple gpu, but fail when using torch.distributed.launch for multi gpu training. I get the following error message:
simple_detector/train.py 273 <module>
main(args)

simple_detector/train.py 141 main
train_one_epoch(model, optimizer, data_loader, device, epoch, args.print_freq, experiment_logger)

/home/joaqo/pano/simple_detector/engine.py 34 train_one_epoch
loss_dict = model(images, targets)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/modules/module.py 727 _call_impl
result = self.forward(*input, **kwargs)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py 617 forward
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py 643 scatter
return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 36 scatter_kwargs
inputs = scatter(inputs, target_gpus, dim) if inputs else []

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 28 scatter
res = scatter_map(inputs)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 15 scatter_map
return list(zip(*map(scatter_map, obj)))

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 17 scatter_map
return list(map(list, zip(*map(scatter_map, obj))))

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 19 scatter_map
return list(map(type(obj), zip(*map(scatter_map, obj.items()))))

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 15 scatter_map
return list(zip(*map(scatter_map, obj)))

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py 13 scatter_map
return Scatter.apply(target_gpus, None, dim, obj)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/_functions.py 92 forward
outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams)

/home/joaqo/.local/lib/python3.6/site-packages/torch/nn/parallel/comm.py 186 scatter
return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams))

RuntimeError:
chunk expects at least a 1-dimensional tensor

Not sure what could be the issue.

Well I don't have any experience in multi-gpu training environments. Also, I don't have resources to reproduce this.
Here are few links which might help you.
I searched for error chunk expects at least a 1-dimensional tensor
https://discuss.pytorch.org/t/dataparallel-multi-gpu-runtimeerror-chunk-expects-at-least-a-1-dimensional-tensor/45340
https://stackoverflow.com/questions/56177305/dataparallel-multi-gpu-runtimeerror-chunk-expects-at-least-a-1-dimensional-tens

I seem  to have fixed by inserting all the zeros in the example you linked to into arrays.
So from this:
target = {"boxes": torch.zeros((0, 4), dtype=torch.float32),
          "labels": torch.zeros(0, dtype=torch.int64),
          "image_id": 4,
          "area": torch.zeros(0, dtype=torch.float32),
          "iscrowd": torch.zeros((0,), dtype=torch.int64)}

to this:
target = {"boxes": torch.zeros((0, 4), dtype=torch.float32),
          "labels": torch.zeros([0], dtype=torch.int64),
          "image_id": torch.tensor([4]),  # Yeah, I even had to do it to image_id!
          "area": torch.zeros([0], dtype=torch.float32),
          "iscrowd": torch.zeros([0], dtype=torch.int64)}

Thanks!

@joaqo could you provide a minimal reproducing example that works on GPU but fails on distributed? This could be a bug

Hi @NicolasHug, yeah I'm pretty sure its a bug on Pytorch's side. The code I'm using is the reference detection project (Using pytorch 1.7.1) adapted to a different dataset. I think you should be able to reproduce it by setting the dataset to COCO, which is the default, and making sure the zeros are not inside an array, as shown in my previous comment.
It could also just be that the reference documentation is incorrect and in fact the model expects the torch.zeros array to be initialized with an array instead of just a plan 0.

Hi @oke-aditya, does the use of images with no bounding boxes in training help? As we are only passing empty tensors in the target!

Hi @sahilg06 .
Yes passing images with empty tensors do help in detection!! And this is a really good trick to improve detection.
I will answer this on both ways, high level intuition so to know why and where it works. Also a precise answer to how it works
:smile: Note that this method is general to object detection task, and should work out of box for many models.
Here is how!
High Level intuition : -
Say you are detecting fruits (E.g. classes are apple, mango, orange)
And you have an image which contains, red tomatoes (I'm not considering tomato as fruit here).
If you have trained your model only on fruits images ignoring tomatoes then it is highly likely that a detection model will get confused with apples and tomatoes (since both are red, have similar shape,). The model never knew about tomato ! So it will make a biased guess that it is an apple!
Now, consider the case where you explicitly labelled these data as 0 tensors. The tomatoes now become a background object!
While training, the model now knows that the tomatoes are not apples! These are to be ignored. So it will learn better and hence differentiate tomatoes and apples much better. This method of marking the object as background class would help the model in reducing false positives.
Technical Answer: -
A detection problems involves 2 stages, classification and regression. It usually has a loss function which consists of 2 parts, classification loss and regression loss. By passing an object as background class, it will help the classification loss to differentiate target from background. When we explicitly pass background an object as background, these samples too help in determining classification loss.
Here is to the regression part!. By passing a tensor [0,0,0,0] for bounding box, this marks the object as a point object (basically a dot) which implies this is not a detectable object. So hence this is also ignored.
And when we do backward on these losses! This information gets propagated and hence the detector recognizes such explicit background objects.
(My PyTorch experience is far less, and hence I cannot provide a benchmark / any comparative analysis.)

Thanks, @oke-aditya
But when we train for an image that contains bounding boxes (say image contains only 1 bbox), we pass the label as something like this [1], now model can learn that it is of 1st class.
But when we pass the label for an image that doesn't contain any bounding box, we pass something like this [] (basically an empty tensor so even if pass torch.ones(0, dtype=torch.int64) instead of torch.zeros(0, dtype=torch.int64) it doesn't matter). So How model learns from this that it is of 0 or background class?

Internally FasterRCNN (and many other object detection models) separate the image into areas called anchors, for each of these anchors they predict if it is one of the foreground classes or the background class. When training the model grabs a bunch of these anchors in different strategic ways, tries to balance a decent propotion between foreground and background classes (look here and here for the proportions in FasterRCNN), and calculates the loss function using them.
So it's trying to learn how well it did on the anchors it predicted as foreground but also on the ones it predicted as background. So you'll see that training on an image with no annotation is not really a problem, its as if you were just setting the proportion of foreground/background to just be background during that one sample. You cannot do it for all samples, but for a subset of them its fine.

Thanks, @joaqo, @oke-aditya. My exact doubt got cleared from this section of the code.https://github.com/pytorch/vision/blob/993325dd82567f5d4f28ccb321e3a9a16984d2d8/torchvision/models/detection/roi_heads.py#L569-L577

Closing as seems that this has been answered / resolved already.


how can I get the ouput of reptyr -l into tmux

I'm writing a script using tmux and gdb to automatically generate a productive debugger. Currently I use sleep infinity to set up an inferior tty. I'd like to switched to reptyr -l. However, i cannot get the output of reptyr -l in order to make the whole setting up process automatic.

The script looks like this ,
#!/usr/bin/env bash

tmux neww -n Debug
tmux splitw -h -p 50
tmux select-pane -t 1
tty1=$(tmux display -p "#{pane_tty}")
tmux splitw -v -p 40
tty3=$(tmux display -p "#{pane_tty}")
tmux select-pane -t 1
tmux splitw -v -p 40
tty2=$(tmux display -p "#{pane_tty}")
# tmux send-keys "reptyr -l" C-m
tmux send-keys "sleep infinity" C-m
tmux select-pane -t 4
tmux send-keys "cgdb -- $1 -tty $tty2 -ex \"dashboard -output $tty1\" -ex \"dashboard expression -output $tty3\"" C-m

If you run reptyr -l with a command line, it will execute that command with the REPTYR_PTY environment variable set.  So you could do something like
reptyr -l sh -c 'echo $REPTYR_PTY > /tmp/reptyr-pty'

and then read the pty out of that tmpfile.


InstallAppleCertificate@2 Doesn't accept variables for parameter certSecureFile

Required Information
Entering this information will route you directly to the right team and expedite traction.
Feature / Bug
InstallAppleCertificate@2
Environment

Server - Azure Pipelines

Agent - Private:

macOS-10.15

Issue Description
The InstallAppleCertificate@2 task doesn't support the ability to pass in a variable for the secure file. Is this on purpose or is this something that could be added? Our Xcode project builds multiple applications from one source and each app has its own signing certificate. I was hoping to utilize this task with matrices to install the relevant certificate per run.
Ideally it would be helpful to be able to do something like this.
- task: InstallAppleCertificate@2
  displayName: '🔐 Install Signing Certificate'
  inputs:
    certSecureFile: $(certificate)
    certPwd: $(certificate_password)

Error logs
There was a resource authorization issue: "The pipeline is not valid. Job Job: Step InstallAppleCertificate input certSecureFile references secure file $(certificate) which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

Hi @Zanchee
Thank you for opening the issue!
I've tried to reproduce the issue but it worked fine for me. For us to investigate the issue further, could you please share the following information if it's possible:

build logs in debug mode;
YAML file containing the pipeline definition;
We're looking forward to hearing from you!

@egor-bryzgalov so I can't grab you a build log b/c it doesn't startup the build. It immediately cancels with the following issue

I pruned my YAML to the following, just to make sure I was focusing JUST on the certificate installs.
pool:
  vmImage: 'macOS-10.15'

variables:
- group: certificate-passwords

strategy:
  matrix:
    APP_1:
        certificate: 'MY_CERTIFICATE_1.p12'
    APP_2:
        certificate: 'MY_CERTIFICATE_2.p12'

steps:
- task: InstallAppleCertificate@2
  displayName: '🔐 Install Certificate'
  inputs:
    certSecureFile: $(certificate)
    certPwd: $(certificate_password)

I am experiencing the same issue!

Here's how to reproduce:
jobs:
  - job:
    strategy:
      matrix:
        mojave:
          imageName: 'macOs-10.14'
          certificateFile: myCertificate
          certificatePassword: myPassword
        catalina:
          imageName: 'macOs-10.15'
          certificateFile: myCertificate1
          certificatePassword: myPassword2
          provisioningProfile: myProfile
    pool:
      vmImage: '$[variables.imageName]'
    steps:
    - task: InstallAppleCertificate@2
      displayName: 'Install an Apple certificate'
      inputs:
        certSecureFile: $(certificateFile)
        certPwd: $(certificatePassword)

From what I can tell, there is some validation that happen on the pipeline definition before the variable substitution so the build doesn't even start. I am not able to find any of that logic in this repo though.

@LaurentGoderre @Zanchee Thank you for the information you've provided! We'll investigate this issue and keep you posted on the results.

Hi @Zanchee @LaurentGoderre
It seems that it's not supported at the moment.
This issue's root cause is in pipeline processing steps order. It's required to have secureFile inputs values defined by the authorization step otherwise errors like one you had would be thrown.
So the matrix is resolved after the authorization step.
You can find more information about pipeline processing here.

Thanks for the  update. We ended up making a dummy matrix yesterday by creating a pipeline that just calls a template with the same parameters. This worked 👍

@Zanchee I'm closing this issue. Please feel free to reopen it if it's necessary. Thank you!

@egor-bryzgalov I would like to re-open this issue. Or at least ask for help. I believe i'm experiencing the same issue but don't see a workaround/solution.
note: this also looks to be the same issue: https://developercommunity.visualstudio.com/content/problem/984321/installsshkey-task-cant-resolve-variable-reference.html
I have a pipeline that deploys an app to a staging environment and later, after approval, to a production environment. This means that putting the variables for the certSecureFile for instance cant be place at the root level. They are different per environment.
But when placed at the stage or job level the InstallAppleCertificate task fails with the above mentioned error.
So how can we use the InstallAppleCertificate task with a certSecureFile: '$(p12FileName)' configuration? Combined with variables defined at the stage or job level since they are different per stage/job.

@egor-bryzgalov I would like to re-open this issue. Or at least ask for help. I believe i'm experiencing the same issue but don't see a workaround/solution.
note: this also looks to be the same issue: https://developercommunity.visualstudio.com/content/problem/984321/installsshkey-task-cant-resolve-variable-reference.html
I have a pipeline that deploys an app to a staging environment and later, after approval, to a production environment. This means that putting the variables for the certSecureFile for instance cant be place at the root level. They are different per environment.
But when placed at the stage or job level the InstallAppleCertificate task fails with the above mentioned error.
So how can we use the InstallAppleCertificate task with a certSecureFile: '$(p12FileName)' configuration? Combined with variables defined at the stage or job level since they are different per stage/job.

Hi @Guuz
I think that you could try to use templates as was suggested by @Zanchee.
As per this article  templates are processed before checking resources authorization. Only variables explicitly included in the pipeline can be used for such inputs. It also makes sense to experiment with variables syntax if it suits your demands.
If it doesn't help you to resolve your issue please share your YAML structure so we can look for possible ways to help you.

Hi @Guuz
I think that you could try to use templates as was suggested by @Zanchee.
As per this article  templates are processed before checking resources authorization. Only variables explicitly included in the pipeline can be used for such inputs. It also makes sense to experiment with variables syntax if it suits your demands.
If it doesn't help you to resolve your issue please share your YAML structure so we can look for possible ways to help you.

I think the YAML might help. since i'm using templates already. but this seems to cause the issue. The below YAML files are obviously shortend by A LOT. just to show you the most important parts:
# Part of the main pipeline that runs. contains other stages for testing for instance.
# And will later deploy to ACC, and with a manual approval build and deploy to PROD.
stages:
  - stage: build_acc
    jobs:
      - job: build_ios_acc
        variables:
          - group: ios_acc_variables
        steps:
          - template: pipeline-helpers/acc/build-ios.yml
      - job: build_android_acc
        variables:
            - group: android_acc_variables
        steps:
          - template: pipeline-helpers/acc/build-android.yml

# Part of the build-ios.yml file.
steps:
  - task: CocoaPods@0
    inputs:
      projectDirectory: 'ios'
  - task: InstallAppleCertificate@2
    inputs:
      # This is what breaks
      certSecureFile: '$(p12FileName)'
      certPwd: '$(p12Password)'
      keychain: 'temp'
      deleteCert: true

The "Authorize resources" button seems to work. But does not fix the issue. The resources have also already been given access to all pipelines anyway.

Moving the variables to the root level will cause this to work. Proving the bug.
But they can't be at the root level because we need to differentiate between ACC and PROD. Let alone that the iOS and Android steps (templates) might use the same variable so even they need to be different. The root scope is not an option, and shouldn't be in scenario's like this.
Thanks for helping and looking into this!

I think the YAML might help. since i'm using templates already. but this seems to cause the issue. The below YAML files are obviously shortend by A LOT. just to show you the most important parts:
# Part of the main pipeline that runs. contains other stages for testing for instance.
# And will later deploy to ACC, and with a manual approval build and deploy to PROD.
stages:
  - stage: build_acc
    jobs:
      - job: build_ios_acc
        variables:
          - group: ios_acc_variables
        steps:
          - template: pipeline-helpers/acc/build-ios.yml
      - job: build_android_acc
        variables:
            - group: android_acc_variables
        steps:
          - template: pipeline-helpers/acc/build-android.yml

# Part of the build-ios.yml file.
steps:
  - task: CocoaPods@0
    inputs:
      projectDirectory: 'ios'
  - task: InstallAppleCertificate@2
    inputs:
      # This is what breaks
      certSecureFile: '$(p12FileName)'
      certPwd: '$(p12Password)'
      keychain: 'temp'
      deleteCert: true

The "Authorize resources" button seems to work. But does not fix the issue. The resources have also already been given access to all pipelines anyway.

Moving the variables to the root level will cause this to work. Proving the bug.
But they can't be at the root level because we need to differentiate between ACC and PROD. Let alone that the iOS and Android steps (templates) might use the same variable so even they need to be different. The root scope is not an option, and shouldn't be in scenario's like this.
Thanks for helping and looking into this!

@Guuz I see that you're using variable groups but as per the article mentioned above Variable groups are themselves a resource subject to authorization, so their data is likewise not available when checking resource authorization. I guess the only way here might be to pass secure files as template parameters somehow. Here you can find more information about template parameters.

@Guuz I see that you're using variable groups but as per the article mentioned above Variable groups are themselves a resource subject to authorization, so their data is likewise not available when checking resource authorization. I guess the only way here might be to pass secure files as template parameters somehow. Here you can find more information about template parameters.

Using a template paramater sounds like it could work. I'll try it out and post the results here. Thanks!
Thought:
It would still feel "broken" though. Since using variables works only at the root level, and not lower. But this is the intended usage of variables.
The paramaters sound like a viable workaround.

Using a template paramater sounds like it could work. I'll try it out and post the results here. Thanks!
Thought:
It would still feel "broken" though. Since using variables works only at the root level, and not lower. But this is the intended usage of variables.
The paramaters sound like a viable workaround.

@egor-bryzgalov i tried to use the template parameters combined with variables. But this resulted in the same error.
I ended up using the parameters and hardcoding the required arguments for them, instead of using variables. since the secure files are only a name reference, this is acceptable. the passwords (which should be secure) do work via variables and then passing them on as parameters.
i hope it makes any sense what i'm saying ^_^ it's kinda hard to explain. the take-away is that i still dont see a way to use devops variables and templates. to re-use build steps and inject them with variable inputs configured outside the pipelines.

@egor-bryzgalov i tried to use the template parameters combined with variables. But this resulted in the same error.
I ended up using the parameters and hardcoding the required arguments for them, instead of using variables. since the secure files are only a name reference, this is acceptable. the passwords (which should be secure) do work via variables and then passing them on as parameters.
i hope it makes any sense what i'm saying ^_^ it's kinda hard to explain. the take-away is that i still dont see a way to use devops variables and templates. to re-use build steps and inject them with variable inputs configured outside the pipelines.

Correct, you can't use a variable explicitly for a certificate, but by creating a template that takes in parameters, and then just creating N jobs each passing in your required parameters to the template, you are essentially doing the same thing as a matrix, but now on each template call you can define the certificate you wish to use.
To be honest. We publish many apps from one code source, and this covers that solution perfectly. I'm not really sure what benefit being able to use a variable would be. I can't think of a use case. Even if you could, it's either a single update to a yml or a single update to the Azure GUI
This has worked perfectly for us now. I think I was simply just too focused on using a matrix that I forgot about the other tools in the shed.

Correct, you can't use a variable explicitly for a certificate, but by creating a template that takes in parameters, and then just creating N jobs each passing in your required parameters to the template, you are essentially doing the same thing as a matrix, but now on each template call you can define the certificate you wish to use.
To be honest. We publish many apps from one code source, and this covers that solution perfectly. I'm not really sure what benefit being able to use a variable would be. I can't think of a use case. Even if you could, it's either a single update to a yml or a single update to the Azure GUI
This has worked perfectly for us now. I think I was simply just too focused on using a matrix that I forgot about the other tools in the shed.

I managed to solve the same issue:

by defining the variables like this:
variables:
  - group: CodeSign
  - name : BuildConfiguration
    value: 'Release'
  - name: SetupBuildConfiguration
    value: 'Release'

stages:
- stage: Stage1

  jobs:
  - job: LLT
    timeoutInMinutes: 0
    steps:
    - checkout: self
      submodules: true
      persistCredentials: true

it's a desktop application building pipeline but I think the issue was the same.

I managed to solve the same issue:

by defining the variables like this:
variables:
  - group: CodeSign
  - name : BuildConfiguration
    value: 'Release'
  - name: SetupBuildConfiguration
    value: 'Release'

stages:
- stage: Stage1

  jobs:
  - job: LLT
    timeoutInMinutes: 0
    steps:
    - checkout: self
      submodules: true
      persistCredentials: true

it's a desktop application building pipeline but I think the issue was the same.

Closing the issue since the workaround has been found.

Please feel free to open a ticket at https://developercommunity.visualstudio.com/ if you'd like to suggest an enhancement or found any issue related to Azure DevOps.

Hi,
The "workaround" is not acceptable (and adaptable) for me... 2 days I'm trying to solve this... without success.
I'm building 18 applications (x2 : Android & iOS = 36 unique apps), based on the same unique Xamarin.Forms code-base.
And each unique application has his unique .Keystore(android) or .mobileprovision(iOS)
So I have only 1 pipeline for Android and 1 pipeline for iOS, and in these 2 pipelines I'm looping foreach application I choose to build (as build parameter) to copy the specific files, to get the specific variablesGroup, to build the Xamarin.android, and to SIGN each of them with their own keystore... all is working fine, except that I'm unable to sign correctly with the good keystore (it takes always the "defaut-keystore") :-(
I've tried with different tasks (androidSignin / CmdLine / DownloadSecure / ...) but everytime it takes the SecureFile in the "pre-build step) and impossible to get/set the securefile in my "each" loop...
So I'm completely stuck, unable to get the corresponding keystore for each application.
Here's a part of YAML, if anybody can help, it would be great 👍 :
#Ask (as BuildParameter) the city/cities that need to be built
# !!!! Add new cities here (in values and in default) after having created the "city-specific-VariableGroup"
parameters:
  - name: CitiesToBuild
    type: object
    values:
      - CityA
      - CityB
      - CityC
      - CityD
      - CityE
    #By default -> build ALL the cities (if new comercea "master functional modification" for example)
    default:
      - CityA
      - CityB
      - CityC
      - CityD
      - CityE

#Do not trigger automatic CI !
trigger: none

#Our own Mac Agent
pool:
  name: Respawnsive
  demands:
  - MSBuild
  - Xamarin.Android
  - JDK

#Get the variables from _Common_VariablesGroup and other locals variables.
variables:
- group: _Common_VariablesGroup
- name: buildConfiguration
  value: 'AppStore'
- name : outputDirectory
  value: '$(build.binariesDirectory)/$(buildConfiguration)'

#Define the build Name
name: $(Build.DefinitionName)_V$(VersionMajor).$(VersionCode)_$(Date:yyyyMMdd)$(Rev:.r)

jobs:
  #Run a new Job foreach selected city !
  - ${{ each city in parameters.CitiesToBuild }}:
    - job:
      displayName: 'Build AppStore Android for ${{ city }}'

      #Clean the workspace between each city
      workspace:
        clean: all

      #Get the variables group for the specific city
      variables:
      - group: ${{ city }}VariableGroup

      # ALL JOB STEPS
      steps:

      #Copy/Overrite the City specific files (images, json, etc.) to local source-code
      - task: CopyFiles@2
        inputs:
          SourceFolder: 'Comercea/Cities/${{ city }}'
          Contents: '**'
          TargetFolder: 'Comercea/FormsKiosk'
          OverWrite: true
          retryCount: '3'

      - task: NuGetToolInstaller@0
        displayName: 'Use NuGet 5.11.0'
        inputs:
          versionSpec: 5.11.0

      - task: NuGetCommand@2
        displayName: 'NuGet restore'
        inputs:
          command: 'restore'
          restoreSolution: '**/*.sln'
          includeNuGetOrg: true
          verbosityRestore: Detailed

      - task: android-manifest-package-name@1
        displayName: 'AndroidManifest.xml Change packageName and appLabel'
        inputs:
          sourcePath: 'Comercea/FormsKiosk/FormsKiosk.Android/Properties/AndroidManifest.xml'
          packageName: '$(PackageId_Android)'
          appLabel: '$(ApplicationName)'
          printFile: true

      - task: android-manifest-version@1
        displayName: 'AndroidManifest.xml Change versionCode and versionName'
        inputs:
          sourcePath: 'Comercea/FormsKiosk/FormsKiosk.Android/Properties/AndroidManifest.xml'
          versionCodeOption: buildid
          versionCode: '$(VersionCode)'
          versionName: '$(VersionMajor).$(VersionCode)'
          printFile: true

      - task: XamarinAndroid@1
        displayName: 'Build Xamarin.Android project **/*Droid*.csproj'
        inputs:
          projectFile: '**/*Droid.csproj'
          outputDirectory: '$(outputDirectory)'
          configuration: '$(buildConfiguration)'
          msbuildVersionOption: latest
          msbuildArguments: '/verbosity:detailed'

      - task: DownloadSecureFile@1
        displayName: 'DownloadingKeyStore'
        name: 'MyKeystoretemp'
        inputs:
          secureFile: '$(Android_KeystoreFilename)'
          retryCount: 3

      - script: |
          echo '-------'
          echo 'Android_KeystoreFilename:'
          echo '$(Android_KeystoreFilename)'
          echo '-------'
          echo 'MyKeystoretemp.secureFilePath:'
          echo '$(MyKeystoretemp.secureFilePath)'
          echo '-------'

      - task: CmdLine@2
        displayName: 'Signing and aligning AAB with commandline'
        continueOnError: true
        inputs:
          script: 'jarsigner -verbose -sigalg SHA256withRSA -digestalg SHA-256 -keystore $(MyKeystoretemp.secureFilePath) -storepass $(Android_KeystorePwd) -keypass $(Android_KeyPwd) $(outputDirectory)/*.aab $(Android_KeystoreAlias)'

      - task: AndroidSigning@2
        displayName: Sign and Align AAB
        enabled: false
        inputs:
          apkFiles: '$(build.binariesdirectory)/$(BuildConfiguration)/*.aab'
          jarsignerKeystoreFile: '$(Android_KeystoreFilename)'
          jarsignerKeystorePassword: $(Android_KeystorePwd)
          jarsignerKeystoreAlias: $(Android_KeystoreAlias)
          jarsignerKeyPassword: $(Android_KeyPwd)
          jarsignerArguments: '-sigalg SHA256withRSA -digestalg SHA-256'

      - task: PublishBuildArtifacts@1
        displayName: 'Publish Artifact ${{ city }}'
        inputs:
          PathtoPublish: '$(outputDirectory)'
          ArtifactName: '${{ city }}'
          publishLocation: 'Container'

Correct, you can't use a variable explicitly for a certificate, but by creating a template that takes in parameters, and then just creating N jobs each passing in your required parameters to the template, you are essentially doing the same thing as a matrix, but now on each template call you can define the certificate you wish to use.
To be honest. We publish many apps from one code source, and this covers that solution perfectly. I'm not really sure what benefit being able to use a variable would be. I can't think of a use case. Even if you could, it's either a single update to a yml or a single update to the Azure GUI
This has worked perfectly for us now. I think I was simply just too focused on using a matrix that I forgot about the other tools in the shed.

Worked for us too, thank you.

The workarounds listed here did not resolve our issue. We were still not able to pass a variable containing the secret. A colleague suggested re-implementing the ADO task using native macOS commands. We settled on the following which allowed us to pass variables as expected:
- task: DownloadSecureFile@1
          name: mySecureFile
          inputs:
            secureFile: 'ios-cert.p12' # name of the secure file

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      CERT_SECURE_FILE="$(mySecureFile.secureFilePath)"
      CERT_PWD="$(p12-password)"
      KEYCHAIN_PATH="/tmp/temporary.keychain"
      KEYCHAIN_PWD=$(uuidgen) # Generate a random password

      create_temp_keychain() {
          # Create a new keychain
          security create-keychain -p "${KEYCHAIN_PWD}" "${KEYCHAIN_PATH}"

          # Set the keychain timeout to 1 hour for this session
          security set-keychain-settings -t 3600 -u "${KEYCHAIN_PATH}"

          # Add the keychain to the search list
          security list-keychains -s "${KEYCHAIN_PATH}"

          # Unlock the keychain
          security unlock-keychain -p "${KEYCHAIN_PWD}" "${KEYCHAIN_PATH}"
      }

      import_certificate() {
          # Import the certificate
          security import "${CERT_SECURE_FILE}" -k "${KEYCHAIN_PATH}" -P "${CERT_PWD}" -T /usr/bin/codesign

          # Set the key partition list to fully trust the imported certificate for code signing
          security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "${KEYCHAIN_PWD}" "${KEYCHAIN_PATH}"
      }

      create_temp_keychain
      import_certificate
      # Find the identity of the signing certificate
      APPLE_CERTIFICATE_SIGNING_IDENTITY=$(security find-identity -v -p codesigning "${KEYCHAIN_PATH}" | head -n 1 | awk -F '"' '{print $2}')
      echo "##vso[task.setvariable variable=APPLE_CERTIFICATE_SIGNING_IDENTITY;]$APPLE_CERTIFICATE_SIGNING_IDENTITY"
  displayName: 'Install Apple Certificate'

Thanks @sherenator for that workaround, it works a treat.
I'd also like this to be re-opened. We pull our secrets in from key vault, so they are not available until execution time. This means the pre-job step that InstallAppleCertificate@2 inserts, that needs the password, fails as the key vault task hasn't run yet.

i just encountered this when attempting to create a deploy build template for multiple stages/applications.


java.net.UnknownHostException: kubernetes.default.svc: unknown error - KubernetesFacade.getNodeOfPod line 164

Hi
I am trying to get the Fabric8 jenkins-pipeline-library to run outside a pure fabric8 configuration, and have created a separate namespace for this on OpenShift Origin 1.2 also with an Jenkins server running outside of OpenShift. The Kubernetes jenkins cloud slave provider plugin works fine, and can launch jenkinsslaves on OpenShift for me, but the "kubernetes pipeline library" is not working for me.
I keep getting an "unknown error" in the KubernetesFacade getNodeOfPod function where I guess the code is trying to lookup the node of the first pod launched, so that it can launch the second pod directly on the same host as the first one. See the attached log files.
jenkinslog.txt
unknownerror.txt
I have tried the setup on the same OpenShift origin installation with the default fabric8 installation, where jenkins is running inside OpenShift in the same namespace as the "build slaves" are launched, and there everything is working just fine.
Any good suggestions to what may be wrong here?
Best regards
Lars Milland

Hi
I have still not solved the problem reported on this issue, so I do hope that there are someone that can give some qualified input to which direction to look.
I have tried to narrow in what could be causing the problem, but without much success yet, so I have then also tried to create small sample Java program that is doing the same thing as the pipeline code is doing through its groovy script when it is to launch the "buildpod". That looks as shown here below, and the I have manually created a "permanent" jenkins-slave on the OpenShift cluster where I execute the Java program with oc rsh into the OpenShift pod, and strangely enough that code works just fine. When the exact same pod is used to run the groovy script shown also here from the associated jenkins system the host lookup fails somewhere below call to:
io.fabric8.kubernetes.pipeline.KubernetesFacade.getNodeOfPod(KubernetesFacade.java:164)
Groovy script:
node('manual') {   kubernetes.pod('buildpod').withImage('fabric8/maven-builder')         .inside { 		    echo "Hello from buildpod" 	}  }
Java sample program with Kubernetes client that strangely enough works:
`		String master = "https://kubernetes.api.mydomain:8443/";
	Config config = new ConfigBuilder().withMasterUrl(master).build();
	try (final KubernetesClient client = new DefaultKubernetesClient(config)) {
	  Pod pod = client.pods().withName("jenkins-jnlp-client-manual").get();
          String nodeName = pod.getSpec().getNodeName();
          Node node = client.nodes().withName(nodeName).get();
	  System.out.println("Node: " + node);
	} catch (KubernetesClientException e) {
		System.out.println(e.getMessage());
	}

`
Best regards
Lars Milland

I'm having the exact same problem, although in my case I'm still using gofabric8 to launch a minikube and them I'm accessing it for a remote Jenkins (which happens to be running in a different virtualbox on my machine). I can also provision slaves on Kubernetes.
Even though my Cloud is setup with the correct Kubernetes URL ("Test Connection" tells me everything is fine) and I've replicated the templates over with the matching KUBERNETES_MASTER value to match the URL, it still appears that when running a build that uses this library it uses the default master URL defined here: https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L98
I suspect that in your case Lars it's a similar situation where what is happening is whilst the KubernetesClient that is created for the Cloud stuff gets the correct environment variables set that when invoked from this library it resorts to the default, which will only work from "within" a fabric8 deployment.
It was pretty tricky to spot that it was this that was the issue, because most of the time I would get a socket time out and only sometimes would it tell me the domain it was trying to hit. Adding a logger also didn't report anything when running, although when using "Test Connection" under the Cloud definition it would produce output.

Sorry I originally missed this issue.  @nickdgriffin you are correct that this shared pipeline library assumes that it is running inside the kubernetes / openshift cluster and not on a remote Jenkins.  We can add some checks before calling this function and ensure that the library has access to the kubernetes token that is automatically mounted into the Jenkins pod.
I'll add a note to the readme to clear this up.


The resolution is unable to recover when network quality is back to good or excellent

Describe the bug
We are trying to use the client-sdk-react-native to build an Android app with 4G networking.
While the resolution seems ok at the beginning of the experiment (network state excellent), when the network quality dropped significantly to the poor state, the resolution also dropped, but when the network quality returned to excellent, the resolution did not increase.
Here is our setting:
const preset = VideoPresets.h720
  const [room] = useState(
    () =>
      new Room({
        dynacast: true,
        videoCaptureDefaults: {
          resolution: preset.resolution,

        },
        publishDefaults: {
          videoEncoding: preset.encoding,
          simulcast: false,
        },
      }),
  );

Server

Version: 1.1.0
Environment: eks
any other information about your deployment setup

Client

SDK: client-sdk-react-native
Version: 0.1.0

To Reproduce
Steps to reproduce the behavior:

Mobile client create and join a room.
Publish by 4g network with excellent network state.
Drop the network quality significantly to poor state,  the resolution also dropped.
The network quality returned to excellent, the resolution did not increase.

Expected behavior
When the network quality returns to excellent, the resolution should increase again.
Screenshots

Excellent network state with good resolution.

Excellent network state with bad resolution.

Additional context
Add any other context about the problem here.

simulcast: false，
When you publish the stream, try to give this parameter to ensure that the quality of the video stream is the image quality you set.

Hi @echo-LuGuang ,
The simulcast  is already set to false.

@benjamin658 did you disable adaptive stream as well? It recovers fine from poor network conditions in our testing.
What are you using as the publishing dimensions/resolution? The screenshot indicates it's much higher than 720p. (indicated by your code)

Hi @davidzhao,
Yes, we disabled the adaptive stream from the subscriber, we also adjusted the framerate to 20, and codec to h264, it seems a little bit better.
We also suspect that our publisher device's power supply is not stable, it could influence the 4g connectivity and camera (maybe), let us do more testing.
Thanks.


Format of dates, times and currencies start configuration missing in documentation

When we run tests with on our computers with European region configs. Tests are failing due to differences with the date and time formats.
We are missing an option to set up timezone and regional config in gauge to everyone in our team will have the same regional config and timezone.

One more info, it fails only when tests are running in headless browser.

Timezone and format are usually set either by the operating system (locales) or if you use any sort of internationalization.
In test code, you could use UTC time instead of local time, and offset it when required. Or in the OS, set the timezone (ex. timedatectl set-timezone <timezone> in linux based systems).
Gauge does not manage either, presently. What sort of configuration would you like here? Perhaps if you can illustrate with an example, it'll be help in understanding this better.

One more info, it fails only when tests are running in headless browser.

@fhavrlent - what browser/driver are you using? Is there any sample that you can share please?

Can someone share some sample code or example how this fails? Like @sriv mentioned this can be set in code. You can also use environments to store and use the time zones.

This runs in Chrome, using Selenium.
We have a date, 1980-04-17 for example, that is in year-month-day format. When I run tests in headless, it seems to be using different input format, so I end up with 12/04/1980 (month/day/year) -> it fixes 17 to 12. When I run it with the normal browser, date inputs are correct, so this one would give me 04/17/1980.

it seems to be using different input format,

Unfortunately, handling dates are hard so it's best that the tests be deliberate about the date formats by using Date libraries provided by the language or using external libraries written for this specific purpose like Joda time or moment js


Fail gracefully if Helm chart already exists

If you run up.sh on an exiting cluster with the flag services_only kraken will fail on:
TASK [roles/kraken.services : Install charts from registry] and potentially (I have not tested this one, but I assume it would show the same behavior): TASK [roles/kraken.services : Install charts from repo]
An example of the error is:
 "stderr": "usage: appr [-h]\n            {pull,run-server,show,inspect,list,delete-package,helm,version,logout,deploy,plugins,push,login,config,channel,jsonnet}\n            ...\nappr: error: \nmessage: 'Error: a release named heapster already exists.\n\n  Run: helm ls --all heapster; to check the status of the release\n\n  Or run: helm del --purge heapster; to delete it\n\n  '\n\nError: plugin \"registry\" exited with error"
We should either ignore these errors and keep running the script or use a conditional check to see if the Helm package already exists before trying to install.

if we are going to ignore the errors, then we should spit out some sort of message at the end so the user doesn't think their charts went up as planned.

imho we should helm [registry] upgrade --install which will succeed either way.

because we are running tiller in a kraken namespace we should always do as @joejulian commented.


View background

In my app, I set the background of white ,but the Page allways hava a little gray color between tow pages,
how can I to cancel that

Hi @AttemptD ,
Can you share an image or some code, please? So if you go to SimpleExample and remove lines 28 and 40 (background and shadow) you'll get this:

As you can see, there's no greyish tone there.

thank you very much

@fermoya thank very much

I'll close this then


Clickable tiles

I believe would be great to be able to click on configured tiles. Other than going to external URLs for more insight, I was thinking to create a main dashboard.json with different groups, and clicking on a group would bring me to the specific group1.json with single tiles.
I couldn't find anything to suggest this feature was recommended before.

Hey!
Since Monitoror first goal is to be a wallboard and not a dashboard, this will not be included in this project.
On the other hand, we plan to create something like "Monitoror PRO" to monetize the project with some advanced features like this one. I need to take time to create a dedicated issue for this, but I'm on vacation, and we want to work on improving onboarding first.
I will spoil a little bit: what I call Monitoror PRO will be a way to get Monitoror as a browser extension or PWA, more like a dashboard, with clickable tiles, etc. Like you said. But hush, it's a secret! :D

+1 as looking for the similar feature where clicking on a red tile monitoring a group of servers (from few to hundreds) could provide the clue which underlying server/alias/label (in order of preference) is failing the ping/port/http tests. FYI I'm used the monitorwall for scanning of runtime acceptance testing endpoint baked into those VMs.
I moved to vigil recently due to this (and alerting) limitation but would like to still use monitoror for the projects where the entire info could be confined to a single screen with the need of alerting as well.


Add links to reddit, facebook and xbox

No idea how battle.net works

Thanks for the PR!
Links to Facebook will not be presented on the website (just because).
Reddit links have just been fixed in this commit, except for Xbox, which Discord does not offically support linking to.
If you have any further fixes, feel free to reopen.


NDRS-1061: Remove ProtoBlock from FinalizedBlock and rename wasm_deploys.

This is a refactoring, and a preparation for simplifying the block types, specifically for replacing ProtoBlock and CandidateBlock with a single type.
This PR removes ProtoBlock from FinalizedBlock, because the new type will contain fields (accusations) that would be redundant as a part of FinalizedBlock. It also renames ProtoBlock::wasm_deploys to deploy_hashes, and transfers to transfer_hashes, so that it is in line with BlockBody.
https://casperlabs.atlassian.net/browse/NDRS-1061

bors r+


Internal: Inspect if hack to insert istanbul ignore comments is still needed

The hack is pretty slow.
https://github.com/SAP/chevrotain/blob/master/gruntfile.js#L197
And maybe it is no longer needed if the TypeScript compiler stopped
swallowing certain comments and disappearing them from the output...

Tested this in an if/else
The typescript compiler still swallows those comments...


Incorrect Typescript type for QueryEvent.timestamp

Bug description
QueryEvent.timestamp's Typescript type is Date, but it is actually a number
How to reproduce
Check the type of QueryEvent.timestamp; see that it is Date
Log a query event, e.g.
prismaClient.$on('query', event => {
  console.log(event);
};

and see that the timestamp field is actually a number
Expected behavior
Either QueryEvent.timestamp should have its type set to number, or it should contain a Date not a number
Prisma information
n/a
Environment & setup

OS: Windows, Linux
Database: PostgreSQL
Node.js version: 16.8.0

Prisma Version
prisma                  : 3.2.1
@prisma/client          : 3.2.1
Current platform        : windows
Query Engine (Node-API) : libquery-engine b71d8cb16c4ddc7e3e9821f42fd09b0f82d7934c (at<EMAIL_ADDRESS>Migration Engine        : migration-engine-cli b71d8cb16c4ddc7e3e9821f42fd09b0f82d7934c (at<EMAIL_ADDRESS>Introspection Engine    : introspection-core b71d8cb16c4ddc7e3e9821f42fd09b0f82d7934c (at<EMAIL_ADDRESS>Format Binary           : prisma-fmt b71d8cb16c4ddc7e3e9821f42fd09b0f82d7934c (at<EMAIL_ADDRESS>Default Engines Hash    : b71d8cb16c4ddc7e3e9821f42fd09b0f82d7934c
Studio                  : 0.435.0
Preview Features        : interactiveTransactions

Thanks for the report @Ezard, I can confirm this issue.
I would say that the static type is correct, the timestamps in the events have been (and still are) Dates in the old BinaryEngine, but the new LibraryEngine introduced a regression where they are either a number or an undefined at run time instead, and target is sometimes undefined instead of string:
{
  timestamp: undefined,
  message: 'Starting a postgresql pool with 17 connections.',
  target: undefined
}
{
  timestamp:<PHONE_NUMBER>556,
  query: 'SELECT "public"."Post"."id", "public"."Post"."title", "public"."Post"."userId" FROM "public"."Post" WHERE 1=1 OFFSET $1',
  params: '[0]',
  duration: '4',
  target: 'quaint::connector::metrics'
}

This happens because the LibraryEngine incorrectly uses Date.now() instead of new Date() for the timestamps of the query events: https://github.com/prisma/prisma/blob/80d0e7bf3be325a038d0c3d68c51d7934cdd8286/packages/engine-core/src/library/LibraryEngine.ts#L246 and doesn't set the timestamp and target at all for other log events: https://github.com/prisma/prisma/blob/80d0e7bf3be325a038d0c3d68c51d7934cdd8286/packages/engine-core/src/library/LibraryEngine.ts#L261
However, there's a separate issue with the BinaryEngine as well. Even though the runtime type matches the static type there, the Date objects themselves are malformed:
{
  timestamp: Invalid Date,
  message: 'Starting a postgresql pool with 17 connections.',
  target: 'quaint::pooled'
}
{
  timestamp: Invalid Date,
  message: 'Started http server on http://<IP_ADDRESS>:54144',
  target: 'query_engine::server'
}
{
  timestamp: Invalid Date,
  query: 'SELECT "public"."Post"."id", "public"."Post"."title", "public"."Post"."userId" FROM "public"."Post" WHERE 1=1 OFFSET $1',
  params: '[0]',
  duration: 14,
  target: 'quaint::connector::metrics'
}

This issue is caused by the BinaryEngine logger prepending the current year for some reason to the ISO date string before passing it to the Date constructor: https://github.com/prisma/prisma/blob/bbdf1c23653a77b0b5bf7d62efd243dcebea018b/packages/engine-core/src/common/errors/utils/log.ts#L122

^ Moved back to doing since the PR is not merged yet.


Dental infection: Copyedits and image replace

Minor formatting changes
Will need copyright-free images to replace these with
Changed author to Hans

Codecov Report

Merging #77 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #77   +/-   ##
=======================================
  Coverage   50.32%   50.32%
=======================================
  Files          14       14
  Lines         155      155
  Branches       43       43
=======================================
  Hits           78       78
  Misses         77       77

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09a53c0...963c4ba. Read the comment docs.


Language theme selection doesn't match properly if you are using dark theme on Android

Steps to Reproduce
Open Bluesky
Go to the Settings >> Languages
Open App Language or Primary Language and if you use a dark theme you'll see the the box is white instead of the black and it's uses the old Android box instead of the new Android box
Attachments

What platform(s) does this occur on?
Android
Device Info
Realme c55
What version of the app are you using?
Build version: <IP_ADDRESS>1; Bundle info: 41243e874 (prod); Bundle date: 24120701; Platform: android; Platform version: 34
Additional Information
No response

I'm not entirely clear if the react-native-picker-select package supports dark mode on Android? @mozzius said here:

doesn't affect android because that prop is iOS only (and that's fine)

it does not


Create palindrome number explanation

Added an example to the existing palindrome number code so that it is easy to understand.

@Thanmaia please add particular extension of the file and raise a pr again :)

Thankyou for remembering @dixitt5 ..As it is my first commit and because of lack of experience I forget to add extension. I have done it now and raised a new pr

@Thanmaia no worries my friend , I'm not able to see your new pr again can you specify what name does it have ?

@dixitt5 Create palindromenumexample.c is the new name..

@Thanmaia  this pr is not showing the change , better if you create a new PR

Okay got it ;)


Automatic(?) translation of object ID to object Name

Describe the bug
Possible error in automatic translation
bmw.0.WBAxxxxxxxxx.properties.vehicleLocation.heading is translated to 'Überschrift'.
Expected behavior
Should be 'Richtung'
Versions:

Adapter version: 2.5.1

fixed


Sentry breaks custom protocols because it calls registerSchemesAsPrivileged which can be called only once

Is there an existing issue for this?

[X] I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
[X] I have reviewed the documentation https://docs.sentry.io/
[X] I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?
Sentry Saas (sentry.io)
Electron SDK Version
4.4.0
Electron Version
24.0.0
What platform are you using?
MacOS
Link to Sentry event
No response
Steps to Reproduce
Create any custom protocol:
  protocol.registerBufferProtocol("foobar", (request, callback) => {
    return callback({ data: Buffer.from("Hello World") });
  });

Mark it as priviliged so it supports fetch API
protocol.registerSchemesAsPrivileged([
    {
      scheme: "foobar",
      privileges: { bypassCSP: true, corsEnabled: true, supportFetchAPI: true },
    },
  ]);

Initialize sentry main module (and renderer module)
import * as Sentry from "@sentry/electron/main";

export function initSentryMain() {
  console.info("Initializing Sentry");

  Sentry.init(sentryConfig);
}

Expected Result
I am able to call my custom protocol with fetch API on renderer eg. fetch("foobar://some-path");
Actual Result
After 2 totally ruined days of debugging this; having no clue it can be caused by sentry, I finally found that the issue happens because in main/ipc.ts of Sentry you do:
/** Enables Electron protocol handling */
function configureProtocol(options: ElectronMainOptionsInternal): void {
  if (app.isReady()) {
    throw new SentryError("Sentry SDK should be initialized before the Electron app 'ready' event is fired");
  }

  protocol.registerSchemesAsPrivileged([
    {
      scheme: PROTOCOL_SCHEME,
      privileges: { bypassCSP: true, corsEnabled: true, supportFetchAPI: true },
    },
  ]);

Which is incorrect. As Electron docs says https://www.electronjs.org/docs/latest/api/protocol#protocolregisterschemesasprivilegedcustomschemes

Note: This method can only be used before the ready event of the app module gets emitted and can be called only once.

As a result - you're silently overwriting previously defined schemes with your own, without warning.
It results in
Fetch API cannot load foobar://fooo. URL scheme "foobar" is not supported.

What makes it even more confusing is that in my case (and I imagine most of the cases) Sentry is only enabled in production, which made my custom protocols work just fine in dev.
Please fix it and please be careful when you use code that overwrites settings defined by the app. You really ruined a lot of time and caused a lot of stress and hair-pulling with this one.

Workaround:
To workaround the issue and keep sentry working, AFTER initializing sentry, call
export function registerPriviligedProtocols() {
  protocol.registerSchemesAsPrivileged([
    {
      scheme: "sentry-ipc",
      privileges: { bypassCSP: true, corsEnabled: true, supportFetchAPI: true },
    },
    // <<--- REST OF MY CUSTOM SCHEMES HERE
  ]);
}

to register both sentry protocols and our own protocols.

Sorry, I didn't notice that registerSchemesAsPrivileged could only be called once and silently allowed it to be called more times!
Probably the easiest way to work around this until we come up with a better solution is to use IPC via preload injection rather than a protocol scheme. You can disable protocol setup by configuring ipcMode: IPCMode.Classic:
const { init, IPCMode } = require("@sentry/electron");

init({
  dsn: "__DSN__",
  ipcMode: IPCMode.Classic,
});

I'm still not sure what to do to improve this since Electron doesn't supply a means to test if registerSchemesAsPrivileged has already been called.
The only mild improvement I can think of (which I'd still like to avoid) would be to patch registerSchemesAsPrivileged when the Sentry library is imported (ie. a side effect) and display a warning if it gets called by SDK consumers.

I encountered the same problem：

I need to register my own custom protocol

And when I use IPCMode.classic, I found that sentry initialized in the rendering process is still fetch "sentry-ipc:" protocol

@Wonder233 have you tried using the above workaround at the bottom of the original post?


Changed regex for userid to allow negative IDs

django-guardian needs a anonymous user, which has a user-id of -1 in the suggested default config:

https://github.com/django-guardian/django-guardian#configuration

This patch adjusts the regex in the urls.py to allow for this negative userid. Without this patch rendering the userlist in the admin fails.

@cato- we will adapt the tests and merge afterwards. Thanks for your RP @cato-


Not much of an issue, more of a question

Hello! I have been messing around with Nadekobot, and I am just wondering: what is this embedded field called? There are fields like CurrentQuestion and QuestionEmbed, not sure which it is. Thanks!

If it's more of a question. Why not ask it in #help in the Nadekolog server?

@Macley-Kun Hi there! I have looked all over the net for a Nadeko Discord Invite, all of them seem to be expired. I would greatly appreciate it if you could send me one though.

https://discord.gg/nadekobot

@Macley-Kun apparently he's not allowed to

With allowed you mean. He didn't do: .iam modder?

@Macley-Kun So he mentioned he was making a bot, they replied and I quote:

we don't offer support for other bots or even making bots


[3.x] Update the visibility logic for the spatial editor navigation controls

Fixes https://github.com/godotengine/godot/issues/69845
main version

Thanks!


Added watsonmarlow323 device for i22 ppump.

Added Watson Marlow 323 pump device for use on I22 - draft PR for concept review.
This device depends on an updated support module interface (support/watson-marlow) that was made more consistent while making this change - release TBC.
Instructions to reviewer on how to test:

TBC
Confirm that device is operable

Checks for reviewer

[ ] Would the PR title make sense to a scientist on a set of release notes
[ ] If a new device has been added does it follow the standards
[ ] If changing the API for a pre-existing device, ensure that any beamlines using this device have updated their Bluesky plans accordingly
[ ] Have the connection tests for the relevant beamline(s) been run via dodal connect ${BEAMLINE}

Thanks @callumforrester I've incorporated your suggested changes, rebased, and linted. Also, made enabled a readable signal based on @DiamondJoseph's comments following chat on Wed.
I just need to test this with an actual device to confirm it communicates correctly after which I'll undraft and submit.

@barnettwilliam Do you still plan to merge this? Since you have addressed Callum's comments and Joseph has approved this you are free to dismiss Callum's review (or you could ask him to take a look again)


Makefile error: setup-packer-configs-ami

What happened: Trying to create a new Ubuntu 20.04 (w/ EKS-A 1.24) AMI with image-builder and encountered [Makefile:207: setup-packer-configs-ami] Error 141
What you expected to happen: An AMI to be produced successfully
How to reproduce it (as minimally and precisely as possible): Follow the instructions in EKS-A documentation, get to the image-builder section and encounter this error. Others are noticing this as well. Can be reproduced using image-builder build --os ubuntu --hypervisor ami --release-channel 1-24 with or without an AMI config and any release channel.
Anything else we need to know?:
Environment:

EKS Anywhere Release: Any
EKS Distro Release: Any

Logs:
image-builder-2.log

@matthenslee this should be fixed now. Would you mind retesting?

Apologies for the delay @raykrueger - this fixed my issue. Thank you!


Document Windows-dependencies

I've managed to get it running by using win-builds and downloading every package, but @5k3105 was unable to do so with TDM.
We should document that it works with the win-builds package (which include a lot, so I don't know which was the "magic fix").

Currently documented at engo.io/00-foreword.
We may want to add it to our README.md as well.


feat(docs): document DEFAULT_LISTEN_ADDR constant

This docs update was missing in the #2479 that added option for configurable
listening address.
Thanks!

:x: Build karma 2518 failed (commit https://github.com/karma-runner/karma/commit/a373a48dbf by @peterblazejewicz)


I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: No such file or directory

Description
I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: No such file or directory
How to reproduce
Java app running well, after Java app dockerization, thrown error.
What do you expect
Dockerized version running well.
What happened instead
final DockerClient dockerClient = DefaultDockerClient.fromEnv().build();
I wonder how can I set the "fromEnv()" method's  external  docker env information as parameter?
Software:

docker version: Docker version 17.06.0-ce, build 02c1d87
Spotify's docker-client version: 8.8.0

Full backtrace
Retrying request to {}->unix://localhost:80
uuid-accredit_1  | 2017-07-23 01:28:14.719 ERROR 1 --- [nio-8095-exec-9] o.a.c.c.C.[.[.[.[dispatcherServlet]      : Servlet.service() for servlet [dispatcherServlet] in context with path [/accredit] threw exception [Request processing failed; nested exception is com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: No such file or directory] with root cause
uuid-accredit_1  |
uuid-accredit_1  | java.io.IOException: No such file or directory
uuid-accredit_1  | 	at jnr.unixsocket.UnixSocketChannel.doConnect(UnixSocketChannel.java:94)
uuid-accredit_1  | 	at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:102)
uuid-accredit_1  | 	at com.spotify.docker.client.ApacheUnixSocket.connect(ApacheUnixSocket.java:76)
uuid-accredit_1  | 	at com.spotify.docker.client.UnixConnectionSocketFactory.connectSocket(UnixConnectionSocketFactory.java:78)
uuid-accredit_1  | 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
uuid-accredit_1  | 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
uuid-accredit_1  | 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
uuid-accredit_1  | 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
uuid-accredit_1  | 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
uuid-accredit_1  | 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
uuid-accredit_1  | 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
uuid-accredit_1  | 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
uuid-accredit_1  | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
uuid-accredit_1  | 	at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
uuid-accredit_1  | 	at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
uuid-accredit_1  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
uuid-accredit_1  | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
uuid-accredit_1  | 	at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
uuid-accredit_1  | 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
uuid-accredit_1  | 	at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
uuid-accredit_1  | 	at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
uuid-accredit_1  | 	at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
uuid-accredit_1  | 	at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
uuid-accredit_1  | 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
uuid-accredit_1  | 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
uuid-accredit_1  | 	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
uuid-accredit_1  | 	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
uuid-accredit_1  | 	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
uuid-accredit_1  | 	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
uuid-accredit_1  | 	at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
uuid-accredit_1  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
uuid-accredit_1  | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
uuid-accredit_1  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
uuid-accredit_1  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
uuid-accredit_1  | 	at java.lang.Thread.run(Thread.java:745)

It seems you are asking how you can configure a DockerClient instance with the details of your connection. If that is what you are asking, please see the relevant section of the User Manual on Creating a Docker Client.

@johnflavin  I try to change the configure to
final DockerClient docker = new DefaultDockerClient("unix:///var/run/docker.sock");
after that dockerized java app again ,then inside docker-compose running  still the same error, does any ip related configure?

Let me see if I can figure out what you are trying to do.
You have some java app that uses docker-client to do something with docker. You tested it on your machine, and it works fine. Then you put that java app into a docker image and now it doesn't work. Is that correct?
If that is correct, then it seems the problem you are seeing is from your app trying to control docker, but it can't because it is inside a container. Have you tried mounting the docker socket into the container? That would let your app control docker again.

@johnflavin I must to say it is a "docker  in docker-compose" mode, and using "-v /var/run/docker.sock:/var/run/docker.sock " to fix up my problem, and thanks for your help again.

@yangboz Hi bude. I have been disturbed by same problem, but I don`t understand your way to solve this problem. Is it mean "DockerClient docker = new DefaultDockerClient("-v /var/run/docker.sock:/var/run/docker.sock");"

@yangboz Hi, buddy. I have been distrubed by same problem, but I can`nt understand your way to solve this question. Is it mean to set DOCKER_HOST value is "-v /var/run/docker.sock:/var/run/docker.sock " ?
DockerClient docker = new DefaultDockerClient("-v /var/run/docker.sock:/var/run/docker.sock");
It like blocked

@yangboz Hi, buddy. I have been distrubed by same problem, but I can`nt understand your way to solve this question. Is it mean to set DOCKER_HOST value is "-v /var/run/docker.sock:/var/run/docker.sock " ?
DockerClient docker = new DefaultDockerClient("-v /var/run/docker.sock:/var/run/docker.sock");
It can`nt work.

@pickmonster in MacOS,  the  "var/run/docker.sock "  is default value. and I am using Spotify's docker-client API.

Im having the same issue when I deploy my Java app inside of a docker container.
"-v /var/run/docker.sock:/var/run/docker.sock" does not work because I have a windows host system and the container is linux (alpine).
Whats the best way to create a docker client that connects to the docker deamon on the host?


Fixed XSS in JPath expression operation

This pull should fix issue #1318 - I can confirm that it mitigates the exploit that po6ix provided but I don't have enough experience in JPath to confirm that it protects from all future variations of this.
A more in-depth explanation of this mitigation is available here.

Thank you very much for this fix. I had a play around with it and ultimately decided to switch to a different jsonpath library instead. We are now using jsonpath-plus which appears to be well supported. The reason I've gone for this route rather than merging your solution is because I feel that having the ability to evaluate expressions is an important part of JPath. I have added an argument to turn them on or off and the POC from #1318 does not result in XSS in either case since the evaluation environment is more locked down in this new library. If the vulnerability remains, I will probably resort to turning off expression evaluation entirely, which would be a shame. Thanks again for your efforts.


Capture certificate DN etc from NGINX as meta data attribute on data receipt if available

We need to sort out capturing certificate DN as a data meta attribute in proxy and stroom for NGINX fronted receipt.
Version 5 included the following attributes from the certificate:
RemoteDN
RemoteCertExpiry
Example attributes from v5:
content-type:application/x-www-form-urlencoded
Environment:DEV
Feed:FEED_NAME
GUID:xxx-xxx-xxx
host:xxx.ddd.fff.com
ReceivedTime:2018-11-11T12:12:12.000Z
RemoteAddress:<IP_ADDRESS>
RemoteHost:xxx.ddd.xxx.xom (edited)
RemoteCertExpiry:2019-12-12T11:11:11.000Z
RemoteDN:CN=fred, OU=Plop, O=Ploop, C=UK
StreamSize:232344
System:FRED
user-agent:beefy_1.0

It sounds as if NGINX can be configured to add the appropriate HTTP headers (RemoteCertExpiry and RemoteDN) when NGINX is doing SSL termination. We also need to figure out how to add these if DW is used directly (see gh-940).

It's easy to configure NGINX to pass on SSL stuff as headers -- this is done using proxy_set_header. Our default stroom-resources NGINX config isn't adding the DN for datafeed but we do it elsewhere, for example:
location /authenticationService/ {
    ...
   proxy_set_header X-SSL-CLIENT-S-DN $ssl_client_s_dn;
   ...
 From the docs:

$ssl_client_escaped_cert
returns the client certificate in the PEM format (urlencoded) for an established SSL connection (1.13.5);
$ssl_client_v_end
returns the end date of the client certificate (1.11.7);
$ssl_client_v_remain
returns the number of days until the client certificate expires (1.11.7);
$ssl_client_v_start
returns the start date of the client certificate (1.11.7);

Capturing these details if using DW without NGINX requires that Jetty be configured to use: SecureRequestCustomizer. It appears that DW is configured to use SecureRequestCustomizer by default in io.dropwizard.jetty.HttpsConnectorFactory line 567.
If this is correct it should be possible to retrieve the necessary attributes with some code like this:
        final X509Certificate[] certs = (X509Certificate[]) request.getAttribute("javax.servlet.request.X509Certificate");

        if (certs!=null && certs.length > 0 && certs[0]!=null)  {
            final X509Certificate cert = certs[0];
            final String remoteDN = cert.getIssuerDN().toString();
            final String remoteCertExpiry = DateUtil.createNormalDateTimeString(cert.getNotAfter().getTime());

Changed nginx config to put the remoteDN and cert expiry in the headers (and stroom code has been changed to rename/format the expiry date) .
Example meta output below.
Content-Type:application/x-www-form-urlencoded
Environment:DEV
Feed:TEST_FEED
GUID:76f6adda-753a-4984-ab11-093f3b102d34
Host:stroom_proxy_upstream
ReceivedTime:2018-11-27T11:56:36.429Z
RemoteAddress:<IP_ADDRESS>
RemoteCertExpiry:2292-09-10T06:39:20.000Z
RemoteDN:CN=A Stroom Client,O=GCHQ,L=Cheltenham,C=UK
RemoteHost:<IP_ADDRESS>
StreamSize:29
System:MY_SYSTEM
User-Agent:curl/7.47.0
X-Forwarded-For:<IP_ADDRESS>
X-SSL-Client-V-End:Sep 10 06:39:20 2292 GMT
X-SSL-Client-V-Start:Nov 27 06:39:20 2018 GMT
X-SSL-Client-Verify:SUCCESS
X-SSL-Protocol:TLSv1.2


Update community_lessons.json

Proposed:
http://hpc-carpentry.org/hpc-intro/
which will auto redirect to https://carpentries-incubator.github.io/hpc-intro/
because of hpc-carpentry.org/pages/hpc-intro.md
Current 404:
http://hpc-carpentry.github.io/hpc-intro/

Thanks for this, it will also (I think) get magically fixed if we implement #8

I'd leave this open for now until we do that, and then I would update all the links at once

@annajiat I updated this to update all the other links as well since I renamed the repo, let me know that you are ok with it.

Thanks @annajiat for checking this


Klusta operation Error in Windows10

Hi, I had a problem with using klusta
As you guided in 'Installation Issue #33', I installed klusta with the 'klustaviewa-setup.exe' file.
And I get an error like below
C:\Program Files\KlustaViewa\WinPython-64bit-<IP_ADDRESS>\python-2.7.6.amd64\Scripts>klusta.exe 0526.prm
11:59:55 [I] creator:446          Using kwik file C:\Program Files\KlustaViewa\WinPython-64bit-<IP_ADDRESS>\python-2.7.6.amd64\Scripts\0526.kwik.
11:59:55 [I] launch:206           Starting spike detection.
11:59:55 [I] launch:53            SpikeDetekt parameters:
11:59:55 [I] launch:55            threshold_strong_std_factor = 4.5
11:59:55 [I] launch:55            use_single_threshold = 1
11:59:55 [I] launch:55            filter_low = 500.0
11:59:55 [I] launch:55            dtype = int16
11:59:55 [I] launch:55            threshold_weak_std_factor = 2.0
11:59:55 [I] launch:55            n_channels = 16
11:59:55 [I] launch:55            filter_lfp_low = 0
11:59:55 [I] launch:55            raw_data_files = 0526.dat
11:59:55 [I] launch:55            chunk_size_seconds = 1
11:59:55 [I] launch:55            filter_high_factor = 0.475
11:59:55 [I] launch:55            prb_file = 0526.prb
11:59:55 [I] launch:55            voltage_gain = 10.0
11:59:55 [I] launch:55            extract_s_after = 16
11:59:55 [I] launch:55            chunk_overlap_seconds = 0.015
11:59:55 [I] launch:55            n_excerpts = 50
11:59:55 [I] launch:55            filter_butter_order = 3
11:59:55 [I] launch:55            weight_power = 2
11:59:55 [I] launch:55            excerpt_size_seconds = 1
11:59:55 [I] launch:55            connected_component_join_size = 1
11:59:55 [I] launch:55            extract_s_before = 16
11:59:55 [I] launch:55            experiment_name = 0526.kwik
11:59:55 [I] launch:55            sample_rate = 30000
11:59:55 [I] launch:55            n_features_per_channel = 3
11:59:55 [I] launch:55            detect_spikes = negative
11:59:55 [I] launch:55            pca_n_waveforms_max = 10000
11:59:55 [I] launch:55            filter_lfp_high = 300
11:59:55 [I] spikedetekt:212      Found 4 live channels in group 0.
11:59:55 [I] spikedetekt:212      Found 4 live channels in group 1.
11:59:55 [I] spikedetekt:212      Found 4 live channels in group 2.
11:59:55 [I] spikedetekt:212      Found 4 live channels in group 3.
11:59:55 [I] spikedetekt:285      Finding the thresholds...
Detecting spikes        : 100%|##########| 204/204 [00:43<00:00,  4.71it/s]
Extracting waveforms    : 100%|##########| 204/204 [00:11<00:00, 17.35it/s]
Performing PCA          : 100%|##########| 4/4 [00:00<00:00, 114.29it/s]
Computing features      : 100%|##########| 204/204 [00:18<00:00,  8.73it/s]
12:01:12 [I] launch:218           Spike detection done! 1123 spikes found in shank 0, 1016 spikes found in shank 1, 12448 spikes found in shank 2, 2127 spikes found in shank 3
12:01:12 [I] launch:249           Starting clustering on shank 0/4.
12:01:12 [I] launch:252           Clustering group 0 (1123 spikes).
12:01:12 [I] launch:123           Starting KK...
ImportError: No module named klustakwik2
C:\Program Files\KlustaViewa\WinPython-64bit-<IP_ADDRESS>\python-2.7.6.amd64\Scripts>
I use Window 10, 64bit.
Can you check this problem?

How did you install the package? Using the all-in-one installer for Windows? If so, could you download it again and reinstall it? I just added klustakwik2 in it.


Change deprecated SessionComponent::setFlash() to FlashComponent on branch 3.0

SessionComponent::setFlash() is deprecated. You should use Flash instead.
I have replaced all $this->Flash->set() calls to one of:
$this->Flash->success()
$this->Flash->warning()
depending on the context they are called on (operation done or exception).
I also updated the tests.
Reference:
https://book.cakephp.org/3.0/en/appendices/3-0-migration-guide.html#sessioncomponent

Hi @rogerpro please fix the build to merge it

As you can see (11 commits) I've been trying & researching the origin of the failing checks.
I understand that the plugin must be tested with many PHP versions, CakePHP versions and databases, and that it must follow coding standards. The only issue it that I'm not familiar with the .travis.yml file. I read https://github.com/FriendsOfCake/travis/blob/master/README.markdown searching for tips but checks are not passing yet. Thats why I'm now asking for help.
Possibly you'll need to undo last commits. But actually the file .travis.yml already needed some update, like including 7.0 & 7.1 PHP and CakePHP 3.4 versions. When having a proper .travis.yml file, I hope that closing this PR will be a snap.


Pass service instance name via ProvisionRequest context

This PR is a

[x] Feature Implementation
[ ] Bug Fix
[ ] Documentation

What this PR does / why we need it:
Pass service instance name via ProvisionRequest context
Which issue(s) this PR fixes

Fixes #2532
Please leave this checklist in the PR comment so that maintainers can ensure a good PR.
Merge Checklist:

[ ] New feature

[ ] Tests
[ ] Documentation

[ ] SVCat CLI flag
[ ] Server Flag for config

[ ] Chart changes
[ ] removing a flag by marking deprecated and hiding to avoid
breaking the chart release and existing clients who provide a
flag that will get an error when they try to update

looks like you needed to add this bit to some of the expected results in the integration tests?

/retest

@jberkhahn not sure why the tests were failing before. just ran them again and they passed

/lgtm
/approve


Users get 403 error when they login for first time and don't have permissions setup

This is a bad user experience. they should see a page telling them to email their region to get permissions.
Better yet, we should have invite codes....

pull request #253


Training Time Discrepancies

Hi @farewellthree,
Thank you for sharing your work. I've been trying to training your STLLM models using the configurations provided - specifically the following two: instructblipbase_stllm_conversation.yaml and instructblipbase_stllm_qa.yaml on 8 A100 GPUs. I observed the training time for the conversation model to be around 16 hours, whereas the paper suggests approximately "6 hours for 2 epochs using Deepspeed's zero-2 setting".
Here is the command I use to initiate training:
deepspeed --master_port=20000 --include=localhost:0,1,2,3,4,5,6,7
I'm looking for a clarification on whether there might be any configuration adjustments (e.g., batch size, optimizer settings) that could help align the training time more closely with what is in the paper. Additionally, could you provide the expected training duration for tboth configurations?
Any suggestions to improve training efficiency would be greatly appreciated. Thank you.

Sorry for the confusion. The training time mentioned in the paper is for the qa_config, and training with the conversation_config does take considerably more time. However, if you exclude the caption_webvid data, you can save a significant amount of time with minimal performance loss.


Using Elasticsearch 7.17.3, GMS generates a lot of log entries "[ignore_throttled] parameter is deprecated"

Describe the bug
I'm running ElasticSearch 7.17.3, as specified in the datahub helm charts. This works, but GMS produces a lot of WARNINGS in the logs:
org.opensearch.client.RestClient:85 - request [POST http://elasticsearch:9200/datahubingestionsourceindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true] returned 1 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
Apparently it's recommended to update the ElasticSearch / Opensearch java clients to remove these warnings. I've tried updating the opensearch client to the latest version (2.11.1), but unfortunately this has breaking changes and doesn't compile.
It doesn't happen when running datahub docker quickstart, as the ES version in the quickstart docker compose files is still on 7.10.1
To Reproduce
Steps to reproduce the behavior:

Set DATAHUB_SEARCH_TAG to 7.17.3
Start Datahub using Docker quickstart
Observe GMS logs

Expected behavior
No deprecation warnings in logs
Additional info
I'm more than happy to contribute to a fix. However I'm missing a bit of context on the implementation, and as stated this wasn't a "zero-touch" upgrade unfortunately.
If someone could give me some pointers, I'd be happy to try and find a fix.

In the Elasticsearch repository there is a pull request which prevents that the ignore_throttled parameter is added to the search request, when it is set to the default value of true (https://github.com/elastic/elasticsearch/pull/84827). However DataHub is using the OpenSearch client (see here= and unfortunately there isn't such a logic which prevents the parameter to be added to the search request (see here). I am not even sure whether you can use Elasticsearch 7.17.3 with the OpenSearch client, because it seems the OpenSearch client is only guaranteed to be compatible up to version 7.10.2 of Elasticsearch (see here). It also seems to be that frozen tiers are not deprecated in OpenSearch...? Maybe the Helm chart should also have an OpenSearch deployment instead of Elasticsearch if the OpenSearch client is used?
Nonetheless a quick, but maybe not nice solution would be to filter the corresponding log message using the logback.xml here:

https://github.com/datahub-project/datahub/blob/b0163c4885f85b4cb78d0ca471b3dff3242d2a3a/metadata-jobs/mae-consumer-job/src/main/resources/logback.xml#L8-L10
https://github.com/datahub-project/datahub/blob/b0163c4885f85b4cb78d0ca471b3dff3242d2a3a/metadata-jobs/mce-consumer-job/src/main/resources/logback.xml#L8-L10
https://github.com/datahub-project/datahub/blob/b0163c4885f85b4cb78d0ca471b3dff3242d2a3a/metadata-service/war/src/main/resources/logback.xml#L11-L13
https://github.com/datahub-project/datahub/blob/b0163c4885f85b4cb78d0ca471b3dff3242d2a3a/datahub-upgrade/src/main/resources/logback.xml#L11-L13

<filter class="com.linkedin.metadata.utils.log.LogMessageFilter">
	<excluded>scanned from multiple locations</excluded>
	<excluded>[ignore_throttled] parameter is deprecated because frozen indices have been deprecated</excluded>
</filter>


What version of lua does loadcaffe use?

I'm on a Mac and brew install lua which is currently on version 5.2.4_3. When I luarocks install loadcaffe or luarocks search loadcaffe nothing comes up. This makes me think that loadcaffe works with an older version..? If so, which one should I use?

remove lua and install torch http://torch.ch

Ohhhhhh! Thanks for the quick reply. I'll do that instead.

Worked like a charm. Thanks again for the quick response!


Remove apache 2 licence

Needed in order to comply with the LF energy guidelines and to prevent license confusion.

closes #228

Does not pass the tests.


feat: Accept URL escape for device command name and resource name

Use router.UseEncodedPath() to match the encoded original path to the routes
Decode the Url path variable before passing to the controller

Close #4371

If your build fails  due to your commit message not passing the build checks, please review the guidelines here: https://github.com/edgexfoundry/edgex-go/blob/main/.github/Contributing.md
PR Checklist
Please check if your PR fulfills the following requirements:

[x] I am not introducing a breaking change (if you are, flag in conventional commit message with BREAKING CHANGE: describing the break)
[x] I am not introducing a new dependency (add notes below if you are)
[ ] I have added unit tests for the new feature or bug fix (if not, why?)
[ ] I have fully tested (add details below) this the new feature or bug fix (if not, why?)
[ ] I have opened a PR for the related docs change (if not, why?) not impact

Testing Instructions

New Dependency Instructions (If applicable)

Codecov Report

Merging #4376 (44004a9) into main (76c26b6) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main    #4376   +/-   ##
=======================================
  Coverage   43.71%   43.71%
=======================================
  Files         116      116
  Lines       10713    10713
=======================================
  Hits         4683     4683
  Misses       5610     5610
  Partials      420      420

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Is there any ETA for custom background feature?

I've already posted this question over here:
https://github.com/twilio/twilio-video-app-react/issues/453
But not sure how to implement it with VideoProcessor API.
https://github.com/twilio/twilio-video.js/issues/1413
Is anyone have implemented the custom background feature or can someone pls guide me on how to implement it?

Hi @SanjayBikhchandani - thanks for the question!
We don't have a set release date for this feature, but it is in active development. We recently released the video-processor library, which can be used to add virtual backgrounds to video tracks: https://www.twilio.com/blog/introducing-virtual-backgrounds-browser-based-video-applications
Soon, we will add a user interface to this app that will let users choose a virtual background. Stay tuned!

Hi @SanjayBikhchandani - just wanted to let you know that our virtual background feature has been released in version 0.5.0:
https://github.com/twilio/twilio-video-app-react/releases/tag/v0.5.0
Enjoy! Please let us know if you have any questions about it.


Check signers are within the last config block

#133
Implement a new block validation policy: "ImplicitOrderer SMARTBFT"


fix(k8sprocessor): fix metadata dependencies

Fixes #512

One comment w.r.t to the tests is that I'd maybe be more generous with comments, i.e. add comments where we're testing this particular feature that is added in this PR so that we know what's being tested.

ping

One comment w.r.t to the tests is that I'd maybe be more generous with comments, i.e. add comments where we're testing this particular feature that is added in this PR so that we know what's being tested.

ping

Here's how I see it: there was a functionality  that was not covered by tests (retrieving metadata for deployments and cronjobs), and this functionality didn't work correctly. I fixed the bug and added the tests for this functionality at the same time. I don't think there is anything else required with regards to this bug.


Nordic PCA10059 support without additional Segger debugger

A collection of pull requests (and issues) related to PCA10059 (Dongle) suport:

[ ] DNM Add nRF52840 USBD driver ( #8787 )
[ ] MCUboot nrf52840_pca10059 support ( https://github.com/runtimeco/mcuboot/pull/318 )
[x] Add add nrf52840_pca10059 board support ( #8812 )

/cc @lemrey @nvlsianpu

@larsgk thanks for creating this.

cc @jhedberg

https://github.com/pawelzadrozniak/zephyr/pull/1 to be merged to #8787 .

I'm stuck with a few of these dongles that I want to flash with zephyr, but I don't have an external debugger (yet). If I understand it correctly, once this is done, I can flash without an additional debugger but I would still need one in the first place to place mcu_boot on the target, right? Using DFU-flash is not sufficient?

@postlund there is an issue with the current USB driver that @pawelzadrozniak. I think then that @lemrey needs to update the documentation on the board in order to describe how to use Zephyr without an external debugger.

Perfect, thank you!

@postlund @larsgk can you retry with current master? a fix was just merged https://github.com/zephyrproject-rtos/zephyr/pull/10934

@lemrey it looks like all the features are in place for this now. We just need to document how to transition from the built-in bootloader to MCUboot in order for people to be able to use this without a debugger.

I can try when I get home
On Tue, Oct 30, 2018, 18:24 Carles Cufí<EMAIL_ADDRESS>wrote:

@lemrey https://github.com/lemrey it looks like all the features are in
place for this now. We just need to document how to transition from the
built-in bootloader to MCUboot in order for people to be able to use this
without a debugger.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zephyrproject-rtos/zephyr/issues/9055#issuecomment-434392122,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA97lm1Io2xGx8xXt1Km82Z5qMjla8zDks5uqItSgaJpZM4VYCnH
.

Is there now a documentation how to replace the stock bootloader without jlink?

@tugstugi not yet, @lemrey is working on it and will report soon

I've tried uploading the latest firmware via j-link but I keep seeing this. It doesn't appear to be working on Linux either
Any idea what might be happening?

Any idea why I can't get the hci_usb sample to work on this board? It appears to fail in the same way on both windows and linux. I'm uploading to the board via j-link and the blinky example works via this method

Hi @Xenoamor, on this board the default peripheral used for HCI over UART is a UARTE peripheral. At the moment we are fixing an issue in the UARTE driver which is preventing the hci_uart sample to work as intented. Could you please check if it works any better with this patch https://github.com/zephyrproject-rtos/zephyr/pull/11475 ?

Thanks, I'll give it a try when I'm back in the office tomorrow! Just to clarify that this is an issue I'm seeing with hci_usb and not hci_uart though

@Xenoamor so this is a completely different issue, not related to using this board without a debugger? Are you flashing using J-Link?

Sorry yes I'm using a J-Link and nrfjprog. Should I make a new issue for this?

@Xenoamor yes please, this is completely unrelated.

Thanks, I'll give it a try when I'm back in the office tomorrow! Just to clarify that this is an issue I'm seeing with hci_usb and not hci_uart though

Sorry @Xenoamor, I had misunderstood. What I said applies for HCI over UARTE.
Follow carlescufi's advice then and create a new issue.


Reduce build matrix to support pyg4ometry

pyg4ometry does not provide wheels for Windows nor certain Python versions and linux/macOS versions. Reduce matrix to pyg4ometry's support Python and macOS/Linux versions.
Expect that tests will still fail, but setup stage should pass.

Hi @yramachers, most issues fixed, just the external services to add (in a separate PR). You should review here as there are some changes to the package structure and scripts involved!

Hi @drbenmorgan I'd like to approve this request (thought I had done it earlier but clearly not). I hope this doesn't interfere with your current efforts.


Fix a couple of Coverity issues in health check plugin, around filenames

This fixes #10349
This fixes #10350

[approve ci cmake]

/home/jenkins/workspace/Github_Builds/cmake/src/plugins/healthchecks/healthchecks.cc
In function 'HCDirEntry* setup_watchers(int)',
    inlined from 'void* hc_thread(void*)' at ../plugins/healthchecks/healthchecks.cc:177:17:
../plugins/healthchecks/healthchecks.cc:140:12: error: 'char* strncpy(char*, const char*, size_t)' output may be truncated copying 4095 bytes from a string of length 4095 [-Werror=stringop-truncation]
  140 |     strncpy(fname, conf->fname, MAX_PATH_LEN - 1);
      |     ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../plugins/healthchecks/healthchecks.cc:140:12: error: 'char* strncpy(char*, const char*, size_t)' output may be truncated copying 4095 bytes from a string of length 4095 [-Werror=stringop-truncation]
cc1plus: all warnings being treated as errors

/home/jenkins/workspace/Github_Builds/cmake/src/plugins/healthchecks/healthchecks.cc
In function 'HCDirEntry* setup_watchers(int)',
inlined from 'void* hc_thread(void*)' at ../plugins/healthchecks/healthchecks.cc:177:17:
../plugins/healthchecks/healthchecks.cc:140:12: error: 'char* strncpy(char*, const char*, size_t)' output may be truncated copying 4095 bytes from a string of length 4095 [-Werror=stringop-truncation]
140 |     strncpy(fname, conf->fname, MAX_PATH_LEN - 1);
|     ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../plugins/healthchecks/healthchecks.cc:140:12: error: 'char* strncpy(char*, const char*, size_t)' output may be truncated copying 4095 bytes from a string of length 4095 [-Werror=stringop-truncation]
cc1plus: all warnings being treated as errors

Yeh, this had me stumped here for a bit, because I didn't change this part of the code. I have an idea now, which I'm trying.


Enhance named_struct to support keys as multiple rows

Is your feature request related to a problem or challenge?
Failed test
statement ok
create table t(name varchar, val int) as values ('a', 1), ('b', 2), ('c', 3);

query error DataFusion error: Execution error: named_struct even arguments must be string literals, got t\.name instead at position 0
select named_struct(name, val) from t;

key could be ColumnarValue::Array (multiple rows), so we could not restrict it to ColumnarValue::Scalar only
let (names, values): (Vec<_>, Vec<_>) = args
        .chunks_exact(2)
        .enumerate()
        .map(|(i, chunk)| {

            let name_column = &chunk[0];

            let name = match name_column {
                ColumnarValue::Scalar(ScalarValue::Utf8(Some(name_scalar))) => name_scalar,
                _ => return exec_err!("named_struct even arguments must be string literals, got {name_column:?} instead at position {}", i * 2)
            };

            Ok((name, chunk[1].clone()))
        })
        .collect::<Result<Vec<_>>>()?
        .into_iter()
        .unzip();

Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response

It is not possible to get the exact array in return_type_from_exprs