Helmcharts not easy to consume, missing logic in processing one of Localization, Configuration or Resource

I would like to have objects 'feeding' each other like so
ComponentVersion -> Localization -> Configuration -> FluxDeployer
Where

Localization localizes values.yaml of a helm chart in my Component
Likewise for Configuration
FluxDeployer sets up the install of my helm chart

This doesn't work because Flux/Helm require that helm charts be in a namespaced repo instead of the being at the root of the registry.
The helm-controller complains about a 404 due to the path it is looking for not existing.
Currently this can be worked around by changing the above to be
ComponentVersion -> Localization -> Configuration -> Resource -> FluxDeployer
where Resource has
extraIdentity:
helmChart: my-helm-chart-name
So the first question, why can't placement of extraIdentity on Configuration have the desired effect?
Second problem is, even after introducing a Resource as described above the deploy fails because the only tag in the repo is 'latest'.
Helm complains that no tags were found in the repo.
This is because Flux/helm require that the tags match a strict semver check and 'latest' doesn't pass this.    I don't understand why the tag is 'latest' when my ComponentVersion is festooned with version info.    Why can't one of those version numbers be utilized?
You can hack around this by adding a version spec on the resource like so
sourceRef:
kind: Configuration
name: *name
resourceRef:
name: chart
version: "1.0.37"
Of course this isn't maintainable so this isn't a solution that one can live with.
And in any case, even adding the version still doesn't give you something that actually is usable.   This because the Resource is picking up the helm chart from the ComponentVersion instad of the Snapshot created from the Configuration.   Comparing resource_controller.go  and localization_controller.go ( see the findObjects func ) provides an idea of why Resource is not producing the desired results.
I can try and work around all of this by adding

HelmRepository and HelmRelease to my component, and
Changing Localization and Configuration to update HelmRepository and HelmRelease instead of my helm chart
Of course the HelmRelease would have a values section that would receive the config updates.

This is what the ocm demo here is doing.   However the problem with this approach is that I am now assuming that ops is using Flux for deploying my service.   I can accept the assumption that helm and ocm are in use but assuming Flux is too much.
Anyhoo as it stands at the processing of at least of the types Localization, Configuration or Resource needs to be changed in order to make it easier to consume helm charts as part of an ocm component.
I'll attach my component.yaml and the manifest file with the ComponentVersion etc.

I'll try testing helm access once I get to this issue. :) The first intended case should have worked 🤔 Not sure why latest was used I'll have to go through the code.
Thanks for reporting it.

@Skarlso and @dee0 , the current way to do configuration and localization for me feels a bit to cumbersome. Thinking of people that want to deploy their Helm Chart, but understood that it makes sense to model their product in OCM should have a very simple possibility to configure and localize their application, similar to how they would by modifying the values.yaml, to specify their intent. If in the background we require CRs with a complex structure and content that establish the intent in the system, then we at least should check if it is possible to at least offer a more simple interface at the surface. Any ideas to solve this Gordian knot are welcome :-)

That is not going to be an easy task. :/ But I think it's gonna be fine maybe. It's just the re-packaging that's missing. A user would have a separate resource that contains configuration values perhaps, and we could merge that with existing ones, I don't know.
Another idea was to basically allow people to describe how to configure something by having the WASM scripts in a resource that would do the configuration for us. They are tiny binaries after all and we could just run them as is and it would configure the requested resource.
That's also an option.

WASM was and still is an interesting idea for the configuration and also still a hot topic out there. I would even buy something super simple for the bread and butter cases and something more complex for the other cases, but I really think that adoption will not raise when we keep the interface as complex as it is.

That might be true, yeah.
I have to do some brainstorming with myself and see if I can come up with something. Neverthless, a pluggable infrastructure is still the way to go I believe. It has been wildly successful in other areas after all.

Hey @Skarlso & @morri-son
Want to restate the use case in our division and then present my naive thoughts on supporting them.
Use case :

There are teams that own the individual services that are to be deployed and there teams that own infrastructure.
The service owning teams are expected to know helm
The service owning teams should not need to know details about the infrastructure.   e.g.  If flux is used to install their helm chart they shouldn't need to know about it
The service owning teams should know that OCM is used and should provide a schema etc that details how to localize and configure their service.   It is Ok if they also need to communicate to the infrastructure team 'extra identity' information and version information about the artifacts in their Component.   ( Note: We want to enable that the Component is the source of truth  )
There is an infrastructure team that is aware of what services are to be installed and that provides infrastructure for providing last mile configuration to those services
Outside of the transported component case, the number of OCI repositories required should be minimized.   i.e.   If I have a Component repository already and I can have my helm chart as a local resource then I should not need yet another repository just for my helm chart.

From my naive user perspective, what is missing to support the above 'nicely' is:

I see here I can specify extra identity info and version information for my resources.    So I can, for example indicate that my dir localresource is a helmchart of version X.    And looking in my cluster I see the extra identity information and version information is already stored in the componentdescriptors.
Unfortunately I can't actually set the helmChart extra identity when defining my component because of this issue.   So even if the ocm controller would consume the info the ocm cli doesn't let me set it.
If the above were fixed and then the mutationreconcilelooper could also be updated to

Use the extra identity information from within the componentdescriptor when constructing the repository name, and
Use the version info from within the componentdescriptors as the tag

And with these mods I could have the simple 'pipeline' of
ComponentVersion -> Localization ( helm chart ) -> Configuration ( helm chart ) -> FluxDeployer
as mentioned in the description of this ticket.
That said, even a change as simple as what is in this PR would make it so my use case is mostly supported.   The only catches would be

If ever helm starts insisting that the repo tag match the version inside Chart.yaml then things would break, and
I need to introduce a Resource in order to get a repo name that helm is happy with ( e.g. ComponentVersion -> Localization -> Resource -> Configuration )    This isn't too bad because it is the infra team that would see this rather than the service teams.

Fwiw, I was hoping to step through the code to see how hard the naive solution above would be to implement.   This is why I left this PR in the 'work in progress' state.   Didn't find the time to do it today but maybe I'll get to it this week sometime.

Hello, let me try to clarify some things.

Outside of the transported component case, the number of OCI repositories required should be minimized. i.e. If I have a Component repository already and I can have my helm chart as a local resource then I should not need yet another repository just for my helm chart.

I assume you are talking about our local registry? That is not something the service or the infrastructure team would be aware off at all. That registry is ocm-controller's local cache and an integration point with flux itself. The FluxDeployer creates Flux components to be able to deploy... with Flux. :) We are aiming to create other deployers like ArgoDeployer or something that can provide other kind of options, but that's in the future. For now, all you would need to be concerned about is defining the properties for a helm chart.

Unfortunately I can't actually set the helmChart extra identity when defining my component because of https://github.com/open-component-model/ocm-project/issues/69 issue. So even if the ocm controller would consume the info the ocm cli doesn't let me set it.

The extra-identity is used by us. OCM the library isn't aware of such a thing. So specifying it there, does nothing. That's why you are getting that error.

ComponentVersion -> Localization ( helm chart ) -> Configuration ( helm chart ) -> FluxDeployer
as mentioned in the description of this ticket.

The reason it doesn't work today is because this feature is incomplete as of now. Like I detailed in my comment, the Resource can produce a valid helm chart, but the others can't yet. :) Once it's implemented your scenario should work nicely. 🤞 😄

If ever helm starts insisting that the repo tag match the version inside Chart.yaml then things would break, and

The version is coming from here:
components:
- name: github.com/open-component-model/helm-test
  version: "v1.0.0"
  provider:
    name: ocm.software
  resources:
  - name: charts
    type: helmChart
    version: 6.3.5
    input:
      type: helm
      version: 6.3.5
      path: charts/podinfo-6.3.5.tgz

So technically you can define whatever version you like when building the helm component using OCM the library.
Then, the resource uses that version as well when it's being reconciled:
apiVersion: delivery.ocm.software/v1alpha1
kind: Resource
metadata:
  name: ocm-with-helm-deployment
  namespace: ocm-system
spec:
  interval: 10m
  sourceRef:
    kind: ComponentVersion
    name: ocm-with-helm
    namespace: ocm-system
    resourceRef:
      name: charts
      version: 6.3.5
      extraIdentity:
        helmChart: podinfo

ComponentVersion -> Localization -> Resource -> Configuration

Hopefully, once everything is implemented, the Resource part could simply be left off. 🤞

Hey @Skarlso
First, let me say 'thank you' for taking the time to respond and having the patience deal with me. :)
"I assume you are talking about our local registry? "
Nope.   I explained this above.   Please see the screenshot above of artifactory.    I circled the extra repository that was created and I explained why this is a problem.
"The extra-identity is used by us. OCM the library isn't aware of such a thing. So specifying it there, does nothing. That's why you are getting that error."
You must be able to see this makes zero sense from my perspective, yes?   Consider what I am seeing as an outsider looking in

CLI documentation says I can set extra identity info but doesn't explain what it is for
Resource documentation, as seen from kubectl explain, says I can set extra identity info including 'helmchart'.  Also doesn't explain what it is for.
Setting helmChart extra identity info on Resource gives me some behaviour I need
But setting helmChart, camel-case, via the CLI gives me an error
As a user I am not appreciative of the difference between OCM and the OCM controller. ( Well I am a bit because I have played with the stuff for too long.    However the people I am trying to convince to adopt OCM have no idea about the distinction )
From what I have seen so far, the extra identity info that I provide to the CLI just gets chucked into the componentdescriptor object in kubernetes but doesn't get used.    That is, for the stuff the cli does accept  is just metadata along for the ride.   So why complain that I use 'helmChart' vs 'helmchart'?    Or why not ConstructRepositoryName recognize 'helmchart' as well as 'helmChart'?

Regarding this example
components:
- name: github.com/open-component-model/helm-test
  version: "v1.0.0"
  provider:
    name: ocm.software
  resources:
  - name: charts
    type: helmChart
    version: 6.3.5
    input:
      type: helm
      version: 6.3.5
      path: charts/podinfo-6.3.5.tgz

it is problematic for these reasons I mentioned above :

If I use the helmChart type then helm chart is stored in a separate repository, it isn't a local resource ( Again see screenshot above etc )
If I pull in the chart as a tgz, instead of a directory, then I can't localize or configure the values.yaml in the chart

So again, from perspective, this should be an easy fix that doesn't break the model

Either let ConstructRepositoryName recognize 'helmchart' as well as 'helmChart' or let the cli accept upper case letters
Have mutation_reconciler_looper use the extra identity and version info I provide in my component description

Then in my component descriptor yaml I can put
name: dmi.csi.sap/dmi-multicloud-service-provisioner2
version: &version "1.0.74" # Doesn't seem to affect tags
provider:
  name: dmi.csi.sap
resources:
- name: chart
  type: dir
  version: "1.0.75"
  extraIdentity:
    helmchart: "mychart"
  input:
    type: dir
    path: ../helm/dmi-multicloud-service-provisioner
    compress: true
    preserveDir: true

and everything will 'just work' :)
Btw, the above is giving me the following  in my componentdescripto, note the extraIdentity and version.     So the information needed to do the right thing is there, just Localize and Configuration need to use it
 componentReferences: []
  name: dmi.csi.sap/dmi-multicloud-service-provisioner2
  provider: dmi.csi.sap
  repositoryContexts: []
  resources:
    - access:
        localReference: sha256:645d3bf820707f82872a2766c423ea4303e4e4b462fa986429470e90420d247c
        mediaType: application/x-tgz
        type: localBlob
      digest:
        hashAlgorithm: SHA-256
        normalisationAlgorithm: genericBlobDigest/v1
        value: 645d3bf820707f82872a2766c423ea4303e4e4b462fa986429470e90420d247c
      extraIdentity:
        helmchart: mychart
      name: chart
      relation: local
      type: dir
      version: 1.0.75

Okay, so I think we have to make something clear. The controller is using OCM LIBRARY. Whatever the CLI does is the CLI's thing. :) The controller is using the Library.
So whatever the CLI does is not necessarily something the controller does. The controller has different goals and purposes.
This means that your component description doesn't need the extra identity the controller needs. Here: https://github.com/open-component-model/ocm-controller?tab=readme-ov-file#helmchart-type-resource it's explained that the controller is using this because it cannot be inferred. This is the only reason for it.
I could extend this of course if it doesn't make any sense. :)
Sorry, I might be dense, but where / what is creating that extra repo?

The chart is in a separate repo, which is something we don't want

But why?
So like I wrote, for now, configuration and localization isn't working with charts yet. I need to add that functionality. Resource though works with charts as long as OCM can fetch them. Whatever it creates internally, is not a concern of the user.

"what is creating that extra repo?"
The extra repository is created by the command
ocm --cred :type=OCIRegistry --cred :hostname=XXX --cred username=YYY --cred 'password=ZZZ' transfer component  ./component-archive XXX/csi-ocm-poc
The chart is in a separate repo, which is something we don't want But why?
Because at least here, there is administrative overhead in having an additional repository that is used for production resources

Need to negation with teams running the registries servers so that you can have it and so that has the capacity you need (  5 tags, 10 tags, .... )
Need to maintain the access control to it
Need to document it and maintain documentation for it
Need to re-explain to administrators and security people who forget why you need or who never read the doc why you need it

Ahhhhh. I see. So ocm cli is creating it. I... can't do anything about that I'm afraid. :D

In the attached image I have circled in red the extra repo. The problem with the extra repo is that then we have to negotiate with the teams managing the various repositories we have to work with to request permission for 2 repositories instead of 1. Basically it will double the 'paperwork' involved in getting this to work.

Hi @dee0 , the first repository "component-descriptors" is created the first time a transfer of a component version to / happens. It only stores the component versions, nothing else. If your component version only contains references to external resources, it will be the only repository created in the OCI registry. Did you transfer the component using the -V / --copy-resources option? This is trying to copy the resources to the same repository name as in the source if available.
component:
  name: ocm.software/demos/echoserver
  provider:
    name: ocm.software
  resources:
  - access:
      imageReference: ealen/echo-server:0.9.2
      type: ociArtifact
    name: image
    relation: external
    type: ociImage
    version: 0.9.2
  version: 0.9.2

Hey @morri-son
Thanks for chiming in.
No, I didn't use either of -V or --copy-resources.   The command line I have been using is :
ocm --cred :type=OCIRegistry --cred :hostname=XXX --cred username=YYY --cred 'password=ZZZ' transfer component  ./component-archive XXX/csi-ocm-poc

I believe this is now fixed with the recent move to OCI repos and a bit of a better handling of helm charts with removing the need to the name and tagging and everything.
@dee0 If you have the time, please re-test this. In the meantime, I'm gonna close it as fixed. Please re-open it if the problem persists. :)

Hey @Skarlso
Should I building and testing the main branch or should I test with the latest release?

Latest release should be okay now.

Hey @Skarlso
My first attempt to use the latest ocm-controller failed.  I am still investigating however what I can tell you  is :

We have 'landscapes' which are collections of k8s clusters of different types.   Types -> different cluster config, different services deployed.
I have on boarded 4  services to OCM, two that only go to one cluster type and two that only go to another.
For one cluster type tests failed because a key service for the type didn't get deployed.   However for the other cluster type there was no problem reported
For the one where the was a problem, all FluxDeployers didn't enter ready state cause HelmRelease v2 isn't present.  I checked the cluster and I see that HelmRelease v2beta1 and v2beta2 are present but not v2
For the one where no problem was reported, I have not yet checked if there was really no problem or if there just is no test that would detect a problem

Will provide updates as I get more data

HelmRelease ->

@Skarlso
Ok, for the cluster that hadn't reported a problem it turned out to fail the same way as the one did.    There was quite some latency between the test completion and the results being collected.
So, first problem with updating to the ocm controller v0.23.7 is that it requires an update to the flux controllers.    To deploy, I am using the output from ocm dry run after running it through a script that 'templatizes' it.     Hopefully if I was using the cli to directly deploy or if I was using the new helm chart I would have received a warning about this.
Yesterday morning I upgraded all of the flux controllers to the latest release and kicked off another test.   Looking at the results this morning I see that the upgrade has broken my POC.    I haven't reviewed all the data yet, but the thing I see immediately is that the FluxDeployers aren't healthy.   They have status like
  status:
    conditions:
      - lastTransitionTime: "2024-06-24T17:48:35Z"
        message: 'failed to create Helm Release object :failed to create reconcile kustomization: HelmRelease.helm.toolkit.fluxcd.io "dmi-broker" is invalid: [spec.chart.spec.sourceRef.name: Invalid value: "": spec.chart.spec.sourceRef.name in body should be at least 1 chars long, spec: Invalid value: "object": either chart or chartRef must be set]'
        observedGeneration: 2
        reason: CreateOrUpdateHelmFailed
        status: "True"
        type: Stalled
      - lastTransitionTime: "2024-06-24T17:48:35Z"
        message: 'failed to create Helm Release object :failed to create reconcile kustomization: HelmRelease.helm.toolkit.fluxcd.io "dmi-broker" is invalid: [spec.chart.spec.sourceRef.name: Invalid value: "": spec.chart.spec.sourceRef.name in body should be at least 1 chars long, spec: Invalid value: "object": either chart or chartRef must be set]'
        observedGeneration: 2
        reason: CreateOrUpdateHelmFailed
        status: "False"
        type: Ready
    observedGeneration: 2

No idea why it is mentioning kustomizations.    I am not using those at all.

That's a typo. It should be HelmRelease. What's your setup look like now? The new version updated the spec and the way we use HelmCharts to align with the new Flux version.
Here is a project that uses helm charts properly: https://github.com/Skarlso/ocm-helm
Component Version
apiVersion: delivery.ocm.software/v1alpha1
kind: ComponentVersion
metadata:
  name: ocm-with-helm
  namespace: ocm-system
spec:
  interval: 10m0s
  component: github.com/open-component-model/helm-test
  version:
    semver: 1.0.3
  repository:
    url: ghcr.io/skarlso/helm-test
    secretRef:
      name: creds

Resource:
apiVersion: delivery.ocm.software/v1alpha1
kind: Resource
metadata:
  name: ocm-with-helm-deployment
  namespace: ocm-system
spec:
  interval: 10m
  sourceRef:
    kind: ComponentVersion
    name: ocm-with-helm
    namespace: ocm-system
    resourceRef:
      name: charts
      version: 6.3.5

Deployer:
apiVersion: delivery.ocm.software/v1alpha1
kind: FluxDeployer
metadata:
  name: fluxdeployer-podinfo-pipeline-backend
  namespace: ocm-system
spec:
  interval: 1m0s
  sourceRef:
    kind: Resource
    name: ocm-with-helm-deployment
  helmReleaseTemplate:
    interval: 5m

To recap my setup, I have this setup

Here is my FluxDeployer
- apiVersion: delivery.ocm.software/v1alpha1
  kind: FluxDeployer
  metadata:
    name: *name
    namespace: *ocm-sys-namespace
  spec:
    interval: 10s
    sourceRef:
      kind: Configuration
      name: *name
      namespace: *namespace
    helmReleaseTemplate:
      timeout: 1h
      chart:
        spec:
          chart: *name
      interval: 10s
      targetNamespace: *namespace
      valuesFrom:
      - kind: Secret
        name: *landscapeSecretName
        valuesKey: *secretKey

The Configuration status indicates they are healthy.

For completeness sake, the FluxDeployer above was from a helm template.   Here is the rendered result
- apiVersion: delivery.ocm.software/v1alpha1
  kind: FluxDeployer
  metadata:
    creationTimestamp: "2024-06-24T17:46:20Z"
    generation: 2
    labels:
      hc.sap.com/hc-provisioned: "True"
    name: dmi-broker
    namespace: ocm-system
    resourceVersion: "19542"
    uid: 5343f112-0df2-4c70-96d2-3dc3d8e7d41e
  spec:
    helmReleaseTemplate:
      chart:
        spec:
          chart: dmi-broker
      interval: 10s
      targetNamespace: broker
      timeout: 1h
      valuesFrom:
        - kind: Secret
          name: dmi-broker-from-landscape
          valuesKey: secret
    interval: 10s
    sourceRef:
      apiVersion: delivery.ocm.software/v1alpha1
      kind: Configuration
      name: dmi-broker
      namespace: broker

Hey @Skarlso
At the moment I think what I am seeing is a new ocm-controller bug.   It isn't filling in the helmReleaseTemplate sourceRef properly.

Gonna need some more info about your CRs. Or logs or something. Otherwise you're on your own.

Logs don't show really show anything more than what is in that status message.
Attaching a zip of the ocm CRs
ocm-crs-2024-06-25.zip
Each resource type is in its own *.yml file.   Fwiw this is what our testing infrastructure is spitting out at the end of a failed test run.
Fwiw, this was working until the upgrade.

This:
      resourceRef:
        extraIdentity:
          helmChart: dmi-broker

This is also no longer correct for the flux deployer:
    helmReleaseTemplate:
      chart:
        spec:
          chart: dmi-broker

Will no longer work. Please check out the repository I linked. There is a working example for both helm chart without and wit
h configuration. :)

Hey @Skarlso
Thanks, but I think your sample may be broken and I don't think it is representative for my case.
I think it is broken because the Localization appears to refer to a non-existent Resource.   It refers to resource-pipeline-backend and when I search for that the only hit is in localization.yaml.
As for why I don't think the samples are representative of my case there are these discrepancies

I am using both Localization and Configuration ( presumably this is normal ) but the examples don't use them in combination
My chart is a local resource.   At the time when I was putting things together it didn't seem that I could localize or configure the chart's values.yaml so long as it was in a tgz file.    So if that isn't the case using a tgz would be fine

       resources:
       - name: chart
         version: *version
         type: dir
         input:
           type: dir
           path: "../helm/dmi-broker"
           preserveDir: true
           compress: true

In your examples the delivery.ocm.software resources are referring to specific chart versions.   I can't do that.   Whatever version happens to be  in the ComponentVersion is what should be used.

Please provide some guidance on how to make this work with whatever change you made to the ocm-controller.

Ahhh sorry. I should have said that only configuration was altered to be working. The localization wasn't touched.
So sorry Dan.

I'll try to whip up a localization for you that's working. But generally I think it should once you alter it the same way configuration does it. Only the location of the image and tag needs to follow the value.yaml's format.

Thanks @Skarlso
Btw, the things I am concerned about are

That Configuration can operate on the output of Localization
That there is no requirement to hard code a chart version in any of the delivery objects ( Localization, Configuration, FluxDeployer )

Hum. That shouldn't matter I think. Dir should work.
The requirement is there because it will try and produce a snapshot that the helm controller than needs to understand. And we do that previously already but that information isn't taken any further. Unless the resource is the same version as the chart than that extra info is not needed.

Hey @Skarlso just want to confirm what I think you said...   In my component description if the resource version matches the helm chart version then I don't have to specify a version in the Resource or the FluxDeployer.    So for example, in my component yaml if I have
name: dmi.csi.sap/dmi-broker
version: &version "1.0.0-1.badcafe"
provider:
  name: dmi.csi.sap
resources:
- name: chart
  version: 1.2.3  # <!-- Matches the helm chart version
  type: dir
  input:
    type: dir
    path: "../helm/dmi-broker"
    preserveDir: true
    compress: true

then I won't need to specify a chart version in the Resource or the FluxDeployer.
Do I have that correct?

Yeah, that should be right. As long as the tag in the snapshot matches the chart, Flux's helm controller shouldn't complain. 🤞

Nope, something went wrong.
I'll dig some more, but what I am seeing is
kubectl --kubeconfig=kubeconfig-644-e1-dmi get -A helmrelease
NAMESPACE    NAME         AGE   READY   STATUS
ocm-system   dmi-broker   29m   False   artifact revision 19090 does not match chart version 1.3.0-1085.xeae3fba.14

I have no idea where 19090 is coming from.   Looking at the component descriptor with
kubectl --kubeconfig=kubeconfig-644-e1-dmi get -n broker componentdescriptor dmi.csi.sap-dmi-broker-1.3.0-1085.xeae3fba.14-17027996690744119944 -o=yaml

I see
spec:
  resources:
  - access:
      localReference: sha256:4b942c25c3e772c826959cb4702a3e3d9f9b59be4a78449a15f7e5be4d5e4e12
      mediaType: application/x-tgz
      type: localBlob
    digest:
      hashAlgorithm: SHA-256
      normalisationAlgorithm: genericBlobDigest/v1
      value: 4b942c25c3e772c826959cb4702a3e3d9f9b59be4a78449a15f7e5be4d5e4e12
    name: chart
    relation: local
    type: dir
    version: 1.3.0-1085.xeae3fba.14

19090 is the observed revision which we are using by default as a tag. Hmm, the resource version should have overwritten that. For now, please use helmVersion instead. I'm on vacation for two weeks, so I'll look at it when I'm back.

Hey @Skarlso
I think
https://github.com/open-component-model/ocm-controller/pull/485
may have broken something.
The error I mentioned above was after changing my POC to look like

Yesterday  I grabbed the latest code with the intent of putting in the place the logic I have expected from the beginning. ( Search for 'festooned' in this description )   However before changing anything, while stepping through the debugger I hit a new problem 🔢
"failed to look up component Version: failed to do request: Head \"https://component-descriptors/v2/dmi.csi.sap/dmi-broker/manifests/1.3.0-1126.xb2559b5.17\": dial tcp: lookup component-descriptors on <IP_ADDRESS>:53: no such host",

ResourceReconciler:reconcile gets this when it is trying to get the resource.   This is happening because cv.Status.ReplicatedRepositoryURL is empty string.
My ComponentVersion is
apiVersion: delivery.ocm.software/v1alpha1
kind: ComponentVersion
metadata:
  creationTimestamp: "2024-06-30T19:08:11Z"
  generation: 1
  labels:
    foo.com/blah: "True"
    hc.sap.com/hc-provisioned: "True"
  name: dmi-broker
  namespace: broker
  resourceVersion: "784586"
  uid: 9f52789d-668e-42ca-a5a7-91e5522c3da3
spec:
  component: dmi.csi.sap/dmi-broker
  interval: 3s
  repository:
    secretRef:
      name: dmi-broker-ocm-registry
    url: hanaservice-dev.common.repositories.cloud.sap/csi-ocm-poc
  version:
    semver: '>=v1.0.0-0'
status:
  componentDescriptor:
    componentDescriptorRef:
      name: dmi.csi.sap-dmi-broker-1.3.0-1126.xb2559b5.17-18125962753876232484
      namespace: broker
    name: dmi.csi.sap/dmi-broker
    version: 1.3.0-1126.xb2559b5.17
  conditions:
  - lastTransitionTime: "2024-07-01T13:37:10Z"
    message: 'Applied version: 1.3.0-1126.xb2559b5.17'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
  observedGeneration: 1
  reconciledVersion: 1.3.0-1126.xb2559b5.17

Fwiw, think I will rebase my branch onto 0.23.7 to avoid this problem.

I'm on vacation for a while, would you mind putting this into a separate issue please? Thanks.

Was dmi-broker an existing component version by any chance? 🤔 Because sadly, this change is breaking as in, if the object exists it won't update the status with the correct value unless re-reconciled.

Not stale


Distributed Lock API updates

Description
The structural changes in the docs to introduce the distributed lock API and components. The howto still needs completing
Issue reference
https://github.com/dapr/docs/issues/2485

@yaron2 - Can you review the above. Then issue a new PR to update the HowTo for locks and is there an HTTP API for locks in this release?


Investigate if URLs can contain @ sign

pulled from IRC, may deserve an issue.
<Codex_> m2tias: right, but ldjam web page has problems with urls that have @ character in it

If true we'll need to check how the @ is munged by the URL parser.

original url from this report was https://www.meshpage.org/mesh.php?id=GVILK@080

Update the regex of at-names along the lines of (^@[^ ]+|[^\w]@[^ ]+) (Either start a line with an at-name or if an at-name is not preceded by a letter/number). The @[^ ]+ should be whatever the current bit is and that part should be within parenthesis too so only that part is grouped.

Update, I think the look-ahead was not working in previous version of the atname so here:s my suggestion for replacement:
/^@([A-Za-z0-9-]+)($|(?=[ ,.?!/":'\\]))/`
for line https://github.com/ludumdare/ludumdare/blob/master/src/internal/marked/InlineLexer.js#L24
but the way we step forward we must invoke the allowance of @ into the uri match and make sure it happens first.

The issue has me a little confused, the uri pattern matches the whole thing  including the at and what comes after and that one seems to have higher priority than the at-name pattern so don't really understand what is happening.


WIP: Request API change

The changes will be the following:
bool mr_run_session_until_timeout(mrSession* session, int timeout);

This function is waiting always the timeout. If a message is received before the timeout, the function will listen for another message. The change is for to listen only one message, exiting before the timeout if the message is received, because the function name seems to indicate this, not its previous behavior.
In order to preserve the before behaviour, a new function will be created:
bool mr_run_session_time(mrSession* session, int time);

Also, if the purpose of an user is only to send the stream data to the agent, currently is done with mr_run_session_until_timeout(session, 0), but this will listening a message if it already exists for its reading. For that, if the user want send without listening anything, a new function is added:
void flash_output_streams(mrSession* session):

Before accept this PR, @lemunozm, create a PR in eProsima/micro-RTPS-doc updating documentation about these new functions.


[Function Overloading] implementation is obviously different to the signatures, but no ts error

Bug Report
🔎 Search Terms
overload, implementation, signature, different, compatible
🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about This-FAQ

⏯ Playground Link
code in playground
💻 Code
// signature
function foo(xx: number): number;
function foo(xx: number, yy: string): string;

// implementation
// ❗it may return `boolean`, differing to the signatures
function foo(xx: number, yy?: string): number | string | boolean {
    if (yy == null) {
        return xx;
    }
    if (yy === "666") {
        return true;
    }
    return yy;
}

🙁 Actual behavior
Obviously, the implementation doesn't satisfy the signatures, but TS didn't "find" it. It confused me, and brought some issues into my project.
I don't know whether it is by-design or a bug, and I didn't find any doc about this behavior. Please  tell me the reason of this design, if it is by-design.
🙂 Expected behavior
// signature
// EXPECTED: Tell me "This overload signature is not compatible with its implementation signature."
function foo(xx: number): number;
function foo(xx: number, yy: string): string;

// implementation
function foo(xx: number, yy?: string): number | string | boolean {
    // ...
}

This is basically the same kind of unsoundness that happens with the "correct" signature:
// signature
function foo(xx: number): number;
function foo(xx: number, yy: string): string;

// implementation
function foo(xx: number, yy?: string): number | string {
    return 42;  // unconditionally returns a number
}

...namely that nothing prevents you from returning the wrong type for the overload that was actually called.  There doesn't really seem to be an easily generalizable mechanism by which you could reject the case in the OP without also rejecting the "correct" signature, either, since both cases involve a return type which is simultaneously wider than both overloads'.  So what would the rule for that be?

Duplicate #52478 and ultimately #13235


CLI options

There are already some very good libraries that will help us do the requirements as long as they allows as they provide the necessary options required to implement all options of unzip

clap
....

Ok with clap.

With clap we need to check if we can do all the options as they are shown in unzip if we are going to do the same options and as close to unzip as possible for seamless migration to our tool for users.

I made a fast read and I didn't see anything to complex. I think it will be more internal logic and build the proper structs for representing the interactions between flags.

Ok, we can try out Clap and try to replicate all the options at first and if something is not fitting we can always improvize.

I will close this as we already are using clap and for what I see it supports everything we need.


Video activity recognition ROS node

This PR adds the video-based human activity recognition ROS node.
Added

projects/opendr_ws/src/perception/scripts/activity_recognition.py

Changed

projects/opendr_ws/src/ros_bridge/src/opendr_bridge/bridge.py: Added to_ros_category, to_ros_category_description and from_ros_category as renamed copies of to_ros_face, to_ros_face_id, and from_ros_face to keep backwards-compatibility. @Pavlos-Tosidis, I suggest that the face-recognition module also adapts the generic category naming and removes the old face-recognition specific ros bridge functions.

docs/reference/rosbridge.md: Added to_ros_category, to_ros_category_description and from_ros_category docs.

projects/opendr_ws/README.md: Added Video Human Activity Recognition to list of nodes.

projects/opendr_ws/src/perception/README.md:  Added short usage description

I fixed the conflict with the merge due to the introduction of the simplification of the imports. I also realized that I forgot to apply the simplification to the ROS scripts so I took the opportunity to change yours and the others as well.


fix(Sequence): Expand collab window to include refSeqs of in-flight ops

Description
Leverages the mechanism added on Client in #19211 for the sequence package. This fixes the 0x54e bug and allows re-enabling a ton of fuzz tests.
To get tests passing, I also had to correct some logic in IntervalCollection which stored pending data for operations even while detached (and thus no ack would ever come).
There are still a small number of fuzz failures which need to be root-caused, that work is tracked by AB#6552.

I have a fix for the remaining 0x54e issue, but while writing a regression test for it I discovered another bug with zamboni that prevents the regression test from functioning properly. Since it's a rarer case, I think it makes sense to just get this in now and follow up with other PRs to fix the other issues.


Add canned_acl attr

Remove acl from tests that do not need it

We will filter individual tests instead, attribute is not necessary


Add Sunchon National University

https://www.scnu.ac.kr/neweng/main.do

@hellomatia Thank you for your patience while waiting for a review and processing your request. Pull request merged. Please start applying for JetBrains educational subscription at https://www.jetbrains.com/shop/eform/students in an hour, to allow the changes to propagate through JetBrains systems.


Stop exception spam on register and unregister methods

The Story
Expect a singleton dialog instance which will be opened by the user on a specific button click.
This singleton dialog instance registers on the eventbus when the dialog starts to show.
What the developer now has to consider is to check if the dialog instance is already registered.
Otherwise an exception will be thrown which is a common pitfall.
It brings the application into an unecessary error state.
And it leads to boiler plate code which itself leads to bad design, because developers have to implement:
if (!eventBus.isRegistered(object)) {
    eventBus.register(object);
}

and
if (eventBus.isRegistered(object)) {
    eventBus.unregister(object);
}

Solution Proposal
Why not implementing the register and unregister methods like this:
    /**
     * Registers on event bus.
     *
     * @param object the object to register
     * @return true if the object is registered successfully, false if the object was already registered.
     */
    public boolean registerBetter(final Object object) {
        if (!isRegistered(object)) {
            register(object);
            return true;
        }
        return false;
    }

    /**
     * Unregisters on event bus.
     *
     * @param object the object to unregister
     * @return true if the object unregistered successfully, false if the object was not registered before.
     */
    public boolean unregisterBetter(final Object object) {
        if (isRegistered(object)) {
            unregister(object);
            return true;
        }
        return false;
    }

With that solution developers can just call register, and they don't have to consider the internal state of the event bus.
Because the internal state of the event bus framework is something that the framework should handle, not the developer who is using it.

i guess  it can't be changed like u said
because eventbus will post stick event when execute register method.
so if there 's a stick event in ur dialog and it need to be executed again and again ,your suggestion will stop it from executing again.
i guess that's why greenbot give a interface of isRegistered


Add guide to run Piped on Google Cloud VM

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:

NONE

Nice work
/lgtm

LGTM except for just one nit

/approve


DNS server in user-defined networks

I can't get the embedded DNS server in a  user-defined network working. This could be a real issue of Runv or an incorrect setup.
I tried to minimize my setup as followed:
# runv --kernel kernel --initrd hyper-initrd.img --debug containerd
# dockerd -s overlay -H unix:///var/run/docker.sock --containerd /run/runv-containerd/containerd.sock --debug

using the default docker bridge works as expected:
# docker run -t --rm busybox nslookup www.google.com
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS>
Name:      www.google.com
Address 1: 2a00:1450:4009:80c::2004 lhr35s07-in-x04.1e100.net
Address 2: <IP_ADDRESS> lhr25s10-in-f164.1e100.net

in a  user-defined network every DNS request fails:
# docker network create --subnet=<IP_ADDRESS>/16 mynet
# docker run -t --rm --network=mynet busybox nslookup www.google.com
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS>

nslookup: can't resolve 'www.google.com'

It's only the embedded DNS that doesn't work.  E.g. nslookup www.googe.com <IP_ADDRESS> works fine as well as the network in general. Maybe somebody else is already aware of this problem.
I'm using runv 0.6.2, latest hyperstart ca80a07e and docker 1.12.2 Unbuntu x86_64
Thanks
peter

Thanks for your report!
The reason is dns request sent out by container bypass the dns resolver of docker. It's a bug of runv network implement, Will try to fix it.
Thanks

@pmorjan Sorry, This is difficult for us to support this feature now. since docker adds netfilter nat rules in container netns and use loopback address as nameserver... We need to consider a compatible way for runv to work together with docker daemon.

update:
docker start process to serve as dns service in container netns. have no idea how to handle this in runv now.
# netstat -apn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:33517        <IP_ADDRESS>:*               LISTEN      14873/dockerd
udp        0      0 <IP_ADDRESS>:36107        <IP_ADDRESS>:*                           14873/dockerd

Thanks for the update. I understand there is no easy way to solve this. Since I no longer need a fix for this issue feel free to close it. Thanks!


Fixes #773 Camera Scan button making it difficult to view the nutrition facts for some products

Description
Added bottom padding of 60 dp (since floating button is 56dp, it would make it more comfortable to view). Now the size of view pager is the same as before.
Screen-shots, if any

Thank you! :)

Merging, thanks @ManavPoddar


Description guidelines: add "usage" to word list

Have you searched existing descriptions for violations of this guideline?

There are two instances:
https://github.com/web-platform-dx/web-features/blob/d88ac272a8dde06cb194511ca8e0f338e2cdbb06/feature-group-definitions/array-iterators.yml#L2
https://github.com/web-platform-dx/web-features/blob/d88ac272a8dde06cb194511ca8e0f338e2cdbb06/feature-group-definitions/typed-array-iterators.yml#L2

@foolip I just pushed changes to both. Naturally, they were difficult to write! Want to take a look?


Add cross-platform support for the editor

Probably not coming anytime soon but cross-platform support for the editor would be nice now that the project was ported to .NET Core. This requires a switch from WPF to a cross-platform UI framework such as https://github.com/AvaloniaUI/Avalonia.

i also want this feature


Add TypeScript 2.1 release notes

//CC @DanielRosenwasser, @sandersn
Some parts copied verbatim from @DanielRosenwasser's blog.
This includes the content already in https://github.com/Microsoft/TypeScript/wiki/What's-new-in-TypeScript.

@DanielRosenwasser and @sandersn, I have simplified the notes for keyof and mapped type, less chatty now.

@DanielRosenwasser let me know if you have more comments and i will get them done in a separate PR.


Internal logging package and a test-utils package to help them

re: #142
Adding two packages, @changesets/logger and @changesets/test-utils.
@changesets/logger
Log messages to stdout.
Usage:
import { warn } from "@changesets/logger";

warn('message part 1', 'message part 2');

@changesets/test-utils
All test utilities, at the moment only one - temporarilySilenceLogs, it will silence all logs done in the changesets file. **It will not highjack console, therefore console.*(xxx) will still. Which in my perspective is needed to not accidentally silence the error message.
// In tests
import { temporarilySilenceLogs } from "@changesets/test-utils";

temporarilySilenceLogs();

[x] Add a logger

[x] add package.
[x] add tests and documentation.
[x] change usage in the cli package.
[x] Method to silent the internal logging in tests.
[x] Document this internal logging.

This PR is in progress, but inputs at any step are welcome.

Also: needs a changeset

The commit history is not pretty at the moment. I feel some of the messages can be improved and some commits be squashed. I am spend time in fixing the messages. If we do a merge. Else we can update message on squash and commit. 🙂

Hey @mitchellhamilton, Thanks for reviewing the PR. Do you think we can merge it?

I was going to wait for @Noviny to ✅ it but it's not really a change that's likely to cause problems so let's merge it.


Implement Default for AssumeNotNull

Just not sure to what extent this is backwards compatible - can't users implement that in their own crate if they own T?

That should be backward compatible. At least the following code does not compile in a downstream crate with an error message related to the fact that you cannot implement a foreign trait for a for foreign type.
struct Foo;

impl Default for diesel::dsl::AssumeNotNull<Foo> {
    fn default() -> Self {
        todo!()
    }
}

This only works if the trait itself has a type parameter or if the type is marked at fundamental. (Beside of that downstream users wouldn't be able to provide a meaningful implementation of Default for that type, as it exposes no public fields and no constructor.)


fileinfo crashes when analyzing x86-64 Mach-O executable (cannot grow llvm::SmallVector)

fileinfo crashes when analyzing a x86-64 Mach-O executable due to insufficient memory to grow llvm::SmallVector.
Input
Run
$ retdec-fileinfo FILE

where FILE is

5B9AC528160A1C6164DF6F7E3A1375207847E57BA3F8CAC7716DB8D129B4D87A (168 kB)

Output
retdec-fileinfo: src/llvm-project/lib/Support/SmallVector.cpp:36: void llvm::SmallVectorBase::grow_pod(void*, size_t, size_t): Assertion `NewElts && "Out of memory"' failed.

Expected output
fileinfo does not crash when analyzing the above file.
Backtrace from gdb
#0  0x000055555584d1b8 in llvm::object::ExportEntry::pushNode(unsigned long) ()
#1  0x000055555584d444 in llvm::object::ExportEntry::pushDownUntilBottom() ()
#2  0x00005555558552aa in llvm::object::MachOObjectFile::exports(llvm::ArrayRef<unsigned char>) ()
#3  0x0000555555855b47 in llvm::object::MachOObjectFile::exports() const ()
#4  0x000055555571a5ca in retdec::fileformat::MachOFormat::dyldInfoCommand (this=0x555556056f30, commandInfo=...) at src/fileformat/file_format/macho/macho_format.cpp:916
#5  0x000055555571afc3 in retdec::fileformat::MachOFormat::loadCommands (this=0x555556056f30) at src/fileformat/file_format/macho/macho_format.cpp:1064
#6  0x000055555571821b in retdec::fileformat::MachOFormat::initStructures (this=0x555556056f30) at src/fileformat/file_format/macho/macho_format.cpp:246
#7  0x000055555571774a in retdec::fileformat::MachOFormat::MachOFormat (this=0x555556056f30, ...) at src/fileformat/file_format/macho/macho_format.cpp:72
[..]

Notes

The input file has only 168 kB, but even 64 GB of RAM (see the configuration below) is not sufficient to analyze it.
Possibly related to #119.

Configuration

Commit: 1647e9f63120421c66d16c6fb9b9777719023c2c (current master)
64b Debian Stretch, GCC 6.3.0, Debug build of RetDec
ulimit -v is 64 GB

Another sample with the same issue: 515a95f91c599ebc5c196fab8fbddc10c8b32db4c95011634c1897992b089694

Another sample with probably the same issue: 67CA94E779B8FA0BA8758A7EC010922FF633039A9023077C44000F48A6B76F43

Another sample with probably the same issue:
FBA4E789B2715FCE4D30E4BF91DF4D7507BE018318065862F38D6DC2AC437CA0

Another sample with probably the same issue:
ACFB78DA95D27F0677C45B00514360018B5C6A546A04CDF1C6E83610F926CB2D

Same as in #119, these are OK in the new LLVM. All of them are also without problems in valgrind. Added some of them to nightly tests.


[CH] Improve the performance of FormatFile::readSchema

Currently FormatFile::readSchema is too slow for orc/parquet because the schema is read from remote hdfs file. Sometimes it will takes more than 10s for a single file.
TODO: get table schema from substrait proto instead of loading from orc/parquet remote file.

let's do it!


updates-arch-combined only appears on 1 monitor at a time

When using multiple monitors, the module 'updates-arch-combined' only appears on one of them.
My config file: https://github.com/zjeffer/dotfiles/blob/master/polybar/config
This only happens with updates-arch-combined. All other modules work as intended.

But you use the same bar at two monitors at the same time?

Yes

Could you change the exec line to exec = echo test?

Test displays on both monitors.

After restarting polybar, the update module always loads about a second after the bar and the other modules. Maybe the update script is slow(er)?
Could it be that this happens to every module that has a long loading time?

To debug this issue, I set the interval to 10 seconds instead of 600, and now when I refresh, it loads the module on 1 monitor, and after 10 seconds it loads it on the other as well.

The reason why the script does not load instant is that it need to pull the information from the update servers first. This needs more time than just print a cpu usage.
You could use $ time /path/to/updates-arch-combined.sh to count the time.
The problem you have is that the script plugin does not run at the same time on different bars.
But i think its not to achieve.

So this is more a polybar issue than the particular script? I'll make an issue in the polybar github, closing.


The size of tensor a (32) must match the size of tensor b (40) at non-singleton dimension 2

Using Ubuntu Deepstack docker 2022.01.1 on esxi with grid Nvidia GPU. RTX A5000
After sometime Deepstack seems to crash with:
File "/app/intelligencelayer/shared/detection.py", line 81, in run_task
det = detector.predict(img_path, threshold)
File "/app/intelligencelayer/shared/./process.py", line 63, in predict
pred = self.model(img, augment=False)[0]
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/app/intelligencelayer/shared/./models/yolo.py", line 149, in forward
return self.forward_once(x, profile)  # single-scale inference, train
File "/app/intelligencelayer/shared/./models/yolo.py", line 176, in forward_once
x = m(x)  # run
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/app/intelligencelayer/shared/./models/yolo.py", line 75, in forward
y[..., 0:2] * 2.0 - 0.5 + self.grid[i].to(x[i].device)
RuntimeError: The size of tensor a (32) must match the size of tensor b (40) at non-singleton dimension 2
Anyway increase logging level to get timestamps?
Anyone have an idea what’s causing this crash?
Any options to catch this and restart deepstack docker instance automatically?

after trying to figure this out I found mentions on pytorch forums that this might happen if the image size is too big.
I reduced the resolution of my 8mp cams to 1980p and alas it's working.
anyway to limit/reduce the image size in deepstack before it hits the AI?

Hi @babbahotep,  this error is as a result of shape mismatch. I'd suggest you make sure the shape of both input tensors match, so in place addition can be executed hitch free.

Input is blue iris i have no control over the input.
this only happens on esxi docker. Ds on windows bare metal works fine.


Setup mass matrices with ARKODE

@OliverEvans96 pointed out that he wanted to use ARKODE with mass matrices and currently that's not setup. It would just require checking for a mass matrix in the ODEProblem and if it has one, converting it to a Sundials matrix and passing it through the API call.

Hi @ChrisRackauckas
Is there any progress on this issue? I would also be interested in using this feature, particularly with sparse mass matrices.

Hey, sorry. I ended up getting very busy. This isn't too difficult of an issue though so I am hoping it will get done if there's a student who's interested in starting work on Sundials.jl for GSoC. If not, I can knock it out sometime in January. Give it a bump then if you're still interested. It should be only around 10 lines of code so it's a great beginner issue but I also don't want to delay someone's research if I can find the hour or so to do it.

Hi @ChrisRackauckas,
Thanks for working on this! I've been trying to get the solver working for systems of the form M*du = K*u, where M and K are constant sparse matrices, but I've run into some issues. A minimal example is:
M = randn(3,3) |> sparse
K = copy(M)
u0 = ones(3)
tspan = (0.0, 1.0)
f! = (du,u,p,t) -> mul!(du, K, u)
u_analytic = (u0,p,t) -> exp.(u0.*t)

odefun = ODEFunction{true,true}(f!, analytic = u_analytic, mass_matrix = M, jac_prototype = K)
prob = ODEProblem(odefun, u0, tspan)
sol = solve(prob, ARKODE(linear_solver = :KLU, mass_linear_solver = :KLU), u0, tspan)

The issue, as far as I can tell, is two-fold:

The ARKODE constructor doesn't allow using :KLU for the linear_solver or mass_linear_solver. It does look to be implemented, though, so I think it just hasn't been added to the list of allowed solvers.
Manually adding :KLU to the allowed solvers, I then get the following ARKODE error:

[ARKODE ERROR]  arkInitialSetup
  The mass matrix solver's setup routine failed.

And I've not been able to resolve this. Any help would be much appreciated!

I don't know why that would occur either. It would take some digging. Maybe the mass matrix isn't what's passed down to the right constructor, so it doesn't build a sparse SUNMatrix? One would need to take a deep look at what's being passed where in the solve.jl code.

Okay, I'll do some digging when I have the time then. I'll let you know if I figure it out, possibly just by making a PR if it's a simple enough fix.


Update Stride to v22.0.0

Pre-release image: ghcr.io/akash-network/cosmos-omnibus:v0.4.13-rc1-stride-v22.0.0


vignette function return "no vignettes found"

metajam vignettes are not listed by the vignette function
library(metajam)

vignette(package="metajam")
# no vignettes found

It appears that devtools:build_vignettes() is creating the rendered vignettes into the .doc directory and not ./inst/doc. When the package is installed then, the ./doc directory is not included. If I manually move the ./doc directory to ./inst/doc and rebuild, the vignettes show up.
I'm not sure how to resolve this problem yet, it may take some perusing of the build_vignettes source.

in fact, we need to pass explicitly building vignettes to devtools:
devtools::install_github('NCEAS/metajam', build_vignettes = TRUE)


Add suzannehamilton as a user

Requesting access so that I can SSH in and delete the workspace for frontend/master, because the git checkout has got into a weird state on one of the agents.

Closing this PR because the repo is for the old CI and I'm trying to SSH to the new one.


Inquiry regarding the feasibility of fine-tuning LLaMA2-7B with a single A100

Hi team,
Great work, but I have a question to consult.
I used --adam_offload option in https://github.com/OpenLLMAI/OpenRLHF/blob/main/examples/scripts/train_sft_llama.sh which mentioned in your blog on Zhihu(https://zhuanlan.zhihu.com/p/650758507) to make it possible to fine-tuning 7B model with a single A100(80G).
However, upon implementing this option, I encountered difficulties as the script seemed to get stuck.
Could you provide more details about it and recommended practices for fine-tuning LLaMA2-7B with a single A100(80G):)

[2023-12-01 05:53:04,949] [INFO] [comm.py:662:init_distributed] Distributed backend already initialized
[2023-12-01 05:53:15,433] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2023-12-01 05:53:15,436] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2023-12-01 05:53:15,436] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2023-12-01 05:53:15,453] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2023-12-01 05:53:15,453] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2023-12-01 05:53:15,453] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:146:__init__] Reduce bucket size 500,000,000
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:147:__init__] Allgather bucket size 500,000,000
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:148:__init__] CPU Offload: True
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:149:__init__] Round robin gradient partitioning: False
[2023-12-01 05:53:49,783] [INFO] [utils.py:802:see_memory_usage] Before initializing optimizer states
[2023-12-01 05:53:49,784] [INFO] [utils.py:803:see_memory_usage] MA 12.86 GB         Max_MA 12.86 GB         CA 12.86 GB         Max_CA 13 GB
[2023-12-01 05:53:49,784] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory:  used = 52.58 GB, percent = 41.9%
[2023-12-01 05:54:12,066] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2440
[2023-12-01 05:54:12,099] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python', '-u', '../train_sft.py', '--local_rank=0', '--max_len', '200', '--dataset', 'OpenOrca', '--dataset_probs', '1.0', '--train_batch_size', '50', '--micro_train_batch_size', '1', '--max_samples', '500', '--pretrain', 'Llama-2-7b-hf', '--save_path', './ckpt/7b_llama', '--save_steps', '-1', '--logging_steps', '1', '--eval_steps', '-1', '--ckpt_path', './ckpt/7b_llama/checkpoints_sft', '--zero_stage', '2', '--max_epochs', '1', '--bf16', '--flash_attn', '--learning_rate', '5e-6', '--adam_offload', '--gradient_checkpointing'] exits with return code = -9

And i use the provided nvidia-docker and modified train_sft_llama.sh is
set -x

read -r -d '' training_commands <<EOF
../train_sft.py \
    --max_len 200 \
    --dataset OpenOrca \
    --dataset_probs 1.0 \
    --train_batch_size 50 \
    --micro_train_batch_size 1 \
    --max_samples 500 \
    --pretrain Llama-2-7b-hf \
    --save_path ./ckpt/7b_llama \
    --save_steps -1 \
    --logging_steps 1 \
    --eval_steps -1 \
    --ckpt_path ./ckpt/7b_llama/checkpoints_sft \
    --zero_stage 2 \
    --max_epochs 1 \
    --bf16 \
    --flash_attn \
    --learning_rate 5e-6 \
    --adam_offload \
    --gradient_checkpointing
EOF
    # --wandb [WANDB_TOKENS]

if [[ ${1} != "slurm" ]]; then
    export PATH=$HOME/.local/bin/:$PATH
    CUDA_VISIBLE_DIVICES=0 deepspeed $training_commands
fi

I have downloaded dataset and model locally.

I have tested it, and it works well on my side, perhaps because your CPU memory is small?

thx for your reply:)


When binaries are built with GH actions, it does not move/upload the DLLs, or the SND folder

Easy fix, just need to remember to rewire these. Opening this issue so I surely don't forget.
The sound library stuff if it can't find the file should be non fatal, but no sound will play of course. Not finding the dll is very very fatal.

no DLLs plz and thx


Any chance you could tag and build a new release?

Please could you tag and build a new release? From the commits/issues, it seems a feature or two has been added since the last release in 2018, and the last commit was in 2019. We're getting close to 2021.
I can try and build one, but it wouldn't be the same as having a new Official Release™...

I will try to find some time to figure this out. I need to renew the code signing certificate for the release, which is a pain. And would you believe me if I told you that I don't currently own a working computer that can run Windows? :|

Indeed I do believe you; don't blame you with the shenanigans Microsoft's been pulling. Thank you so much for replying. Best wishes, and Merry Christmas!

@dziemborowicz - Microsoft has free VM's running Windows 10 & dev tools (like Visual Studio 2019 & VS Code) available on their web site. That should get you a "working computer" running Windows quickly, if that's feasible for you. https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/

Apologies to everyone waiting for this release for so long. I have got a working(-ish) Windows computer again, I've got my updated code signing certificate, and this should be ready for a new release this weekend. Just fixing and verifying some bugs introduced with the recent changes.

v1.11.0 is now tagged and released.


Uploader sort order issue

It seems that there is a defect in the directory sorting within uploader.py that results in older directories and files not being cleaned up.  I suspect it has something to do with the listdir_by_creation_date() not actually sorting by creation date, but by last modified date instead.  Basically, if a number of days worth of drives backs up on the EON, and upload speed of the internet connection isn't fast enough to actually catch up, things go haywire.  I have files and folders that are over a week old, yet I see drives that have uploaded from drives today in cabana.  I've also noticed that the timestamps on the directories don't match the folder names AT ALL.

I assume the modified date was changed when logs were bzipped. Currently this is done on the fly.
Therefore I assume this issue is now fixed. Feel free to reopen when you still have problems.

I don't think the zipping is the cause of the issue.  It seems that the uploader is uploading the rlog files from all existing directories, then uploading the video files from the directories.  It deletes the rlog files as it goes, which updates the modified date of the directory.  If the upload speed is not fast enough to completely empty the sdcard folder before the next drive occurs, the existing video files will remain until after the rlog files from the next drive get uploaded.  In my case, I was accruing days worth of video files that would never upload, so I just had to keep purging them manually.
I think the fix for this would be to A) fix the directory sort issue, and B) upload the rlog and video files as pairs.


duration encoding/parsing is broken

encode_Duration only encodes days,
parse_Duration only encodes days,months,weeks or years

note: briefly discussed with @ewenmaclean and @c-cube, the current use of duration to represent FIX's tenor is not good, we should have a custom tenor type implemented as a sum type, and the encoding/parsing of duration should not be conflated with this


How to convert Series to compliant Column?

Say I have a function which takes a pandas Series:
def calculate_std_of_series(ser: pd.Series):
    return ser.std(ddof=1)

I'd like to convert this to support any standard-compliant library. How can I do that?
I'd like to be able to do something like
def calculate_std_of_series(ser: Any):
    ser_compliant = ser.__series_standard__()
    return ser_compliant.std(correction=1)

Currently, I can't do that, because we don't have __series_standard__.
Are we OK to require the libraries implementing the Standard have a __series_standard__ method in whichever object they use to back their Column objects?

Context of why this is necessary: in plotly, it's possible to pass a Series to 'x' / 'y' / etc:
df = pd.DataFrame({'x': [1, 2, 3], 'y': [1, 3, 2]})
px.line(df, x='x', y=pd.Series([1,2,3]))

I expect there'll initially be pushback (just something I've come to expect 😄 ) - if you do disagree that we should do have __series_standard__, could you please suggest an alternative for writing library-agnostic functions which operate on Series?
And if the answer is that we don't want to support that, then could you please suggest how plotly could support
px.line(df, x='x', y=pd.Series([1,2,3]))

in a library-agnostic manner?

One alternative would be to pass a namespace to the function:
def calculate_std_of_series(ser: Any, namespace):
    ser_compliant = namespace.dataframe_from_dict({'col': ser}).get_column_by_name('col')
    return ser_compliant.std(correction=1)

but this feel unnecessarily complicated (not to mention that the particular namespace may not be known at the given place in the code)

Didn't we just add .column to address a gap here? It feels like we should have all the pieces already without needing another dunder method.
It's hard to say more without more context; it sounds like this is a private function? And is there a need for backwards compatibility with it?  Your proposed return ser_compliant.std(correction=1) changes the return type, and it's not clear to me if that's fine here.
Perhaps we should talk through how array-api-compat and the scikit-learn/SciPy usages of that work. We went through several iterations of adoption approaches for questions like these, and it would be useful to take the learnings from that.

.column only works if you already have a standard-compliant Column

Your proposed return ser_compliant.std(correction=1) changes the return type, and it's not clear to me if that's fine here.

it doesn't, the return value here is just a single value:
>>> ser = pd.Series([1,2,3])
>>> ser.mean()
2.0
>>> ser.rename('col').to_frame().__dataframe_standard__().get_column_by_name('col').mean()
2.0

We already have a way to go from non-compliant DataFrame to compliant DataFrame: __dataframe_standard__.
I'm saying: we need a similar way to go from non-compliant Series to compliant Series

We already have a way to go from non-compliant DataFrame to compliant DataFrame: __dataframe_standard__.

To clarify: we have that as an example on how to go about adoption, and not in the standard itself.

I'm saying: we need a similar way to go from non-compliant Series to compliant Series

That's fine to add in the purpose and scope section I'd say, same as for dataframes. And then in dataframe-api-compat it can be used for specific libraries that are supported there.
I think you can just implement it there and see how it works? It's not a standardization question.

I think you can just implement it there and see how it works? It's not a standardization question.

But there is no equivalent __dataframe_standard__ for Series (column objects), so I don't think you can currently "just implement it"?

To clarify: we have that as an example on how to go about adoption, and not in the standard itself.

I don't think it's just an example: if a library has a compliant namespace, then __dataframe_standard__ is required for going from non-compliant to compliant
https://github.com/data-apis/dataframe-api/blob/04650bae8c0252e64ffda451621936e9962bc1ed/protocol/purpose_and_scope.md#L53-L67

I don't think you can currently "just implement it"?

For now I'll implement it the same way that I've implemented __dataframe_standard__, i.e. by monkeypatching: https://github.com/data-apis/dataframe-api-compat
pd.Series.__column_namespace__ = pandas_standard.convert_to_standard_compliant_column
pl.Series.__column_namespace__ = polars_standard.convert_to_standard_compliant_column

(this is good enough for testing, but longer term we would upstream __dataframe_standard__ and __column_standard__ methods to each library with a Standard-compliant namespace)

Oh wait, I was going by Marco's statement and a quick grep. It's actually in the protocol "purpose and scope" section, not in the API standard one: https://github.com/data-apis/dataframe-api/blob/2c94312572ff58031e19cc8a3b7fe1226164f4fe/protocol/purpose_and_scope.md#L75.
So there's no analogy at all here, the API standard has nothing for a dataframe either? And to go to a standard object, the answer so far is "use a constructor function".

if a library has a compliant namespace, then __dataframe_standard__ is required for going from non-compliant to compliant

That's an example, not a required method.

I think we discussed that as essentially a requirement, though

It's actually in the protocol "purpose and scope" section, not in the API standard one:

I just noticed the same (looking for that example), but shouldn't we move that to the spec docs?(or have it in both places)

If __dataframe_standard__ isn't required, then how is someone meant to write a library-agnostic dataframe function?
Let's take the example from the blog post:
def remove_outliers(df, column: str):
    z_score = (df[column] - df[column].mean())/df[column].std()
    return df[z_score.between(-3, 3)]

Could you please write out exactly how to make this function library-agnostic without using __dataframe_standard__?

__dataframe_standard__ is on the front page of the blog post https://data-apis.org/blog/dataframe_standard_rfc/, and is mentioned there in "step 1" of the "use the DataFrame Standard" solution
If it's not required, then we need to align on this as soon possible as it looks like there's been some serious miscommunication

FWIW this blog post doesn't appear at https://data-apis.org/blog/ or is not linked on the home page, so it's not easy to find (unless you know it linked from some other post)

Thanks, gh-169 does clarify this. We've indeed agreed on it, but we have simply forgotten to document that agreement. So I guess we need to do that, and then having the analogous method for Column seems pretty obvious (I hope it'd be called __column_standard__ rather than __series_standard__ though).

agree on __column_standard__ instead of __series_standard__
I'll document this better then, but to summarise:

__dataframe_namespace__: check if a dataframe is already standard compliant
__column_namespace__: check if a column is already standard compliant
__dataframe_standard__: go from not-necessarily-compliant dataframe to compliant dataframe
__column_standard__: go from not-necessarily-compliant column to compliant column
DataFrame.dataframe: go from compliant dataframe to not-necessarily-compliant dataframe
Column.column: go from compliant column to not-necessarily-compliant column

Do we need a separate __column_namespace__? It would return the same object as __dataframe_namespace__, and we could add another attribute (eg ndim) to distinguish dataframe from column
Other option for __column_standard__ is the longer __dataframe_column_standard__ (it's longer, but I think clearer)

sure, we can rename later, for now I'm just trying to plug holes in what we've got so that the standard can be useful

Thanks, that list LGTM! If we had that bullet list in a single place, that would really clarify things.

"use a constructor function"

Good point, I suspect that we may return to that point again and will want a direct Column constructor at some point.

I hope I'm not coming across as confrontational

not at all!

This issue also mentioned a "column_namespace" dunder, which isn't resolved yet?

isn't it?
https://github.com/data-apis/dataframe-api/blob/2c94312572ff58031e19cc8a3b7fe1226164f4fe/spec/API_specification/dataframe_api/column_object.py#L23-L25


Implement basic Hoon testing pattern.

Current status:  Hoon test cases have been implemented.  These need to be double-checked and then run against the target Jock files.

This has essentials for testing code, barring a supporting Python script which generates Hoon test cases.
There's some question still of what the best use pattern would look like, and that tends to mili


treeview started as closed need another refresh (or re-opening) to show content

Issue Type: Bug
I'm developping an extension, and i've noticed a problem that i can also reproduce with the gitlens default extension in vscode. (therefore prompting me to think it's an issue with vscode and not my extension)
Make sure the 'commits' section of gitlens is closed when
closing vscode.
Restart vscode, goto the git lens view, open the commit tree view. Notice it's empty.
You'll have to hit the 'refresh icon' on the right of its title bar to populate it correctly. This was not the case a couple weeks ago.
(Thanks for your great product nonetheless you guys!)
 Can you point me to another tree view where you also see this issue?

@alexr00 Thank you for your time and consideration!
I've investigated this last night to try and get to the bottom of it. Turns out I had an event tied to the change of visibility of the treeview, and if visible === true, then I launched a  'refresh root' of the tree, which seemed to 'glitch out' that said refresh cycle. (is it intended/normal?)
(I re-use the same instance of the tree provider for two instances of the tree view: one in the explorer panel, and another one in a view-container that My extension provides. And so I thought I had to refresh the tree manually if uncovered by the user, either by unfolding it or by switching to the other view-container.)
I've noticed that, if the treeview was never populated, vscode launches a 'refresh cycle' itself upon revealing a treeview. (i verified by adding console.logs in the 'getChildren' method of my treeview provider)  And me also calling  this._onDidChangeTreeData.fire(); was causing glitching in the treeview it seems, preventing it to refresh properly.
Workaround : I've solved it by debouncing my calls to  this._onDidChangeTreeData.fire() by 80 milliseconds.
(perhaps I only should try to refresh if I know my tree structure has changed - instead of also refreshing when I detect a visibility change that tells me a treeview was uncovered / unfolded and made visible.)
I'll let you close this issue if you see fit, as I'm not sure anymore if it's actually a bug that calling the refresh-root procedure as it's actually already refreshing bugs the treeview by not refreshing at all.
Many thanks again for your attention to the issue I raised! You Rock!! :)

Thanks for the investigation @boltex!
I'll take some time to make sure the behavior you're seeing is intended before closing.

Sidenote: I wonder if this is related to https://github.com/microsoft/vscode/issues/127204 (which is a duplicate of   https://github.com/microsoft/vscode/issues/110450) ... Which is the one big remaining bug in treeviews/sidebar that makes extensions somewhat look 'buggy'... :/

Started investigation, but can't finish right now. Some notes for when I pick it back up:

When two refresh requests come in back to back, we cancel the first one.
Cancellation: https://github.com/Microsoft/vscode/blob/8f774f132e036347c018b9cc69c79a178322dacb/src/vs/workbench/api/common/extHostTreeViews.ts#L523-L524
Where the cancellation is checked: https://github.com/Microsoft/vscode/blob/8f774f132e036347c018b9cc69c79a178322dacb/src/vs/workbench/api/common/extHostTreeViews.ts#L499-L509

@alexr00  Another sidenote, in case it can help with this directly-related issue: @eamodio also seems to consider this a bug https://github.com/eamodio/vscode-gitlens/issues/1582 when I tried to reproduce & pointed out this behavior in his extension.

/verified
@alexr00 I want to thank you personally for this fix. Helped a lot with my extension. :)


Fixed type issue for RuleDocsInfo

🚨 IMPORTANT: Please do not create a Pull Request without creating an issue first.
Any change needs to be discussed before proceeding. Failure to do so may result in the rejection of the pull request.
Description
Type issue fix for RuleDocsInfo
Fixes # (1146)
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Screenshots/Sandbox (if appropriate/relevant):
Adding links to sandbox or providing screenshots can help us understand more about this PR and take action on it as appropriate
How Has This Been Tested?

TS Build errors relating to description field is gone (when strict flag is turned on)
No new TS Build errors

Test Environment:

OS: MacOS Monterey
@graphql-eslint/.3.11.2:
NodeJS: 16.16

Checklist:

[ ] I have followed the CONTRIBUTING doc and the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests and linter rules pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

Hey, thanks, I'll take care of this PR when I will enable strict: true in v4!


TypeError when using from_map on enumerate object

What happened:
When attempting to use from_map with an enumerate object as the input iterable, I run into an uncaught TypeError when trying to compute the length of the iterable:
TypeError                                 Traceback (most recent call last)
Input In [1], in <cell line: 10>()
      6     x = t[1]
      8     return pd.Series([x] * size)
---> 10 dd.from_map(func, enumerate(["A", "B"]))

File ~/dask/dask/dataframe/io/io.py:953, in from_map(func, args, meta, divisions, label, token, enforce_metadata, *iterables, **kwargs)
    949     raise ValueError(
    950         f"All elements of `iterables` must be Iterable, got {type(iterable)}"
    951     )
    952 try:
--> 953     lengths.add(len(iterable))
    954 except AttributeError:
    955     iterables[i] = list(iterable)

TypeError: object of type 'enumerate' has no len()

What you expected to happen:
I would expect this error to be caught, as it seems like the except case would've resolved this by creating a list from the enumerate object.
Minimal Complete Verifiable Example:
This fails:
import pandas as pd
import dask.dataframe as dd

def func(t):
    size = t[0] + 1
    x = t[1]

    return pd.Series([x] * size)

dd.from_map(func, enumerate(["A", "B"]))

Creating a list from the enumerate object before calling from_map works:
import pandas as pd
import dask.dataframe as dd

def func(t):
    size = t[0] + 1
    x = t[1]

    return pd.Series([x] * size)

dd.from_map(func, list(enumerate(["A", "B"])))

Environment:

Dask version: latest main
Python version: 3.9
Operating System: ubuntu 20.04
Install method (conda, pip, source): source

Looks like a little bug. Do you or @rjzamora have time to open a PR to fix?


Solutions for simplification notebook

@sylee957 do you have solutions for the simplification notebook? If so, I think you forgot to push it.

I didn't present every exercises with solutions though.

We should do that.
By the way, I think the simplification notebook can be trimmed a bit. It's a little too long to fit into the time period. We can move the less important bits to the end so that we can show them if we have time. Some things like assumptions should be moved to the front so that we definitely get to them. Or we could even move assumptions into the intro notebook since they are really important.

I'm about to step on a plane. I will look at seeing if I can make a "solvers" notebook, and we can discuss what to do with it. Some of my later notebooks are somewhat short so we might have room to include it. Or we can trim something. Feel free to update the existing notebooks.

Anyway, why did you delete the exercise about printing and copy-pasteing the expression in LaTeX though
(I think that it's quite common usage though.)

I'm doing the redux on the part

Introduction: Remove sympify because that are better be done in parsing
Introduction: Remove function symbols because that is rarely used in the tutorial

I've also moved some stuff like pattern matching, to the end of the section such that
it is only going to be explained if there is enough time.

In fact, assumptions are the things I want to drop because it is quite complicated and shows ambiguous behavior
(Trying to fiddle with assumption for how to make some simplify, solve, integral work wasn't good experience for using CAS)
If simplify is explained only with polynomials and trigonometry, it isn't needed anyway.
Maybe it can be only needed before things like integral, but if I'd keep the easy and obvious things, I'd keep only polynomials and trigonometry.

I would only think of
polynomial solve (solveset or roots)
linear solve (linsolve or rref)
Because they are the most commonly used solvers we get query of.
On Sun, Jul 9, 2023 at 7:33 PM Aaron Meurer @.***>
wrote:

I'm about to step on a plane. I will look at seeing if I can make a
"solvers" notebook, and we can discuss what to do with it. Some of my later
notebooks are somewhat short so we might have room to include it. Or we can
trim something. Feel free to update the existing notebooks.
—
Reply to this email directly, view it on GitHub
https://github.com/sympy/scipy-2023-tutorial/issues/16#issuecomment-1627882455,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIKTPTPZ2TII7H66EJLD5HTXPNEULANCNFSM6AAAAAA2D2CNGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
S.Y. Lee

Anyway, why did you delete the exercise about printing and copy-pasteing the expression in LaTeX though
(I think that it's quite common usage though.)

I only meant to delete the codegen stuff, because that's shown in the later notebook. If I deleted some stuff about latex that was an accident.

Assumptions is one of the most important things to know about when using SymPy. So we definitely need to show it.
I pushed up a notebook for solvers. We can figure out tomorrow what to do with it.

I have added the solutions so far


Eilidh testing for editor guide. added full stop

I'm testing making edits as someone outwith the organisation for the editors' guide. Please ignore.

@EilidhRoss1 👏


Delivery services dashboard widget

What does this PR do?
This adds a delivery services widget to the dashboard page of traffic portal. It displays a list of all of the user's delivery services and shows a bandwidth chart when a service is clicked.
Fixes #(issue_number)
Which TC components are affected by this PR?

[ ] Documentation
[ ] Grove
[ ] Traffic Analytics
[ ] Traffic Monitor
[ ] Traffic Ops
[ ] Traffic Ops ORT
[x] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[ ] Other _________

What is the best way to verify this PR?
Run traffic portal locally, login, and see the new widget near the top of the page.
Check all that apply

[ ] This PR includes tests
[ ] This PR includes documentation updates
[ ] This PR includes an update to CHANGELOG.md
[x ] This PR includes all required license headers
[ ] This PR includes a database migration (ensure that migration sequence is correct)
[ ] This PR fixes a serious security flaw. Read more: www.apache.org/security

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2441/
Test FAILed.

a couple things

can you add some labels to the tooltip?

maybe something like
16:02 UTC
45 Gbps (or whatever metric is being displayed)

I'm not sure the time on the tooltip is actually UTC.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2494/
Test FAILed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2495/
Test PASSed.

can you add the chart.js library via the bower.json approach? it makes it easier to  upgrade the dependency if we need to. here is an example of how to do that
https://github.com/apache/trafficcontrol/pull/2871

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2511/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2512/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2514/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2515/
Test FAILed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2516/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2518/
Test FAILed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2529/
Test FAILed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2530/
Test FAILed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2534/
Test FAILed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2543/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2573/
Test FAILed.

retest this please


Oil spawn in unbuildable location

Just need to make all the terrain in that area Rocks instead of the weird Tiles it currently has under it. Oil Rigs can only spawn on those tiles.


Opening multiple windows causes crash on Linux

Hi there folks, I was trying to combine an Avalonia linux app (https://avaloniaui.net) and use Photino as an embedded browser. I can start one instance of the Photino window and it works fine. However, when I close the window and try to open a new one, the entire app crashes with no stack trace. I've debugged the issue to be something in the C++ native Linux code, but all my attempts at solving the issue in C++ have not been successful. I'd greatly appreciate any ideas you may have in fixing this. Thanks for the great work!

Hi @RatikKapoor. Thank you for the kind words. We have not worked with Avalonia, but are interested in tracking this down for you as we're working to get a new 2.0 version tested and ready and would like to include all the fixes we can. Can you tell us which Linux distro you're using and if you're using Photino.NET or Photino.Blazor? So that we can try to reproduce.

Hi @MikeYeager apologies for the late reply. My specs are as follows:
System: Ubuntu 20.04 VM
Version: Photino.NET
I've managed to isolate how to repro the issue. Firstly, if I'm not running the Photino window on the main Avalonia thread, the second time I launch a window it will definitely crash. Secondly, if I call photinoWindow.Dispose() it will also crash second time around. I'm guessing here since IDisposable is implemented Dispose() is automatically called when the object falls out of scope. Calling it manually + a second time probably causes some null ref issue/segmentation fault Photino second time it is launched.
I think the priority would be being able to launch a Photino window in a separate thread using Task.Run(...) reliably multiple times. If it helps I can push a sample where I can repro this issue and provide it to you!
This likely belongs in a separate issue but is an OnNavigated callback planned for the 2.0 release? This would be very useful in cases where the web control is redirected somewhere for google or SAML authentication and then we want to handle that redirect event in C#. Currently there's no event listener for when the browser is navigated. I have fully functional code for this in my local clone of this repo. I would be happy to PR this functionality and see it merged upstream here since this is a requirement for our use case :).
Thanks again for your help!

That makes sense. Photino needs to run on an STA thread, generally, the UI thread as it's a UI app. It also makes sense that calling Dispose() more than once is causing a crash. Dispose() will be different in V2. We'll try to add some code to prevent a crash if it's called multiple times. It's mainly being used in V1 to clean up child windows. We've now got the Windows portion of V2 working and almost ready for prime time. Linux and Mac coming next, but those tend to be a lot easier. Multi-window support on Windows was a major feature for V2 and required some big internal changes. It seems to be working very well now. The first window created runs the message loop and when that window is closed, all of the child windows close as well.
You can register custom schemes (other than http, https, file, etc.) and handlers in all versions, but we don't have an OnNavigated for standard schemes. I will look into adding handlers for standard schemes. It will have to be supported on all 3 native browser controls. Should be easy to hook up if it's supported. New features will be much easier to add in V2 if we can get it out the door.

Gotcha! This all sounds great, in that case I'll wait for a V2. If there's anything I can do to help test please let me know!

@RatikKapoor v2 of Photino has been released. Please give it a try and let us know if you see any issues.


problem in getting events from server for particular club in club_info.js

$http is creating error when passed in the club_info.js controller.

The $http should be used in postPromise the problem is getting bigger and bigger. My suggestion is to use backend to store clubs and society so they can be changed without any problems in future.


Adding frame number in timelapse

Hi,
I find it useful to get the frame number in the timelapse. That way it is way easier to find interesting images for meteors or other events.
To do this, I've just added this parameter to the ffmpeg command:
-vf "drawtext=fontfile=Arial.ttf: text='%{frame_num}': start_number=1: x=(w-tw)/10: y=h-(2*lh): fontcolor=black: fontsize=10: box=1: boxcolor=white: boxborderw=5"
The info was obtained from https://stackoverflow.com/questions/15364861/frame-number-overlay-with-ffmpeg
I would love to see this integrated as an option. If it is not, then this issue will be a good reference for those that want to change the timelapse script manually.

Hi,
I find it useful to get the frame number in the timelapse. That way it is way easier to find interesting images for meteors or other events.
To do this, I've just added this parameter to the ffmpeg command:
-vf "drawtext=fontfile=Arial.ttf: text='%{frame_num}': start_number=1: x=(w-tw)/10: y=h-(2*lh): fontcolor=black: fontsize=10: box=1: boxcolor=white: boxborderw=5"
The info was obtained from https://stackoverflow.com/questions/15364861/frame-number-overlay-with-ffmpeg
I would love to see this integrated as an option. If it is not, then this issue will be a good reference for those that want to change the timelapse script manually.

Can you provide the full FFMPEG command? I am not sure at what part of the command to add this to. Front, Middle, etc.

A future version of Allsky will include a TIMELAPSE_PARAMETERS variable in config.sh that can be used to add parameters to the ffmpeg command.


iMODSEAWAT: dsp-dmcoef is written instead of al

In GitLab by @VinceKaandorp on Dec 18, 2018, 14:57
This doesn't seem right:
[dsp]
    al_l1 = c:\testbank_imod-WQ\Notebook\HenryCase\dsp\dsp-dmcoef_l1.idf

The wrong path is written for al_l* fields, possibly for the other [dsp] fields as well.

In GitLab by @visr on Jul 22, 2019, 11:50
closed

In GitLab by @Huite on Dec 18, 2018, 15:52
The problem appears to be that it does not throw an error when other "dsp" are defined, but these do get added to the "dsp" package dictionary inside of imod.run.
These values are almost always scalar, so for now it should just throw an error such as "dmcoef" is not a field of "dsp", or something.
This also has to be fixed in the HenryCase example!


enables highly_variable_genes_seurat_v3 to accept pseudocounts

fixes #1642 .
I'll reply to the comments here @adamgayoso

shouldn't both not be Optional?

I removed optional type from both span and check_values

On another note, in this current state check_nonnegative_integers is an unused import. Do you guys check for this with flake8?

we don't have pre-commit in place, we are discussing it here #1563 . I do check flake8 but clearly didn't do it this time.

I think a bug has been introduced @ivirshup @giovp. Namely, the call to check_nonnegative_integers(X) was removed (should be here on line 61) if returns False, raise the warning

this should be fixed now
Sorry again for very sloppy handling of this, should be ready to review

we don't have pre-commit in place, we are discussing it here #1563 . I do check flake8 but clearly didn't do it this time.

I thought at one point you guys were checking flake8 with CI.

I thought at one point you guys were checking flake8 with CI.

We were, sorta. The CI tool we were using had pretty stochastic reporting (which isn't really what we want in a CI tool). Hopefully it'll be back in a more reliable form soon: https://github.com/theislab/scanpy/issues/1563#issuecomment-784922713

@meeseeksdev backport to 1.7.x


API for multilanguage?

How can i check if player have other language? Or not exist an api for that?

Do you mean "I want to send messages that match the language of the player, not the language of the server"?

Yes

Ok, I'll implement this on v2.4.2

Done this on 3.0.0

Actually not. I mean about on floatingtext support multilanguage, like this plugin on nukkit: https://nukkitx.com/resources/holograms.42/
Not about commands language. @fuyutsuki

@NycuRO
It's not planned to be provided by the standard function.
Please implement by yourself using TexterApi etc.


InvalidOperationException: No Entity Framework provider found for the ADO.NET provider with invariant name 'System.Data.SqlClient'.

Hello - I'm getting this runtime exception.
My solution details:
Target Framework = .NET 4.6.1

VS 2017 15.2 (26430.120
A class library project (repositories library) using DbContext (using Entity Framework 6.3, not EF Core)
ASP.NET Core 1.1 API project referencing the repositories project.
Running any query using DbContext throws this exception.

Adding the <entityFramework> in the App.config of the API project doesn't help. I suspect that the build is not copying EntityFramework.SqlServer.dll to the API project's bin folder.
Adding Entity Framework package to the API project directly does copy the EntityFramework.SqlServer.dll to the bin folder during build. However, I don't prefer to do that. Rather I would like to maintain the indirection through the repository project.
Any suggestions? (again, I'm not using EF Core, can't take chances in this critical project).
Thanks

This issue was moved to aspnet/EntityFramework6#311


Remove usage of apache httpclient for more product usages

Currently we use apache.commons.httpclient for http connection. It would be better to use the JDK internal HttpCLient or one of Intellij bundled version.
So it is possible to support product like Jetbrains Gateway e.g.

usage of bundled http client in IDEA product (currenty apache httpclient)


WIP: IFU-528: presave function

Todo figure our draft flow
Code is triggered by save

Currently the code will add the new paragraph aded to Finnish version to all other languages to the same position.
The problem is that without the draft flow the paragraph most likely containing Finnish text will end up in other languages published if there is no draft.
One way would just use this code with outside links and for everything else do the paragraphs manually


Style elements within text editors using the atom-text-editor::shadow selector or the .atom-text-editor.less file extension.If you want to target overlay elements, target them directly or as descendants of atom-overlay elements.

In stylesheets/supercollider.less: Style elements within text editors using the atom-text-editor::shadow selector or the .atom-text-editor.less file extension.
If you want to target overlay elements, target them directly or as descendants of atom-overlay elements.

I've got all of these done and will be doing a release shortly.  thanks!
On Sat, May 30, 2015 at 2:36 PM bgc<EMAIL_ADDRESS>wrote:

In stylesheets/supercollider.less: Style elements within text editors
using the atom-text-editor::shadow selector or the .atom-text-editor.less
file extension.
If you want to target overlay elements, target them directly or as
descendants of atom-overlay elements.
—
Reply to this email directly or view it on GitHub
https://github.com/crucialfelix/atom-supercollider/issues/39.


docs(cli): add aliases for options

https://github.com/webpack/webpack-cli/pull/2700
To be merged post the release.

Let's also add for --output https://github.com/webpack/webpack-cli/pull/2709