Hue Bridge -  API version upgrade to 1.53.0

Issue
After updating / restart of my Homebridge I am getting the following warning. See logs (API version recommendation)
Log Messages

[30/09/2022, 21:35:06] [Homebridge-Hue] Philips hue: Signify Netherlands B.V. BSB002 bridge v1953188020, api v1.53.0
[30/09/2022, 21:35:06] [Homebridge-Hue] Philips hue: warning: not using recommended Hue bridge api version 1.52.0

Hadn’t seen that before, but indeed, my Hue bridge has upgraded its firmware as well.  The bastards haven’t published any release notes for over two years, since 1.39.0, so I’ll have to check for any changes myself.  In the meantime, please report if you run into any issues.

I'll do.
Thanks @ebaauw !

Hi, short feedback:
Alonso working fine and as expected with the newest firmware.
Thanks JO

In v0.13.47.


DISTX-415 Reduce the amount of time it takes to add new nodes

In this commit, we optimize the salt execution times by removing the high state apply after each phase of recipe execution.
There is only one high state apply in all the nodes after a scale-up.
We could still optimize the individual scripts like metering which always removes and adds the same package in all the nodes of the cluster for every scale-up.
This is the first time modifying something in salt, so I will focus on further optimization after getting initial feedback.
Also removed the deprecated pre/post-execution phases. Made sure that there are no existing salt scripts for such phases.

./gradlew build && test scale-up in a private stack to see the time getting reduced from 5 mins to 4 mins.

The integration tests failed because of the optimizations. I could fix them if this patch is generally agreed upon.
test_1                | 23:21:32.919 [TestNG-test=newway mock testcases-2] ERROR c.s.i.cloudbreak.context.TestContext [RecipeClusterTest.testRecipePreTerminationRecipeHasGotHighStateOnCluster] - MockVerification
test_1                | java.lang.RuntimeException: /saltapi/run with bodyBody: state.highstate Times: -1 request should have been invoked exactly 5 times, but it was invoked 3 times

So we add recipe grains and then we remove them. If there is no highstate in between what is going to execute the recipes?

I think it would make a real difference if we would target only new nodes during upscale, like it's mentioned here: https://github.com/hortonworks/cloudbreak/issues/4675

@cegganesh84 , for metering, it's not the same package, so if it removes something and add something, that's totally valid to keep. (i thought probably it's time to remove the old package cleanup, but if it's still executing the remove, we cannot do that...as those different rpms using the same folders)

So we add recipe grains and then we remove them. If there is no highstate in between what is going to execute the recipes?

So @keyki I thought this line executes the recipes            saltCommandRunner.runSaltCommand(sc, new SyncAllRunner(targetHostnames, allNodes), exitCriteriaModel, maxRetry, exitCriteria);. I thought this applies only to the newly added grains as opposed to the full state. If that is not the case it is my misunderstanding and would like to know the method to apply only the existing state.

I think it would make a real difference if we would target only new nodes during upscale, like it's mentioned here: #4675

That is a very good suggestion. It would certainly remove the overhead for the salt masters, and in few cases like external repo like pypi.

@cegganesh84 , for metering, it's not the same package, so if it removes something and add something, that's totally valid to keep. (i thought probably it's time to remove the old package cleanup, but if it's still executing the remove, we cannot do that...as those different rpms using the same folders)

Ah, I see. I did not look at it more closely. Thanks! Some investigation is still needed as to why I see the rpm -i is still getting executed on new nodes.

So we add recipe grains and then we remove them. If there is no highstate in between what is going to execute the recipes?

So @keyki I thought this line executes the recipes  saltCommandRunner.runSaltCommand(sc, new SyncAllRunner(targetHostnames, allNodes), exitCriteriaModel, maxRetry, exitCriteria);. I thought this applies only to the newly added grains as opposed to the full state. If that is not the case it is my misunderstanding and would like to know the method to apply only the existing state.

So SyncAllRunner just makes sure that all states and pillars and caches are up to date, but it does not execute anything. To target only the new instances with highstate would be like salt 'host1,host2,host3' state.highstate. The target needs to change from '*' to specific minion IDs. However, I'm afraid we'll run into troubles as some things need to be done on the master instance as well when new nodes are added to the cluster. Like we generate the TLS agent tokens on the master instance for all instances and we copy them over from there.

.... However, I'm afraid we'll run into troubles as some things need to be done on the master instance as well when new nodes are added to the cluster. ...

I meant it only for recipe execution

So we add recipe grains and then we remove them. If there is no highstate in between what is going to execute the recipes?

So @keyki I thought this line executes the recipes  saltCommandRunner.runSaltCommand(sc, new SyncAllRunner(targetHostnames, allNodes), exitCriteriaModel, maxRetry, exitCriteria);. I thought this applies only to the newly added grains as opposed to the full state. If that is not the case it is my misunderstanding and would like to know the method to apply only the existing state.

So SyncAllRunner just makes sure that all states and pillars and caches are up to date, but it does not execute anything. To target only the new instances with highstate would be like salt 'host1,host2,host3' state.highstate. The target needs to change from '*' to specific minion IDs. However, I'm afraid we'll run into troubles as some things need to be done on the master instance as well when new nodes are added to the cluster. Like we generate the TLS agent tokens on the master instance for all instances and we copy them over from there.

Thank you, that was helpful. I have added a new command StateAllRunner to run the specific recipe execution phase. Also will take up targeting the specific scaled up host in another PR.

LGTM. I'll merge once it's green. Re-triggered IT.

For larger node count it turned out that the highstate is inefficient, there are multiple reasons for it:

it seems that many things happening on master side and basically Salt master is consuming a lot of  CPU, even if the machine has 8 cores we are saturating it
we have a lot of inefficient states, which take a lot of time even they do nothing
it seems that minions are downloading files from master and it is also slow
we have a few states that are continuously doing something like fluentd states (although they are not slow in general)

What we learned that the pillar size is what has caused a lot of performance issues. If the pillar size is large then even if you execute a single state the master is struggling.
We tried to do state caching, changing pillar structures from yaml to json etc. but basically they didn't help that much. I think we need to put focus again and analyze the states and check the caching again, maybe even updating Salt from 2017 version to a newer one to see whether there are some performance differences. But I think I can raise a separate jira for this.
Further optimization possibility: completely avoid recipe execution if the end-user has not attached any recipe.
Although separating recipes probably ok, I wouldn't like to go down further on this path, I mean separating things out instead of just executing a highstate, because that makes the orchestration really simple, and it is super easy to keep compatibility.

Thanks @akanto for looking into this.
@keyki thanks for the review. I see that I could scale up fine with this PR, but not scale down, so looking into the issue. This is the error I am getting,
ew node(s) could not be removed from the cluster. Reason com.sequenceiq.cloudbreak.orchestrator.exception.CloudbreakOrchestratorFailedException: Failed: Orchestrator component went failed in 3 min(s), message: There are missing nodes from job (jid:<PHONE_NUMBER>0357866451), target: [gk1custom-compute2.gk1priv.xcu2-8y8x.cloudera.site]
Node: gk1custom-compute2.gk1priv.xcu2-8y8x.cloudera.site Error(s):
Name: ipa host-del gk1custom-compute2.gk1priv.xcu2-8y8x.cloudera.site --updatedns && ipa-client-install --uninstall -U
Comment: Command "ipa host-del gk1custom-compute2.gk1priv.xcu2-8y8x.cloudera.site --updatedns && ipa-client-install --uninstall -U" run
Stderr: ipa: ERROR: did not receive Kerberos credentials

@keyki I have fixed one issue with respect to how salt state command was executed.
Still, the node decommission does not work.
I see that at some places there is a kinit before ipa commands, and it is missing in other places.
Present
    - unless: test -f /var/log/dnsrecord-add-executed || { echo $PW | kinit {{salt['pillar.get']('sssd-ipa:principal')}} && ipa dnsrecord-find {{salt['pillar.get']('sssd-ipa:domain')}} --a-rec=$(hostname -i); }

Absent
    - name: runuser -l root -c 'ipa host-del {{ salt['grains.get']('fqdn') }} --updatedns && ipa-client-install --uninstall -U'

Could this be the reason for the failure with, ```Stderr: ipa: ERROR: did not receive Kerberos credentials

For the context, I scale up the cluster and immediately scale it down.

@keyki scale down worked after I added kinit in ipa_leave command. I will look at the integration test failures if any later today.

Thank you.

@keyki I have fixed the failing tests. Please review!


Replace in-house rand with absl::Random for backoff

Let's give it a try!

grpc_build_artifacts_multiplatform: running

Basic Tests Ruby Linux failure doesn't look relevant to this PR.


bazel: bazel_3 -> bazel_4

Description of changes

Updates the default version of bazel in nixpkgs from 3 to 4. 3 isn't LTS, but 4 is: https://blog.bazel.build/2020/11/10/long-term-support-release.html.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

cc @NixOS/bazel

@ofborg build python3Packages.tensorflow python3Packages.tensorboard python3Packages.jaxlib gvisor envoy pythonPackages.tensorflow pythonPackages.tensorboard pythonPackages.jaxlib bazel-watcher

Need to update some fetch derivation hashes.

Updated bazel-watcher here: https://github.com/NixOS/nixpkgs/pull/170192.


"ppartementen" should be appartementen in the chart

Great spot!
In the locales the label is marked as "Appartementen", the 'A' falls of the chart in this case.
I see 2 options to solve this:

Place an enter in the label; This might be a bit ugly, but is easy to fix.
Alter the space that is given to these labels in the files that create these graphs. This could be quite hard.

What do you think @mabijkerk ?

Maybe this is something @louispt1 can look into?


Speed extremely slow with populating relation dropdown (several minutes)

Describe the bug
We have a template that uses four different list fields - each one allowing the administrator to specify any number of items from a collection to be displayed on a page. The list is defined with a relation field widget, similar to the code below. When there were fewer than 30 items in the respective collections and fewer than 5 or so items selected for each of the four lists this was slow, but acceptable (taking a second or two to populate the relation field dropdown). With 140 and 60 items in each of the referenced collections (stories and news), and 77 selected from one collection and 8 from the smaller collection the waiting time for drop-down population seems to be on the order of several minutes, and this population takes place every time a new item is added to the list, or an item's relation drop-down is clicked to edit it. There seems to be no caching of values between drop-downs despite pulling from the same source:
      - label: "Featured Stories⁺"
        singular_label: "Featured Story"
        hint: "⁺will be displayed in the order provided above."
        name: "featured_stories"
        widget: list
        ui: fields
        required: true
        allow_add: true
        fields:
        - label: "Altas post/story"
          name: "story"
          widget: "relation"
          collection: "atlas"
          search_fields: ["title"]
          value_field: "uuid"
          display_fields: ["title", "thumb_new"]
          hint: "Start typing the title of the Atlas post/story then select from the dropdown list"
        - {label: "Don't show thumbnail in list", name: "no_thumb", widget: "boolean", default: false, required: false}

To Reproduce

Create a collection of 150 items,
Create a field with the above displayed definition, referencing this collection
Start adding items to the list and watch the performance take longer and longer until it's basically unusable for the average site editor

Expected behavior
I would expect that the performance for selecting in this sort of situation be good enough at least through a thousand or more items in the collection and at least a few hundred or more items selected from that collection
Applicable Versions:

netlify-cms-app 2.15.54
netlify-cms-core 2.53.0
netlify-cms 2.10.174
Git provider: Gitlab (was also a problem on Github, though may have been very slightly faster on github)
OS: Windows 10, Linux, MacOS
Browser version Chrome, Firefox, Safari (all the most recent versions)

CMS configuration
See relevant snipet, above
Additional context
If required, we can provide a copy of the repository with all config and collection files for analysis to a Netlify CMS engineer (not for direct public release)

Hi @delwin, related to https://github.com/netlify/netlify-cms/issues/4635

There seems to be no caching of values between drop-downs despite pulling from the same source:

Initial load can be slow as it needs to load all entries from the related connection. Once entries are loaded they should be cached, so that's definitely something we should look into.
I also experience GitLab to be slower than GitHub (I don't have official benchmarks though).
I'll try to create a test repo to simulate this case.
Can you share the browser network traffic (by opening the browser developer tools and going to the network tab)? The browser should indicate if requests are cached or not. For example this is how Chrome shows cached requests:

I can share the browser network traffic, but it goes on for ages - 10s of
thousands of requests. I'll try to get this for you. It's definitely not
caching.
I can provide you with a cloned repository for you to try this on, as well.
On Fri, 12 Nov 2021 at 14:52, Erez Rokah @.***> wrote:

Hi @delwin https://github.com/delwin, related to #4635
https://github.com/netlify/netlify-cms/issues/4635
There seems to be no caching of values between drop-downs despite pulling
from the same source:
Initial load can be slow as it needs to load all entries from the related
connection. Once entries are loaded they should be cached, so that's
definitely something we should look into.
I also experience GitLab to be slower than GitHub (I don't have official
benchmarks though).
I'll try to create a test repo to simulate this case.
Can you share the browser network traffic (by opening the browser
developer tools and going to the network tab)? The browser should indicate
if requests are cached or not. For example this is how Chrome shows cached
requests:
[image: image]
https://user-images.githubusercontent.com/26760571/141477653-954681f8-5b23-42a3-9799-21e7f5e3dbf9.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/netlify/netlify-cms/issues/5920#issuecomment-967133458,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC6GKEU5C5M2D4FJ5FENVDULULYZANCNFSM5GQRRPNQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

In fact, I'm still waiting for the admin page to completely load and I'm on 37k requests, most of the most recent are, of course, receiving 429 Too Many Requests responses.

The final 43k requests took almost 6 minutes to "finish" (many with a 429 response from gitlab). Once done I am not getting further requests when using a new dropdown to select a post except for the following requests (after clearing the network traffic from the console, to ensure I get the right requests):

Thanks @delwin, that's helpful - it seems you're hitting GitLab's rate limits.
Looks like these changed not so long ago: https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
I'll set up a reproduction and see if we can be more efficient. I'll try to post some answers next week.

Thanks Erez
On Fri, 12 Nov 2021 at 17:01, Erez Rokah @.***> wrote:

Thanks @delwin https://github.com/delwin, that's helpful - it seems
you're hitting GitLab's rate limits.
Looks like these changed not so long ago:
https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
I'll set up a reproduction and see if we can be more efficient. I'll try
to post some answers next week.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/netlify/netlify-cms/issues/5920#issuecomment-967226529,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC6GKH5DCF3YKJO5PHPUQTULU22LANCNFSM5GQRRPNQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

The final 43k requests took almost 6 minutes to "finish" (many with a 429 response from gitlab). Once done I am not getting further requests when using a new dropdown to select a post except for the following requests (after clearing the network traffic from the console, to ensure I get the right requests):

Ok, so I finally was able to dig into this.
I created a test repo here. It has a collection referencing another collection that has 500 entries.
What's you're seeing (with the 429 response) is GitLab rate limiting the initial load. The CMS identifies this situation and uses a backoff mechanism to handle it (basically slows everything down).
After the initial load, not only the CMS caches the requests using the browser IndexedDB (not the built-in cache mechanism my initial image shows), it also saves a local representation of the git tree for that collection.
That means, that in further attempts to retrieve the collection the CMS only retrieves the difference (the changes to the folder) instead of retrieving all the files. The reason for this, is that GitLab paginates listing of files, so without this optimization we'll need to retrieve all pages (in comparison GitHub lets you list 100K files in a single request).
This is how it looks like in local storage:

The requests you're seeing here are used by the CMS to calculate the difference between the local version to the remote version.
There isn't a simple solution for this, as the relation widget needs to load all files to create the relation.
One possible option is to use a file collection with a list widget and reference items in the list (see more here), instead of a folder collection with multiple entries.
Another option is to reach out to GitLab to see if the rate limits are configurable, or if see if they can provide a way to retrieve multiple files in a single request. This might be possible via the GraphQL API, but will require a lot of effort.
Please et me know if that helps.

Hi Erez,
I have tried understanding the couple options you sent, particularly the
one of using a file collection with a list widget. I've looked through and
tried to figure out what you might mean by trying a number of things,
without being able to get anywhere. But maybe you can explain what you
mean, more in depth?
We have a folder collection because editors of the site create new items in
it all the time. I thought maybe you were referencing a new type of file
collection, or making a file collection dynamically from a folder
collection from which we can then select, but I'm not finding a way to make
this work.
So can you expand on that possibility?
We switched to GitLab because on GitHub we had one of two problems
(depending on the version of CMS we pegged things to - a pre-GrpahQL
version or the latest verison). One issue was the same 429 rate limit, but
in a much more severe form, not allowing editors to proceed in any way.
That was with GraphQL not enabled. The other issue was with the Media
library, which didn't work on GitHub, but worked on Gitlab when allowing
the latest (GraphQL-allowed) CMS versions to run. With those two crippling
errors, we had to switch to GitLab, which, except for this one page, allows
us to function very well.
Delwin
On Thu, 18 Nov 2021 at 21:06, Erez Rokah @.***> wrote:

The final 43k requests took almost 6 minutes to "finish" (many with a 429
response from gitlab). Once done I am not getting further requests when
using a new dropdown to select a post except for the following requests
(after clearing the network traffic from the console, to ensure I get the
right requests):
Ok, so I finally was able to dig into this.
I created a test repo here
https://gitlab.com/erezrokah/netlify-cms-reproductions/-/tree/netlify_cms/issue_5920.
It has a collection referencing another collection that has 500 entries.
What's you're seeing (with the 429 response) is GitLab rate limiting the
initial load. The CMS identifies this situation and uses a backoff
https://github.com/netlify/netlify-cms/blob/05e7629cf413b8fc3cfa9ee2b15b6f5ef09e549c/packages/netlify-cms-lib-util/src/API.ts#L80
mechanism to handle it (basically slows everything down).
After the initial load, not only the CMS caches the requests using the
browser IndexedDB (not the built-in cache mechanism my initial image shows
https://github.com/netlify/netlify-cms/issues/5920#issuecomment-967133458),
it also saves a local representation
https://github.com/netlify/netlify-cms/blob/05e7629cf413b8fc3cfa9ee2b15b6f5ef09e549c/packages/netlify-cms-lib-util/src/implementation.ts#L511
of the git tree for that collection.
That means, that in further attempts to retrieve the collection the CMS
only retrieves the difference (the changes to the folder) instead of
retrieving all the files. The reason for this, is that GitLab paginates
listing of files, so without this optimization we'll need to retrieve all
pages (in comparison GitHub lets you list 100K files in a single request).
This is how it looks like in local storage:
[image: image]
https://user-images.githubusercontent.com/26760571/142483748-e3490f17-620b-4b26-b86e-1d5f4e93cc9b.png
The requests you're seeing here
https://github.com/netlify/netlify-cms/issues/5920#issuecomment-967224646
are used by the CMS to calculate the difference between the local version
to the remote version.
There isn't a simple solution for this, as the relation widget needs to
load all files to create the relation.
One possible option is to use a file collection with a list widget and
reference items in the list (see more here
https://www.netlifycms.org/docs/widgets/#relation), instead of a folder
collection with multiple entries.
Another option is to reach out to GitLab to see if the rate limits are
configurable, or if see if they can provide a way to retrieve multiple
files in a single request. This might be possible via the GraphQL API
https://docs.gitlab.com/ee/api/graphql/, but will require a lot of
effort.
Please et me know if that helps.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/netlify/netlify-cms/issues/5920#issuecomment-973217330,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC6GKBPW35T7S7DGMNEUZTUMVMBTANCNFSM5GQRRPNQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I have done some more testing, re-creating the repository again on Github so that the GraphQL API can be used, to check to see how this helps.
The response time to the editor is just as abysmal, to the point of being unusable.
The number of requests made to Github are definitely vastly reduced (to around 800 from around 40000 without GraphQL on Gitlab), but this doesn't seem to help with the responsiveness.
Other notes:

Testing was done by going directly to the page editing URL, not by navigating to the page via the collections list.
Once a relationship drop down has been populated, all other drop downs defined in the same manner respond much more quickly, though still on the order of several seconds. That is much faster than the 5 minutes initial response time, though
Once moving off the page editor (moving back to the collections list, whether one is abandoning the edit, saving it, or publishing it first), coming back to the page editor does not seem to benefit from any caching. The initial population of the list once again takes around 5 minutes.
Enabling GraphQL seems to make the Media library unusable - no images are displayed.

When going to the Collections list page first I notice that a caching message is indicated the first time going to the collections list It seems that perhaps the entries are only cached when listing them in the Collections screen?? When doing that I get an initial "This may take several minutes" message, and when scrolling down through each "page" of entries, a "Loading Entries..." message that takes a minute before being replaced with another page of the entries, finally taking several minutes before I'm able to get to the bottom of the list of 150 items.
However, even when these entries are cached for the Collections list screen, that cache does not seem to be used in the drop downs (the relationship widget), as using them even after the caching in the list still takes 5 minutes to populate.
So this is no better on Github with GraphQL than it is on Gitlab (where I guess GraphQL is not yet supported by Netlify CMS).
Delwin

Hi @delwin and sorry for the very late response, I was out of office for a week+.
For the file collection suggestion - the config will look like:
collections:
  - name: posts
    label: Posts
    label_singular: 'Post'
    folder: content/posts
    create: true
    fields:
      - label: Title
        name: title
        widget: string
      - label: 'Featured Stories⁺'
        singular_label: 'Featured Story'
        hint: '⁺will be displayed in the order provided above.'
        name: 'featured_stories'
        widget: list
        ui: fields
        required: true
        allow_add: true
        fields:
          - label: 'Story'
            name: 'story'
            widget: 'relation'
            collection: 'stories'
            file: 'stories'
            search_fields: ['stories.*.title']
            value_field: 'stories.*.title'
            display_fields: ['stories.*.title']
  - name: stories
    label: Stories
    files:
      - file: content/stories/stories.json
        name: stories
        label: Stories
        fields:
          - label: Stories
            name: stories
            widget: list
            fields:
              - label: Title
                name: title

Notice the wildcard * char to reference a list.
For differences between GitHub and GitLab when reaching rate limits - GitHub has a 1 hour window, so if you reach the limit you'd have to wait up to 1 hour for that window to reset.
As for GitHub with GraphQL - let me look into the issues, but please consider the CMS doesn't currently take full advantage of the GraphQL API.
We're not grouping multiple queries into a single request as much as we should yet, and in some cases we can't use the GraphQL API at all since it doesn't support everything the REST API supports. For example it doesn't support binary blobs (like images), see https://docs.github.com/en/graphql/reference/objects#blob.
I'll port my test repo to GitHub and see if it works as expected with/without GraphQL and also check the media library issue,

Thanks @erezrokah
So that's what I thought you might mean with the file collection, but that means that every time a story is added, deleted or has its title changed (we don't use title for the relationship field because of this), we'd have to manually update the stories.json file, which isn't feasible for a manual process without errors and seems like an unwieldy process.
It is something I have considered doing, having a server process running regularly to generate such a stories.json file and commit its changes to the repository, but it really doesn't seem like an ideal solution.
Or am I missing something in your suggestion?

Hi @delwin, that's a good point.
For that scenario the recommendation is to use an ID widget like this one and use that field for the relation.
Would that cover your use case?

Hey @erezrokah,
As I indicated we already don't use the title for the relationship field, and we do have an id widget that is used for that purpose.
The issue I raise is that in order to make this sort of "solution" work, the stories.json file needs to be manually updated every time a story is added, deleted, or a title changed (so that the title being searched appears correctly). This cannot happen within the CMS workflow, and would need to be done outside of the CMS, so it seems very unwieldy.

Hi,
we have the same issue with Gitlab. We have an object consisting of an object and a list of objects which have the relation field. Opening the initial object takes quite some time. Each relation type has several hundred potential relation files. We implemented this way because we needed a 1:n relation.
        fields:
        - label: "Section"
          name: "section"
          widget: "object"
          required: false
          fields:
            - label: "Section title"
              name: "title"
              widget: "object"
              required: false
              fields:
              - {label: "Label", name: "label", widget: "string", hint: "The title of the content section", required: false}
              - {label: "Link", name: "link", widget: "string"}
            - label: "Content"
              name: "content"
              widget: "list"
              allow_add: true
              fields:
              - label: "Article"
                name: "article"
                widget: "object"
                required: false
                fields:
                - label: "Select Article"
                  name: "uuid"
                  widget: "relation"
                  collection: "article"
                  searchFields: ["title"]
                  valueField: "uuid"
                  displayFields: ["title"]
                  required: false
                - label: "Select card size"
                  name: "size"
                  widget: "select"
                  required: false
                  options:
                  - { label: "One column wide", value: 1 }
                  - { label: "Two colums wide", value: 2}
                  - { label: "Three columns wide", value: 3 }
              - label: "Podcast"
                name: "podcast"
                widget: "object"
                required: false
                fields:
                - label: "Select Podcast"
                  name: "uuid"
                  widget: "relation"
                  collection: "podcast"
                  searchFields: ["title"]
                  valueField: "uuid"
                  displayFields: ["title"]
                  required: false
                - label: "Select card size"
                  name: "size"
                  widget: "select"
                  required: false
                  options:
                  - { label: "One column wide", value: 1 }
                  - { label: "Two colums wide", value: 2}
                  - { label: "Three columns wide", value: 3 }

Yes, I get the impression more and more that Netlify CMS was a fun experiment that had some hope, but really only works well for small sites with very little structure, unfortunately. The reliance on git was a cool idea, but is obviously not very scalable.

@erezrokah we are hosting our own Gitlab instance. There are no Gitlab API limits set for authenticated users. Any other Gitlab settings I might tweak?
Thanks a lot!

Hi @delwin and @lorenzode 👋
@lorenzode for the performance issues with GitLab, in order to figure out the relation the CMS needs to read all the files. This is especially slow on initial load (before caching kicks it). The only way I can think to improve this is to use the GraphQL API. We're currently using GitLab REST API, which means we need to list all files (and paginate on them 100 at a time), and then read each file in a separate request. Lets say you have 1000 files in a collection, we'll need 10 requests just to list them (those need to happen serially) and then another 1000 to read the content of the files (those are done in parallel, but still...).
With the GraphQL API we should be able to group multiple queries into a single request in theory. I haven't dug into GitLab's GraphQL API and what it supports, so I can't really say at the moment of the benefits.
I did open a separate issue for it here, in case someone would like to dig in further.
@delwin I'm not sure I understand the need to sync the stories.json file manually. Having a file collection will let you edit that file within the CMS. Instead of a folder collection with a file per story, you'd have a file collection with a list of items, each one representing a story object.

I'm seeing this happen in an environment with only 30 or so files, but many relation fields, the number of requests doesn't seem to stop. Keeps crashing chrome.

@erezrokah I will explain the need for a synchronization by starting with a more complete structure of the site, although I believe you have seen the complete structure before with a different bug:

There are many "Stories" that each have a page on the web site

these are therefore in a folder collection so that each generates a page

There are some pages in the web site that also reference these stories in a list

these page templates use the Relation widget to choose the selected Stories from the folder collection
only the Title, ID and whether there is a particular image associated with them is needed to be used by the Relation widget to help the user select items
These Relations widgets are in a List widget where the admin can add and remove as many as they want

There are getting close to 200 "Stories" (articles) and one page in particular has about 50 of them selected in its list.
There is another collection of "News" that works in a similar way to "Stories" and has the same issue, but the "News" items selected on other pages is usually a somewhat shorter list (maybe around 10)

So the issue is that the Stories are and must be in a folder collection, since they produce web pages. We cannot transform the Stories to a file collection because they are not simply used in a list on one or two pages. This means that if we want to use a file collection to make the Relationship Widget more performant, we would need to have it created on-the-fly whenever the Folder collection is updated (particularly when a new Story is added, one is deleted or a title is updated). The File collection cannot functionally replace the Folder collection

Thanks @delwin, that makes sense. Using a file collection will require changing the way pages are generated during the build.
The function to optimize would be this one:
https://github.com/netlify/netlify-cms/blob/0053ebbb46f189469c86518eb458daa34a8fec4d/packages/netlify-cms-backend-gitlab/src/implementation.ts#L199
It's a bit complex at the moment since it does what I described in https://github.com/netlify/netlify-cms/issues/5920#issuecomment-973217330 (caching each collection locally and only getting a diff).
If we can list files and get their content using a GraphQL API query, that should boost performance quite a bit, and it could be a drop-in replacement for that function.
I wonder if at some point we'll hit these limits, but it's worth a try.
I'll look at the GitLab GraphQL API and see what we can do.

Hey @erezrokah,
I have now made a couple file collections that mirror the folder collections so that we can use and test that.
BTW, we use hexo as a site generator, and I have created a small hexo generator plugin to generate the file collection files we would need (one for each type of article - only one of which I've mentioned in this thread). I have thus created file collections for these relationships to use, mirroring the folder collections, but only containing the fields necessary for the relationship widget (Title, ID). Note: This is not yet a complete solution, as I have to manually commit the changes to these file collecitons to git for them to be picked up.
The Stories file collection contains the ~150 stories in it.
Using this configuration where the relationship field picks up values from a file collection instead of the folder collection is quite surprisingly no faster when the editor is first loaded for a page (the relationship widgets aren't yet cached). One the select widget is clicked in and typing is started to search for an article, it still takes several minutes to load the widget with the choices.
One this several minutes has passed, then searches only take several seconds (2-3 at best, 5 at worst). This still seems exceeding slow for a process where only 150 titles are searched for possible matches, so I'm not sure what the widget is doing. This time is consistent across all such widgets on the page (remember the relationship widget is in a list widget, and there are at least two of these lists using the same relationships on this page).
Once the editor page is exited, however, going back to the editor for the page starts everything all over - needing several minutes to load data for the first time a relationship widget is activated, then serveral seconds after that.
So there seems to be no caching of a relationship widget that uses identical collections and search conditions between page edits, and the cache is rebuilt every time a page is edited and the time to construct it is exceeding long considering it now only needs to load one file to get the information it needs (150 records, and about 26kB in size).
BTW, I have also made sure that there is not preview for the page configured in case the preview was adding any time to it. So the "preview" is the default preview, listing values (in this case IDs).
Here is the config of the file collection and the corresponding generated collection:
Definition of the file collection:
  - name: "data"
    label: "💾 Global Settings"
    editor:
      preview: false
    files:
    - name: "story_index"
      label: "⛔ Stories Index"
      file: "source/_data/story_index.json"
      fields:
      - label: "Stories List for use in admin (not edited manually)"
        name: posts
        widget: list
        ui: posts
        required: true
        fields:
        - {label: "ID", name: "uuid", widget: "string"}
        - {label: "Story Title", name: "title", widget: "string"}
        - {label: "Type", name: "layout", widget: "string"}

Use of the file collection in a page:
        - label: "Stories"
          singular_label: "Story"
          name: "spotlight_stories"
          widget: list
          ui: fields
          required: true
          allow_add: true
          fields:
          - label: "Story"
            name: "story"
            widget: "relation"
            collection: "data"
            file: "story_index"
            search_fields: ["posts.*.title"]
            value_field: "posts.*.uuid"
            display_fields: ["posts.*.title"]

story_index.json:
{
	"posts": [
		{
			"title": "Title 1",
			"uuid": "id-1",
			"layout": "post"
		},
		{
			"title": "Title 2",
			"uuid": "id-2",
			"layout": "post"
		},
		{
			"title": "Title 3",
			"uuid": "id-3",
			"layout": "post"
		},
		...
	]
}

Hi @delwin, thank you for the additional information.
I'm currently looking into GitLab's GraphQL API to see if it provides better performance, see https://github.com/netlify/netlify-cms/issues/6034#issuecomment-995918416.
I'll update once/when I make some progress.

Thanks @erezrokah !

Hello 👋 I currently have a draft PR that uses the GitLab GraphQL to retrieve 50 files at a time (to avoid hitting query complexity limit).
There are still improvements to be made (like saving to a local cache instead of in-memory cache), but it should help speed up initial load quite a bit.
If you're interested, the code is here https://github.com/netlify/netlify-cms/pull/6059/files#diff-570651114617393164ce3002abb929d59e73590014360152082e250e3075d772R468
Also, it will require some more testing to see if query complexity is impacted by the content size of files (my test repo has small files at the moment).
It would be great to get some early feedback on this, so if anyone would like to try it out at this stage, you can follow on contributing guide to try it out. See https://github.com/netlify/netlify-cms/blob/782e87c48a14937fcc7167ae5a7960a692e8054c/CONTRIBUTING.md#debugging
You'll need to have a similar config to this:
backend:
  name: gitlab
  branch: <branch>
  repo: <repo>
  use_graphql: true

I'm going to try take a look at this in the next weeks. Thanks @erezrokah

Initial support for the GitLab GraphQL API is released in<EMAIL_ADDRESS>See the docs for more information.
Going to keep the issue open to get some feedback.

Thanks so much! We will test this in the new year and give feedback here.Am 28.12.2021 12:51 schrieb Erez Rokah @.***>:
Reopened #5920.
—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.Message ID: @.***>

Hi @erezrokah . Happy New Year!
So we did test this, even so much as having the team of editors try it on the live administration. Unfortunately enabling graphql caused several issues in a few places, so we've had to disable that:

the collection for one of the widgets (at least) reported zero records found when using graphql, but not when diabling it. This was when a collection was not in a list widget, but on its own
the collections in list widgets kept duplicating records during administration, adding already chosen records again at random places in the list, seemingly as if by magic (words of the admins). This is what we saw previously in a different bug where this was happening without graphql enabled, and seem to have been some sort of race condition where slowly returned responses were being handled after they should have been discarded due to an administrator or other operation initiating a new request. Unfortunately I do not see that bug in the issues, anymore, so can't provide you with a reference.

@erezrokah
We tried 2.10.183 on an instance where we have this structure: List > Object > List > Objects with relation field. We host our own Gitlab. Each relation points to 50 to 100 files. I definitely see an improvement in speed when loading the form although there is still some loading time when revealing the last object widget and until the relations are displayed in the relation widget. It's not snappy but it makes this form usable for us again.
Let me know if you want me to supply additional debugging information. Many thanks!

Hi everyone, sorry for the delay. This issue got pushed back due to other priorities.

reported zero records found when using graphql

This is probably related to us hitting the query complexity limit. We'll need to add some error handling and fall back to retrieving less entries (the failure is probably reflected in the query response in the browser network tab).
I'll try to improve it by the end of this week.

@erezrokah might this be a related problem? https://github.com/netlify/netlify-cms/issues/6410

Still having this issue as well
Related #4097

We still find this a crippling problem for editors, taking minutes (tens of minutes) for relationship fields (relation widgets, displayed as a type of custom drop-down) to populate in trying to select a post in a page. We have tried things like paring down the git repository, ensuring that editors have good amounts of memory for caching, etc.
In the most recent trials I have done as a developer who sees the issues on my machine, too, I have found that the issue is no longer requests being made from the browser on the initial use of a relationship field. In fact, I have always told the editors to give the CMS a few minutes (10-15 or more) to populate the cache after the first time they connect, so that all data can be cached properly. This seems to happen.
Once the cache is populated, however, the drop-down relation widget still take 10-15 minutes to populate (or more, depending on computer and browser) the first time one is used (remember, we use relation widgets as part of a list widget, so adding another relation to the list after one has already been added successfully works much faster). The second time a relation is added to the list and a portion of title typed, the drop-down populates within 5-10 seconds.
However, using the same template, or coming back to the same page within the same editing session, the initial population time returns to 10-15 minutes or more.
During all of this drop-down (relation widget) population time, there is almost no network traffic. None, in fact, other than a occasional "user" ping once every 5 minutes.
So this population time is all some sort of JavaScript processing, completely unrelated to Gitlab network traffic and responses.

There is a possible solution for this problem. We created an own relation widget that only uses the slug as search, display and value field. It is basically a select widget that loads the options by listing all files from a folder collection. Unfortunately, there is no function provided in the props to list all files from a directory in the git repository. I called the GitLab API directly. We have 1400+ files in the referenced collection and it works within a few seconds.
Warning: This code only works for GitLab SaaS but it can be adpted to other backends.
simple_relation/SimpleRelationControl.tsx
import React from 'react';
import { Map, List, fromJS } from 'immutable';
import { find } from 'lodash';
import Select from 'react-select';
import { reactSelectStyles } from 'netlify-cms-ui-default';
import { validations } from 'netlify-cms-lib-widgets';
import { CmsWidgetControlProps } from 'netlify-cms-core'
import { API } from 'netlify-cms-backend-gitlab'

type FileEntry = { id: string; type: string; path: string; name: string };

function optionToString(option) {
  return option && option.value ? option.value : null;
}

function convertToOption(raw) {
  if (typeof raw === 'string') {
    return { label: raw, value: raw };
  }
  return Map.isMap(raw) ? raw.toJS() : raw;
}

function getSelectedValue({ value, options, isMultiple }) {
  if (isMultiple) {
    const selectedOptions = List.isList(value) ? value.toJS() : value;

    if (!selectedOptions || !Array.isArray(selectedOptions)) {
      return null;
    }

    return selectedOptions
      .map(i => options.find(o => o.value === (i.value || i)))
      .filter(Boolean)
      .map(convertToOption);
  } else {
    return find(options, ['value', value]) || null;
  }
}

interface Props extends CmsWidgetControlProps<any> {
  t: any
  setActiveStyle(): void
  setInactiveStyle(): void
}

export default class SimpleRelationControl extends React.Component<Props> {

  state: {
    initialOptions: string[]
  }

  constructor(props: Props) {
    super(props)

    this.state = {
      initialOptions: []
    }
  }

  isValid = () => {
    const { field, value, t } = this.props;
    const min = field.get('min');
    const max = field.get('max');

    if (!field.get('multiple')) {
      return { error: false };
    }

    const error = validations.validateMinMax(
      t,
      field.get('label', field.get('name')),
      value,
      min,
      max,
    );

    return error ? { error } : { error: false };
  };

  handleChange = selectedOption => {
    const { onChange, field } = this.props;
    const isMultiple = field.get('multiple', false);
    const isEmpty = isMultiple ? !selectedOption?.length : !selectedOption;

    if (field.get('required') && isEmpty && isMultiple) {
      onChange(List());
    } else if (isEmpty) {
      onChange(null);
    } else if (isMultiple) {
      const options = selectedOption.map(optionToString);
      onChange(fromJS(options));
    } else {
      onChange(optionToString(selectedOption));
    }
  };

  async componentDidMount() {
    const { field, onChange, value } = this.props;
    if (field.get('required') && field.get('multiple')) {
      if (value && !List.isList(value)) {
        onChange(fromJS([value]));
      } else if (!value) {
        onChange(fromJS([]));
      }
    }
    const user = JSON.parse(localStorage.getItem('netlify-cms-user'))
    const directory = field.get('folder')
    if (!!user && user.backendName == 'gitlab' && !!directory && this.state.initialOptions.length == 0) {
      const api = new API({
        token: user.token,
        branch: field.get('branch'),
        repo: field.get('repo'),
        squashMerges: false,
        initialWorkflowStatus: '',
        cmsLabelPrefix: '',
        useGraphQL: false
      })
      //console.log(api)
      const response: Promise<FileEntry[]> = api.listAllFiles(directory)
      const result = await response
      //console.log(result)
      this.setState({
        initialOptions: result.map(it => it.name.replace(/\.[^/.]+$/, ""))
      })
    }
  }

  render() {
    const { field, value, forID, classNameWrapper, setActiveStyle, setInactiveStyle } = this.props;
    //const fieldOptions = field.get('options');
    const isMultiple = field.get('multiple', false);
    const isClearable = !field.get('required', true) || isMultiple;

    const options = [...this.state.initialOptions.map(convertToOption)];
    const selectedValue = getSelectedValue({
      options,
      value,
      isMultiple,
    });

    return (
      <Select
        inputId={forID}
        value={selectedValue}
        onChange={this.handleChange}
        className={classNameWrapper}
        onFocus={setActiveStyle}
        onBlur={setInactiveStyle}
        options={options}
        styles={reactSelectStyles}
        isLoading={this.state.initialOptions.length == 0}
        isMulti={isMultiple}
        isClearable={isClearable}
        placeholder=""
      />
    );
  }
}

simple_relation/schema.ts
export default {
  properties: {
    multiple: { type: 'boolean' },
    min: { type: 'integer' },
    max: { type: 'integer' },
    folder: { type: 'string' },
    branch: { type: 'string' },
    repo: { type: 'string' },
  },
  required: ['folder', 'branch', 'repo'],
};

cms.ts
import CMS from 'netlify-cms-app'
import NetlifyCmsWidgetSelect from 'netlify-cms-widget-select';
import SimpleRelationControl from './simple_relation/SimpleRelationControl';
import simpleRelationSchema from './simple_relation/schema';

// Initialize the CMS object
CMS.init()

// @ts-ignore
CMS.registerWidget('simple_relation', SimpleRelationControl, NetlifyCmsWidgetSelect.previewComponent, simpleRelationSchema)

usage in config.yml
...
fields:
- label: Source
  name: source
  widget: simple_relation
  folder: *sourceFolder
  repo: *backendRepo
  branch: *backendBranch
...


Create detect_mshta_renamed.test.yml

❤

This PR was created by Automated Detection Testing Service :robot:
Tested detections:
Detect mshta Rename :white_check_mark:

diskUsage: 253952
runDuration: 1.305s
scanCount: 641


Add support for detecting pnpm build tool

As subject
Didn't add tests as there are none in the gem yet

Thanks! ❤


Upgraded SSP tutorial - Aligned with the new implementation of a node…

…mon service provider - pre review
Still need to add the correct url to css file (to be hosted on github) and will be done when  updated tutorial code is merged in

todo:

[ ] redraw diagram in ascii
[ ] add stylesheet link once code repo PR is merged


N38: Subfuse

Part of https://github.com/rust-gamedev/rust-gamedev.github.io/issues/1157

@ozkriff Oops! Just made the image smaller.


Fix gcc 4.8.5 build

Fix for build using g++ 4.8.5 and maybe 4.8.4 #194
g++ 5.3.1 does not generate an error here.

Good catch, I'll merge this over shortly.

This has been updated internally and will be going out shortly. Thanks for your contribution!


runtime properties

Hi,
Where do I place the common runtime properties file and other runtime properties files ? I need to setup the deep storage as hdfs , thus need to configure the common runtime properties.
Regards
Rahul

Ignore it , we figured it out that we can specify it on the supervisor.conf file .
-Rahul


Openless PSA crypto APIs implementation

Description
This PR is the second part of the work related to #3265, the first part being #3527.
This PR implements the openless PSA crypo APIs that take as input or return a key identifier instead of a key handle (see #3265 for details).
Status
Done
Requires Backporting
No, PSA only

The CI is as good as it can. Apart from Mbed OS tests, "merge TLS Testing" and "pr-merge" are falling because of the "ABI-API" testing as expected as this PR changes the definition of API parameter types.

Given the size of the changes, I strongly prefer additional commits to rebasing. I'll allow rewriting commits if you insist, but they would make the re-review a lot more difficult.

Just a word on this. I am on the side of the reviewers on this. As a reviewer you prefer additional commits, I'll do additional commits. As the author of the PR, I have a deep enough understanding of the changes to be comfortable with both approaches. As long as there is not "too much" back and forth (I don't know if this is going to be the case or not) it should be fine for the second reviewer with additional commits as well. If it is not the case and the second reviewer would be more comfortable with a rewrite of the history, I may do one at some point with the goal of the result of the rewriting to be the same as what you last reviewed.

I believe this is ready for re-review. I first pushed new patches addressing @gilles-peskine-arm  comments ending up at commit 13c5f76. All the CI tests passed on this commit but there was conflicts in merging it into the development branch.
I then rebased on top of the development branch ending up at  bd22ab7  resolving the five conflicts with development. The CI ran successfully on that commit (both head and merge tests but for the ABI-API-checking test as expected).
I then force-pushed some changes in test code that was added and used handles instead of key identifiers. The diff of this last step can be seen here. On this commit all tests are passing but in the merge test job: ABI-API-checking test and three other tests because of CI infrastructure glitches. As those 3 last tests were passing on the penultimate version I don't feel the need to re-trigger the CI and think it is ok like this.

I've simplified the commit history (from 49 to 37 commits) and forced-pushed the result. The head is now def441e. It is equivalent to the previous head cd65c6c (the previous head can be found here) but for the renaming of "transient key identifiers" to "volatile key identifier" and the diff can be found here./cd65c6cbcbac319b67923a0ca99c8da2d4cfbfb1..def441e7293ccc1edd59b2216c7222364ed8f0c9).

New force-push from from def441e to a5b894f to fix the commit message of commit "psa: Fix tests/handling of lifetime incompatible with location". No code change thus the force-push diff is empty.

I've addressed all comments and the CI is passing but the ABI-API check as expected. Ready for review.

This comment is indeed not at the right place (psa_key_handle_t is not platform specific now) and wrong as psa_key_handle_t is not an integer type anymore when MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER is defined. As this type is on its way out (now defined in crypto_compat.h) I think it is ok if it does not appear anymore in the DOXYGEN documentation. Thus I just removed this comment,

I've force pushed a change to the last commit adding the change logs and added a new commit fixing and improving  documentation. Thus I believe I've addressed all comments but the ones related to access counter naming and key attributes reset. @bensze01 waiting on your final review and your opinion about the naming discussion to proceed.

Forced push from  2a61649 to eb9a307:
2a61649 can be found here.
Starting from 2a61649 I have removed the commit "tests: psa: Reset systematically key attributes" (873e688) and then added the new commit that add the psa_reset_key_attributes() where needed.

Approved. I'd prefer to improve the lock/acquire terminology, but since this is purely internal, I'm ok with doing that in a follow-up.

Great. I agree with you that the lock/acquire terminology can and must be improved and I am going to work on it in the coming days. @bensze01 ok to approve on your side as well?

Force push  from eb9a307 to 3a4f0e3:
I've just rework the penultimate commit "Improve/fix documentation" to address Bence's last comments.
You can find the diff between eb9a307 to 3a4f0e3 here.

CI passed except for expected ABI/API changes (psa_key_handle_t → psa_key_identifier_t or mbedtls_svc_key_id_t, changes in internal structures). I filed https://github.com/ARMmbed/mbedtls/issues/3904 for the improvements around acquire/lock naming. This is ready for merge.


GROQ select

[ ] https://sanity-io.github.io/GROQ/GROQ-1.revision1/#global_select()
[ ] https://sanity-io.github.io/GROQ/GROQ-1.revision1/#Pair

https://github.com/sanity-io/groq-js/blob/28dee3d75e9e32722dfb2291e1f58e1418e11bb8/src/nodeTypes.ts#L201
Seems to parse to SelectNode and SelectAlternativeNode


[SPARK-33980][SS] Invalidate char/varchar in spark.readStream.schema

What changes were proposed in this pull request?

invalidate char/varchar in spark.readStream.schema just like what we've done for spark.read.schema in da72b87374a7be5416b99ed016dc2fc9da0ed88a
Why are the changes needed?

bugfix, char/varchar is only for table schema while spark.sql.legacy.charVarcharAsString=false
Does this PR introduce any user-facing change?

yes, char/varchar will fail to define ss readers when spark.sql.legacy.charVarcharAsString=false
How was this patch tested?

new tests

cc @cloud-fan @maropu @HyukjinKwon thanks~

cc @cloud-fan @maropu @HyukjinKwon thanks~

I think it's fine since at least it's consistent. But just for doubly sure, in which context char and varchar are supported? Only in pure SQL?

we currently run into such a situation.

I think it's fine since at least it's consistent. But just for doubly sure, in which context char and varchar are supported? Only in pure SQL?

we currently run into such a situation.

Only in pure SQL?

Only in table schema. User-specified schema can't support char/varchar.

Only in pure SQL?

Only in table schema. User-specified schema can't support char/varchar.


[add-resource] New Plugin/Theme

I would like to add my new theme to the Grav Repository
Here are the project details:
tnware/grav-theme-mdux
https://github.com/tnware/grav-theme-mdux
https://github.com/tnware/grav-theme-mdux/releases/tag/0.9.8
I created a theme coined "material design user interface". I started this theme in Summer of 2016 and, after a period of hiatus, I have worked to bring it up to functionality on the current version of Grav.
I would like for this to be available in the theme repository, especially considering there seem to be no themes reflecting Google/Android Material Design.
I plan to continue contributions to bring code structure up to date.
Thank you,
Tyler Woods

Added to GPM.  Thanks.


Auto approve group membership

@alvacouch Changes have been made as you suggested.

@alvacouch Can I merge this?


About Status messages

Explain:

The different types of Status messages based on context (DM, group chat, Communities)
What are the options when using messages.
What makes messaging in Status different from other apps (how messaging and storage work)

The FAQ section for this topic is getting bigger, so I'll move this section to a new article https://github.com/status-im/help.status.im/issues/232.


can't find /api/spec.json when i add url_prefix in blueprint

no matte how, i add 'url_prefix' in program, what i get always is error message that "GET /api/spec.json HTTP/1.1" 404 -“.The same problem also exists in the sample. Please tell me what is wrong
flask-restful-swaffer-learn.txt

Have you tried /blueprint/api/spec.json?


Screenshots for F-Droid

You can add Screenshots for F-Droid, too, if you want.
https://f-droid.org/en/docs/All_About_Descriptions_Graphics_and_Screenshots/
https://gitlab.com/snippets/1895688
That was simply information about what is still possible. You would then only have to enter the screenshots in fastlane. This is just one way that you can also add screenshots to the app view if you can. If you added screenshots (then F-Droid is happy) and if you can't add any -> then you can close the issue because it was only an info. Thank you that your app is available in
fdroid. Very much for your attention.
it is only information.

Thanks, I'm transferring to conversations.
We have normally the screenshots in place for F-Droid, not quite sure why it's not picking them up @mondlicht-und-sterne


Upgrade dependencies

This also includes replacing tsdx which is blocking upgrading TypeScript, ESLint, etc.:

bundling: microbundle or tsup
linting: eslint & eslint-config-zenflow
testing: unit testing needs to test TS source which also relies on Babel + babel-plugin-macros + ts-interface-builder/macro

Completed in release v2.0.1 https://github.com/zenflow/composite-service/releases/tag/v2.0.1


Make sqlalchemy an optional dependency

We are using csvkit to work with CSV files and we believe we are not using any sqlalchemy-related functionality. If this library can provide significant functionality without it, could it be made an extra optional requirement?

Is there a particular reason that installing sqlalchemy is a problem? I'm generally averse to having extra requirements that have to be installed.

Well, it's little not-so-important things:

We're developing a remote access tool and our users are surprised when sqlalchemy is installed
If we were to package something using csvkit we would like to avoid having to package SQL Alchemy
Someone might already be using an older version of SQL Alchemy than what you require and might have problems using csvkit as a dependency

In any case, consider this more of a suggestion/for your consideration rather than a request :)

I'll close for now, but if it becomes a bigger issue, I can re-open!


Use concrete constrained extensions to get rid of protocol conformance trick

Now that SR-1009 is resolved, we can remove the FIXME and the extension can constrain to  the concrete Version type without using any protocols.

@swift-ci please test

@kaandedeoglu Thanks! sorry for the delayed merge


updated _initialize() def

added return type to the method definition

Fix for the https://github.com/testomatio/php-reporter/issues/7 , successfully tested locally.


Add and Test Mini Magic

Test to see if minimagic gem is more suitable.

Closed. User can override phcpress basic settings.


Strange cases of profanity

[PROFANE] are you open for business on saturday
[PROFANE] hello. I have no hot water
[PROFANE] Why do I not have hot water
[PROFANE] my hot water is not working
[PROFANE] what type of HVAC systems exist
[PROFANE] who has stephanie worked for?
[PROFANE] what comapnies has she worked for?
[PROFANE] is sun devil in phoenix?
[PROFANE] What was Stanley Connor doing?

Most of these actually seem to be false positives!

[PROFANE] are you open for business on saTURDay
[PROFANE] hello. I have no hoT WATer
[PROFANE] Why do I not have hoT WATer
[PROFANE] my hoT WATer is not working
[PROFANE] what type of HVAC systemS EXist
[PROFANE] who hAS Stephanie worked for?
[PROFANE] what comapnies hAS She worked for?
[PROFANE] is sun devil in pHOEnix?
[PROFANE] What wAS Stanley Connor doing?

I'm making a PR right now to fix these false positives.
 One could say "you suck" and "I hate you" to a friend without it having any malice.

Fixed in v1.2.1. I've also credited you in the release note for bringing these to my attention!


Laziness tweaks

Make LazyList.#::'s argument non-lazy, as it will actually be evaluated eagerly; add a truly lazy prepend method.
Somewhat related: stub out View.toString so views don't get forced prematurely.

@SethTisue Do you think we could fix the issue in scalac 2.13?

If the issue can not be resolved in the compiler, then we should use the same workaround as we use in the current Stream collection, as noted by @szeiger in #127.

I changed it to use a Stream-like implicit trick and added the classic Fibonacci example (which now works) to the tests as well.  It's a bit irritating that the first element is still strict, but this applies to Stream as well.

So, the bad news is that the new tests don't pass with dotty.  The even worse news is that they don't pass under dotty even with scala.collection.immutable.Stream...

What if you explicitly instantiate the correct type of elements? List[BigInt](0, 1, 1, 2)

Hey @pnf, we discussed this issue and came to the conclusion that we should not have lazy right-associative operators unless they are properly supported by the language.
Consequently, we should remove the #:: and just have a prepend method on LazyList.

@pnf I think we should completely remove the #:: operator (and rename the homonym extractor). The workaround that relies on an wrapping class is only partially lazy.

@julienrf I'm not so sure... The #:: operator is "sufficiently lazy" as to be useful for Stream, where it gets extensive use in real code, and it could be made genuinely lazy with a little compiler work.
That said, I'm unclear on whether LazyList is intended to replace Stream or to supplement it, and, if the latter, why one would choose one over the other.

LazyList replaces Stream.
I really dislike having an #:: operator that’s only partially lazy. I think that could lead to very confusing situations. I prefer the semantics of operations on LazyList to be clearly always lazy excepted for “terminal” operations (like fold).

It seems like there are a few probable routes by which it would become fully lazy by the time the new collections are stable: eg.the underlying SI might be fixed, or we may be able to use scalameta. If LL is to replace Stream and we remove the right-associative syntax, the message to users is that we're removing functionality they use, for their own good, because of a bug in another part of the ecosystem that we don't feel like addressing (for some meaning of "we").

On Jun 28, 2017, at 8:55 AM, Julien Richard-Foy<EMAIL_ADDRESS>wrote:
LazyList replaces Stream.
I really dislike having an #:: operator that’s only partially lazy. I think that could lead to very confusing situations. I prefer the semantics of operations on LazyList to be clearly always lazy excepted for “terminal” operations (like fold).
―
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Alright, that’s good for now, could you please squash your commits into a single one?

Thanks @pnf, I squashed your commits and merged them to master: https://github.com/scala/collection-strawman/commit/950e7ddb432e0d1e1ebff34d292ff457908e07ba


print: Fix dark mode, opacity issues.

This PR improves topic-printing in multiple ways:

it isolates all dark-mode CSS to screen media only (@media screen), meaning that printed topics should always only ever apply the light mode--keeping the print CSS free from a whole bunch of adjustments that we'd have to make otherwise.
it fixes a bug with Firefox, where print output can appear pixelated/fuzzed out on text with opacity set, such as the EDITED/MOVED messages and timestamps.
it saves paper by setting height: auto in the print styles for html and body, which were otherwise creating a blank N+1th page--even on topics with a single short message.
it overrides some additional things (primarily margins) that Safari was accounting for, based on how wide the viewport was at the time of printing.

CZO Discussion

Before (Actual PDF)
After (Actual PDF)

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Looks great!

Merged, thanks @karlstolley and @amanagr!


ui: Hide default scheme and metrics path

Signed-off-by: Julien Pivotto<EMAIL_ADDRESS>

I think this would be unexpected, and this approach is also in general not scalable. We've a lot of default properties across our various repos.


v0.2.0 reports version v0.0.1 in about menu

The tag download ( https://github.com/downloads/pje/WavTap/WavTap 0.2.0.pkg ) still reports an older version.

This should be fixed by newer releases (> 0.3.0).


Updating error handler to use the appropriate regex

What
This PR represents a small fix that appears to satisfy the problem reported in Issue #6.
If you look closely at the error reporting code in parserTools.js, you will notice that there are two similarly-named regex patterns for matching on error message output.
It turns out the function nearlyError2Object was expecting a specific type of error message, matching on it, and then using some of that match data to compose a new kind of error object. However, when the match fails -- as it does in the example in Issue #6 -- the program is trying to access indices of an undefined object.
Solution
This PR switches to using the alternate available regex pattern instead. It's a small change, and I'm not sure how it will affect the wider code since I'm not able to get any tests to run.
Here is an example program I used (at the root of the repo) to verify that the new form was indeed working:
const { ATF2JTF } = require("./src/Converters/ATF2JTF.js");

const example = `&P298489 = CBCY 03, p. 208, NBC
@seal
1. ha-ba-lu5-ge2
2. ensi2 Adab{ki}
3. ur-pa4-mu-ra
4. dub-sar
5. ARAD2-zu`;

ATF2JTF(example, '', false);

Misc
This PR only deals with the error reporting error, and not the fact that the above example with a seal fails to parse.

This fixes the behavior for lexer errors but breaks it for grammar errors:
const { ATF2JTF } = require("./src/Converters/ATF2JTF.js");

const example = `@seal 1
1. ha-ba-lu5-ge2
2. ensi2 adab{ki}
3. ur-pa4-mu-ra
4. dub-sar
5. ARAD2-zu`;

ATF2JTF(example, '', false);

I think the check if (message.includes('Instead, I was expecting')) is supposed to distinguish between nearley errors and lexer (moo) errors, but it does not. Maybe change that to if (message.match(re_invalid_syntax)), then we are also immediately sure that nearlyError2Object will work (or at least the step that is causing #6).

@larsgw Good catch. I've pushed a commit that does pretty much what you've suggested and checks for both matches. So we should be getting one or the other now.

If nearlyError2Object is intended only to work on errors that match re_invalid_syntax (which it might) I think solving it in processError().

@larsgw You are probably right (I think I misunderstood you before). There are two new commits that deal with this. Basically, we rename these regexes to be more descriptive -- one for nearley/parser errors and one for moo/lexer errors. As you suggested, now the branching occurs by way of processError and not in nearlyError2Object.
Also some minor reformatting. Sorry if that's annoying I can undo it.


Plot Viewer no longer an option with latest update

Issue Type: Bug
The ability to use plot viewer on plots in jupyter notebook is not an option anymore. I have tried hovering over the plot as well an double clicking and drag-and-droping the plot but the plot viewer just isn't there anymore.
VS Code version: Code 1.57.0 (b4c1bd0a9b03c749ea011b06c6d2676c8091a70c, 2021-06-09T17:18:42.895Z)
OS version: Linux x64 4.15.0-144-generic snap
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (12 x 4299)

GPU Status
2d_canvas: unavailable_softwaregpu_compositing: disabled_softwaremultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: disabled_offrasterization: disabled_softwareskia_renderer: enabled_onvideo_decode: disabled_softwarevulkan: disabled_offwebgl: unavailable_softwarewebgl2: unavailable_software

Load (avg)
1, 1, 1

Memory (System)
31.27GB (21.73GB free)

Process Argv
--force-user-env --unity-launch --crash-reporter-id 8549cb84-3041-4f25-a346-0734e581d59b

Screen Reader
no

VM
0%

DESKTOP_SESSION
ubuntu

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu

XDG_SESSION_TYPE
x11

Extensions (4)

Extension
Author (truncated)
Version

python
ms-
2021.5.926500501

vscode-pylance
ms-
2021.6.1

jupyter
ms-
2021.6.999406279

hexeditor
ms-
1.7.0

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
pythonvspyt639:30300192
pythontb:30283811
vspre833:30321513
pythonptprofiler:30281270
vshan820:30294714
pythondataviewer:30285071
vscus158:30321503
vscorehov:30309549
vscod805:30301674

Hi @Eandreas1857, you have been updated to the new Native Notebooks interface :) What were you hoping to do with the Plot Viewer?

Hello @claudiaregio, the plot viewer is extremely useful when I have multiple plots within a notebook and want to be able to click between a few in another tab in VS quickly. Further, most of my plots are quite large and are difficult to look at in the notebook output, the plot viewer made it easy to open a plot and zoom in on particular parts of the plot. Honestly, this feature was my favorite part of using jupyter notebook in VScode rather than in a browser!

Glad to hear you enjoy the plot viewer :) Do you do anything with the plots after looking at them? What plotting libraries are you using? Have you tried the matplotlib widget?

Currently, I only need to view the plots, not interact with them. I am only using matplotlib and have not used the widget. I only need the capability to be able to open the plots as was available with the plot viewer.

Currently, I only need to view the plots, not interact with them. I am only using matplotlib and have not used the widget. I only need the capability to be able to open the plots as was available with the plot viewer.

Do you need to open the plots on the side so you can save them or
is the purpose to:

Be able to view is on the side, then re-run the cell & compare the difference between the two plots?
Or is it so that you can save the plot (image) to disc?
Or other?

Basically I'm trying to understand the why you are opening the image on the side. Thanks
FYI - in our insiders version we've added support to be able to save plots (without having to open them - please install VS Code Insiders and try this out).

I would like to be able to view plots on the side, then re-run the cell and compare the difference between the two plots, as well as compare with other plots scattered throughout the notebook. Also, the capability of zooming plots in and out is nice when my plots are large. I currently don't really care about saving them, though I will install the VScode insiders.

+1

+1

I am having the same issue with my vscode instance:
Version: 1.57.1
Commit: 507ce72a4466fbb27b715c3722558bb15afa9f48
Date: 2021-06-17T13:26:56.255Z
Electron: 12.0.7
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.8.0-59-generic snap
Plot Viewer was a very useful feature. Why was it moved to insiders if it already was on the main version?

Same here, would really appreciate Plot Viewer back.

+1

+1

+1

+1

+1

+1

+1

+1


Sema: Fix bugs when applying solutions containing type(of:) [4.0]

Description: Swift 4 mode brings new semantics for 'type(of:)' that make it behave more like an ordinary function lookup. The new implementation did not properly propagate the inferred type to the argument expression, causing crashes.

Scope of the issue: Affects users migrating Swift 3.2 code to 4.0.

Risk: Low, this makes the type(of:) code path behave a bit more like the normal function call path, coercing call arguments using the same mechanism.

Tested: New tests added.

Reviewed by: @DougGregor

Radar: rdar://problem/32435723

@swift-ci Please test

LGTM, thanks!


Have popup that says bibtex copied to clipboard

In Firefox. May get fixed with #8

Fixed with another issue.


Making Page for notifications.

@codervivek5 Please assign me the issue

where should you will add this notification.....this is already added in dashboard

Yes in Dash Board Only after Pressing Notification option the notifications should be visible

Here

@codervivek5 Lets discuss more in the discord

be ready for meeting at near 10pm
I'll ping for meet then kindly join

or meet then kindly join

Yes okay


CI: Drop sudo: false

Removes an old setting from Travis. See https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration

I think we have to follow https://github.com/rubygems/rubygems/issues/2534
Do you think you can try?

Thanks @olleolleolle


[MAISTRA-2.4] CVE proxy protocol sanitise and Timeout workaround

CVE - Proxy protocol: sanitise non utf8 chars in TLVs
Workaround for test timeouts in ci

/retest


Add missing SNAME macro optimization for StringName in some functions

In commit 6631f66c2a9d54dc80d57df60376c84ce1252d08 , reduz changed all theme , signal emission and call_deferred usages to use the new SNAME optimization for StringName.
I find that some occurrences were missing for these usages.
I've added the SNAME macro to these occurrences.
There's a lot of other functions that use StringName parameters ( especially Object.connect() and disconnect() which are widely used). Should macro usage be added to all StringName everywhere ?

especially Object.connect() and disconnect() which are widely used

SNAME only makes sense on functions that are called often, while connect/disconnect are usually called only once, or rarely. So I don't think we need that in that case.
That said, changes in this PR seems good.

Thanks!


Lists tasks owned by an advisor organization

The requirements owner would like to display tasks managed by an advisor organization at the advisor's organization's page.  This feature was removed in #450.  (See also #236).
The requirements owner is responsible for adding, modifying and inactivating tasks in ANET and the system of record which is an Excel spreadsheet.
However, the requirement owners are not in charge of the change management process.  The approval of adding, modifying and inactivating tasks are managed by individual advisor organizations and a review board.
The requirement owners have now assigned the tasks in ANET to other advisor organizations.
Another problem is most advisor organizations will not be responsible for tasks.  The problem before was when an organization didn't have tasks, it begged many questions.  For example, "Where are my organization's tasks?"

@VassilIordanov Issue resolved


Fixing CMake error

Fixes #286 .

Can we leave it configurable like -DCUDA_ARCH=xx?

Can we leave it configurable like -DCUDA_ARCH=xx?
Try this:
https://github.com/ztxz16/fastllm/pull/475
Supports -DCMAKE_CUDA_ARCHITECTURES=xx


Missing tags for versions 1.10 - 1.12

The latest tag is 1.9, you could please add the newer releases as tags as well? That makes comparing the versions easier. Thanks!

Howdy @sebix - I just added tags for those releases. Thanks for the issue!


Implement StrongholdPathListeners and stats/timing ui improvements

Fixes #28

Implements listeners for StrongholdPath, removing rendering+stats code out of MixinMinecraftServer.
Implements an in-game timer

Also fixes #45


about loss_in and loss_out in paper

Hello. Thank you for your hard-working!
In your paper page 5, you stated that
'The two loss formulations are not, however, equivalent. Because log is a concave function, Jensen’s Inequality [23] implies that  L_in ≤ L_out. One would thus expect L_out to be in the superior supervised loss function'
this might be a silly question, but I am wondering that why L_in ≤ L_out indicates loss_out is superior loss function ? In your paper page 6, you showed loss_out is more stable for traning and I understood it. But I can't connect this idea with being the superior loss function.

Hi, thanks for the question. Actually the correct loss is actually a simple cross-entropy. Hope this makes it easier to understand


fixed publicFile filename in keygen.go

should be publicFile instead of privateFile.

Nice catch!!! Can you put your name/email into the contributors file please.


How this cache task will identify if package.json is updated ?  ?

How this cache task will identify if package.json is updated ?  ?
In my current project, packaje.json is getting updated very frequently, like at-least once in a week.
In this case, when i am adding cache task, and when it is restoring old packages from cache, build is getting failed.
I want to know is there any option were this cache task will check if any new update in package.json file and create new cache is there is any update to package.json file.
Azure pipeline cache....

Please add this tag to this issue  Area: PipelineCaching

Follow this example: https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#nodejsnpm

This is not a support forum. Please look at https://developercommunity.visualstudio.com/spaces/21/index.html


flag to apply combination CR evaluation

Story
As a user of Combo,
I want to be able to apply the evaluation of Combo,
So that there is an extra layer of optional automation.
Summary
This relies on #2 being complete. As it currently stands, #2 will make it so that the template and combinations can be evaluated but will not be applied to the cluster. This task should add an optional flag that will enable the creator to have Combo automatically apply the evaluation to the cluster.
Acceptance Criteria

Can set a flag to apply the evaluation to the cluster
Has E2E test(s) to validate the logic and implementation

One thing to note here is that since combo can template out any arbitrary resource, the ability to then create that resource would require sufficiently permissive RBAC: essentially * on create for all APIs. Currently the RBAC for combo is nicely scoped and limited to only its own APIs:
rules:
- apiGroups: ["combo.io"]
  resources: ["*"]
  verbs: ["get", "watch", "list", "update"]

I'm not sure the additional RBAC requirements block the feature entirely but it's worth thinking about from a security point of view.

See https://github.com/operator-framework/combo/pull/78#issuecomment-1081875995 for a description of the state of the first implementation of this feature.


Package Initialization Error

Diagnostics
Request: GET atom://teletype/portal/REDACTED/protocol-version

Versions
Teletype version: v0.13.4
Atom version: 1.45.0
Platform: win32

Thanks for reaching out!
We require the template to be filled out on all new issues and pull requests. We do this so that we can be certain we have all the information we need to address your submission efficiently. This allows the maintainers to spend more time fixing bugs, implementing enhancements, and reviewing and merging pull requests.
Thanks for understanding and meeting us half way 😀


Refactored types in frame to use size_t (#fix 34)

It compiles to ARM, but
I have a doubt about line 124.
writeDataOrThrow(ptr, cast(ulong) frame.payload.length);

perhaps it should be changed to cast(size_t)?

Regarding your question about line 124, the writeDataOrThrow call there does need to write a ulong, since the spec requires 64 bits in that part of the frame. ARM only seems to complain when a long or ulong is used instead of size_t when doing array indexing operations.


Autoflip cannot rapidly switch between two faces in scene.

I was attempting to run autoflip on a scene like the one below which rapidly switches between two faces to autocrop to portrait 9:16.
https://www.youtube.com/watch?v=VlSkPA60ujQ
However, I can't seem to tweak the advanced settings to get it to switch to between the two faces. It ends up slow panning or ending up somewhere in the middle of the scene with no faces.
Setting FACE_CORE_LANDMARKS is_required: TRUE will solve the issue but results in padding.
Is there anyway to get the scene_camera_motion_analyzer to rapidly switch between two faces without padding or forcing it to a different aspect ratio. Would like it to crop it as 9:16.
Thanks!

Hi @darkrider87,
Thank you for sharing this issue with us.  I did some testing on your example video and I believe the root problem is autoflip is not detecting shot boundaries in this dark pub scene.  Specifically, when the camera switches focus back and forth (between Jesse Eisenberg and Rooney Mara) you may notice the scenes color and brightness are very similar and the ShotBoundaryDetection calculator fails to tell a shot change occurred.
I'd recommend adjusting your ShotBoundaryCalculatorOptions to be a bit more aggressive for this clip.  The specific settings I used were:
window_size: 3
min_shot_measure: 0.95
min_motion_with_shot_measure: 0.0001

Also, because this scene is pretty focused on faces, I disabled object detection into the SignalFusingCalculator (you could also just reduce their weights closer to zero):
  input_stream: "shot_change"
  input_stream: "face_regions"
#  input_stream: "object_regions"
  output_stream: "salient_regions"

With those changes, the results look much improved:
https://photos.app.goo.gl/SVe4baL5jwRf5gDP7
In the long term, we hope to develop more reliable ways to detect shot boundaries across all scene types. In the mean time, let me know if I can provide anymore guidance on this video.
-Nathan Frey

This is awesome! Thanks a lot.

Our team did come across a few more issues. Would it be possible to set up a quick call to discuss this in more detail? You can find me on LinkedIn...
http://linkedin.com/in/muskan-srivastava

Hi @darkrider87, we would now recommend using a new shot detection calculator based on the TransNetV2 deep model.  You can find that calculator and example code here:
https://github.com/googleinterns/intern-for-design/tree/master/autoflip_integrated_calculators
-Nathan Frey


Reconciliation of request failed: failed to get machine for node

Hello together,
We deploy Kubernetes Clusters on Hetzner with kubermatic/kubeone which uses by default the kubermatic/machine-controller addon.
I don't know if this is an issue, because (i think so) everything works fine.
But when I view the logs of the machine controller i see the following logs occuring every second:
I0622 12:51:39.408545       1 node_csr_approver.go:103] Reconciling CSR csr-dnfp5
E0622 12:51:39.409188       1 node_csr_approver.go:89] Reconciliation of request /csr-dnfp5 failed: failed to get machine for node 'control-plane-1': failed to get machine for given node name 'control-plane-1'
I0622 12:51:43.525874       1 node_csr_approver.go:103] Reconciling CSR csr-sw2mt
E0622 12:51:43.527863       1 node_csr_approver.go:89] Reconciliation of request /csr-sw2mt failed: failed to get machine for node 'control-plane-3': failed to get machine for given node name 'control-plane-3'
I0622 12:51:44.107206       1 node_csr_approver.go:103] Reconciling CSR csr-jb2h6
E0622 12:51:44.108756       1 node_csr_approver.go:89] Reconciliation of request /csr-jb2h6 failed: failed to get machine for node 'control-plane-2': failed to get machine for given node name 'control-plane-2'

And my kube-system namespace is flooded with csr's. I don't think this is good for my clusters.
Regards,
Dome

@dcardellino what kubeone version are you running?

@kron4eg We are running the following version:
kubeone version
{
  "kubeone": {
    "major": "1",
    "minor": "4",
    "gitVersion": "1.4.3",
    "gitCommit": "717787f2287964e5793d80ec8ca2c2169936b0ac",
    "gitTreeState": "",
    "buildDate": "2022-05-11T14:18:03Z",
    "goVersion": "go1.18.1",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "43",
    "gitVersion": "v1.43.2",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

@kron4eg Any updates here?

@dcardellino is it possible that you tesd-drive dev build from the master?

@kron4eg
Is there a Make target to build from master branch?

make build will build dist/kubeone

I'm not really sure why kubelet keep generating new certificate requests, probably it's some kind of a bug in the kubelet. But I've observed that if I approve all pending certificates their generation is stopped.

@dcardellino Since #2199, we're not able to reproduce the issue any longer (neither manually, nor in the E2E tests). Can you give KubeOne 1.5.0 a try?

@xmudrii Sorry, I forgot to mention that I recently approved the pending CSRs manually with kubectl certificate approve <csr-name>. And now everything works fine! But thank you!

Hey, I just wanted to let you know that I had the same issue today with KubeOne 1.5.4 and k8s 1.25.11. Sadly, I also manually approved the CSRs before finding this issue.
I still have about 280 pending requests, but they seemingly get deleted when becoming older than 24h. I also have these log entries:
2024-01-31 15:15:21.715	E0131 14:15:21.715009       1 node_csr_approver.go:89] Reconciliation of request /csr-5bjq6 failed: failed to get machine for node 'staging-control-plane-3': failed to get machine for given node name 'staging-control-plane-3'
2024-01-31 15:15:21.715	I0131 14:15:21.714902       1 node_csr_approver.go:103] Reconciling CSR csr-5bjq6
2024-01-31 15:15:21.614	E0131 14:15:21.614419       1 node_csr_approver.go:89] Reconciliation of request /csr-zsg56 failed: failed to get machine for node 'staging-control-plane-2': failed to get machine for given node name 'staging-control-plane-2'
2024-01-31 15:15:21.614	I0131 14:15:21.614329       1 node_csr_approver.go:103] Reconciling CSR csr-zsg56
2024-01-31 15:15:21.515	E0131 14:15:21.515034       1 node_csr_approver.go:89] Reconciliation of request /csr-s99s4 failed: failed to get machine for node 'staging-control-plane-1': failed to get machine for given node name 'staging-control-plane-1'
2024-01-31 15:15:21.515	I0131 14:15:21.514952       1 node_csr_approver.go:103] Reconciling CSR csr-s99s4

even after running KubeOne again (exact command: kubeone apply --manifest kubeone.yaml --tfjson output.json --upgrade-machine-deployments --auto-approve) and deleting the MachineController pod manually, so it gets recreated/restarted.
Either the fix didn't work or my MachineController somehow is running an old version (its image is quay.io/kubermatic/machine-controller:v1.56.2 currently). :thinking:


Plus Minus Code - WHVX+64J-MINSK

It is the same short code, except for the deterministic "minus" part containing the locality. Locality is a table lookup, so this requires adding a table with localities the the standard. Filtered by population (>1mln people or the largest city in the square) to keep the list short. The has few advantages.

Plus Minus Codes can be decoded by offline tools
The code become machine readable (unlike short Plus Code with locality) and long code more human readable

Short Plus Ccodes with locality like "WHVX+64J, Minsk, Belarus" are not machine readable, because it is impossible to understand if text after the first comma is locality without making a request to external service. With Plus Minus Code WHVX+64J-MINSK you don't need the external heuristics.

This repository is related to plus codes. What you are referring to is not a plus code. I recommend that this issue be closed as out of project scope.

@fulldecent this is a proposed backward compatible extension to Google's plus code. Where should I send it to if not this project?

I think the Google Groups makes sense https://groups.google.com/forum/#!forum/open-location-code

From (newly updated) https://github.com/google/open-location-code/wiki/FAQ#reference-location-dataset :
The open source libraries support conversion to/from addresses using the latlng of the reference location. Callers will need to convert place names to/from latlng using a geocoding system.
Providing a global dataset isn't within scope of this project. For a potential free alternative, see Open Street Map and derived geocoding service Nominatim.

Not sure I need plus codes then. https://what3words.com/ looks like a good alternative.

Not sure I need plus codes then. https://what3words.com/ looks like a good alternative.

Depends on your use case. Ultimately W3W is not open source, nor is it scalable (i.e. variable granularity/accuracy) and it's not free. Its also not (IMHO) as easy to integrate into a database given the variable string lengths 🤷🏻♂


Cannot create Publisher using built-in message types

(Following up on #7)
I cannot create a publisher which uses a built-in message type (e.g. Float64, String, etc.). Running:
#!/usr/bin/env julia
using RobotOS
@rosimport std_msgs.msg: Float64
rostypegen()
using std_msgs.msg

pub_topic = "out_float"

pub = Publisher{Float64Msg}(pub_topic, queue_size = 10)

gives the following error:
ERROR: LoadError: Type (std_msgs.msg.Float64Msg) is not generated
 in error at ./error.jl:21
 in _get_rospy_class at /home/mpc/.julia/v0.4/RobotOS/src/gentypes.jl:608
 in call at /home/mpc/.julia/v0.4/RobotOS/src/pubsub.jl:9
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in process_options at ./client.jl:308
 in _start at ./client.jl:411
while loading /home/mpc/GitHub/ROS-Julia/workspace/src/mpc_julia_sim/scripts/test/test_pub_type.jl, in expression starting on line 19

I can, however, use the data type constructor without any issue. That is,
pub_data = Float64Msg()

works.

Thanks for noticing and reporting the bug. Hopefully this fix is a little cleaner way of doing the renaming.


Small landmark model for face recognition

Hi, is there any reason why the method to generate face encodings (def face_encoding) in face_recognition/api.py uses the small landmark model?
Would it generate more accurate encodings when using the large model?

The small model is faster.
The face landmarks are used as control points to do a simple affine transform on the face image to align it to a normalized position, so using the larger model with more face points shouldn't change the result since those extra points aren't used in the transform anyway.