Disk Space Issue : Migrate one image at a time

Instead of downloading the entire v1 registry with all images and tags, can we download one at a time and push to the v2 registry?
The new process would look like :

Query v1 registry for all repositories and their tags
Pull one image (with a particular tag) to local docker engine
Retag it and push to v2
Delete the image from local docker engine
Repeat this for all images in v1

This addresses the major issue of Disk space on the docker engine host not being sufficient to hold entire v1 registry.
We use S3 for storing our images and all the images cannot fit on a single host's disk.

I've thought about that as a potential way to migrate the images but that only works if you have a v1 registry and a v2 registry using different URLs.  I think it is still a good approach as I agree that needing a disk large enough to hold your entire registry could be very painful.
As a way to work around this, your process makes sense but my hesitation is that you will end up pulling many layers multiple times if a large number of your images have the same base layers.  For example, if you base all of your images on debian:jessie, it will pull the base every time since you are removing it every time you pull an image down.  I am not aware of a good way to detect what might be a commonly used base to keep those image layers cached during the pull besides having the user provide what those images might be.
There are a few methods I've also thought about in how to break it up into batches:

n images to pull, re-tag, and push; then delete all of the images locally
Loop through all images in the same namespace before deleting locally
Loop through all tags of the same repository before deleting locally
Check the local disk space and ensure that X amount of disk space is free before removing the currently cached images locally

I don't love any of these solutions at the moment and am trying to think of a better way to handle large registries.

Thanks. I did not think about the local caching of image layers. I agree that none of 1-4 are ideal, mostly because of DNS switching issue when the URLs are the same, I do like the idea no. 1 though, which is a very simple way of implementing batch migrations.

If we use https://github.com/mrbobbytables/dockproxy, will DNS switching still be a problem?
Although I agree that pulling layers multiple times is a bigger issue.

I've run in the same issue.
A solution could be to export the ?q= parameter and read it from the env, so, I can migrate group of images
IMAGE_LIST="$(curl -s https://${AUTH_CREDS}@${V1_REGISTRY}/v1/search?q=${QUERY_SEARCH:''} | jq -r '.results | .[] | .name')"

Ah yes, that would be a great idea to break the chunks into more manageable portions. Thanks!  I'll implement that shortly as an optional parameter.

:+1:

So I actually decided to go about this slightly differently.  While I think that using the available search filter is good, I think that using grep to search allows for greater flexibility due to the ability to use regex.  As far as I can tell, the v1 registry can't handle regex.

Filter is tricky to use. Simpler solution would be splitting image list into chunks of X items. This way you'll have cache locality (base layers are reused to some degree) and limited disk space requirements while the only setting is required: number of images in a chunk.

Unfortunately it really isn't that easy in all cases due to many other
variables in play. The biggest being that you might want to replace a v1
with a v2.  There is a PR that was submitted that runs into that issue and
I was waiting on updates for that before doing anything else around that.
On Sunday, October 18, 2015, Ian Babrou<EMAIL_ADDRESS>wrote:

Filter is tricky to use. Simpler solution would be splitting image list
into chunks of X items. This way you'll have cache locality (base layers
are reused to some degree) and limited disk space requirements while the
only setting is required: number of images in a chunk.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/migrator/issues/14#issuecomment-149042169.

--
Matt Bentley
Solutions Engineer
Docker, Inc.
317.966.8225

For what it is worth, I do still want to add the ability to migrate images in chunks when the source and destination registries differ.

This is My solution for this problem.
https://github.com/duyanghao/migrator/tree/Tencent
There are a few methods I've also thought about in how to break it up into batches:

n images to pull, re-tag, and push; then delete all of the images locally
Loop through all images in the same namespace before deleting locally
Loop through all tags of the same repository before deleting locally
Check the local disk space and ensure that X amount of disk space is free before removing the currently cached images locally

I combine the 2 and 4 methods:
Loop through all images in the same namespace before deleting locally under the precondition of 100G amount of free disk space.
@mbentley @ankushagarwal

This is my solution for this problem: @ankushagarwal @mbentley
https://github.com/duyanghao/migrator/tree/Tencent
There are a few methods I've also thought about in how to break it up into batches:

n images to pull, re-tag, and push; then delete all of the images locally
Loop through all images in the same namespace before deleting locally
Loop through all tags of the same repository before deleting locally
Check the local disk space and ensure that X amount of disk space is free before removing the currently cached images locally

I combine the method 2 and 4:
Loop through all images in the same namespace before deleting locally under the precondition of 100G amount of free disk space.

Added https://github.com/docker/migrator/pull/81 to be able to set the number of images you would like to  pull incrementally.


Perform case-insensitive comparion for action names

Fix for #2521

We should add a unit test for this.

Added unit test

This PR has been merged and released as part of version 3.7.105

@normj The issue persists, and the changes were not found in the master branch.


Idempotence tests incorrectly succeed when Ansible callback plugin is not default

Prerequisites

[X] This was not already reported in the past (duplicate check)
[X] It does reproduce it with code from main branch (latest unreleased version)
[X] I include a minimal example for reproducing the bug
[X] The bug is not trivial, as for those a direct pull-request is preferred
[X] Running pip check does not report any conflicts
[X] I was able to reproduce the issue on a different machine
[X] The issue is not specific to any driver other than 'default' one

Environment
❯ molecule --version
molecule 6.0.2 using python 3.11
    ansible:2.15.4
    azure:23.5.0 from molecule_plugins
    containers:23.5.0 from molecule_plugins requiring collections: ansible.posix>=1.3.0 community.docker>=1.9.1 containers.podman>=1.8.1
    default:6.0.2 from molecule
    docker:23.5.0 from molecule_plugins requiring collections: community.docker>=3.0.2 ansible.posix>=1.4.0
    ec2:23.5.0 from molecule_plugins
    gce:23.5.0 from molecule_plugins requiring collections: google.cloud>=1.0.2 community.crypto>=1.8.0
    podman:23.5.0 from molecule_plugins requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0
    vagrant:23.5.0 from molecule_plugins

Linux 4.18.0-348.23.1.el8_5.x86_64 #1 SMP Fri Jul 22 09:30:58 CDT 2022 x86_64 x86_64 x86_64 GNU/Linux

❯ python3 --version
Python 3.11.2
What happened
What happened?
We have CI that uses molecule to automatically test pull requests to our internal company repositories of Ansible roles. We discovered that idempotent tests were showing as passed in CI, but when locally they would fail. This was narrowed down to the difference being that locally, we use the default ansible.builtin.default callback, while we use ansible.builtin.junit callback in CI.
What did we expect to happen?
We expected tests to fail in CI as they were locally.
Root cause?
I believe this is the root cause. In idempotence.py, we have the following function:
https://github.com/ansible/molecule/blob/01631c7c8e1d7c053bf3c57e4adafcc7badc70d4/src/molecule/command/idempotence.py#L62-L78
Specifically, the following line:
changed = re.search(r"(changed=[1-9][0-9]*)", output)

molecule is detecting the changed_when status from the ansible output by specifically looking for the changed=[some_number] format that exists in the default callback. However, with other callback plugins Ansible doesn't print those lines and instead uses other formats. For example, in ansible.builtin.json we instead get:
   "stats": {
        "localhost": {
            "changed": 2,
            "failures": 0,
            "ignored": 0,
            "ok": 4,
            "rescued": 0,
            "skipped": 2,
            "unreachable": 0
        }
    }

I'm happy to put in PRs for this, but I'd like to hear what the maintainers think about how to restructure this function. Writing a regex per parser seems highly undesirable and a pain to maintain. String parsing doesn't seem like the most ideal way to go about this, but because there's so many different callback plugin types that might be unavoidable.
Reproducing example
Gist here: https://gist.github.com/chrwang/bad5de8f073bd6a8cedc1332f2737768
Directory structure:

❯ ll
total 12
-rw-r--r-- 1 chwang swdev 285 Oct  5 09:26 converge.yml
-rw-r--r-- 1 chwang swdev 153 Oct  5 09:24 molecule.yml
-rw-r--r-- 1 chwang swdev 167 Oct  5 09:25 verify.yml
❯ cd ../
❯ ll
total 0
drwxr-xr-x 2 chwang swdev 64 Oct  5 09:22 idempotence_test
❯ cd ../
❯ ll
total 0
drwxr-xr-x 3 chwang swdev 30 Oct  5 09:27 molecule
❯ pwd
/tmp/molecule_test

Command run for success:
❯ PY_COLORS=1 molecule test -s idempotence_test

Command(s) run for failure:
❯ export ANSIBLE_STDOUT_CALLBACK=ansible.builtin.junit
❯ PY_COLORS=1 molecule test -s idempotence_test

Out of interest does setting ANSIBLE_CALLBACK_RESULT_FORMAT have an effect on this? I find the output more readable when it is set to yaml rather than the default which I guess is json.

Out of interest does setting ANSIBLE_CALLBACK_RESULT_FORMAT have an effect on this? I find the output more readable when it is set to yaml rather than the default which I guess is json.

It does not in this case, as the format of the ok/changed/failed lines in the play recap seem to be independent of ANSIBLE_CALLBACK_RESULT_FORMAT and rather depends on the callback plugin itself instead. i.e. in both json and yaml modes I see the following format printed for the recap:
PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

I think that env var controls output format from tasks themselves rather than the recap, and as molecule is looking for the lines from the recap, whatever gets printed by the task itself shouldn't matter (unless you happen to print changed=2 exactly in a debug message or something...)

I PR would be welcomed here. The right way to address this should be to enforce use of a specific callback (included in core), so we can analyze the output.

RFC: Use the ansible.posix.json callback? That seems the most easily parseable / standard one to use.


Support for cyclic graphs and recursive workflows

Describe the solution you'd like
I would like to be able to perform a rule recursively. I am currently trying to develop a bioinformatics pipeline which involves collating multiple datasets into one for the sake of downstream analysis. One limitation is the software I am using, PLINK-1.9, can only merge two file-sets together at a time. I am trying to keep this as dynamic as possible and have created a recursively firing rule with a custom naming system, naming logic and input functions to determine the name of the input files needed for a given output file within the recursion loop. Eventually, a final output will be produced which breaks the cyclic graph (is valid input for downstream processes for requested files).
Describe alternatives you've considered
I have considered running the recursion logic using the run: property. It will most likely work (haven't attempted it yet), however I am working on a cluster environment and would prefer if I could leverage that to submit each recursion independently to speed up computational times. If I use run:, that's not really an option anymore.
Additional context
I understand there are definitely inherit risks with recursive programming, especially unintentional recursion. Given that, would it not be an option to add some sort of flag or label syntax to a rule to indicate recursion is explicitly intended and/or include a int value of the intended recursion depth? That way there is at least some constraint preventing complete and unintentional computational melt-downs.

I ran into something similar where I was combining samples in pairs and combining the results in pairs, etc, until I had one output.  I think generally any cyclic or recursive rule would eventually have to terminate and could, therefore, be unraveled into a DAG.  I'm not sure the syntatic sugar would add much to readability or coding.  For my application, I'm assuming you have a list of samples to run, called torun.  The intermediate files contain the results for start to end, as s_{start}_e_{end}.out.
def intersect_input(wildcards):
    start = int(wildcards.start)
    end = int(wildcards.end)
    right_start = 2**math.floor(math.log2(end-start)) + start
    left_end = right_start-1

    # get the input file
    if start == left_end:
        left = paths['input'].format(
            id=torun[start],
            chromosome=wildcards.chromosome)
    else:
        left = paths['combined_temp'].format(
                start=start,
                end=left_end,
                chromosome=wildcards.chromosome)

    if right_start == end:
        right = paths['input'].format(
            id=torun[end],
            chromosome=wildcards.chromosome)
    else:
        right = paths['combined_temp'].format(
                start=right_start,
                end=end,
                chromosome=wildcards.chromosome)

    return [left, right]

rule intersect_sub:
    input:
        intersect_input

    output:
        temp(paths['combined_temp'])

    wildcard_constraints:
        start='\d+',
        end='\d+'

    shell:
        'cat {input} > {output}'  # your code goes here

rule intersect:
    input:
        lambda wc: paths['combined_temp'].format(
            start=0,
            end=len(torun)-1,
            chromosome=wc.chromosome)

    output:
        paths['combined']

    shell:
        'cp {input} {output}'

The rule intersect requests the combined file with start 0 and end the length of the input list.  That is built recursively with the intersect_sub rule, which finds the midpoint of start and end and requests the files (start, mid) and (mid, end).  If mid is start or end, the request resolves into an input file (terminating the recursion).
Since each rule is separate it can be run in parallel and submitted to a cluster as separate rules.  I hope this can be a good starting point for your rule.  Also note that the intersect_sub rule is actually recursive, so snakemake can already support your feature.  It's up to you to make sure it terminates though!

In my case I've avoided cyclic rules using configuration step before main simulation. So snakemake actually unwraps supposedly undefined cycle into DAG, provides configuration file as a result and then snakemake works well.
I'd add that from Computer Science point of view cycles and recursions producing undetermined complexity so I appreciate current Snakemake approach.

I'd add that from Computer Science point of view cycles and recursions producing undetermined complexity so I appreciate current Snakemake approach.

Agree on the complexity added, but there are some cases where users may want to run cyclic workflows. Weather forecast models are normally run ad-infinitum as more data is added, having a cyclic workflow where tasks trigger themselves, or trigger other tasks that eventually produce a cycle.
That's the main use case for Cylc, but Cylc isn't data-oriented like Snakemake, and also doesn't scale out as well. So it would be useful to have an option with cycles like Cylc, but with the other features of Snakemake IMO.
StackStorm is another workflow manager, more automation oriented, supporting cloud, HPC, containers, multiple integrations, etc. It has a workflow engine called Orquesta that also supports cycles.
The implementation definitely adds complexity, but there are existing workflow managers like these two examples that show that it should be doable. In computer science, and other fields, there are also some papers that show techniques ^1 where a cyclic workflow can be de-constructed into one or multiple DAG's (I think others mentioned something similar in the issues here).
So perhaps we could support something like that, read a workflow, and if it detects cycles, just unroll the graph and re-trigger the same DAG multiple times until it reaches a condition that stops recursion/cycles, or keep running until the user stop it.


Last repository tag picked over first reachable when determining version

Hi,
I have the following situation:
I have a Git repository, where the master branch serves as the development head, with tags for release points, and maintenance branches for previous major releases that receive bugfixes and tags for updates.
.
.
C
| B'
| |
B / A'
|   |
A __/
|
o

Where A, B and C are releases (let's presume v1.0.0, v2.0.0 and v3.0.0 respectively), and A' and B' are maintenance commits for the respective releases (let's assume v1.0.1 and v2.0.1).
When I generate an SBOM document for the module, as it is at B', the version detection malfunctions, and determines the version of the module to be whatever the latest tag is (in the example, this is C, i.e. v3.0.0). I've verified that it is the version control system tags that affect this, as after removing all tags from the repository, the result was a pseudo-version.
Is this the Go tooling that's doing this, or the cyclonedx-gomod utility incorrectly determining the version of the module?
Apologies in advance if this isn't the cyclonedx-gomod utilities fault - my Go-fu isn't great, and I'm just managing this aspect of the subject project.

Hey, @nscuro!
Surely!
The repository in question is git.zabbix.com/scm/ap/plugin-support.
The master branch is the development head, and branches release/6.0 and release/6.4 are the maintenance branches.
There was an attempt at one point to start properly versioning these with tags, for publishing to the module index, but that got canceled, hence why the tags are in a bit of a disarray. If you check out the release/6.0 branch and make, it should generate an sbom.json file (the Makefile contains only rules for generating these).
If you look at the metadata.component and metadata.version fields
jq '.metadata.component | .["bom-ref"], .version' sbom.json

the version that is detected is v6.4.3-0.20230425081451-9e40ea6472c9 (6.4.3 which is the last tag that was issued, to commit 9c948d2bdd6, without a v prefix, and 9e40ea6472c9 is actually v6.4.2).
The version I would expect to be detected for that branch is 6.0.18-0..., as that is the last tag on that branch.
The cyclonedx-gomod utility behaves more like git log --tags, because, as man 1 git-log states:
--tags[=<pattern>]
           Pretend as if all the refs in refs/tags are listed on the command line as <commit>.
           If <pattern> is given, limit tags to ones matching given shell glob.
           If pattern lacks ?, *, or [, /* at the end is implied.

This would then make sense, as if you issue git log --tags, you get a commit log of tagged commits and commits leading up to the tags that were part of a merge.
_I'm reading the version detection code in (GetLatestTag in internal/gomod/version.go mainly) to properly understood what is being derived.

Hey, @nscuro!
Sorry for the lack of activity; I was ill the last couple of weeks, so this didn't really move along anywhere during that time.
What I've found is the following:
In internal/gomod/version.go:GetVersionFromTag we start off by determining the commit the HEAD reference is currently pointing at. Using that, we then attempt to get the latest tag, relative to that commit object, via GetLatestTag.
In internal/gomod/version.go:GetLatestTag  we start off by getting a list of all the tags in the repository, with the list being in order from oldest to latest tag (probably just an implementation detail).
We then iterate over, skipping those tags that do not conform to the SemVer specification. For the tags that do, we then determine whether it occurs before or at the commit pointed to by HEAD. This is where the issue lays.
We determine whether the the tagged commit object occurs before or at the HEAD commit by whether it is older than the commit at HEAD based on HEADs commit and author date. The issue here, as in my case, is that the commit on the maintenance branch is chronologically newer than the latest tag in the repository, making this test succeed.
The additional issue here is that we process a list of all tags, regardless of whether they are reachable from the HEAD commit, and only rely on the date.
I briefly looked at the go-git module documentation, and didn't see a way of retrieving tags only reachable from a specified commit. The LogOptions type has a From field which, when specified, does do this, but this isn't exactly what we want.
The only option I see here, with what is available to us, is requesting a full log of the repository, up to the commit object HEAD is pointing at, and then collecting references to the tagged commit objects, discarding those that can't be reached from the HEAD commit (go-git doesn't provide a method to determine this).
That seems like quite a hack to me, just to get an appropriate tag list.
Maybe this should be filed as an issue/feature request with the go-git project people? This would require change in cyclonedx-gomod itself regardless of the path you take.


Introduce variables

The variables within the packerfiles now are the ones needed.
This prepares for multiple purpose image building (needed) with multiple distros (if needed).

@cla-bot please-recheck-my-cla

Thanks for the review @MostAwesomeDude !


how to use butterknife in a library module?

Now, I have put my baseactivity in a serparated module,and I inject the butterknife in the baseactivity,Then I would implements the baseactivity in my app module, The problem is that it throw NullPointerException  on widgets which defined by butterknife,anyone can help me?

This configuration is not supported. Dupe of #808.

Hi @JakeWharton any updates on this yet? May I ask what's the issue that's blocking this configuration? Thanks.


license issue in fury-core jar

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
The META-INF/LICENSE in https://repository.apache.org/content/repositories/orgapachefury-1043/org/apache/fury/fury-core/0.6.0/fury-core-0.6.0.jar is the source license. There should be a dedicated license file for the jars.
I see that the janino and kryo licenses are included in META-INF/licenses but they should be mentioned in META-INF/LICENSE.
Component(s)
Java
Minimal reproduce step
n/a
What did you expect to see?
better LICENSE in jar
What did you see instead?
n/a
Anything Else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

We have text The text of each license is also included in licenses/LICENSE-[project].txt in java/fury-core/src/main/resources/META-INF/LICENSE:
The following components are provided under the BSD-3-Clause License. See project link for details.
The text of each license is also included in licenses/LICENSE-[project].txt.

* kryo (https://github.com/EsotericSoftware/kryo)
    Files:
      java/fury-core/src/main/java/org/apache/fury/collection/FuryObjectMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/IdentityMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/IdentityObjectIntMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/LongMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/ObjectIntMap.java
      java/fury-core/src/main/java/org/apache/fury/type/Generics.java

* janino (https://github.com/janino-compiler/janino)
    Files:
      Shaded classes under org/apache/fury/shaded/org/codehaus/janino/*

Do we need to update it into specific file name in this LICENSE file?

We have text The text of each license is also included in licenses/LICENSE-[project].txt in java/fury-core/src/main/resources/META-INF/LICENSE:
The following components are provided under the BSD-3-Clause License. See project link for details.
The text of each license is also included in licenses/LICENSE-[project].txt.

* kryo (https://github.com/EsotericSoftware/kryo)
    Files:
      java/fury-core/src/main/java/org/apache/fury/collection/FuryObjectMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/IdentityMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/IdentityObjectIntMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/LongMap.java
      java/fury-core/src/main/java/org/apache/fury/collection/ObjectIntMap.java
      java/fury-core/src/main/java/org/apache/fury/type/Generics.java

* janino (https://github.com/janino-compiler/janino)
    Files:
      Shaded classes under org/apache/fury/shaded/org/codehaus/janino/*

Do we need to update it into specific file name in this LICENSE file?

the LICENSE doesn't mention shaded/org/codehaus/commons/ - there are classes in fury-core that are shaded versions of commons compiler classes. I guess this is part of Janino but we should update the Janino section to mention shaded/org/codehaus/commons/.

I see, thanks. I will fix this soon.


Update README.md

一份力量，ref http://www.xinhuanet.com/politics/2015-06/06/c_127884245.htm

Thanks for your contribution! Happy new year~

Thanks for your contribution! Happy new year~


Docs request: Workspaces -- do we add albatross to tool.uv.sources

With uv==0.4.0 looking at https://docs.astral.sh/uv/concepts/workspaces/#workspace-layouts, it details:

albatross in src
bird-feeder in packages/bird-feeder/src
seeds in packages/seeds/src

The docs don't mention if:

albatross should be added to tool.uv.sources
If workspace = true is required, and when to use path

The below is what I got working, but it does not match the example shown here: https://docs.astral.sh/uv/concepts/workspaces/#workspace-sources
[tool.uv.sources]
bird-feeder = {workspace = true}
albatross = {workspace = true}

[tool.uv.workspace]
members = ["packages/*"]

More specifically, albatross = {workspace = true} is not shown in the docs. But when I remove it and run uv lock, I get:
error: Failed to build: `aviary-hotpotqa @ file:///path/to/repo/packages/bird-feeder`
  Caused by: Failed to parse entry for: `albatross`
  Caused by: Package is not included as workspace package in `tool.uv.workspace`

What do you have in the pyproject.toml for file:///path/to/repo/packages/bird-feeder?

Here is that pyproject.toml:
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools>=64", "setuptools_scm>=8"]

[project]
dependencies = [
    "albatross",
]
dynamic = ["version"]
name = "bird-feeder"
requires-python = ">=3.11"

[tool.ruff]
extend = "../../pyproject.toml"

[tool.setuptools.packages.find]
where = ["src"]

[tool.setuptools_scm]
root = "../.."
version_file = "src/bird-feeder/version.py"

Cool, so in that pyproject.toml, you should include:
[tool.uv.sources]
albatross = { workspace = true }

The sources should be defined wherever the dependencies are included. Does that make sense?

Yeah that makes sense! Thanks for working through it
I think then we should update the example pyproject.toml's in https://docs.astral.sh/uv/concepts/workspaces/#getting-started to include that

Good idea!

Cool, so in that pyproject.toml, you should include:
[tool.uv.sources]
albatross = { workspace = true }

The sources should be defined wherever the dependencies are included. Does that make sense?

OK, that matches up with the source code in https://github.com/astral-sh/uv/blob/main/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder/pyproject.toml.
Then I have a question: Is it not OK to move the tool.uv.sources declaration to the root like in this commit: https://github.com/thorbenk/uv-workspaces-question/commit/0ca5611a69c6f5c77ad1d09cb2589291c11974e9
I want to achieve the following:

each package does not itself know that it is part of a workspace (actually, each package for me is a git submodule, and released as an independent python package)
Only the workspace root ties together all packages into a workspace

Am I misunderstanding the workspace concept?

@thorbenk -- I think that should work because members inherit sources from the root. Did it not work?

@charliermarsh: Sorry, I did not write that clearly.

However, I was wondering if this was intentional.
Happy to hear that it is!
Actually, I was using the setup like above and it was working well for me in an internal project.
Then today I ported to using the new explicit indexes and I ran into issues, prompting me to wonder whether I was using supported behavior.
I will try to create a minimal reproducible example for this.
Thanks for an amazing uv experience!


srpm builds started to fail on f38 with packaging.version.InvalidVersion: Invalid version: '287.32.g9eba04e'

What happened? What is the problem?
A few days ago, most of our projects started to fail srpm builds during upstream GitHub PR tests. E. g. in https://github.com/cockpit-project/cockpit-machines/pull/1021 , whose SRPM build fails like this:
2023-04-14 18:38:19.213 upstream.py       DEBUG  No ref given or is not glob pattern
2023-04-14 18:38:19.213 upstream.py       DEBUG  We're about to `git-describe` the upstream repository /tmp/tmpma26yg7g.
2023-04-14 18:38:19.213 commands.py       DEBUG  Command: git describe --tags --abbrev=0 --match=?*
2023-04-14 18:38:19.217 logging.py        DEBUG  287
2023-04-14 18:38:19.218 upstream.py       DEBUG  Version: 287
2023-04-14 18:38:19.218 upstream.py       DEBUG  Sanitized version: 287
2023-04-14 18:38:19.218 upstream.py       DEBUG  Name + version = cockpit-machines-287
[...]
2023-04-14 18:38:25.011 upstream.py       INFO   Created archive:
2023-04-14 18:38:25.011 upstream.py       INFO   	parsed   path: /tmp/tmpma26yg7g/cockpit-machines-287.32.g9eba04e.tar.xz
2023-04-14 18:38:25.012 upstream.py       INFO   	absolute path: /tmp/tmpma26yg7g/cockpit-machines-287.32.g9eba04e.tar.xz
2023-04-14 18:38:25.079 commands.py       DEBUG  Command: git describe --tags --long --match *
2023-04-14 18:38:25.082 logging.py        DEBUG  287-32-g9eba04e
2023-04-14 18:38:25.088 commands.py       DEBUG  Command: git describe --tags --long --match *
2023-04-14 18:38:25.091 logging.py        DEBUG  287-32-g9eba04e
2023-04-14 18:38:25.091 base_git.py       DEBUG  Running ActionName.fix_spec.
2023-04-14 18:38:25.091 base_git.py       DEBUG  Running default implementation for ActionName.fix_spec.
2023-04-14 18:38:25.285 upstream.py       INFO   Version in upstream registries is '287'.
2023-04-14 18:38:25.309 upstream.py       INFO   Version in spec file is '287.32.g9eba04e'.
2023-04-14 18:38:25.310 utils.py          ERROR  Preparation of the repository for creation of an SRPM failed: Invalid version: '287.32.g9eba04e'

AFAIK our packit.yaml does not do anything magic wrt. the version, and it hasn't changed in months (other than a Fedora version list update in February). Our make dist rule does call git describe to figure out the upstream version, but it already adjusts to RPM rules with tr '-' '.' -- an upstream version like "287.32.g9eba04e" should be totally fine.
The log says it uses packit 0.72.0-1.fc38. Locally I have F37, but packit-0.72.0-1.fc37.noarch is the same upstream version, so it should behave the same. When I run packit srpm in cockpit-machines.git, it succeeds, but curiously produces a different version format:
2023-04-17 07:48:11.012 upstream.py       INFO   Created archive:
2023-04-17 07:48:11.012 upstream.py       INFO   	parsed   path: /var/home/martin/upstream/cockpit-machines/cockpit-machines-287.29.g00f191f.tar.xz
2023-04-17 07:48:11.012 upstream.py       INFO   	absolute path: /var/home/martin/upstream/cockpit-machines/cockpit-machines-287.29.g00f191f.tar.xz
2023-04-17 07:48:11.639 upstream.py       INFO   Version in upstream registries is '287'.
2023-04-17 07:48:11.710 upstream.py       INFO   Version in spec file is '287.29.g00f191f'.
2023-04-17 07:48:11.710 upstream.py       WARNING Version '287.29.g00f191f' in spec file is outdated.
2023-04-17 07:48:11.710 upstream.py       INFO   Picking version '287' from upstream registry.
2023-04-17 07:48:11.900 api.py            INFO   Directory with sources: /var/home/martin/upstream/cockpit-machines
2023-04-17 07:48:11.986 command_handler.py INFO   Nothing to clean.
2023-04-17 07:48:11.987 srpm.py           INFO   SRPM: /var/home/martin/upstream/cockpit-machines/cockpit-machines-287-1.20230417074811108593.main.29.g00f191f.fc37.src.rpm

So I'm a bit confused what to do here. Do we need to change anything on our side? Is that a regression in the service? Or in packit itself?
Thanks!
@skobyda @kkoukiou FYI
What did you expect to happen?
SRPM build should succeed.
Example URL(s)

https://github.com/cockpit-project/cockpit-machines/pull/1021
https://download.copr.fedorainfracloud.org/results/packit/cockpit-project-cockpit-machines-1021/srpm-builds/05785729/builder-live.log.gz
https://github.com/cockpit-project/cockpit-podman/pull/1268

Steps to reproduce
Any upstream PR against cockpit-podman or cockpit-machines
What is the impacted category (job)?
Copr builds
Workaround

[ ] There is an existing workaround that can be used until this issue is fixed.

Participation

[ ] I am willing to submit a pull request for this issue. (Packit team is happy to help!)

@nforro thanks for the details, so the fix here would be to catch the InvalidVersion exc in packit and sort lower in that case, right?

I did toolbox create -r 38 and inside of cockpit-podman.git, sudo dnf install -y packit npm make, packit srpm, and that reproduces the error. You are actively discussing that, but please let me know if we should change anything on our side. I.e. do we even have a say which version we are going to get, or is that packit's decision?

@nforro thanks for the details, so the fix here would be to catch the InvalidVersion exc in packit and sort lower in that case, right?

Yes, exactly.

@martinpitt Packit uses the higher version (upstream release vs the version in specfile), I can't think of any workaround you could do (@nforro, do you have any idea?)...but @nforro is already working on a fix in Packit for this, hopefully we will get this into production tomorrow, we will update you and sorry for the issues!

@lbarcziova No worries, I was just wondering if we do anything odd. Thanks a lot for the quick analysis here!


When can MAVSDK-Go be used for production?

When can MAVSDK-Go be used for production?

Closing because it is a duplicate of https://github.com/mavlink/MAVSDK-Go/issues/5


upgrade hive to 1.1.12

https://github.com/open-cluster-management/backlog/issues/14856

/hold

/unhold

/assign @zhiweiyin318 @elgnay

/approve
/lgtm


update from asyncio_mqtt to aiomqtt

@BenjiU  I tried this with in a mqtt-io homeassistant addon, and it seems to work fine, thanks!


set Markers in the Map

Hi I am new to UI5. I am working on using the google maps.  Currently i embedded the map onto my shell and appended the markers like below.
<gmaps:Map id="map1" class="googleMap" height="600px" width="100%" zoom="4" lat="10" lng="20" markers="{/coords}" zoomControl="true" ready="onMapReady">         <gmaps:Marker class="mark" lat="{latitude}" lng="{longitude}"  click = "onMarkerClick" animation="0" ></gmaps:Marker>     </gmaps:Map>
The issue i am facing is if i want to add an extra marker from user input without removing the already loaded marker.
I created a new marker
var oMarker = new openui5.googlemaps.Marker({ lat: '16', lng: '80'  });
But when i try to insert this marker to the map this.oMap.setMarker(oMarker) it gives me an error
i have seen in your sample app2 . you iterated throught all the markers using getMarkers(); Is there any way to set the marker to the map???

Hello @vinny366 , @jasper07
May I know why you closed this issue, as I am also facing the same problem and the library does not offer a way to add additional markers.
for Ex. I want to have a different icon and click operation for the current position. so essentially its a different marker. but problem I face is that I have to add all markers to a single model and then create a template using that model.
I want to just add a another marker in the end for current location  with out all this hassle.
If you have an answer please let me know.
Thanks
Prateek


Favorites need common name

Favorites only show scientific name, need common name to appear.

Fixed in master.


Filter streets like wandrer.earth to optimize for cycling-friendly roads

Sorry for the glut of input here...
I tried adjusting the custom_filter and useful_tags_way in optiburb.py to achieve the same road map as in wandrer.earth:
        self.custom_filter = (
            f'["highway"]["area"!~"yes"]["highway"!~"footway|service|pedestrian|motorway|motorway_link|trunk|trunk_link|bridleway|steps|corridor|elevator|escalator|motor|proposed|construction|abandoned|platform|raceway|stairs|proposed|demolished|escape|bus_guideway|sidewalk|crossing|bus_stop|traffic_signals|stop|give_way|milestone|platform|speed_camera|rest_area|traffic_island|services|drain|street_lamp|razed|abandoned|corridor"]'
            f'["bicycle"!~"no"]["service"!~"private|parking_aisle"]["access"!~"private"]'
        )

        # not all of this data is used but it looks interesting.

        useful_tags_way = ['bridge', 'tunnel', 'oneway', 'lanes', 'ref', 'name', 'highway', 'maxspeed', 'access', 'area', 'landuse', 'width', 'est_width', 'junction', 'surface',]

The above code changes removed a lot of sidewalk/walkway/service roads that were not included in wandrer.earth (and that I'd prefer not to ride or are not bike-legal), and were reflected in the .svg file, but then the .gpx track only visited some of the nodes, and also did not visit now-isolated sections of the suburb. Something screwy must have happened later on.
Ideally optiburb.py would filter just as wandrer.earth (see https://wandrer.earth/scoring). They outline OSM highway tags for inclusion very clearly.

Good idea.. I'll have a look later and see how it goes.

How's this look?  I've put in the explicit exclusions, but not the inclusions.
        self.custom_filter = (

            '["highway"]'

            '["area"!~"yes"]'

            '["highway"!~"motorway|motorway_link|trunk|trunk_link|bridleway|footway|service|pedestrian|'
            'steps|stairs|escalator|elevator|construction|proposed|demolished|escape|bus_guideway|'
            'sidewalk|crossing|bus_stop|traffic_signals|stop|give_way|milestone|platform|speed_camera|'
            'raceway|rest_area|traffic_island|services|yes|no|drain|street_lamp|razed|corridor|abandoned"]'

            '["access"!~"private|no|customers"]'

            '["bicycle"!~"dismount|use_sidepath|private|no"]'

            '["service"!~"private|parking_aisle"]'

            '["motorroad"!="yes"]'

            '["golf_cart"!~"yes|designated|private"]'

            '[!"waterway"]'

            '[!"razed"]'
        )

Awesome, sorry for disappearing here. I've run the code again on some suburbs and it's so much better! Keep up the great work!


Continue firing what read at channelRead method in CommandHandler so that customerized handler can receive messages

Feature Request

I added IdleStateHandler to pipeLine to moniter "idle state" ,but could not get real 'readIdle' , because IdleStateHandler could not receive msg in channleRead method.

If you would like us to spend some time helping you to diagnose the problem, please spend some time describing it and, ideally, providing a minimal sample that reproduces the problem.


Toolbar Customization Issue When Selecting Text Inside Table Cells

When text is selected within a table cell, the toolbarInlineForSelection: true setting fails to trigger the inline toolbar specific to the text selection. Instead, the toolbar that appears is the one customized for the entire cell.
Jodit Version:  4
Browser: All
OS: All
Is React App: True
Code

    Company
    Contact
    Country

    Alfreds Futterkiste
    Maria Anders
    Germany

    Centro comercial Moctezuma
    Francisco Chang
    Mexico

Steps to Reproduce:

Create an HTML table with multiple rows and columns.
Insert some text into one of the table cells.
Select the text inside the table cell.
Observe the behavior of the toolbar.

Expected Behavior: When text is selected inside a table cell, the toolbar should appear with customization options specifically for the selected text within the cell.
Actual Behavior: Currently, the toolbar appears with customization options for the entire table cell, rather than just the selected text.

Any update?


Add option to disable generating defaultResolvers

I can totally understand the use of the generated defaultResolvers, but it would be nice to have an option to disable them.
For example a project I'm working on doesn't use them, but now when I rename a property of a type, I have to generate the resolver types just because the defaultResolvers are using invalid properties.

but now when I rename a property of a type, I have to generate the resolver types just because the defaultResolvers are using invalid properties.

If I understand correctly, take the following example, imagine I'm not using the default resolvers and imagine I change my model type such that it affects the parent herein... that could cause parent.first and parent.last to become invalid.
export namespace HumanNameResolvers {
  export const defaultResolvers = {
    first: (parent: HumanName) => parent.first,
    last: (parent: HumanName) => parent.last,
  }

  //...

  export interface Type {
    first: (
          parent: HumanName,
          args: {},
          ctx: Context,
          info: GraphQLResolveInfo
        ) => string | Promise<string>)

    last: (
          parent: HumanName,
          args: {},
          ctx: Context,
          info: GraphQLResolveInfo
        ) => string | Promise<string>)

  }
}


What is the meaning of each column is the .tsv file for vqa ?

The first 3 column of the first row is 101096 , 101096 , what color are the letters of the stop sign?
The third column is the question , and which one is image_id , which one is question_id ?

@JY9087 The first column is question_id, the sencond column is image_id. More details are provided in here


Feature Request: Definitions of the PodDisruptionBudget in the Helm Chart

I think the PodDisruptionBudget is necessary to evict external-secrets pods  safely.
So, I would like to define  the PodDisruptionBudget in the Helm Chart and add a parameter which controls if Helm Chart creates PodDisruptionBudget or not.

I will create a PR later.

Merged will be in next release.


sorting by price does not work correctly

What happened?
a larger number may be followed by a smaller one
Please reproduce in steps
link to https://beta.kodadot.xyz/bsx/stats?page=1
clicked on Price
checked the sorting result
Expected Behavior
I expected that they would be sorted from less to more
What browsers are you seeing the problem on?
Chrome
At which address did you encounter bug?
beta.kodadot.xyz
Are you logged in?
Yes
Which wallet you are using?
PolkadotJS
At which chain did you encounter bug?
Basilisk
Screenshots

Relevant log output
No response
Payment link for reward
https://kodadot.xyz/transfer/?target=<My_Kusama_Address_check_https://github.com/kodadot/nft-gallery/blob/main/CONTRIBUTING.md#creating-your-ksm-address>
Code of Conduct

[X] I agree to follow this project's Code of Conduct

👋


Can't set border radius for Tamagui Button Component

Current Behavior
I can't set border radius for the button component. I've tried through the tamagui.config.js, I've tried through the borderRadius and br props, I've also tried through the default react native style prop by setting style={{ borderRadius: 999 }}, none of these methods seem to be working.
Expected Behavior
I'm expecting a pill like button after setting border radius values to 999, but it just stays adamant on the default value.
Tamagui Version
^1.39.7

Reproduction
To reproduce, simply call the button component and try setting the borderRadius, br and style props to reflect a large border radius.

System Info
System:
    OS: macOS 11.7.2
    CPU: (8) x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
    Memory: 86.45 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.14.2 - /usr/local/bin/node
    Yarn: 1.22.18 - /usr/local/bin/yarn
    npm: 9.5.1 - /usr/local/bin/npm
  Browsers:
    Chrome: 114.0.5735.198
    Safari: 16.2
  npmPackages:
    @babel/core: ^7.20.0 => 7.22.9
    @tamagui/animations-react-native: ^1.39.7 => 1.43.10
    @tamagui/babel-plugin: ^1.39.7 => 1.43.10
    @tamagui/config: ^1.39.7 => 1.43.10
    @tamagui/font-inter: ^1.39.7 => 1.43.10
    @tamagui/lucide-icons: ^1.43.8 => 1.43.10
    @tamagui/react-native-media-driver: ^1.39.7 => 1.43.10
    @tamagui/shorthands: ^1.39.7 => 1.43.10
    @tamagui/theme-base: ^1.39.7 => 1.43.10
    @tamagui/themes: ^1.39.7 => 1.43.10
    @types/react: ~18.2.14 => 18.2.15
    @types/react-native: ^0.72.2 => 0.72.2
    babel-plugin-module-resolver: ^5.0.0 => 5.0.0
    babel-plugin-transform-inline-environment-variables: ^0.4.4 => 0.4.4
    expo: ^49.0.2 => 49.0.3
    expo-font: ^11.4.0 => 11.4.0
    expo-status-bar: 1.6.0 => 1.6.0
    react: 18.2.0 => 18.2.0
    react-dom: ^18.2.0 => 18.2.0
    react-native: 0.72.3 => 0.72.3
    react-native-reanimated: ~3.3.0 => 3.3.0
    react-native-svg: ^13.10.0 => 13.10.0
    react-native-web: ^0.19.6 => 0.19.6
    tamagui: ^1.39.7 => 1.43.10
    typescript: ^5.1.3 => 5.1.6

prop order is important, so borderRadius must come after size for example

I used the borderRadius and br props after the size prop and it's still not applying the style.


Bug: Pocket since high-water-mark gets set even when indexing fails

Describe the bug
I'm setting up Pocket importing for the first time, meaning I'm importing a lot of old links, some of which are on now-defunct websites. When one of them fails, the entire import fails, but the since value in pocket_api.db is still set, meaning when I try to re-import my Pocket feed, it only retrieves new items, leaving me with no URLs archived.
Steps to reproduce

Set up Pocket config, per #528
Have a URL in pocket on a domain which refuses connections, or does not exist
Import from Pocket:$ archivebox add --depth=1 pocket://myUserName
[+] [2021-04-28 14:00:05] Adding 1 links to index (crawl depth=1)...
    > Saved verbatim input to sources/1619618411-import.txt
    > Parsed 169 URLs from input (Pocket API)

[*] Starting crawl of 169 sites 1 hop out from starting point
    > Downloading http://my-working-url.com/ contents
    > Saved verbatim input to sources/1619618411-crawl-my-working-url.com.txt
    > Parsed 12 URLs from input (Generic TXT)
    > Downloading http://my-defunct-url.com/ contents
[!] Failed to download http://my-defunct-url.com/

     HTTPConnectionPool(host='my-defunct-url.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb4dbf9b8>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Remove broken URL from Pocket
Try importing again:$ archivebox add --depth=1 pocket://myUserName
[+] [2021-04-28 14:39:35] Adding 1 links to index (crawl depth=1)...
    > Saved verbatim input to sources/1619620775-import.txt
                                                                                0.1% (0/240sec)
[X] No links found using Pocket API parser
    Hint: Try a different parser or double check the input?

    > Parsed 0 URLs from input (Pocket API)
    > Found 0 new URLs not already in index

[*] [2021-04-28 14:39:35] Writing 0 links to main index...
    √ ./index.sqlite3

ArchiveBox version

ArchiveBox v0.6.2
Cpython Linux Linux-5.4.79-v7l+-armv7l-with-glibc2.28 armv7l
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.6.2          valid     /usr/local/bin/archivebox
 √  PYTHON_BINARY         v3.9.4          valid     /usr/local/bin/python3.9
 √  DJANGO_BINARY         v3.1.8          valid     /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py
 √  CURL_BINARY           v7.64.0         valid     /usr/bin/curl
 √  WGET_BINARY           v1.20.1         valid     /usr/bin/wget
 √  NODE_BINARY           v15.14.0        valid     /usr/bin/node
 √  SINGLEFILE_BINARY     v0.3.16         valid     /node/node_modules/single-file/cli/single-file
 √  READABILITY_BINARY    v0.0.2          valid     /node/node_modules/readability-extractor/readability-extractor
 √  MERCURY_BINARY        v1.0.0          valid<EMAIL_ADDRESS> √  GIT_BINARY            v2.20.1         valid     /usr/bin/git
 √  YOUTUBEDL_BINARY      v2021.04.07     valid     /usr/local/bin/youtube-dl
 √  CHROME_BINARY         v89.0.4389.114  valid     /usr/bin/chromium
 √  RIPGREP_BINARY        v0.10.0         valid     /usr/bin/rg

[i] Source-code locations:
 √  PACKAGE_DIR           22 files        valid     /app/archivebox
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates
 -  CUSTOM_TEMPLATES_DIR  -               disabled

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled
 -  COOKIES_FILE          -               disabled

[i] Data locations:
 √  OUTPUT_DIR            5 files         valid     /data
 √  SOURCES_DIR           28 files        valid     ./sources
 √  LOGS_DIR              1 files         valid     ./logs
 √  ARCHIVE_DIR           0 files         valid     ./archive
 √  CONFIG_FILE           204.0 Bytes     valid     ./ArchiveBox.conf
 √  SQL_INDEX             204.0 KB        valid     ./index.sqlite3

The URLs should still be added to your database, so if you need to download those urls, you can run archivebox update to go through your db and do that process.
@pirate My understanding is this isn't specific to the Pocket API implementation though, is it?

The URLs should still be added to your database, so if you need to download those urls, you can run archivebox update to go through your db and do that process.

I thought so as well, but when the add fails, no snapshots get added:
$ archivebox update
[i] [2021-04-28 17:12:12] ArchiveBox v0.6.2: archivebox update
    > /data

[!] No Snapshots matched your filters: [] (exact)
$ archivebox status
[*] Scanning archive main index...
    /data/*
    Index size: 236.0 KB across 3 files

    > SQL Main Index: 0 links        (found in index.sqlite3)
    > JSON Link Details: 0 links     (found in archive/*/index.json)

[*] Scanning archive data directories...
    /data/archive/*
    Size: 0.0 Bytes across 0 files in 0 directories

    > indexed: 0                     (indexed links without checking archive status or data directory validity)
      > archived: 0                  (indexed links that are archived with a valid data directory)
      > unarchived: 0                (indexed links that are unarchived with no data directory or an empty data directory)

    > present: 0                     (dirs that actually exist in the archive/ folder)
      > valid: 0                     (dirs with a valid index matched to the main index and archived content)
      > invalid: 0                   (dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized)
        > duplicate: 0               (dirs that conflict with other directories that have the same link URL or timestamp)
        > orphaned: 0                (dirs that contain a valid index but aren't listed in the main index)
        > corrupted: 0               (dirs that don't contain a valid index and aren't listed in the main index)
        > unrecognized: 0            (dirs that don't contain recognizable archive data and aren't listed in the main index)

[*] Scanning recent archive changes and user logins:
    /data/logs/*
    UI users 1: archivebox
    Last UI login: archivebox @ 2021-04-28 12:50

    ...

Yeah I think it's missing a try: except: block around the crawl step, because it's a preprocessing step done before the actual archiving it doesn't have our usual "if snapshot fails, continue anyway" logic around it.
For now you may have to revert to adding URLs one at a time when using --depth=1 mode until this is fixed in the next version. To fix it for now / re-update urls that are existing you can try using the archivebox add --update-all ... or archivebox add --overwrite ... flags depending on what your desired behavior when encountering a previously-added URL is.

To fix the immediate issue of not being able to add these pages / re-update urls that are already present, you can try using the archivebox add --update-all ... or archivebox add --overwrite ... flags

I tried this as well, but it doesn't work either, as the URLs aren't being indexed at all.

Raw output from --update-all and --overwrite
$ archivebox add --depth=1 --update-all pocket://myUserName
[i] [2021-04-28 17:33:44] ArchiveBox v0.6.2: archivebox add --depth=1 --update-all pocket://myUserName
    > /data

[+] [2021-04-28 17:33:48] Adding 1 links to index (crawl depth=1)...
    > Saved verbatim input to sources/1619631228-import.txt
    > Parsed 0 URLs from input (Failed to parse)
    > Found 0 new URLs not already in index

[*] [2021-04-28 17:33:48] Writing 0 links to main index...
    √ ./index.sqlite3
$ archivebox add --depth=1 --overwrite pocket://myUserName
[i] [2021-04-28 17:34:04] ArchiveBox v0.6.2: archivebox add --depth=1 --overwrite pocket://myUserName
    > /data

[+] [2021-04-28 17:34:08] Adding 1 links to index (crawl depth=1)...
    > Saved verbatim input to sources/1619631248-import.txt
    > Parsed 0 URLs from input (Failed to parse)
    > Found 0 new URLs not already in index

[*] [2021-04-28 17:34:08] Writing 0 links to main index...
    √ ./index.sqlite3

This is why I think it's Pocket-specific: because the Pocket API consumer saves the since value, it stops reporting the existence of the old URLs at all, even though it never successfully indexed them. Files have been written in data/sources, but data/archive is still empty.
More to the point, if I delete the sources/pocket_api.db file and run add again, the behaviour changes: it starts attempting to index all the 1-hop-out URLs again, rather than just saying No links found.

Ah yeah, you're right, this is pocket api specific then.


[REQUEST] can we load a deepspeed ckpt without deepspeed?

Hello, is there a way to load the deepspeed checkpoint without deepspeed?  I'd like to somehow just load it with standard Pytorch.  Or do we somehow need to convert it first?  Deepspeed has some overhead that I'd like to get around for doing zero-shot evals downstream.

@exnx, please see https://deepspeed.readthedocs.io/en/latest/model-checkpointing.html#zero-checkpoint-fp32-weights-recovery

Thanks for sharing this!
More specifically, I was wondering if there's a way to load the deepspeed checkpoint without need to initializing deepspeed?  ie without deepspeed.init_distributed().
Sometimes I get errors using deepspeed, like mpi issues, and for just loading ckpts it would be nice not need to do all that.

Also, what if you use a zero stage 1, or not at all, ie zero stage 0?

from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint

# remove all the deepspeed magic data
state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir)

model = model.cpu() # move to cpu
model.load_state_dict(state_dict)

MODEL_PATH = "your_model_without_DS.pth"

# Save the model's state_dict
torch.save(model.state_dict(), MODEL_PATH)

# Load the model's state_dict
model.load_state_dict(torch.load(MODEL_PATH))

My understanding is that it get_fp32_state_dict_from_zero_checkpoint only works on zero 2 or 3 optimizers?  (not 0 or 1)?

zero 0 or zero 1, the model parameters are not partitioned. It means the full FP32 state dictionary is readily accessible without additional reconstruction steps.

@ycool, thanks for providing a workaround.
Closing this issue.


[Event Request] CodeUnit 10801 - RIB Key - Check() - OnBeforeCheck |  CheckRemaining

Hello, would it be possible to have two events "OnBeforeCheck" and "CheckRemaining" in Codeunit 10801 "RIB Key" , procedure Check(), Like this :

Cordially.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Don't export GraphQLClientRequestHeaders in src/index.ts

Description
I can't use import { GraphQLClientRequestHeaders } from "graphql-request", because I get error: TS2614: Module '"graphql-request"' has no exported member 'GraphQLClientRequestHeaders'
type GraphQLClientRequestHeaders doesn't  export in src/index.ts
Reproduction Steps/Repo Link
I configure project by Pure ESM package:

in package.json

{
  "type": "module",
  "engines": [
    "node": ">=16"
  ],
  "dependecies": {
    "graphql-request": "^6.1.0"
  }
}

in tsconfig.json

{
  "compilerOptions": {
    "module": "node16",
    "moduleResolution": "node16"
  }
}

This has been fixed under #616


Washer fails to connect

I've gone through the example and added the washer to my phone, but it fails to connect. Here are the logs
I replaced the washer-start example with the washer-done example, just in case I typed something wrong, but it still throws this error:
{
  "insertId": "1q5xl8qg1z19oc2",
  "jsonPayload": {
    "locale": "en-US",
    "executionLog": {
      "executionResults": [
        {
          "latencyMsec": "83",
          "requestId": "11498087478043784573",
          "executionType": "PARTNER_CLOUD",
          "actionResults": [
            {
              "action": {
                "actionType": "STATE_QUERY"
              },
              "status": {
                "externalDebugString": "Error querying agent backend. State: URL_UNREACHABLE, reason: 2",
                "statusType": "EXECUTION_BACKEND_FAILURE"
              },
              "device": {
                "deviceType": "WASHER"
              }
            }
          ]
        }
      ]
    }
  },
  "resource": {
    "type": "assistant_action_project",
    "labels": {
      "project_id": "smart-home-example-39c50"
    }
  },
  "timestamp": "2021-02-28T22:40:56.720383524Z",
  "severity": "ERROR",
  "logName": "projects/smart-home-example-39c50/logs/assistant_smarthome%2Fassistant_smarthome_logs",
  "receiveTimestamp": "2021-02-28T22:40:56.720383524Z"
}

Hi @daniel-frenkel,
We weren't able to reproduce the issue, but the logs (EXECUTION_BACKEND_FAILURE & URL_UNREACHABLE) show that it's likely caused by an incorrect fulfillment URL setting on the Actions Console. Please check your settings there.


Add recreated pilot patients to test

This is a set of synthetic test patients recreated from pilot site.

Closing this PR, since we decided not to merge all of these patients into our main branch. We will instead choose a subset of these patients to serve as examples of realistic test patients that we include in our main branch. We will also create additional patient scenarios, since the one's in this set are not representative of all logic paths in our CDS.


Document the registry

We probably don't need to implement this for a while, but I want to have it in the docs so that the plan is clear.

r? @Gankra

Oh sorry, I saw this on friday but was too fried to look at it. This is... a surprising development but it's... possible? Having it as a random static file in this repo is deeply strange though. You would think you would want to host in on an actual website/service...?


Added notifications to slack for travis-ci failed builds

This PR adds notifications to slack in case of travis-ci failed builds on the master or release branch.

Codecov Report

Merging #1923 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1923   +/-   ##
=======================================
  Coverage   34.20%   34.20%
=======================================
  Files         146      146
  Lines        9115     9115
  Branches      128      128
=======================================
  Hits         3118     3118
  Misses       5580     5580
  Partials      417      417

Flag
Coverage Δ

#moduleA
35.88% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 56ce87a...0db2e14. Read the comment docs.


Misbehavior for sidebar when collapsed

Describe the bug
The sidebar, when collapsed is acting weird:

It shows the badge
It makes the whole children as its tooltip (even the icon)

To Reproduce
Steps to reproduce the behavior:

Create a CRA project with flowbite-react
Create the sidebar menu as described in the documentation (with icons and labels)
Change its attribute to be collapsed <Sidebar collapsed={true} ...>...</Sidebar>
Hover the mouse hover to see the issue

Expected behavior

When collapse it should NOT display the Badge, ONLY the icon
The content of Tooptip must be or the original children or/and also allow it to be overwrite by Sidebar.Item property

Screenshots

Additional context
n/a

I found some tests are reporting false positive here. userEvent.focus(something) is not reliable. We should add Tooltip integration tests in cypress to actually cover user interactions like hover


Problem running the script on RP3

I don't have much experience with Linux, so I'd really love to have this script working on my device.
I ran this command:
sudo ./picloud.sh -s "<IP_ADDRESS> admin password"
And got this:

Running setup...
mkdir: cannot create directory ‘/mnt/MyCloud’: File exists
mount error(115): Operation now in progress
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
rmdir: failed to remove ‘/mnt/MyCloud’: Device or resource busy
Mounting MyCloud failed during setup

I doubt I am doing something wrong, as this is pretty straight forward. What should I check/what data can I give so a solution could be found?

Investigated a bit:
/mnt/MyCloud is my device mounted successfully.
When I do:
cd /mnt/MyCloud
mkdir test

It works, and I can see that dir from a different computer.
However, when trying to execute:
sudo ./picloud.sh
I get this output:

Creating backup. This may take a while.
Unable to find suitable address.
mount error(115): Operation now in progress
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Mounting MyCloud failed

I find it odd, as I have checked, and MyCloud is mounted just fine

I think this is the solution:
The source code has:
dd if=/dev/mmcblk0p7 bs=1M | gzip > /mnt/MyCloud/images/"$(date +%d-%b-%y_%T)".gz
The code should run: lsblk to get the hard drive, because in my case I have:

mmcblk0
|--mmcblk0p1
|--mmcblk0p2

So when I ran this command:
sudo dd if=/dev/mmcblk0 bs=1M | gzip > /mnt/MyCloud/"$(date +%FT%T)".gz
(I changed the drive, and the date format for a sortable format)
It works.

Glad you got it working 👍
It looks like different SD cards may have different partition aliases (eg mmcblk0p1) where the operating system resides. Perhaps the script needs some integrated logic to figure out which partition contains the OS. If you have any ideas or want to submit a pull request you're more than welcome. I also like the improvements you made to the date formatting.

Is there a reason to backup just the OS partition? Why not backup the entire thing?

I added 5 PRs. merge whichever you want.

Thanks! I tested backing up the whole file system instead of a particular partition and it had no issues. So I agree that it may as well back up the entire thing.
I'll take a look at the PRs shortly.


Installation failed (poetry 0.12.17, Linux Mint 19.2)

[x] I am on the latest Poetry version.

[x] I have searched the issues of this repo and believe that this is not a duplicate.

[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Linux Mint 19.2

Poetry version: 0.12.17

Issue
I failed to install poetry with Python 3.8 and found a relevant issue. I uninstalled it. I then tried to install poetry with Python 3.6, but I get the same kind of error: ModuleNotFoundError: No module named 'cleo'
I was conviced during last PyConFr to use poetry which seems great. So I want to install it.
Here is what happened:
 vincent  vincent-port  ~  $  python3.8 get-poetry.py --uninstall
# We are sorry to see you go!

This will uninstall Poetry.

It will remove the `poetry` command from Poetry's bin directory, located at:

$HOME/.poetry/bin

This will also remove Poetry from your system's PATH.

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

$HOME/.poetry/bin

This path will then be added to your `PATH` environment variable by
modifying the profile files located at:

$HOME/.profile
$HOME/.bash_profile

You can uninstall at any time with `poetry self:uninstall`,
or by executing this script with the --uninstall option,
and these changes will be reverted.

Installing version: 0.12.17
  - Downloading poetry-0.12.17-linux.tar.gz (8.37MB)

Poetry (0.12.17) is installed now. Great!

To get started you need Poetry's bin directory ($HOME/.poetry/bin) in your `PATH`
environment variable. Next time you log in this will be done
automatically.

To configure your current shell run `source $HOME/.poetry/env`

 vincent  vincent-port  ~  $  source $HOME/.poetry/env
 vincent  vincent-port  ~  $  poetry completions bash -vvv
Traceback (most recent call last):
  File "/home/vincent/.poetry/bin/poetry", line 12, in <module>
    from poetry.console import main
  File "/home/vincent/.poetry/lib/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/home/vincent/.poetry/lib/poetry/console/application.py", line 6, in <module>
    from cleo import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'

Thanks for your attention and for this project.

The 0.12.17 release does not support Python 3.8 and you should, if possible, use the latest 1.0.0 beta release.

I'll try to use the beta release.
But in the example I provided:

Installed poetry with Python 3.8 (not shown in the issue)
Uninstalled poetry with Python 3.8
Installed poetry with Python 3.6
Failed to use poetry

So there is a problem with either the uninstallation (with Python 3.8), or the reinstallation with Python 3.6, or with my computer. Any advice?

I ran into a similar issue with poetry 0.12.17 installation on macOS Mojave with Python 3.8.0. I resolved it by following these instructions:

Remove the $HOME/.poetry directory
Re-install poetry with POETRY_PREVIEW enabled $ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | POETRY_PREVIEW=1 python

Thanks @sdispater as well as this comment on another issue https://github.com/sdispater/poetry/issues/553#issuecomment-548446831

@xyzst @sdispater Thanks, it works with the beta version.

The 0.12.17 release does not support Python 3.8 and you should, if possible, use the latest 1.0.0 beta release.

Then that needs to be mentioned in the online documentation: https://poetry.eustace.io/docs/#system-requirements


write_h5ms error

I referred to the online tutorial (https://stereopy.readthedocs.io/en/latest/Tutorials(Multi-sample)/Batch_Effect.html) to integrate spatial transcriptomics data from multiple sections of one sample (.h5ad), and everything was fine, but I encountered an error when saving as .h5ms. At the same time, I have confirmed that there are no files with the same name in the folder. The version of Stereopy is 1.3.0.
import sys
import os
import seaborn as sns
import stereo as st
from natsort import natsorted
from matplotlib import pyplot as plt
from stereo.core.ms_data import MSData
from stereo.core.ms_pipeline import slice_generator
%matplotlib notebook
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
data_dir = '/home/bin50/mouse2/data'
data_list=[]
for fn in os.listdir(data_dir):
data_list.append(os.path.join(data_dir, fn))
ensure data order by naming them regularly
data_list = natsorted(data_list)
construct MSData object
ms_data = MSData(_relationship='other', _var_type='intersect')
when come to loaded data object
ms_data = MSData(_data_list=[data1, data2], _names=['s1', 's2'], _relationship='other', _var_type='intersect')
add all samples into MSData
for sample in data_list:
ms_data += st.io.read_h5ad(file_path=sample,
bin_type='bins',
bin_size=50
)
st.io.write_h5ms(ms_data,output='test.h5ms')
File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()
File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()
File h5py/h5d.pyx:135, in h5py.h5d.create()
ValueError: Unable to create dataset (name already exists)

That did not mean there is a same name of file, it may have same name of items in the data read from your file, but the error information you provided is too few to find the reason, could you post all the error messages here? If possible, providing your files with me is better.

Thanks, here is all the error information, and my input data is named as follows: T01.h5ad-T15.h5ad.

Thanks, here is all the error information, and my input data is named as follows: T01.h5ad-T15.h5ad.
ValueError                                Traceback (most recent call last)
Cell In[22], line 1
----> 1 st.io.write_h5ms(ms_data,output='test.h5ms')
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/stereo/io/writer.py:304, in write_h5ms(ms_data, output)
302 for idx, data in enumerate(ms_data.data_list):
303     f['sample'].create_group(f'sample{idx}')
--> 304     write_one_h5ad(f['sample'][f'sample{idx}'], data, use_raw=True, use_result=True)
305     # if isinstance(data, AnnBasedStereoExpData):
306     #     write_one_anndata(f['sample'][f'sample{idx}'], data)
307     # else:
(...)
310 #     f.create_group('sample_merged')
311 #     _write_one_h5ad(f['sample_merged'], ms_data._merged_data)
312 if len(ms_data.scopes_data) > 0:
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/stereo/io/writer.py:109, in _write_one_h5ad(f, data, use_raw, use_result, key_record)
107         sn_data = pd.DataFrame(sn_list, columns=['batch', 'sn'])
108         h5ad.write(sn_data, f, 'sn', save_as_matrix=True)
--> 109 h5ad.write(data.genes, f, 'genes')
110 h5ad.write(data.cells, f, 'cells')
111 if data.position_z is None:
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/functools.py:875, in singledispatch..wrapper(*args, **kw)
871 if not args:
872     raise TypeError(f'{funcname} requires at least '
873                     '1 positional argument')
--> 875 return dispatch(args[0].class)(*args, **kw)
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/stereo/io/h5ad.py:58, in _(v, f, k)
56 @write.register(Gene)
57 def _(v, f, k):
---> 58     write_genes(f, k, v)
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/stereo/io/h5ad.py:117, in write_genes(f, k, v, dataset_kwargs)
115 g.attrs['encoding-type'] = 'gene'
116 g.attrs['version'] = 'v2'
--> 117 write_dataframe(g, 'var', v.to_df(), dataset_kwargs)
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/stereo/io/h5ad.py:176, in write_dataframe(f, key, df, dataset_kwargs, save_as_matrix)
174 else:
175     for col_name, (_, series) in zip(col_names, df.items()):
--> 176         write_series(group, col_name, series, dataset_kwargs=dataset_kwargs)
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/stereo/io/h5ad.py:182, in write_series(group, key, series, dataset_kwargs)
179 def write_series(group, key, series, dataset_kwargs=MappingProxyType({})):
180     # group here is an h5py type, otherwise categoricals won’t write
181     if series.dtype == object:  # Assuming it’s string
--> 182         group.create_dataset(
183             key,
184             data=series.values,
185             dtype=h5py.special_dtype(vlen=str),
186             **dataset_kwargs,
187         )
188     elif is_categorical_dtype(series):
189         # This should work for categorical Index and Series
190         categorical: pd.Categorical = series.values
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/h5py/_hl/group.py:183, in Group.create_dataset(self, name, shape, dtype, data, **kwds)
180         parent_path, name = name.rsplit(b'/', 1)
181         group = self.require_group(parent_path)
--> 183 dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
184 dset = dataset.Dataset(dsid)
185 return dset
File /data/pxd/miniconda3/envs/py3.8/lib/python3.8/site-packages/h5py/_hl/dataset.py:165, in make_new_dset(parent, shape, dtype, data, name, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times, external, track_order, dcpl, dapl, efile_prefix, virtual_prefix, allow_unknown_filter, rdcc_nslots, rdcc_nbytes, rdcc_w0)
162 else:
163     sid = h5s.create_simple(shape, maxshape)
--> 165 dset_id = h5d.create(parent.id, name, tid, sid, dcpl=dcpl, dapl=dapl)
167 if (data is not None) and (not isinstance(data, Empty)):
168     dset_id.write(h5s.ALL, h5s.ALL, data)
File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()
File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()
File h5py/h5d.pyx:135, in h5py.h5d.create()
ValueError: Unable to create dataset (name already exists)

The error may occur on one of your files, running st.io.write_h5ad for each file to find the file where the error occur, then print the data.genes of the error file and post here.

Thank you for your advice. I have iterated through all the input .h5ad files as follows: data = st.io.read_h5ad(input_file, flavor='scanpy', bin_type='bins', bin_size=50, spatial_key='spatial'), and running st.io.write_h5ad to write the files. Unfortunately, each file still results in the same error message mentioned earlier.
The data.genes for one of the files is as follows (eg.)
By the way, using the command adata = st.io.stereo_to_anndata(data, output='./scanpy_anndata.h5ad') successfully saves the data. Is there a similar method to save the integrated data as anndata?
Additionally, there is no image data in the .uns of the data, does this mean that it is not possible to visualize the spatial gene expression through Scanpy (sc.pl.spatial)?

Thank you for your advice.
I have iterated through all the input .h5ad files as follows: data = st.io.read_h5ad(input_file, flavor='scanpy', bin_type='bins', bin_size=50, spatial_key='spatial'), and running st.io.write_h5ad to write the files. Unfortunately, each file still occur the same error.

The data.genes for one of the files is as follows (eg.)

By the way, using the command adata = st.io.stereo_to_anndata(data, output='./scanpy_anndata.h5ad') successfully saves the data. Is there a similar method to save the integrated data as anndata?
Additionally, there is no image data in the .uns of the data, does this mean that it is not possible to visualize the spatial gene expression through Scanpy (sc.pl.spatial)?

I have known the cause of the error, the name of index of the data.genes is dumplicate with one of its columns, before creating MSData, running data.adata.var.index = None for each data object can solve your problem.

Thank you for your reply, it has resolved my issue. By the way, "data.adata.var.index.name = None" works for me.


remove pulumi deplendency, use third_party instead

close issue: https://github.com/KusionStack/kusion/issues/58

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

6 of 21 (28.57%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.2%) to 76.768%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/kusionctl/cmd/init/options.go
1
3
33.33%

pkg/scaffold/loader.go
1
3
33.33%

pkg/scaffold/external_templates.go
4
15
26.67%

Totals

Change from base Build<PHONE_NUMBER>:
-0.2%

Covered Lines:
3734

Relevant Lines:
4864

💛  - Coveralls


fix(vite-node): allow returning id not wrapped in promise

This is a small change to the type of resolveId. I believe it is only ever awaited, which means it should be safe to synchronously return an id.
I didn't think it was worth adding @vitest/utils to benefit from the Awaitable type but happy if you think otherwise.
(Might we also want to allow void?)

I am not sure why this is needed. The only reason why resolveId exists as an option is to support await import(customVariable). ViteNode will throw an error in future versions if it didn't return anything: #3307
If you have a specific use case where you use it, maybe it's better to expose another API.

I raised the PR as I was turning on type checking at this place in our codebase:
https://github.com/nuxt/nuxt/blob/a793e7ad48c56a034fd547081ed32b47a810c23e/packages/vite/src/runtime/vite-node.mjs#L41
We currently use this hook to spy on ids being resolved so we can give better errors in the event that there is an error resolving the id.

We currently use this hook to spy on ids being resolved so we can give better errors in the event that there is an error resolving the id.

I know you have that - this is why I am pointing out that it might not be the best solution, because it will fail in the next version.

I know you have that - this is why I am pointing out that it might not be the best solution, because it will fail in the next version.

Even now it doesn't work like you expect. resolveId is not called most of the time since ids are already resolved.

Well, this PR was mostly aimed at either fixing the type to match how it is currently used - or surfacing whether our usage is wrong - so I'm very happy to be told that we shouldn't use it as we do (if that's what I'm hearing from you).

Well, this PR was mostly aimed at either fixing the type to match how it is currently used - or surfacing whether our usage is wrong - so I'm very happy to be told that we shouldn't use it as we do (if that's what I'm hearing from you).

As I said, the fix is fine if we are talking about syntax. It is allowed to return a value straight away, but it's usage like this is questionable considering Vite's resolution is async (which is what this function is used for). This is why I was concerned at first.
I think we can merge PR, but you should be aware that your usage of this method is incorrect.


Mistake in documentation?

I'm using texar-pytorch v0.1.0.
In the example here:
@executor.on_event(Event.Epoch, 'end')
def log_at_end_of_epoch(executor):
    logging.info("Epoch %d done", executor.status.epoch)

executor.status.epoch should be executor.status['epoch'] since executor.status is a dict and epoch is a key rather than an attribute.
Here's the error I met in experiment.

Yes, you're right. Sorry about this mistake. Would you like to create a pull request for this?


[nix] replace streamLayeredImage to buildImage

Proot have too many bugs to be fixed, it is better to use qemu to build the docker image.

Since we are using qemu now. How about providing multiple arch(aarch64,x86,riscv) in our release?


[RFC] Transition frontier listener

Oh, one more thing. We sort of discussed this, but what do you think about formalizing this using the same diff abstraction we need for the persistence layer? For example, imagine we have a diff type 'a diff which has the variants Add of 'a | Remove of 'a | Set_root of 'a. The interface would then be a function which takes a diff with the type Breadcrumb.t diff. In the case of the persistent listener, it would map that diff into a Minimal_breadcrumb.t diff or something. It's not really necessary for the other listeners, but I think it makes a nice reusable abstraction. What do you think though?

Trying my best to incorporate feedback. The lifetime/ownership of these objects is the part that still the most unclear to me, but here's another design that might better match what you are imagining.
This goes with the assumption that there won't be very many of these views and that they'd be completely static, but lmk if it's way off.
Here are the signatures we could add to Protocols.Coda_transition_frontier:
module type Transition_frontier_iview = sig
  type t
  val create: unit -> t
  val handle_diff: t -> node frontier_diff -> unit
end

type 'a frontier_diff =
  | Add of 'a
  | Remove of 'a
  | Set_root of 'a
  | Destroy

type iviews =
  { snark_pool_refcount: Snark_pool_refcount.t
  ; transaction_pool_refcount: Transaction_pool_refcount.t
  ; frontier_persister: Frontier_persister.t}

This would be combined with the addition of functions like this to Transition_frontier.ml:
module Transition_frontier = struct
...
  let notify_iviews_of_diff iviews diff =
    Snark_pool_refcount.handle_diff iviews.snark_pool_refcount diff;
    Transaction_pool_refcount.handle_diff iviews.transaction_pool_refcount diff;
    Frontier_persister.handle_diff iviews.frontier_persister diff;

  (* Called inside Transition_frontier.create *)
  let create_iviews () =
    { snark_pool_refcount = Snark_pool_refcount.create (),
    ; transaction_pool_refcount = Transaction_pool_refcount.create (),
    ; frontier_persister = Frontier_persister.create ()}
end

Here's a rough example of what an implementation of Transition_frontier_iview could look like for the Snark_pool reference counter.
  module Snark_pool_refcount = struct
    module type S = Transition_frontier_iview
    type t = {ref_table: Work.t int Hashtbl.t
              ; removed: Work.t Hash_set.t }

    let create () =
      { ref_table = Hashtbl.create ()
      ; removed = Hash_set.create ()}

    let handle_diff t diff = match diff with
      | Set_root node (* ??? *)
      | Add node ->
        let work = somehow_get_work node in
        List.iter(work, ~f:(fun w -> Hashtbl.update t w ~f:(function
          | Some n -> n + 1
          | None -> 1)))
      | Remove node ->
        let work = somehow_get_work node in
        List.iter(work, ~f:(fun w -> Hashtbl.change t w ~f:(function
          | None | Some 1 -> Hash_set.add t.removed w; None
          | Some n -> Some (n - 1))))
      | Destroy -> ()
  end

Your changes look good to me, here are my three small things to add:

I dislike the iview name (what is i? acronyms!) extension is probably what I'd vote for.
If you put [@@deriving fields] on the record with the extensions, we can invoke the listeners with a fold over the fields and get compile errors if we forget to update when we add a new extension (see lib/daemon_rpcs/daemon_rpcs.ml for an example of this)
Another nit, prefer nested modules with type t to named types (see docs/style_guide.md)

The extensions should receive as an argument the transition frontier that was subject to the event?
When the extension methods are called, is the corresponding action already fully completed?
Must we block the relevant transition frontier methods on calling into the extensions, or can they be scheduled to occur later?

Must we block the relevant transition frontier methods on calling into the extensions, or can they be scheduled to occur later?

If they must block, we should lift the extension methods into Deferred so that we can do asynchronous work without having a don't_wait_for that potentially invalidates any code that depending on blocking-until-extensions-are-done.

The lifetime/ownership of these objects is the part that still the most unclear to me, but here's another design that might better match what you are imagining.

The transition frontier controller "owns" both the transition frontier and the extensions to it. The lifetime of these two values should be tied to each other, which is to say that, when the transition frontier is created, the extensions, and when the transition frontier is destroyed, the extensions should be as well (destruction is implicit though).

Looks like this is converging in the correct direction. I would second what @bkase said about using [@@deriving fields] as a way of constraining the application of diffs to extensions.
To respond to @cmr:

The extensions should receive as an argument the transition frontier that was subject to the event?

Yes

When the extension methods are called, is the corresponding action already fully completed?

Yes

Must we block the relevant transition frontier methods on calling into the extensions, or can they be scheduled to occur later?

Diffs are generated by the relevant transition frontier methods, so yes. They could be split later if need be, but I don't see any reason to require that now. The processor thread can perform the transition frontier actions and defer another thread to handle passing the diff to the extensions if we need to split it up, but I think that the extensions we need will be fast enough to update in thread that we can just do that.

@cmr @nholland94 Thanks for feedback, one question about this:

The extensions should receive as an argument the transition frontier that was subject to the event?

Yes

The original idea was to just pass the diff (ie, the breadcrumb that was removed/added). Is that not enough information and we think that most extensions will need to able to also access the transition frontier itself, or is there another reason?

We added the extensions mechanism and never merged this rfc... and it's going to be different again when my changes are merged


OOM when running test_net.py and demo.py

How much memory does this use while inferencing? It is trying to allocate 12GB and there is no option for changing batch sizes in these scripts either (test_net.py and demo.py). I changed batch size in the config but still same result.
I ran this on an RTX3080:
python3 test_net.py --config-file configs/sparse_inst_r50_giam.yaml --num-gpus 1 MODEL.WEIGHTS output/sparse_inst_r50vd_dcn_giam_aug/model_0006499.pth INPUT.MIN_SIZE_TEST 512

Error:
[08/06 12:41:22 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(512, 512), max_size=853, sample_style='choice')]
[08/06 12:41:22 d2.data.common]: Serializing 30 elements to byte tensors and concatenating them all ...
[08/06 12:41:22 d2.data.common]: Serialized dataset takes 2.45 MiB
/home/siddharth/.local/lib/python3.8/site-packages/detectron2/structures/image_list.py:88: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  max_size = (max_size + (stride - 1)) // stride * stride
/home/siddharth/.local/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "test_net.py", line 196, in <module>
    test_sparseinst_speed(cfg, fp16=args.fp16)
  File "test_net.py", line 157, in test_sparseinst_speed
    output = model(images, resized_size, ori_size)
  File "/home/siddharth/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "test_net.py", line 82, in forward
    result = self.inference_single(
  File "test_net.py", line 116, in inference_single
    pred_masks = F.interpolate(pred_masks, size=ori_shape, mode='bilinear',
  File "/home/siddharth/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 3731, in interpolate
    return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)
RuntimeError: CUDA out of memory. Tried to allocate 11.85 GiB (GPU 0; 9.78 GiB total capacity; 431.67 MiB already allocated; 6.52 GiB free; 462.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I think the size of the input image is very big and when it is trying to upsample, it crashes, fixing this is going to be a pain since I will need to resize the coco annotations as well. Any way to just evaluate on resized images (along with resized annotations)?
Also, I am also getting this error:
AssertionError: A prediction has class=2, but the dataset only has 2 classes and predicted class id should be in [0, 1].
However, when I set my NUM_CLASSES to 2 while training, I get CUDA error: device-side assert triggered


Margin property expands to max-resolution

TAB on margin expands to max-resolution: res;

If I disable emmet everything works just fine.
I also have installed:
autocomplete-css\
autocomplete-html\
autocomplete-plus\

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Doing more research this is not actually a bug.
Because emmet actually requires m to expand to margin in a css file. Like he needs mb for margin-bottom (http://docs.emmet.io/)
But it is weird to see the auto complete box with the thing you need expanding to something else.
Not sure how this use case can be handled.
Easiest way for now is to remove tab from emmet.

Hi. Weird enough, 6 years later, I have the exact same problem that started only since the 2.0 release but on Sublime Text. I was searching around to see if someone had that issue and how to solve it 👀


Please tag, both old versions and future ones

It would be very useful if each of the releases would get an own tag as this helps a lot to follow the changes between them,
For older versions I volunteer to create a shell script using the technique from https://stackoverflow.com/a/21759466/5027456 using the history of package.json to look up the commit hashes for each release (if I know that it is very likely to be applied).
For future ones: just tag directly before the release.
What do you think?

Coming back here as I've just wanted to check some differences between versions, which would be very easy to do if the versions would have a tag...
Ping @WebFreak001 Do you consider "tagging nicely"? As noted I'd be available to provide a script to tag old versions if I know that it is very likely to be applied.

just tags would be good to have already. I wanted to see if now creating a new CI action would affect new tags/releases being created on old commits. Having an action generating a zipped extension to install would be nice before making the tags if that is the case.

Having an action generating a zipped extension to install would be nice before making the tags if that is the case.

For CI/CD it is normally the other way around: you tag, possibly with a special schema like "vNN.NN" and then the action gets auto-triggered to publish via osvx for example and add a release from the tag which includes the vsix, too (tags on GitHub always lead to a "zipped" snapshot).
So, should I start to write the script for post-adding the old tags?

Now done (both the tag old/new and the inclusion of the .vsix as artifact).


Train with my own dataset

Hi, I try to train with my own dataset, and I make a folder "videos/new_videos", which has content like below pic:

But when I run the script, it appears:

I don't know what's wrong with this, how do I solve this problem?
(I give the args.dataset_path=videos/new_videos, a folder not the tar file, does this has relationship with the problem?)

Hi, I have solved this problem.(Make a .tar file).
Thanks a lot.


:book: Fixes and clarifications to the Argo CD Agent integration solution

Summary
Fixes and clarifications to the Argo CD Agent integration solution: https://github.com/open-cluster-management-io/ocm/tree/main/solutions/argocd-agent

/lgtm


DSET-3995 refactor - add dummy architecture plantUML picture

move plantUml diagram to github from https://www.notion.so/sentinelone/Documentation-Snippets-a56cfbf7817b4f909023d389c3dfbc56#3c7ecbf506e84e058007ae67cb91ef4a

Codecov Report

Merging #31 (87445a1) into main (8d16acf) will decrease coverage by 1.40%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #31      +/-   ##
==========================================
- Coverage   73.42%   72.02%   -1.40%
==========================================
  Files          10       10
  Lines        1358     1358
==========================================
- Hits          997      978      -19
- Misses        302      319      +17
- Partials       59       61       +2

see 2 files with indirect coverage changes


Some providers already out of business

Some of the provides already out of business.

Opneload
StreaMango
RapidVideo
StreamCherry
VeryStream

thanks, removed


Add vue.js to javascript frameworks

http://vuejs.org/

HI @icebob,
Putting the time it takes to learn Vue.js aisde, plus writing a generator for it, just to give you an idea here is how many modules there are to generate a React app in Mega Boilerplate:

Adding a new JavaScript framework is no small task.

Ok, I think. It was an idea. Maybe a vuejs master make it :)

I ran a quick survey but there doesn't seem to be much interest for Vue.js relatively speaking. Going to close this issue. If enough people are interested in Vue.js I will consider accepting a PR.

Hey @sahat!
I've been using megaboilerplate without a JS framework since it's first release, and would like to help you adding Vue to it.
As I can see, there's a lot of work to be done, but that's not the biggest challenge for me here. I can't find any contributing guideline (I've already made a PR some time ago, but at the time it wasn't a big piece of code) so, would you mind giving me some details of what need to be done, if I should work and commit to a branch before pulling to master, and so on?
Thank you.

@estevanmaito did you get a respond on adding vue?

+1 for Vue

+1 Vue would be a fantastic addition to the boilerplate.


Remove webhook validation code

Proposed changes
Problem: The webhook validation code has been removed from the upstream Gateway API repo, causing import errors when we try and run the conformance tests against the main branch of the Gateway API repo.
Solution: Remove the web hook validation code from our repo, and remove the web hook assumption panics.
Testing: Added additional unit tests where appropriate, ran the conformance test suite against the GW API main branch
Closes #1512
Checklist
Before creating a PR, run through this checklist and mark each as complete.

[x] I have read the CONTRIBUTING doc
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have checked that all unit tests pass after adding my changes
[x] I have updated necessary documentation
[x] I have rebased my branch onto main
[x] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes
If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NONE

One validation is not supported by CEL and needs to be implemented manually: https://github.com/kubernetes-sigs/gateway-api/issues/2277