Jakarta Webservices Specification

A preview of the proposed specification text can be found here:

osgi.cmpn-8.1.0.pdf

I also added TCKs for a simple webservice registration and the both Handlertypes now, to prove basic implementation of the spec here is an implementation of the spec that fulfills the current TCK

https://github.com/laeubi/osgi-xml-webservice

A preview of the proposed specification text can be found here: osgi.cmpn-8.1.0.pdf
I also added TCKs for a simple webservice registration and the both Handlertypes now, to prove basic implementation of the spec here is an implementation of the spec that fulfills the current TCK
https://github.com/laeubi/xmla-services-workground/tree/osgi_spec_impl/ws/runtime/registrar

I updated the spec to reflect the recent changes osgi.cmpn-8.1.0.pdf

I updated the naming discussed now and updated the specification osgi.cmpn-8.1.0.pdf

I have now moved my TCK and reference implementation to this repository:

https://github.com/laeubi/osgi-xml-webservice

this should make it able for the build to succeed here.

@stbischof @timothyjward I think I need some help here it looks like my new added jakarta.xml.soap-api (version 3) now conflicts somehow with CDI (version 1.4) both are in the central index and I would have assumed bnd to sort that out but it seems not the case.

[ ] Shorten examples by using multi-catch
[ ] Make clear that registering a (invalid) handler will result in an error
[ ] Add a property type that enables the pickup of Handler interfaces by the whiteboard / probably combine with the filter expression type
[ ] jax-ws-ap --> jakarta xml webservices API
[ ] runtime API doc is missing

I have published the implementation here:

https://github.com/eclipse-osgi-technology/jakarta-webservices/pull/1

There are also issues with the implementation, which currently does not work due to incompatibilities between metro and JAXB.

I don't understand this, as I also wrote some TCKs for the implementation and also @stbischof has used the impl already im quite confident it works, but be aware that you likely want the org.glassfish.hk2.osgiresourcelocator in your setup to let all the jakarta stuff find its implementations. (see also https://github.com/osgi/osgi/issues/685 where I proposed a more general way to make OSGi integrate better wit J2EE)

Could you link the latest test and setup you made Christoph?

There are also issues with the implementation, which currently does not work due to incompatibilities between metro and JAXB.

I don't understand this, as I also wrote some TCKs for the implementation and also @stbischof has used the impl already im quite confident it works, but be aware that you likely want the org.glassfish.hk2.osgiresourcelocator in your setup to let all the jakarta stuff find its implementations. (see also #685 where I proposed a more general way to make OSGi integrate better wit J2EE)

My test registers an "Echo" service identical to the one in the specification:

It fails to register the endpoint due to "missing return" on a generated class. This turns out to be because the test bundle doesn't import any JAXB packages. If I add DynamicImport-Package: jakarta.xml.bind.* then it progresses further
It fails to register the endpoint due to Metro being unable to set up a JAXBContext. This appears to be due to JAXB trying to load a class from Metro without using the correct class loader.

Could you link the latest test and setup you made Christoph?

I'm pushing my work in progress to https://github.com/timothyjward/osgi/tree/tw/jakartaws-ct

Could you link the latest test and setup you made Christoph?

Yes I will add some more code to the implementation, I just wanted to mention that metro is/was not used at all, this is actually just some delegation of the jakarta stuff that is not really usefull here, we want to simply use the reference implementations.

Yes I will add some more code to the implementation, I just wanted to mention that metro is/was not used at all, this is actually just some delegation of the jakarta stuff that is not really usefull here, we want to simply use the reference implementations.

Normally you will need to pick an implementation, or put a lot of effort into supporting different implementations. The servlet whiteboard uses Jetty, the Jakarta REST whiteboard uses Jersey, neither of them are pluggable. This allows them to fix or work around issues to that users of the whiteboard don't have to.
Aries JPA tries to be pluggable, and it's very hard to maintain as a result.

@timothyjward I'm not sure I understand the reference implementation already is an implementation and it works, of course one can decide to use other implementations as well but most of them are not very integrated with OSGi or have very large dependency chains and require special configuration. Aynways maybe things become clearer once I add the integration tests to the implementation repo.

@timothyjward I'm not sure I understand the reference implementation already is an implementation and it works, of course one can decide to use other implementations as well but most of them are not very integrated with OSGi or have very large dependency chains and require special configuration. Aynways maybe things become clearer once I add the integration tests to the implementation repo.

Sorry if I wasn't clear - normally when implementing a whiteboard specification such as this one, the OSGi whiteboard implementation would have to target a specific implementation of the Jakarta specification. For example, the Felix servlet whiteboard implementation uses Jetty directly, and cannot use TomCat or Grizzly.
My expectation is that an implementation of the XML Web Services whiteboard will have to make a similar decision, and be tied to a specific implementation of jakarta.xml.ws. Otherwise the overhead of working around bugs becomes an exercise for the user of the whiteboard, which can be very painful.

Could you link the latest test and setup you made Christoph?

The runnable test are now here:

https://github.com/eclipse-osgi-technology/jakarta-webservices/pull/7


feat: Create a local thrift service provider for RemoteFunction

Summary:
This diff is to extract the common logic related to starting RemoteFunctionService to a single place, so that we can later reuse it.

Create a Class RemoteFunctionServiceProvider and extract the logic related to starting a local thrift server for RemoteFunctionService there.
Create a Singleton remoteFunctionServiceProviderSingleton so that the server is only started once per process.
Expose helper methods getRemoteFunctionServiceParamsForLocalThrift()
Use this in RemoteFunctionTest.cpp

Differential Revision: D65927269

address comments


CloudFormation template fails.

31 UTC+0100
MetaflowStack
ROLLBACK_IN_PROGRESS
The following resource(s) failed to create: [DefaultGateway, BatchS3TaskRole, ComputeEnvironment, NLB, Subnet1RTA, SageMakerExecutionRole, MetadataSvcECSTaskRole, Subnet2RTA]. . Rollback requested by user.

15:58:30 UTC+0100
ComputeEnvironment
CREATE_FAILED
An error occurred (ClientException) when calling the CreateComputeEnvironment operation: Error executing request, Exception : Instance type can only be one of [m5d.8xlarge, r4, r5, optimal, r5d.24xlarge, r4.16xlarge, r5a.2xlarge, m5.xlarge, m5ad.12xlarge, r5ad.large, i3.4xlarge, r5ad.12xlarge, m5a.24xlarge, m5.large, c5d.large, r5.large, m5d.2xlarge, c5.large, g4dn.4xlarge, g4dn.2xlarge, c5.metal, m5.2xlarge, r5d.4xlarge, c5d.24xlarge, r5d.metal, x1.32xlarge, c5, g4dn.xlarge, m5d.12xlarge, m5.metal, m5ad.xlarge, m5.12xlarge, m5a.8xlarge, c5d.9xlarge, m5ad.4xlarge, c5d.2xlarge, r5ad.4xlarge, d2, m5ad.24xlarge, r5d.2xlarge, r5ad, c5.24xlarge, r5.16xlarge, r4.xlarge, r5.xlarge, m5a.2xlarge, m5ad.large, r4.8xlarge, m5d.xlarge, m5.8xlarge, c5.2xlarge, m5d.16xlarge, m5a.xlarge, r4.2xlarge, i3.2xlarge, r5ad.24xlarge, m5, m5d.4xlarge, r5d.large, m5a.12xlarge, c5d, c5d.xlarge, i3.xlarge, r5ad.xlarge, r5a.8xlarge, d2.4xlarge, r5a.16xlarge, m5.16xlarge, m5d.large, r5a.xlarge, c5.12xlarge, r5.8xlarge, r5d.xlarge, c5.4xla

2019-12-19 16:50:19 UTC+0100
metaflowstack
ROLLBACK_IN_PROGRESS
The following resource(s) failed to create: [DefaultGateway, ComputeEnvironment, NLB, RDSMasterInstance, Subnet2RTA]. . Rollback requested by user.

2019-12-19 16:50:17 UTC+0100
Subnet2RTA
CREATE_FAILED
Resource creation cancelled

2019-12-19 16:50:06 UTC+0100
RDSMasterInstance
CREATE_FAILED
Resource creation cancelled

2019-12-19 16:50:06 UTC+0100
DefaultGateway
CREATE_FAILED
Resource creation cancelled

2019-12-19 16:50:06 UTC+0100
NLB
CREATE_FAILED
Resource creation cancelled

2019-12-19 16:50:06 UTC+0100
ComputeEnvironment
CREATE_FAILED
An error occurred (ClientException) when calling the CreateComputeEnvironment operation: Error executing request, Exception : Security group ids are required., RequestId: c0db4815-52da-45db-9443-277d7a2f9003

@queueburt Can you assist on this?

@ideivitcom Looks like you don't have access to C4 instance types. As a workaround, you can change the instance types in the template to the instances available in your account and give it a try.

From the AWS C4 Page - C4 instances are currently available in the US East (N. Virginia), US West (Oregon), US West (N. California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Sydney), and Asia Pacific (Tokyo) AWS regions. Are you not in any of these regions?

I am from Spain. Using the Amazon free layer.
I will check your solution.

Selecting oregon to deploy doesn't work

What's the error that you are seeing now?

019-12-20 10:32:12 UTC+0100
metaflowstack
ROLLBACK_IN_PROGRESS
The following resource(s) failed to create: [ComputeEnvironment, NLB, RDSMasterInstance]. . Rollback requested by user.

2019-12-20 10:32:11 UTC+0100
NLB
CREATE_FAILED
Resource creation cancelled

2019-12-20 10:32:11 UTC+0100
RDSMasterInstance
CREATE_FAILED
Resource creation cancelled

2019-12-20 10:32:11 UTC+0100
ComputeEnvironment
CREATE_FAILED
Operation failed, ComputeEnvironment went INVALID with error: CLIENT_ERROR - Invalid IamInstanceProfile: ecsInstanceRole

Have you followed the README, more specifically the pre-requisites? - https://github.com/Netflix/metaflow-tools/tree/master/aws/cloudformation#prerequisites

Yes. Now it works.
Selecting Oregon and following: https://docs.aws.amazon.com/es_es/batch/latest/userguide/instance_IAM_role.html
Thank you for yout assist


[feat] selectively disable API routes

By adding something like this into config:
disabledRoutes:
  - /op
  - /kick

Would just "turn off" those routes. This would supercede any other selective route-based configurations such as the stuff mentioned in #75

Closing this in favor of #6 - I'll ensure this is possible with the new auth system


Feature/add hasura default types

Resolves #93

Codecov Report

Merging #103 (577cc61) into main (e924d8c) will increase coverage by 0.26%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #103      +/-   ##
==========================================
+ Coverage   58.57%   58.83%   +0.26%
==========================================
  Files          69       69
  Lines         770      775       +5
  Branches      130      130
==========================================
+ Hits          451      456       +5
  Misses        291      291
  Partials       28       28

Impacted Files
Coverage Δ

.../query-generator/src/generator/defaultFactories.ts
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e924d8c...577cc61. Read the comment docs.


Add list_benchmarks to compare models easily

This is a simple function for listing ImageNet benchmarks found here for users to see

adding pandas as a requirement might not be great, and the function is simple enough to be done without pandas. could you rework this to use the standard library? urllib to download the file, and then the csv module to load the data, for example.

@kaczmarj Adding pandas to the list of dependinces will provide room in the future for more features to be added.
I can do what you want if this what you think is best for the project.


Fancy Zones: All templates should have vertical + horizonal options, and editable splits

OS: Windows 10 Pro 64bit
PowerToys: 0.16.0
# Summary
I have a PLP (portrait-landscape-portrait) monitor setup. With 16:9 widescreen displays. This ends up looking like the letter "H", with the crossbar of the H higher than the middle of the side screens. The extreme top and bottom of the side screens is harder to read than the middle ~70% of the display because of the viewing angle.
Fancy zones (0.16.0) doesn't have good layout options to deal with this.
# Proposed technical implementation

There should be vertical and horizonal versions of all template layouts (currently there is no row-style Priority Grid with a middle row larger than 33% of screen height)
The divider lines in the template layouts need to be movable. One option is to have a custom copy of a template made automatically if the user edits a layout based off a template.
If the user creates an asymmetric layout, there should be an option to create a linked clone, with the clone mirrored (left/right or top/bottom) from the starting layout. The clones should stay linked by default so editing one half of the pair automatically updates the other.
It would be desirable to have an option to break the link between pairs so the clone becomes a standalone (non-linked) copy that can be edited independently.

Hi @incansvl
thank you for your feedback.

The divider lines in the template layouts need to be movable.

they are:

Regarding:

One option is to have a custom copy of a template made automatically if the user edits a layout based off a template.

I doubt we will implement such a feature since it will be easier to allow to create a copy of a custom template, that is more flexible than creating new templates automatically.
3 and 4  are nice o have, but at this stage there are several higher priority features for the editor v2, I doubt we will have the time to support this feature in the near future.

Aaah, me being dim. In the table layout I happened to choose, the "Name this layout" popup menu came up directly on top of the splitter drag handle, so I didn't see it. I ended up achieving the same aim by using the custom editor and manually creating a table...
User error on that one.

User error.

We need to improve the UI because we should never hide the splitters.


Allow verbose log in console

Currently it does  not print debug logs to
the console, only crash report. We should allow by passing
a flag as argument for more verbose log.
It helps when testing.

I think the need for this was removed with #75

The console still has a hardcoded INFO level. I think we need to specify a --debug flag so it will print everything in the console. Currently I manually enable it when working by changing log.py. And technically I could be monitoring file logs/username.log, however it's much more convenient to be looking to the console screen when testing.

This is added in the upcoming v1.2.0. Closing the issue to clean things up.


Links on main page not clickable

Happened to be on the p5.js website today and I noticed the links in the main copy (section#home) are not clickable.

eep! thank you

fixed with https://github.com/processing/p5.js-website/pull/31


Merge Constantine backend with BN254 curve based PoR

The backend is basically ready to be merged in #101
However, Constantine is Nim 1.4+ only at the moment. Thus maybe we should merge all the changes to have the backends, but keep the Constantine backend on a branch.

Ah, that is a shame. We should get as much of it merged, but don't activate it for now... What I'd really like to settle on is a good backend agnostic front end api, that we can rely on....


Update sizing.rst

Codecov Report

Merging #307 (8cc8e21) into main (355e8fa) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #307   +/-   ##
=======================================
  Coverage   49.56%   49.56%
=======================================
  Files          95       95
  Lines       17102    17102
=======================================
  Hits         8476     8476
  Misses       8626     8626

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


crypto-square: tests are now generated

Why?
All tests should be generated as per #396
What?
This PR contains a generator for the crypto-square exercise.
How Has This Been Tested?
bin/generate crypto-square
rake test:crypto-square
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (non-breaking change which adds functionality)
[ ] Breaking change (fix or enhancement that would cause existing functionality to change)
[ ] Gardening (code and/or documentation cleanup)

References and Closures
#396
Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My change relies on a pending issue/pull request
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

Hi, @ajwann! It looks like this PR is using an older version of the canonical data. I submitted https://github.com/exercism/ruby/pull/738, which uses the latest version of the canonical data. Would you mind taking a look and letting me know what you think?

It looks like this is a case of "great minds think alike" and we have 2 PRs to generate tests for the same problem.
#738 Looks like a currently more complete and functional solution so I'll close this one.
Thanks @ajwann for your work on this. :heart:

@nholden I should have though to pull from master in my x-common repo! Oh well, nice work with the PR.


Delete images (repository + data) from registry

What are the ways to delete the pushed images and repositories from registry?
I guess, Docker does not provide any API to do that operation. Any ways, may be using curl, to delete image data from registry?

I don't have an answer but have a similar question.
I pushed the busybox image to my private 2.0 registry (distribution).
This will work for me.  I get a huge json payload.
curl -XGET http://private-host:5000/v2/busybox/manifests/latest

The API doc says that DELETE should work.  So I just substitute above and get
curl -XDELETE http://private-host:5000/v2/busybox/manifests/latest
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

So I think that's your answer but it's not working for me.

Delete will remove the tag, but not the actual payloads.
So, your image will no longer be available, it will still exist on your backend storage.

Some people have contributed garbage-collector scripts to do the cleanup. You will find them clicking the "delete" label over here.

I think the issue here is that I'm getting a method not supported on the DELETE request not that the blobs are sticking around.  If you have an example curl example, that's be great.  I could be doing something wrong here.  If the OP disagrees, I'll open another issue.

@squarism so, you are using the golang registry2.
Please head over to https://github.com/docker/distribution and/or irc #docker-distribution
TLDR: delete is not supported yet, in design phase: https://github.com/docker/distribution/issues/210

Aaah!  And I blogged the url to the distribution repo too!  I'm sorry, I knew the difference, I failed to see what issue list I was in.  :sob:  Thanks for the tldr too.  :+1:

;)

+1 with my operation and output:
adolph@geek:docker_registry$ curl -v -X DELETE http://localhost:5000/v2/busybox/manifests/sha256:0fc02bc170932a0d727de2201c65bfe3fe31448dfed6f6bcbc1b37c2668927f5
* Hostname was NOT found in DNS cache
*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 5000 (#0)
> DELETE /v2/busybox/manifests/sha256:0fc02bc170932a0d727de2201c65bfe3fe31448dfed6f6bcbc1b37c2668927f5 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Dec 2015 05:57:29 GMT
< Content-Length: 78
<
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
* Connection #0 to host localhost left intact
adolph@geek:docker_registry$ curl -v -X DELETE http://localhost:5000/v2/busybox/manifests/list
* Hostname was NOT found in DNS cache
*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 5000 (#0)
> DELETE /v2/busybox/manifests/list HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Dec 2015 05:59:35 GMT
< Content-Length: 78
<
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
* Connection #0 to host localhost left intact

I sent the same request with @adolphlwq 's request, and got the same response
curl -v -X DELETE http://myregistry/v2/busybox/manifests/sha256:blablabla...

{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

Hi,
I have tried like this "curl -X DELETE localhost:5000/v1/repositories/jenkins_ora_jdk/tags/latest" but this works for tag deletion and not the image data for that tag. So i written my own script to do that Job. Since i am only the one who tested this script please use with your own risk
This is how i start registry image
mkdir -p /docker/registry/latest && docker run -d -p 5000:5000 -v /docker/registry/latest:/tmp registry
## NOTE
1. Registry service must be running
2. Volume must be mounted locally ( -v /docker/registry/latest:/tmp )
docker_registry_delete_image.sh.txt
Thanks,
Phani Kumar

@CodeJuan You must set the registry config.xml with

Hi,
just adding my experiences:

started registry:2 with my own config and own volume mounted storage.
Pushing images into my registry works ok, my storage contains the data (approx. 100MB)

Now trying to delete:

curl -v -s -X DELETE "localhost:5000/v2/collctor/blobs/sha256:cbb2acf4..."  (see layerDigests above)
curl -v -s -X DELETE "localhost:5000/v2/collctor/manifests/sha256:66846..." (see Docker-Content-Digest above)

After the above steps, we still have an entry in the catalog:
curl -v -s -X GET "localhost:5000/v2/_catalog" .. which still returns the deleted image
curl -s -X GET localhost:5000/v2//manifests/latest still returns seemingly valid data
but
a docker pull localhost:5000/ does not work any more.
And, the storage space (my ~100MB) are still in use and are not deleted.
Any ideas?

This repo here is deprecated, and any message here is likely to go unnoticed.
Either report problems on the docker/distribution repo, or on irc (#docker-distribution)

https://github.com/EugenMayer/docker_registry_cli/blob/master/DockerRegistryRequest.rb#L75 waits for this, so pretty interested in getting this done

@EugenMayer I've been messing around with this too. It looks like the digest that you return as a header from: get v2/<repoName>/manifests/<tagName> is the wrong digest.
Instead, if you docker pull that image from your registry and grab the Digest it spits out after downloading you should should be able to delete using that with:
curl -u <username:password> -X DELETE https://yourregistry/v2/<repository>/manifests/<newdigest>
If anyone knows of a simple way to get this digest other than pulling the image, please let me know.
 On the docker host that is running your registry, run:
docker exec -it <name of container or container ID> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml. It shows that you have N blobs eligible for deletion. Run the command again without --dry-run and it will delete the blobs. Now that they're deleted, if you run the dry-run again you should see that the namespace you deleted has no more blobs.

as requested i created a new issue #1068, maybe you continue work with me there @jshapiro26

ups, used the legacy issue, @jshapiro26  see https://github.com/docker/distribution/issues/1637 instead

My approach how to delete image tag from docker registry (tested on registry version 2.4.0):
1) Get tags list
GET <protocol>://<registry_host>/v2/<repo_name>/tags/list
2) Get manifest for selected tag
GET <protocol>://<registry_host>/v2/<repo_name>/manifests/<tag_name>
3) Copy digest hash from response header
Docker-Content-Digest: <digest_hash>
4) Delete manifest (soft delete). This request only marks image tag as deleted and doesn't delete files from file system. If you want to delete data from file system, run this step and go to the next step
DELETE <protocol>://<registry_host>/v2/<repo_name>/manifests/<digest_hash>
Note! You must set headers for request - Accept: application/vnd.docker.distribution.manifest.v2+json
5) Delete image data from file system
Run command from the host machine:
docker exec -it <registry_container_id> bin/registry garbage-collect <path_to_registry_config>
Note! Usually, <path_to_registry_config>=/etc/docker/registry/config.yml

Yes, i already implemented this in my docker_registry_cli:
https://github.com/EugenMayer/docker_registry_cli/blob/master/commands/DockerRegistryCommand.rb#L57
So this works - one of the things i missed was the Accept header: application/vnd.docker.distribution.manifest.v2+json]
But other then that, it works quiet butifully

There is no API available to trigger the GC so I think it's the only possibility to use docker exec .. ? I'm able to put it in a cron job or something but I don't know how I can get my registry in read-only mode, execute it and disable read-only (were need to push images to the registry). Or is the only solution to stop the container, start it as read-only, docker exec .., stop container and start it in a normal way? This would take "a lot of" time I assume. (I want to run the GC every time we delete an image)

For, the GC is utterly broken in 2.5 - i would assume the registry being completely useless after more then 2 months - you have to wipe it a redo all images. There are severe issues with blobs, pushed will not work because "the layers are already up to date". Using the GC is completely useless, the only thing actually really kinda works is going to /var/lib/registry/repos.. and delete the folder manually - restart the registry container.

@lorenzvth7 and @EugenMayer, I ended up writing an interactive script for the process I mentioned above and will share it soon and post back here, maybe it'll be useful for someone. The issue i've had is that my team typically just uses the latest tag. The latest starts bloating and there isn't a good way to clean up old versions of latest.
I ended up just parsing the output of the --dry-run garbage collection command and comparing the SHAs against the SHA I get from latest when I run a docker pull. This is because Docker-Content-Digest: <digest_hash> never seems to be correct.
I noticed your comment about the header: Accept header: application/vnd.docker.distribution.manifest.v2+json and will check that out. Maybe it'll solve all the problems!

@jshapiro26 I'm able to get the right sha of my image with tag:latest.
I'm using this command:
curl -k -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://myreg/v2/proj2/base/manifests/latest 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'
sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe

Check in my registry
$ docker exec -it 86deeac4f25f /bin/sh
/ # cd /var/lib/registry/docker/registry/v2/repositories/proj2/base/_manifests/revisions/sha256/52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe/

Hope it helps for your. My issue is the following now:
I delete the sha:
curl -k -v --silent -X DELETE https://myreg/v2/proj2/base/manifests/sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe
This is going fine:
> Accept: */*
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0

I use the GC to clean up:
docker exec -it 86deeac4f25f bin/registry garbage-collect  /etc/docker/registry/config.yml
$ docker exec -it 86deeac4f25f bin/registry garbage-collect /etc/docker/registry/config.yml
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c8/c8144262002cd241e607d7d3ecda450ce4ae8edf7dac8dbf46897d498ac667d8  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/ce/cee0974db2b868f0408f7e3eaba93c11fce3a38f612674477653b04c10369da0  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/06/064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/39/390957b2f4f0cd72b8577795cd8076cdc21d45c7823bbb5c895a494ae6038267  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3d1d7dffe2da492324b1fd62a2939ddc6e50549746cefa9a67684253dfa40b88  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/52/52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/6b/6bc128ff8a43e6b26862384494bd2cbfabc5e9d5277121881cbd0fcbf1442508  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/a5/a52d570eb492f9fb3cebb345c96395fc6156eef00ac42758e1a2d0f4190ecbf6  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392

Check (seems good to me):
/var/lib/registry/docker/registry/v2 # du -sh
63.0K

But now the real issue is repushing:
The push refers to a repository [myreg/proj2/base]
5d3accbaa61c: Layer already exists
3af563f9cad6: Layer already exists
447f88c8358f: Layer already exists
df9a135a6949: Layer already exists
dbaa8ea1faf9: Layer already exists
8a14f84e5837: Layer already exists
latest: digest: sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe size: 1576

Everything seems to exist already in the registry (isn't it cleaned up?).
The sha is the same which seems normal to me (it's the same image).
But checking in the registry?
/var/lib/registry/docker/registry/v2 # du -sh
65.0K

The most of space is taken in /bin/registry
/bin # du -sh registry
26.0M	registry

Something I don't understand.
after deleting the sha and rerunning the GC nothing seems to happen (the tag and _manifest is gone in the registry, but it seems not necessary to delete layers again, the space in / remains the same?).

Thanks for the clarification about sending the header! I'll try that out. It sounds like you likely have an older version of that image that would be untagged now since you've deleted latest. If you remove that image again then ssh to your registry and run docker exec -it registry_registry_1 bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml does it return any SHAs or blobs under myreg/proj2/base? That might be where the "Layer already exists" is coming from.

I pushed the image only with the latest tag. Than I'm able to delete it correctly but after repushing just the same image with just the same tag it goes wrong. Everything seems to exist, All the folders in /var/lib/registry/docker/registry/v2/blobs/sha256/xx are empty. There is a new tag and a new manifest in the registry but it does not take any space. Even pulling the image from the registry does not work..
When I push another image with the same name and same tag (also latest in this case) it's pushing in a fine way (I build it with --no-cache, don't know if it's relevant). I'm able to pull it and everything so this issue only seems to appear when I try to push, delete, repush and pull the exactly same image.
Maybe it's all a bit unclear but to resume:
It seems to work fine for me (pushing, pulling, deleting, clearing) in any case except when I delete image:x and repush image:x (which is the same image, not only the same name). After the "repush" everything still seems to exist but I'm unable to pull.

did you check the output of the --dry-run of garbage collection?
@EugenMayer here is the script I wrote to actually "garbage collect" https://github.com/jshapiro26/registry-tools/blob/master/clean_docker_registry.rb It's a bit of a hassle as it requires user input, but much faster than other methods i've seen.
I've set the garbage-collect task to run via cron on the docker host that my registry runs on, this script will mark the appropriate SHAs and blobs to be deleted when that cron runs.

@EugenMayer I am running the same issue as yours with the "being unable to repush an image", as the deleted layers tells it is already there. When I list the images, it does not appear neither.
I have a similar shell script, will try to find out the --dry-run trick if it solves my repush issue.

@jshapiro26 I am tempted to rewrite your ruby script in shell.

@jshapiro26 your ruby script suffers from the same defect as @EugenMayer mentioned. When I repush the same image, I get the same message "Image already exists", and when I try to pull it from the repo, I got the error "Error: image ubuntu not found".

@zoobab What image tag of registry are you using? I am using registry:2 and have never ran into the issue you're referring to.  The script doesn't actually run the garbage collection after all the images and blobs are marked. I solve this by have a cron run the garbage collection from the docker host early in the morning everyday.
To clarify what I wrote the script for, it is used to remove all versions of an image that are not tagged "latest". Therefore if you're pushing an image with a tag that's called "6.2" or something like that, those images will be removed in addition to all the untagged versions of "latest" leaving only the image with the latest tag for the repo you're "cleaning".
I am unable to replicate what you're seeing @zoobab with this script. I use it daily and have an automated build of new images weekly that push successfully. Can you give me some more details so that I might be able to help?

@jshapiro26 yes, I clean up after without --dry-run so that the disk siez diminishes (which is my goal here). My docker registry version is 2.5.0. When I repush the image after the garbage cleanup, I have this error "Layer already exists".

are you pushing a new build of the image with a different sha? The script leaves the image with the latest tag in-tact on the registry. If you are pushing the same image up it will throw the error you're seeing.

@jshapiro26 no, I am trying to repush an old image that was deleted, which has the same sha. In your case, you are repushing "latest" which has a new sha because it was freshly built.

@zoobab did you restart the registry after the garbage collection ?
I have the same behavior as you do if I run the registry GC and don't restart it. After it is restarted, no problem (which is totally understandable as the GC is done outside from the server process, which might have some cache).

@eesprit I can confirm a restart of the registry solves the "docker push oldimage" problem!

I thought maybe purging the upload cache would be the root cause of the "docker push oldimage" problem? Is there a way to purge this cache? I have seen some bits in the config.yml file.
maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false

There is no way to purge the cache manually?

I'm trying to delete following your instructions, but get a 403 Forbidden error.
I've set the REGISTRY_STORAGE_DELETE_ENABLED: true in the yaml file that starts up the registry.

I builded a docker-reg-gc shellscript for that stuff.
https://github.com/TranceMaster86/docker-reg-gc

Is there someone who knows what's the best way to delete unused revisions?
As far as I understand the normal way is to delete a revision based on its tag and after that you're able to run the GC which will clean all the rest.
But in DEV we are retagging an image on the same tag (latest). So if we try to use our delete-job (where you have to specify a tag) it only deletes one image (the newest/latest).
All the rest is stillt here because their revisions aren't deleted.

Look at my script there https://github.com/TranceMaster86/docker-reg-gc
It removes everything that is older than 150 days....or the count of days you want it to do. So the newst things are left in the registry

hi do you see that : https://docs.docker.com/registry/spec/api/#/deleting-an-image

Ive been using reg (https://github.com/jessfraz/reg/tree/master/vendor) to mark the blobs I want to delete from the registry by doing:
docker run --rm reg/delete -u username -p password --registry
myregistry.com rm image

I get the output of:
Deleted chrome@sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Then I ran the command in my server where the docker registry container is, to run the garbage collector like:
     sudo docker exec -it registry bin/registry garbage-
collect --dry-run /config/config.yml

And I got a lot of output back like:
DEBU[0007] filesystem.Stat("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=15.631837ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=6977da41-fd94-4926-99af-059821e18046 trace.line=137
account-portal: marking manifest sha256:17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee
DEBU[0007] (*manifestStore).Get                          environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry
DEBU[0007] filesystem.GetContent("/docker/registry/v2/repositories/account-portal/_manifests/revisions/sha256/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/link")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=86.1µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=ebd03857-56b6-4957-b791-9c48fdb152e9 trace.line=82
DEBU[0007] filesystem.Stat("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=74.19µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=96520ca1-6ba1-470c-869e-cf0a49846390 trace.line=137
DEBU[0007] filesystem.GetContent("/docker/registry/v2/blobs/sha256/17/17954313e0f8926875ce0743fe85693e0e9f695908c8f1544360d067a66b5eee/data")  environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry trace.duration=20.820776ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=6752e130-1b77-49c2-adda-3b9195aac1df trace.line=82
DEBU[0007] (*schema2ManifestHandler).Unmarshal           environment=development go.version=go1.7.3 instance.id=3296f17d-2ba7-4b3f-8530-991cb2ff8e04 service=registry
account-portal: marking blob sha256:90769d06ffb7dd1f8d1bc4f7b18e52fb94b651191640b680f028806622b99236
account-portal: marking blob sha256:3b8d91fe75724cafe136236e31d5212cb322d68aa3bfc4bfda97a0a53a55e234
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:dfc6505cfad4d185913ccd772541a0d32e64d4e04d9c249728dd7e194c77340f
account-portal: marking blob sha256:47b142d56ce3cc7766143d5fcd836541dac5006fa66076d4261e9e3d4a24a0cd
account-portal: marking blob sha256:6397456b1db08c3b5fb104497d15eb849a317c35ffeb18b608e6013593b82300
account-portal: marking blob sha256:f1074061d22a0681d732d72a8d11abf0a2f77ee3652ee827c62d696c19697a93
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:7b60f5a7551c3bb15dcd5ecd46a2ec18118cc652f6bdbff23bf16c895a8a1586
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:3b36e36d988def39cf035c9a82076bbff95861bd9092701516d937f7dff24a73
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:283fbeeb0d74113ed38891d145bcbff04a96178d2f1e19bb9dc5564ebd1959ab
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
account-portal: marking blob sha256:68ad5d9d0e66645895611177d50dfc46bf1bbf9cd1275e43da416af0d2b2c30a
account-portal: marking blob sha256:6fae118190fe88ba8487a3a2cfac7b1f2d922d313963cd81329fd37b2516ca3d
account-portal: marking blob sha256:9e14db6259dc2696e6fe816ddb106ec1810a4f39075695c66cbeccd58f54aad1

and at the end I get the recap like this:
12530 blobs marked, 0 blobs eligible for deletion
So according to the output no images were removed from my registry. I confirmed by running df -h and yes I still using the same amount of disk space.
So, I don't know what I'm doing wrong, I've followed all the instructions in the docker documentation about the Garbage Collector I don't know what do to do get space back, please help.

@messyhead,
I also got 403 responses for the DELETE requests. Turns out I was making the requests through the registry-frontend, which silently overwrote the config variable that should have enabled the DELETE HTTP verb.
The solution was to speak to the registry API directly instead of through the frontend. (ie. port 5000 instead of 80 or 443.)

I followed the steps in 224280919 and 224325910. However it ends up deleting all tags for a repository instead of just one. Garbage collection removes some unused layers as expected and the next push with a new set of tags only shows the new tags set, not the ones from earlier. Furthermore, after I push a image with newer tags, they can't be listed. Let me illustrate, please let me know what I'm doing something wrong here.
/ # /bin/registry --version
/bin/registry github.com/docker/distribution v2.6.1

$ curl -s  http://example.com:5000/v2/myrepo/tags/list | python -m json.tool
{
    "name": "myrepo",
    "tags": [
        "latest",
        "2017-05-16-19-02-latest",
        "2017-05-16-19-04-latest"
    ]
}

$ curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -D - \
  -o /dev/null http://example.com:5000/v2/myrepo/manifests/2017-05-16-19-02-latest
HTTP/1.1 200 OK
Content-Length: 1970
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b"
X-Content-Type-Options: nosniff
Date: Wed, 17 May 2017 02:24:06 GMT

$ curl -vs -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X DELETE \
  http://example.com:5000/v2/myrepo/manifests/sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to example.com (<IP_ADDRESS>) port 5000 (#0)
> DELETE /v2/myrepo/manifests/sha256:cb5ca4aa9346b738fbaf2a18711a301b23d452293b1526407b6bd5213dc7a58b HTTP/1.1
> Host: example.com:5000
> User-Agent: curl/7.53.1
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Wed, 17 May 2017 02:24:32 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host example.com left intact

# ALL tags are gone, was expecting to see the remaining two
$ curl -s http://example.com:5000/v2/myrepo/tags/list| python -m json.tool
{
    "name": "myrepo",
    "tags": null
}

Now push a new image with some tags. The tags are listed but can't be queried.
$ curl -s  http://example.com:5000/v2/myrepo/tags/list| python -m json.tool
{
    "name": "myrepo",
    "tags": [
        "latest",
        "2017-05-16-19-39-latest"
    ]
}

$ curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -D - -o /dev/null http://example.com:5000/v2/myrepo/manifests/2017-05-16-19-39-latest
HTTP/1.1 404 Not Found
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 17 May 2017 03:04:48 GMT
Content-Length: 184

My previous comment (301976594) is valid on registry:2.6.1. I cannot reproduce it on registry:2.5.1, so looks like a regression. I'll stick to using 2.5.1 for now

An addendum to my earlier post, two points:

The entire process from my previous note worked flawlessly with registry:2.5.1 (in the last step I am able to fetch the digest for the newly uploaded images)
the reason 3 tags disappeared when I deleted one digest was that they were all referring to the same image. My understanding earlier was that images hang around till all relevant tags are deleted. Turns out it is the other way around, tags go away when the image is deleted.

Anyway it's fine now and I will use 2.5.1 for now

what kind of b.s. is this again? this is a basic function of any tool.

@CodeJuan can you help with deleting?
How to get token with delete rights? As I understand, we need to set sope:pull, push,<some-delete-keyword>, is it?

This oneliner command may help
https://gist.github.com/jaytaylor/86d5efaddda926a25fa68c263830dac1#gistcomment-2662407


Reconsider "Sparse files SHOULD NOT be used" restriction?

https://github.com/opencontainers/image-spec/commit/f5a91ffe001afe2308628b112f3d9534fb94c2da added the following sentence to layer.md:
[Sparse files](https://en.wikipedia.org/wiki/Sparse_file) SHOULD NOT be used because they lack consistent support across tar implementations.

GIven that most implementations use Go and now Go supports sparse files (https://github.com/golang/go/issues/13548), can we reconsider this?
@vbatts @stevvooe

In #255, @vbatts made this an implementation-support issue instead of a filesystem-support issue, because he wasn't aware of many filesystems that didn't support sparse files.  The recent Go support for sparse files has turned up a number of cases that don't support sparse files (e.g. see golang/go@718d9de60fd4337d9044cdc2c685177dd2177ef6 giving up on trying to parameterize this, with the previous parameterization suggesting FFS and HFS+ lack support, and AUFS has only partial support).  However, if that's the argument we want to make, we'd still want to adjust the spec to mention inconsistent OS/FS support instead of inconsistent tool support.

I'm still keen to keep that sparse files SHOULD not be used. If an
implementation uses them, they are accepting the risk of checksums not
validating or similar.
On Thu, Sep 21, 2017, 18:03 W. Trevor King<EMAIL_ADDRESS>wrote:

In #255 https://github.com/opencontainers/image-spec/pull/255, @vbatts
https://github.com/vbatts made this an implementation-support issue
instead of a filesystem-support issue, because he wasn't aware of many
filesystems that didn't support sparse files
https://github.com/opencontainers/image-spec/pull/255#r77822800. The
recent Go support for sparse files has turned up a number of cases that
don't support sparse files (e.g. see golang/go@718d9de
https://github.com/golang/go/commit/718d9de60fd4337d9044cdc2c685177dd2177ef6
giving up on trying to parameterize this, with the previous
parameterization suggesting FFS and HFS+ lack support, and AUFS has only
partial support). However, if that's the argument we want to make, we'd
still want to adjust the spec to mention inconsistent OS/FS support instead
of inconsistent tool support.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/opencontainers/image-spec/issues/733#issuecomment-331294553,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEF6VUyQrWoZ94HtDkqaSHYDnnsEjlxks5skt01gaJpZM4Pe286
.

On Thu, Sep 21, 2017 at 11:31:03PM +0000, Vincent Batts wrote:

If an implementation uses them, they are accepting the risk of
checksums not validating or similar.

That's only a problem for unpack/repack, though right?  That seems
like less of a “passing image data between parties” issue and more of
a “some consumers may create sub-optimal images if they use an image
containing sparse files as a base” issue.  It's a risk, and I'm fine
calling it out (which we don't do at the moment), but I'm not sure
it's a SHOULD-worthy risk.  Folks will still be able to unpack the
image (assuming their local OS/FS is capable of supporting sparse
files or they have a big enough disk to hold the unsparsed file) and
run containers from it, which seems like the primary use-case
image-spec is enabling.  And if folks occasionally pack a new image
based on the content and happen to not recycle the same tar blob, it
still seems like a CAS win to not be holding unsparsed copies.

In general, sparse files should not be used. You're still welcome to use them but I don't see the benefit, especially with compression.
@AkihiroSuda Is there are particular use case that you had in mind?

adduser (with large uid value), which seeks /var/log/lastlog depending on uid value: https://github.com/moby/moby/issues/5419

@AkihiroSuda What if we added a section that says implementations should be able to consume them?
Considering the sparse support of sparse files, I'm a little reticent to remove this from the specification. It doesn't say you can't use them, but generating them may come with consequences.

OK, closing this issue.
However, if we encounter real compatibility problem across different implementations, I'd like to reopen this issue with some new annotation convention (org.opencontainers.image.has-sparse-tar-layer-blah-blah-blah)

Understandable
On Mon, Sep 25, 2017, 22:49 Akihiro Suda<EMAIL_ADDRESS>wrote:

OK, closing this issue.
However, if we encounter real compatibility problem across different
implementations, I'd like to reopen this issue with some new annotation
convention (org.opencontainers.image.has-sparse-tar-layer-blah-blah-blah)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/opencontainers/image-spec/issues/733#issuecomment-332070855,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEF6dbyhCxzRsYkLeLJqzXWThaY0ctIks5smGYfgaJpZM4Pe286
.


added maxpooling to nnj and corresponding tests

Codecov Report

Merging #18 (e2965e3) into master (e4afdd7) will increase coverage by 1.21%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #18      +/-   ##
==========================================
+ Coverage   61.23%   62.44%   +1.21%
==========================================
  Files           9        9
  Lines        1135     1177      +42
==========================================
+ Hits          695      735      +40
- Misses        440      442       +2

Flag
Coverage Δ

Linux
62.44% <100.00%> (+1.38%)
:arrow_up:

Windows
62.44% <100.00%> (+1.21%)
:arrow_up:

cpu
62.44% <100.00%> (+1.21%)
:arrow_up:

macOS
62.44% <100.00%> (+1.38%)
:arrow_up:

pytest
62.44% <100.00%> (+1.21%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

tests/test_nnj.py
100.00% <ø> (ø)

stochman/nnj.py
98.88% <100.00%> (+0.20%)
:arrow_up:

tests/test_curves.py
98.05% <0.00%> (-1.95%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4afdd7...e2965e3. Read the comment docs.


Monitoring locking

What feature do you want to see added?
It would be useful to have such type of lockable resource, that is locked/unlocked once by some job and other jobs are only monitoring resource's state and wait until it is unlocked, but not locking it themselves.
For example when running job, which perform HTTP Server maintenance, it locks Server resource and all other jobs, which use that Server are stopped until maintenance is finished.
Upstream changes
No response

I am not sure how it shall works, why you does not use lock() in both jobs?


Remove persistent error display

Error message, once displayed in the page, is persistent even if a new input is provided or a the decrypt button is clicked again.

Removed persistent error message


Add support for Enum Entity Columns

This PR is a part of https://github.com/SpineEventEngine/core-java/pull/698.
It adds functionality necessary to support enum entity columns to the querying mechanism.
This PR also migrates jdbc-storage to the latest versions of core-java and base. For the transition to be proper the Java version was updated to 1.8 and Guava version - to 25.1-jre (same as in stated repositories).
The version of jdbc-storage advances to 0.10.41-SNAPSHOT

@armiol PTAL.

@armiol PTAL. I believe poor diff coverage is the result of some issues with covering switch-case statement, as logically those line are covered.


fix(ratings): Whole number ratings displays as half ratings

Changes description 🧑💻

Fix issue about whole number ratings displays as half ratings
Context 🤔

Closes #151
Checklist ✅

[X] Make sure you are requesting to pull a topic/feature/bugfix branch. Please, don't request directly from your main!
[X] Check commits & PR names matches our requested structure. It must follow the https://www.conventionalcommits.org pattern.
[X] Check compose contract convention. It must follow conventions described here.
[X] Check your code additions will fail neither code linting checks.
[X] I have reviewed the submitted code.
[X] I have tested on a phone device/emulator.
I have tested on a tablet device/emulator.
I have tested on a large screen device/emulator.
If it includes design changes, please ask for a review design-system-core-team-design GitHub team.

Screenshots 📸

Test with number = 3.0f

Other info 👋

You can download the apk here :
sample-debug.apk :
https://app.bitrise.io/app/e9ae480151be4514/build/7aedd64b-b78a-41e0-85f2-695e8e9c7879/artifact/a9dd51855b53a8b9/p/4ca461b8738461c910ce97ae87dad0ae

@ManonPolle Is it possible to get a version with this fix?


Vertex and attribute compression

This scheme is backwards compatible so old meshes will continue to work.
Users can get a performance boost by reimporting assets
The rationale behind this PR can be found here: https://files.godot.foundation/s/259N9YQEPcBZwMx
In short, the main reason behind making this PR is bandwidth reduction (passing less data around the GPU per draw call). Bandwidth reduction is valuable in two cases:

If the scene is bandwidth limited, reducing bandwidth can improve performance
Reducing bandwidth reduces power consumption on mobile and power-constrained devices.

From our perspective, reducing bandwidth also has the added benefit that we can now confirm that we are not generally bandwidth bound (i.e. we aren't boundwidth bound in all scenes) which means we can focus our next performance improvements elsewhere (VGPRs / instructions).
Usage notes
After merging this PR all meshes will automatically start using a slightly different vertex attribute layout:
P = Vertex position
N = normal
T = tangent

// Before this PR
PNTPNTPNTPNT

// After this PR
PPPPNTNTNTNT

This is the recommended layout for mobile devices as it means that shadow passes and tiling passes only ever have to read vertex positions and it doesn't hurt performance on Desktop.
The renderer will automatically use this layout even if the mesh doesn't. To accommodate that, I have added a mesh format version number so we can perform automatic upgrades between versions. This avoids the issue we had last summer when integrating octahedral compression. Three things to note about this though:

When running scenes with the old mesh format, users will get warnings for each mesh that needs to be upgraded
Meshes need to save the new mesh format to stop getting the warning. In some cases this means they will have to reimport the mesh. Upon saving a mesh though, the new format will be used.
Once the new format is used, users can't go back to an earlier version without re-importing their mesh.

In addition to the vertex layout change, we also introduced optional compression for vertices, normals, tangents, and UVs. The importers will automatically detect if the mesh is a candidate for compression and apply the appropriate flags. For UVs, we can compress them as long as they are all within the 0-1 range. While this is technically a lossy compression, there won't be a visible difference on textures with a size smaller than 65536 pixels, so it is safe to apply). For vertices we do two things:

Scale the vertices into the 0-1 range using the surface's AABB (this allows us to store the meshes in RGB16 instead of RGB32
Store Normal (vec3) + Tangent (vec4) as Axis (vec3)+ Angle (float). The Axis replaces the normal and tangent fits into the A channel of the vertex.

I implemented this compression scheme in a way that does not require additional shader variants, so there will be no effect on shader compile time. However, this means that there are more calculations in the shader, and VGPR usage may be affected on some compilers.
When the compression flags are not used, the vertex format remains the same as it was before this PR.
Performance
So far I have tested on my laptop using an integrated GPU and an NVidia 3050m. There is no difference when using the forward+ renderer, a slight improvement when using the compatibility renderer, and about a 10% improvement when using the mobile renderer. This is likely due to the fact that I had to refactor the mobile renderer to pass the necessary data.
On the TPS demo on integrated graphics, I am seeing an improvement from ~17 FPS to ~20FPS (Forward+).
Using the Synty Vikings demo When using the mobile renderer, I get an increase from 40 ms per frame to 37 ms per frame.
Running the TPS demo on mobile (Pixel 7 and Pixel 1) I see no performance difference (likely because both are instruction bound, not bandwidth bound).
~More performance testing is needed before merging. But so far, in all cases the performance is either the exact same or slightly better.~
Todos:

~Add additional optimization by de-interleaving vertex positions (this won't be backwards compatible and so we need to add version identifiers to mesh data~
~Add a setting to force disable optimization on an imported mesh. Generally, the auto-detection works quite well. But it relies on the mesh having both proper normals and tangents. If the mesh has improper tangents, the result is also bad.~ In the end this setting wasn't needed. The issue came from a scene with non-continuous UVs, so the generated tangents were bogus. The solution was to allow GLTFs to respect the "ensure tangents" setting. Previously tangents were always generated if not supplied.
~more performance testing with heavier scenes on mobile devices (especially scenes with more shadows).~

Pushed an update to address comments and CI issues. Should be ready for substantive review now.

Thanks for the reviews AThousandShips!
Just pushed another version with docs and fixes for CI. This PR should be ready to go now. I don't expect to find any performance regressions in my final testing.
One catch, I am away on holidays from September 7 to 19. So if we merge this anytime soon I won't be around to jump on bugs. So we may want to merge this the week of the 18th so that I can be available to fix any bugs that arise. While I have done my best to test extensively, it is inevitable that bugs will appear from such a large change to a fundamental system

Changing gltf_document.cpp caused it to be recompiled, and a new MSVC warning was found (not caused by this PR but could be fixed here too to pass CI):
Error: modules\gltf\gltf_document.cpp(2958): error C2220: the following warning is treated as an error
Warning: modules\gltf\gltf_document.cpp(2958): warning C4334: '<<': result of 32-bit shift implicitly converted to 64 bits (was 64-bit shift intended?)

I think this kind of error can be fixed by changing 1 << l to 1U << l.

I’ll take a glance.

I didn't test and I only looked at the GDExtension compatibility code, but those bits look good to me!

While this is technically a lossy compression, there won't be a visible difference on textures with a size smaller than 65536 pixels, so it is safe to apply

you mean x and y size or total pixel count?

While this is technically a lossy compression, there won't be a visible difference on textures with a size smaller than 65536 pixels, so it is safe to apply

you mean x and y size or total pixel count?

x and y size

The change of parameter type to SurfaceTool.commit() requires a compat binding for GDExtension, but it seems to generate a conflicting compat method for C#

I don't know what's going on here, the method list contains the method commit twice (same hash and everything). Actually, could it be that we're not considering the metadata when calculating the method's hashes?

Rebased and addressed Bastiaan's comment. Maybe the rebase will fix the C# issue?

From what I understand, the method hash is calculated from the argument type but it's not taking the type metadata into consideration:
https://github.com/godotengine/godot/blob/4c3dc26367518e006f8555c12f5d2df0b8a28192/core/object/method_bind.cpp#L43
So for int32 and int64, the computed hash is the same. Since the registered method and the compatibility method shared the same hash, the C# bindings generator ends up generating the registered method twice.
I don't know how to change the hash computation without breaking compatibility though.

CC @dsnopek Do you think there is a way to make the binder take arg types into account without breaking compatibility? Similar to your recent PR fixing the hash for default arguments?

The binder always uses int64 as arg type, so this is a problem on the C# side.

To clarify you should not  have any problem with old code using this, new and old versions of the C# code generated will continue working.

Do you think there is a way to make the binder take arg types into account without breaking compatibility? Similar to your recent PR fixing the hash for default arguments?

Like Juan says, in GDExtension, all integer arguments are encoded as int64_t (ie. cast to int64_t, transferred across the boundary, then cast back to correct type). So, for the purposes of GDExtension we do want the hashes to be the same. I think the problem here stems from C# using the same data as GDExtension, while having slightly different requirements.
Could the C# code generator maybe have a list of exceptions that prevent the automatic generation, and then some handwritten code could be added for this case?

To clarify you should not have any problem with old code using this, new and old versions of the C# code generated will continue working.

That's not how compatibility works in C#, if the method doesn't exist in the new bindings it will not be found by old code.

Could the C# code generator maybe have a list of exceptions that prevent the automatic generation, and then some handwritten code could be added for this case?

We could avoid generating compatibility methods that have the same hash as an existing non-compat method. However, I feel like this defeats the purpose of https://github.com/godotengine/godot/pull/80527. The goal with that PR was automating this so contributors didn't have to add compatibility methods to C# manually (see Compat.cs).

Like Juan says, in GDExtension, all integer arguments are encoded as int64_t (ie. cast to int64_t, transferred across the boundary, then cast back to correct type). So, for the purposes of GDExtension we do want the hashes to be the same.

But if an argument has different type metadata, doesn't that make it different enough to guarantee having different hashes? In fact, this PR is providing a compatibility method precisely because it changes the metadata of an argument that used to be int32_t and is now int64_t. So if the metadata didn't matter, why do we need to add a compatibility method?

I think the problem here stems from C# using the same data as GDExtension, while having slightly different requirements.

The problem is that I can't retrieve the compatibility method when it has the same hash as a non-compat method, because ClassDB::get_method_with_compatibility will always retrieve the non-compat method. I feel like having multiple methods with the same hash registered in ClassDB is not right.
But I understand that changing the way we compute the method hash is not a solution, since it would also change the hash for every other method in extension_api.json and therefore break compatibility everywhere. So I don't know how to solve this, other than going back to adding compat methods manually for C#.

But if an argument has different type metadata, doesn't that make it different enough to guarantee having different hashes?

On a purely technical level, for GDExtension, we don't need for these hashes to be different in this case, since the encoding used when calling the function won't change.
That said, we definitely didn't get the hashing algorithm perfectly correct. :-) There are cases where the hash will change (for example, changing from one enum type to another), but technically, we don't need it to be different because the encoding is the same. It may or may not be worth trying to fix those cases eventually.

So if the metadata didn't matter, why do we need to add a compatibility method?

I don't think we actually need a compatibility method here, if the only change is an argument going from int32_t to int64_t. I think it could probably be removed and GDExtension would be fine. If the validator gives a "hash has changed" error, I think that'd be a bug in the validator.

@raulsntos

That's not how compatibility works in C#, if the method doesn't exist in the new bindings it will not be found by old code.

Sure but this is exclusively C# -> C# thing. I can only see it being a problem if you have a .net DLL using another .net DLL. This seems a little wierd for me, still you fix it simply rebuilding both things. For the C# -> C++ side of thing nothing changes since it should be as if this change does not exist for C#.

But if an argument has different type metadata, doesn't that make it different enough to guarantee having different hashes? In fact, this PR is providing a compatibility method precisely because it changes the metadata of an argument that used to be int32_t and is now int64_t. So if the metadata didn't matter, why do we need to add a compatibility method?

We did not need to add a compatibility method, if it was added it was a mistake, but I did not see this in the PR, maybe I missed it, let me check.

@raulsntos @dsnopek My apologies, there was one in the .inc file, I missed it. You should not need to do anything on your side.
@clayjohn The problem is that you added a compatibility method for a function that changed from uint32_t to uint64_t, this is not required because for the binder everything is int64_t anyway, so its generating issues with the same method getting bindings twice. Keep in mind if the method takes a default value and this changes, then you do need to add one as this changes hash.
@clayjohn @YuriSizov Sounds like this should be added to do con compat methods (if you change integer or float type, the method hash is the same and no compatibility method is required).
@clayjohn If you remove the compatibility methods that only change integer type, your PR will pass CI.

Sure but this is exclusively C# -> C# thing. I can only see it being a problem if you have a .net DLL using another .net DLL. This seems a little wierd for me, still you fix it simply rebuilding both things. For the C# -> C++ side of thing nothing changes since it should be as if this change does not exist for C#.

Yes, but you may not be able to rebuild everything. If the user is consuming a library from NuGet, then they can't rebuild that library, they can only wait for the library to be updated. Updating the library will make it incompatible with older versions of Godot, so the author may not want to do that.

We did not need to add a compatibility method.

Maybe not for C++, but C# still needs it. If the compat method won't be registered, then we need to add a C# method manually to Compat.cs and that's more work for contributors (most may not be familiar with C#).

Maybe not for C++, but C# still needs it. If the compat method won't be registered, then we need to add a C# method manually to Compat.cs and that's more work for contributors (most may not be familiar with C#).

I think this may be acceptable for now, since we don't want to block this PR on an infrastructural issue. So if we can work around it with a Compat.cs method, I think this is what we should do, and hope to resolve this layer.

This diff adds compat methods for ImporterMesh::add_surface and SurfaceTool::commit. The other methods can't have compat overloads because the only thing that changes is the return type, so we'll break compat there.
diff --git a/modules/mono/glue/GodotSharp/GodotSharp/Compat.cs b/modules/mono/glue/GodotSharp/GodotSharp/Compat.cs
index 48b47b166a8..aa03c3f97f1 100644
--- a/modules/mono/glue/GodotSharp/GodotSharp/Compat.cs
+++ b/modules/mono/glue/GodotSharp/GodotSharp/Compat.cs
@@ -44,6 +44,16 @@ partial class Geometry3D
     }
 }

+partial class ImporterMesh
+{
+    /// <inheritdoc cref="AddSurface(Mesh.PrimitiveType, Godot.Collections.Array, Godot.Collections.Array{Godot.Collections.Array}, Godot.Collections.Dictionary, Material, string, ulong)"/>
+    [EditorBrowsable(EditorBrowsableState.Never)]
+    public void AddSurface(Mesh.PrimitiveType primitive, Godot.Collections.Array arrays, Godot.Collections.Array<Godot.Collections.Array> blendShapes, Godot.Collections.Dictionary lods, Material material, string name, uint flags)
+    {
+        AddSurface(primitive, arrays, blendShapes, lods, material, name, (ulong)flags);
+    }
+}
+
 partial class MeshInstance3D
 {
     /// <inheritdoc cref="CreateMultipleConvexCollisions(MeshConvexDecompositionSettings)"/>
@@ -106,6 +116,13 @@ partial class SurfaceTool
     {
         AddTriangleFan(vertices, uvs, colors, uv2S, normals, new Godot.Collections.Array<Plane>(tangents));
     }
+
+    /// <inheritdoc cref="Commit(ArrayMesh, ulong)"/>
+    [EditorBrowsable(EditorBrowsableState.Never)]
+    public ArrayMesh Commit(ArrayMesh existing, uint flags)
+    {
+        return Commit(existing, (ulong)flags);
+    }
 }

 partial class Tree

Updated to:

combine compression options into one
rebase on master
remove compatibility methods where the only change is uint32_t->uint64_t
add C# compatibility methods

As far as I know, this should be good once I squash the commits. To keep things easy to review (and avoid accidentally deleting necessary compat methods) I have left this in multiple PRs
@raulsntos can you confirm that the C# compat looks acceptable now?
@dsnopek Can you confirm again that the GDExtension compatibility is still fine?

This should be ready to go now. I have tested extensively including:

TPS Demo
Large Synty demo scene
Various GLTfs including those with blend shapes, skeletons, etc.
Bistro scene

And with all three renderers. In the current state I haven't caught any issues to be aware of.
One big thing to make users aware of is that we are switching to a new mesh format version. This means that when they run existing scenes they will get a warning that their mesh has been upgraded to a new version and then need to re-save or re-import the mesh.

Thanks!


Dashboard 'Pages' and 'Navigation' not working properly

What I'm trying to achieve
Add a child Page to the navber in "Pages" and "Navigation" in the Configuration section.
There are 2 problems:

The content of the page doesn't show up, it's just a blank section
The navigation element never appears as a child, even though it's indented in the UI as a child, like it's described in the documentation

Steps to reproduce the problem

Simply add a Page with content and try to add a child navigation. Doesn't work.

System information
Operating system: MacOS
Browser: Chrome + Safari (Tried both)

@r-zammit Hi there! Apologies for the delayed response. Could you please tell me which version of Saleor are you using?


Export: better performance by avoiding pointless image->python data->image roundtrips

This PR creates a "happy path" for exporting images where we don't have to round-trip through Python data, which is much faster and uses much less memory.
Fixes #790 and helps with export times generally.
Problem
The glTF spec requires that eg. the metalllicRoughness texture have the metalness in B and the roughness in G. The exporter will assemble the appropriate image for this even if the material graph reads the metalness from, say, the R channel of one image and the roughness from the G channel of another.
To do this, all images are decoded to python data. The code sequence for this looks roughly like this:
creating export image:
    pixel_data = np.array(blender_image.pixels[:])              (★1)

pixel data may be modified to insert channels

encoding export image:
    if created from blender image that's on disk:
        read back data from the disk

    tmp_image = bpy.data.images.new(...)
    tmp_image.pixels = pixel_data.img.flatten().tolist()        (★2)
    save tmp_image to disk and read it back

The lines marked with a ★ are slow and eat up loooots of memory. Eg. blender_image.pixels[:] is going to allocate a tuple of width×height×channels floats. It's really bad.
As you can see, sometimes it was possible to skip ★2 but everything did ★1.
However, for many images, like all color textures, normals, or metal/rough/occlusion textures that are already packed the way the exporter likes, this is totally wasted effort: the image data is converted to python data, nothing gets changed, and the python data is converted back to a new temp image, which is then encoded to disk and read back.
In this case it would have been better to just encode the original image!
Solution
Change ExportImage to store some declarative data saying how it's channels should be filled in (ex. it says its R channel should be filled by the G channel of such-and-such Blender image) instead of storing a numpy array. I tried to write a good docstring here, so see it for details.
Then when it comes time to encode, we examine that data and determine whether all we need to do is save some existing Blender image (happy path), or whether we need to do the pixel manipulation to repack the channels (unhappy path).
The unhappy path is basically the same as before, but the happy path can be significantly better :)
Results
Results seem pretty good. Obviously they depend on how much you can stay on the happy path but since that should include the most common texture types (base color, normal, etc.) it should be pretty often.
Export times:

Before
After

#790 (8192x8192 normalmap)
OOM
3s

DamagedHelmet
15s
7s

VC
17s
6s

Sponza
91s
38s

Reviews and testing are appreciated!

I will have a look, but because material/texture is not my main domain of expertise, I hope @emackey or @donmccurdy can have a look

I wrote a bash script for testing if anyone else wants to try. Make sure you have the faster-export-image branch, cd in the glTF-Blender-IO folder, and run test.sh FILE. FILE will be loaded and then exported with both master and the faster-export-image branch, and the .pngs will be compared with imagemagick.
So far the only difference I have found is desirable IMO: this branch will sometimes figure out it can use an existing Blender image without change, where master would have changed it.
For example, if you import MetalRoughSpheres.glb, you get this image

On master, if you export, the red channel will have been filled with ones

whereas on this branch it exports exactly as it was.

Oh, here's a difference: when Blender textures are missing, it was possible to have an empty ExportImage which would export as a single pixel image. In master, the image is just thrown away in this case. I'll add a commit to fix it.

So I tested it with dirty images (ie. images that have been painted on and not saved; in the Image Editor there will be a * next to the Image menu if the current image is dirty) and I found another bug. After export, any image that goes down the happy path will have its dirty flag cleared since it was saved to a temp file.
This can be fixed by just doing image.pixels[0] = image.pixels[0] on images that were dirty to set their dirty flags again. This is can be pretty slow for some reason. Luckily we only pay this cost if an image was dirty. I added a commit to do this.
Btw, you might have noticed that the first commit also fixed a bug in master where it would use an image from disk even if the image was dirty.

Saving a copy of an image turns out to be surprisingly hard :(
Besides clearing the dirty flag, saving images would also turn generated images into from-files-on-disk images, so I've given up on using image.save() to save images.
Instead I'm now using image.save_render() which doesn't modify the image at all so it won't have any of those problems. It does use the render settings from a scene though, so this now creates a temp scene, sets all the settings we need, and then uses save_render(). See the commit message on 685e9ec for details.
(Doing it this way is slightly slower unfortunately; Sponza has gone to about 40s vs 38s before; the others I tested are about the same.)
The reason the tests are failing on 2.79 is that the test expects an RGB image and somewhere between 2.79 and 2.8 Blender learned to do
if you call img.save_render()
    if the render color mode is RGBA
        if img is an RGB image
            save the image as an RGB file instead of an RGBA file

which makes 2.8 pass, but since 2.79 doesn't know how to do this, the PNG files it saves when it saves a copy will be RGBA images instead of RGB. As far as I could tell, there is no field that exposes if the image data is RGB or RGBA on a Blender image, so I don't know how I could fix this. (Other than having an unnecessary A channel though, I don't think there's anything actually wrong with 2.79).
I'd really love any help writing _save_copy_of_image. Does anyone else have any ideas? Is there some way to ask what the correct way to save a copy of an image from Python is?

saving images would also turn generated images into from-files-on-disk images

You can get back to generative using bpy.data.images['Untitled'].source = 'GENERATED'

clearing the dirty flag

This is dirty too, but we can change pixels[0] to any value and back to original one, it will get back the  dirty flag

You can get back to generative using bpy.data.images['Untitled'].source = 'GENERATED'

You can't if the image was dirty though. When you set image.source = 'GENERATED' the image gets recreated from the generation parameters, which means any changes you painted on top of it would be lost.

This is dirty too, but we can change pixels[0] to any value and back to original one, it will get back the dirty flag

Yeah, just image.pixels[0] = image.pixels[0] works (though like I said before, it is pretty slow; seems anything touching pixels is).
The other problem is I don't know if this is an exhaustive list of things that are messed up by image.save().

@scurest Is image.save_render() only blocked by the 2.79 behavior described in your comment above?  We'll be dropping 2.79 soon, as you know.
The other thing I thought of is that the UI actually has something similar to what we want, saving an image as a copy from a menu item.  There's a matching Python operator:

bpy.ops.image.save_as(save_as_render=False, copy=True, filepath="/path/file.png", relative_path=True, show_multiview=False, use_multiview=False)

I know that addons are discouraged from using ops for stuff like this, but it means there's code in there somewhere (here? here?) for this, even if there doesn't seem to be a non-ops Python binding for it.
Let's not let version 2.79 hold us back if that's the only stumbling block on save_render though.

There's a matching Python operator

Yeah, I'd love to use it but I can't figure out how. Do you know?

Is image.save_render() only blocked by the 2.79 behavior described in your comment above?

AFAIK yes.
I'm not sure if the 2.79 behaviour should really be considered blocking. It's unfortunate that an alpha channel is added when it isn't needed, but it doesn't really hurt anything except the file size. This also only happens in what should be the uncommon case, when we can't find an existing file to use for the image (eg. it was packed from channels, it was dirty, it was generated).
If another way can't be found... If the test could be relaxed to only check if the images are pixel-for-pixel equal instead of checking if the files are byte-for-byte equal, it should pass on 2.79 too. (I think that would be a good idea anyway; checking if the PNGs files are byte-for-byte equal is too restrictive. For example, the tests will break even if all we do is change the compression setting for PNGs since the file will get encoded differently even though obviously the pixels don't change.)

Yeah, I'd love to use it but I can't figure out how. Do you know?

You have to override the context, see https://docs.blender.org/api/current/bpy.ops.html#overriding-context

There's a matching Python operator

Yeah, I'd love to use it but I can't figure out how. Do you know?

I haven't used these much but my understanding is that bpy.ops.* is tied specifically to user UI operations, such as saving an image from an Image Editor or UV Editor UI panel.  This may be a dead-end.

checking if the PNGs files are byte-for-byte equal is too restrictive

Yeah, I put in the byte-for-byte check when I was looking to test a "happy path" where the file was simply copied from the source, and then I probably reused it and never wrote a more detailed pixel-for-pixel check.  Some sort of relaxing of tests would be fine with me, we could even isolate this test to only Blender 2.8x and above, and skip it for 2.79.

@julienduroure Can you post the code for it?

@scurest Have a look here:
https://gist.github.com/julienduroure/83661b26fa5e1e442dcdd7873936a5a6
I didn't test in the context of addon exporting

@julienduroure Thanks a ton! Do you know how to control the file format?

Actually, you know what, sorry, I've dragged this PR off into the weeds. I didn't expect saving a copy of an image to be so hairy, but this case is not that important. We can just do
tmp_img = image.copy()
tmp_img.filepath/file_format = whatever
tmp_img.save()

which still avoids going through Python data like master has to.
The important thing here is the restructuring of ExportImage to hold a description of how to build the image instead of holding raw pixel data so that we can avoid decoding every single image to a Python tuple. That's plenty for one PR.
I'm going to close this and open a fresh PR with just that. Trying to figure out how to save a copy of an image can always be done separately. Sorry for the noise.


You should unblock ieji.de and koyu.space

Commit https://github.com/scafroglia93/blocklists/commit/e327ff50286980a457a94231184785c5dfbf222d
Having one bad account on a public platform does not justify a block against the platform itself, especially when the admins were not notified of this at publishing, and when they do become aware, the accounts have been removed.
cc. @solsocog @koyuawsmbrtn

Can confirm. Please contact me before putting my site on your blocklist or publishing news articles about it.

Deleted it as soon as I was notified of the abuse, I'd also welcome an attempt of contacting beforehand.

I am just going back to the reports made by the zsscaler team
I find the report very rude and in any case no one forces you to follow my work.
The problem has been solved; I advise you not to use my blocklists for the future
Thanks


Connector shows "connection refused" error if using only MQTT

When using MQTT, the sensor block turns off http but if the port 7575 is mapped in docker compose, a Connector block error will be shown:
 [inputs.http] Error in plugin: [url=http://sensor:7575]: Get "http://sensor:7575": dial tcp <IP_ADDRESS>:7575: connect: connection refused
This has no functional issues but may confuse some users. We could either switch balenaSense to only use MQTT or look into making adjustments to the Connector block. Either way, a low priority.

i have same issue but can't not access, jellyfish page


Investigate NetworkHandler

Why isn't it minirpclib?

If someone more experienced in networking would be willing to take a look at this, I'd appreciate it.


Export/import in another namespace the self signed certificate

Feature request
We should find a way to pass the self signed certificate to different namespaces and applications like Tekton as the task git cloning the repository fails and reports this error:
fatal: unable to access 'https://gitea.cnoe.localtest.me:8443/q-shift/my-quarkus-app-job.git/': SSL certificate problem: self signed certificate

Ideally, we should have a command or a controller able to to export the certificate and key like and import i in another namespace
Some ideas to explore:

https://external-secrets.io/latest/api/pushsecret/
https://github.com/zakkg3/ClusterSecret
Cert Manager can also import a TLS Secret in an ingress resource, etc
Tanzu uses this conteoller https://github.com/carvel-dev/secretgen-controller as it supports the features:

Supports generating certificates, passwords, RSA keys and SSH keys
Supports exporting and importing secrets across namespaces
Exporting/importing registry secrets across namespaces => https://github.com/carvel-dev/secretgen-controller/blob/develop/docs/secret-export.md
Supports generating secrets from data residing in other Kubernetes resources

This was one of the advantages of using cert manager. Cert manager allows you to replicate certificate to specified namespaces. This is very easy to do so I don't think we need cert manager to implement it.
Regardless of whether we go with cert manager or not, we will need a new flag in idpbuilder to specify which namespaces to sync certificates to. Or we can try and create certs in all namespaces but this is probably not a good idea.


help: I can not pnpm install @nuxt/devtools

📚 What are you trying to do?
If i use npm/pnpm install @nuxt/devtools, it will have error with this
.../node_modules/fast-folder-size postinstall$ node get-sysinternals-du.js │ du64.exe not found at D:\Test\courseManage-Nuxt\node_modules\.pnpm\registry.npmmirror.…   │ node:internal/url:611 │       throw new ERR_INVALID_URL(input); │       ^ │ TypeError [ERR_INVALID_URL]: Invalid URL │     at new NodeError (node:internal/errors:405:5) │     at new URL (node:internal/url:611:13) │     at new HttpsProxyAgent (D:\Test\courseManage-Nuxt\node_modules\.pnpm\registry.npmm…   │     at exports.downloadDuZip (D:\Test\courseManage-Nuxt\node_modules\.pnpm\registry.np…   │     at exports.default (D:\Test\courseManage-Nuxt\node_modules\.pnpm\registry.npmmirro…   │     at Object.<anonymous> (D:\Test\courseManage-Nuxt\node_modules\.pnpm\registry.npmmi…   │     at Module._compile (node:internal/modules/cjs/loader:1256:14) │     at Module._extensions..js (node:internal/modules/cjs/loader:1310:10) │     at Module.load (node:internal/modules/cjs/loader:1119:32) │     at Module._load (node:internal/modules/cjs/loader:960:12) { │   input: '<IP_ADDRESS>:7890', │   code: 'ERR_INVALID_URL' │ } │ Node.js v18.17.1 └─ Failed in 157ms at<EMAIL_ADDRESS>ELIFECYCLE  Command failed with exit code 1.
May there have some problems with the package 'fast-folder-size' ?
My environment is windows 10 and  Node.js v18.17.1.
🔍 What have you tried?
If i use without @nuxt/devtools , it can run success.
My package.json 👇
{   "name": "nuxt-app",   "private": true,   "scripts": {     "build": "nuxt build",     "dev": "nuxt dev",     "generate": "nuxt generate",     "preview": "nuxt preview",     "postinstall": "nuxt prepare"   },   "devDependencies": {     "nuxt": "^3.7.0"   } }
ℹ Additional context
No response

#409 resolved this issue, you can try reinstalling the latest version of @nuxt/devtools @2016WUJI01


Release should retry creating the pull request for updating go dependencies

As part of the release process we create PRs to bump released components versions.
Sometimes this process fails (although the PR might get created).
In this case we should have some retry logic (perhaps even try to force delete the PR, because sometimes it gets created in a screwed up state).

Some of the failures should be fixed by the recent dependency bumping rework (one PR per project), but it’s still possible for this to fail if go get fails for some reason so it would be useful to retry. (The failures addressed by the dependency rework wouldn’t be fixed by retries.)

I don't think we've seen this error, or any release issues, in quite a while. Unless we actually see the error, I'm not sure it's worth adding complexity to mitigate it.