Garbage Collector removes all tags

Description
I Activated Garbage Collector on Portus Background process with keep_latest: 5 and older_than: 100
But it deletes all images older_than 100 ignoring the keep_latest flag. In result I have old repositories wiped all completely
Steps to reproduce

Using Portus 2.4 with two processes: Portus and Portus Background
Enable Garbage Collector with following options:

  garbage_collector:
    enabled: true
    older_than: 100
    keep_latest: 5
    tag: ""

Restart Portus Background process
All images older than 100 days are deleted, keep_latest is ignored.

Expected behavior: I expected that an old repository it kept latest 5 tags
Actual behavior:  All images older than 100 days are deleted

Here initial logs:
[Initialization] Running: 'Registry events', 'Garbage collector'
[catalog] Removed the tag 'master'.
[catalog] Removed the image 'bsdash'.
[catalog] Removed the tag 'master'.
[catalog] Removed the image 'bsdash-gitlabreceiver'.
[catalog] Removed the tag '5.13'.
[catalog] Removed the tag '4.0.28'.
[catalog] Removed the tag '4.0.29'.
[catalog] Removed the tag '4.0.30'.
[catalog] Removed the tag '5.14'.
Handling 'delete' event:
{
  "id": "16748f9c-95cd-496d-ba29-2d46881f657b",
  "timestamp": "2019-10-14T17:04:02.347313389+02:00",
  "action": "delete",
  "target": {
    "digest": "sha256:2aa456083567cc6c63aafca541805fe551088abd347f32d5adca245ee3cc8100",
    "repository": "bs/bsdash"
  },
  "request": {
    "id": "2220cb85-5421-405f-8b0b-ade62cb5c504",
    "addr": "<IP_ADDRESS>:46500",
    "host": "registry.bscompany.eu:5000",
    "method": "DELETE",
    "useragent": "Ruby"
  },
  "actor": {
    "name": "portus"
  },
  "source": {
    "addr": "itportus01:5000",
    "instanceID": "5dfbe8ef-e81d-4ce6-84b6-470c720ea725"
  }
}

Deployment information
Deployment method: Portus is deployed as a standalone Container (not Compose) which connects to local MariaDB and Registry.
Configuration:
email:
  from<EMAIL_ADDRESS>  name: Portus
  reply_to: ''
  smtp:
    enabled: false
    address: smtp.example.com
    port: 587
    domain: example.com
    ssl_tls: ''
    enable_starttls_auto: false
    openssl_verify_mode: none
    ca_path: ''
    ca_file: ''
    user_name: ''
    password: "****"
    authentication: login
gravatar:
  enabled: true
delete:
  enabled: true
  contributors: false
  garbage_collector:
    enabled: false
    older_than: 30
    keep_latest: 5
    tag: ''
ldap:
  enabled: false
  hostname: ldap_hostname
  port: 389
  timeout: 5
  encryption:
    method: ''
    options:
      ca_file: ''
      ssl_version: TLSv1_2
  base: ''
  admin_base: ''
  group_base: ''
  filter: ''
  uid: uid
  authentication:
    enabled: false
    bind_dn: ''
    password: "****"
  group_sync:
    enabled: true
    default_role: viewer
  guess_email:
    enabled: false
    attr: ''
oauth:
  local_login:
    enabled: true
  google_oauth2:
    enabled: false
    id: ''
    secret: ''
    domain: ''
    options:
      hd: ''
  open_id:
    enabled: false
    identifier: ''
    domain: ''
  openid_connect:
    enabled: false
    issuer: ''
    identifier: ''
    secret: ''
  github:
    enabled: false
    client_id: ''
    client_secret: ''
    organization: ''
    team: ''
    domain: ''
  gitlab:
    enabled: false
    application_id: ''
    secret: ''
    group: ''
    domain: ''
    server: ''
  bitbucket:
    enabled: false
    key: ''
    secret: ''
    domain: ''
    options:
      team: ''
first_user_admin:
  enabled: true
signup:
  enabled: false
check_ssl_usage:
  enabled: true
registry:
  jwt_expiration_time:
    value: 15
  catalog_page:
    value: 100
  timeout:
    value: 2
  read_timeout:
    value: 120
machine_fqdn:
  value: portus.bscompany.eu
display_name:
  enabled: false
user_permission:
  change_visibility:
    enabled: true
  create_team:
    enabled: true
  manage_team:
    enabled: true
  create_namespace:
    enabled: true
  manage_namespace:
    enabled: true
  create_webhook:
    enabled: true
  manage_webhook:
    enabled: true
  push_images:
    policy: allow-teams
security:
  clair:
    server: ''
    health_port: 6061
    timeout: 900
  zypper:
    server: ''
  dummy:
    server: ''
anonymous_browsing:
  enabled: true
background:
  registry:
    enabled: true
  sync:
    enabled: true
    strategy: initial
pagination:
  per_page: 10
  before_after: 2

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952
env_portus: environment file used for customizing Portus Foreground:
PORTUS_MACHINE_FQDN_VALUE=portus.bscompany.eu
PORTUS_PUMA_HOST=<IP_ADDRESS>:3000

PORTUS_SECRET_KEY_BASE=***
PORTUS_KEY_PATH=/certificates/***
PORTUS_PASSWORD=***

CCONFIG_PREFIX=PORTUS

PORTUS_DB_ADAPTER=mysql2
PORTUS_DB_HOST=portus.host.local
PORTUS_DB_USERNAME=***
PORTUS_DB_PASSWORD=***
PORTUS_DB_DATABASE=portus_production
PORTUS_DB_POOL=5

RAILS_SERVE_STATIC_FILES=true

PORTUS_PUMA_TLS_KEY=/certificates/***
PORTUS_PUMA_TLS_CERT=/certificates/***

PORTUS_LOG_LEVEL=info

TZ=Europe/Rome

We are running portus with:
docker run -d --restart=always -v /opt/ssl:/certificates:ro -v /srv/portus/config/config.yml:/srv/Portus/config/config.yml \
       -p 3000:3000 --name portus --env-file=/srv/portus/config/env_portus opensuse/portus:2.4

env_background: environment file used for customizing Portus Background:
PORTUS_MACHINE_FQDN_VALUE=portus.bscompany.eu
PORTUS_PUMA_HOST=<IP_ADDRESS>:3000
PORTUS_BACKGROUND=true

PORTUS_SECRET_KEY_BASE=***
PORTUS_KEY_PATH=/certificates/***
PORTUS_PASSWORD=***

CCONFIG_PREFIX=PORTUS

PORTUS_DB_ADAPTER=mysql2
PORTUS_DB_HOST=portus.host.local
PORTUS_DB_USERNAME=***
PORTUS_DB_PASSWORD=***
PORTUS_DB_DATABASE=portus_production
PORTUS_DB_POOL=5

RAILS_SERVE_STATIC_FILES=true

PORTUS_PUMA_TLS_KEY=/certificates/***
PORTUS_PUMA_TLS_CERT=/certificates/***

PORTUS_LOG_LEVEL=info

TZ=Europe/Rome

Then we are running portus background:
docker run -d --restart=always -v /opt/ssl:/certificates:ro -v /srv/portus/config/config.yml:/srv/Portus/config/config.yml \
       --name portus_background --env-file=/srv/portus/config/env_background opensuse/portus:2.4

Thanks in advance
Roberto

This is affecting us very badly.  any update on this?

Description
I Activated Garbage Collector on Portus Background process with keep_latest: 5 and older_than: 100
But it deletes all images older_than 100 ignoring the keep_latest flag. In result I have old repositories wiped all completely
Steps to reproduce
1. Using Portus 2.4 with two processes: Portus and Portus Background

2. Enable Garbage Collector with following options:

  garbage_collector:
    enabled: true
    older_than: 100
    keep_latest: 5
    tag: ""

1. Restart Portus Background process

2. All images older than 100 days are deleted, keep_latest is ignored.

* **Expected behavior**: I expected that an old repository it kept latest 5 tags

* **Actual behavior**:  All images older than 100 days are deleted

Here initial logs:
[Initialization] Running: 'Registry events', 'Garbage collector'
[catalog] Removed the tag 'master'.
[catalog] Removed the image 'bsdash'.
[catalog] Removed the tag 'master'.
[catalog] Removed the image 'bsdash-gitlabreceiver'.
[catalog] Removed the tag '5.13'.
[catalog] Removed the tag '4.0.28'.
[catalog] Removed the tag '4.0.29'.
[catalog] Removed the tag '4.0.30'.
[catalog] Removed the tag '5.14'.
Handling 'delete' event:
{
  "id": "16748f9c-95cd-496d-ba29-2d46881f657b",
  "timestamp": "2019-10-14T17:04:02.347313389+02:00",
  "action": "delete",
  "target": {
    "digest": "sha256:2aa456083567cc6c63aafca541805fe551088abd347f32d5adca245ee3cc8100",
    "repository": "bs/bsdash"
  },
  "request": {
    "id": "2220cb85-5421-405f-8b0b-ade62cb5c504",
    "addr": "<IP_ADDRESS>:46500",
    "host": "registry.bscompany.eu:5000",
    "method": "DELETE",
    "useragent": "Ruby"
  },
  "actor": {
    "name": "portus"
  },
  "source": {
    "addr": "itportus01:5000",
    "instanceID": "5dfbe8ef-e81d-4ce6-84b6-470c720ea725"
  }
}

Deployment information
Deployment method: Portus is deployed as a standalone Container (not Compose) which connects to local MariaDB and Registry.
Configuration:
email:
  from<EMAIL_ADDRESS>  name: Portus
  reply_to: ''
  smtp:
    enabled: false
    address: smtp.example.com
    port: 587
    domain: example.com
    ssl_tls: ''
    enable_starttls_auto: false
    openssl_verify_mode: none
    ca_path: ''
    ca_file: ''
    user_name: ''
    password: "****"
    authentication: login
gravatar:
  enabled: true
delete:
  enabled: true
  contributors: false
  garbage_collector:
    enabled: false
    older_than: 30
    keep_latest: 5
    tag: ''
ldap:
  enabled: false
  hostname: ldap_hostname
  port: 389
  timeout: 5
  encryption:
    method: ''
    options:
      ca_file: ''
      ssl_version: TLSv1_2
  base: ''
  admin_base: ''
  group_base: ''
  filter: ''
  uid: uid
  authentication:
    enabled: false
    bind_dn: ''
    password: "****"
  group_sync:
    enabled: true
    default_role: viewer
  guess_email:
    enabled: false
    attr: ''
oauth:
  local_login:
    enabled: true
  google_oauth2:
    enabled: false
    id: ''
    secret: ''
    domain: ''
    options:
      hd: ''
  open_id:
    enabled: false
    identifier: ''
    domain: ''
  openid_connect:
    enabled: false
    issuer: ''
    identifier: ''
    secret: ''
  github:
    enabled: false
    client_id: ''
    client_secret: ''
    organization: ''
    team: ''
    domain: ''
  gitlab:
    enabled: false
    application_id: ''
    secret: ''
    group: ''
    domain: ''
    server: ''
  bitbucket:
    enabled: false
    key: ''
    secret: ''
    domain: ''
    options:
      team: ''
first_user_admin:
  enabled: true
signup:
  enabled: false
check_ssl_usage:
  enabled: true
registry:
  jwt_expiration_time:
    value: 15
  catalog_page:
    value: 100
  timeout:
    value: 2
  read_timeout:
    value: 120
machine_fqdn:
  value: portus.bscompany.eu
display_name:
  enabled: false
user_permission:
  change_visibility:
    enabled: true
  create_team:
    enabled: true
  manage_team:
    enabled: true
  create_namespace:
    enabled: true
  manage_namespace:
    enabled: true
  create_webhook:
    enabled: true
  manage_webhook:
    enabled: true
  push_images:
    policy: allow-teams
security:
  clair:
    server: ''
    health_port: 6061
    timeout: 900
  zypper:
    server: ''
  dummy:
    server: ''
anonymous_browsing:
  enabled: true
background:
  registry:
    enabled: true
  sync:
    enabled: true
    strategy: initial
pagination:
  per_page: 10
  before_after: 2

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952
Thanks in advance
Roberto

Hi @robgiovanardi ?I have a feeling ,but no more time just wanted to give you an improtant tip :

Is the PORTUS_BACKGROUND=true environement variable set ?
It is improbable, that you can operate inproduction a portus, with one standalone container : the documentation will confirm you that a second container, at least, is required. They call it the background. At that's why docker-compose at least is natural for a prod deployment.

Hope this will help

Hi @Jean-Baptiste-Lasselle
thanks for your answer. I confirm we have two containers:
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                    NAMES
96e14d4670ad        opensuse/portus:2.4   "/init"             7 months ago        Up 2 weeks          3000/tcp                 portus_background
e23352be8846        opensuse/portus:2.4   "/init"             7 months ago        Up 4 weeks          <IP_ADDRESS>:3000->3000/tcp   portus

Only portus_backgound container has PORTUS_BACKGROUND=true set and this is the one container with garbage_collector feature enabled:
  garbage_collector:
    enabled: true
    older_than: 100
    keep_latest: 5
    tag: ""

So, yes, we are running garbage collection on Portus Background dedicated container and the problem is still here..

We also have the same problem and can confirm that the keep_latest setting doesn’t work.

Hi @Jean-Baptiste-Lasselle
thanks for your answer. I confirm we have two containers:
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                    NAMES
96e14d4670ad        opensuse/portus:2.4   "/init"             7 months ago        Up 2 weeks          3000/tcp                 portus_background
e23352be8846        opensuse/portus:2.4   "/init"             7 months ago        Up 4 weeks          <IP_ADDRESS>:3000->3000/tcp   portus

Only portus_backgound container has PORTUS_BACKGROUND=true set and this is the one container with garbage_collector feature enabled:
  garbage_collector:
    enabled: true
    older_than: 100
    keep_latest: 5
    tag: ""

So, yes, we are running garbage collection on Portus Background dedicated container and the problem is still here..

hi @robgiovanardi thank you so much for ur feedback, i mean it's very interesting, i haven't tested the feature yet, but it's at least very important business case.
But i can try n help you :

next step : how is communication established between ur registry and portus_background containers ?  I wanna ask here, because both of your portus and portus_background are using same port number
and maybe that's why you went standalone, so that port conflict does not blow everything. Plus, if tried to access "from the outside", your portus_background is unreachable, it's always your portus container who gets the requests on to 3000. Is it why?

@robgiovanardi all in all, since some images (all of them) are deleted, the communication should be ok, i think you've got your hands on a real bug inside portus' code here. Still, you config is odd with those redundant prot numbers, i think it's worth checking.
I have antoher idea, too : for your portus container (not the portus_background) , do you have any occurrence of the string garbage_collector  ?

Hi @Jean-Baptiste-Lasselle
thanks for your answer. I confirm we have two containers:
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                    NAMES
96e14d4670ad        opensuse/portus:2.4   "/init"             7 months ago        Up 2 weeks          3000/tcp                 portus_background
e23352be8846        opensuse/portus:2.4   "/init"             7 months ago        Up 4 weeks          <IP_ADDRESS>:3000->3000/tcp   portus

Only portus_backgound container has PORTUS_BACKGROUND=true set and this is the one container with garbage_collector feature enabled:
  garbage_collector:
    enabled: true
    older_than: 100
    keep_latest: 5
    tag: ""

So, yes, we are running garbage collection on Portus Background dedicated container and the problem is still here..

hi @robgiovanardi thank you so much for ur feedback, i mean it's very interesting, i haven't tested the feature yet, but it's at least very important business case.
But i can try n help you :

next step : how is communication established between ur registry and portus_background containers ?  I wanna ask here, because both of your portus and portus_background are using same port number
portus foreground is externally facing, so the 3000 port exposed; portus_background has no exposed port because it should communicate with registry, which is located on the docker host.
and maybe that's why you went standalone, so that port conflict does not blow everything. Plus, if tried to access "from the outside", your portus_background is unreachable, it's always your portus container who gets the requests on to 3000. Is it why?
I supposed portus_background doesn't need to be externally accessible. Am I wrong?

I forgot to mention environment variable used for background and foreground, I'm adding that to Description

@robgiovanardi all in all, since some images (all of them) are deleted, the communication should be ok, i think you've got your hands on a real bug inside portus' code here. Still, you config is odd with those redundant port numbers, i think it's worth checking.
I have another idea, too : for your portus container (not the portus_background) , do you have any occurrence of the string garbage_collector in the config file /srv/Portus/config/config.yml ?

Actually, yes, because we are using the very same config.yaml for both foreground and background, and running them with different environment variables, I just updated initial description including those envs.
Let me try a different config.yml

background

Excellent news @robgiovanardi  !!!! Indeed, my idea was that the comunication between your registry and the portus_background actually never happens, and here is what it involves:

There is one hugedifference (extremely important) between portus container and the portus_background container : the portus_background has the - PORTUS_BACKGROUND=true  environement variable. The portus container hasnot, (and must not) have that environmentvariable value set to true,i think it defaults to false.
So okay : that a massive difference, it makes what runs into those two contaners just as different as windows and linux.
And that 'swhy it is SO importantthat youmake sure that the registry communicates with the portus_background : the docker-compose  ad deployment examples in the portus project for that sake, are awfully misleading, using same (network) identities for completely different services, for example.
So her is what I think : your no.1 prirority is to make sure communication happens between your registry and the portus_background
I have a question : reading how you speak about it, I think it's possible the private docker registry which communicates with your portus, was already there long before you tried portus, am I right? If yes,then you confirm this private docker registry you wan to operate is not in the docker-compose.yml (which has portus inside) ?

One final amusement remark : The examples in the portus distrib are awful..,And it's funny I heave a feeling thy were kind of teared off a legacy docker swarm cluster... I Might be wrong. Or not . :)
Anyway,I actually understand the point of view of OpenSUSE, and I am here to support community, because we are going to make portus work together, and that is great(ly strategic in the container planet). And I am so thanking OpenSUSE guys for what they dropped us. Plus i started years ago because of OpenStack.
OpenSUSE guys will understand the message.

@robgiovanardi all in all, since some images (all of them) are deleted, the communication should be ok, i think you've got your hands on a real bug inside portus' code here. Still, you config is odd with those redundant port numbers, i think it's worth checking.
I have another idea, too : for your portus container (not the portus_background) , do you have any occurrence of the string garbage_collector in the config file /srv/Portus/config/config.yml ?

Actually, yes, because we are using the very same config.yaml for both foreground and background, and running them with different environment variables, I just updated initial description including those envs.
Let me try a different config.yml

yes, doyour thing with the config.yml :

make sure the , I'm almost sure the garbage_collector config is present for the portus_background,
and remove it completely from the portus container's config.yml
then I'm almost sure of whats gonna happen now : you wil try and tyry again to geta garbage collection,but it won't ever happen anymore. (test with images you don't care to delete, only them with the tag parameter)

having accurate results on that test, will help me a lot. Still is sure that we will have to ironize your network setup, so it's more explicitly telling the operator who 's talking to who and for what purpose. You security guys will like that too.

background

Excellent news @robgiovanardi !!!! Indeed, my idea was that the comunication between your registry and the portus_background actually never happens, and here is what it involves:

There is one hugedifference (extremely important) between portus container and the portus_background container : the portus_background has the - PORTUS_BACKGROUND=true  environement variable. The portus container hasnot, (and must not) have that environmentvariable value set to true,i think it defaults to false.
So okay : that a massive difference, it makes what runs into those two contaners just as different as windows and linux.
And that 'swhy it is SO importantthat youmake sure that the registry communicates with the portus_background : the docker-compose  ad deployment examples in the portus project for that sake, are awfully misleading, using same (network) identities for completely different services, for example.
So her is what I think : your no.1 prirority is to make sure communication happens between your registry and the portus_background

Let me better understand: communication would happens from portus_background to registry, right? Not from registry to portus_background?
If this is true, then i can boot up a portus_background without externally facing port but still network: now the portus_background can queries mysqldb, find the registry network settings (attached screenshot, in this case, this one: )

and then portus_background can do anything he want with the registry

I have a question : reading how you speak about it, I think it's possible the private docker registry which communicates with your portus, was already there long before you tried portus, am I right? If yes,then you confirm this private docker registry you wan to operate is not in the docker-compose.yml (which has portus inside) ?

Yes, you right, the registry is installed as legacy application, with zypper, not deployed with docker nor docker-compose

One final amusement remark : The examples in the portus distrib are awful..,And it's funny I heave a feeling thy were kind of teared off a legacy docker swarm cluster... I Might be wrong. Or not . :)
Anyway,I actually understand the point of view of OpenSUSE, and I am here to support community, because we are going to make portus work together, and that is great(ly strategic in the container planet). And I am so thanking OpenSUSE guys for what they dropped us. Plus i started years ago because of OpenStack.
OpenSUSE guys will understand the message.

Hi @Jean-Baptiste-Lasselle
I can confirm that the problem is still present:

completely removed garbage entries from config.yml for portus:

  #garbage_collector:
  #  enabled: true

  #  # Remove images not pulled and older than a specific value. This value is
  #  # interpreted as the number of days.
  #  #
  #  # e.g.: If an image wasn't pulled in the latest 30 days and the image wasn't
  #  # updated somehow in the latest 30 days, the image will be deleted.
  #  older_than: 30

  #  # Keep the latest X images regardless if it's older than the value set in
  #  # `older_than` configuration.
  #  keep_latest: 15

  #  # Provide a string containing a regular expression. If you provide a
  #  # valid regular expression, garbage collector will only be applied into tags
  #  # matching a given name.
  #  #
  #  # Valid values might be:
  #  #   - "jenkins": if you anticipate that you will always have a tag with a
  #  #     specific name, you can simply use that.
  #  #   - "build-\\d+": your tag follows a format like "build-1234" (note that
  #  #     we need to specify "\\d" and not just "\d").
  #  tag: ""

started portus_background with garbage enabled and with an external port:

docker run -d --restart=always -v /opt/ssl:/certificates:ro -v /srv/portus/config/config_background.yml:/srv/Portus/config/config.yml -p3001:3000 --name portus_background --env-file=/srv/portus/config/env_background opensuse/portus:2.4

The problem is still there
Things to notice: I enabled debug mode on portus_background: it drop this query to find if there's some images to delete:
(0.2ms)  SELECT COUNT(*) FROM `tags` WHERE `tags`.`marked` = 0 AND (updated_at < '2019-11-18 10:07:00.047409')

I used older_than: 30 garbage configuration, so why the updated_at time 2019-11-18. But i can't see other logs that indicate the keep_latest
Some debug logs
   (0.4ms)  SELECT COUNT(*) FROM `tags` WHERE `tags`.`marked` = 0 AND (updated_at < '2019-11-18 10:03:08.231707')
  User Load (0.5ms)  SELECT  `users`.* FROM `users` WHERE `users`.`username` = 'portus' LIMIT 1
  Tag Load (0.7ms)  SELECT `tags`.* FROM `tags` WHERE `tags`.`marked` = 0 AND (updated_at < '2019-11-18 10:03:08.231707')
  Repository Load (0.3ms)  SELECT  `repositories`.* FROM `repositories` WHERE `repositories`.`id` = 1 LIMIT 1
  SQL (1.5ms)  UPDATE `tags` SET `tags`.`marked` = 1 WHERE `tags`.`digest` = 'sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a' AND `tags`.`repository_id` = 1          Registry Load (0.3ms)  SELECT  `registries`.* FROM `registries`  ORDER BY `registries`.`id` ASC LIMIT 1
  Namespace Load (0.4ms)  SELECT  `namespaces`.* FROM `namespaces` WHERE `namespaces`.`id` = 3 LIMIT 1
  Tag Load (1.1ms)  SELECT  `tags`.* FROM `tags` WHERE `tags`.`digest` = 'sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a' AND `tags`.`repository_id` = 1  ORDER BY `tags`.`id` ASC LIMIT 1000
[catalog] Removed the tag 'latest'.
  Tag Load (0.2ms)  SELECT  `tags`.* FROM `tags` WHERE `tags`.`id` = 1 LIMIT 1
   (0.1ms)  BEGIN
  ScanResult Load (0.2ms)  SELECT `scan_results`.* FROM `scan_results` WHERE `scan_results`.`tag_id` = 1
  SQL (0.6ms)  DELETE FROM `tags` WHERE `tags`.`id` = 1
   (4.6ms)  COMMIT

with tag_load it select tags marked = 0 updated 30 days ago; then at the end it run a DELETE FROM tags where tags.id =1. Seems no keep_latest checks are done at all

hi @robgiovanardi  :
Communication between registry and background_portus

Let me better understand: communication would happens from portus_background to registry,
right? Not from registry to portus_background?

Absolutely, accurately;
About checking if your background can reach registry (at least, not even database)
Your portus webpage screenshot :

Does not make think that your background_portus can reach the registry. Do you have any reason to think otherwise (actually asking, maybe i'm forgetting about something) ? To me , it could just be the portus container being able to query the portus database, why else would we have such a configuration for the portus (if not to reack database) :
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5

About your background_container

On the machine on which you execute the command :

docker run -d --restart=always -v /opt/ssl:/certificates:ro -v /srv/portus/config/config_background.yml:/srv/Portus/config/config.yml -p3001:3000 --name portus_background --env-file=/srv/portus/config/env_background opensuse/portus:2.4

You have created a directory with path /srv/Portus/config/env_background ?  I mean the path just looks like a path inside, not outside a container.

regardless from this path question, about --env-file, I would need to have a look at the content of  /srv/Portus/config/env_background

last thing : if you wanna exchange by email, to quickly discuss some details but not in public,
jean.baptiste.lasselleATgmail.com

hi @robgiovanardi just to keep you informed, I am totally working on the matter, still haven't finished complete automation, and I today found something that sounds very good, in relation with garbage collection :  https://github.com/SUSE/Portus/issues/2275#issuecomment-586648423
What those
# Sync config
- PORTUS_BACKGROUND_REGISTRY_ENABLED=true
- PORTUS_BACKGROUND_SYNC_ENABLED=true
- PORTUS_BACKGROUND_SYNC_STRATEGY=update

made me think ... Maybe there exists things like

PORTUS_BACKGROUND_GARBAGE_COLLECTION_ENABLED ,
PORTUS_BACKGROUND_GC_ENABLED=true # GC for Garbage Collection
PORTUS_BACKGROUND_GC_OLDER_THAN=100
PORTUS_BACKGROUND_GC_KEEP_LATEST=5

@robgiovanardi I think I found your solution!!!!, and oh my god, the idea I just wrote  5minutes ago, inspred by https://github.com/SUSE/Portus/issues/2275#issuecomment-577829167 , gave full award !!!!
have a look out there : https://github.com/Ashtonian/server-setup/blob/bc9ac031a18f1c686da5a662d3cf969009a50c38/portus/docker-compose.yml
So Yesssss! there exist PORTUS_BACKGROUND_GARBAGE_COLLECTION_XXXX variables to activate and configure Garbage collection !!!  :D :D :D thank you sooo much  @kylegoetz
And so, what you need to do, is to ad the following env. variables to both your background and your portus services in docker-compose.yml
      - PORTUS_DELETE_ENABLED=true
      - PORTUS_DELETE_CONTRIBUTORS=false
      - PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
      - PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=30
      - PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5

Honestly, I'll try as soon aspossible to set that only for the background, just to check if it works, cause there a copy-paste pattern smell around this example... ^^
I have a lot of other  work on Portus, so I can't do that this weekend, I am so dying that you do that and give me feedback even before I run it :smile:
The whole docker-compose.yml, so we don't lose it
I found it seraching github with string PORTUS_BACKGROUND_REGISTRY_ENABLED, and got only 4 results in code, in the whole of github.com as of 15/02/2020!!
Even funnier, :laughing: , none of those 4 results are in portus documentation
version: "3.7"

services:
  portus:
    image: opensuse/portus:2.4.3
    # env_file:
    #   - ./portus.env
    environment:
      - PORTUS_MACHINE_FQDN_VALUE=portus.ashlab.dev
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5
      - PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
      - PORTUS_KEY_PATH=/certificates/portus.ashlab.dev/privatekey.key
      - PORTUS_PASSWORD=${PORTUS_PASSWORD}
      - PORTUS_CHECK_SSL_USAGE_ENABLED=false

      - PORTUS_SIGNUP_ENABLED=false
      - RAILS_SERVE_STATIC_FILES=true

      - PORTUS_GRAVATAR_ENABLED=true
      - PORTUS_DELETE_ENABLED=true
      - PORTUS_DELETE_CONTRIBUTORS=false
      - PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
      - PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=30
      - PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5
      - PORTUS_ANONYMOUS_BROWSING_ENABLED=false

      - PORTUS_OAUTH_GITHUB_ENABLED=true
      - PORTUS_OAUTH_GITHUB_CLIENT_ID=${PORTUS_OAUTH_GITHUB_CLIENT_ID}
      - PORTUS_OAUTH_GITHUB_CLIENT_SECRET=${PORTUS_OAUTH_GITHUB_CLIENT_SECRET}
      - PORTUS_OAUTH_GITHUB_ORGANIZATION=karsto
      # # - PORTUS_OAUTH_GITHUB_TEAM=''
      # # - PORTUS_OAUTH_GITHUB_DOMAIN=''

      #       - PORTUS_SECURITY_CLAIR_SERVER=http://clair:6060
    # ports:
    #   - 3000:3000
    depends_on:
      - db
    links:
      - db
    volumes:
      - traefik_certs_raw:/certificates:ro
      # - secrets:/certificates:ro
    networks:
      - portus
      - public
    labels:
      - "traefik.enable=true"
      # - "traefik.http.middlewares.sslHeaders.headers.SSLHost=portus.ashlab.dev"
      - "traefik.http.routers.portus.rule=Host(`portus.ashlab.dev`)"
      - "traefik.http.routers.portus.middlewares=https_redirect, sslHeaders"
      - "traefik.http.routers.portus.service=portus"
      - "traefik.http.routers.portus.tls=true"
      - "traefik.http.routers.portus.tls.certresolver=le"
      - "traefik.http.services.portus.loadbalancer.server.port=3000"
      - "traefik.http.services.portus.loadbalancer.server.scheme=http"
      - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
      - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
      # - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
      # - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
      # - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
      # - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
      # - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
    deploy:
      labels:
        - "traefik.enable=true"
        # - "traefik.http.middlewares.sslHeaders.headers.SSLHost=portus.ashlab.dev"
        - "traefik.http.routers.portus.rule=Host(`portus.ashlab.dev`)"
        - "traefik.http.routers.portus.middlewares=https_redirect, sslHeaders"
        - "traefik.http.routers.portus.service=portus"
        - "traefik.http.routers.portus.tls=true"
        - "traefik.http.routers.portus.tls.certresolver=le"
        - "traefik.http.services.portus.loadbalancer.server.port=3000"
        - "traefik.http.services.portus.loadbalancer.server.scheme=http"
        # - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
        # - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
        # - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
        # - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
        # - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
        # - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
        # - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
        # - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
        # - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
        # - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
  background:
    image: opensuse/portus:2.4.3
    depends_on:
      - portus
      - db
    environment:
      # Theoretically not needed, but cconfig's been buggy on this...
      - CCONFIG_PREFIX=PORTUS
      - PORTUS_MACHINE_FQDN_VALUE=portus.ashlab.dev
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5
      - PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
      - PORTUS_KEY_PATH=/certificates/portus.ashlab.dev/privatekey.key
      - PORTUS_PASSWORD=${PORTUS_PASSWORD}
      #       - PORTUS_SECURITY_CLAIR_SERVER=http://clair:6060
      # - PORTUS_CHECK_SSL_USAGE_ENABLED=false
      - PORTUS_GRAVATAR_ENABLED=true
      - PORTUS_DELETE_ENABLED=true
      - PORTUS_DELETE_CONTRIBUTORS=false
      - PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
      - PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=30
      - PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5

      - PORTUS_OAUTH_GITHUB_ENABLED=true
      - PORTUS_OAUTH_GITHUB_CLIENT_ID=${PORTUS_OAUTH_GITHUB_CLIENT_ID}
      - PORTUS_OAUTH_GITHUB_CLIENT_SECRET=${PORTUS_OAUTH_GITHUB_CLIENT_SECRET}
      - PORTUS_OAUTH_GITHUB_ORGANIZATION=karsto
      # - PORTUS_OAUTH_GITHUB_TEAM=''
      # - PORTUS_OAUTH_GITHUB_DOMAIN=''
      - PORTUS_ANONYMOUS_BROWSING_ENABLED=false

      - PORTUS_BACKGROUND=true
      - PORTUS_BACKGROUND_REGISTRY_ENABLED=true
      - PORTUS_BACKGROUND_SYNC_ENABLED=true
      - PORTUS_BACKGROUND_SYNC_STRATEGY=update-delete
    links:
      - db
    # env_file:
    #   - ./portus.env
    volumes:
      - traefik_certs_raw:/certificates:ro
    networks:
      - portus

  db:
    image: library/mariadb:10.0.33
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
    # env_file:
    #   - ./portus.env
    environment:
      - MYSQL_DATABASE=portus_production
      - MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
    volumes:
      - mariadb:/var/lib/mysql
    networks:
      - portus

  # clair: TODO:
  #   image: quay.io/coreos/clair
  #   restart: unless-stopped
  #   depends_on:
  #     - postgres
  #   links:
  #     - postgres
  #     - portus
  #   ports:
  #     - "6060-6061:6060-6061"
  #   volumes:
  #     - /tmp:/tmp
  #     - ./clair/clair.yml:/clair.yml
  #   command: [-config, /clair.yml]

  registry:
    image: library/registry:2.7.1
    # env_file:
    #   - ./portus.env
    environment:
      # REGISTRY_HTTP_ADDR: registry.ashlab.dev
      # Authentication
      REGISTRY_AUTH_TOKEN_REALM: https://portus.ashlab.dev/v2/token
      REGISTRY_AUTH_TOKEN_SERVICE: registry.ashlab.dev
      REGISTRY_AUTH_TOKEN_ISSUER: portus.ashlab.dev
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certificates/portus.ashlab.dev/certificate.crt

      # Portus endpoint
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://portus.ashlab.dev/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s
    volumes:
      - traefik_certs_raw:/certificates:ro
      - registry:/var/lib/registry
      - secrets:/secrets:ro
      - ./config.yml:/etc/docker/registry/config.yml:ro
    ports:
      # - 5000:5000
      - 5001:5001 # required to access debug service
    links:
      - portus:portus
    networks:
      - portus
      - public
    labels:
      - "traefik.enable=true"
      # - "traefik.http.middlewares.sslHeaders.headers.SSLHost=registry.ashlab.dev"
      - "traefik.http.routers.registry.rule=Host(`registry.ashlab.dev`)"
      - "traefik.http.routers.registry.middlewares=https_redirect, sslHeaders"
      - "traefik.http.routers.registry.service=registry"
      - "traefik.http.routers.registry.tls=true"
      - "traefik.http.routers.registry.tls.certresolver=le"
      - "traefik.http.services.registry.loadbalancer.server.port=5000"
      - "traefik.http.services.registry.loadbalancer.server.scheme=http"
      # - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
      # - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
      # - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
      # - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
      # - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
      # - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
      # - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"
    deploy:
      labels:
      - "traefik.enable=true"
      # - "traefik.http.middlewares.sslHeaders.headers.SSLHost=registry.ashlab.dev"
      - "traefik.http.routers.registry.rule=Host(`registry.ashlab.dev`)"
      - "traefik.http.routers.registry.middlewares=https_redirect, sslHeaders"
      - "traefik.http.routers.registry.service=registry"
      - "traefik.http.routers.registry.tls=true"
      - "traefik.http.routers.registry.tls.certresolver=le"
      - "traefik.http.services.registry.loadbalancer.server.port=5000"
      - "traefik.http.services.registry.loadbalancer.server.scheme=http"
      # - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https" # Standard move to default when traefik fixes behavior
      # - "traefik.http.middlewares.https_redirect.redirectscheme.permanent=true"
      # - "traefik.http.middlewares.sslHeaders.headers.framedeny=true"
      # - "traefik.http.middlewares.sslHeaders.headers.sslredirect=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSSeconds=315360000"
      # - "traefik.http.middlewares.sslHeaders.headers.browserXSSFilter=true"
      # - "traefik.http.middlewares.sslHeaders.headers.contentTypeNosniff=true"
      # - "traefik.http.middlewares.sslHeaders.headers.forceSTSHeader=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSIncludeSubdomains=true"
      # - "traefik.http.middlewares.sslHeaders.headers.STSPreload=true"

volumes:
  secrets:
    driver: local
    driver_opts:
      type: "none"
      o: "bind,rw"
      device: "/mnt/workspace/portus/secrets"
  traefik_certs_raw:
    driver: local
    driver_opts:
      type: "none"
      o: "bind,ro"
      device: "/mnt/workspace/traefik_certs_raw/"

  mariadb:
  registry:

networks:
  public:
    external: true
  portus:

@robgiovanardi so apply environment variables I gave you, inserting them into your env_background file

@Jean-Baptiste-Lasselle Hey - is there a reason why the latest garbage collection code (https://github.com/SUSE/Portus/commit/d8470717b228d56a780bb50983dea9b2d8577adb) isn't in the 2.4.3 release? As far as I can tell the release was made in May, but this code was merged in January? This is a pretty crucial patch. Can we get a 2.4.4 release?

release

hi Matt @diranged , Actually I am not an OpenSUSE engineer,or am not 'yet?) part of the official portus support or dev team. So I can't make release. Looks like you're gonna have to make a personal release n your infrastructure, building portus from source (tagged 2.4.4-private-release?)
I think there 's a docker image portus:2.5, though there is no 2.5 yet

Hi @Jean-Baptiste-Lasselle thanks for your help.
About environment variables, you probably read the doc: http://port.us.org/docs/Configuring-Portus.html:

In Portus we follow a naming convention for environment variables: first of all we have the PORTUS_ prefix, and then we add each key in uppercase. So, for example, the previous example can be tweaked by setting: PORTUS_FEATURE_ENABLED and PORTUS_FEATURE_VALUE

So using
- PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=5

is equal to use:
delete:
  garbage_collector:
    keep_latest: 5

Anyway I tested the env var but nothing changed: keep_latest was ignored and all tags was deleted.
@diranged You're right, that fundamental commit isn't in the 2.4.3 release.. I'll try to build the latest release and coming back for a feedback.

ignored

@robgiovanardi @diranged so thank you both of you I think you are right, the keep_latest shallwork only when we get a release with that commit meaning until then, we have to make a build from source to benefit that feature.
@robgiovanardi so thank you about :

In Portus we follow a naming convention for environment variables: first of all we have the PORTUS_ prefix, and then we add each key in uppercase. So, for example, the previous example can be tweaked by setting: PORTUS_FEATURE_ENABLED and PORTUS_FEATURE_VALUE

So ok, we can infer which env. variables to use, fromthe config files descriptions :

here : http://port.us.org/docs/Configuring-Portus.html#delete-support
and here :  http://port.us.org/features/removing_images.html#automating-the-removal-of-images-and-tags

hi @robgiovanardi Did you try and use portus:2.5 docker image, instead of building portus from source, to see if you get the keep_latest feature ?
I mean :

there is a portus:2.5 docker image, on docker hub
there is no 2.5;* release in https://github.com/SUSE/Portus/releases
So maybe it's the whole point of portus:2.5 docker image, on docker hub , to make that feature early access (though it's now a quite old, "early-access")

Hi @Jean-Baptiste-Lasselle I still have no time to do my tests, but thanks for points me out that docker release. This will speed up my tests

Hi @Jean-Baptiste-Lasselle I still have no time to do my tests, but thanks for points me out that docker release. This will speed up my tests

Very interesting test automation case though :
Test Suite Bundle 1: short-term GC

Suite 1: without tag regexp
Suite 2: with tag regexp dronie-*

Suite 1

test ENV :

PORTUS_DELETE_ENABLED=false : the test starts with 5 days, during the first 3 days, any deletion forbidden, and  we push oci container images everyday for 5 days. at day 4 at midnight plus one minute, we set PORTUS_DELETE_ENABLED=true and restart the portus and background services. So garbage collection should start at day 6, midnight, plus one minute. And always keep last 3 tags for any repository (OCI container image).
PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=2
PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=3
PORTUS_DELETE_GARBAGE_COLLECTOR_TAG=''

test-setup :

day 0 :

registry + portus provisioning.

also, create 2 users, the first super admin, and another beeio. Create team beebee, add beeio to beebee, and create the hive namespace for the beebee team. Finally creating a token named buzz, for the beeio user. All this using Portus API

pushing 3 images every day, for 5 days, from day 0 using beeio username and its buzz  token

day 0 :

export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE

### NODE ON ALPINE

docker pull node:8-alpine
docker tag node:8-alpine $OCI_SERVICE/hive/node:8-alpine
# docker logged-in
docker push $OCI_SERVICE/hive/node:8-alpine

### HELM ON ALPINE

docker pull alpine/helm:3.1.1
docker tag alpine/helm:3.1.1-alpine $OCI_SERVICE/hive/helm:3.1.1-alpine
docker push $OCI_SERVICE/hive/helm:3.1.1-alpine

### ATLANTIS TERRAGRUNT

docker pull exositebot/atlantis-terragrunt:version-1.4.1
docker tag exositebot/atlantis-terragrunt:version-1.4.1 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.1
docker push $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.1

* day 1 :

export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in

### NODE ON ALPINE

docker pull node:9-alpine
docker tag node:9-alpine $OCI_SERVICE/hive/node:9-alpine
docker push $OCI_SERVICE/hive/node:9-alpine

### HELM ON ALPINE

docker pull alpine/helm:3.1.0
docker tag alpine/helm:3.1.0-alpine $OCI_SERVICE/hive/helm:3.1.0-alpine
docker push $OCI_SERVICE/hive/helm:3.1.0-alpine

### ATLANTIS TERRAGRUNT

docker pull exositebot/atlantis-terragrunt:version-1.4.0
docker tag exositebot/atlantis-terragrunt:version-1.4.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.0

day 2 :

export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in

### NODE ON ALPINE

docker pull node:10-alpine
docker tag node:10-alpine $OCI_SERVICE/hive/node:10-alpine
docker push $OCI_SERVICE/hive/node:10-alpine

### HELM ON ALPINE

docker pull alpine/helm:3.0.3
docker tag alpine/helm:3.0.3-alpine $OCI_SERVICE/hive/helm:3.0.3-alpine
docker push $OCI_SERVICE/hive/helm:3.0.3-alpine

### ATLANTIS TERRAGRUNT

docker pull exositebot/atlantis-terragrunt:version-1.3.0
docker tag exositebot/atlantis-terragrunt:version-1.3.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.3.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.3.0

day 3 :

export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in

### NODE ON ALPINE

docker pull node:11-alpine
docker tag node:11-alpine $OCI_SERVICE/hive/node:11-alpine
docker push $OCI_SERVICE/hive/node:11-alpine

### HELM ON ALPINE

docker pull alpine/helm:2.15.2
docker tag alpine/helm:2.15.2-alpine $OCI_SERVICE/hive/helm:2.15.2-alpine
docker push $OCI_SERVICE/hive/helm:2.15.2-alpine

### ATLANTIS TERRAGRUNT

docker pull exositebot/atlantis-terragrunt:version-1.1.0
docker tag exositebot/atlantis-terragrunt:version-1.1.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.1.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.1.0

day 4 :

export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE
# docker logged-in

### NODE ON ALPINE

docker pull node:12-alpine
docker tag node:12-alpine $OCI_SERVICE/hive/node:12-alpine
docker push $OCI_SERVICE/hive/node:12-alpine

### HELM ON ALPINE

docker pull alpine/helm:2.15.1
docker tag alpine/helm:2.15.1-alpine $OCI_SERVICE/hive/helm:2.15.1-alpine
docker push $OCI_SERVICE/hive/helm:2.15.1-alpine

### ATLANTIS TERRAGRUNT

docker pull exositebot/atlantis-terragrunt:version-1.2.0
docker tag exositebot/atlantis-terragrunt:version-1.2.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.2.0
docker tag $OCI_SERVICE/hive/atlantis-terragrunt:version-1.2.0

All test images :

export OCI_SERVICE=docker.culturebase.org
# you access portus web ui through https://$PORTUS_SERVICE/
export PORTUS_SERVICE=portus.culturebase.org
export EXISTING_NAMESPACE

### NODE ON ALPINE

docker pull node:8-alpine
docker pull node:9-alpine
docker pull node:10-alpine
docker pull node:11-alpine
docker pull node:12-alpine

docker tag node:8-alpine $OCI_SERVICE/hive/node:8-alpine
docker tag node:9-alpine $OCI_SERVICE/hive/node:9-alpine
docker tag node:10-alpine $OCI_SERVICE/hive/node:10-alpine
docker tag node:11-alpine $OCI_SERVICE/hive/node:11-alpine
docker tag node:12-alpine $OCI_SERVICE/hive/node:12-alpine

### HELM ON ALPINE

docker pull alpine/helm:3.1.1
docker pull alpine/helm:3.1.0
docker pull alpine/helm:3.0.3
docker pull alpine/helm:2.15.2
docker pull alpine/helm:2.15.1

docker tag alpine/helm:3.1.1-alpine $OCI_SERVICE/hive/helm:3.1.1-alpine
docker tag alpine/helm:3.1.0-alpine $OCI_SERVICE/hive/helm:3.1.0-alpine
docker tag alpine/helm:3.0.3-alpine $OCI_SERVICE/hive/helm:3.0.3-alpine
docker tag alpine/helm:2.15.2-alpine $OCI_SERVICE/hive/helm:2.15.2-alpine
docker tag alpine/helm:2.15.1-alpine $OCI_SERVICE/hive/helm:2.15.1-alpine

### ATLANTIS TERRAGRUNT

docker pull exositebot/atlantis-terragrunt:version-1.4.1
docker pull exositebot/atlantis-terragrunt:version-1.4.0
docker pull exositebot/atlantis-terragrunt:version-1.3.0
docker pull exositebot/atlantis-terragrunt:version-1.1.0
docker pull exositebot/atlantis-terragrunt:version-1.2.0

docker tag exositebot/atlantis-terragrunt:version-1.4.1 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.1
docker tag exositebot/atlantis-terragrunt:version-1.4.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.4.0
docker tag exositebot/atlantis-terragrunt:version-1.3.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.3.0
docker tag exositebot/atlantis-terragrunt:version-1.1.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.1.0
docker tag exositebot/atlantis-terragrunt:version-1.2.0 $OCI_SERVICE/hive/atlantis-terragrunt:version-1.2.0

On day 6, we run a first tests suite, starting 9:00am  :

we check all images inventory, and keep that as a oci-images.json file using registry's  Catalog API endpoint, (not Portus API)
we call Portus API, to list all container images , according Portus ,and save that asjson into portus.oci-images.json
we have an expected-oci-inventory.day6.json, containing the list of all images expected to be found in the registry on day 6, that is to say, all images but those pushed on day 0.
we compare expected-oci-inventory.day6.json, and oci-images.json, generate a nice diff report
we compare expected-oci-inventory.day6.json, and portus.oci-images.json, generate a nice diff report

On day 0, day 1, day 2, day 3, day 4, and day 5, we run similar test,using same json diff technique

Shared Helper for load tests
preambule
Pour me  permettre de préparer des longues listes d'images, avec leurs numéros de version,voci commentje peux faire :
# preparation of huge test dataset for
# tests like loead tests on portus

export namespace=library
export repo_name=ubuntu

curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

export namespace=library
export repo_name=notary
curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

export namespace=library
export repo_name=centos
curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

export namespace=library
export repo_name=debian
curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

export namespace=library
export repo_name=archlinux
curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

export namespace=library
export repo_name=registry
curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

export namespace=dkron
export repo_name=dkron
curl  -L -si https://registry.hub.docker.com/v1/repositories/$namespace/$repo_name/tags > all.${namespace}.${repo_name}.tags.json

ls -allh *.tags.json

# And we can have 1,2,5,10,20, 30, 50, 100, 150, 200, 300, etc...500 simultaneous docker clients constantly pulling and pushing any randomly picked tag among this list

Hi @Jean-Baptiste-Lasselle I tried to use portus:2.5 but got:

https://github.com/SUSE/Portus/issues/2197
https://github.com/SUSE/Portus/issues/2200

So I can't proceed with that. Anyway thanks for your support

Hi @Jean-Baptiste-Lasselle I tried to use portus:2.5 but got:
* #2197

* #2200

So I can't proceed with that. Anyway thanks for your support

Hi  @robgiovanardi It's just a pleasure to support users who feedback so fast and share. Plus I love the General issue with the Portus project, it's like very significant, I believe, about the revolution currently happening all over the world with the cloud.
So, Ok, I duly note your results about 2.5, and definitely will feedback my further work on this issue :  Yes, we will get the keep_latest feature, should it take me to take over the whole portus project.
So next: I'll reproduce those two issues #2197 and #2200
I am so not surprised by #2197 , because while solving this issue I found out that there is (February 2020) no rails official image based on ruby above 2.4. I bet without even reading it, it's exactly the problem for #2197 , that is rails framework version should be upgraded (surely what the opensuse developers do on a regular basis), because of whatever dependency requiring minimum rails version blabla.
One of the cloud's critical challenge : mastering the dependency hell.
Note there's here a distribution management problem of the OpenSUSE project Portus, that we are currently, along with our friend @diranged , clearly identifying
The dream problem for a devops like me. :)

@robgiovanardi just to say i now read both :

#2197 : I am now saying 75% chances problem is exactly what I described above
and #2200  : no worries, this one is damn easy, I'll provide fix

All in all I'd say 99% chances I provide you with a fix for your setup, before end of next week
andi'lldo that with a repo here :   https://github.com/orgs/pokusio

@robgiovanardi Hi, ok :

I tested with portus source code last version on master
I just tried every Dockerfile I found, even https://github.com/openSUSE/docker-containers/tree/master/derived_images/portus
I even had to try and build from source in a new custom ruby / debian image :

goal is to just perform a standard build from source of a rails web application.
I need time to repair that, it's a real mess, there's not even one page in the developers documentation explaining you how to build from source, except yarn commands that are useless as far as ruby is concerned.

So :

this will take time,  to finally be able to benefit the  keep_latest feature.
this can be managed in a fairly acceptable way,for small middle size teams :

turn off garbage collector (so sparing some CPU memory machine ressources)
running every N days a batch job (dkron.io), completely independently from portus, does this :

the batch runs on an infra that has no adherence with portus infrastructure (machine is out there somewhere, we don't care, just need to be able to reach it)
using registry catalog api v2, retrieve list of all images that are OLDER_THAN=XX days, filter list to keep only images OLDER_THAN=XX days, and then keep only latest N tags  in list. Let's call that list .eligible.keep_latest.json
docker pull all those images, and push them to a fresh new, empty portus, or a protus that has a  much shorter (or longer?) OLDER_THAN. First portus reachable at portus.mycompany.org/registry.mycompany.org, This other portus at freeze.portus.mycompany.org/freeze.registry.mycompany.org
send notification to users, to :

give them the list .eligible.keep_latest.json,
tell them that starting monday 9:00am, for the next 24hours, those images will still be available, but read only
tell them that starting monday 9:00am, for the next 24hours, those images will still be available, but guaranteed on freeze.registry.mycompany.org, and unrecommended/unguaranteed on  registry.mycompany.org
those notifications should become webhooks : that way devops can use the webhooks to check maintainance status, and switch to freeze.registry.mycompany.org to docker pull the said images. Note they can still keep on docker pushing new images to registry.mycompany.org, during the garbage collection/curation period.

Monday 9:00am  : turn on garbage collector in docker-compose.yml, reset all PORTUS_DELETE_GARBAGE_COLLECTOR_* to turn on garbage collector, and docker-compose restart.
Let the Portus background do his job, while robots kiss back webhooks.
when finished, turn off the GArbage collector, docker push back the kept_latest images,
and let the SYNC feature of Portus update Portus webapp as usual.
tear down freeze.registry.mycompany.org

I'd think, for a dream tool for that, of  a K8S batch service manager / executor, with audit capabilities, inside a little K8S cluster, say 3VM on 2 different physical machines, plus dkron.io .

Hi @Jean-Baptiste-Lasselle
There's a DB workaround: I noticed that tags with 'marked = 1' wouldn't take in account by garbage collector process, so we can do following queries to Database:

given repository_id = 7, updates all tags to marked 0:update tags set marked = 0 where repository_id = 7;

Now mark all to 1 except the latest 15:update tags set marked = 1 where repository_id = 7 order by created_at desc limit 15;

Restart portus_background process for force a garbage collection: now our latest 15 tags are preserved because have flag 'marked = 1' and garbage collection process will ignore them.

Now I need to write a query/stored procedure which updates the latest N rows for every repository_id; then that query can be scheduled by a cronjob

Hi @Jean-Baptiste-Lasselle
There's a DB workaround: I noticed that tags with 'marked = 1' wouldn't take in account by garbage collector process, so we can do following queries to Database:
1. given repository_id = 7, updates all tags to marked 0:
   ```
   update tags set marked = 0 where repository_id = 7;
   ```

2. Now mark all to 1 except the latest 15:
   ```
   update tags set marked = 1 where repository_id = 7 order by created_at desc limit 15;
   ```

3. Restart portus_background process for force a garbage collection: now our latest 15 tags are preserved because have flag 'marked = 1' and garbage collection process will ignore them.

Now I need to write a query/stored procedure which updates the latest N rows for every repository_id; then that query can be scheduled by a cronjob

f***  (please allow me) I really mean like a thank you there :

I haven't tried anything you last added about db work around
But I mean I had a really,really hard day yesterday, testing every single Dockerfile from the SUSE Team
And it really is cheer up / supportive to have answers so quick , and shareful from

I really think it is very important that there is a serious alternative to harbor, and should it take weeks, to tackle down the entire software factory of Open SUSE / Portus, I will.
Actually, I already know enough to probably re-develop the whole thing, but  :

if I completely grind portus soruce code hitting with hundreds of automated, tests, to prove that there is a necessity, to re-develop from scratch the whole thing,
well then I'll have at least 70% of the re-developing already done. I'll have a complete sofware factory able to track down all features, in details, And I'll just roll , and manage contributors work, on the rails of TDD, and the git flow

So Thank you, before any test result,  Rob
(The Open SUSE servers downlaod.opensuse.org going down yesterday at 22:00 more / less...  harsh)

Hi @Jean-Baptiste-Lasselle
There's a DB workaround: I noticed that tags with 'marked = 1' wouldn't take in account by garbage collector process, so we can do following queries to Database:
1. given repository_id = 7, updates all tags to marked 0:
   ```
   update tags set marked = 0 where repository_id = 7;
   ```

2. Now mark all to 1 except the latest 15:
   ```
   update tags set marked = 1 where repository_id = 7 order by created_at desc limit 15;
   ```

3. Restart portus_background process for force a garbage collection: now our latest 15 tags are preserved because have flag 'marked = 1' and garbage collection process will ignore them.

Now I need to write a query/stored procedure which updates the latest N rows for every repository_id; then that query can be scheduled by a cronjob

And btw : excellent info on the db, I will use both DB and Catalog API, cause then I can compare, and consolidate, before launching the grunt batch job / process I above described, to gracefully emulate keep_latest, and have a transition solution, before I get more serious.

hashtag autobots all on portus ^^

Just to write it down,like that, I have a feeling openSUSE is conducted a huge migration on containers, so that it works with podman,  or any other #nobigfatdaemon oci runtime ecosystem. I think that's what the left maintainers like @mssola are concerned about, just  migrating portus distributed containers in that broader context. like what they have in mind constantly is Kubernetes

Alsotoshare with community, the tools list I'm gonna test to manage batch jobs :

https://github.com/volcano-sh/volcano : alternative to https://github.com/spotify/styx maybe i'll choose that over styx, because you can  also practivce big data, with spark, see https://github.com/TommyLike/spark-operator-volcano-demo .
https://github.com/spotify/styx : Voilà ce qu'il faut que j'utilise The service takes responsibility for triggering and possibly also re-triggering invocations until a successful exit status has been emitted or some other limit has been reached.
https://github.com/atlassian/escalator : try n use with styx, to see if it improves autoscaling of  styx.

Hi  @robgiovanardi  I have news here :

I finally succeeded building a portus:2.5 docker image,
how to build it exactly like I did :  https://github.com/pokusio/opensuzie-oci-library/releases/0.0.1
And I just finished testing it : hourra! Protus starts, I create ffirst super admin, teams namespace users, start docker push pull, it all worked without any crash or unexpected logs (and it boots up quite fast, but maybe due to my specific home provisioning)
And I'll soon push to quay.io and docker.io

https://github.com/pokusio/opensuzie-oci-library/issues/1

The Real Build from source : Additional info related to /init script fix

there is a .ruby-version at root of protus repo : it is used inside the Docker container, with something called RVM, like nvm, but for ruby.
in this .ruby-version file, we, on master, have 2.6.2.
That value has to be set to a value consistent with those applied for running, that is, the 2.5.0 and 2.5.3 versions.
So there, we might end up building different images for portus and background.

Pipeline cycle identified
Ok, the OpenSUSE Team use the https://github.com/SUSE/Portus source code repo, to version control pipeline recipe, in particular :

the pipeline steps that package portus into a linux opensuse package, see https://github.com/SUSE/Portus/blob/master/packaging/suse/package_and_push_to_obs.sh
they publish the portus linux package to the opensuse linux package repository, still using https://github.com/SUSE/Portus/blob/master/packaging/suse/package_and_push_to_obs.sh
and they build the portus image, installing portus into it, using package manager zypper.

Undocumented use of RVM confirmed
https://github.com/SUSE/Portus/blob/3de71a6fb1f9865ca194fe015d8014d06c4f3ef2/.travis.yml#L4

Victory :D
@robgiovanardi Tested and re-tested, versioned and released, https://github.com/pokusio/opensuzie-oci-library/releases/tag/0.0.2

It all works !! no errors in logs, both portus and background
I even checked the webui, I am running portus version 2.5, and I can even say the exact commit (on master I believe ...) is b87d37e4e692b4fe5616b6f0970cb606688c344a :

git clone https://github.com/SUSE/Portus.git
cd Portus
export GIT_COMMIT_ID='b87d37e4e692b4fe5616b6f0970cb606688c344a'
git branch --contains $GIT_COMMIT_ID

the clair scanner didn't seem too busy, when I was running portus:2.4.3, and now.... Even Clair  Works GREAAAAT :D

confirmed 2.5 version of portus source code, I can even tell the commit ID I believe it is b87d37e4e692b4fe5616b6f0970cb606688c344a :

And look, now Clair scanning Rocket Chat official distribution for docker ... full of vulnerabilities, my dear, as I expected ^ ^ :

Now, I can't wait you test that on your side :)

Hi, I'm a bit confused. I'm setting up an instance of Portus. I've been battling with it for quite a while, the example compose file gets you so far but there has been a lot of trial and error to get things working.
Now I am looking at setting up the garbage collection and I stumbled on this bug. I'm running opensuse/portus:2.4. So what is the recommended way to set this up given the limitations that we know exist? I'm wondering if I can leave the keep_latest option and just use a combination of older_than and tag?
I'm planning to use git flow and have a tag for each branch. If I set a regex for the tag option that only includes feature, release and hotfix branches then only these should be considered for GC, leaving master and develop safely alone.
I can't tell from this bug report if this will work or not?

I can't tell from this bug report if this will work or not?

Hi @benthurley82 ,
I today found your message.
I will try n help you the fastest way, so will propose :

I ask a question
you answer to just exactly that question, the less words, the better. You just trust I know how to drive.
then
I ask another question
and you answer tu just exactly [...]
etc ...

this will repeat around 6 to 12 times, until you have informations you need to get where you wannt to get, in you cicd.
So let's start.

question 1 : For this question, forget anything about Garbage collection, or any deletion at all, of any OCI image. Do you have any docker-compose.yml with which you successfully run Portus , and by sucessfully, I mean :

you can push images
you can pull images
you can CRUD users, teams, namespaces, images, Tokens, through the Web UI
when you push a new tag of an image, the new tag appears in your Portus Web UI just as you would expect.
you can create a Token, and docker login (then push pull etc) using that token
you have TLS certs on all three registry, background, portus

question 2 : Does the bullet list below answers your question ? (yes, or no)

Indeed, you cannot do what you can do with the keep_latest option, using a combination of tags and older_than, it is logicallly impossible :

using the tag config option, the only thing you can say is "Hey Portus, don't garbage collect any tag that matches exactly this regular expression".
but the problem is, that, for every interger N, whatever regular expression you choose as configuration, there is only one that can match all the N youngest in the future : *.*.*.
I will give a proof that will not assume you use the semver standard, or any other standard, just that you can match any of your version numbers, using a regular expression. Weak enough, fair enough, isn't it ?
First, I say that the versioning system you use, will feed a set of released versions, if ordered by datetimes (of tagging/release), will always remain a well ordered set, as the set theory understands the term, with additional properties :
(well ordered) among your version numbers, there is one that you call "the first version".
(well ordered) for any of your version number, there exists, and there is only one, other version number that you call the "next" version. Except if it is the latest of all your versions.
(well ordered) At any time, there exist one and only one version number, which you call "the latest" version.
Ok, Let bensregexp be a regular expression (pick one, write it on a paper as you read this), and MYSTERY_VERSION_NUMBER any of your existing version numbers.
there was a date time  DTomega, at which MYSTERY_VERSION_NUMBER was the latest version. Because that was true at the very instant you created / tagged / released that new version number.
So, At DTomega,   MYSTERY_VERSION_NUMBER  matched bensregexp.  And so, MYSTERY_VERSION_NUMBER  matched bensregexp today,
and that will remain true, at any date/time, for all of your version numbers.

So you see, whatever regular expression you may ever choose, it cannot ever allow you to "filter the 5 latest at anytime", unless you don't ever release more than 5 version numbers, in other words, you abort the project.
and that's why the tag config. option is used to just filter environments, not version : dev is dev, and will all remain just dev, just new versions of it.
what you do with your versioning system, is that you endow your version number with meta data,
For example, I would say that in dev-3.4.7, 3.4.7 is a version number of a software, endowed with the dev label, used to "slip in" the info that this docker image is the dev execution environment  for the 3.4.7 version of your software.
The docker image version number, implicitly is infered from the sofware it runs' versions, in the widely adopted standard (the git commit id maven plugins ? same in almost all todays lagnuages, javascript golang etc).
Think of alpine-12.0.4 , is alpine a version number ?
And well that regular expression is not of any use obviously, because that configuration will lead to not ever garbage collect at all.
That was the very point of @robgiovanardi , opening this issue, because Rob has the exact same use case as you.  So he realized he has to be able to use the keep_latest config. option.

Alright, so you HAVE, to use the keep_latesttag, to get the behavior you (and bob) actually want, which is  (isn't it ?) :
* hey portus, for all images (repositories), you delete all tags that are older than 100 days
* hey portus, regardless of their age (even in two years) you never, ever delete the 5 latest (youngest) tags, for any repository .

So rob opened an issue, because he also realized through tests, that for all version of portus, either the keep_latest is not available (version too old), or that config. option is bugged.
We (rob,  @diranged , and myself) discovered a few weeks ago :

that actually, there exists a version of the portus source code that has a repaired keep_latest config. option.
that, that version has a commit hash, which is on master branch, is not tagged, and not merged back to any other branch, like the v2.5 branch for example.
that this exact version of the source code of portus, is not distributed in any docker image on docker hub or quay.io, by the OpenSUSE Team.
So to be able to use the keep_latest option, we had to build our own docker image of portus, starting from its from source.

Now :

you will find here a total rebuild of portus docker image, starting from portus source code, which has a working keep_latest feature :  https://github.com/SUSE/Portus/issues/2241#issuecomment-593370789 :
and the sum up of what happened, why and when : https://github.com/SUSE/Portus/issues/2261#issuecomment-612624170


InvalidParameterEception even if value is ok. Email to long exception

Describe the bug
Hi, i get an Email should not be longer than 256 characters on a 256 character long email. This is a bug.
private readonly IAmazonCognitoIdentityProvider _cognitoService;

//...

var userAttributes = new List<AttributeType>
        {
            new() { Name = "email", Value = email }, // email is for example 256 character long
            new() { Name = "custom:role", Value = userType.ToString().ToLower() }
        };

        var signUpRequest = new SignUpRequest
        {
            UserAttributes = userAttributes,
            Username = userName,
            ClientId = _clientId,
            Password = password,
            SecretHash = CalculateSecretHash(userName)
        };

        try
        {
            var response = await _cognitoService.SignUpAsync(signUpRequest); // will throw:
            // Amazon.CognitoIdentityProvider.Model.InvalidParameterException: 1 validation error detected: Value 'email = "asdöklfaslkdf@asdfjalksdjfklajskldfkasjlasdklfjklasjdfkljalskjdfljakdfklaksdflkajskldfjlkasjfdkljasdlkfjklasdjfkljasdkl
fjalskdjfklasdjfklsadjfkldsjflkasklfjasdfjalksdjfklajskldfkasjlasdklfjklasjdfkljalskjdfljaksdfklaksdflkajskldfjlkasjfdkljasdlkfjkeajsj.de"' at 'filter' failed to satisfy constraint: Member must have length less than or equal to 256

Please fix that.
Regression Issue

[ ] Select this option if this issue appears to be a regression.

Expected Behavior
I expect that the Error Message will be correct. The error message says that it would not fail if the email is 256 characters long, but it failed.
Current Behavior
email to long exception for a not to long email
Reproduction Steps
call the SignUpAsync method from IAmazonCognitoIdentityProvider
Possible Solution
Allow or dissallow emails with that length.
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
AWSSDK.CognitoIdentityProvider version 3.7.404.1
Targeted .NET Platform
.NET 8
Operating System and version
Windows 11

@joshua-classen Good afternoon. The issue is not reproducible using the code below (used AWSSDK.CognitoIdentityProvider version 3.7.404.2). This also assumes that Username is selected as SignIn option while creating user pool):
using Amazon.CognitoIdentityProvider;
using Amazon.CognitoIdentityProvider.Model;
using System.Security.Cryptography;
using System.Text;

namespace CognitoIDPTest
{
    internal class Program
    {
        static void Main(string[] args)
        {
            Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
            Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
            Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

            IAmazonCognitoIdentityProvider _cognitoService = new AmazonCognitoIdentityProviderClient();
            string email256Chars =<EMAIL_ADDRESS>            string userName = "<<username>>";
            string password = "<<password>>";
            string clientId = "<<appclientid>>";
            string clientSecret = "<<appclientsecret>>";

            var userAttributes = new List<AttributeType>
            { new() { Name="email", Value = email256Chars } };

            var signUpRequest = new SignUpRequest
            {
                Username = userName,
                Password = password,
                ClientId = clientId,
                UserAttributes = userAttributes,
                SecretHash = GetUserPoolSecretHash(userName, clientId, clientSecret)
            };

            var response = _cognitoService.SignUpAsync(signUpRequest).Result;
        }

        public static string GetUserPoolSecretHash(string userID, string clientID, string clientSecret)
        {
            string s = userID + clientID;
            byte[] bytes = Encoding.UTF8.GetBytes(clientSecret);
            byte[] bytes2 = Encoding.UTF8.GetBytes(s);
            return Convert.ToBase64String(new HMACSHA256(bytes).ComputeHash(bytes2));
        }
    }
}

However, while using an email<EMAIL_ADDRESS>shared by you, we get error Invalid email address format. error. I'm unsure if this is because of long domain name in your email address.
Kindly note that the AWSSDK.CognitoIdentityProvider is autogenerated from service models and the above error is thrown by the AWS Cognito service.
Do you get this error other than the one you originally reported initially? Could you try using short domain name?
If required, I could open an internal ticket to Cognito team to get their inputs.
Thanks,
Ashish

I found my error on a different location. Before I signup a user I want to check if the user with this mail already exists and I call this function before.
public async Task<UserType?> FindUserByEmailAsync(string email)
    {
        _logger.LogDebug("Searching for user by email: {Email}", email);

        var request = new ListUsersRequest
        {
            UserPoolId = _userPoolId,
            Filter = $"email = \"{email}\""
        };

        try
        {
            var response = await _cognitoService.ListUsersAsync(request); // will throw  at 'filter' failed to satisfy constraint: Member must have length less than or equal to 256
            // for FILTER = email =<EMAIL_ADDRESS>
            // because this is then longer than 256 characters
            if (response.Users.Any())
            {
                _logger.LogInformation("User found with email: {Email}", email);
                return response.Users.First();
            }

            _logger.LogInformation("No user found with email: {Email}", email);
            return null;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error searching for user by email: {Email}", email);
            throw;
        }
    }

The Email can be of a valid format for example<EMAIL_ADDRESS>but then the Filter string is to long and it will throw an exception. Basically you can not filter for correct emails.

This is really a bug in the filtering technology. It says that the Filter String can only be of max 256 characters lenght.
Type: String
Length Constraints: Maximum length of 256.
Required: No
Source: https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ListUsers.html
This is a problem because the user is allowed to save an email that has 256 characters. But you can not filter these email because the Filter attribute gets longer because "email = " gets attached.

Issue is reproducible using code below when using email with 256 characters in Filter:
public static async Task<UserType?> FindUserByEmailAsync(IAmazonCognitoIdentityProvider cognitoService, string userPoolId, string email)
{
    var request = new ListUsersRequest
    {
        UserPoolId = userPoolId,
        Filter = $"email = \"{email}\""
    };

    try
    {
        var response = await cognitoService.ListUsersAsync(request);

        // because this is then longer than 256 characters
        if (response.Users.Any())
        {
            return response.Users.First();
        }

        return null;
    }
    catch (Exception)
    {
        throw;
    }
}

@joshua-classen As you rightly pointed out, per AWS Cognito > ListUsers service API reference, Filter string can have maximum length of 256. This is a service API limitation, AWS SDK for .NET cannot do much about it. Please provide Feedback using link at top left of AWS Cognito > ListUsers service API reference page. This would open a ticket with the correct team and this way you would get updates on any progress made on the issue.
Thanks,
Ashish


FULL ARSENAL

I think this pr will fix almost all of the issues in this mod, so if there's no issue or bugs, please consider releasing it.
Added

Added Keys for texts added in Patch 15 and Patch 15b(a.k.a FULL ARSENAL Update)
Added Keys for Alter menu(incl, outside sandbox)

Changed

Changed act1_limboFourth_alternateRevolver to hud_alternateVersion because it's used in 3 different level in 3 different layer.
the previous key is reused for short description for Alt revolver.
Changed order of options texts in template.json to match the in game options

Fixed

Fix Magnet Nailgun/Sawblade Launcher's ammo text is unreadable(nailgun.cs, NailgunPostfix)
Fix mandalore's subtitles
Fix Gabriel's subtitles are never working outside it's level
Fix Language Button doesn't appear after patch 15
Fix Options panels are not getting patched after patch 15

Removed

Removed some unused strings in template.json.

Thank you so much for the contribution again, your work is invaluable!
Do you have a Discord account? We'd like to contact you on the project's discord server

Yeah and I have already joined in the project's server(if I didn't messed up with the links)
Username is 5n9 and displayname is Susinopo.

Thank you so much for the contribution again, your work is invaluable! Do you have a Discord account? We'd like to contact you on the project's discord server


Update the upgrade instruction docs to be clearer about how to actually do the upgrade

Going through the upgrade instructions, I found that there are a few locations that are out of date and also not clear enough:

The folder got renamed last release, so we can update the instructions to the 2.4 naming
I wanted to be clearer about what the instructions are for (2.4.0 to 2.5.0), and point folks at the previous if they were on 2.3
Updated the migration tool instructions to be clearer about how to run it (since it's not actually immediately obvious how to migrate).
There's a line where we tell people to watch out when deleting the Library/ folder, but we do that after telling them a few steps after telling them to delete the Library folder... which isn't very helpful.

/azp run mrtk_docs


Cannot start Kube-controller container of Calico v2.6.3  --- cannot find the kube config file

Kube-controller (v1.0.1) of Calico v2.6.3 does not work with Kube v1.6x/1.7 on both amd64 and s390x
Expected Behavior

On Calico v2.3 it was tested by following https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/integration#configuring-kubernetes and worked fine.
It seems that the backward comparability was broken.
Current Behavior

The container of kube-controller cannot be started, the container logs show:
 2017-12-08 19:50:27.051 [INFO][34] main.go 66: Loaded configuration from environment
 config=&config.Config{LogLevel:"info", ReconcilerPeriod:"5m",
 EnabledControllers:"policy,profile,workloadendpoint,node", WorkloadEndpointWorkers:1,
 ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:""}
 2017-12-08 19:50:27.052 [INFO][34] client.go 202: Loading config from environment
 2017-12-08 19:50:27.054 [FATAL][34] main.go 78: Failed to start error=failed to build kubernetes
   client config: invalid configuration: no configuration has been provided

It seems that the kube-controller cannot find the kube config file.
Possible Solution

Please provide a workable yaml file in the instruction document.
Steps to Reproduce (for bugs)

Following  the instruction https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/hosted

download calico.yam file, update the etcd endpoints, then add
      - name: KUBECONFIG
       value: /var/run/kubernetes/kubeconfig
     - name: ENABLED_CONTROLLERS
       value: policy,profile,workloadendpoint,node
   volumeMounts:
     # Mount in the etcd TLS secrets.
     - mountPath: /var/run/kubernetes/
       name: test-volume
volumes:
 # Used by calico/node.
 - name: test-volume
   hostPath:
     path: /var/run/kubernetes/

into the calico-policy-controller part.
2. cp  kubeconfig  /var/run/kubernetes/
3.  Then kubectl apply -f calico.yaml
You will see the container of kube-controller exits and logs show the error as the above.
By the way, as the developer suggested to define env variables like:
export KUBECONFIG=/var/run/kubernetes/kubeconfig
export ETCD_ENDPOINTS=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:2379

does not work.
Context

Your Environment

Calico version v2.6.3, k8s-policy (v1.0.1)
Orchestrator version (e.g. kubernetes, mesos, rkt):  Kubenetes 1.6.13 (1.17.x)
Operating System and version:  amd64 Ubuntu 16:04
Link to your project (optional):

@samding01 why do you need to mount in a kubeconfig file? I'd expect the correct serviceaccount token to be provided by the system for API access, unless you've not enabled the generation of api tokens?

@caseydavenport  Originally I followed the instruction without the part in the (1) (mount mounting the kubeconfig file), it did not work,  then the developer suggested to mount the folder.
What is the workable calico.yaml?

@samding01 if you're installing this "self-hosted" on Kubernetes as a Pod, I don't think you need to set KUBECONFIG, nor do you need the volume mount.
It should "just work".

Follow the  Integration Guide and face same problem with start up calico-kube-controllers..
Any ideas?

Try @samding01 's workaround with mounting the kubeconfig file and work. So the problem is that calico-kube-controllers wants to read this file and the Guide does not tel this details..

There are two name: calico-kube-controllers parts, it should put the above (1) to the 1st part,
which  made me confused.

@samding01 is this still a problem? From your last statement it sounds like you might have figured out the issue.

you can close this


Releases are not tagged in git

This would be nice have for quickly referencing the version of code for any given release.  Reconstructing would be nice, definitely should be done moving forward.

Yes, good suggestion. I haven't been doing this but I should. Just went and tagged all previous releases, and will do so from now on. (please do point it out if I forget!)

:+1: Thanks!


First comment in auto-generated files causes conflicts in git

The first comment in auto-generated files, e.g.: # Generated by ariadne-codegen on 2023-09-18 16:52, causes problems when two people auto-generate the same file with different content. Changes that could easily be merged in git without the comment, result in conflicts when the comments are there. Setting include_comments to false in config is not ideal, as then there is no info for code reviewers etc. that the code in a given file was auto-generated. It should be possible to keep the comments, but without the date (anyway, the approximate date of the change can be found in git).

We could make include_comments accept a string value, eg. "stable" to make this header comment stable (or deterministic) to avoid silly git conflicts on those.

I suggest the header comment in following syntax, so it can have a stable build output as long as meaningful content:
# Generated by ariadna-codegen v0.9.0
# Schema path: https://example.com/gql
# Schema checksum: d41d8cd98f00b204e9800998ecf8427e


removed explore pages from built site

Description

Makes all the rendering page options dependent on dev

Checklist

[x] I have added a changeset

@ItsMeBrianD seems like tests broke in 96a3943?


Trait constraints for functions

Blocked by:

#3143
#3031

closes #1163

Closing in favor of #3194.


icon: add ability to override aria-label

Expected behavior
So that semantic information about what the icon is used for is apparent to a screenreader.
Actual behavior
We have generic labels only (eg, "right arrow").  We need to allow users to override when they know more.  Beyond that, we can look at the internal usage of Icon in other DS components and make sure that we're overriding the labels helpfully once this is enabled.
Related packages
-icon

We're looking through our audit right now and came across this so I wanted to give it a 👍

Feature now supported.


fix clean deployment

fixes #154

restore "fix certifi" task to config.rb
only run "fix certifi" task in source.rb if the certificate already exists

Nice one Ben! This works well. Working in a green field deployment for me now. and when I call reinstallswift. If you could update the commit message to be a little more imformative, happy to land this!

Great work Ben!


Add QR sharing support to Android visualizer

This pull request only adds QR sharing to the Android visualizer. There'll be a separate one for the Designer (coming later), but want to get this merged in Android earlier (Designer will need more work).
With these changes, users can go to https://adaptivecards-sharing.azurewebsites.net/designer/ and scan the QR code.
You can try it by installing the app from the Store (that's a compiled version of this branch's code) https://play.google.com/store/apps/details?id=io.adaptivecards.adaptivecardsvisualizer_master

@almedina-ms can you review this QR sharing feature addition to Android? This new feature will probably help facilitate testing the Android renderer :)


(maint) Codebase Hardening

Changes made to ensure that no malformed commands are passed through to the system.
Certain commands were left undivided as the commands did not get correctly interpreted.

Not sure where these 6 failures are coming from?

Spec test failures replicated on main branch


[3.0] Support caching of dynamic property

Support caching of dynamic properties to avoid unnecessary remote interaction costs.

Codecov Report

Merging #7760 (109cbf2) into 3.0 (8a4a6b5) will decrease coverage by 0.48%.
The diff coverage is 25.00%.

:exclamation: Current head 109cbf2 differs from pull request most recent head 4ae3e9b. Consider uploading reports for the commit 4ae3e9b to get more accurate results

@@             Coverage Diff              @@
##                3.0    #7760      +/-   ##
============================================
- Coverage     58.16%   57.68%   -0.49%
- Complexity      402      403       +1
============================================
  Files          1069     1070       +1
  Lines         43203    43219      +16
  Branches       6432     6413      -19
============================================
- Hits          25130    24930     -200
- Misses        15280    15491     +211
- Partials       2793     2798       +5

Impacted Files
Coverage Δ
Complexity Δ

...dubbo/common/reporter/FrameworkStatusReporter.java
0.00% <0.00%> (ø)
0.00 <0.00> (?)

...t/migration/DefaultMigrationAddressComparator.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...apache/dubbo/common/config/ConfigurationUtils.java
77.14% <66.66%> (+2.14%)
0.00 <0.00> (ø)

...ache/dubbo/config/utils/ConfigValidationUtils.java
71.61% <100.00%> (-0.65%)
0.00 <0.00> (ø)

...egistry/client/migration/MigrationRuleHandler.java
27.14% <100.00%> (ø)
0.00 <0.00> (ø)

...figcenter/file/FileSystemDynamicConfiguration.java
0.00% <0.00%> (-59.05%)
0.00% <0.00%> (ø%)

...ntext/event/AwaitingNonWebApplicationListener.java
46.96% <0.00%> (-25.76%)
0.00% <0.00%> (ø%)

...fig/configcenter/TreePathDynamicConfiguration.java
74.28% <0.00%> (-14.29%)
0.00% <0.00%> (ø%)

...ng/transport/dispatcher/all/AllChannelHandler.java
82.75% <0.00%> (-13.80%)
0.00% <0.00%> (ø%)

...bo/config/event/listener/LoggingEventListener.java
37.50% <0.00%> (-12.50%)
0.00% <0.00%> (ø%)

... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a4a6b5...4ae3e9b. Read the comment docs.


DataContainer' object is not subscriptable

While running single_gpu_test.py, I got this error :
loading annotations into memory...

This is my configuration file:
inferencing with  single_gpu_test
cfg = Config.fromfile('./configs/detr/detr_r50_8x2_150e_coco.py')
cfg.dataset_type = 'CocoDataset'
cfg.classes = ("cystic_mass", "solid_mass")
cfg.model.bbox_head.num_classes = 2
Training data
PREFIX = '/media/josi/5CEAB0FEEAB0D58C/Dataset/blue_phantom/prepared/'
cfg.data.train.img_prefix = PREFIX + 'train/images/annotations/'
cfg.data.train.classes = cfg.classes
cfg.data.train.ann_file = PREFIX + 'train/annotations/instances_annotations.json'
cfg.data.train.type = 'CocoDataset'
Validation data
cfg.data.val.img_prefix = PREFIX + 'valid/images/annotations/'
cfg.data.val.classes = cfg.classes
cfg.data.val.ann_file = PREFIX + 'valid/annotations/instances_annotations.json'
cfg.data.val.type = 'CocoDataset'
Testing data
cfg.data.test.img_prefix = PREFIX + 'test/images/annotations/'
cfg.data.test.classes = cfg.classes
cfg.data.test.ann_file = PREFIX + 'test/annotations/instances_annotations.json'
cfg.data.test.type = 'CocoDataset'
#cfg.data.test.pipeline = get_loading_pipeline(cfg.data.train.pipeline)
build data and initialize detector
#cfg.work_dir = "./outputs_phantom"
datasets = build_dataset(cfg.data.test)
data_loader = build_dataloader(
datasets,
samples_per_gpu=2,
workers_per_gpu=1,
dist=False,
shuffle=False)
cfg.model.train_cfg = None
model = build_detector(cfg.model, test_cfg=cfg.get('test_cfg'))
model.CLASSES = [datasets][0].CLASSES
checkpoint_file = '/media/josi/5CEAB0FEEAB0D58C/mmdetection_/results_detr/latest.pth'
checkpoint = load_checkpoint(model, checkpoint_file, map_location='cuda')
#data['img_metas'][0] = data['img_metas'][0].data[0]
outputs = single_gpu_test(model, data_loader, out_dir= './results_phantom' )
out_file = './results_phantom/result.pkl'
mmcv.dump(outputs, out_file)
Any help will be much appreciated

This error is coming from mmdetection/models/detectors/base.py, line 134 object is not subscriptable, is a way to fix it?

It seems the same issues as https://github.com/open-mmlab/mmdetection/issues/4737.
You can try to upgrade the version of mmcv or change the version of pytorch.

Thank you @jbwang1997  the issue was solved

Hi @JosianeUwineza, how did you solve this issue?
Thanks.

@JosianeUwineza how did you solve the issue? was it adjusting your version of pytorch?

solved by upgrading mmcv to 1.7.0

谢谢@jbwang1997 问题已解决

I have the same problem, how can I solve it

谢谢@jbwang1997 问题已解决

这个错误来自mmdetection/models/ detectors/base.py，第134行对象不可下标，有办法修复吗？

I have the same problem, how can I solve it

Hello, I am using mmcv-full 1.5.3 which actually fix the mentioned issue, https://github.com/open-mmlab/mmcv/pull/882. But this error still occurs.


Tree-shaking package imports that include file extensions

Original Discussion: https://github.com/pikapkg/snowpack/discussions/965
/cc @papierschiff

For example, create a new module test at the node_modules directory with one file, add one method to this file, export this method and use it in your code. I get the following behaviour for imports

import * as TestModule from 'test'; // is working
import * as TestModule from 'test/index'; // is working
import * as TestModule from 'test/index.js'; // is not working("Generated an empty chunk: "te

Bug is most likely coming from this line in src/rollup-plugins/rollup-plugin-wrap-install-targets.ts:
        const allInstallTargets = installTargets.filter(
          (imp) => imp.specifier.replace(/\.js$/, 'js') === key,
        );

We convert a .js -> js for packages like tippy.js that get installed as tippyjs.js. It looks like we need to restrict this to only a package name. In the past, we've done that by calling:
import validatePackageName from 'validate-npm-package-name';
if (validatePackageName(imp.specifier).validForNewPackages) {
  // this is a package name, not a path within a package

c.ts
export * from './a';
export * from './b';
index.ts
import { TestBComponent } from './c';
snowpack will install  a.ts & b.ts
treeshake is not work


Added a check to switch between BuildStatus and BuilderStatus.

Refs #5

r? @brson
(rust_highfive has picked a reviewer for you, use r? to override)

I've tested this successfully in dev. Hopefully deploying to prod today.
Sorry for taking so long.

Rebased, merged and deployed. Thanks!


Incomplete (stripped) 1st production build

I found a weird bug where the 1st production build with npm run production will produce incomplete build/css/main.css. However subsequent builds will correct the issue itself.
I am trying with the sample jigsaw blog $ ./vendor/bin/jigsaw init blog. I deleted the build/ folder and run npm run production for the 1st time

Notice the 11.4kb main.css
This is what I get in next build with npm run production

Here we have 12.4kb main.css
Here is the screenshot of code compare in VS code of these 2 different main.css

My working environment:
OS: Window 10
node version: 10.15.3
npm version: 6.8.0
Laravel mix version: 4.0.14

Transferring this here, since it seems to be related to purgeCss, which is unique to the blog template. It doesn't affect a stock Jigsaw install.

@iainheng can you confirm if you still have this issue? The blog template has changed a fair bit since we were last looking at this issue and I can no longer reproduce it on a fresh install.

@bakerkretzmar sorry for my belated reply. Yes. I can confirm this issue is no longer exists after few tries I tried just now. 👍🏻

Glad to hear it! Closing this now, if you're still seeing this on older sites, upgrading to use the new laravel-mix-jigsaw package should be enough to fix it. Feel free to comment or open another issue if you run into any problems doing that!


Colab .cache weights and config

@JaMe76
I am using colab when I try to make my pipeline and try to load layout-parser models but I don't know where should I load it. As it has no ".cache" directory. can you let me know if I create a folder and load weights and config files from that folder it will be handy. Can you share how to load?

Check this tutorial.
It shows you how to save your models manually in the .cache and how to add the model to the ModelCatalog.

To get the .cache you can find it like that
from deepdoctection.utils.settings import dd_cache_home

print(dd_cache_home)

You can also change the .cache dir by changing the env variable before setting everything else up: DEEPDOCTECTION_CACHE.


Proposal to allow passing rotate option in float (or in radians)

I want to draw a LINE shape from point A to point B, but as there is no such functionality, I'm using x, y, w, h and rotate options.
So I'm calculating angle and offsets after rotation to position a line correctly.
But there are some inaccuracies in angle because of rotate should have integer type .
On top of this, I propose to make it more precise.
Maybe there is more handy way to solve this simple task, so I'll appreciate if somebody shares it to me.

The library can create lines:
API Shapes
Also check examples


Deploying the application

I was wondering how would I deploy the application. On the example it says that I have to have the dlls inside the system32 on windows folder and on linux it should be installed, I tried to put the dll inside the project folder and it doesn't load, is this right or there's a way to deploy everything together?

You need to copy mpv-1.dll to the directory with mpvjs.node

Sorry for my ignorance but what you mean?

?

See https://github.com/Kagami/mpv.js#windows
mpv-1.dll is name of the file from lachs0r builds.