Keys generated by KeyLocality.Random are just available locally

Everything with random keys is fine so far, despite the fact that you cannot use them easily in a global verify method. We have this issue in AsyncAtomicLongTest and AtomicLongTest which are not working with multiple members.
We should implement an easy-to-use way to generate random keys, which are at least collectable within the verify() method. I don't think that we need to share random keys during run(), if this would create a big overhead.
Ideas:

Save the keys locally and sync them between run() and verify() (costs memory)
Pre-generated keys which are synchronized once via Hazelcast during setup() (costs memory)
Realtime shared keys which are synchronized via Hazelcast (costs performance)
Add a unique PREFIX to all keys, iterate over list of DistributedObject and match with key.startsWith(PREFIX) to retrieve key set or objects (not very elegant)
Pseudo-Random with shared seed, so we can re-produce the same keys again and again (sounds wrong)

What do you think?

generate a random seed, store the seed and then use it in verify()?
this way each test run would use a different set of keys yet you would be able to re-generate it in the verify() method?

So you prefer the Pseudo-Random way, too.
We would have to sync the random seed between the members, maybe via IMap.putIfAbsent(testID, seed). Also the random generator for StringKeys seems to be a private static final Random in StringUtils. We would have to use our own Random generator (once created with the seed in setup() and again in verify()). Maybe we can put this in the TestContainer or TestContext, so it will be automatically re-generated before verify().
It sounds manageable if we provide some nice methods for that, so we don't have too much additional logic in the tests.

Perhaps we can generate the random seed based on the test id?

We have shared key locality.


Update README.md

Related issue

What changes does this PR do?
Screenshots (If any)
Questions

kindly add you name as per alphabetical order:)
__
  Summary: :hammer_and_wrench: This code needs a fix  |  Tags: Not maintainable

Hey @shubham2-ctrl. In order to merge your PR, kindly remove the blank space.
__
  Summary: :hammer_and_wrench: This code needs a fix

Hey @shubham2-ctrl. In order to merge your PR, kindly remove the blank space.
__ sema-logo Summary: 🛠 This code needs a fix

Done


add my pixel at (42,30)

Checklist

[x] I ran npm test locally and it passed without errors.

[x] I only edited the _data/pixels.json file.

[x] I entered the username in the pixels.json that I'm also using to create this pull request.
[x] I acknowledge that all my contributions will be made under the project's license.

add my pixel at (42,30)


Remove TSM Wiki Page

https://github.com/bcgov/bc-vcpedia/wiki/TSM-Protocol-Credential-Governance-Framework-(Primary-Document)

Complete


Docker Daemon

File: desktop/mac/install.md
At no point do you mention that this is just docker client and that docker daemon wont be installed.
Lots of time wasted trying to realise mac install is not a full install:
https://stackoverflow.com/questions/44084846/cannot-connect-to-the-docker-daemon-on-macos

Can you provide more details? The Docker Desktop installation installs both the docker cli and a minimal Linux VM that runs the Docker Engine (Docker Daemon). The "failed to connect to the docker daemon" error can occur if the VM failed to start (you can see the status through the "whale" icon in your menu bar, which should show "Docker Desktop is Running").

Thanks for sharing the issue. Closing this as stale unless more details are provided.


Upgrade to XF 3.1

I was going to add a missing property ListView.SelectionMode, but found this was new in XF 3.1 (link).
So my question is:
Can we update EXF to use the latest version of XF (templates and all), or is it tied to XF 3.0 for a reason?

I think it's OK to upgrade (though maybe we aim to stay about 2 months behind the curve?)

Alright, I agree that we should wait a little for things to settle down. XF 3.1 is only 1 month old for now.
In the meantime, I'll try to find an easy way to extend existing controls without updating EXF (saw some examples in AllControls)


Compilation error : actual and formal argument lists differ in length

Summary
Hello! While trying to compile LeaveEvent for lab 2 I got this error and I'm unsure of what it means. LeaveEvent inherits from my Abstract Class Event that has the properties customer and time.
Screenshots

Hi, this error just means that you have passed the wrong number of inputs to your constructor argument! You may notice the lines that say
required: Customer, double
found: Customer
This means that you are only initialising with a Customer, and omitting the double that is required by your constructor method! Try looking at the constructor definition and your constructor call to ensure that you are passing the correct number and types of input arguments. Hope this helps! :)

Thanks for the explanation!


Rockrobo v1 map issue after upgrade to valetudo 0.6.0

I've lost the maps in IObroker for my two Rockrobo v1 after upgrading to valetudo 0.6.0.
With v0.5.1 the maps were existent in IObroker, with 0.6.0 the are gone.

The maps are available in the web frontend.

Maybe i have the same problem.
I updated to Valetudo 0.6.1.
The objects are created but the values are empty.
In the Log i can see following lines...

mihome-vacuum.0 | 2020-08-11 16:05:13.542 | warn | (5105) Could not receive Mappointer, giving up
mihome-vacuum.0 | 2020-08-11 16:04:52.805 | warn | (5105) Could not receive Mappointer, giving up
mihome-vacuum.0 | 2020-08-11 16:04:47.543 | info | (5105) create state for carpet_mode
mihome-vacuum.0 | 2020-08-11 16:04:46.973 | info | (5105) set nächster Timer: Nicht verfügbar
mihome-vacuum.0 | 2020-08-11 16:04:46.535 | info | (5105) Connected
mihome-vacuum.0 | 2020-08-11 16:04:46.494 | info | (5105) connecting, this can take up to 10 minutes ...
mihome-vacuum.0 | 2020-08-11 16:04:46.493 | warn | (5105) Time difference between Mihome Vacuum and ioBroker: -3 sec
mihome-vacuum.0 | 2020-08-11 16:04:46.482 | info | (5105) New generation or new fw(3.5.8,004018) detected, create new states goto and zoneclean
mihome-vacuum.0 | 2020-08-11 16:04:46.445 | info | (5105) Cloud control disabled
mihome-vacuum.0 | 2020-08-11 16:04:46.443 | info | (5105) Expert mode disabled, states deleted
mihome-vacuum.0 | 2020-08-11 16:04:46.364 | info | (5105) starting. Version 2.1.1 in /opt/iobroker/node_modules/iobroker.mihome-vacuum, node: v12.18.3, js-controller: 3.1.6

@Tschipel have you the same entries in the log?

ihome-vacuum.0
2020-08-13 11:02:01.310
info
(155826) create state for carpet_mode

mihome-vacuum.0
2020-08-13 11:01:58.820
info
(155826) set nächster Timer: Nicht verfügbar

mihome-vacuum.0
2020-08-13 11:01:58.452
error
(155826) TypeError: Cannot read property 'pixels' of undefined

mihome-vacuum.0
2020-08-13 11:01:58.319
warn
(155826) Time difference between Mihome Vacuum and ioBroker: -117 sec

mihome-vacuum.0
2020-08-13 11:01:58.319
info
(155826) Connected

mihome-vacuum.0
2020-08-13 11:01:58.314
info
(155826) New generation or new fw detected, create new states goto and zoneclean

mihome-vacuum.0
2020-08-13 11:01:58.302
info
(155826) Disconnect

mihome-vacuum.0
2020-08-13 11:01:58.297
info
(155826) Create state clean_home for controlling by cloud adapter

mihome-vacuum.0
2020-08-13 11:01:58.296
info
(155826) Expert mode enabled, states created

mihome-vacuum.0
2020-08-13 11:01:58.267
info
(155826) starting. Version 2.0.7 in /opt/iobroker/node_modules/iobroker.mihome-vacuum, node: v12.18.3, js-controller: 3.1.6

mihome-vacuum.0
2020-08-13 11:01:58.082
warn
(155826) This object will not be created in future versions. Please report this to the developer.

mihome-vacuum.0
2020-08-13 11:01:58.081
warn
(155826) Object mihome-vacuum.0.info.dnd is invalid: obj.common.type has an invalid value (switch) but has to be one

mihome-vacuum.1
2020-08-13 11:02:02.541
info
(155841) create state for carpet_mode

mihome-vacuum.1
2020-08-13 11:02:00.051
info
(155841) set nächster Timer: Nicht verfügbar

mihome-vacuum.1
2020-08-13 11:01:59.633
error
(155841) TypeError: Cannot read property 'pixels' of undefined

mihome-vacuum.1
2020-08-13 11:01:59.550
warn
(155841) Time difference between Mihome Vacuum and ioBroker: -28 sec

mihome-vacuum.1
2020-08-13 11:01:59.549
info
(155841) Connected

mihome-vacuum.1
2020-08-13 11:01:59.546
info
(155841) New generation or new fw detected, create new states goto and zoneclean

mihome-vacuum.1
2020-08-13 11:01:59.536
info
(155841) Disconnect

mihome-vacuum.1
2020-08-13 11:01:59.530
info
(155841) Create state clean_home for controlling by cloud adapter

mihome-vacuum.1
2020-08-13 11:01:59.529
info
(155841) Expert mode enabled, states created

mihome-vacuum.1
2020-08-13 11:01:59.500
info
(155841) starting. Version 2.0.7 in /opt/iobroker/node_modules/iobroker.mihome-vacuum, node: v12.18.3, js-controller: 3.1.6

mihome-vacuum.1
2020-08-13 11:01:59.262
warn
(155841) This object will not be created in future versions. Please report this to the developer.

mihome-vacuum.1
2020-08-13 11:01:59.261
warn
(155841) Object mihome-vacuum.1.info.dnd is invalid: obj.common.type has an invalid value (switch) but has to be one

@B4n4n3 these are my two logs
As i understand the complete map structure got a redesign by Hypfer, unfortunately i'm not skilled enough to implement the new way (which is available in the valetudo frontend) to iobroker.

https://github.com/Hypfer/Valetudo/releases/tag/0.6.0
New Data Format
I've finally found the time to rework the Map Data format as well as the robot state format.
Both previously being heavily influenced by roborock, the new and improved formats are a huge step for easier adaption
of Valetudo to new Vacuums as well as implementation of new features.
In fact it has already proven itself in other work that has been done for this release and decreased memory pressure quite a bit.
This change is also of course a breaking change.
Make sure to update any dependant applications/integrations/etc. as well.`

Same for me here with my Roborock S50:
2020-08-25 20:02:43.435 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":192,"method":"get_status"}
--
2020-08-25 20:02:43.443 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5287,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":192}
2020-08-25 20:03:03.437 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":193,"method":"get_status"}
2020-08-25 20:03:03.448 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5289,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":193}
2020-08-25 20:03:23.440 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":194,"method":"get_status"}
2020-08-25 20:03:23.443 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":195,"method":"miIO.info"}
2020-08-25 20:03:23.446 - debug: mihome-vacuum.0 (31036) Next WiFi check: 25.08 20:04
2020-08-25 20:03:23.463 - debug: mihome-vacuum.0 (31036) Receive <<< {"partner_id":"","id":195,"code":0,"message":"ok","result":{"hw_ver":"Linux","fw_ver":"3.3.9_001886","ap":{"ssid":"Katzenkorb","bssid":"B0:BE:76:77:C9:3F","rssi":-54},"netif":{"localIp":"<IP_ADDRESS>","mask":"<IP_ADDRESS>","gw":"<IP_ADDRESS>\n<IP_ADDRESS>"},"model":"roborock.vacuum.s5","mac":"7C:49:EB:9D:94:B8","token":"72385577627935684c76695048544c45","life":58080}}
2020-08-25 20:03:23.466 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5290,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":194}
2020-08-25 20:03:43.446 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":196,"method":"get_status"}
2020-08-25 20:03:43.456 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5291,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":196}
2020-08-25 20:04:03.447 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":197,"method":"get_status"}
2020-08-25 20:04:03.474 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5293,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":197}
2020-08-25 20:04:23.452 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":198,"method":"get_status"}
2020-08-25 20:04:23.455 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":199,"method":"miIO.info"}
2020-08-25 20:04:23.457 - debug: mihome-vacuum.0 (31036) Next WiFi check: 25.08 20:05
2020-08-25 20:04:23.480 - debug: mihome-vacuum.0 (31036) Receive <<< {"partner_id":"","id":199,"code":0,"message":"ok","result":{"hw_ver":"Linux","fw_ver":"3.3.9_001886","ap":{"ssid":"Katzenkorb","bssid":"B0:BE:76:77:C9:3F","rssi":-54},"netif":{"localIp":"<IP_ADDRESS>","mask":"<IP_ADDRESS>","gw":"<IP_ADDRESS>\n<IP_ADDRESS>"},"model":"roborock.vacuum.s5","mac":"7C:49:EB:9D:94:B8","token":"72385577627935684c76695048544c45","life":58140}}
2020-08-25 20:04:23.483 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5294,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":198}
2020-08-25 20:04:43.458 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":200,"method":"get_status"}
2020-08-25 20:04:43.474 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5295,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":200}
2020-08-25 20:04:58.345 - debug: mihome-vacuum.0 (31036) stateChange mihome-vacuum.0.map.loadMap {"val":true,"ack":false,"ts":1598378698339,"q":0,"from":"system.adapter.admin.0","user":"system.user.admin","lc":1598359704084}
2020-08-25 20:04:58.346 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":201,"method":"get_map_v1"}
2020-08-25 20:04:58.356 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":201}
2020-08-25 20:04:58.859 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":202,"method":"get_map_v1"}
2020-08-25 20:04:58.861 - debug: mihome-vacuum.0 (31036) 0. Mappointer_nomap___map_upload_handler
2020-08-25 20:04:58.882 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":202}
2020-08-25 20:04:59.386 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":203,"method":"get_map_v1"}
2020-08-25 20:04:59.387 - debug: mihome-vacuum.0 (31036) 1. Mappointer_nomap___map_upload_handler
2020-08-25 20:04:59.410 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":203}
2020-08-25 20:04:59.913 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":204,"method":"get_map_v1"}
2020-08-25 20:04:59.915 - debug: mihome-vacuum.0 (31036) 2. Mappointer_nomap___map_upload_handler
2020-08-25 20:04:59.964 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":204}
2020-08-25 20:05:00.046 - info: host.iobroker instance system.adapter.dwd.0 started with pid 2808
2020-08-25 20:05:00.467 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":205,"method":"get_map_v1"}
2020-08-25 20:05:00.468 - debug: mihome-vacuum.0 (31036) 3. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:00.484 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":205}
2020-08-25 20:05:00.988 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":206,"method":"get_map_v1"}
2020-08-25 20:05:00.990 - debug: mihome-vacuum.0 (31036) 4. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:01.026 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":206}
2020-08-25 20:05:01.529 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":207,"method":"get_map_v1"}
2020-08-25 20:05:01.530 - debug: mihome-vacuum.0 (31036) 5. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:01.544 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":207}
2020-08-25 20:05:02.046 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":208,"method":"get_map_v1"}
2020-08-25 20:05:02.047 - debug: mihome-vacuum.0 (31036) 6. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:02.054 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":208}
2020-08-25 20:05:02.558 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":209,"method":"get_map_v1"}
2020-08-25 20:05:02.559 - debug: mihome-vacuum.0 (31036) 7. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:02.586 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":209}
2020-08-25 20:05:02.990 - info: dwd.0 (2808) starting. Version 2.4.10 in /opt/iobroker/node_modules/iobroker.dwd, node: v10.21.0, js-controller: 3.1.6
2020-08-25 20:05:03.088 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":210,"method":"get_map_v1"}
2020-08-25 20:05:03.089 - debug: mihome-vacuum.0 (31036) 8. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:03.104 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":210}
2020-08-25 20:05:03.460 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":211,"method":"get_status"}
2020-08-25 20:05:03.471 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":[{"msg_ver":2,"msg_seq":5297,"state":8,"battery":100,"clean_time":2852,"clean_area":42030000,"error_code":0,"map_present":1,"in_cleaning":0,"in_returning":0,"in_fresh_state":1,"lab_status":0,"fan_power":101,"dnd_enabled":0}],"id":211}
2020-08-25 20:05:03.607 - debug: mihome-vacuum.0 (31036) sendMsg[1] >>> {"id":212,"method":"get_map_v1"}
2020-08-25 20:05:03.607 - debug: mihome-vacuum.0 (31036) 9. Mappointer_nomap___map_upload_handler
2020-08-25 20:05:03.631 - debug: mihome-vacuum.0 (31036) Receive <<< {"result":["map_upload_handler"],"id":212}
2020-08-25 20:05:03.632 - warn: mihome-vacuum.0 (31036) Could not receive Mappointer, giving up

Maybe @MeisterTR  needs to edit how the mappointer has been read out in the past

What I noticed was when I set up Valetudo and it started the first cleaning run, the map came up but got stuck then. After that no update at all...

I also got this error .... no Map data loaded.
Gen1 Device with valetudo 2021.02.0
 warn | (26254) Could not receive Mappointer, giving up
Hope to get info how to resolve this


startGenericLaser is blocking

I'm trying to use sick_scan_xd without ROS. As far as I understand startGenericLaser should return immediately, but it never does for some reason. Running it from another thread and calling stopScannerAndExit does not help either. Am I overlooking something? I'm testing in linux with an LMS511.

My bad, rosSpin was blocking not startGenericLaser, which is not necessary to use without ROS.


m.request() mangles urls containing colon in the path

version: 0.2.4
browser: any
When calling m.request() with a colon in the url, the url is modified. I'm guessing this is a result of the parameter substitution for m.route(), but I wasn't expecting this for m.request(). Even if it is an intended feature, there doesn't seem to be a way to escape the colon in the url.
example:
m.request({url:"http://example.org/some:path/", data:{q:1}})
expected:
GET http://example.org/some:path/?q=1
got instead:
GET  http://example.org/someundefined/?q=1

Fixed in origin/next

Works perfectly, thanks for the quick response!


Fix ABI stability statement

Currently, it's misleading. See: https://nullderef.com/blog/plugin-start/#_abi_unstability_its_much_worse_than_it_seems

Hmmm, not super happy with my patch or yours. You can load plugins dynamically just by using #[repr(C)]. abi_stable just makes it easier, safer, and more backward-compatible.


Need to upgrade the package.

Hi,
Get some issue in this package, this package is no compatible with angular 9 please check and upgrade it please
Renderer need to change to Renderer2
And i need force : "enableIvy": false,
And i also need to comment this section
// this.keyDownListener = this.renderer.listenGlobal("window", "keydown", function (e) { return _this.onKeyDown(e); });
Because of this issue this.renderer.listenGlobal is not a function
I attached the ss please find the attachment.

Thanks.

Hi! Try this package please:
https://github.com/kolkov/ngx-gallery

Hi,
It worked. I am going to close the issue. 👍
Thanks.


The dashboard DOS attacks the service during initial load

Description
helm-dashboard seems to want to get all the information for all the services from the initial call.
This causes a DOS on the service itself.
The CPU spikes and I never get results for the resources view.
This seems to be initiated from datadog-rum-v4.js, initialized by the analytics part.
We're using komodorio/helm-dashboard:1.3.2 docker image
This causes the whole system to stop responding until kubernetes has restarted the dashboard.
I'm trying to modify the deployment and disabling tracking, but this is very annoying.. if you want tracking, you should probably modify this to not DOS the service..
Screenshots

Additional information
No response

Hi
It has nothing to do with analytics. It's just how the app works by querying health status for each of the releases. There's no recursion, just loop once over a list. It may be longer if your list is long.
I guess you have many releases installed, are you? How many?
It this moment I'm not sure what would be the best solution to address it. Probably we should only query the health status for those releases that are visible on the list. It's something that could be part of V2 effort (https://github.com/komodorio/helm-dashboard/pull/233).
One of the workaround is to narrow down the list of namespaces in scope via --namespace parameter.
Another immediate option to overcome this is Komodor's platform, where you have the same functionality without scalability issues.

:/
We deploy multiple times a day, so the list is long.
Looking at the code, it's doing a lot of things that can be optimized.
To get the health of a single service, the app is fetching all helm apps and all helm releases and parsing them before processing and checking the health status of that single app.
There are some architectural decisions you can make to improve this such as:

Cache helm releases for 20 seconds (helps with bursts).
Do a request collapsing - if there are multiple requests to the k8s api for the same data, fold them into a request queue and return all the calling functions the same data.
For a health check, only check health. Don't download the whole catalog for every single application.

This is a cool concept that can be improved a bit by doing tiny performance optimizations.
I don't think we'll be able to use this or Komodor's platform (I guess it's running the same thing behind the scenes).
We'll find another solution :cry:

@SGudbrandsson About Komodor platform - no, it does not use the same code and same approaches, thus it does not suffer from this problem.
Thanks for sharing your observations, it will help to improve the product.

You were absolutely right about fetching all releases to find single, I'm already fixing that in #373

Wow, you're amazing!
Thanks for checking this out.
I'm also in contact with sales at Komodor.io regarding the platform to see if it fits our use case (we use GKE autopilot, so node based pricing doesn't work)

hi @undera !
any chance to get a new release? :wink:

hi @undera !
any chance to get a new release? 😉

Hell yeah!! I've opened this Pandora box

There you go, after 2 retries it have completed.


Sparse Tensor: request for reviews

Following please find a summary of changes made for sparse tensor feature on CPU.
Note: this PR contains duplicate code in PR #7015
Frontend Interfaces
NDArray Class Hierarchy
-> NDArrayBase (root)
-> NDArray
-> SparseNDArray
-> RowSparseNDArray, CSRNDArray
Sparse Storage Types Overview
Two storage formats are supported: row_sparse and csr.

row_sparse tensors, typically used for sparse gradient updates (similar to tf.IndexedSlice)

csr tensors, with the standard csr storage format.

Functionality Overview
operators with sparse inputs
A list of operators in this PR include:

Arithmetics: elemwise_add(row_sparse), dot(csr, row_sparse), dot(csr, dense)
Conversion: cast_storage(csr<->dense), cast_storage(row_sparse<->dense)
Indexing: sparse_retain(row_sparse), slice(csr)
Creation: zeros(row_sparse), zeros(csr)

iterators and IO utils functions

NDArrayIter with CSRNDArray
LibsvmIter which reads files in libsvm format and produces CSRNDArray
load/save RowSparseNDArray/CSRNDArray

kvstore and optimizer with sparse gradient updates

sgd_update(row_sparse) and sgd_mom_update(row_sparse) will update rows of the weight where the gradient of the row contains nonzeros.
kv.push with row_sparse gradient, which only sends over the nonzero gradients over the network
kv.row_sparse_pull with row_ids, which only pulls back a RowSparseNDArray with specified row_id over the network

unsupported functionality
There's a list of basic functionality supported in NDArray, but is either not supported or inefficient in SparseNDArray:

get/set an element in CSRNDArray/RowSparseNDArray by index (not supported)
reshape in CSRNDArray/RowSparseNDArray (not supported)
_slice, _at in CSRNDArray/RowSparseNDArray (not supported)
inplace add/sub/mul/div/mod, broadcast_to in CSRNDArray/RowSparseNDArray (not efficient)

Please refer to the TODO list for a list of ongoing/planned operators.
Backend Interfaces
NDArray
NDArray uses storage_type to determine what storage format to use to represent a tensor.
enum NDArrayStorageType {
  kUndefinedStorage = -1,  // undefined storage
  kDefaultStorage,         // dense
  kRowSparseStorage,       // row sparse
  kCSRStorage,             // csr
};

Some of the important interfaces added to NDArray are:
NDArray {
  // constructor
  NDArray(const NDArrayStorageType stype, const TShape &shape, Context ctx,
          bool delay_alloc = true, int dtype = mshadow::default_type_flag,
          std::vector<int> aux_types = {}, std::vector<TShape> aux_shapes = {},
          TShape storage_shape = TShape(mshadow::Shape1(0)))
  // memory allocation
  void CheckAndAllocData(const TShape &storage_shape) const;
  void CheckAndAllocAuxData(size_t i, const TShape &aux_shape) const;

  struct Chunk {
    Storage::Handle shandle;
    // storage handles for aux data (e.g. indptr, indices)
    std::vector<Storage::Handle> aux_handles;
    NDArrayStorageType storage_type;
    std::vector<int> aux_types;
    TShape storage_shape;
    std::vector<TShape> aux_shapes;
    void CheckAndAllocData(const TShape &shape, int dtype);
    void CheckAndAllocAuxData(size_t i, const TShape &shape);
  };
...

 void CopyFromTo(const NDArray &from, NDArray *to) {
    // if storage_types of from & to don't match, convert to destination storage type
    // perform copy...
 }

Executor and NNVM
operator interface
To deal with tensors of non-default storage, operators need to register with the following interface:
using FComputeEx = std::function<void (
        const nnvm::NodeAttrs& attrs,
        const OpContext& ctx,
        const std::vector<NDArray>& inputs,
        const std::vector<OpReqType>& req,
        const std::vector<NDArray>& outputs)>;

Since memory usage for sparse tensor cannot be calculated during graph binding phase, operator is expected to use CheckAndAllocAuxData and CheckAndAllocData to perform memory allocation for result tensors at runtime.
infer_storage_type pass
Besides FInferShape and FInferType, an operator needs to register FInferStorageType if it involves non-default storage_type. ctx could be used to infer if MKL experimental memory should be used.  The attribute __storage_type__ of mx.sym.Variable is read for this pass.
using FInferStorageType = std::function<bool (
        const NodeAttrs& attrs,
        const Context& ctx,
        std::vector<int>* in_attrs,
        std::vector<int>* out_attrs)>;

When binding a graph, InferStorageType pass will be applied to infer graph entries with unknown storage_type. The pass is implemented in MXNet instead of NNVM. infer_type and infer_shape passes are both moved from NNVM to MXNet.
plan_memory pass
For node entries of non-default storage_type, they're annotated with kDynamicStorageID; memory are neither requested nor released. No memory sharing optimization is applied.
memory allocation and sharing
For node entries of kDefaultStorage(i.e. dense storage), memory allocation and sharing is done as usual. For node entries of non-default storage_type, memory allocation is delayed to runtime. No memory sharing is applied to these entries even when shared_exec is provided.
executor dispatch and storage fallback
For each node in graph, if it involves non-default storage in either its inputs or outputs, FComputeExExecutor will be created if FComputeEx attribute is registered.
If FComputeEx is not registered, an FComputeExcutor/FStatefulComputeExcutor will be created, inputs with non-default storage will be converted to NDArrays with default (i.e. dense) storage; the outputs will be converted back to non-default storage if necessary. The converted NDArrays are temporary and will be released when there's no reference to them.
kvstore
Sparse gradient update requires a parameter to be initialized as row_sparse format in kvstore. If an optimizer is registered with FComputeEx, then the update may only involve the rows with non-zero gradients. Gradients are sent in row_sparse format to reduce the stress on network bandwidth.
For distributed training, kRowSparsePushPull mode is introduced so that row ids of the row_sparse gradients are encoded as keys by kvstore workers. When the kvstore server receives the message, it constructs a row_sparse gradient NDArray based on the row ids decoded from it and applies the optimizer to the weight. The memory for parameters are pre-allocated on parameter servers, instead of storing weight row by row.
TODO list
Operators

elemwise binary & unary operators (row_sparse): done, to be merged in another PR
negative, square, abs, sqrt, round, ceil, floor, fix, elemwise_add, elemwise_sub, elemwise_div, elemwise_mul, _maximum, _minimum, _mul_scalar
elemwise binary & unary operators (csr): ongoing
negative, square, broadcast_mul, _mul_scalar
inplace broadcast_add, broadcast_sub, _mul_scalar operators
broadcast operators (csr)
broadcast operators (row_sparse)
elemwise operators(csr)
_sync_copy_from with scipy.sparse.csr_matrix input
asscipy (csr)

kvstore and optimizer

lazy initialization - postpone parameter initialization to kvstore server
storing row_sparse weight by rows to support weights of > 2 billion rows. This avoid pre-allocating memory for row_sparse weight and reduces memory consumption at kvstore servers.
other optimizers for sparse gradient update (e.g. adam)

List of important files
NDArray related files:

ndarray.h
c_api.h
ndarray.cc
common/utils.h

Executor related files:

c_api_executor.h
attach_op_exec_pass.cc
graph_attr_types.h
op_attr_types.h
executor.h
graph_executor.h
graph_executor.cc
infer_graph_attr_pass.cc
inplace_addto_detect_pass.cc

Kvstore related files:

comm.h
kvstore_dist.h
kvstore_dist_server.h
kvstore_dist_local.h
(pending PR #7015)

@ZihengJiang @szha @anirudh2290 @stefanhenneking @madjam @cjolivier01
@tqchen @jermainewang @piiswrong @mli @pluskid @ykim362 @formath @reminisce
It's probably not feasible to review all the operators implemented, but I'd appreciate if important changes made to NDArray, executor and kvstore are reviewed. Any help is appreciated!

Great job! Did you consider use the storage format proposed by http://people.csail.mit.edu/fred/tensor-compiler-techreport.pdf ? It seems to me that adding more storage formats is quite tedious (e.g. CSC, BCSR, DCSR, etc.). It will be great that we can define the storage format in a hierarchical way so future extension will be much easier.

@jermainewang  Thanks for the great reference! I can see how the proposed storage format in the paper could be used to express many types of common sparse storage format like CSC and CSR. As for "define the storage format in a hierarchical way", I can see the four main categories in this paper: dense, dense / dense, sparse / sparse dense / sparse, sparse. Are you suggesting we assign the NDArrayStorageType enums to some of the categories here?

The storage format can be formalized as a vector. Each element is a pair contains "dim" and "dense/sparse".
For example:

Row-major equals to [<0, dense>, <1, dense>]
Col-major equals to [<1, dense>, <0, dense>]
CSR equals to [<0, dense>, <1, sparse>]
CSC equals to [<1, dense>, <0, sparse>]

If having more tensor dimensions (e.g. convolution),

BCSR equals to [<0, dense>, <1, sparse>, <2, dense>, <3, dense>]

I found most variants of current sparse tensor formats can be formalized as this. The paper talks about coordinate, ellpack and dia formats in the future work, which they believe can also be supported.
In terms of implementation, we could change the enum to the vector above, define some common formats (like CSR, CSC), and define the "==" operator. This should simulate the behavior of enum type.

@jermainewang Thanks for the paper reference. I can see the mapping between our currently defined enum kDefaultStorage, kRowSparseStorage, kCSRStorage and vectors [<0, dense>, <1, dense>], [<0, sparse>, <1, dense>], [<0, dense>, <1, sparse>]. I have a few questions regarding the proposal though:

kRowSparseStorage mapps to [<0, sparse>, <1, dense>] only when the ndarray is 2D, but the enum kRowSparseStorage actually means "sparse in the first dim and dense in the rest of the dims". It seems difficult to use a fixed-size vector to represent this generalized case for N-D tensors, where N is only known at runtime.

Has this formalization been proven to work for any storage format? My concern is that if we want to add a storage format that cannot be formalized in the vector representation in the future, how do we deal with the exception?

It is not clear to me that how COO format fits in the scheme? It seems that this kind of storage format is not mentioned/considered in the paper.

@reminisce These are actually great catches!

I think you are right. What I could think about is to use a integer (32/64 bits) to represent the format. Each bit means sparse or dense. In this case, we can support up to 32/64 dimension tensors, which may be enough?

2/3. I am not sure about this. I raised an issue in there project so we can discuss directly with the authors. (https://github.com/tensor-compiler/taco/issues/108)

I think the format layout by taco is interesting and we should adopt this more general scheme

These are good discussions. A more general format will make adding more storage types easier. We should also take special formats such as MKL experimental memory into consideration since it uses StorageType, too. Also we need to evaluate the cost of interface changes and performance if we want to use a vector instead of integer for NDArraryStorageType.

what's the use case for more general sparse formats? Scipy doesn't seem to support N-D sparse tensors. Are there other libraries that supports this?
The number of sparse formats grow exponentially and the interaction between them (for binary op) grow quadratically. This only make sense if you can generate kernels. Taco seems to do this for cpu but requires compiler toolchain at runtime. Is it feasible to include it in MXNet? Is is feasible to do it for GPU?

I think we agree a more general sparse scheme is good. The question is how much efforts it costs and how important is it. I think the change can be limited to only the data structure. It is fine our hand-written code only targets specific format combination.
I think having Taco in MXNet is great. I am not sure about the GPU story. Does TF support sparse operation on GPU?

@jermainewang  Both PyTorch and Caffe2 have sparse feature on GPU, but looks like tacoers haven't worked on GPU yet.

Hi everyone,
I've updated the branch to include code changes to address comments from reviewers (except the discussion re: taco's more general storage format).
Recent changes also include:

added sparse ndarray classes to python api markdown
added support storage fallback with mutable inputs
updated documents for sparse ndarray classes
removed todense(), to_rsp(), to_csr() and replaced with tostype()
changed executor debug str from macro to env var

Known issues to fix include:

storage fallback fails on batch norm operator https://github.com/apache/incubator-mxnet/issues/7346

I should acknowledge that this PR contains code from @reminisce(sparse dot, sparse retain, cast_storage, sparse elemwise sum, ndarray and infer_storage refactor and many other things) and @stefanhenneking (many optimized kernels for gpu sparse dot and cast_storage operator). This project won't go this far without their significant effort.

Since this cannot be rebased I'll just squash it into one commit

Hi @eric-haibin-lin,  @jermainewang, @piiswrong, all,
I'm one of the taco developers. I know that this pull request is closed, but I wanted to add a comment on taco. We're thinking about taco on GPUs, since it is a requirement for entry for at least ML. We think it can be done, but that it's going to be a fair amount of work to make it fast. We'll likely be working on it this spring.

test_kvstore_gpu.test_row_sparse_pull seems to a bit  flaky and has been failing randomly for egin recent following builds 475, 482. I'll create an issue for this.


Fastnlight: Watch (?-i:tbCSd)(?#cl.gy)

Fastnlight requests the watch of the watch_keyword (?-i:tbCSd)(?#cl.gy). See the MS search here and the Stack Exchange search in text, in URLs, and in code.

(?-i:tbCSd)(?#cl.gy) has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ


WIP resolves Camel 2.x issue 15033 generate index tables 2

Generate the index tables in branch 2.x similarly to latest and 3.2.x.
There is a mysterious unresolved change here: two options disappeared from the spring-event spring boot configuration.  Once this is explained and resolved the commits can be squashed and merged.
The problematic change is in rev bd1b8aa8d9d5246aea7496111d7ba99d638a69d2.
You can see the page in preview at https://camel-preview-2.s3-us-west-2.amazonaws.com/components/2.x/spring-event-component.html.

Its probably okay as the spring-even component is never really in use.


x/tools/gopls: CLI: gopls check go.mod hangs

The gopls command line can check go files with gopls check foo.go, and should theoretically be able to also check modfiles. However, gopls check go.mod just hangs indefinitely.
We should either add support for checking modfiles, or return a meaningful error.
CC @stamblerre

Interesting! @stamblerre can I pick this up?

I actually went ahead and created a PR, but I need help writing tests, can someone point to where the tests for this section of the code lie?


How do you install startcoin on Linux?

I can not find a .deb file and I can not find instructions that work.

You have to compile it from source. This is done by following this document: https://github.com/startcoin-project/startcoin/blob/master/doc/build-unix.md


ios 8 problem

hello,
this snippet of code doesn't work with ios8
    <script>
        var a = new BigNumber(0.5);
        var b = new BigNumber(0.5);
        a = a.plus(b);
        console.log(+a);
    </script>

console.log print NaN
with a bit of debug i found that the problem is here
xc[b] %= BASE;
here log of safari console

xc[b]
<<PHONE_NUMBER>00000
BASE
<<PHONE_NUMBER>00000
xc[b] % BASE
< 2.5333397e-315
a % 1
< 0
1 % 1
< 0

note the 2.5333397e-315 as a result of mod operation....
seems a safari bug, but as you know, it's easy if your lib support browser os rather than ask to steve jobs (rest in peace) to fix safari in the next ios version :dancers:
for now my (hugly) fix is
if (BASE == xc[b])
xc[b] = 0;
else
xc[b] %= BASE;
in "plus" function

How strange. Unfortunately, I can not try it myself at the moment.
What does 1e14 % 1e14 produce in the console?
Please post the result of adding console.dir(a) to your snippet so I can see the c, e and s properties of a.

modified the code as follow

output: (i have try to 'beautify' the report)
---------------------------------------------------------------- KO (a is NaN)
[Log] printing +a (test.html, line 10)
[Log] NaN (test.html, line 11)
---------------------------------------------------------------- KO (please note c[1])
[Log] printing console.dir(a) (test.html, line 13)
[Log] BigNumber (test.html, line 14)
c: Array[2]
0: 1
1: 2.5333397e-315
length: 2
proto: Array[0]
e: 0
s: 1
proto: BigNumber
---------------------------------------------------------------- OK
[Log] printing 1e1%1e1 (test.html, line 17)
[Log] 0 (test.html, line 18)
---------------------------------------------------------------- OK
[Log] printing 1e5%1e5 (test.html, line 20)
[Log] 0 (test.html, line 21)
---------------------------------------------------------------- OK
[Log] printing 1e7%1e7 (test.html, line 23)
[Log] 0 (test.html, line 24)
---------------------------------------------------------------- OK
[Log] printing 1e8%1e8 (test.html, line 26)
[Log] 0 (test.html, line 27)
---------------------------------------------------------------- OK
[Log] printing 1e9%1e9 (test.html, line 29)
[Log] 0 (test.html, line 30)
---------------------------------------------------------------- KO!!!
[Log] printing 1e10%1e10 (test.html, line 32)
[Log] 2.65249474e-315 (test.html, line 33)
---------------------------------------------------------------- KO!!!
[Log] printing 1e11%1e11 (test.html, line 35)
[Log] 1.9230586856e-314 (test.html, line 36)
---------------------------------------------------------------- KO!!!
[Log] printing 1e12%1e12 (test.html, line 38)
[Log] 1.3428254615e-314 (test.html, line 39)
---------------------------------------------------------------- KO!!!
[Log] printing 1e13%1e13 (test.html, line 41)
[Log] 1.900263809e-314 (test.html, line 42)
---------------------------------------------------------------- KO!!!
[Log] printing 1e14%1e14 (test.html, line 44)
[Log] 2.5333397e-315 (test.html, line 45)
---------------------------------------------------------------- KO!!!
[Log] printing 1e15%1e15 (test.html, line 48)
[Log] 3.166674627e-315 (test.html, line 49)

Thanks.
It looks like the % operator has problems with operands over 32 bits, which seems to be a surprisingly rudimentary failing of the JavaScript implementation.
I am reluctant to apply a fix to work round it here at this stage, but further details/confirmation of this issue would be welcome.

if you can share the snippet code with the fix, i will apply it locally eventually.
this issue unfortunatly prevent me to use this lib, as i work in mobile env.
I don't think that my fix is correct, probably it will be not too much fork for you to put out a fix for that, even locally or in a special version

What are the results given for each of the following?
1 % 1e14
2147483647 % 1e14
2147483648 % 1e14
4294967295 % 1e14
4294967296 % 1e14

here we go
[Log] printing 1 % 1e14 (test.html, line 17)
[Log] 1 (test.html, line 18)
[Log] printing<PHONE_NUMBER> % 1e14 (test.html, line 20)
[Log]<PHONE_NUMBER> (test.html, line 21)
[Log] printing<PHONE_NUMBER> % 1e14 (test.html, line 23)
[Log]<PHONE_NUMBER> (test.html, line 24)
[Log] printing<PHONE_NUMBER> % 1e14 (test.html, line 26)
[Log]<PHONE_NUMBER> (test.html, line 27)
[Log] printing<PHONE_NUMBER> % 1e14 (test.html, line 29)
[Log]<PHONE_NUMBER> (test.html, line 30)

code modified as follow
console.log("printing 1 % 1e14");
console.log(1 % 1e14);

console.log("printing<PHONE_NUMBER> % 1e14");
console.log(2147483647 % 1e14);

console.log("printing<PHONE_NUMBER> % 1e14");
console.log(2147483648 % 1e14);

console.log("printing<PHONE_NUMBER> % 1e14");
console.log(4294967295 % 1e14);

console.log("printing<PHONE_NUMBER> % 1e14");
console.log(4294967296 % 1e14);

Oh, so it just seems to fail when the first operand equals the second operand.
1e14 % 1e14        // 2.5333397e-315 fail
(1e14+1) % 1e14    // 1 okay
(1e14-1) % 1e14    //<PHONE_NUMBER>9999 okay

Is that correct?

confirmed!!


Added CSS to project

This PR adds the CSS style sheet and updated the HTML file folder for our Pre-work study guide application.

CSS code for Pre-work study guide completed


svcat provision command enhancement for service parameters via config file or config map

Bug Report
What happened:
svcat provision cli does not allows to specify either:

json file path to read the parameters for service creation
configmap name to read the parameters for service creation

There was a related issue for the first item https://github.com/kubernetes-sigs/service-catalog/issues/2543 but it was closed without providing the promised enhancement.
What you expected to happen:
Enhance the API for https://github.com/kubernetes-sigs/service-catalog/blob/master/docs/cli.md#provision-a-service to support to read configuration either from specified file location or kubernetes configmap

/livecycle frozen

This project is being archived, closing open issues and PRs.
Please see this PR for more information: https://github.com/kubernetes/community/pull/6632


[Bug] ImagePullBackoff when pulling an image from private ECR with SOCI Index being present.

Description
I have set up CRI-based authentication to make Soci Snapshotter work with our private ECR Registry. We use EKS.
I noticed that one of the containers was stuck in the ImagePullBackoff state. This was the event.
Failed to pull image "111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77": [failed to pull and unpack image "111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77": failed to resolve reference "111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77": unexpected status from HEAD request to https://111111111.dkr.ecr.eu-west-1.amazonaws.com/v2/tfy-truefoundry-billing-server-1egxs/manifests/75159d954d0b3aed9dfc9ca7df2654b5220f174f-77: 403 Forbidden, rpc error: code = Unavailable desc = failed to pull and unpack image "111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77": failed to prepare extraction snapshot "extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0": layer "551" unavailable: unavailable]

These are the logs from the Containerd, Soci snapshotter and Kubelet.
[root@ip-10-2-45-158 /]# journalctl -u containerd | grep "extract-639718280-mVNW"
Feb 22 08:40:19 ip-10-2-45-158.eu-west-1.compute.internal containerd[3145]: time="2024-02-22T08:40:19.668913836Z" level=error msg="PullImage \"111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77\" failed" error="rpc error: code = Unavailable desc = failed to pull and unpack image \"111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77\": failed to prepare extraction snapshot \"extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0\": layer \"551\" unavailable: unavailable"

[root@ip-10-2-45-158 /]# journalctl -u soci-snapshotter | grep "extract-639718280-mVNW"
Feb 22 08:40:19 ip-10-2-45-158.eu-west-1.compute.internal soci-snapshotter-grpc[2797]: {"key":"k8s.io/620/extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0","level":"info","msg":"preparing filesystem mount at mountpoint=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/551/fs","parent":"sha256:649fd1aff815275641e05b690d7ddb2755b6d9ae6da2c6c32baac61709e4b647","time":"2024-02-22T08:40:19.642853068Z"}
Feb 22 08:40:19 ip-10-2-45-158.eu-west-1.compute.internal soci-snapshotter-grpc[2797]: {"image":"111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77","key":"k8s.io/620/extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0","layerDigest":"sha256:cf35370accea37c1d01c15961f837154edafcf047ec934bacc449aa497ea0282","level":"info","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/551/fs","msg":"skipping mounting layer as FUSE mount: no ztoc for layer","parent":"sha256:649fd1aff815275641e05b690d7ddb2755b6d9ae6da2c6c32baac61709e4b647","time":"2024-02-22T08:40:19.642922896Z"}
Feb 22 08:40:19 ip-10-2-45-158.eu-west-1.compute.internal soci-snapshotter-grpc[2797]: {"error":"skipping mounting layer sha256:cf35370accea37c1d01c15961f837154edafcf047ec934bacc449aa497ea0282 as FUSE mount: no ztoc for layer","key":"k8s.io/620/extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0","level":"warning","msg":"failed to prepare remote snapshot","parent":"sha256:649fd1aff815275641e05b690d7ddb2755b6d9ae6da2c6c32baac61709e4b647","remote-snapshot-prepared":"false","time":"2024-02-22T08:40:19.642959154Z"}

[root@ip-10-2-45-158 /]# journalctl -u kubelet | grep "extract-639718280-mVNW"
Feb 22 08:40:19 ip-10-2-45-158.eu-west-1.compute.internal kubelet[3287]: E0222 08:40:19.669367    3287 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unavailable desc = failed to pull and unpack image \"111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77\": failed to prepare extraction snapshot \"extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0\": layer \"551\" unavailable: unavailable" image="111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-billing-server-1egxs:75159d954d0b3aed9dfc9ca7df2654b5220f174f-77"

I restarted the Pod and got the same error again.
I drained the Node where the Pod was scheduled to move it to another Node.
Surprisingly it worked this time. The new Node, where the Pod got scheduled, also had Soci Snapshotter.
Steps to reproduce the bug
No response
Describe the results you expected
No ImagePullBackoff
Host information

OS: AL 2
Snapshotter Version: soci-snapshotter-grpc version v0.5.0 77f218d0b11c88d17a832ad7456dbd2165d19c86
Containerd Version: containerd github.com/containerd/containerd 1.7.11 64b8a811b07ba6288238eefc14d898ee0b5b99ba

Any additional context or information about the bug
No response

I noticed similar issues when using Docker Hub as a registry. In this case, I did not have the SOCI Index for the image present in the repository.
Pod Event:
Failed to pull image "docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18": rpc error: code = Unavailable desc = failed to pull and unpack image "docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18": failed to prepare extraction snapshot "extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378": layer "3496" unavailable: unavailable
Error: ErrImagePull

[root@ip-10-2-17-73 /]# journalctl -u containerd | grep "extract-779240847-Oy6"
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal containerd[3677]: time="2024-02-22T06:46:17.804322115Z" level=error msg="PullImage \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\" failed" error="rpc error: code = Unavailable desc = failed to pull and unpack image \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\": failed to prepare extraction snapshot \"extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378\": layer \"3496\" unavailable: unavailable"

[root@ip-10-2-17-73 /]# journalctl -u containerd | grep "extract-779240847-Oy6"
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal containerd[3677]: time="2024-02-22T06:46:17.804322115Z" level=error msg="PullImage \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\" failed" error="rpc error: code = Unavailable desc = failed to pull and unpack image \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\": failed to prepare extraction snapshot \"extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378\": layer \"3496\" unavailable: unavailable"

[root@ip-10-2-17-73 /]# journalctl -u soci-snapshotter | grep "extract-779240847-Oy6"
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"key":"k8s.io/3573/extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378","level":"info","msg":"preparing filesystem mount at mountpoint=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/3496/fs","parent":"sha256:7cb425d35fb31431c07c144f811e9aae2e16b39afbc5ff4ece873589fba5d29f","time":"2024-02-22T06:46:17.782280909Z"}
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"unable to fetch SOCI artifacts: cannot fetch list of referrers: no existing referrers","key":"k8s.io/3573/extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378","level":"warning","msg":"failed to prepare remote snapshot","parent":"sha256:7cb425d35fb31431c07c144f811e9aae2e16b39afbc5ff4ece873589fba5d29f","remote-snapshot-prepared":"false","time":"2024-02-22T06:46:17.782349136Z"}

I am also noticing a lot of logs like the ones below,
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"failed(layer:\"sha256:1c74526957fc2157e8b0989072dc99b9582b398c12d1dcd40270fd76231bab0c\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\"): unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\", digest:\"sha256:1c74526957fc2157e8b0989072dc99b9582b398c12d1dcd40270fd76231bab0c\"): failed to refresh connection","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mount-point":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/118/fs","msg":"layer is unavailable","time":"2024-02-22T14:45:04.290019320Z"}
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-aditya-ws-civo-usage-tracker-zh31f:5a8ad2dfd46440ee0eee86586874582fb84e6d27-7\", digest:\"sha256:aa8e0026efede8b3da7364fd0ec879657b2c9be209b5cc1e2ec83bed6dfcf6a9\")","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/323/fs","msg":"failed to refresh the layer \"sha256:aa8e0026efede8b3da7364fd0ec879657b2c9be209b5cc1e2ec83bed6dfcf6a9\" from \"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-aditya-ws-civo-usage-tracker-zh31f:5a8ad2dfd46440ee0eee86586874582fb84e6d27-7\"","time":"2024-02-22T14:45:04.290009572Z"}
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"failed(layer:\"sha256:aa8e0026efede8b3da7364fd0ec879657b2c9be209b5cc1e2ec83bed6dfcf6a9\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-aditya-ws-civo-usage-tracker-zh31f:5a8ad2dfd46440ee0eee86586874582fb84e6d27-7\"): unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-aditya-ws-civo-usage-tracker-zh31f:5a8ad2dfd46440ee0eee86586874582fb84e6d27-7\", digest:\"sha256:aa8e0026efede8b3da7364fd0ec879657b2c9be209b5cc1e2ec83bed6dfcf6a9\"): failed to refresh connection","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/323/fs","msg":"check failed","time":"2024-02-22T14:45:04.290061220Z"}
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"failed(layer:\"sha256:aa8e0026efede8b3da7364fd0ec879657b2c9be209b5cc1e2ec83bed6dfcf6a9\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-aditya-ws-civo-usage-tracker-zh31f:5a8ad2dfd46440ee0eee86586874582fb84e6d27-7\"): unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-aditya-ws-civo-usage-tracker-zh31f:5a8ad2dfd46440ee0eee86586874582fb84e6d27-7\", digest:\"sha256:aa8e0026efede8b3da7364fd0ec879657b2c9be209b5cc1e2ec83bed6dfcf6a9\"): failed to refresh connection","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mount-point":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/323/fs","msg":"layer is unavailable","time":"2024-02-22T14:45:04.290095138Z"}
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\", digest:\"sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4\")","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/116/fs","msg":"failed to refresh the layer \"sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4\" from \"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\"","time":"2024-02-22T14:45:04.290119127Z"}
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"failed(layer:\"sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\"): unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\", digest:\"sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4\"): failed to refresh connection","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/116/fs","msg":"check failed","time":"2024-02-22T14:45:04.290136467Z"}
Feb 22 14:45:04 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"failed(layer:\"sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\"): unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"111111.dkr.ecr.eu-west-1.amazonaws.com\", ref:\"111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-srihari-ws-emotion-class-svc-kg6df:21907524966ac31c03786bf29313489274f1694d-1\", digest:\"sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4\"): failed to refresh connection","key":"sha256:8eaf2218e5b78e1a434f669af263ecf4bfa31c5615b8db65beabb8595d3db19b","level":"warning","mount-point":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/116/fs","msg":"layer is unavailable","time":"2024-02-22T14:45:04.290146660Z"}

Is this normal even after setting up CRI-based authentication?

I noticed similar issues when using Docker Hub as a registry. In this case, I did not have the SOCI Index for the image present in the repository.
Pod Event:
Failed to pull image "docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18": rpc error: code = Unavailable desc = failed to pull and unpack image "docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18": failed to prepare extraction snapshot "extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378": layer "3496" unavailable: unavailable
Error: ErrImagePull

[root@ip-10-2-17-73 /]# journalctl -u kubelet | grep "extract-779240847-Oy6"
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal kubelet[3842]: E0222 06:46:17.804623    3842 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unavailable desc = failed to pull and unpack image \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\": failed to prepare extraction snapshot \"extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378\": layer \"3496\" unavailable: unavailable" image="docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18"

[root@ip-10-2-17-73 /]# journalctl -u containerd | grep "extract-779240847-Oy6"
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal containerd[3677]: time="2024-02-22T06:46:17.804322115Z" level=error msg="PullImage \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\" failed" error="rpc error: code = Unavailable desc = failed to pull and unpack image \"docker.io/truefoundrycloud/tfy-truefoundry-abcd-onboarding-svc-8eavt:e2fcd343b85e5a48c73e8b0a9421a4969b5ac9a8-18\": failed to prepare extraction snapshot \"extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378\": layer \"3496\" unavailable: unavailable"

[root@ip-10-2-17-73 /]# journalctl -u soci-snapshotter | grep "extract-779240847-Oy6"
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"key":"k8s.io/3573/extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378","level":"info","msg":"preparing filesystem mount at mountpoint=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/3496/fs","parent":"sha256:7cb425d35fb31431c07c144f811e9aae2e16b39afbc5ff4ece873589fba5d29f","time":"2024-02-22T06:46:17.782280909Z"}
Feb 22 06:46:17 ip-10-2-17-73.eu-west-1.compute.internal soci-snapshotter-grpc[3323]: {"error":"unable to fetch SOCI artifacts: cannot fetch list of referrers: no existing referrers","key":"k8s.io/3573/extract-779240847-Oy6- sha256:ae5b1556685ad8a7631a85f3e89607495b2d4ad931bcaf2c89acf6ea14471378","level":"warning","msg":"failed to prepare remote snapshot","parent":"sha256:7cb425d35fb31431c07c144f811e9aae2e16b39afbc5ff4ece873589fba5d29f","remote-snapshot-prepared":"false","time":"2024-02-22T06:46:17.782349136Z"}

This is something I started noticing after enabling CRI-based authentication. I had the soci snapshotter for some time in the cluster, and I did not face this issue before.
When a new pod with a newly built image of the same Docker Hub registry got scheduled into this node again, I faced the same problem. I had no soci index in the repository for the same image. Once I drained the problematic node and the pod got scheduled to another new node, with soci-snapshotter running there, the issue was not there anymore.
Sometimes, the snapshotter can get into this bad state after running for some time. I should have restarted the snapshotter, rather than draining the node, to see if the issue was resolved, but it slipped my mind.

@debajyoti-truefoundry Hey, thanks for letting us know. We are looking into this. Could you provide a full dump of the snapshotter and containerd logs?

Please find the snapshotter log attached. @turan18
I did not have containerd logs with me. I will attach it as soon as this issue starts happening again.
soci-snapshotter.log

Please find the snapshotter log attached. @turan18
I did not have containerd logs with me. I will attach it as soon as this issue starts happening again. soci-snapshotter.log

This is the full dump of all the logs. I have not filtered it for a particular image.

@turan18 Adding logs from containerd and soci-snapshotter.
containerd.log
soci-snapshotter.log

When a new pod with a newly built image of the same Docker Hub registry got scheduled into this node again, I faced the same problem. I had no soci index in the repository for the same image. Once I drained the problematic node and the pod got scheduled to another new node, with soci-snapshotter running there, the issue was not there anymore.

Sometimes, the snapshotter can get into this bad state after running for some time. I should have restarted the snapshotter, rather than draining the node, to see if the issue was resolved, but it slipped my mind.

Hey @debajyoti-truefoundry, so a couple of things:

CRI credentials are stored in memory and so they do not persist over node reboot.

CRI credentials are static. If you were to run a pod with a set of images and then attempt to run the same pod with the same set of images on the same node, after some time (enough for the credentials to expire; which should be 12 hours in the case of ECR), the snapshotter would still attempt to use the in-memory expired CRI credentials from the first pod run.

The error you are seeing: failed to prepare extraction snapshot "extract-639718280-mVNW sha256:e3d7a120de01b04d536f8f8479b4f942dc3a4c6c52a4070e73c39e70f5a298b0": layer "551" unavailable: unavailable], is thrown when the snapshotter cannot establish a connection with the remote registry. In this case, it appears that the issue is related to authentication as we see tons of 403 responses from ECR. The snapshotter most likely attempted to use an older/expired set of CRI credentials for the image tfy-truefoundry-billing-server-1egxs.
That is to say, unfortunately, CRI credentials are not a stable mechanism for getting credentials to the snapshotter and we do not recommend using it in a production environment. For the time being, we recommend disabling CRI credentials and setting up a docker credential helper on the nodes (see: ecr-credential-helper for ECR), if you haven't done so already.
Apologies, for the troubles.

Thanks, @turan18, for your detailed response.

In this case, it appears that the issue is related to authentication as we see tons of 403 responses from ECR. The snapshotter most likely attempted to use an older/expired set of CRI credentials for the image tfy-truefoundry-billing-server-1egxs. This is also why you were unable to reproduce the error on newer nodes, since the images were re-pulled and the CRI credentials were re-populated.

We do not have any credential refresher in our cluster and keep the credentials as a kubernetes.io/dockerconfigjson type Secret.
I assume the CRI credential is basically what we have in the Secret, not a different token derived from the given credentials with a different lifetime. Please correct me if this assumption is wrong.
So, if the credential expired, even on the new node, it should have used the expired credential and given an ErrImagePull. But I am not observing this. Somehow, it works on the new node.
We faced the same issue with a private image for Docker Hub for which we were using a long-living credential that could not have expired. Again, we noticed the same behaviour once I drained the node and the pod got scheduled on a new node; the issue did not happen again.

Hey @debajyoti-truefoundry  sorry for the delay.

I assume the CRI credential is basically what we have in the Secret; it is not a different token derived from the given credentials with a different lifetime. Please correct me if this assumption is wrong.

All EKS worker nodes configure the kubelet to talk to a ECR credential provider binary, which dynamically fetches ECR credentials. The credential helper binary comes pre-installed on all EKS nodes (see: https://github.com/awslabs/amazon-eks-ami/blob/8d7b5f89f511ef018905c8e24a6c1917e3b8bbdb/scripts/install-worker.sh#L391-L405). Now, although you have pod secrets configured, the kubelet is most likely exec'ing the credential helper binary to get credentials, instead of calling out to the API server to get your pod secrets. Generally, pod secrets are not a good mechanism for storing short-lived secrets, like ECR credentials, unless you wish to incur the burden of updating them yourself.

So, if the credential expired, even on the new node, it should have used the expired credential and given an ErrImagePull. But I am not observing this. Somehow, it works on the new node.

The reason it works on the new node is because the images are re-pulled using the credentials obtained by the image credential helper binary automatically configured on the node.

https://github.com/awslabs/soci-snapshotter/issues/1084#issuecomment-1959009826 for which we were using a long-living credential that could not have expired. Again, we noticed the same behaviour once I drained the node and the pod got scheduled on a new node; the issue did not happen again.

Yeah, I initially said: "then attempt to run the same pod with the same set of images on the same node, after some time (enough for the credentials to expire; which should be 12 hours in the case of ECR), the snapshotter would still attempt to use the in-memory expired CRI credentials from the first pod run.", it is actually a little more nuanced than that.
Containerd solely refers to layers/snapshots by their content/digest. What this means is that shared layers between two different images will still have the same digest. When containerd attempts to unpack a layer to disk, it first checks to see if the layer has already been unpacked, using the digest of the layer as the key. If it sees that the key exists, then it will move on to the next layer. The problem with this is that it doesn't exactly fit the model of SOCI. To the SOCI snapshotter, layers are not just content but a set of configurations that are needed to fetch content. This includes things like the ECR repository URL and, more importantly, the credentials to the ECR repository/registry. What this then effectively means is that shared layers, regardless of what image they are a apart of, will use the same underlying SOCI configuration, meaning the same set of credentials.
This is exactly what's happening in your case. The image that resides in docker hub still shares layers with the images previously pulled with ECR with SOCI. Because of this, containerd assumes that there is no point of downloading/unpacking the layer, since, in it's eyes, the content already exists. So the layer in the docker image ends up sharing the same mount point and thus the same set of expired credentials, as the layer in the ECR image that was pulled with SOCI.
Now, although there isn't an easy way to address the shared layers problem, at least without making some changes in upstream containerd, we can still make some changes to the SOCI snapshotter, like avoiding making any requests to the registry after all the content for the layer has been downloaded. This still isn't a universal solution, however, as there may be cases where, for some reason, the SOCI snapshotter couldn't fetch all the content in the background and will need to do so on demand at some point after the credentials have expired.
Currently, the best mechanism and the one that we recommend for getting credentials through to the snapshotter, is setting up a credential provider on the node. This way, on credential expiration, the snapshotter can retrieve a fresh set of credentials.
Apologies, if any of this was confusing, I'm happy to help clarify any questions that may arise.

Hey @debajyoti-truefoundry  sorry for the delay.

I assume the CRI credential is basically what we have in the Secret; it is not a different token derived from the given credentials with a different lifetime. Please correct me if this assumption is wrong.

All EKS worker nodes configure the kubelet to talk to a ECR credential provider binary, which dynamically fetches ECR credentials. The credential helper binary comes pre-installed on all EKS nodes (see: https://github.com/awslabs/amazon-eks-ami/blob/8d7b5f89f511ef018905c8e24a6c1917e3b8bbdb/scripts/install-worker.sh#L391-L405). Now, although you have pod secrets configured, the kubelet is most likely exec'ing the credential helper binary to get credentials, instead of calling out to the API server to get your pod secrets. Generally, pod secrets are not a good mechanism for storing short-lived secrets, like ECR credentials, unless you wish to incur the burden of updating them yourself.

So, if the credential expired, even on the new node, it should have used the expired credential and given an ErrImagePull. But I am not observing this. Somehow, it works on the new node.

The reason it works on the new node is because the images are re-pulled using the credentials obtained by the image credential helper binary automatically configured on the node.

https://github.com/awslabs/soci-snapshotter/issues/1084#issuecomment-1959009826 for which we were using a long-living credential that could not have expired. Again, we noticed the same behaviour once I drained the node and the pod got scheduled on a new node; the issue did not happen again.

Yeah, I initially said: "then attempt to run the same pod with the same set of images on the same node, after some time (enough for the credentials to expire; which should be 12 hours in the case of ECR), the snapshotter would still attempt to use the in-memory expired CRI credentials from the first pod run.", it is actually a little more nuanced than that.
Containerd solely refers to layers/"snapshots" by the digest of their uncompressed content (it's a bit more complicated than that) . What this means is that shared layers, starting from the base layer, between two different images will still have the same snapshot key/"digest". When containerd attempts to unpack a layer to disk, it first checks to see if the layer has already been unpacked, using the digest of the uncompressed layer as the key. If it sees that the key exists, then it will move on to the next layer. The problem with this is that it doesn't exactly fit the model of SOCI. To the SOCI snapshotter, layers are not just content but a set of configurations that are needed to fetch content. This includes things like the ECR repository URL and, more importantly, the credentials to the ECR repository/registry. What this then effectively means is that shared layers, regardless of what image they are a apart of, will use the same underlying SOCI configuration, meaning the same set of credentials.
This is exactly what's happening in your case. The image that resides in docker hub still shares layers with the images previously pulled with ECR with SOCI. Because of this, containerd assumes that there is no point of downloading/unpacking the layer, since, in it's eyes, the content already exists. So the layer in the docker image ends up sharing the same mount point and thus the same set of expired credentials, as the layer in the ECR image that was pulled with SOCI.
Now, although there isn't an easy way to address the shared layers problem between images, at least without making some changes in upstream containerd, we can still make some changes to the SOCI snapshotter, like avoiding making any requests to the registry after all the content for the layer has been downloaded. This still isn't a universal solution, however, as there may be cases where, for some reason, the SOCI snapshotter couldn't fetch all the content in the background and will need to do so on demand at some point after the credentials have expired.
Currently, the best mechanism and the one that we recommend for getting credentials through to the snapshotter, is setting up a credential provider on the node. This way, on credential expiration, the snapshotter can retrieve a fresh set of credentials.
Apologies, if any of this was confusing, I'm happy to help clarify any questions that may arise.

sorry for the delay.

No worries. I appreciate the detailed response.

Now, although you have pod secrets configured, the kubelet is most likely exec'ing the credential helper binary to get credentials, instead of calling out to the API server to get your pod secrets.

Right. I tested this. Seems like that is what is happening. We use imagePullSecrets anyway, and I have no way to verify whether this binary is used or the imagePullSecret is used. 😅
[root@ip-10-2-28-241 /]# journalctl -u kubelet | grep image-credential
Mar 01 04:37:55 ip-10-2-28-241.eu-west-1.compute.internal kubelet[3063]: I0301 04:37:55.777992    3063 flags.go:64] FLAG: --image-credential-provider-bin-dir="/etc/eks/image-credential-provider"
Mar 01 04:37:55 ip-10-2-28-241.eu-west-1.compute.internal kubelet[3063]: I0301 04:37:55.777996    3063 flags.go:64] FLAG: --image-credential-provider-config="/etc/eks/image-credential-provider/config.json"
[root@ip-10-2-28-241 /]# cat req.json
{ "apiVersion": "credentialprovider.kubelet.k8s.io/v1", "kind": "CredentialProviderRequest", "image": "1111111111.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-sfy-manifest-service-v5nza:c4c2732e1d8639e70bb2845f93213c381b533dcf-221" }
[root@ip-10-2-28-241 /]# cat req.json | /etc/eks/image-credential-provider/ecr-credential-provider
I0301 11:34:37.717329  244493 main.go:124] Getting creds for private image<PHONE_NUMBER>.dkr.ecr.eu-west-1.amazonaws.com/tfy-truefoundry-sfy-manifest-service-v5nza:c4c2732e1d8639e70bb2845f93213c381b533dcf-221
{"kind":"CredentialProviderResponse","apiVersion":"credentialprovider.kubelet.k8s.io/v1","cacheKeyType":"Registry","cacheDuration":"6h0m0s","auth":{"1111111111.dkr.ecr.eu-west-1.amazonaws.com":{"username":"AWS","password":"redacted"}}}

What this then effectively means is that shared layers, regardless of what image they are a apart of, will use the same underlying SOCI configuration, meaning the same set of credentials.
Now, although there isn't an easy way to address the shared layers problem between images, at least without making some changes in upstream containerd, we can still make some changes to the SOCI snapshotter, like avoiding making any requests to the registry after all the content for the layer has been downloaded.

Right, this seems to be a big issue and should be documented. I went through https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md#cri-based-authentication, and this was not documented there. CRI-based Auth can only work reliably if the cluster has private images only from a single registry, ignoring the credential expiry issue.

Currently, the best mechanism and the one that we recommend for getting credentials through to the snapshotter, is setting up a credential provider on the node. This way, on credential expiration, the snapshotter can retrieve a fresh set of credentials.

Do you have any documentation for this? How do I integrate different credential providers for registries from different providers (Docker Hub, ECR)?

@turan18 Is this limitation present in other snapshotters too?
https://github.com/containerd/nydus-snapshotter
https://github.com/containerd/stargz-snapshotter

https://github.com/containerd/stargz-snapshotter/issues/1583 Is this related?

Right, this seems to be a big issue and should be documented. I went through https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md#cri-based-authentication, and this was not documented there. CRI-based Auth can only work reliably if the cluster has private images only from a single registry, ignoring the credential expiry issue.

CRI creds work fine, even with shared layers, as long as the registry credentials are not expired. The issue is with credential rotation.

Do you have any documentation for this? How do I integrate different credential providers for registries from different providers (Docker Hub, ECR)?

Unfortunately, we do not right now. You can follow the documentation on the ecr-credential-helper repository on how to get the helper installed and set up on the node. I'm not too sure how DockerHub handles credentials. If they are long lived/do not expire you can run docker login on the node with your username/id and password (see: https://docs.docker.com/reference/cli/docker/login/).

Is this limitation present in other snapshotters too?

https://github.com/containerd/nydus-snapshotter
https://github.com/containerd/stargz-snapshotter

Yes. Both stargz and nydus also rely on the same mechanisms to get credentials (credential helper or CRI creds). There are some on-going discussions upstream about making the process of obtaining credentials in snapshotter's more streamlined, but there isn't anything concrete just yet (see: https://github.com/containerd/containerd/issues/6251, https://github.com/containerd/containerd/issues/5251).

https://github.com/containerd/stargz-snapshotter/issues/1583 Is this related?

Yes, it looks like that's the same issue and the proposed fix was what I mentioned earlier, which is to ensure we not make requests once the content for the image is fully downloaded on disk. This is a viable work-around for the time being and something we will work on as-well.

I also want to understand whether https://github.com/awslabs/soci-snapshotter/pull/1035 helps resolve this issue.

Not really. All that will do is ensure that images that cannot be lazy loaded at-all are managed by containerd. If the image has already been pulled with the SOCI snapshotter, like in your case, and you attempt to to run another image with shared layers after the credentials have expired, containerd will still see the content as pulled and it will use the same underlying SOCI configuration.

CRI creds work fine, even with shared layers, as long as the registry credentials are not expired. The issue is with credential rotation.

Right. So there are two issues.

ECR creds are expiring within 6/12 hours. The credential that the soci-snapshotter is caching will get expired.
Shared layers between images present in DockerHub and ECR.

The reason image pulls from ECR were failing was due to 1. Image pulls from Docker Hub were failing due to 1 and 2.
As I have mentioned above, The credentials we set for Docker Hub never expire.

This is exactly what's happening in your case. The image that resides in docker hub still shares layers, starting from the base layer, with the images previously pulled with ECR with SOCI. Because of this, containerd assumes that there is no point of downloading/unpacking the layer, since, in it's eyes, the content already exists. So the layer in the docker image ends up sharing the same SOCI mount point and thus the same set of expired credentials, as the layer in the ECR image that was pulled with SOCI.

In this case, a layer from the DockerHub image shares the SOCI mount point of a layer from ECR with expired credentials. In the container's eye, these are logically the same layer. In this case, any extra API call to the registry will fail.

Yes, it looks like that's the same issue and the proposed fix was what I mentioned earlier, which is to ensure we do not make requests once the content for the image is fully downloaded on disk. This is a viable work-around for the time being and something we will work on as-well.

Will you be able to give a timeline for this? This will open up adoption for us.

Not really. All that will do is ensure that images that cannot be lazy loaded at-all are managed/unpacked by containerd.

Understood. I brought this up because I was thinking with the mental model that if there is any error from the soci-snapshotter, it can defer the process to containerd.

Unfortunately, we do not right now. You can follow the documentation on the ecr-credential-helper repository on how to get the helper installed and set up on the node.

Is this even solving the issue at the soci-snapshotter layer?
I can see that the Authn details are cached here. https://github.com/awslabs/soci-snapshotter/blob/1d4d288d008d5f9b67f7179e25e2b6f22e8d1365/service/keychain/cri/v1alpha/cri.go#L80
The cache is populated here. https://github.com/awslabs/soci-snapshotter/blob/1d4d288d008d5f9b67f7179e25e2b6f22e8d1365/service/keychain/cri/v1alpha/cri.go#L130C37-L130C44
So, the cached credentials are only updated if an image is pulled. Looking at https://github.com/containerd/stargz-snapshotter/issues/1583, it seems credentials will only get updated if we Always pull the Image.

Now, although you have pod secrets configured, the kubelet is most likely exec'ing the credential helper binary to get credentials, instead of calling out to the API server to get your pod secrets.

Kubelet is already using a credential helper.

Yes, it looks like that's the same issue and the proposed fix was what I mentioned earlier, which is to ensure we do not make requests once the content for the image is fully downloaded on disk. This is a viable work-around for the time being and something we will work on as-well.

The only fix here seems to be, "Do not connect the registry again for layer A if soci-snapshotter has already set it up". Can soci-snapshotter work with that assumption?

The only fix here seems to be, "Do not connect the registry again for layer A if soci-snapshotter has already set it up". Can soci-snapshotter work with that assumption?

It should, I've opened up an issue and will try and get a PR out soon to test this.

It should, I've opened up an https://github.com/awslabs/soci-snapshotter/issues/1116 and will try and get a PR out soon to test this.

Thanks @sondavidb. I am worried about another scenario.

Image A is cached on Node N. Assuming that it was already set up using the soci-snapshotter before and at present, there are no running containers in Node N using Image A.
A Pod having a container with Image A gets scheduled on Node N.
Assuming that the credentials of Image A's registry have expired or Kubelet is not passing credentials, given that Image A is already cached in the Node, will the soci-snapshotter still work?

@sondavidb I'm having trouble reproducing since the expiry time is 12h for an ECR token, but even with the amazon-ecr-credential-helper we're seeing that after the authorization expires it's never refreshed.
These are the logs in question. It's different than what's in this issue, so happy to open a new one if needed.
Mar 28 17:20:52 ip-10-0-1-2.us-west-2.compute.internal soci-snapshotter-grpc[690]: {"level":"info","msg":"Received status code: 403 Forbidden. Authorizing...","time":"2024-03-28T17:20:52.858737822Z"}
Mar 28 17:20:52 ip-10-0-1-2.us-west-2.compute.internal soci-snapshotter-grpc[690]: {"error":"failed to refresh URL: status 403 Forbidden","key":"sha256:[sha256]","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/51/fs","msg":"failed to connect to blob","time":"2024-03-28T17:20:52.859196457Z"}
Mar 28 17:20:52 ip-10-0-1-2.us-west-2.compute.internal soci-snapshotter-grpc[690]: {"error":"unable to create remote fetcher: failed to redirect: request to registry failed: unknown \"unknown\": giving up request after 1 attempt(s): context canceled (host \"0123456789012.dkr.ecr.us-west-2.amazonaws.com\", ref:\"0123456789012.dkr.ecr.us-west-2.amazonaws.com/repo/image:tag\", digest:\"sha256:[sha256]\")","key":"sha256:[sha256]","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/52/fs","msg":"failed to refresh the layer \"sha256:[sha256]\" from \"0123456789012.dkr.ecr.us-west-2.amazonaws.com/repo/image:tag\"","time":"2024-03-28T17:20:52.858771180Z"}

(same instance, different occurrence)
Mar 28 15:41:12 ip-10-0-1-2.us-west-2.compute.internal soci-snapshotter-grpc[690]: {"error":"unable to create remote fetcher: failed to redirect: request to registry failed: failed to handle challenge: failed to find supported auth scheme: not implemented (host \"0123456789012.dkr.ecr.us-west-2.amazonaws.com\", ref:\"0123456789012.dkr.ecr.us-west-2.amazonaws.com/repo/image:tag\", digest:\"sha256:[sha256]\")","key":"sha256:[sha256]","level":"warning","mountpoint":"/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/79/fs","msg":"failed to refresh the layer \"sha256:[sha256]\" from \"0123456789012.dkr.ecr.us-west-2.amazonaws.com/repo/image:tag\"","time":"2024-03-28T15:41:12.206163948Z"}

Restarting soci-snapshotter.service fixes the issue. Pods are running with pullPolicy: Always.
# uname -a
Linux [snip] 6.5.0-1016-aws #16~22.04.1-Ubuntu SMP Wed Mar 13 18:54:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# containerd --version
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8
# soci-snapshotter-grpc --version
soci-snapshotter-grpc version v0.5.0 77f218d0b11c88d17a832ad7456dbd2165d19c86
# kubelet --version
Kubernetes v1.28.5-eks-5e0fdde

Sorry for the delay in response @debajyoti-truefoundry

Thanks @sondavidb. I am worried about another scenario.
Image A is cached on Node N. Assuming that it was already set up using the soci-snapshotter before and at present, there are no running containers in Node N using Image A.
A Pod having a container with Image A gets scheduled on Node N.
Assuming that the credentials of Image A's registry have expired or Kubelet is not passing credentials, given that Image A is already cached in the Node, will the soci-snapshotter still work?

As of now, no, I don't have a reason to believe it would work, for the same reason that we always check the registry. With the fix from stargz I think this might work, but I wonder if it poses any security threats. Have made a PR for this in the meantime #1147

I'm having trouble reproducing since the expiry time is 12h for an ECR token, but even with the amazon-ecr-credential-helper we're seeing that after the authorization expires it's never refreshed.

We don't do any form of credential rotation. Whatever credentials are pulled with the image are the ones we use for the lifecycle of the container.

I'm having trouble reproducing since the expiry time is 12h for an ECR token, but even with the amazon-ecr-credential-helper we're seeing that after the authorization expires it's never refreshed.

We don't do any form of credential rotation. Whatever credentials are pulled with the image are the ones we use for the lifecycle of the container.

Setting aside K8s, it does seem like https://github.com/awslabs/soci-snapshotter/pull/945 made an attempt to. I split out this conversation to a new issue (https://github.com/awslabs/soci-snapshotter/issues/1148) so I don't muddle it more, apologies!

Hey @debajyoti-truefoundry, we just released v0.6.0, wondering if you could confirm that this mitigates this particular issue?

Thanks!
I got a little sidetracked on investigation for 1148 but I'm taking a look. I will confirm I can repro the solution, though.


Add llvm-version arg

A few notes on LLVM 15 after some testing:

All the targets are using minified LLVM by default, except x86_64-unknown-linux
minified LLVM 15 does not have all the necessary files to build esp-idf projects, its missing libLLVM-15.so
In Windows, libs are compressed in .zip while complete is .tar.xz

Pehaps this should be named clang version? AFAIK, and based off of @JurajSadel's testing this will not change the LLVM version used by the Rust compiler, but instead the version of clang (I think). Is this the intention?

Perhaps this should be named clang version? AFAIK, and based on @JurajSadel's testing this will not change the LLVM version used by the Rust compiler, but instead the version of clang (I think). Is this the intention?

It is supposed to change the LLVM version, AFAIK. Either rustc is not picking it correctly or I am doing something wrong, but I am installing the latest release of LLVM 15 instead of LLVM 14 (minified or complete).


Replace golang.org/x/text version

Refs:

https://github.com/stolostron/backlog/issues/26347

Signed-off-by: Justin Kulikauskas<EMAIL_ADDRESS>

/cherrypick release-2.6


Device parameters when connected

Hello, I am having some trouble retrieving device data once the device is connected. When I run startDeviceScan () I console log scanned devices (i only have one) and it has data in it like this:
{
id: "00:15:85:14:9C:09",
isConnectable: null,
localName: "BT05",
manufacturerData: "SE0AFYUUnAk=",
mtu: 23,
name: "BT05" ,
overflowServiceUUIDs: null,
rssi: -57,
serviceData: {0000b000-0000-1000-8000-00805f9b34fb: "AAAAAA=="},
serviceUUIDs: ["0000ffe0-0000-1000-8000-00805f9b34fb"],
solicitedServiceUUIDs: null,
txPowerLevel: 0,
_manager: BleManager-object,
}
after that I do device. connect() and it is successful, but when I console log device I have this data:
{
id: "00:15:85:14:9C:09",
isConnectable: null,
localName: null,
manufacturerData: null,
mtu: 23,
name: "BT05",
overflowServiceUUIDs: null,
rssi: null,
serviceData: null,
serviceUUIDs: null,
solicitedServiceUUIDs: null,
txPowerLevel: null,
_manager: BleManager-object
}
after that  i tried device.discoverAllServicesAndCharacteristics() but i got same result.
I saw one similar issue and also tried device.services() but i got error:
BleError: Services not discovered for device 00:15:85:14:9C:09
at parseBleError (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:120653:18)
at BleManager._callPromise$ (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:121001:51)
at tryCatch (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:20190:19)
at Generator.invoke [as _invoke] (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:20365:24)
at Generator.prototype.(anonymous function) [as throw] (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:20233:23)
at tryCatch (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:20190:19)
at invoke (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:20266:22)
at blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:20278:15
at tryCallOne (blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:21463:14)
at blob:http://localhost:8081/132206d4-b879-458d-87b6-d78f1fbb4e95:21564:17

localName: "BT05",
manufacturerData: "SE0AFYUUnAk=",
overflowServiceUUIDs: null,
rssi: -57,
serviceData: {0000b000-0000-1000-8000-00805f9b34fb: "AAAAAA=="},
serviceUUIDs: ["0000ffe0-0000-1000-8000-00805f9b34fb"],
solicitedServiceUUIDs: null,
txPowerLevel: 0,
``
are only visible during scanning. Please call ` device.discoverAllServicesAndCharacteristics()` and after it successfully completes call `device.services()`. It should work.

Thx i already did that and it worked :)


Fixed pathing issue where when codecept is started from directory other than test root, baseFolder would not be found

+some formatting

Closing as changes are in pull request https://github.com/Percona-Lab/codeceptjs-resemblehelper/pull/51


feat: Implement addMemo conditional

Closes: #331
I wonder if this is the right place for this UI element

I made a few adjustments:

the addMemo value will be copied over from the Mint to the AppMint when a new app gets created.
the switch is moved to the Mint Settings

For now, I disabled the switch as we don't have a userUpdateAppMint operation yet - let's discuss on a call how we wrap actually toggling this setting.


Fixtures can't be marked

The corresponding tests should already be marked anyways.
Contributor Checklist:

[ ] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases

Reviewer Checklist:

[ ] /azp run libertem.libertem-live-data passed

/azp run libertem.libertem-live-data


Question: Do we need to rebuild OpenPCDet after each commit you push to the repository in order to take effect?

This is maybe a stupid question,
but I wonder if updates MMLab members push to this repository
only take effect after a rebuild of OpenPCDet via the following commands in the root folder.
rm -r build
python setup.py clean
python setup.py develop

Does anyone know the answer (including the logic behind this)?

No, you just need to re-build the codes only if we modified the cuda/cpp codes under pcdet/ops/.
For other python/pytorch modifications at most time, you doesn't need to re-build your project.