Make TensorFlow Lite available as Swift Package Manager package

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information

TensorFlow version (you are using): Lite 2.3.0
Are you willing to contribute it (Yes/No): yes

Describe the feature and the current behavior/state.
Right now, TensorFlow Lite is available as a CocoaPod or as source for iOS developers. CocoaPods cannot be used from inside a Swift Package Manager (SPM) package. Since SPM is the official package manager from Apple and the way forward, it would be nice to have TensorFlow Lite available as an SPM package or an XCFramwork, in addition to the current CocoaPods distribution.
Will this change the current api? How?
No
Who will benefit with this feature?
iOS developers who are using SPM for module manager or who are not using CocoaPods anymore.
Any Other info.
I have tried creating my own SPM package or XCFramework from the source but could not get it to work because of the native code. But I'm confident it should be possible.

This isn't a solution, but I spent a fair amount of time digging into this trying to find a workaround.
I was able to get a custom build of TFLite built and packaged as an XCFramework in a private SPM module, but it required a handful of hacks to get working properly. I'm not really comfortable with bazel though, so I did all of this with writing a bash script over a handful of bazel operations.
My workflow is:

Build a device static framework with

bazel build --config=ios --ios_multi_cpus="armv7,arm64" //tensorflow/lite/experimental/ios:TensorFlowLiteC_framework
# copy output file to temporary build dir

Build a simulator static framework with

bazel build --config=ios --ios_multi_cpus="i386,x86_64" //tensorflow/lite/experimental/ios:TensorFlowLiteC_framework
# copy output file to temporary build dir

Write some Info.plist files into the framework folders since bazel seems to omit them and Xcode will silently fail to install your app on a device if they are missing.

Create an xcframework with something like

xcodebuild -create-xcframework \
   -output "${OUTPUT}" \
   -framework "${BUILD_DIR}/dev/TensorFlowLiteC.framework" \
   -framework "${BUILD_DIR}/sim/TensorFlowLiteC.framework

At this point, I made a sample private git repo that vendored the swift code and the TensorFlowLiteC.xcframework file and then added a Package.swift file to make it work as a source for SPM.  The instructions on the apple developer site give some helpful instructions for shipping a binary framework with SPM.

One limitation is that the xcframeworks will require you to manually disable the iOS simulator for arm64 architecture because tensorflow (rooted in a problem in bazel-core) assumes that arm64 is always an ios device, not a simulator. See the bug I filed here for details: https://github.com/bazelbuild/rules_apple/issues/980
To "disable the iOS simulator for arm64", you need to set the following Xcode build setting:
EXCLUDED_ARCHS[sdk=iphonesimulator*] = i386 arm64

@yyoon and @teijeong have we done any investigation into supporting the Swift Package Manager?

We haven't done any investigation, and I'll need to learn more about Swift Package Manager to understand how much effort it would be required to support it.
cc/ @morganchen12 who might have some experience about this.

Probably the most challenging part of adding SPM support is you will need to add SPM support for all of your dependencies. SPM packages are also defined per-repository, so one Package.swift file at the root of the tensorflow repo would be responsible for representing Tensorflow, TensorFlowLiteSwift, TensorFlowLiteObjC, TensorFlowLiteC, and any other modules that are buildable in a Swift target.
It looks like TensorFlowLiteSwift and TensorFlowLiteObjC have very shallow dependency graphs, so adding SPM support should be straightforward.

Today I hit the error building the framework with Xcode 12.3 Building for iOS Simulator, but the linked and embedded framework 'ITLogin.framework' was built for iOS + iOS Simulator. Which was working with Xcode 12.2 With Xcode 12.3 it is not a feature anymore, it is a bugfix.
It is time to move to XCFrameworks. Lots of big and small companies are already doing this.
https://gist.github.com/evnik introduced conversion script as a temporary solution: https://gist.github.com/evnik/6762d5c3a4b21f61f13b100e03b62c38

Today I hit the error building the framework with Xcode 12.3 Building for iOS Simulator, but the linked and embedded framework 'ITLogin.framework' was built for iOS + iOS Simulator. Which was working with Xcode 12.2 With Xcode 12.3 it is not a feature anymore, it is a bugfix.
It is time to move to XCFrameworks. Lots of big and small companies are already doing this.
https://gist.github.com/evnik introduced conversion script as a temporary solution: https://gist.github.com/evnik/6762d5c3a4b21f61f13b100e03b62c38

Yeah, I think XCFramework makes more sense than Swift Package Manager approach, as XCFramework supports packaging compiled binaries as we currently do today.
As far as I can tell, CocoaPods supports XCFrameworks already, but bazel still doesn't support building for iOS arm64 simulator (for M1-based macs) nor building XCFrameworks natively. We might need to do some manual work in the interim, with something like the script @greenzeal linked.
@greenzeal In what exact situation were you seeing that specific error? Can you be a bit more specific and help me reproduce that issue? What does your project structure look like?

Yeah, I think XCFramework makes more sense than Swift Package Manager approach, as XCFramework supports packaging compiled binaries as we currently do today.
As far as I can tell, CocoaPods supports XCFrameworks already, but bazel still doesn't support building for iOS arm64 simulator (for M1-based macs) nor building XCFrameworks natively. We might need to do some manual work in the interim, with something like the script @greenzeal linked.
@greenzeal In what exact situation were you seeing that specific error? Can you be a bit more specific and help me reproduce that issue? What does your project structure look like?

@yyoon
I am trying to create my framework, but have a dependency on TensorFlowLiteC.framework.
to reproduce:

Xcode 12.3
Having a TensorFlowLiteC.framework embedded
Build for iOS Simulator with release configuration

The compiler will complain like this: Building for iOS Simulator, but the linked and embedded framework 'ITLogin.framework' was built for iOS + iOS Simulator.
Another simple way to reproduce:

Xcode 12.3
create dummy Xcode project
pod init
add to pods pod 'TensorFlowLiteObjC' (it will download its dependency TensorFlowLiteC.framework)
pod install
Build for iOS Simulator with release configuration

I tried different ways to find a solution like building the Bazel and then TensorFlowLiteC from the source, but all lead to that error.
Besides tried to build TensorFlowLiteC normally and then using the patch mentioned here after with lipo added third architecture to simulator build in the XCFramework.
I would really appreciate if anyone has a workaround. Or maybe it's possible to build TensorFlowLiteC.framework using Xcode (xcodebuild)?

@yyoon
I am trying to create my framework, but have a dependency on TensorFlowLiteC.framework.
to reproduce:

Xcode 12.3
Having a TensorFlowLiteC.framework embedded
Build for iOS Simulator with release configuration

The compiler will complain like this: Building for iOS Simulator, but the linked and embedded framework 'ITLogin.framework' was built for iOS + iOS Simulator.
Another simple way to reproduce:

Xcode 12.3
create dummy Xcode project
pod init
add to pods pod 'TensorFlowLiteObjC' (it will download its dependency TensorFlowLiteC.framework)
pod install
Build for iOS Simulator with release configuration

I tried different ways to find a solution like building the Bazel and then TensorFlowLiteC from the source, but all lead to that error.
Besides tried to build TensorFlowLiteC normally and then using the patch mentioned here after with lipo added third architecture to simulator build in the XCFramework.
I would really appreciate if anyone has a workaround. Or maybe it's possible to build TensorFlowLiteC.framework using Xcode (xcodebuild)?

btw also I will be happy if there is a way, even a manual one. And will be happy to try to automate it and share it here.

btw also I will be happy if there is a way, even a manual one. And will be happy to try to automate it and share it here.

@greenzeal I could create a dummy project as you described and confirmed that I could reproduce.
Do you really need to build your framework for arm64 simulator?
If not, you could go to the target build settings, and add arm64 as an excluded architecture for Any iOS Simulator SDK for both debug and release configs.

If you have a pressing need to build for arm64 simulator, you might want to try this:
https://github.com/bazelbuild/rules_apple/issues/980#issuecomment-738645357
This would let you build a TensorFlowLiteC slice for arm64 simulator (haven't had a chance to try it myself). Then, you could add the build output to the "Frameworks and Libraries" list of your framework project.

@greenzeal I could create a dummy project as you described and confirmed that I could reproduce.
Do you really need to build your framework for arm64 simulator?
If not, you could go to the target build settings, and add arm64 as an excluded architecture for Any iOS Simulator SDK for both debug and release configs.

If you have a pressing need to build for arm64 simulator, you might want to try this:
https://github.com/bazelbuild/rules_apple/issues/980#issuecomment-738645357
This would let you build a TensorFlowLiteC slice for arm64 simulator (haven't had a chance to try it myself). Then, you could add the build output to the "Frameworks and Libraries" list of your framework project.

Excluding arm64 is a temporary option indeed.
I had tried with a patch, but had no luck yet to make it work. Will try again definitely.

Excluding arm64 is a temporary option indeed.
I had tried with a patch, but had no luck yet to make it work. Will try again definitely.

What is the current situation of the issue? @yyoon @teijeong @jdduke
I have tried to create TensorFlowLiteC.xcframework following the gist shared by @greenzeal (message) and steps described by @mrosales . I have created TensorFlowLiteC Swift Package and TensorFlowLite Swift Package (which depends on the former, TensorFlowLiteC) and successfully built, run and archived my app. However, it has failed at Upload to App Store stage (you can find error logs at the end). It would be nice to have official TensorFlowLiteC.xcframework to depend on. Most of the popular packages such as firebase have SPM support for their binaryTargets.
I have used official TensofFlowLiteC.framework distributed in Cocoapods and lipo to create simulator and ios frameworks, and added an Info.plist having relevant fields for a framework. Note that I had to separate TensorFlowLiteC.xcframework to its own package to work around some build issues on Xcode.
Errors
I have tried several workarounds to eliminate some errors like first two below.
I applied the solution in swift forums to eliminate:
ERROR ITMS-90680: "Invalid directory. The bundle Payload/MyApp.app/PlugIns/TensorFlowLiteC.framework is not contained in a correctly named directory. It should be under "Frameworks"."

and it seems worked out.
And, I am not sure if it is relevant but I tried to set ENABLE_BITCODE=NO for my project to get rid of the error below:
ERROR ITMS-90635: "Invalid Mach-O Format. The Mach-O in bundle "MyApp.app/Frameworks/TensorFlowLiteC.framework" isn’t consistent with the Mach-O in the main bundle. The main bundle Mach-O contains arm64(bitcode), while the nested bundle Mach-O contains arm64(machine code). Verify that all of the targets for a platform have a consistent value for the ENABLE_BITCODE build setting."

But as you can see below, now I have something similar and several others:
ERROR ITMS-90635: "Invalid Mach-O Format. The Mach-O in bundle "MyApp.app/Frameworks/TensorFlowLiteC.framework" isn’t consistent with the Mach-O in the main bundle. The main bundle Mach-O contains arm64(machine code), while the nested bundle Mach-O contains arm64(machine code). Verify that all of the targets for a platform have a consistent value for the ENABLE_BITCODE build setting."
ERROR ITMS-90171: "Invalid Bundle Structure - The binary file 'MyApp.app/Frameworks/TensorFlowLiteC.framework/TensorFlowLiteC' is not permitted. Your app can’t contain standalone executables or libraries, other than a valid CFBundleExecutable of supported bundles. Refer to the Bundle Programming Guide at https://developer.apple.com/go/?id=bundle-structure for information on the iOS app bundle structure."
ERROR ITMS-90124: "The binary is invalid. The executable 'MyApp.app/Frameworks/TensorFlowLiteC.framework/TensorFlowLiteC' has type 'OBJECT' that is not valid. Only 'EXECUTE' is permitted."
ERROR ITMS-90125: "The binary is invalid. The encryption info in the LC_ENCRYPTION_INFO load command is either missing or invalid, or the binary is already encrypted. This binary does not seem to have been built with Apple's linker."
ERROR ITMS-90210: "Missing load commands. The executable at 'MyApp.app/Frameworks/TensorFlowLiteC.framework' does not have the necessary load commands. Try rebuilding the app with the latest Xcode version. If you are using third party development tools, contact the provider."```

Also looping in @miaout17

Is there any progress?

Please?

Hi, we start distributing XCFrameworks through Cocoapods for nightly builds https://github.com/CocoaPods/Specs/tree/master/Specs/1/6/0/TensorFlowLiteC/0.0.1-nightly.20220513
https://github.com/CocoaPods/Specs/tree/master/Specs/3/f/8/TensorFlowLiteSelectTfOps/0.0.1-nightly.20220513. Will release 2.9.0 with XCFrameworks soon.

@yishuangP thank you!

We've already released 2.9.0 with XCFrameworks (https://github.com/CocoaPods/Specs/tree/master/Specs/1/6/0/TensorFlowLiteC/2.9.0, https://github.com/CocoaPods/Specs/tree/master/Specs/3/f/8/TensorFlowLiteSelectTfOps/2.9.0, https://cocoapods.org/pods/TensorFlowLiteSwift, https://cocoapods.org/pods/TensorFlowLiteObjC). Note that this version doesn't have a ios-sim-arm64 slice yet but the nightly releases already have it.

Should this be closed given that the request was for a Swift package and there isn't a Swift package yet? It sounds like you're planning to add one which is great, I'm just wondering what issue is tracking that work?

From the original title ("Swift Package Manager package or XCFramework", emphasis mine), my guess is there is currently no issue tracking adding Swift Package Manager support.
It is possible for SPM to declare a dependency on a remotely hosted xcframework zip file (see the Declare a Binary Target in the Package Manifest section of that document), but we'd need someone who owns this repo to specify where the built XCFramework is hosted. That information is not currently documented – I recommend re-opening this issue until that documentation has been created @yishuangP

Hi sorry for the late reply, we do have plan to support SPM but it is deprioritized recently. We'll work on it asap. I'll reopen this issue to track this work

It is possible for SPM to declare a dependency on a remotely hosted xcframework zip file (see the Declare a Binary Target in the Package Manifest section of that document), but we'd need someone who owns this repo to specify where the built XCFramework is hosted. That information is not currently documented

Yes we do have a remotely hosted xcframework zip file and it is public, you can see it in the cocoapods podspec. For example, in https://github.com/CocoaPods/Specs/blob/master/Specs/1/6/0/TensorFlowLiteC/2.9.0/TensorFlowLiteC.podspec.json, the link is https://dl.google.com/tflite-release/ios/prod/tensorflow/lite/release/ios/release/15/20220517-175828/TensorFlowLiteC/2.9.0/c88775ef9b845ba7/TensorFlowLiteC-2.9.0.tar.gz. Let me know if it doesn't work.

Hi @yishuangP! Thank you for your work on this, and I appreciate your re-opening this issue.
I tried adding the following binaryTarget to a Package.swift file as a target:
.binaryTarget(
    name: "TensorFlowLiteC",
    url: "https://dl.google.com/tflite-release/ios/prod/tensorflow/lite/release/ios/release/15/20220517-175828/TensorFlowLiteC/2.9.0/c88775ef9b845ba7/TensorFlowLiteC-2.9.0.tar.gz",
    checksum: "03acd48ec2a40b90b8572b3725e7c3cded2d402c")

I generated the checksum by running shasum on the downloaded tar.gz file (if this is the intended distribution mechanism, the checksum should likely be added to documentation somewhere).
However, when I tried to build the project I got the following error:

unsupported extension for binary target 'TensorFlowLiteC'; valid extensions are: 'zip'

I'm also curious: is there a plan to release the TensorFlowLiteSwift package as an XCFramework (or Swift Package)?

I've created a sample Swift Package repo you can play with here:
TensorFlowLitePackageTest.zip
Once you unzip it, you'll need to put the unarchived TensorFlowLiteC-2.9.0 folder in the same directory as the Package.swift

Hi Dan, really sorry that you have to download the tar.gz file and unpack it to use it in Swift package. Hopefully we'll release it as Swift package soon. For TensorFlowLiteSwift, we just publish the sources and it depends on other XCFrameworks (like TensorFlowLiteC). Eventually we'll release it as Swift package.

Makes sense to me! Looking forward to the TensorFlowLiteSwift Swift Package, but can start playing with building-from-source and utilizing an XCFramework in the meantime. Thank you for the explanation – again, I appreciate your work on this!

Hi. I was wondering if there is any progress on this? specifically, an official URL that we can add as a depndency that holds the latest stable release in XCFramework form? or perhaps any updats on where you stand as far as adding SPM support?

Can we get an update on this? As more time passes the need for this library to be available as Swift Package grows.

1

+10000

+1

I've created a sample Swift Package repo you can play with here: TensorFlowLitePackageTest.zip
Once you unzip it, you'll need to put the unarchived TensorFlowLiteC-2.9.0 folder in the same directory as the Package.swift

I'm trying to create something like this for an app I'm working on. I can import this properly, and get it to build, but it seems to have an issue with installation. I'm not sure if this is a signing issue or what not. Have you managed to actually get this working in an app and not just as a package?

I've created a sample Swift Package repo you can play with here: TensorFlowLitePackageTest.zip
Once you unzip it, you'll need to put the unarchived TensorFlowLiteC-2.9.0 folder in the same directory as the Package.swift

@dfed
I'm trying to create something like this for an app I'm working on. I can import this properly, and get it to build, but it seems to have an issue with installation. I'm not sure if this is a signing issue or what not. Have you managed to actually get this working in an app and not just as a package?

I've only tried this in a package – I'm still using Cocoapods for this part of my app.

I've created a sample Swift Package repo you can play with here: TensorFlowLitePackageTest.zip
Once you unzip it, you'll need to put the unarchived TensorFlowLiteC-2.9.0 folder in the same directory as the Package.swift

@dfed
I'm trying to create something like this for an app I'm working on. I can import this properly, and get it to build, but it seems to have an issue with installation. I'm not sure if this is a signing issue or what not. Have you managed to actually get this working in an app and not just as a package?

I've only tried this in a package – I'm still using Cocoapods for this part of my app.

Ok Thanks.
@yishuangP and the rest of the team. There seems to be overwhelming need to allow users to pull these packages via SPM. What can we do to get this up and running? all we really need is a way to declare this as a dependancy so we can distribute our own packages.
Unfotunately trying to add the xcframeworks as binaryTargets will compile, but fails to install on device, with obscure errors.

+1
All our other dependencies are handled via SPMs packages and tflite is our last cocoapods link. Is there anything the community here can do to support? As previous posters already said: there is an overwhelming need to pull in those packages via SPM. This ticket was opened 895 days ago and has lots of reactions, upvotes, and community members that have offered their help. It would be great to find a solution for this.

@mmaetzler and others. I have put together a package, which builds but won't install onto a device for some obscure reason. If anyone wants to try and tackle this, so that we might have a community workaround for now:
https://github.com/arctop/TensorFlowLiteSwiftPackage
Right now, I've just copied over the files from TF that are needed. If we can get it to run, we can add some automation to pull the latest releases and what not.

@arctop-sk, did you manage to fix the issue? Tried your approach and got the code signing issue.
I plan to try to codesign tensorflow xcframeworks manually, have you tried this approach?

@arctop-sk, did you manage to fix the issue? Tried your approach and got the code signing issue.
I plan to try to codesign tensorflow xcframeworks manually, have you tried this approach?

Nope. I haven't continued down this path due to time restrictions.
If you manage to properly sign and run that, please let us know. Might also have an issue getting this archived and validated for the store, so I would test that.
Good luck!

Unfortunately, no luck with signing the xcframework manually.
As far as I understood, the issue is that frameworks for a particular architecture are missing Info.plist. I've tried adding those and got another code-sign validation error.

+1

+1

@pkgoogle @yishuangP  This is absolutly nonsensical.
A feature request that has been open for 3 years, isn't even a big feature or a development of any sort of core functionality, but rather just a bit of putting together an automated script and a wrapper, uploading it here, and releasing us all from these workarounds.
There are a dozen of engineers here, that with the proper access and support will HAPPILY do the work and contribute. In fact some of us have already spent hours trying to work this for the good of the community.
Furthermore, the lack of communication of something that is trivial, with anyone from the team not having updated or participated in this thread for almost a full year, is downright unacceptable.

Maybe this can help you https://github.com/kewlbear/TensorFlowLiteC

@pkgoogle @yishuangP Is there a way to get an official response from the team? Even just to say "Sorry we won't do anything for the time being" (though an ETA would be much appreciated). This issue has been tagged with "Awaiting tensorflower" for 4 months now...

+1

If you manage to export the current pod as .xcframework making it a SPM is very easy

If you manage to export the current pod as .xcframework making it a SPM is very easy

It is very easy to roll out your own SPM.
Problem is that won't correctly sign and you cannot deploy that to the App Store.
Works great in debug but useless in actual apps.

+1

+1

Is there any hope to have SPM anytime in the future?

I gave up on TensorFlow creating SPM, I created my own SPM repo that passes code signing issues in App Store and uploads just fine https://github.com/tareksabry1337/TensorFlowLiteC

@tareksabry1337 Could you elaborate on how you did it? Maybe the guys here will try and implement it if they have the solution (if anyone at tensorflow actually still reads this thread - cc @yishuangP)

+1

+1

+54

+1
Can we get an update on this please?
It's been quite a while since SPM has been out.

Can we please add SPM support to at least Tensorflow Lite Swift [https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/swift]
Since everyone is now starting to use SPM and dropping cocoapods, this will really help all iOS devs.
(Its been 4years since the issue was opened and I am willing to help)
Thanks!
Attn: @yyoon @yishuangP @pkgoogle

Literally having to choose CoreML because of this limitation.
SPM support is more critical to iOS ecosystem than it might seem.
CocoaPods has gone into maintenance mode earlier this year and we will only see more people move to SPM from here on as that's the only option around with an actual feature and maintenance roadmap.

I wonder if this SPM issue is part of the reason why there is no SPM yet? I think it would be impossible at the moment to bundle the binary frameworks inside the SPM because of that issue.

@mmaetzler I literally packaged TensorFlow under SPM and it's currently in app that's on the AppStore, I can't fathom why such simple request is being completely ignored.

@tareksabry1337 I did the same. I have an app submitted to the store with my own (internal) tflite SPM. But I was not able to create an SPM with TensorFlowLiteSelectOps because the binary size requires us to use git lfs and SPM does not support git lfs files. So if your model requires a good chunk of the select ops, then you are out of luck.

@pkgoogle Continuing the conversation here as you suggested, https://github.com/tensorflow/tensorflow/issues/78787#issuecomment-2448261271
Could you please tell us if we can expect this to be resolved in the next 6 months? Its crazy that after 4 years and so many developers requesting & waiting for so long, there is no clarity. There must be some valid reason for this, right?
If there is a reason, please let us know so we don't waste each other's time.
Thanks.

HI @lothrop,
Thanks for raising this issue. Are you aware of the migration to LiteRT? This transition is aimed at enhancing our project's capabilities and providing improved support and focus for our users. As we believe this issue is still relevant to LiteRT we are moving your issue there. Please follow progress here.
Let us know if you have any questions. Thanks.


Fix mark private orgs feature

For https://github.com/sindresorhus/refined-github/issues/2830.
Details: https://github.com/sindresorhus/refined-github/issues/2830#issuecomment-591111897.

Thank you for the fast PR!

I'm not seeing these URLs yet; this change would break the feature for me.

@fregante The change has been reverted on my GitHub as well. I have updated the replace regex to work in both cases if it were to be restored.


chore(release): v0.6.5

:robot: I have created a release beep boop
0.6.5 (2023-11-02)
Performance Improvements

add config for netlify avoid refresh page become page 404. (b528ed6)
use vite-react-swc. (b528ed6)

Miscellaneous Chores

merge gh-87. (b528ed6)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/riccox/meilisearch-ui/releases/tag/v0.6.5 :sunflower:


Apply about screen design

Idea Description
please apply cool design for about screen following figma instruction.
Reference images and links
Figma

I think there is a lot of misunderstanding about this issue.
This issue is for the whole screen design. And the other issues are just for click listeners.

I would like to do this task.

Thanks! Assigned 👍

Sorry, as stated in the PR summary section, some of the implementation is still incomplete. 🙇♂


fixed git push issues

What is this?
hotfix to fix the fact we autocommited git to git, which caused issues on push to remote
Why?
the push to remote was broken
How did you test to ensure no regressions?
tested the banana init locally, then pushed to remote
If this is a new feature what is one way you can make this break?

deployed as 0.0.12


Added a New "Preloader" Matched with the Theme of The Website

Description
Hey @sunny0625 @khushi-joshi-05

Closes #1573

This PR introduces a new preloader matched to the website theme i.e. Foods and Resturants, to enhance the user experience on the website. The preloader is displayed as a full-screen overlay with a white background

Added a video as the preloader content.
The preloader is styled to cover the entire viewport with a white background.
Included CSS transitions to handle the fade-out effect of the preloader smoothly.
Implemented a timeout to hide the preloader after 5 seconds, with a fade-out effect.
Integrated the preloader across all the pages of the website to maintain consistency
Made it responsive

https://github.com/user-attachments/assets/59e674d7-88cb-4bca-9765-291e99467f68
Responsive
https://github.com/user-attachments/assets/f8686647-928d-42f7-b828-75fa92afda43

Type of PR

[x] Feature enhancement

Screenshots / videos (if applicable)
[Attach any relevant screenshots or videos demonstrating the changes]
Checklist

[x] I have gone through the contributing guide
[x] I have updated my branch and synced it with project main branch before making this PR
[x] I have performed a self-review of my code
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[x] I have commented my code, particularly in hard-to-understand areas.

Hey @sunny0625 @khushi-joshi-05 pls  take a look and review it

@sunny0625 @khushi-joshi-05 review this PR

@sunny0625 Please merge this PR

@sunny0625 Please take a look at this PR and Merge it!

@sunny0625 @khushi-joshi-05 I have resolve the conflicts pls merge this

@sunny0625 If possible could you please increase the level? as per considering the changes!


theme issue

I use cattpuccin theme and your plugin show this

with ugly selection color. What theme color you show and how I can change it?

We can just decide on a better color to use by default


fix: LSDV-4836: Conditional annotation with visibleWhen choice-selected works with Taxonomy

PR fulfills these requirements

[X] Commit message(s) and PR title follows the format [fix|feat|ci|chore|doc]: TICKET-ID: Short description of change made ex. fix: DEV-XXXX: Removed inconsistent code usage causing intermittent errors
[ ] Tests for the changes have been added/updated (for bug fixes/features)
[ ] Docs have been added/updated (for bug fixes/features)
[X] Best efforts were made to ensure docs/code are concise and coherent (checked for spelling/grammatical errors, commented out code, debug logs etc.)
[X] Self-reviewed and ran all changes on a local instance (for bug fixes/features)

Change has impacts in these area(s)
(check all that apply)

[ ] Product design
[X] Frontend

Describe the reason for change
Make visibleWhen work with Taxonomy.
What does this fix?
Currently Taxonomy does not support visibleWhen, which throws errors and does not work in the Labelling config.
Support visibleWhen in Taxonomy so that it can be used to control other Tag's visibility conditions.
Does this PR introduce a breaking change?
(check only one)

[ ] Yes, and covered entirely by feature flag(s)
[ ] Yes, and covered partially by feature flag(s)
[X] No
[ ] Not sure (briefly explain the situation below)

What level of testing was included in the change?
(check all that apply)

[ ] e2e
[ ] integration
[ ] unit

Which logical domain(s) does this change affect?
Taxonomy

Codecov Report
Patch coverage: 96.29% and project coverage change: +0.07 :tada:

Comparison is base (cdc27bd) 65.35% compared to head (d50fad7) 65.43%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1273      +/-   ##
==========================================
+ Coverage   65.35%   65.43%   +0.07%
==========================================
  Files         411      412       +1
  Lines       25583    25621      +38
  Branches     6592     6607      +15
==========================================
+ Hits        16719    16764      +45
+ Misses       8864     8857       -7

Impacted Files
Coverage Δ

src/env/development.js
0.00% <0.00%> (ø)

src/tags/control/Choices.js
76.74% <ø> (ø)

src/mixins/SelectedChoiceMixin.js
96.29% <96.29%> (ø)

src/mixins/Visibility.js
78.37% <100.00%> (-0.20%)
:arrow_down:

src/regions/Result.js
80.71% <100.00%> (+0.27%)
:arrow_up:

src/tags/control/Taxonomy/Taxonomy.js
95.65% <100.00%> (+0.49%)
:arrow_up:

... and 3 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Visual Aids Install issue

(book_env) aarons-mbp:Hands-On-Data-Analysis-with-Pandas-2nd-edition kineret$ pip3 install visual-aids
ERROR: Could not find a version that satisfies the requirement visual-aids (from versions: none)
ERROR: No matching distribution found for visual-aids

Hi there. It looks like you aren't following the setup instructions. The visual_aids package is part of this repository, so you need to have the files on your machine locally and point pip to where you have it. It looks like you have the files locally, so you can try this, but the best would be to follow the instructions in chapter 1:
pip install ./visual-aids

Thanks.


Problem when importing baal 1.7.0 with Python 3.7

Describe the bug
You say that baal supports Python >= 3.7.  However, when I try to import baal 1.7.0, I get the following error:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\...\lib\site-packages\baal\__init__.py", line 1, in <module>
    from .active.dataset import ActiveLearningDataset
  File "C:\...\lib\site-packages\baal\active\__init__.py", line 4, in <module>
    from .active_loop import ActiveLearningLoop
  File "C:\...\lib\site-packages\baal\active\active_loop.py", line 12, in <module>
    from .dataset import ActiveLearningDataset
  File "C:\...\lib\site-packages\baal\active\dataset\__init__.py", line 1, in <module>
    from .numpy import ActiveNumpyArray
  File "C:\...\lib\site-packages\baal\active\dataset\numpy.py", line 6, in <module>
    from baal.active.dataset.base import SplittedDataset
  File "C:\...\lib\site-packages\baal\active\dataset\base.py", line 2, in <module>
    from typing import Union, List, Optional, Any, TYPE_CHECKING, Protocol
ImportError: cannot import name 'Protocol' from 'typing' (C:\Users\manow\miniconda3\envs\py37_ml_TfGpuMain\lib\typing.py)

Protocol is not included in the typing library in Python 3.7.
To Reproduce
Install baal 1.7.0 in a Python 3.7 environment.  Start Python and try to import baal.
Some advices:

Please don't use custom data or custom paths.
Use random arrays or even np.ones, np.zeros.
The example should run with Baal (and deps) alone.
Should be Python3 compatible.
Should not be OS specific.
The file should reproduce the bug with *high fidelity.
Link to a gist would be appreciated.

Expected behavior
baal imports properly
Vesion (please complete the following information):

OS: Windows 10
Python: 3.7
Baal version: 1.7.0

Additional context
Add any other context about the problem here.

Ah good catch! We only support 3.8.
I updated the README in this commit
Thank you!


[Service Bus] Porting Fulfill promise when disconnect occurs after drain is requested from track 1

More info - #8552 and #8885

/azp run js - servicebus - tests

/azp run js - servicebus - tests

/azp run js - servicebus - tests

/azp run js - servicebus - tests

Failed on windows (once).
Ran this test in a loop for 100 times locally, the test failed thrice with the same error.(97/100 success rate)

/azp run js - servicebus - tests

/azp run js - servicebus - tests

/azp run js - servicebus - tests

/azp run js - servicebus - tests

/azp run js - servicebus - tests

Browser integration test step is shown as failed even though none of the tests failed.
It seems unrelated to this PR.

/azp run js - servicebus - tests

275fda2 fixes the issue with the skipped browser tests.


[ogre] make zziplib, freeimage, assimp, freetype and less strict resource manager as feature

Describe the pull request

What does your PR fix?

Some Ogre functionalities are not mandatory and they require additional dependencies which could be problematic, most notably zzip which have a GPL licensing. This PR add features, with default set "as before", so people could easily choose which dependencies they could afford.
Another feature is the ability to use less strict resource manager, so some legacy code would work, while preserving fast lookup for new code.

Which triplets are supported/not supported? Have you updated the CI baseline?

All currently supported triplets should still be supported.

Have you tested these features?

Have you tested these features?

We don't use it, but, as I said, the previous version of the package did build them. I've just added a way to deactivate them.

Hi @dweckmann
I tested this features on my local and failed with the following error:
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
DirectX9_D3DX9_LIBRARY (ADVANCED)
    linked by target "RenderSystem_Direct3D9" in directory F:/ogre/vcpkg/buildtrees/ogre/src/eddf310f0b-6ab1152694.clean/RenderSystems/Direct3D9

Note: I installed ogre[d3d9,java,python,csharp,overlay,zziplib,strict].
Could you please look into this?

DirectX9

Well, this option come from the original recipe and the fail seems linked to CMake not finding Directx 9 libraries. Not sure I can do anything, because it is either a cmake issue or platform issue. BTW, I have no access right now to a dev environment with the old directx 9  libraries installed. Maybe should we contact the original author of the port file ?

Note: I installed ogre[d3d9,java,python,csharp,overlay,zziplib,strict].
Could you please look into this?

Hello again. I can confirm the behavior is the same prior my PR. [d3d9] only works when cmake could find the DirectX9 headers. I guess is it perfectly normal that you encounters the quoted error and this is linked your environment so it is not really an"issue".
What should I do now ?

Note: I installed ogre[d3d9,java,python,csharp,overlay,zziplib,strict].
Could you please look into this?

Hello again. I can confirm the behavior is the same prior my PR. [d3d9] only works when cmake could find the DirectX9 headers. I guess is it perfectly normal that you encounters the quoted error and this is linked your environment so it is not really an"issue".
What should I do now ?

Thanks for your further investigation about this issue.
Have you tested other features on your machine? Do they work fine?

Thanks for your further investigation about this issue.
Have you tested other features on your machine? Do they work fine?

Have you tested other features on your machine? Do they work fine?

As I said, I've really tested that the additional features are not built when not used. Our project build and run correctly when using ogre[core]. I assume, since the optional features where already there before, that the previous contributor have thoroughly tested the now optional features. However, I have at least tested the build of some feature combination [core,assimp,zziplib,assimp], but not the usage because I don't have another project that make use of them. BTW this is what the CI test since the optional features are default features.
Please, tell me if I can do something else.

Have you tested other features on your machine? Do they work fine?

As I said, I've really tested that the additional features are not built when not used. Our project build and run correctly when using ogre[core]. I assume, since the optional features where already there before, that the previous contributor have thoroughly tested the now optional features. However, I have at least tested the build of some feature combination [core,assimp,zziplib,assimp], but not the usage because I don't have another project that make use of them. BTW this is what the CI test since the optional features are default features.
Please, tell me if I can do something else.

Thanks for your response.
Since these new features are set as default features, there is no need to test them now.
I have checked that ogre[d3d9] is broken with the same error on current master branch.
So this PR is good for me.

Thanks for your response.
Since these new features are set as default features, there is no need to test them now.
I have checked that ogre[d3d9] is broken with the same error on current master branch.
So this PR is good for me.

I have checked that ogre[d3d9] is broken with the same error on current master branch.

Thanks,
To be clear, ogre[d3d9] is not technically broken, just you must have d3d9 available on your platform, which require some setup.

I have checked that ogre[d3d9] is broken with the same error on current master branch.

Thanks,
To be clear, ogre[d3d9] is not technically broken, just you must have d3d9 available on your platform, which require some setup.

I experienced a failure in a downstream CI (see https://github.com/robotology/robotology-superbuild-dependencies-vcpkg/pull/39) due to the fact that this PR enabled by default the strict resource mode, while before by default the strict mode was set to OFF. While I see the use of the strict option, I think that for the time being it could make sense for it to be opt-in by default for backward compatibility, as any user of vcpkg's ogre port until now could be relying on it, while any user interested in strict mode can enable it if necessary. What do you think @dweckmann @vicroms ?

I experienced a failure in a downstream CI (see https://github.com/robotology/robotology-superbuild-dependencies-vcpkg/pull/39) due to the fact that this PR enabled by default the strict resource mode, while before by default the strict mode was set to OFF. While I see the use of the strict option, I think that for the time being it could make sense for it to be opt-in by default for backward compatibility, as any user of vcpkg's ogre port until now could be relying on it, while any user interested in strict mode can enable it if necessary. What do you think @dweckmann @vicroms ?

I experienced a failure in a downstream CI (see robotology/robotology-superbuild-dependencies-vcpkg#39) due to the fact that this PR enabled by default the strict resource mode, while before by default the strict mode was set to OFF. While I see the use of the strict option, I think that for the time being it could make sense for it to be opt-in by default for backward compatibility, as any user of vcpkg's ogre port until now could be relying on it, while any user interested in strict mode can enable it if necessary. What do you think @dweckmann @vicroms ?

For more visibility, I have opened a PR in https://github.com/microsoft/vcpkg/pull/15789 .

I experienced a failure in a downstream CI (see robotology/robotology-superbuild-dependencies-vcpkg#39) due to the fact that this PR enabled by default the strict resource mode, while before by default the strict mode was set to OFF. While I see the use of the strict option, I think that for the time being it could make sense for it to be opt-in by default for backward compatibility, as any user of vcpkg's ogre port until now could be relying on it, while any user interested in strict mode can enable it if necessary. What do you think @dweckmann @vicroms ?

For more visibility, I have opened a PR in https://github.com/microsoft/vcpkg/pull/15789 .


Correctly pick platform

Uses the library name currently https://github.com/segmentio/integration-amplitude/blob/master/lib/mapper.js#L91.
Should use context.os or something equivalent.

In our use case, we have clients (android, ios, and web) as well as the an API sending data to amplitude via segment.
When requests hit our API we know where they came from (which device, app version, etc) and I would like to pass that platform value through to amplitude.
If you were to just use context.os, things could get weird trying to distinguise an ios app via mobile web requests coming from ios devices.

@f2prateek would it be possible to use something like context.device.type? We have a couple of customers who are seeing values like 'analytics-ruby' for platform, but are expecting Android, iOS, Web.

yup context.device.type will work - although it's not populated by a.js afaik. https://segment.com/docs/spec/common/

@f2prateek we could fall back on something like Web if the value doesn't exist for device type, something like this:
platform: facade.proxy('context.device.type') || 'Web'
What do you think? If that looks good I'll submit a PR for it

I wouldn't assume that it's from the web if it's not provided. None of our server side libraries send it by default and all those would get attribtued to web.

@f2prateek is there a way for the server-side libraries to know what the platform is? To give some context, say we have a customer that uses the Javascript integration. They have a user that visits their site on an Android mobile browser. For events logged, the platform should be Web. device type, device family, those would describe the Android device. OS would show like the browser.

You'd want to use a combination of context.userAgent, context.device.type, context.os.name.
The flow would be something like :

Use context.device.type if provided.
Use context.os.name if provided.
Try to parse from userAgent
Optional: map known values of context.library.name (i.e. analytics.js -> web, analytics-android -> android, etc.)

You'll have to map context.os.name to the appropriate platform as well.

@f2prateek thanks for pushing this live. We're still seeing events with platform analytics-node and such. Does that mean those events just don't have the device info? Or is there another library I need to update? Thanks!

Yup that would mean those events don't have the device info.


Set the tftp directory owner/group via attributes

Description
This change allows the tftp directory owner/group values to be modified by an attribute. Defaults are modified to be in line with OS defaults (per discussion https://github.com/chef-cookbooks/tftp/pull/5).
Check List

[X] All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
[ ] New functionality includes testing. (not applicable)
[ ] New functionality has been documented in the README if applicable (not applicable)
[X] The CLA has been signed. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD

1.6.0 released


Support mypy 0.910 to 0.930 including CI tests

Some cleanup from #3573, also stealing logic from #3572.
I normally don't think changing the versions of mypy which the plugin is compatible with would be a problem in a minor release, but since we've already made two beta release of v1.9, breaking mypy compatibility now would be ugly.

Right now, pypi is partially down, hence all the failures, see here.

@christianbundy, @PrettyWood I'd love you feedback on this.
I still need to get tests passing once pypi is back up and add a change description.

LGTM. I'll merge tomorrow.


⚠ Jenkins Plugins Artifactory has degraded performance

In 8ff2494, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1424 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 5da7068 after 12 minutes.


SIWE on React package not persist after reload

Link to minimal reproducible example
https://github.com/reown-com/web-examples/tree/main/dapps/appkit-siwe/react
Summary
After the user completes the signature and gets verified, the session should persist even if the user reloads the page, as managed by Next.js with NextAuth. However, currently, it does not, and the sign modal reopens.
(I tried with the previous version and the latest of all the packages on the examples repo)
List of related npm package versions
"@reown/appkit": "1.5.2",
"@reown/appkit-adapter-wagmi": "1.5.2",
"@reown/appkit-siwe": "1.4.1",
"@wagmi/core": "^2.13.
"wagmi": "^2.13.2"

Screencast from 2024-11-27 19-39-01.webm

please check our docs and examples again. We fix them. sorry for the inconvenience . 🫶

https://docs.reown.com/appkit/react/core/siwe
https://github.com/reown-com/web-examples/tree/main/dapps/appkit-siwe


Empty Sack of Holding glitch

We are seeing from time to time that the Empty Sack of Holding will simply eat whatever inventory/spawner it picks up. Its not always a constant. THOUGH, we are seeing and hearing about
this from servers running the pack using Sponge.

I'm not seeing this in just vanilla. I know you said its not constant though.
What modpack are you using so i can download and test it there?

Mech & Magic.
Im almost willing to bet sponge I just want sure of your position on sponge support.

The sponge team opened a bug report with me before& they helped me out; they are great people.  I want to support it but i havent installed it on my machine or ever used it.  Ill download the pack from AT launcher and then look into if i can rig up some sort of localhost sponge.

Awesome sounds great!  Love cyclic and I know my users love using it.  Only real issue we have seen was this.  Thanks.

I have it loaded on  a sponge server, i used this for setup https://docs.spongepowered.org/stable/en/server/getting-started/implementations/spongeforge.html . so far ive been playing off and on hand havent seen the sack of storage destroy something yet.  i want to trach down this issue if i can anyway.  Is there anything in your logs that would help? or maybe a video of it happening.

I know something close to, if you have a inventory item in your sack and try to pick another one (if you have two sacks) this gonna make one of the sacks disapear and give you the sack with the last inventory item picked up.


Added some changes for new wagtail version.

Why close you this???

My mistake, I went back to make your pull.

I have to do again?


MyQ "Taking over 10 seconds" log file errors

Home Assistant release with the issue:
0.70.1

Last working Home Assistant release (if known):
Unknown, have always had this issue
Operating environment (Hass.io/Docker/Windows/etc.):
Ubuntu machine in virtual environment

Component/platform:
https://www.home-assistant.io/components/cover.myq/

Description of problem:
Log file has been loaded with these errors for as long as I can remember.  Door usually works fine, and reports accurately, other than rare occasions.  But this makes it very difficult to actually find issues in the logs.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
cover:
  - platform: myq
    username: !secret email_address
    password: !secret myq_pass
    type: chamberlain

Traceback (if applicable):

Additional information:
Log:
Attached (Formatting wasn't liking me)
Log Snippet.txt

I'm having this issue recently too. It has become increasingly prevalent over the past few weeks. Prior to 0.67 it was very rare.

Have been getting this forever.    Hope someone has a solution.


GodSVG Optimization roadmap

GodSVG has a pretty poor performance. It will easily choke when SVGs get even slightly complex. It's time to establish what the current situation is, and propose some ideas on how to improve things.
This PR is going to focus on...
The fundamental bottlenecks
There are four performance-intensive tasks that are fundamental to GodSVG and not really able to be worked around in any way. That is...

Parsing and highlighting
Rendering and displaying the SVG
Rendering the handles and contours
Building the inspector

There are a number of very situational ways to optimize each of these by avoiding full inspector rebuilds, re-rendering the full SVG, etc. but this post won't focus on those.
This post focuses not on how to avoid the worst case scenarios, but how to optimize them.
Parsing and highlighting
GodSVG works simultaneously with a text representation of the SVG and a native representation using RefCounteds: Tags and Attributes. Attributes and Tags can be a few different types, but they share most of the relevant logic. There are parsers for translating between text and these classes. Path attributes also have two representations: the value (a string for the path def) and the path commands, and there are separate parsers for translating between the two. As for highlighting, there is a parser too that returns a big color map dictionary.

Attributes have a string representation and usually an extra representation, like an Array[PathCommand] for pathdata. The parsing between the two is always done when one or the other changes.
SVG parsing from classes to text is done when any changes are made to the SVG that didn't come from the CodeEdit, otherwise, a parsing from text to classes is done.
Anytime the SVG text changes, SVGHighlighter runs.

In general, these tasks currently don't seem to be a major bottleneck. Some developments might make the parsing slightly more intensive, but right now it pales in comparison to the other bottlenecks and I don't have major ideas for optimizing it.
Rendering and displaying the SVG
Things get interesting here: GodSVG is based on Godot, which uses the open-source ThorVG library. GodSVG redraws the graphic every time anything about it is modified. Moreover, the graphic is redrawn whenever you zoom in.
What does the redrawing do? It uses Image.load_svg functions. For scaling, it takes the current zoom level, fixes edges' alpha, and uses a TextureRect to display the texture. The texture is also limited to 4096x4096 even though Godot allows up to 16384x16384.
Why the alpha edge fixing and size limit? Because otherwise, transparent pixels are partially white.
The cleanest way to optimize this would be to remove the alpha fixing. If I do though, white antialiasing around the edges starts to become clearly visible which is unacceptable. (https://github.com/godotengine/godot/issues/82579)

So, this could yield something like a 70% performance boost for free, but I'm not sure if it's a fundamental issue or on Godot's end.
Rendering only the part of the SVG visible on the screen. Even if the above point were implemented, no matter how you twist and turn it, even a 16x16 graphic at max zoom will need to be turned into a 1024x1024 image before being turned into a texture. An image this big will still be slow, not to speak of bigger SVGs.
When you zoom this much, only a part of the texture will be on screen. So a solution would be to give the TextureRect information about the visible rect of the screen. This would need to be updated while you're panning, however, and this might mean the graphic would get redrawn every frame. And what this means is that GodSVG should be able to render a viewport-sized version of the SVG every frame. Can it do that? No.
This is not the case now, but when it becomes the case, this should be possible by working out which rect is visible in SVG coords and using it in display_texture.gd to parse an identical SVG with different viewbox, height and width, and display that. This would also allow infinite scaling to be technically possible.
Multithreading: With or without the above points, the drawing can be multithreaded, by splitting the graphic to be displayed into a few strips and letting WorkerThreadPool work on them.
However, having these cutoffs is quite problematic because of a bug in ThorVG whose fix isn't yet added in Godot. https://github.com/thorvg/thorvg/issues/1785
So all of these points should result in big gains, but they are unfortunately all blocked by upstream bugs in one way or another.
Rendering the handles and contours
In the past, this was done with a whole second SVG, now it's done with Godot draw commands. As far as I see, the performance of this approach is now very solid even for massive SVGs.
So in general, this shouldn't be a priority to optimize, however, because it's redrawn every time anything about the SVG changes or when it's zoomed in, it can still tank the performance a little for really huge SVGs.
Since huge SVGs aren't a priority to support, the only optimization I'm looking out for is the way curve handles are drawn. Currently, they are added to an array that's pretty much ready to be used for draw_multiline(). But it's not, because draw_multiline() can't do antialiasing, so it uses a loop of draw_line(). This Godot PR might solve this: https://github.com/godotengine/godot/pull/84497.
A more complex optimization, which I tried to do once and failed, would be to bundle subsequent path operations into polylines. I wouldn't try it again.
Building the inspector
The inspector with its tag and attribute editors easily becomes expensive. It's rebuilt when the SVG layout changes, that is, when tags are added, deleted, or moved. It's also redrawn when the SVG text changes. This means it's mainly important to optimize to improve the experience of using the code editor. But it also becomes the main bottleneck for big SVGs.
Unlike rendering and displaying, this seems to unavoidably cost at least O(n). And there seems to be no way to limit the amount of operations done in any way. There are ways to improve the lag spikes by doing the rebuild more gradually, but I want to first investigate if I can improve the baseline performance.
For now, no idea how to do that. I'm investigating it, I've even done some small steps towards resolving it, and it's high-priority.
Conclusion
In short, drawing and parsing don't need major optimizations currently. The major issues are in rendering the graphic and in populating the inspector. There are several ideas on how to optimize the rendering, but most of them are blocked by upstream bugs. For the inspector, right now I have no idea.
A caveat to this is the fact the profiler doesn't seem to work correctly during lag spikes. It keeps saying that lag spikes are caused by physics, even though there are no physics in the project whatsoever. So I've not been able to pinpoint the reasons for the inspector lag, parsing, or drawing any more precisely. This was all discovered with microbenchmarking.

Edited a bit to update some things.

Edited significantly because of a few developments.

Closing this issue, as everything it points out is in a bunch of TODO comments now, that I occasionally check.


Flutter integration_test package conflict with my code and cannot be resolved by tweaking package versions - so the whole app cannot run

Hi, thanks for the wonderful flutter framework! I have an app (with null-safety), and want to use integration_test package to do some testing. The pubspec.yaml looks like the following, see below. However, it has errors when flutter pub get (see below).
I cannot use the non-null-safety version (2.x) of archive package, because if I do so, my app code will fail to run in null safety mode! I can accept that my tests run in non-null-safety mode, but I cannot tolerate my app code run in non-null-safety mode.
Thanks for any suggestions!
Steps to Reproduce

Run flutter create bug.
Update the files as follows: pubspec.yaml

dependencies:
  archive: ^3.1.2 # I use archive package in my app code.
  ...

dev_dependencies:
  build_resolvers: ^2.0.0
  build_runner: ^1.11.5
  flutter_test:
    sdk: flutter
  integration_test:
    sdk: flutter
  ...

flutter pub get

Expected results:
no error
Actual results:
Because every version of flutter_driver from sdk depends on archive 2.0.13 and my_app depends on archive ^3.1.2, flutter_driver from sdk is forbidden.

I cannot use the non-null-safety version (2.x) of archive package, because if I do so, my app code will fail to run in null safety mode! I can accept that my tests run in non-null-safety mode, but I cannot tolerate my app code run in non-null-safety mode.

Logs

N/A

no errors

[✓] Flutter (Channel stable, 2.0.4, on Mac OS X 10.15.7 19H1030 darwin-x64, locale zh-Hans-CN)
    • Flutter version 2.0.4 at /Users/tom/opt/flutter
    • Framework revision b1395592de (6 周前), 2021-04-01 14:25:01 -0700
    • Engine revision 2dce47073a
    • Dart version 2.12.2
    • Pub download mirror https://pub.flutter-io.cn
    • Flutter download mirror https://storage.flutter-io.cn

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/tom/Library/Android/sdk
    • Platform android-30, build-tools 29.0.3
    • ANDROID_SDK_ROOT = /Users/tom/Library/Android/sdk
    • Java binary at: /Users/tom/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/201.7042882/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.4, Build version 12D4e
    • CocoaPods version 1.10.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Users/tom/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/201.7042882/Android
      Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] IntelliJ IDEA Ultimate Edition (version 2020.3)
    • IntelliJ at /Users/tom/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.56.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.22.0

[✓] Connected device (2 available)
    • iPhone 11 (mobile) • 14CE77E4-C2C5-4267-B565-520FA291E237 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-4 (simulator)
    • Chrome (web)       • chrome                               • web-javascript • Google Chrome 90.0.4430.212

• No issues found!

You're using flutter_driver from Flutter 2.0.4, which is using archive 2.0.13.
https://github.com/flutter/flutter/blob/b1395592de68cc8ac4522094ae59956dd21a91db/packages/flutter_driver/pubspec.yaml#L22
Unfortunately the bump to the null safe dependency of archive from flutter_driver didn't make it into Flutter 2.0.4 (see #71379), sorry about that. You should be able to try the dev/beta channel, or wait for the next upcoming stable release of flutter.

@jiahaog My app is in production environment, so definitely we cannot use dev/beta. And when will next stable release come? Testing is so important that we want it right now :/

You can try just using a dependency override for archive, e.g.
  dependency_overrides:
    archive: ^3.0.0

@fzyzcjy
Above solution should resolve the issue you are facing.

@dnfield @darrenaustin Thanks, this solves the problem!


MM-32664 - Initial implementation of Backup Restore Tool

Summary

Initial implementation of Backup Restore Tool.
The tool uses pg_dump for backup and pg_restore for restoration. The backup file is uploaded and fetched from S3 compatible storage.
Commands:

backup-restore-tool backup [FLAGS]
backup-restore-tool restore [FLAGS]

Ticket Link

https://mattermost.atlassian.net/browse/MM-32664

One general question: this is intended to be used manually, at least for now, correct?

It is mainly intended to be run as K8s job, created by Provisioner. The Dockerfile and other build related stuff will come in next PR.


Release request for 1.0.1

We started using swift-async-algorithm in ci.swift.org to build Swift SDK generator now. (https://github.com/apple/swift/pull/72728)
However, we currently use main branch instead of a tagged revision to checkout a revision including https://github.com/apple/swift-async-algorithms/commit/6ae9a051f76b81cc668305ceed5b0e0a7fd93d20 fix.
This is not ideal situation for the build stability, so could you publish 1.0.1 of swift-async-algorithm including https://github.com/apple/swift-async-algorithms/commit/6ae9a051f76b81cc668305ceed5b0e0a7fd93d20?
Thanks in advance :pray:

cc @FranzBusch @phausler

It has been tagged now: https://github.com/apple/swift-async-algorithms/releases/tag/1.0.1


Choice rules

Enabling the use of choice rules in Alpha.

Coverage increased (+2.2%) to 72.688% when pulling da9e48aa9fae56545681e84c4179f6a3eeb7d353 on choice_rules into 34b13ad00e3ca1bab2315e217a8ff2ebbafe0c31 on master.

Codeclimate indicating failure, because a method requires more than 25 lines really seems a bit over the top.

Nice! After looking at your latest changes regarding transformations, I also thought that choice rules were the next thing to approach.

This branch needs a rebase. Should I do that or will you?

I will try to merge master and resolve the conflicts. Thanks for the offer, though.

@lorenzleutgeb I think the code is ready to merge, please have a look.

Regarding the second note, I think the current problem is that we actually want to say "rule head" but since this could so far only be an atom, we said Atom instead in all places. When realizing disjunction, the same problems will surface again: a disjunction of atoms may occur as a rule head but all those Atom methods cannot be used on it. Choices and (a limited form of) disjunction can be removed by translating the input program, thus it seems to me, that we could just introduce an empty interface RuleHead and both Atom and ChoiceHead simply extend/implement the RuleHead interface.

I don't think that we would benefit from an empty interface. We'd probably end up with instanceof checks anyway. But we could introduce a Head type which then translates to a list of Atoms, where a non-disjunctive head translates to just a singleton list.

Done.

The AppVeyor build failed hitting some timeout and I cannot restart the build. @lorenzleutgeb could you restart it?

Addressed the raised issues, please check again.


Uncaught TypeError: Cannot read property 'enable' of undefined

[Enter steps to reproduce:]

...
...

Atom: 1.33.0 x64
Electron: 2.0.11
OS: Ubuntu 18.04.1
Thrown From: atom-clock package 0.1.16
Stack Trace
Uncaught TypeError: Cannot read property 'enable' of undefined
At /home/sky/.atom/packages/atom-clock/lib/atom-clock-view.js:140

TypeError: Cannot read property 'enable' of undefined
    at AtomClockView.setTooltip (/packages/atom-clock/lib/atom-clock-view.js:140:50)
    at /packages/atom-clock/lib/atom-clock-view.js:35:12
    at emitter.on (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:335221)
    at Function.simpleDispatch (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:1189772)
    at Emitter.emit (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:1191213)
    at Config.emitChangeEvent (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:337709)
    at Config.endTransaction (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:332480)
    at Config.transact (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:332099)
    at Config._resetSettings (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:333732)
    at Config.resetUserSettings (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:333499)
    at disposables.add.applicationDelegate.onDidChangeUserSettings.e (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:1:644526)
    at ipcMessageEmitter.on.t (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:1:660858)
    at Function.simpleDispatch (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:1189772)
    at Emitter.emit (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:11:1191213)
    at EventEmitter._ipcMessageEmitter.get_ipcRenderer.on (/snap/atom/207/usr/share/atom/resources/app/static/<embedded>:1:657482)
    at emitThree (events.js:136:13)
    at EventEmitter.emit (events.js:217:7)

Commands

Non-Core Packages
atom-clock 0.1.16
colorblind-wombat-dark-syntax 0.6.0
Hydrogen 2.6.0
javascript-snippets 1.2.1
teletype 0.13.3

Thank you for reporting the issue. Unfortunately, I was not able to reproduce it on my system. Is it still present in the latest Atom version (1.35.1)?

Will check and report.

As there are no further updates on this, I will close.


Problem with C api

Visual Studio linker can't find the C api function.
I think this is a problem like mentioned here:
https://githubmemory.com/repo/lu-zero/cargo-c/issues/58

which version of cargo-c are you using ? How are you producing the library?

sorry, I think, I used an old one.
I now switched to the latest one. At the moment the linker is missing GetUserProfileDirectory().

OK, I added UserEnv.lib to my project and now I can save AVIF images with rav1e encoding :-)
But it takes much time to code. Is this library a release or debug version?

Still a little bit slow, but I managed to save pictures with metadata and transparency.

But it takes much time to code. Is this library a release or debug version?

You should give us more information cargo cbuild, as cargo build does, defaults to debug profile, you may pass --release to request a release build.
cargo cinstall, as cargo install does, defaults to --release. Which version of cargo-c are you using?

sorry, but I work with Visual Studio with Windows.
So I am not able to compile the library myself. I used the download version from here.

Given that this is a support question rather than a bug, I'm going to close this. If you still need help, I recommend you stop by our IRC channel #daala on Libera.Chat. You can also use a web client to join with a web browser.


Make killing shims more resilient

Currently, we send a single SIGKILL to the shim process
once and then we spin in a loop where we use kill(pid, 0)
to detect when the pid has disappeared completely.
Unfortunately, this has a race condition since pids can be reused causing us
to spin in an infinite loop when that happens.
This PR removes this check and simply relies on kill succeeding or not.
This fixes https://github.com/containerd/cri/issues/1427

Attaching some more relevant information that we found when debugging this issue
$ pidof containerd
1567
$ strace -fp 1567 -e trace=kill
strace: Process 1567 attached with 301 threads
[pid 13252] kill(13737, SIG_0)          = 0
[pid 11864] kill(13737, SIG_0)          = 0
[pid  6042] kill(13737, SIG_0)          = 0
[pid 11864] kill(13737, SIG_0)          = 0
[pid  6042] kill(13737, SIG_0)          = 0
[pid  7318] kill(13737, SIG_0)          = 0
[pid  8004] kill(13737, SIG_0)          = 0
[pid 11864] kill(13737, SIG_0)          = 0
[pid  8004] kill(13737, SIG_0)          = 0
[pid 20957] kill(13737, SIG_0)          = 0
[pid 11864] kill(13737, SIG_0)          = 0
[pid 11864] kill(13737, SIG_0)          = 0
[pid  6042] kill(13737, SIG_0)          = 0
[pid  6042] kill(13737, SIG_0)          = 0

and
$ ps aux | grep 13737
root     13737  0.0  0.0 110356  6576 ?        Sl   07:18   0:00 containerd-shim -namespace k8s.io -workdir /var/lib/container-runtime/containerd/io.containerd.runtime.v1.linux/k8s.io/051b2f90455d240b29f3130682e851c0e7fd6bc0e64121b5e77bcfb516c49b18 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/containerd

Containerd logs are full of
ERROR [2020-04-22T12:43:34.234344073Z] containerd.io/containerd/containerd: "StopPodSandbox for \"d50318c73c890f6a829fab36a980332b87b8133ff5f5b6c72f8693c3e1140d49\" failed" error="rpc error: code = Canceled desc = failed to stop container \"4f1dd102b497be8256c290e4da16128dd36c340575a8d0c36edd2ccae51e2f05\": an error occurs during waiting for container \"4f1dd102b497be8256c290e4da16128dd36c340575a8d0c36edd2ccae51e2f05\" to be killed: wait container \"4f1dd102b497be8256c290e4da16128dd36c340575a8d0c36edd2ccae51e2f05\": context canceled"

I uploaded the stack trace dump from a containerd in this state here: https://gist.github.com/ashrayjain/f1bac2cc5bec2af5445268a3e1bc7fef

Hey @Zyqsempai , I think I signed my commit already. Is there anything else needed to satisfy CI?

You will need to rebase on master to pass CI once we merge #4206. Sorry--upstream change in golangci-lint broke our dev-tool-install script.

@estesp no problem, done!

Codecov Report

Merging #4204 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4204   +/-   ##
=======================================
  Coverage   38.34%   38.34%
=======================================
  Files          90       90
  Lines       12728    12728
=======================================
  Hits         4881     4881
  Misses       7181     7181
  Partials      666      666

Flag
Coverage Δ

#windows
38.34% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e094d36...46ce63c. Read the comment docs.

Codecov Report

Merging #4204 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4204   +/-   ##
=======================================
  Coverage   38.34%   38.34%
=======================================
  Files          90       90
  Lines       12728    12728
=======================================
  Hits         4881     4881
  Misses       7181     7181
  Partials      666      666

Flag
Coverage Δ

#windows
38.34% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e094d36...c3d0845. Read the comment docs.

Is this prblem also relevant to the v2 shim?

I spent some time trying to look into that late last week; my understanding is that the v2/ code doesn't have this same flow of waiting for a complete exit.
My problem with the PR is that it effectively is changing the semantics of "waitForExit" to "keepTryingToKillForAwhile" :) I'm not sure if @crosbymichael has had a chance to look at this as he is more intimately familiar with this code.

@estesp to provide some more info here. We ran the killsnoop bpf tool (https://github.com/iovisor/bcc/blob/master/tools/killsnoop.py) to inspect and track all the kill signals being sent by containerd and the responses they were getting from the kenel.
Here is the output from an occurrence of this issue.
$ journalctl -u killsnoop | grep 10516
killsnoop.sh[922]: TIME      PID    COMM             SIG  TPID   RESULT
killsnoop.sh[922]: 08:39:02  24136  containerd       9    10516  0
killsnoop.sh[922]: 08:39:02  24136  containerd       0    10516  0
killsnoop.sh[922]: 08:39:02  24136  containerd       0    10516  0
killsnoop.sh[922]: 08:39:02  24357  containerd       0    10516  0
killsnoop.sh[922]: 08:39:02  419    containerd       0    10516  0
killsnoop.sh[922]: 08:39:02  30071  containerd       0    10516  0

$ strace -fp $(pidof containerd) -e trace=kill
strace: Process 1595 attached with 224 threads
[pid   699] kill(10516, SIG_0)          = 0
[pid   699] kill(10516, SIG_0)          = 0
[pid   699] kill(10516, SIG_0)          = 0
[pid  4991] kill(10516, SIG_0)          = 0
[pid 26715] kill(10516, SIG_0)          = 0
[pid 14796] kill(10516, SIG_0)          = 0
[pid 14796] kill(10516, SIG_0)          = 0
[pid   699] kill(10516, SIG_0)          = 0
[pid   699] kill(10516, SIG_0)          = 0
[pid   699] kill(10516, SIG_0)          = 0
[pid 26715] kill(10516, SIG_0)          = 0
[pid 26715] kill(10516, SIG_0)          = 0
[pid 14796] kill(10516, SIG_0)          = 0
[pid 14796] kill(10516, SIG_0)          = 0
[pid 26715] kill(10516, SIG_0)          = 0

As you can see, containerd sent a kill -9 to the pid 10516 and got back 0 as the result, however this process was not killed and kept running until manual action was taken. :(
Would this PR be acceptable if we made it clear that the method was retrying the kill as opposed to just waiting for exit?

@estesp @BenTheElder @crosbymichael @fuweid is there anything else i can do to help push this forward?

I spent a few minutes looking at the call path (services/tasks/local.go Delete -> runtime/v1 tasks Delete calls into the shim.KillShim which leads to signalShim with SIGKILL) trying to understand if there is a reasonable place to capture a failed kill and insert some form of retry. If the ctx in that flow had a timeout then that would be another route, but apparently it doesn't by default, hence the original problem of spinning forever on the unix.Kill with signal "zero".
It does turn out that this code (signalShim) is only called with SIGKILL as the StopShim function is never used from any caller in containerd at the moment. Otherwise, I assume the same problem might be observed with StopShim using SIGTERM; an infinite wait if the process doesn't respond/exit on SIGTERM.
Seems maybe the right path is to make signalShim a time-out based kill function (since all it is being used is for killing pids) that retries itself until success or exits with error after a timeout or a specified # of retries? Thoughts @fuweid?

I am not sure why kill -9 doesn't work and still want to find the root cause. :)
Back to the issue, for shim v1, the containerd doesn't retry kill -9.
In CRI plugin, the task.Delete is called by CRI plugin event handler when the container init task has been exited. And task.Delete will call t.shim.KillShim(ctx) and we can see containerd send kill -9 signal.
If the task.Delete fails, the container status will not be updated to exited and CRI plugin will try it. Checkout the following code, if the t.shim.Delete return errors or c.ShimInfo return errors, the kill -9 will not be called.
// Delete the task and return the exit status
func (t *Task) Delete(ctx context.Context) (*runtime.Exit, error) {
        rsp, shimErr := t.shim.Delete(ctx, empty)
        if shimErr != nil {
                shimErr = errdefs.FromGRPC(shimErr)
                if !errdefs.IsNotFound(shimErr) {
                        return nil, shimErr
                }
        }
        t.tasks.Delete(ctx, t.id)
        if err := t.shim.KillShim(ctx); err != nil {
                log.G(ctx).WithError(err).Error("failed to kill shim")
        }
        if err := t.bundle.Delete(); err != nil {
                log.G(ctx).WithError(err).Error("failed to delete bundle")
        }
        if shimErr != nil {
                return nil, shimErr
        }
        t.events.Publish(ctx, runtime.TaskDeleteEventTopic, &eventstypes.TaskDelete{
                ContainerID: t.id,
                ExitStatus:  rsp.ExitStatus,
                ExitedAt:    rsp.ExitedAt,
                Pid:         rsp.Pid,
        })
        return &runtime.Exit{
                Status:    rsp.ExitStatus,
                Timestamp: rsp.ExitedAt,
                Pid:       rsp.Pid,
        }, nil
}

func (c *Client) signalShim(ctx context.Context, sig syscall.Signal) error {
        info, err := c.ShimInfo(ctx, empty)
        if err != nil {
                return err
        }
        pid := int(info.ShimPid)
        // make sure we don't kill ourselves if we are running a local shim
        if os.Getpid() == pid {
                return nil
        }
        if err := unix.Kill(pid, sig); err != nil && err != unix.ESRCH {
                return err
        }
        // wait for shim to die after being signaled
        select {
        case <-ctx.Done():
                return ctx.Err()
        case <-c.waitForExit(pid):
                return nil
        }
}

So I think we should check the event handler in CRI plugin and what error we meet.
Maybe related to https://github.com/containerd/containerd/pull/4198 because I check the log from https://gist.github.com/ashrayjain/f1bac2cc5bec2af5445268a3e1bc7fef and found
goroutine 3324776 [semacquire, 139 minutes]:
sync.runtime_Semacquire(0xc00185b618)
	/home/travis/.gimme/versions/go1.12.16.linux.amd64/src/runtime/sema.go:56 +0x3b
sync.(*WaitGroup).Wait(0xc00185b610)
	/home/travis/.gimme/versions/go1.12.16.linux.amd64/src/sync/waitgroup.go:130 +0x67
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func2(0xc00185b610, 0xc002015b00)
	/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:246 +0x2d
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
	/home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:245 +0x1a7

@ashrayjain would you mind to provide more information about CRI plugin event log? like failed to handle container TaskExit event something? Thanks
cc @estesp

ping @ashrayjain

Hey @fuweid, apologies for the delay.
I'm waiting for another repro of this issue on our cluster to provide you with some more info on this.

@fuweid @estesp So we just had another repro of this issue.
Here is what we found:
Containerd was stuck trying to kill pid 9750
$ strace -fp $(pidof containerd) -e trace=kill
strace: Process 1603 attached with 131 threads
[pid  6034] kill(9750, SIG_0)           = 0
[pid  6034] kill(9750, SIG_0)           = 0
[pid  6034] kill(9750, SIG_0)           = 0
[pid 28558] kill(9750, SIG_0)           = 0
[pid 28558] kill(9750, SIG_0)           = 0
[pid 30413] kill(9750, SIG_0)           = 0
[pid 30413] kill(9750, SIG_0)           = 0
[pid 30413] kill(9750, SIG_0)           = 0
[pid 30413] kill(9750, SIG_0)           = 0
[pid 30413] kill(9750, SIG_0)           = 0

Based on the killsnoop tool, we know that containerd sent a kill -9 to this pid and got back a 0 exit code
11:21:14  12062  containerd       9    9750   0      R (running)
11:21:14  12062  containerd       0    9750   0      R (running)
11:21:14  5450   containerd       0    9750   0      S (sleeping)
11:21:14  28560  containerd       0    9750   0     S (sleeping)
11:21:14  28560  containerd       0    9750   0     S (sleeping)

On trying to look at the containerd logs, we saw
"shim containerd-shim started" address=/containerd-shim/21e89a11376d496299190e55578927b75988ab9664374ee924299553fdbfe45d.sock debug=false pid=9750

Based on how the code generates 21e89a11376d496299190e55578927b75988ab9664374ee924299553fdbfe45d, we were able to confirm that this was for sandbox id 3c6ced659501c69a1b0342c46cbb2c47c9dde045cf92f22fe82e96a4a1e22068
func (b *bundle) shimAddress(namespace string) string {
	d := sha256.Sum256([]byte(filepath.Join(namespace, b.id)))
	return filepath.Join(string(filepath.Separator), "containerd-shim", fmt.Sprintf("%x.sock", d))
}

and k8s.io/3c6ced659501c69a1b0342c46cbb2c47c9dde045cf92f22fe82e96a4a1e22068 hashes to 21e89a11376d496299190e55578927b75988ab9664374ee924299553fdbfe45d
For this pod, if we run crictl inspectp, we get the following redacted info
{
  "status": {
    "id": "3c6ced659501c69a1b0342c46cbb2c47c9dde045cf92f22fe82e96a4a1e22068",
...
    "state": "SANDBOX_READY",
  },
  "info": {
    "pid": 9768,
    "processStatus": "deleted",
...
}

There was no process on this host with pid 9768 at this time.
Additionally, for the sandbox in question, we also have
11:21:14.276625531Z] containerd.io/containerd/containerd: "shim reaped" id=3c6ced659501c69a1b0342c46cbb2c47c9dde045cf92f22fe82e96a4a1e22068

which is at the exact time we observe the first kill -9 in killsnoop above.
Trying to find the "current" pid 9750 on the host, we find
$ pstree -pa -G -s 9750
systemd,1
  └─containerd,1603
      └─containerd-shim,9371 -namespace k8s.io -workdir ...
          └─tini,9438 -s -- ...
              └─java,9661 ...
                  └─{java},9750

So this is actually a thread being used by an unrelated container.
From this container's inspect output, we have
    "createdAt": "2020-05-23T11:21:13.699473306Z",
    "startedAt": "2020-05-23T11:21:14.141006424Z",

So in summary, what appears to have happened here is that containerd tried to kill the shim with pid 9768/9750 for sandbox 3c6ced659501c69a1b0342c46cbb2c47c9dde045cf92f22fe82e96a4a1e22068 (where one of those pids was probably a thread?) and succeeding in doing so, however at roughly  the same time, a different container started (the java container above) and one of the threads in that container got the same pid (9750). This meant that containerd got stuck in its kill -0 loop since from the kernel's perspective, that pid exists, even though containerd just deleted it.
@fuweid it seems when you first suggested that the pid might be getting reused, you were on to something ;)
Do you folks have ideas on how to handle this situation better?

Was the
    "startedAt": "2020-05-23T11:21:14.141006424Z",

part of the output of 'crictl inspectp' ?
(I ran the same command on a node with cri-o container but didn't see this field).
I wonder if the start time can be obtained and disambiguate whether the same process id was reused.

It was part of the crictl inspect output for the container

@ashrayjain  OK, It is pid-reuse issue... 😂
I think @tedyu 's idea is reasonable to prevent dead loop to kill -0.
But it there any log related to failed to handle container TaskExit event? It is the key to know why the StopContainer action always timeout. Thanks!

@fuweid we don't appear to have any logs like that in our logs when this happens.

@ashrayjain The >= 1.2.10 version containerd doesn't use TaskExit from containerd. Which version are you using right now?

@fuweid I see, we run containerd 1.3.3 currently.

@ashrayjain the code is here https://github.com/containerd/cri/blob/release/1.3/pkg/server/container_start.go#L145.
So, I am still thinking why it is timeout to update status. When it is timeout at first time, and record has been deleted from containerd. Next time, StopContainer will be successful because cri-plugin will ignore 404 error and update the status.

I think there are three solutions to fix this issue:

use shimv2 :)

change blocked mode to async mode by the following change, but if the reuse-shimv1-pid process is long-live, it will consume more cpu resource because waitForExit files syscall by 10ms.

From 4194b4a4a9f72ddaf050a2265823acf19b1a22eb Mon Sep 17 00:00:00 2001
From: Wei Fu<EMAIL_ADDRESS>Date: Wed, 27 May 2020 17:01:27 +0800
Subject: [PATCH] bugfix: run waitForExit in other goroutine

:)

Signed-off-by: Wei Fu<EMAIL_ADDRESS>---
 runtime/v1/shim/client/client.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/runtime/v1/shim/client/client.go b/runtime/v1/shim/client/client.go
index 116e8408..82436727 100644
--- a/runtime/v1/shim/client/client.go
+++ b/runtime/v1/shim/client/client.go
@@ -330,7 +330,7 @@ func (c *Client) signalShim(ctx context.Context, sig syscall.Signal) error {
 }

 func (c *Client) waitForExit(pid int) <-chan struct{} {
-	c.exitOnce.Do(func() {
+	go c.exitOnce.Do(func() {
 		for {
 			// use kill(pid, 0) here because the shim could have been reparented
 			// and we are no longer able to waitpid(pid, ...) on the shim
--
2.24.1

check starttime for process

From 47e3e530c5e70101acaf987e2349a437c0d518ab Mon Sep 17 00:00:00 2001
From: Wei Fu<EMAIL_ADDRESS>Date: Wed, 27 May 2020 16:21:13 +0800
Subject: [PATCH] bugfix: check starttime before kill -0

The containerd has to wait for containerd-shim to exit by calling
waitForExit. But in the time window between `kill -9` and `waitForExit`,
the original pid might be reused by other process and the `waitForExit`
is never end because the once function is called by blocked mode :).

`kill -0` is not harmful but it will consume more cpu resource if the
reusing-shimv1-pid process is long-running type.

To fix this isuee, we should get shimv1's starttime before `kill -9`,
and then check starttime in `waitForExit` at every single loop. If the
starttime changes, exit the `waitForExit`.

Signed-off-by: Wei Fu<EMAIL_ADDRESS>---
 runtime/v1/shim/client/client.go | 121 +++++++++++++++++++++++++------
 1 file changed, 100 insertions(+), 21 deletions(-)

diff --git a/runtime/v1/shim/client/client.go b/runtime/v1/shim/client/client.go
index 116e8408..474667d1 100644
--- a/runtime/v1/shim/client/client.go
+++ b/runtime/v1/shim/client/client.go
@@ -22,6 +22,7 @@ import (
 	"context"
 	"fmt"
 	"io"
+	"io/ioutil"
 	"net"
 	"os"
 	"os/exec"
@@ -29,14 +30,12 @@ import (
 	"strconv"
 	"strings"
 	"sync"
-	"syscall"
 	"time"

-	"golang.org/x/sys/unix"
-
 	"github.com/containerd/ttrpc"
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
+	"golang.org/x/sys/unix"

 	"github.com/containerd/containerd/events"
 	"github.com/containerd/containerd/log"
@@ -289,16 +288,6 @@ func (c *Client) IsAlive(ctx context.Context) (bool, error) {
 	return true, nil
 }

-// StopShim signals the shim to exit and wait for the process to disappear
-func (c *Client) StopShim(ctx context.Context) error {
-	return c.signalShim(ctx, unix.SIGTERM)
-}
-
-// KillShim kills the shim forcefully and wait for the process to disappear
-func (c *Client) KillShim(ctx context.Context) error {
-	return c.signalShim(ctx, unix.SIGKILL)
-}
-
 // Close the client connection
 func (c *Client) Close() error {
 	if c.c == nil {
@@ -307,7 +296,8 @@ func (c *Client) Close() error {
 	return c.c.Close()
 }

-func (c *Client) signalShim(ctx context.Context, sig syscall.Signal) error {
+// KillShim kills the shim forcefully and wait for the process to disappear
+func (c *Client) KillShim(ctx context.Context) error {
 	info, err := c.ShimInfo(ctx, empty)
 	if err != nil {
 		return err
@@ -317,29 +307,118 @@ func (c *Client) signalShim(ctx context.Context, sig syscall.Signal) error {
 	if os.Getpid() == pid {
 		return nil
 	}
-	if err := unix.Kill(pid, sig); err != nil && err != unix.ESRCH {
+
+	statData, err := ioutil.ReadFile(procStatPath(pid))
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return errors.Wrapf(err, "failed to get proc stat for pid %v", pid)
+	}
+
+	startTime, err := parseProcStartTime(string(statData))
+	if err != nil {
+		return err
+	}
+
+	// NOTE(fuweid): The pid might be other process in this time window.
+	// Well, it is possible if the shim is dead after we call the ShimInfo.
+	// Call ShimInfo again to make sure that pid is correct.
+	//
+	// The better solution for this issue is to use shimv2 instead of v1 :).
+	if _, err := c.ShimInfo(ctx, empty); err != nil {
+		return err
+	}
+
+	if err := unix.Kill(pid, unix.SIGKILL); err != nil && err != unix.ESRCH {
 		return err
 	}
+
 	// wait for shim to die after being signaled
 	select {
 	case <-ctx.Done():
 		return ctx.Err()
-	case <-c.waitForExit(pid):
+	case <-c.waitForExit(pid, startTime):
 		return nil
 	}
 }

-func (c *Client) waitForExit(pid int) <-chan struct{} {
-	c.exitOnce.Do(func() {
-		for {
+func (c *Client) waitForExit(pid int, startTime uint64) <-chan struct{} {
+	go c.exitOnce.Do(func() {
+		defer close(c.exitCh)
+
+		var statPath = procStatPath(pid)
+		var lctx = context.TODO()
+
+		for retry := 0; retry < 50; {
+			time.Sleep(10 * time.Millisecond)
+
+			statData, err := ioutil.ReadFile(statPath)
+			if err != nil {
+				if os.IsNotExist(err) {
+					log.G(lctx).Debugf("shim pid=%v has gone, stop waitForExit", pid)
+					return
+				}
+				log.G(lctx).WithError(err).Debugf("failed to read data from %v", statPath)
+
+				retry++
+				continue
+			}
+
+			st, err := parseProcStartTime(string(statData))
+			if err != nil {
+				log.G(lctx).WithError(err).Debugf("failed to parse stat data")
+
+				retry++
+				continue
+			}
+
+			if st != startTime {
+				log.G(lctx).Debugf("expected stattime of pid=%v is %v, but got %v when pid-reuse",
+					pid, startTime, st,
+				)
+				return
+			}
+
 			// use kill(pid, 0) here because the shim could have been reparented
 			// and we are no longer able to waitpid(pid, ...) on the shim
 			if err := unix.Kill(pid, 0); err == unix.ESRCH {
-				close(c.exitCh)
 				return
 			}
-			time.Sleep(10 * time.Millisecond)
 		}
 	})
 	return c.exitCh
 }
+
+// parseProcStartTime parses /proc/[pid]/stat data and returns start time of
+// pid.
+//
+// Based on https://www.man7.org/linux/man-pages/man5/proc.5.html description,
+// the starttime is the (22) field.
+func parseProcStartTime(data string) (uint64, error) {
+	var st uint64
+
+	i := strings.LastIndex(data, ")")
+	if i <= 2 || i >= len(data)-1 {
+		return st, errors.Errorf("invalid /proc/[pid]/stat data: %q", data)
+	}
+
+	// For example,
+	//
+	// 3204 (containerd) S 1 3204 3204 0 -1<PHONE_NUMBER> 8753 10878 0 0 2142 21404 4 7 20 0 20 0<PHONE_NUMBER>851904 13478<PHONE_NUMBER>3709551615<PHONE_NUMBER>4800<PHONE_NUMBER>7212<PHONE_NUMBER>06416 0 0 0<PHONE_NUMBER> 0<PHONE_NUMBER> 0 0 0 17 2 0 0 0 0 0<PHONE_NUMBER>5480<PHONE_NUMBER>6033<PHONE_NUMBER>7744<PHONE_NUMBER>14343<PHONE_NUMBER>14369<PHONE_NUMBER>14369<PHONE_NUMBER>14462 0
+	//
+	// The parts will be start at index 3.
+	parts := strings.Split(data[i+2:], " ")
+	if len(parts) <= 22-3 {
+		return st, errors.Errorf("invalid /proc/[pid]/stat data: %q", data)
+	}
+
+	if _, err := fmt.Sscanf(parts[22-3], "%d", &st); err != nil {
+		return 0, errors.Wrapf(err, "failed to parse start time")
+	}
+	return st, nil
+}
+
+func procStatPath(pid int) string {
+	return filepath.Join("/proc", strconv.Itoa(pid), "stat")
+}
--
2.24.1

The third one is complicated and 1-2 are easier. For existing shimv1 user, I would like to take option 2 because the logic doesn't change...
ping @AkihiroSuda @crosbymichael @dmcgowan @estesp @mikebrow PTAL

@fuweid i noticed that 1.4.0 will default runtime to v2. Are there any major changes to v2 between 1.3.x and 1.4.0? If it is ready for use with 1.3.x, we are happy to try and switch to v2 now

@ashrayjain It is hard to say what big change of shimv2 between two releases. Most of nit bugfixes. It is like shimv1. But I suggest to use shimv2 because it has better design, including resource clean. And the core code base is github.com/containerd/containerd/pkg/process, which shares between shimv1 and shim-runc-v2.
Sure, v2 is ready to be used in 1.3.x :)

@fuweid
Ok we can try to switch some of our k8s clusters to v2 and see how it goes :)
In the meantime, i'm happy to change this pr according to the second option you proposed. What do you think of including a time limit on the waitForExit?
and if we timeout, we just log a warning and assume the process is gone and continue?

What do you think of including a time limit on the waitForExit?
@ashrayjain SGTM 👍

Actually, if we make the waitForExit async, does it serve any purpose anymore?
Should we just remove waitForExit?

@fuweid please take a look when you get the chance :)

@fuweid would appreciate a look here when you get a chance.
Thanks for continuing to push this forward, I think we are almost there with the fix :)

@mikebrow @fuweid please take a look again. I switched it to rely on the timeout already present on the context.
I still kept the ticker because i think that's more idiomatic Go than using a time.Sleep for this.

Additionally, i've fixed the initial delay issue, so now there won't be an initial 10ms wait

LGTM
This is fine for v1 but the proper fix is to move to v2 for the runtime shim :)

Thanks for your first PR @ashrayjain !

@ashrayjain Thanks for your patience.

@containerd/containerd-release is it good to cherry-pick into release/1.3?

@fuweid is there another 1.3.x release scheduled in the near future?

@ashrayjain https://github.com/containerd/containerd/pull/4307 has been merged. No sure about the 1.3.5 release schedule :)


Error while refreshing data on the AKS dashboard

Describe the bug
I was able to install the Infrastructure and Governance dashboard, but when it comes to the AKS dashboard, I have not been successful in installing it. The bug is with the refresh stage, where there is an error somewhere in the M Code.
To Reproduce
Steps to reproduce the behavior:

Go to 'ccodashboard\dashboards\CCODashboard-AKS '
Open 'CCO AKS Dashboard 5.1'
Click 'Refresh All'
See error

Expected behavior
Data is expected to be fetched, which it actually is, as I can see it in the advanced editor, but the overall refresh fails, as can be seen in the screen shot
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [win10]
App: PowerBI Desktop
Version [N/A]

Session log
Additional context
Add any other context about the problem here.

Hi @Jakar2 the report needs to be updated because it was developed with an old Power BI version.  You can fix manually this issue selecting "upgrade report". I verified that all the API queries are working properly

I will release a new template version ASAP


Adding Square Root Algorithm

This new algorithm explores a very intricate part of critical thinking about a mathematical function we often take for granted. When you get down to the basics, creating a square root function is a very difficult thing to implement, and consequently is a good algorithm to try and implement. Many might wonder why we would do something like this when there are functions that already do this for us; however, it's about the critical thinking that goes behind this problem.

Can't run it to check at the moment. The logic seems alright. If there are any problems an issue can be opened!


Add pull request template and issue templates

This is an automatic batch pull request.

Pull-request updated, HEAD is now c8f378bddd4f1751777fbf26a0cae1814bf30c02


Is FFlag modification ui planned?

No, I don't really see it as necessary. If you do need to modify FFlags, add "ClientSettings\ClientAppSettings.json" to your mod folder, and you can set any FFlags you need there.


update pygame

pygame is not available in PyPi for Python 3. Installation instructions should be added.
If pygame gets/can be replaced, this issue becomes obsolete.
DoD:

installation instructions for (Python 3) pygame for at least one OS

Done implicitly via travis.yml in #3, but needs to be documented explicitly.


Revert "Update index.json"

Reverts elementary/mvp#392

Will revert when pulling new translations from Transifex.


Make FakeFeatureFlag overrides composable with FakeFeatureFlagsOverrideModule

In the current setup, all test feature flag overrides have to live in the same guice module. For large services we end up duplicating feature flag overrides in many places.
This change allows us to have one FakeFeatureFlagsModule and many FakeFeatureFlagsOverrideModule. Modules at any level can install their own overrides independently using the familiar api of FakeFeatureFlagsModule#withOverride

I am going to walk back the withOverrides deprecation to keep it open to installing the module multiple times (without being confusing) but will merge as-is to unblock