How to train custom models

How to Train Your Dataset.

https://github.com/facebookresearch/segment-anything/issues/5

All rights reserved.
This source code is licensed under the license found in the
LICENSE file in the root directory of this source tree.
import torch
from torch import nn
from torch.nn import functional as F
from typing import Any, Dict, List, Tuple
from .image_encoder import ImageEncoderViT
from .mask_decoder import MaskDecoder
from .prompt_encoder import PromptEncoder
class Sam(nn.Module):
mask_threshold: float = 0.0
image_format: str = "RGB"
def __init__(
    self,
    image_encoder: ImageEncoderViT,
    prompt_encoder: PromptEncoder,
    mask_decoder: MaskDecoder,
    pixel_mean: List[float] = [123.675, 116.28, 103.53],
    pixel_std: List[float] = [58.395, 57.12, 57.375],
) -> None:
    """
    SAM predicts object masks from an image and input prompts.

@zhenzi0322 , @SBoulanger , @HIMANSHUSINGHYANIA
Hey guys, 👋
Built a quick app to finetune SAM (https://app.instalabel.ai/) on custom data (for those who don't wanna go through the hassle of changing and debugging the code 😅). The app allows users to directly train a fine-tuned SAM model by following a few simple steps:
Log in or create an account
Upload around 15-20 samples from your dataset to the platform
Upload individual ground truth segmentation masks for each sample
Click 'Train annotator'. This will finetune SAM to the data you sent. May need to wait for 15-20 minutes for it to train. ⏳
Go to the 'Annotation' tab and upload any other images you want to try the finetuned SAM on. You can test it out with a few bounding box prompts. Otherwise, you can save the finetuned '.pth' file yourself to use in your projects by pressing the 'Download Trained Checkpoint' button in the annotation tab!
The app is still a little buggy and I'm working to fix them (like refreshing the page after uploading all ground truth masks before you can press the 'Train annotator' button). However, it should still be largely usable! 👍
Hoping this platform can help developers have a simple UI to get finetuned checkpoints for SAM and annotate their data with a fine-tuned Segment Anything Model 🎯
Hope it helps! 🙌

@aadilmehdis Is your source code for fine-tuning available somewhere?

@zhenzi0322 , @SBoulanger , @HIMANSHUSINGHYANIA
Hey guys, 👋
Built a quick app to finetune SAM (https://app.instalabel.ai/) on custom data (for those who don't wanna go through the hassle of changing and debugging the code 😅). The app allows users to directly train a fine-tuned SAM model by following a few simple steps:
Log in or create an account Upload around 15-20 samples from your dataset to the platform Upload individual ground truth segmentation masks for each sample Click 'Train annotator'. This will finetune SAM to the data you sent. May need to wait for 15-20 minutes for it to train. ⏳ Go to the 'Annotation' tab and upload any other images you want to try the finetuned SAM on. You can test it out with a few bounding box prompts. Otherwise, you can save the finetuned '.pth' file yourself to use in your projects by pressing the 'Download Trained Checkpoint' button in the annotation tab! The app is still a little buggy and I'm working to fix them (like refreshing the page after uploading all ground truth masks before you can press the 'Train annotator' button). However, it should still be largely usable! 👍 Hoping this platform can help developers have a simple UI to get finetuned checkpoints for SAM and annotate their data with a fine-tuned Segment Anything Model 🎯
Hope it helps! 🙌
Updates:

Most of the code used were from open-source repos already available earlier in this thread, (@giswqs)! Just whipped up a platform really quick to make it possible for users to directly download fine-tuned SAM checkpoints without having to deal with the hassle of setup.
Also received a few issues about how users can directly upload their individual ground truth segmentation masks for each sample. I used LabelMe for this. An example of a groundtruth mask is shown here.

📢 Update Note:

Thank you for the positive feedback and interest in our platform! I really appreciate the support. However, I want to address a concern regarding scalability and multiple users. While the platform initially worked well with a few users, the sudden increase in the number of users has exposed some limitations. I am actively working on scaling up the platform to accommodate the more users.

Hope that helps! Feel free to let me know if you guys are facing any other issues with the platform. Can try to address them in time!

📢 Update Note:
Hello everyone! 👋
I've scaled up the platform to handle a larger number of users and made sure that the platform is robust enough for a smoother experience. I've also added a feature that sends you an email notification when a job has started and when it has finished fine-tuning. 🔔 It might take a couple of minutes for the email to come through, so please hang tight!
Furthermore, in order to manage GPU load more efficiently, I am currently resizing all images to a uniform 256x256 resolution. I am working to support varied dimensions and will incorporate this feature soon.
I'd really appreciate it if you could take it for a spin! Any feedback or issues, big or small, can be addressed directly to me at<EMAIL_ADDRESS>Your input is extremely invaluable as I continue to improve the platform!
Thanks

Excuse me,are the original images uploaded at the same time as the labeled images? and how to name the labeled images?

Excuse me,
After uploading the original image, click the green button below the image to upload the corresponding annotated image?
But after I do that,nothing is happening.

@zhenzi0322 , @SBoulanger , @HIMANSHUSINGHYANIA
Hey guys, 👋
Built a quick app to finetune SAM (https://app.instalabel.ai/) on custom data (for those who don't wanna go through the hassle of changing and debugging the code 😅). The app allows users to directly train a fine-tuned SAM model by following a few simple steps:
Log in or create an account Upload around 15-20 samples from your dataset to the platform Upload individual ground truth segmentation masks for each sample Click 'Train annotator'. This will finetune SAM to the data you sent. May need to wait for 15-20 minutes for it to train. ⏳ Go to the 'Annotation' tab and upload any other images you want to try the finetuned SAM on. You can test it out with a few bounding box prompts. Otherwise, you can save the finetuned '.pth' file yourself to use in your projects by pressing the 'Download Trained Checkpoint' button in the annotation tab! The app is still a little buggy and I'm working to fix them (like refreshing the page after uploading all ground truth masks before you can press the 'Train annotator' button). However, it should still be largely usable! 👍 Hoping this platform can help developers have a simple UI to get finetuned checkpoints for SAM and annotate their data with a fine-tuned Segment Anything Model 🎯
Hope it helps! 🙌
Updates:

Most of the code used were from open-source repos already available earlier in this thread, (@giswqs)! Just whipped up a platform really quick to make it possible for users to directly download fine-tuned SAM checkpoints without having to deal with the hassle of setup.
Also received a few issues about how users can directly upload their individual ground truth segmentation masks for each sample. I used LabelMe for this. An example of a groundtruth mask is shown here.

📢 Update Note:

Thank you for the positive feedback and interest in our platform! I really appreciate the support. However, I want to address a concern regarding scalability and multiple users. While the platform initially worked well with a few users, the sudden increase in the number of users has exposed some limitations. I am actively working on scaling up the platform to accommodate the more users.

Hope that helps! Feel free to let me know if you guys are facing any other issues with the platform. Can try to address them in time!

Hi can we upload mask labeled? you mentioned this "Upload individual ground truth segmentation masks for each sample" but I only make manual annotation there. Please advise
Thank you

Hi @ramdhan1989,
Previously, uploading a mask was an option, but it led to several inconsistencies. To address this, I've updated the platform to support direct annotation. Before fine-tuning a model, you'll need to manually annotate a small set (5-10 images) from your dataset.
To expedite this initial annotation process, I've introduced AI-assisted annotation tools. Simply draw a bounding box using the AI-assisted tool (marked by a Wand Icon, as shown in the image), and it will automatically generate the polygon annotation.
After your initial annotations are set, click on 'Train Annotator'. Keep an eye on your emails for updates on the training status. Once training is complete, you have two options for annotating the rest of your dataset: use 'Instant Annotate' for quick, automated annotation of the entire dataset (accessible in the Annotate tab, indicated by a blue button above the images), or opt for interactive annotation using the 'Instant Annotate' tool within the annotation interface (icon with a wand with sparkles).
Feel free to reach out if you need further clarification or assistance.

I didn't receive an email with a download link after 15 minutes


Research Amazon EC2 and Azure server requirements and pricing

Research Amazon EC2 and Azure server requirements and pricing.
Either way we need something more powerful than our current VPS (1CPU, 1GB RAM, 20GB HDD, no virtual memory).
Determine the specifications needed and the costs associated with those specifications.
Research whether using EC2 or Azure vs some other VPS/Dedicated server provider is the best option.

Closing this as resolved.
Ended up going with an r3.2xlarge spot instance with an m3.xlarge front end. Fault tollerence is enabled, but realistically with solid bid prices, the spot should never go down.


Generated index.js for Cognito triggers is missing context & callback params

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
16.13.1
Amplify CLI Version
7.6.9
What operating system are you using?
Mac
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
No manual changes made
Amplify Categories
auth
Amplify Commands
add
Describe the bug
Adding the auth category with a Cognito Pre-signup or Post confirmation trigger generates an index.js that passes only the event param to each trigger (not context & callback):
exports.handler = async event => {
  await Promise.all(modules.map(module => module.handler(event)));
  return event;
}

Thus, the user's trigger code in custom.js cannot invoke the callback.
Expected behavior
The generated index.js that calls the trigger(s) should pass context & callback (in addition to event):
exports.handler = async (event, context, callback) => {
  await Promise.all(modules.map(module => module.handler(event, context, callback)));
  return event;
}

Reproduction steps

add auth
Choose Y for Lambda triggers question
Select Pre-signup or Post Confirmation
For "functionality" choose Create Your Own
Finish, then look at amplify/backend/function/<MyName>/src/index.js
Notice it only passes the event param (not context & callback)

GraphQL schema(s)

# Put schemas below this line

Log output

# Put your logs below this line

Additional information
No response

The callback should not be needed. Please see the Async handlers section of this page. It probably makes sense to make context available though.

The Lambdas should work fine with synchronous code. Ultimately, callbacks, Promises, and async/await are just different ways of doing the same thing in JavaScript.
I've changed this from a feature request to a bug based on your most recent comment. The custom trigger should be updated to use an async function instead of the callback, and the context argument should be available.


cmake: exclude external project build directories

Update

[x] Template - Update existing .gitignore template

Details
https://cmake.org/cmake/help/latest/module/ExternalProject.html
In practice, cmake uses a path like <project name>-prefix for building external projects.

@joeblau Hey, why are patch files preferred to editing the original file directly?

Because the original file is managed by GitHub.  If you run the the linter by running
./.travis/sync-github.sh
moban

You'll get a conflict with the canonical file from GitHub @ https://github.com/github/gitignore/blob/master/CMake.gitignore


Salary Policy created

Description
Salary policy created
Checklist:

[x] I have performed a self-review of my own code.

This is not needed now. Closing


upgrade simple-peer to fix issue with wrtc

Before with opts.wrtc:
webrtc-swarm $ node example.js
/home/substack/projects/webrtc-swarm/node_modules/simple-peer/index.js:419
      self.remoteAddress = remote.ipAddress
                                 ^

TypeError: Cannot read property 'ipAddress' of undefined
    at setActiveCandidates (/home/substack/projects/webrtc-swarm/node_modules/simple-peer/index.js:419:34)
    at /home/substack/projects/webrtc-swarm/node_modules/simple-peer/index.js:439:28
    at Array.forEach (native)
    at /home/substack/projects/webrtc-swarm/node_modules/simple-peer/index.js:434:11
    at /home/substack/projects/webrtc-swarm/node_modules/simple-peer/index.js:397:7
    at /home/substack/projects/node-webrtc/lib/peerconnection.js:311:7

but this was fixed upstream in simple-peer 6.0.1. The simple-peer 5 to 6 jump was related to removing automatic json parsing, which shouldn't affect webrtc-swarm.

out in 2.4.0


Travis - spark script should stop on error

What is this PR for?
There has been a few cases Travis fails to download Spark release but doesn't stop. Stopping would make it easier to track down and check the exit code.
What type of PR is it?
Improvement
Todos

[x] - Update spark download & start/stop script

Is there a relevant Jira issue?
N/A
How should this be tested?
Run Travis CI
Screenshots (if appropriate)
N/A
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

https://s3.amazonaws.com/archive.travis-ci.org/jobs/107893675/log.txt
$ ./testing/startSparkCluster.sh $SPARK_VER $HADOOP_VER
SPARK_HOME is /home/travis/build/apache/incubator-zeppelin/spark-1.3.1-bin-hadoop2.3

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
./testing/startSparkCluster.sh: line 59: /home/travis/build/apache/incubator-zeppelin/spark-1.3.1-bin-hadoop2.3/sbin/start-master.sh: No such file or directory
./testing/startSparkCluster.sh: line 63: /home/travis/build/apache/incubator-zeppelin/spark-1.3.1-bin-hadoop2.3/sbin/start-slave.sh: No such file or directory

travis_time:end:2d16bb10:start=1454972345135573912,finish=1454972360543622554,duration=15408048642

The command "./testing/startSparkCluster.sh $SPARK_VER $HADOOP_VER" failed and exited with 127 during .

Download from Apache mirror timeout still seem to be a big source of issues,
https://api.travis-ci.org/jobs/108686270/log.txt?deansi=true
+wget -q http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz

No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

I have a hack, what if instead of
wget -q http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz
we use
wget -q -r -np -N   http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz &
wget -q -r -np -N   http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz &
wget -q -r -np -N   http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz &
wget -q -r -np -N   http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz &
wget -q -r -np -N   http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz

Which will make wget download using 5 simultaneous connections, which may result in downloading it quicker.
or
THREADS=5
WGETFLAGS=’-r -np -N’
URL="http://mirror.tcpdiag.net/apache/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.3.tgz"
for i in `seq 1 $((THREADS-1))`
do
  Command=”$Command””wget $WGETFLAGS $URL”‘ & ‘
done
Command=”$Command””wget $WGETFLAGS $URL”
eval $Command

References: http://blog.netflowdevelopments.com/2011/01/24/multi-threaded-downloading-with-wget/

Good call, this might be better? http://stackoverflow.com/questions/7577615/parallel-wget-in-bash?answertab=votes#tab-top

Another download timeout
+wget -q http://mirror.sdunix.com/apache/spark/spark-1.3.1/spark-1.3.1-bin-hadoop2.3.tgz

No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

Another approach would be https://docs.travis-ci.com/user/caching/. It seems travis-ci capable of cache directory when it is configured to use container based infrastructure.

Right, ideally. My understanding is though caching is available only for the travis-ci.com which they charge for and container based infra cannot use sudo - neither seems to work with us at this point?

Not 100%sure, but my understanding is sudo: false enables container based infrastructure as well as cache.
For example, Apache Flink project looks like using cache capability of travis-ci https://github.com/apache/flink/blob/master/.travis.yml

@prabhjyotsingh this seems to fail quite a bit:
20:01:31,655 ERROR org.apache.zeppelin.ZeppelinIT:198 - Exception in ZeppelinIT while testAngularDisplay
org.openqa.selenium.ElementNotVisibleException: Element is not currently visible and so may not be interacted with
Command duration or timeout: 10 milliseconds
Build info: version: '2.48.2', revision: '41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06'
System info: host: 'testing-worker-linux-docker-77898bf2-3380-linux-5', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-40-generic', java.version: '1.7.0_76'
Session ID: b2de8969-6562-4b77-9902-910c5df7750e
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true, cssSelectorsEnabled=true, databaseEnabled=true, browserName=firefox, handlesAlerts=true, nativeEvents=false, webStorageEnabled=true, rotatable=false, locationContextEnabled=true, applicationCacheEnabled=true, takesScreenshot=true, version=31.0}]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:206)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:158)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:647)
	at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:326)
	at org.openqa.selenium.remote.RemoteWebElement.sendKeys(RemoteWebElement.java:121)
	at org.apache.zeppelin.ZeppelinIT.testAngularDisplay(ZeppelinIT.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: org.openqa.selenium.ElementNotVisibleException: Element is not currently visible and so may not be interacted with
Build info: version: '2.48.2', revision: '41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06'
System info: host: 'testing-worker-linux-docker-77898bf2-3380-linux-5', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-40-generic', java.version: '1.7.0_76'
Driver info: driver.version: unknown
	at <anonymous class>.fxdriver.preconditions.visible(file:///tmp/anonymous7743485075556311307webdriver-profile/extensions/fxdriver@googlecode.com/components/command-processor.js:9981)
	at <anonymous class>.DelayedCommand.prototype.checkPreconditions_(file:///tmp/anonymous7743485075556311307webdriver-profile/extensions/fxdriver@googlecode.com/components/command-processor.js:12517)
	at <anonymous class>.DelayedCommand.prototype.executeInternal_/h(file:///tmp/anonymous7743485075556311307webdriver-profile/extensions/fxdriver@googlecode.com/components/command-processor.js:12534)
	at <anonymous class>.DelayedCommand.prototype.executeInternal_(file:///tmp/anonymous7743485075556311307webdriver-profile/extensions/fxdriver@googlecode.com/components/command-processor.js:12539)
	at <anonymous class>.DelayedCommand.prototype.execute/<(file:///tmp/anonymous7743485075556311307webdriver-profile/extensions/fxdriver@googlecode.com/components/command-processor.js:12481)
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 151.157 sec <<< FAILURE! - in org.apache.zeppelin.ZeppelinIT

Thanks @felixcheung for pointing it out.  I was neither able to reproduce it on my local, nor was able to do it on local travis. Hence, raise this PR https://github.com/apache/incubator-zeppelin/pull/719. I'll try and fix it there.

another download error https://s3.amazonaws.com/archive.travis-ci.org/jobs/109754384/log.txt:
SPARK_HOME is /home/travis/build/apache/incubator-zeppelin/spark-1.3.1-bin-hadoop2.3
tar (child): spark-1.3.1-bin-hadoop2.3.tgz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Let's merge this if no more discussion. At least we could isolate download or spark start failure with this change.
As for container base I could investigate that further. There is one pending PR that is including sudo commands though that would not work with it.


[BUG] Wasserstein Distance is ill-defined for categorical data and should be excluded

SynapseML version
0.11.1
System information

Language version: Irrelevant
Spark Version: Irrelevant
Spark Platform: Irrelevant

Describe the problem
The Wasserstein distance, sometimes referred to as earth mover's distance (under some conditions), is perhaps most easily understood by imagining distributions as different ways of piling up earth. The earth mover's distance is the minimum cost of transforming one distribution to the other by shuffling earth around, where cost is defined as the amount of dirt moved multiplied by the horizontal distance over which it is moved. In this analogy the effects of gravity are ignored (in reality it takes work to lift dirt up).
Lets take two simple examples. Our goal is to flatten the initial distribution defined over metric space spanning from 1 to 3 into a uniform distribution and measure the cost it took.
Example 1:
Here we have an excess of dirt at 2 which we can move into 3. The cost of flattening the distribution is a single unit.

  o
o o        o o o
o o o      o o o
-----  ->  -----
1 2 3      1 2 3

Example 2:
Here the excess of dirt is initially at 1, which needs to needs to be first moved to 2 and then finally to 3. The cost of flattening the distribution is 2 units.

o             o
o o         o o        o o o
o o o       o o o      o o o
-----  ->   -----  ->  -----
1 2 3       1 2 3      1 2 3

We can confirm this result with the scipy.stats.wasserstein_distance function:
# Example 1
wasserstein_distance([1,1,2,2,2,3],[1,1,2,2,3,3])
0.166666

# Example 2
wasserstein_distance([1,1,2,2,2,3],[1,1,2,2,3,3])
0.333333

What is important to note is that the Wasserstein distance of Example 2 is twice that of Example 1, as it was in our toy example.
Now we can understand why the Wasserstein distance can't really be defined for categorical variables. Imagine that instead of the range between 1 and 3 the distribution is over three colors: red, green, and blue. What is the cost moving earth from green to blue? Is it the same as moving it from red to blue? Will everyone agree on a the same answer? It is no longer clear whether Example 2 should cost twice as Example 1. We confirm this issue by computing the Wasserstein distance for Example 1 and Example 2 as for categorical columns as defined by the DistributionBalanceMeasures functions of SynapseML:
# Example 1
Example1 = [1,1,1,2,2,3]
df1 = spark.createDataFrame(Example1, IntegerType()).toDF("earth")

wasserstein_distance_example1 = (
    DistributionBalanceMeasure()
    .setSensitiveCols(['earth'])
    .transform(df1).select("FeatureName","DistributionBalanceMeasure.wasserstein_dist")
)

wasserstein_distance_example1.show(truncate=False)
+-----------+-------------------+
|FeatureName|wasserstein_dist   |
+-----------+-------------------+
|earth      |0.11111111111111112|
+-----------+-------------------+

# Example 2
Example2 = [1,1,2,2,2,3]
df2 = spark.createDataFrame(Example2, IntegerType()).toDF("earth")

wasserstein_distance_example2 = (
    DistributionBalanceMeasure()
    .setSensitiveCols(['earth'])
    .transform(df2).select("FeatureName","DistributionBalanceMeasure.wasserstein_dist")
)

wasserstein_distance_example2.show(truncate=False)
+-----------+-------------------+
|FeatureName|wasserstein_dist   |
+-----------+-------------------+
|earth      |0.11111111111111112|
+-----------+-------------------+

The Wasserstein distance is the same for Example 1 and Example 2, which is not what we expect.
Code to reproduce issue
Here is another way of showcasing the issue, by coding up a function that can compute the Wasserstein Distance for these simple examples:
def wasserstein_dist(x,y):
    sorted_x = sorted(x)
    sorted_y = sorted(y)

    if len(sorted_x) != len(sorted_y):
        raise ValueError("The two arrays must have the same length.")

    sum = 0
    for i in range(len(sorted_x)):
        sum += abs(sorted_x[i] - sorted_y[i])

    return sum / len(sorted_x)

Example1 = [1,1,1,2,2,3]
Example2 = [1,1,2,2,2,3]
y = [1,1,2,2,3,3]

wasserstein_dist(Example1,y)
0.33333333

wasserstein_dist(Example2,y)
0.16666666

In contrast, what are DistributionBalanceMeasures functions of SynapseML are computing is:
from collections import Counter

def categorical_fractions(categorical_data):

    # Count the occurrences of each feature
    feature_counts = Counter(categorical_data)

    # Calculate the total number of features
    total_features = len(categorical_data)

    # Calculate the fraction each feature occurs
    fractions = {feature: count / total_features for feature, count in feature_counts.items()}

    return fractions

def cat_wasserstein_dist(x_probs, y_probs):

    # Combine the keys from both dictionaries
    all_keys = set(x_probs.keys()) | set(y_probs.keys())

    # Calculate the absolute differences for each key
    absolute_differences = [abs(x_probs.get(key, 0) - y_probs.get(key, 0)) for key in all_keys]

    # Calculate the mean of the absolute differences
    mean_difference = sum(absolute_differences) / len(absolute_differences)

    return mean_difference

Example1 = [1,1,1,2,2,3]
Example2 = [1,1,2,2,2,3]
y = [1,1,2,2,3,3]

Example1_probs = categorical_fractions(Example1)
Example2_probs = categorical_fractions(Example2)
y_probs = categorical_fractions(y)

cat_wasserstein_dist(Example1_probs, y_probs)
0.11111111

cat_wasserstein_dist(Example2_probs, y_probs)
0.11111111

Note that both definitions contain similar mathematical expressions, especially the component computing mean(abs(X - Y)) (present in the source code here), but they are fundamentally computing different things. In the Wasserstein distance, the difference between X and Y is being computed over the metric space (the horizontal direction in the earth pile distribution), while for the definition used in DistributionBalanceMeasures it is within a category. This would correspond to the vertical direction in the earth pile distribution. As currently defined, the Wasserstein Distance computed by DistribuionBalanceMeasures is more akin to what the Jensen-Shannon Distance.
Recommendation
My recommendation is to completely avoid trying to define the Wasserstein distance for categorical features. To properly define it, we require the probability distribution to be defined over a metric space, which categorical variables do not create. In principle, we could define distances between categories and thus define a metric analogous to the Wasserstein distance, but this is really more trouble than it is worth and would be a source of endless confusion for users. We already have measures that are properly defined for categorical variables, like the Jensen-Shannon Distance, so we should use those.
Other info / logs
No response
What component(s) does this bug affect?

[ ] area/cognitive: Cognitive project
[X] area/core: Core project
[ ] area/deep-learning: DeepLearning project
[ ] area/lightgbm: Lightgbm project
[ ] area/opencv: Opencv project
[ ] area/vw: VW project
[ ] area/website: Website
[ ] area/build: Project build system
[ ] area/notebooks: Samples under notebooks folder
[ ] area/docker: Docker usage
[ ] area/models: models related issue

What language(s) does this bug affect?

[ ] language/scala: Scala source code
[ ] language/python: Pyspark APIs
[ ] language/r: R APIs
[ ] language/csharp: .NET APIs
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[X] integrations/synapse: Azure Synapse integrations
[X] integrations/azureml: Azure ML integrations
[X] integrations/databricks: Databricks integrations

@perezbecker Thanks for filing boss, tagging in creator @ms-kashyap


[bug] running tests fails with a linker error

Upd: it seems to be there for a while https://users.rust-lang.org/t/linker-dies-when-using-knuffel/101753/3
Upd2: in my project i was able to WORK AROUND the issue by using lld linker like so
# .cargo/config.toml
[target.aarch64-apple-darwin]
linker = "rust-lld"

Im on M2 Apple chip, latest xcode command line tools etc, rust version 1.81.0
repro:
# in the knus repo root
cargo test

it seems that the actual test that fails is coming from the readme doc.
#[derive(knus::Decode)]
enum TopLevelNode {
    Route(Route),
    Plugin(Plugin),
}

#[derive(knus::Decode)]
struct Route {
    #[knus(argument)]
    path: String,
    #[knus(children(name="route"))]
    subroutes: Vec<Route>,
}

#[derive(knus::Decode)]
struct Plugin {
    #[knus(argument)]
    name: String,
    #[knus(property)]
    url: String,
}

fn main() -> miette::Result<()> {
let config = knus::parse::<Vec<TopLevelNode>>("example.kdl", r#"
    route "/api" {
        route "/api/v1"
    }
    plugin "http" url="https://example.org/http"
"#)?;
 Ok(())
 }

here is how:
error: linking with `cc` failed: exit status: 1

--- a bunch of files to link ---

0  0x104d22074  __assert_rtn + 72
          1  0x104c8f994  ld::DynamicAtomFile::makeNamedAtom(std::__1::basic_string_view<char, std::__1::char_traits<char>>, ld::file_format::Scope, bool) + 488
          2  0x104c5f758  ___ZNK2ld10InputFiles11SliceParser15parseObjectFileEPKN6mach_o6HeaderE_block_invoke.48 + 228
          3  0x104c462e4  ld::KindToRelocs::addFixupFromSymbolNum(ld::AtomPlacement const&, ld::DynamicAtom*, unsigned int, unsigned int, long long) const + 116
          4  0x104c5b7f0  ld::InputFiles::SliceParser::parseObjectFile(mach_o::Header const*) const + 8944
          5  0x104c6b830  ld::InputFiles::parseAllFiles(void (ld::AtomFile const*) block_pointer)::$_8::operator()(unsigned long, ld::FileInfo const&) const + 440
          6  0x197000698  _dispatch_client_callout2 + 20
          7  0x197014b9c  _dispatch_apply_invoke3 + 336
          8  0x197000658  _dispatch_client_callout + 20
          9  0x197001ea0  _dispatch_once_callout + 32
          10  0x197013bc8  _dispatch_apply_invoke + 252
          11  0x197000658  _dispatch_client_callout + 20
          12  0x1970124e8  _dispatch_root_queue_drain + 860
          13  0x197012b20  _dispatch_worker_thread2 + 156
          14  0x1971af39c  _pthread_wqthread + 228
          ld: Assertion failed: (name.size() <= maxLength), function makeSymbolStringInPlace, file SymbolString.cpp, line 74.
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

my quick search gave two seemingly relevant links:

https://github.com/leptos-rs/leptos/issues/3148
https://users.rust-lang.org/t/intel-macos-linking-error/102342

the latter might give a hint why: too much codegen via inlining vs using Box<dyn Trait>.
I tried to wrangle the error, but it seems that it is beyond my immediate humble abilities, still wanted to document that issue

Hi @twop ! This is indeed something I've been battling to get working in the CI (it's the reason macos CI is currently disabled)!
It was also an issue with the original knuffel repo, and I used the same work-around as you listed above, unfortunately that wasn't seeming to work when I was testing macos builds in CI?
That file is also theoretically already part of this repo: https://github.com/TheLostLambda/knus/blob/main/.cargo/config.toml
In your Upd2, were you able to fix the issue on this repo, or was that just you fixing it for another repository that depends on this one?

that was for the usage in my separate repro project. im wondering though if that setting is honored for doc tests.

Closed by #14


Explore/document CSP requirements for loading compression webworker

We currently require a Content Security Policy of worker-src blob: in order to load our Compression webworker. If it fails to load, the SDK will fallback and send uncompressed payloads.
This rule is a bit too relaxed and we may need to load the worker script from the Sentry CDN. We need to be make this doesn't negatively impact the dev experience too much (we probably need to keep loading via blob for development).

The worker generally is a topic to touch on again. IMHO it may make sense to extract this out of the SDK and host it somewhere else, would simplify a lot of stuff.


Error with yarn build with Next 14

Describe the bug
I have the following error when I run yarn build, I ma using NextJs 14.1.0:
❯ yarn build
yarn run v1.22.19
$ node --require ./swagger-build.js ./node_modules/.bin/next build
node:internal/modules/cjs/loader:1080
  throw err;
  ^

Error: Cannot find module 'next/dist/compiled/undici'
Require stack:
- /Users/gabriele.manna/Documents/Projects/convexin/next-swagger-doc/examples/next14-app/swagger-build.js
- internal/preload
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:119:18)
    at Object.<anonymous> (/Users/gabriele.manna/Documents/Projects/convexin/next-swagger-doc/examples/next14-app/swagger-build.js:1:18)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at internalRequire (node:internal/modules/cjs/loader:174:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/gabriele.manna/Documents/Projects/convexin/next-swagger-doc/examples/next14-app/swagger-build.js',
    'internal/preload'
  ]
}

Node.js v18.18.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Reproduction
cd examples/next14-app && yarn build
System Info
System:
    OS: macOS 14.3
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 16.47 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.18.0 - ~/.n/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.8.1 - ~/.n/bin/npm
  Browsers:
    Chrome: 121.0.6167.184
    Safari: 17.3

Used Package Manager
yarn
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

Thanks @geberele for the report. I will look this issue sometime this week.

Hi @geberele Could you take a try with latest commit? It should work now.


Provide a mechanism to record a 'snapshot' of the DAG

GF has built their own version of this.
Basically for a given run of raw inputs, they record the repo/commitID for all of the pipeline output repos as well. This represents the 'state' of a set of inputs ... both the raw inputs and all the outputs (intermediate, final).
You can't just look at the latest commits since there could be multiple batches working through the DAG at any given time.
It seems like we could provide a way to 'tag' the input/output commits to expose this information very easily as part of FlushCommit().
The one component missing from this is the name/hash of the images used in the transforms. To truly be able to reproduce a run, we'll need that as well. I can't think of a simple way to encode that information alongside the commits, so this is starting to feel like a new Snapshot object in the db.

Good discussion w Matt + Joey
Inspecting a commit shows you the provenance. There may be a way to include the image name/hash per commit. Not necessarily the image is reported for each of the commtis included under 'provenance' but at least for the commit that is being inspected.
If we showed the image name + hash for inspect commit, a user could construct a complete snapshot as needed.

Ok, so we can provide basically all of this functionality today, just not in a single command. But that's ok. As long as you can re-create the state (and all of the processing), the rest is just better tooling.

Use Cases
As an example, I've run the fruit stand demo to show 2 use cases:

Given a complete set of input commits, return all of the output commits + jobs

# First get all of the output commits of the input commit(s)
$pachctl flush-commit data/master/0
BRANCH                             REPO/ID                                     PARENT              STARTED             FINISHED            SIZE
master                             data/master/0                               <none>              About an hour ago   About an hour ago   874 B
4fcdd66a9cb240d49580e65e9ae71f3a   sum/4fcdd66a9cb240d49580e65e9ae71f3a/0      <none>              About an hour ago   About an hour ago   12 B
5581dee7f0d94c44a1d82e75270ba8bc   filter/5581dee7f0d94c44a1d82e75270ba8bc/0   <none>              About an hour ago   About an hour ago   200 B
# Then for each of the output commits, look for the job ID that created the commit:
$pachctl list-job
ID                                 OUTPUT                                      STARTED             DURATION            STATE
adb86255ce181d7e7719efda3f203951   sum/4fcdd66a9cb240d49580e65e9ae71f3a/1      About an hour ago   3 seconds           success
609440a680caf705c3f3b451d70b732c   sum/4fcdd66a9cb240d49580e65e9ae71f3a/0      About an hour ago   3 seconds           success
4ab2d81bf69bd012096aa7d95cc57467   filter/5581dee7f0d94c44a1d82e75270ba8bc/1   About an hour ago   6 seconds           success
67c30d70ba9d2179aa133255f5dc81db   filter/5581dee7f0d94c44a1d82e75270ba8bc/0   About an hour ago   7 seconds           success
# And using that job ID inspect the job to get the all of the transform info for the job
$pachctl inspect-job 609440a680caf705c3f3b451d70b732c
ID: 609440a680caf705c3f3b451d70b732c
Started: About an hour ago
Duration: 3 seconds
State: success
ParallelismSpec: constant:1
Inputs:
NAME                COMMIT                               PARTITION           INCREMENTAL
filter              5581dee7f0d94c44a1d82e75270ba8bc/0   FILE                DIFF
Transform:
{
  "cmd": [
    "sh"
  ],
  "stdin": [
    "for fruit in apple orange banana; do",
    "   { cat /pfs/prev/$fruit || echo 0; cat /pfs/filter/$fruit; } | awk '{s+=$1} END {print s}' > /pfs/out/$fruit",
    "done"
  ]
}

This is manual, but allows you to completely reconstruct all input/output commits, and the image is referenced in the output of the inspect-job call. However, we don't currently report the hash of the image, and that is something we probably do want to support. (If you've only specified your image w names or name:tag, we wont necessarily be able to tell exactly which version of the image you're using. On the flip side, if you do need that level of granularity you can refer to an image by its hash and this will work. But we want to make this better. This is listed in the todo section below).

Given a Pipeline's output commit somewhere on the DAG, provide a list of all the commits + jobs upstream of this commit

# First find the commit you care about. In this case we'll choose the latest on the sum repo
$pachctl list-commit sum
BRANCH                             REPO/ID                                  PARENT                               STARTED             FINISHED            SIZE
4fcdd66a9cb240d49580e65e9ae71f3a   sum/4fcdd66a9cb240d49580e65e9ae71f3a/0   <none>                               About an hour ago   About an hour ago   12 B
4fcdd66a9cb240d49580e65e9ae71f3a   sum/4fcdd66a9cb240d49580e65e9ae71f3a/1   4fcdd66a9cb240d49580e65e9ae71f3a/0   About an hour ago   About an hour ago   12 B
# Inspect that commit to get all commits in its provenance
$pachctl inspect-commit sum 4fcdd66a9cb240d49580e65e9ae71f3a/1
Commit: sum/4fcdd66a9cb240d49580e65e9ae71f3a/1
Parent: 4fcdd66a9cb240d49580e65e9ae71f3a/0
Branch: 4fcdd66a9cb240d49580e65e9ae71f3a
Started: About an hour ago
Finished: About an hour ago
Size: 12 B
Provenance:  filter/5581dee7f0d94c44a1d82e75270ba8bc/1  data/master/1
# For each of the commits listed that are the output of a pipeline, find the corresponding job
$pachctl list-job | grep filter/5581dee7f0d94c44a1d82e75270ba8bc/1
4ab2d81bf69bd012096aa7d95cc57467   filter/5581dee7f0d94c44a1d82e75270ba8bc/1   About an hour ago   6 seconds           success
# Inspect the job to get the job/image info
$pachctl inspect-job 4ab2d81bf69bd012096aa7d95cc57467
ID: 4ab2d81bf69bd012096aa7d95cc57467
Parent: 67c30d70ba9d2179aa133255f5dc81db
Started: About an hour ago
Duration: 6 seconds
State: success
ParallelismSpec: constant:1
Inputs:
NAME                COMMIT              PARTITION           INCREMENTAL
data                master/1            BLOCK               DIFF
Transform:
{
  "cmd": [
    "sh"
  ],
  "stdin": [
    "for fruit in apple orange banana; do",
    "   grep $fruit /pfs/data/sales | awk '{print $2}' >>/pfs/out/$fruit",
    "done"
  ]
}

Todo

Update the docs to provide these common debugging workflows as examples

Update JobInfo to store the hash of the image (we'll need to see if we can get the hash by querying the docker registry somehow ... seems like we should be able to get it)

In the future, I think this will make a great tool for our web UI. There is also a case to be made that you might want a command to take a 'snapshot' as input and re-run something. But that is loosely defined right now, so I'll defer that decision until that use case becomes clearer.

We discussed this again last week. That discussion centered around the hash of a docker image and whether or not we could easily expose this information.
The goal here is for a user to be able to reproduce all of the data and the exact version of the code that was running to create the output data.
@jdoliner keep me honest - but from what I remember I think our conclusion was:

Support exposing the image hash per job

we think we can get the image hash from the k8s job API
if this is easy, we could expose this information in pachctl inspect-job

Recommend conventions around versioning images and development

In development, use pachctl update-pipeline ... --push-images which will automatically create a unique tag for your image each time you do an update
In production, you'll care a lot more about which version of the code was deployed. In this case, use semantic versions + docker tags

Both of these cases should be documented

The new migration feature marshalls a whole cluster into a file, thereby snapshotting it.


Fix equal_approx and zero_approx for doubles

When Godot is compiled with 64-bit floats, is_equal_approx and is_zero_approx still use CMP_EPSILON, which is "only" 0.00001. Doubles have way more decimal places of precision, so a lower threshold should be used by default.
One example of problem this solves is in the editor:
Place a Node3D at (100,000; 100,000; 100,000), and center the camera on it. You can rotate around, but panning is completely unresponsive. The camera is not moving, unless you rotate it. This is not a rendering issue because the camera doesn't update either when the screen redraws.
In node_3d_editor_plugin.cpp, this comparison is done to check if the camera should be updated:
https://github.com/godotengine/godot/blob/73ece5ba1962eb3c19ac529b8cb4ac18aefd5cac/editor/plugins/node_3d_editor_plugin.cpp#L330
Which leads to here:

This is not a small panning: the cursor is moving by 0.1 units, while looking at a node a few units away. Yet because CMP_EPSILON turns tolerance into such a high value, the numbers are considered equal, when they really aren't, even if there are plenty of decimal places.
There are likely other similar issues cropping up when large numbers show up, so I thought of fixing the double and float versions of is_equal_approx and is_zero_approx to use specific epsilons by default.
I don't know what to do about CMP_EPSILON and the others though. Seems like it sometimes matters, sometimes it doesn't. So I didn't touch them for now.

Looks like every test comparing floats are failing now, because the expected results are not written with enough decimals 👀

I haven't reviewed the discussion in-depth yet, but please note that CMP_EPSILON is not meant to be the ~minimal difference between two valid representations of a floats/doubles.
It's meant to be a high enough maximal permission error to smooth over precision issues that result from multiple operations (especially complex math operations on transforms). As such it's meant to be at least one, if not two, orders of magnitude bigger than the "precision" of the representation of a given float/double.
If it's set too low, everything will break :)

Agree with both @aaronfranke and @akien-mga 's points...
Especially that the epsilons are not designed to be the minimum recognisable difference between values in a data type, and the fact the resolution is higher does not necessarily imply that the epsilon should be lower.
My gut feeling is that at least during introduction, taking baby steps might be a safer course of action than taking a giant leap, e.g. maybe reusing existing functions and their 32 bit epsilons if we can (as much as possible) to start with might be a good way of limiting the permutations which can give rise to bugs.
So if we could, for example, keep the same default functions and epsilons as 32 bit (which we know should work), then gradually introduce some new explicit precise functions to a few specific areas that could obviously benefit from smaller epsilons (e.g. world coords, basis). We could then test these out, and monitor for regressions, then only once these were all confirmed working, move onto the next area etc.
E.g.:
is_equal_approx - use 32 version
is_equal_precise - use 32 or 64 bit depending on compile option

These could of course be renamed later, but it might be a crucial stage introducing such features, and the process of identifying and fixing regressions (easier to pin down when we have changed 1 area of the code, rather than simultaneously changed 200 maybe :grinning: ).
There are probably things in the unit tests that have not been made correctly for 64 bit real builds, and imo we should err on the side of fixing the tests (perhaps with 32 and 64 bit test versions), rather than have the engine changed to bodge around dodgy unit tests.
There's also the problem of how these epsilons are handled in user code (e.g. gdscript) but I guess we can be in a better place to answer that once it is working satisfactorily in the core engine.

I'm aware that CMP_EPSILON is not the true epsilon, for that reason I didn't really look for an even lower value for CMP_EPSILON_64. I also believed that the choice to use CMP_EPSILON rather than a lower one when compiling with float=64 would be the exception, rather than the default.
So you think I should leave CMP_EPSILON and instead go the other way around, fixing cases that need lower epsilon as we find them? I had no problem so far with the current PR, apart from the tests.
If that's the choice we make, the editor camera problem needs either to be able to pass a custom epsilon to Vector2::is_equal_approx IF building with 64-bit floats (making it annoying to use), or we have to create these new methods, even tho new methods would still not use the true epsilon. What do we prefer?
Either way, bugs involving comparisons are going to show up, so I need to know which pattern to follow to fix them. I started using float=64 from now on, to find as much as I can (and also because I'd like this to be well supported for the kind of game I want to make). Here is a video of the latest try I had.
If not using lower epsilon in is_equal_approx, tests will "pass" again. But is that also wanted? We may have to fix them later.

BTW don't take my word for it how we should handle this, I know absolutely no more about it than you do! :grinning:
I've never used doubles for large worlds (only moving origin or fixed point). I think we just have to just discuss it between us over a few days, get reduz thoughts on this as well, and come up with an agreement on the best compromise. (Maybe it will come up in PR meeting?)

I kinda have the feeling that this is unnecessary. I mean CMP_EPSILON does not really have a number that should be tiny enough that makes a difference between float and double. Both can represent this. Its generally more of a number for  practical purposes you can use to consider equality.


[Teams-AI-Bot] Upgrade the referenced TeamsAI package to GA version

Requirement
Upgrade the version of TeamsAI package to the latest GA version 1.3.0.
Context
Currently, the bot still uses the alpha version of the TeamsAI package since its' initial release. Now, the package has been GAed.
The GA version of TeamsAI package introduces a bunch of breaking changes, which will require significant effort to accommodate in this version upgrade.
Additionally, upgrading to this version will allow us to also upgrade the bot builder package to its latest version, enabling support for SNI certificate authentication for app registration.

This has been postponed, and we will revisit this task later.


there be a preflight check for ufw, similar to firewalld

There should be checkUfw, similar to https://github.com/replicatedhq/kURL/blob/123b11e4d39d5cafd84058e5cf43d69b541eb4e3/scripts/common/preflights.sh#L7

Resolved by https://github.com/replicatedhq/kURL/pull/1453. Closing