[FEAT] Support for embedding-based similarity functions

Is your proposal related to a problem?
I would like to be able to use Splink with embedding-based similarity functions, specifically with duckdb and Athena backends.
For example, to evaluate the similarity between two product description titles, I can use a deep embedding model (e.g., sent2vec) to embed the titles as numeric arrays. Then, I can compute similarity between the titles as the dot product normalized dot product (cosine similarity) between the two.
The embedding does not have to be managed by Splink. I can compute the embeddings and append them as an array feature column to my data.
Alternatives considered
I can use custom SQL with Splink when defining my comparisons, but it would be more convenient and user-friendly to have this in a comparison library.
Describe the solution you'd like
I'd like the Athena and duckdb comparison libraries to contain a dot product or cosine similarity function that can be applied to array or list columns.
Additional context
It seems very simple to implement this functionality in SQL. However, Splink's architecture is a bit obscure and I'm not sure how I would be able to incorporate this to comparison libraries myself. I would be able to do it with some guidance.

This seems like a useful idea. Just FYI I am in the process of writing some documentation that serves as a dev guide for adding new comparisons and comparison-levels to hopefully make this process a little bit more accessible - I'll try to finish that off sooner rather than later.

This is a really interesting idea and I'd love to see how it pans out.
So it seems like the fundamental challenge is that database needs to support a similarity function that takes two vectors of embeddings as an input, and outputs a similarity score.
There's a rough and ready discussion of how you could use these functions here.
In particular, the kind of syntax you'd be looking for would be like this   (but note the cosine_similarity function DOES NOT EXIST!)  - not in particular sql_condition part:
comparison_product_name = {
    "output_column_name": "product_name",
    "comparison_description": "product name comparison",
    "comparison_levels": [
        {
            "sql_condition": "product_name_l IS NULL OR product_name_r IS NULL",
            "label_for_charts": "Null",
            "is_null_level": True,
        },
        {
            "sql_condition": "product_name_l = product_name_r",
            "label_for_charts": "Exact match",
            "tf_adjustment_column": "product_name",

        },
        {
            "sql_condition": "cosine_similarity(product_name_embeddings_array_l = movie_name_embeddings_array_r) > 0.9",
            "label_for_charts": "cosine_similarity > 0.9",

        },

        {"sql_condition": "ELSE", "label_for_charts": "All other comparisons"},
    ],

}

This approach only works if you have additional comparison columns which are 'independent' of product name, e.g price, sku or something.  Because i general a splink model with a single comparison doesn't work.
If you only have the product name, i suppose you could consider trying to get the embeddings down to a small array and splitting out to columns (embedding_1, embedding_2 etc.) and comparing them separately numerically.  (Though I must admit, I don't understand embeddings well, so don't know if this is a viable strategy)
In terms of implementing a similarity functino:
The functions that work on nested data are listed here.  Sadly, I don't immediately see one that looks useful for your purpose, but there may be a creative use that I haven't thought of.
It looks like there are some planned functions that may be of help here - in particular arrayZip when combined with arrayMap feels like it could be useful.  You could perhaps comment on that discussion to see if there's been any progress.
I also found this:
https://blog.eto.ai/vector-similarity-search-with-duckdb-44dec043532a

Thanks for the links @RobinL ! I missed the part of the documentation with the custom comparison as SQL.
With AWS Athena (Trino SQL), I think zip_with and reduce would work well for implementing a dot product.

Nice - yeah, that def looks possible.
This is pretty nasty, but it occurred to me that in dialects that don't support zip_with you could probably get to the same functionality with an array_map (list transform) function, so long as you stored your embeddings as arrays of tuples in the form [(embedding_value, counter), ...]
array_l = [(embedding_0, 0), (embedding_1, 1), ... ]
Then something like this would iterate across the elements
list_transform(array_l, x -> array_r[x[1]])

@RobinL That is good to know. I'm sure there in interest to provide more functionality for machine learning data preprocessing and NLP as well. Trino has some good stuff for ngram tokenization and min-hashing: https://trino.io/docs/current/functions/setdigest.html

This is also worth keeping an eye on: https://twitter.com/AlexMonahan/status/1621141511000961026
Might make it easier to write a udf (e.g. for cosine distance)

If you only have the product name, i suppose you could consider trying to get the embeddings down to a small array and splitting out to columns (embedding_1, embedding_2 etc.) and comparing them separately numerically. (Though I must admit, I don't understand embeddings well, so don't know if this is a viable strategy)

I think you are correct, the individual components of the embedding are going to be correlated, so I think you need to treat all the components together as one Comparison.

@mamonu has now created 'first try' at a cosine similarity function for the Spark backend which can be found here:
https://github.com/moj-analytical-services/splink_scalaudfs/blob/embeddings/jars/scala-udf-similarity-0.1.1-EMBEDDINGSDEV.jar
Here's a working Splink model using the above jar:
splink model
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import DoubleType
import splink.spark.spark_comparison_library as cl
from splink.spark.spark_linker import SparkLinker

conf = SparkConf()
# This parallelism setting is only suitable for a small toy example
conf.set("spark.driver.memory", "12g")
conf.set("spark.default.parallelism", "16")
conf.set("spark.jars", "jars/scala-udf-similarity-0.1.1-EMBEDDINGSDEV.jar")

sc = SparkContext.getOrCreate(conf=conf)

spark = SparkSession(sc)
spark.sparkContext.setCheckpointDir("./tmp_checkpoints")

spark.udf.registerJavaFunction(
    "vector_cosine_sim",
    "uk.gov.moj.dash.linkage.VectorCosineSimilarity",
    DoubleType()
)

spark.udf.registerJavaFunction(
    "jaro_winkler",
    "uk.gov.moj.dash.linkage.JaroWinklerSimilarity",
    DoubleType()
)

df = spark.read.parquet("fake_1000_with_embeddings.parquet")

df.createOrReplaceTempView("df")
sql = """
select *,
cast(openai_embeddings as array<double>) as embeddings
from df
"""
df = spark.sql(sql)

df.createOrReplaceTempView("df")

settings = {
    "link_type": "dedupe_only",
    "comparisons": [
        cl.jaro_winkler_at_thresholds("first_name", 0.8),
        cl.jaro_winkler_at_thresholds("surname", 0.8),
        cl.levenshtein_at_thresholds("dob"),
        cl.exact_match("city", term_frequency_adjustments=True),
        cl.levenshtein_at_thresholds("email"),
        cl.distance_function_at_thresholds("embeddings","vector_cosine_sim", [1.0, 0.99, 0.98, 0.95, 0.9, 0.85, 0.8, 0.7], include_exact_match_level=False)

    ],
    "blocking_rules_to_generate_predictions": [
        "l.first_name = r.first_name",
        "l.surname = r.surname",
    ],
    "retain_matching_columns": True,
    "retain_intermediate_calculation_columns": True,
    "em_convergence": 0.01
}

linker = SparkLinker(df, settings)
deterministic_rules = [
    "l.first_name = r.first_name and levenshtein(r.dob, l.dob) <= 1",
    "l.surname = r.surname and levenshtein(r.dob, l.dob) <= 1",
    "l.first_name = r.first_name and levenshtein(r.surname, l.surname) <= 2",
    "l.email = r.email"
]

linker.estimate_probability_two_random_records_match(deterministic_rules, recall=0.6)

linker.estimate_u_using_random_sampling(max_pairs=1e6)

linker.match_weights_chart()

@OlivierBinette  you may be interested ☝

What is preventing us from implementing the cosine similarity in SQL, so it works on all backends? Unless I'm missing something it wouldn't be that hard to generate some SQL based on column names that the user provides. Are we just worried about performance?
import duckdb

def _dot(lefts, rights):
    def _clause(left, right):
        # for testing do it literally
        return f"{left} * {right}"
        # would use this for real
        # return f"l_{left} * r_{right}"

    clauses = (_clause(*pair) for pair in zip(lefts, rights, strict=True))
    return "(" + " + ".join(clauses) + ")"

def _norm(cols):
    return f"SQRT({_dot(cols, cols)})"

def make_cosine_similarity_sql(
    column_names: list[str], right_column_names: list[str] | None = None
) -> str:
    if right_column_names is None:
        lefts = column_names
        rights = column_names
    else:
        lefts = column_names
        rights = right_column_names

    return f"{_dot(lefts, rights)} / ( {_norm(lefts)} * {_norm(rights)})"

conn = duckdb.connect()
conn.execute(
    "CREATE TABLE 'data' AS SELECT * FROM READ_CSV_AUTO('https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv')"
)
left_cols = ["petal_length", "petal_width"]
right_cols = ["sepal_length", "sepal_width"]
cs = make_cosine_similarity_sql(left_cols, right_cols)
print(cs)

conn.sql(f"SELECT *, {cs} AS similarity from data")

results in
(petal_length * sepal_length + petal_width * sepal_width) / ( SQRT((petal_length * petal_length + petal_width * petal_width)) * SQRT((sepal_length * sepal_length + sepal_width * sepal_width)))
┌──────────────┬─────────────┬──────────────┬─────────────┬───────────┬────────────────────┐
│ sepal_length │ sepal_width │ petal_length │ petal_width │  species  │     similarity     │
│    double    │   double    │    double    │   double    │  varchar  │       double       │
├──────────────┼─────────────┼──────────────┼─────────────┼───────────┼────────────────────┤
│          5.1 │         3.5 │          1.4 │         0.2 │ setosa    │  0.896248789296663 │
│          4.9 │         3.0 │          1.4 │         0.2 │ setosa    │ 0.9181234236331912 │
│          4.7 │         3.2 │          1.3 │         0.2 │ setosa    │ 0.9025637129150959 │
│          4.6 │         3.1 │          1.5 │         0.2 │ setosa    │ 0.8958521726609922 │
│          5.0 │         3.6 │          1.4 │         0.2 │ setosa    │ 0.8860111771934351 │
│          5.4 │         3.9 │          1.7 │         0.4 │ setosa    │  0.923229445290974 │
│          4.6 │         3.4 │          1.4 │         0.3 │ setosa    │ 0.9108675898478961 │
│          5.0 │         3.4 │          1.5 │         0.2 │ setosa    │ 0.8939896049388586 │
│          4.4 │         2.9 │          1.4 │         0.2 │ setosa    │  0.904392513917151 │
│          4.9 │         3.1 │          1.5 │         0.1 │ setosa    │ 0.8787709575388238 │
│           ·  │          ·  │           ·  │          ·  │   ·       │          ·         │
│           ·  │          ·  │           ·  │          ·  │   ·       │          ·         │
│           ·  │          ·  │           ·  │          ·  │   ·       │          ·         │
│          6.7 │         3.1 │          5.6 │         2.4 │ virginica │ 0.9995949828334421 │
│          6.9 │         3.1 │          5.1 │         2.3 │ virginica │ 0.9999989949665131 │
│          5.8 │         2.7 │          5.1 │         1.9 │ virginica │ 0.9968761695450432 │
│          6.8 │         3.2 │          5.9 │         2.3 │ virginica │ 0.9976798018232167 │
│          6.7 │         3.3 │          5.7 │         2.5 │ virginica │ 0.9990176195924825 │
│          6.7 │         3.0 │          5.2 │         2.3 │ virginica │ 0.9999896398613478 │
│          6.3 │         2.5 │          5.0 │         1.9 │ virginica │ 0.9998931348496853 │
│          6.5 │         3.0 │          5.2 │         2.0 │ virginica │ 0.9978730208564014 │
│          6.2 │         3.4 │          5.4 │         2.3 │ virginica │ 0.9951085010602806 │
│          5.9 │         3.0 │          5.1 │         1.8 │ virginica │ 0.9914171734791748 │
├──────────────┴─────────────┴──────────────┴─────────────┴───────────┴────────────────────┤
│ 150 rows (20 shown)                                                            6 columns │
└──────────────────────────────────────────────────────────────────────────────────────────┘

@NickCrews I think that would work well for generic SQL backends. For SQL engines that support array data types (like Trino/AWS Athena), it would be more convenient to have the embedding arrays contained in a single column.

That's true. For that case, each backend might have slightly different syntax, so we should make it possible for each backend to override with their own implementation. But it can still happen in the SQL level, I don't think we need to write custom UDFs for most cases

I think we could keep the same API as the function I wrote above, but if you supply a single string (instead of Iterable[str] like I have there), then it is interpreted as a single column holding array values

Interesting.  It hadn't occurred to me that rather than using an 'array_transform'/'array_reduce' type function, you could spread over multiple columns, and more importantly, I hadn't thought about the possibility that you can treating an array column as if they're just a group of columns (e.g. replace petal_length and sepal_length etc. with arr_col[0] and arr_col[1] etc. in your formulation.
I agree that approach seems like it's more likely to work in a backend-agnostic way.  Note sure what performance would look like - one of the problems with embeddings is they're often long (e.g. openAI's are of length 1536 ).  I guess for relatively small linking problems, performance would probably be acceptable given cosine distance is a relatively simple calculation so it feels like it's of value.
@OlivierBinette  in your work with embeddings, are they long, or are you finding a way of compressing them?  I haven't tried it by I wondered if there was any milage in using dimensionality reduction techniques.  Though probably only relevant for very large input datasets, otherwise you might as well leave them 'raw'.

Native implementations (eg the one in scala above) are going to scale O(length of vector) just like the above SQL. You are always gonna need the same number of basic operations. I think it's more of a question of overhead. We could run that SQL in scala and compare it to the native implementation as an easy first test.

@RobinL The embeddings can be quite long. PatentsView uses the sent2vec package to embed and compare patent titles. These are of length 100 by default if I remember correctly. These embeddings can get larger with bigger models (as with OpenAI's models as you mention), so scalability in the embedding dimension is very important.

Worth noting for people reading this thread that Python udfs coming to duckdb very soon: https://twitter.com/holanda_pe/status/1655602267083489282

@RobinL, @samnlindsay, duckdb has added list_cosine_similarity as a function and Trino has added cosine_similarity.
I'd be interested working with your team to showcase this functionality, using deep embedding models (via openAI's API or HuggingFace) with Splink for complex comparisons. I'm quite busy finishing my PhD but I could find the time if we plan for something useful.

@OlivierBinette  nice.  yes a showcase would be great!
We're actually thinking about refactoring some of the comparisons because it's currently a bit tricky to implement new ones,  although there's always the option of writing SQL directly into the settings dictionary, which will be fine for a showcase.
I'll contact you direct so we can set up a chat

Hi @RobinL,
Apologies for re-opening this conversation and for the long message, but I'm curious if you have any advice regarding the use of the jar file and your code from above.
More specifically, when I use the vector_cosine_sim udf in my comparisons, the Linker doesn't find any instances of the given thresholds in my data, even though I'm quite confident that they exist.
I don't think it's an issue with registration of the jar/udf because I'm able to run the following:
def configure_spark_session(driver_memory="16g", executor_memory="8g") -> SparkSession:
    """Configure a Spark session with specific settings.

    Args:
        driver_memory (str): Memory allocated to driver.
        executor_memory (str): Memory allocated to executor.
        checkpoint_dir_path (str): Path of model chceckpoint data.

    Returns:
        spark (SparkSession): The configured Spark session.
    """
    conf = SparkConf()
    conf.set("spark.driver.memory", driver_memory)
    conf.set("spark.executor.memory", executor_memory)
    conf.set("spark.sql.codegen.wholeStage", "true")
    conf.set("spark.sql.warehouse.dir", tempfile.mkdtemp())
    conf.set("spark.driver.maxResultSize", "2g")
    conf.set("spark.jars", "./scala-udf-similarity-0.1.1-EMBEDDINGSDEV.jar")
    conf.set("spark.driver.extraClassPath", "./scala-udf-similarity-0.1.1-EMBEDDINGSDEV.jar")
    conf.set("spark.default.parallelism", "800")
    sc = SparkContext.getOrCreate(conf=conf)
    sc.setLogLevel("ERROR")
    spark = SparkSession(sc)
    spark.sparkContext.setCheckpointDir(tempfile.mkdtemp())
    return spark

spark = configure_spark_session()

df = spark.read.parquet('./full.parquet').select('unique_id', 'name', 'title_embeddings')

df = df.withColumn('title_embeddings', df['title_embeddings'].cast(T.ArrayType(T.DoubleType())))

df.printSchema()

root
|-- unique_id: string (nullable = true)
|-- name: string (nullable = true)
|-- title_embeddings: array (nullable = true)
 |    |-- element: double (containsNull = true)

from pyspark.sql.types import DoubleType
spark.udf.registerJavaFunction(
    "vector_cosine_sim",
    "uk.gov.moj.dash.linkage.VectorCosineSimilarity",
    DoubleType()
)

excerpt = df.sample(False, 0.01)

excerpt_l = excerpt.select(F.col('title_embeddings').alias('title_embeddings_l'), F.col('unique_id').alias('unique_id_l'))
excerpt_r = excerpt.select(F.col('title_embeddings').alias('title_embeddings_r'), F.col('unique_id').alias('unique_id_r'))

cross_joined = excerpt_l.crossJoin(excerpt_r)

cross_joined.show(5)

+--------------------+--------------------+--------------------+--------------------+
|  title_embeddings_l|         unique_id_l|  title_embeddings_r|         unique_id_r|
+--------------------+--------------------+--------------------+--------------------+
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[-0.1078699529170...|2c7cd150b8ca89dc0...|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.47958111763000...|40fb2575b7f4787f2...|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.19567984342575...|43ba4ec689260af41...|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.59064656496047...|5125345379d0d15e9...|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.23891282081604...|7ca7ebb81a398d15a...|
+--------------------+--------------------+--------------------+--------------------+

df_with_similarity = cross_joined.withColumn(
    "cosine_similarity",
    F.expr("vector_cosine_sim(title_embeddings_l, title_embeddings_r)")
)
df_with_similarity.show(5)

+--------------------+--------------------+--------------------+--------------------+------------------+
|  title_embeddings_l|         unique_id_l|  title_embeddings_r|         unique_id_r| cosine_similarity|
+--------------------+--------------------+--------------------+--------------------+------------------+
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[-0.1078699529170...|2c7cd150b8ca89dc0...|1.0000000000000002|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.47958111763000...|40fb2575b7f4787f2...|0.8944097413898618|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.19567984342575...|43ba4ec689260af41...|0.8861883455703176|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.59064656496047...|5125345379d0d15e9...|0.8801735873211239|
|[-0.1078699529170...|2c7cd150b8ca89dc0...|[0.23891282081604...|7ca7ebb81a398d15a...|0.8433293128231946|
+--------------------+--------------------+--------------------+--------------------+------------------+

As you can see, the udf seems to be registered and is running correclty.

However, when then when I set up my Linker object and run parameter estimation, I get the following results:
EMBEDDING_COMPARISON = cl.distance_function_at_thresholds(
    "title_embeddings",
    "vector_cosine_sim",
    [0.96, 0.92, 0.88, 0.84, 0.80, 0.76, 0.72, 0.68, 0.5, .0001],
    include_exact_match_level=False,
)

settings = {
    "link_type": "dedupe_only",
    "comparisons": [EMBEDDING_COMPARISON],
    "retain_matching_columns": True,
    "retain_intermediate_calculation_columns": True,
    "max_iterations": 100,
    "probability_two_random_records_match": 1e-5
}

linker = SparkLinker(df, settings)

linker.estimate_u_using_random_sampling(1e6)

----- Estimating u probabilities using random sampling -----
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.92 (comparison vector value: 9). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.88 (comparison vector value: 8). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.84 (comparison vector value: 7). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.8 (comparison vector value: 6). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.76 (comparison vector value: 5). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.72 (comparison vector value: 4). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.68 (comparison vector value: 3). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.5 (comparison vector value: 2). This usually means the comparison level was never observed in the training data.
u probability not trained for title_embeddings - Vector_cosine_sim <= 0.0001 (comparison vector value: 1). This usually means the comparison level was never observed in the training data.

By chance is this something you observed in any of your experiments or is there anything glaringly incorrect in what I've shared above?
I'm happy to help troubleshoot this and work on a PR if there's a bug or something missing in the codebase!
Many thanks in advance,
Jackson

Here's what's produced when I run print(EMBEDDING_COMPARISON.human_readable_description)
Comparison 'Title_Embeddings within vector_cosine_sim thresholds 0.96, 0.92, 0.88, 0.84, 0.8, 0.76, 0.72, 0.68, 0.5, 0.0001 vs. anything else' of `title_embeddings`.
Similarity is assessed using the following ComparisonLevels:
    - 'Null' with SQL rule: `title_embeddings_l` IS NULL OR `title_embeddings_r` IS NULL
    - 'Vector_cosine_sim <= 0.96' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.96
    - 'Vector_cosine_sim <= 0.92' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.92
    - 'Vector_cosine_sim <= 0.88' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.88
    - 'Vector_cosine_sim <= 0.84' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.84
    - 'Vector_cosine_sim <= 0.8' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.8
    - 'Vector_cosine_sim <= 0.76' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.76
    - 'Vector_cosine_sim <= 0.72' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.72
    - 'Vector_cosine_sim <= 0.68' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.68
    - 'Vector_cosine_sim <= 0.5' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.5
    - 'Vector_cosine_sim <= 0.0001' with SQL rule: vector_cosine_sim(`title_embeddings_l`, `title_embeddings_r`) <= 0.0001
    - 'All other comparisons' with SQL rule: ELSE

I tested again:
df_with_similarity = cross_joined.withColumn(
    "cosine_similarity",
    F.expr("vector_cosine_sim(title_embeddings_l, title_embeddings_r) <= .96")
)

to see if the Boolean could be output, and it did indeed work:
+--------------------+--------------------+--------------------+--------------------+-----------------+
|  title_embeddings_l|         unique_id_l|  title_embeddings_r|         unique_id_r|cosine_similarity|
+--------------------+--------------------+--------------------+--------------------+-----------------+
|[-0.2301929146051...|08c2cb77c857104eb...|[-0.2301929146051...|08c2cb77c857104eb...|            false|
|[-0.2301929146051...|08c2cb77c857104eb...|[0.42279487848281...|3484bbf51a31aba09...|             true|
|[-0.2301929146051...|08c2cb77c857104eb...|[0.99581563472747...|37c8ee1fdd49c8ceb...|             true|
|[-0.2301929146051...|08c2cb77c857104eb...|[0.91193675994873...|6036e32bb4a9bb492...|             true|
|[-0.2301929146051...|08c2cb77c857104eb...|[0.56825220584869...|61de65185976fa16f...|             true|
+--------------------+--------------------+--------------------+--------------------+-----------------+

Is it possible that the vector_cosine_sim function somehow doesn't get registered with the SQL engine in Splink and just works on the PySpark session (even though I think they're basically the same thing)?
Thanks very much for the help!

I think it's the <= that's the problem, it's the wrong way around, should be >=, if you look at the function signature, there's an argument like higher is better or something.  Maybe there's a bug, but try changing that option

You're absolutely right. I do believe that's the issue.
I'm trying to change that parameter now; however, it doesn't seem to be flipping the operator as would be expected. I'll keep investigating and will get back to you as to whether it's a user error or a small bug.
Thanks again!

PR above should address the issue I was mentioning! All working on my side now. Thanks a lot for the help!


Scene; calling empty() causes erroneous log output

When calling scene.empty() I get the following trace:
Entity.hx:517:    i / entity / entity has no parent or scene, currently no core events will reach it.

The reason, as far as I can gather, is as follows:
In Scene::empty(), this happens:
remove( entity );
entity.destroy();

Where remove() sets the scene of entity to null.
Then, entity.destroy() triggers the following function calls on Entity:

destroy()

onremoved()

_unlisten()

_find_emit_source() which has the following code:

        if(scene != null) {

            source = scene;

        } else if(parent != null) {

          // ...

        } else {
            log('entity has no parent or scene, currently no core events will reach it.');
        }

Because the entity's scene variable was set to null, the log-statement will be outputted.

ah, yep that's not right, will fix that thanks!


How to recover children items after recoverd parent item

I have groups model and it has parent/children associations. When I destroy parent item, all children items destroy . But, I recover parent item, children items not recover. I tried options of recover in document.

What happens if you do this:
<EMAIL_ADDRESS>:recursive => true)

This was happening on a project to me as well.
Parent.rstore does not exist (or so it seems) in actas_as_paranoid, Parent.recover does, but does not do the trick.
I've decided to migrate to the paranoia gem, which is almost seamless (couple of method renaming).
Best movement though would be moving to the discard gem, sinca paranoia is in maintenance mode, but it takes more work to do the migration, so I'll do it in a next iteration.

There was an issue with recovering children that had (automatic) validation on presence of the parent. This has been fixed in #227.


3 More Paper Friendly Font Colors are added & Contact Form's Style is improved to Align with Overall Design.

For users, there was a limited option to choose font colours that match papers. According to my research, colours like "Orange", "Violet" & "Dodgerblue" are also a good choice for users. Also, pink is the least used one since it is lighter and hard to read.
In Contact Page, the Experience Form's design didn't match the overall design. I mean the Input Boxes had sharp edges even though they get curved when focussed.
But It would be better to have a small border-radius at the beginning and more border-radius on focus.
I am a GSSOC's Participant.

The pink colour is darkened by adding 'deep pink' without any change on the Display name.
Color's display name's first letter is capitalized for making it more readable and professional.
'dodgerblue' is added in the name of "Light Blue" where normal blue is also there.
'dark violet is added in the name of 'Violet"

All Colors:
colors


vendor + update previous commit

whoops, sorry :(

@xuto2 I guess after this merges we would need to bump AgentBaker in RP too?

/lgtm

I've prepared the pull request with the RP toggle and compared it against this branch for testing purposes, if you're able to cut a tag when this merges that would be great!

/lgtm

@alexeldeib use tag v0.0.32 if you need to vendor


Support for edits

This PR adds support for edits in FastSurfer.
Edits are enabled by the --edits flag for run_fastsurfer.sh.
Edits supported:
<asegdkt_segfile> via <asegdkt_segfile-w/o-ext>.manedit.
white matter segmentation (wm.mgz) via wm.manedit.mgz
white matter control points (via the standard FreeSurfer control points file)
talairach registration via mri/mri/transforms/talairach.xfm (overwriting talairach registration via mri/mri/transforms/talairach.xfm)
brain.finalsurfs.mgz via brain.finalsurfs.manedit.mgz
This PR also move the allow_root check into functions.sh and introduces run_it and run_it_cmdf function variants that are more clear wrt. argument splitting (see SC2206, SC2207 and similar).

Hi, thanks for the pr. It would be great to fix the spelling errors and also add a readme for edits, which can be pretty short in the first iteration, with a 1-2 sentence description of each possible edit.

Spelling should be fixed, documentation will need to be a future thing for now.

This branch is currently a mess of fixes and features for 6 different things. Those need to be separated into individual branches and PRs.
At a minimum, this version seems to work for longitudinal processing (still unclear for edits).

I rebased this to current dev which disentangles the edits from the other changes. Please check :

that everything is as wanted and
no purposeful changes from the earlier commits are undone

Also update your local with a hard reset.

Maybe there should be a check if the fastsurfer versions are the same between the original run and the rerun? @m-reuter
If this is just a warning or a message this will likely get ignored.
If this is a failure, it might get more annoying than not.
Solution might be to write a message to the log and let the user confirm it (enter).
But if nothing is attached to stdin, the process proceeds no matter what so only interactive sessions stop (this is how install scripts sometimes solve this)...

Maybe there should be a check if the fastsurfer versions are the same between the original run and the rerun?

I think warning message in the logs is fine. It should only stop if versions are not compatible any longer (and it would fail somewhere during processing). Otherwise, if someone edits things, that is changing more than most of our versions.


[BUG] Mount / Replication failure with k3s when using Ubuntu 22 Server ISO -> DeadlineExceeded desc = context deadline exceeded

Mount / Replication failure with k3s when using Ubuntu 22 Server ISO
When deployed a helm chart with pvc it is failing in the initial try but succeeding if left that for 15 - 30 min automatically.
If we have liveness probe  and readiness probe it is keep on failing.
Below is the error when described the pod
MountVolume.MountDevice failed for volume "pvc-" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Note: I am facing this issue only when we use Ubuntu 22 Server ISO.
I installed in vmware hypervisor. Tried updating with latest HWE Kernels as well.
Didnt face this issue in multupass jammy vm and in AWS Ubuntu 22 server vm.
In the failed ISO if used microk8s it is working fine.
Note: Getting this issue when using multiple nodes only.

Install 2 VMs using ubuntu 22.04 server iso
https://ubuntu.com/download/server
Install k3s in node 2 nodes as a cluster
Node 1:
Install k3s in first node using the below command
sudo curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--cluster-init  --disable servicelb --disable traefik --disable local-storage" sh -
Get Token using the below command to join second node:
sudo cat /var/lib/rancher/k3s/server/node-token
Node 2:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--server https://<first_node_ip>:6443 --token  --disable servicelb --disable traefik --disable local-storage" sh -
Install longhorn helm chart.
Install postgresql helm chart for validation
helm install psql oci://registry-1.docker.io/bitnamicharts/postgresql
Expected behavior
All the pods should come up successfully

Original behaviour
Failing initially for 15 min then coming up by itself automatically if we dont have readiness probe
kubectl get pvc data-psql-postgresql-0
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-psql-postgresql-0   Bound    pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a   8Gi        RWO            longhorn       8m51s
kubectl get pv pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                            STORAGECLASS   REASON   AGE
pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a   8Gi        RWO            Delete           Bound    default/data-psql-postgresql-0   longhorn                9m2s
Pod Events:
Events:
Type     Reason                  Age                    From                     Message

Warning  FailedScheduling        6m50s                  default-scheduler        0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
Warning  FailedScheduling        6m28s (x2 over 6m49s)  default-scheduler        0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
Normal   Scheduled               6m24s                  default-scheduler        Successfully assigned default/psql-postgresql-0 to vm2
Normal   SuccessfulAttachVolume  6m12s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a"
Warning  FailedMount             2m11s (x2 over 4m12s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning  FailedMount             40s                    kubelet                  MountVolume.MountDevice failed for volume "pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/db6bebfce3ba747df5bb36e75f75e2edddc3a8196c8b4fe88e7bdb0e0ffb327a/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/db6bebfce3ba747df5bb36e75f75e2edddc3a8196c8b4fe88e7bdb0e0ffb327a/globalmount: /dev/longhorn/pvc-e1900254-fcb7-412a-a04b-ca9d29eec81a already mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/db6bebfce3ba747df5bb36e75f75e2edddc3a8196c8b4fe88e7bdb0e0ffb327a/globalmount.
Normal   Pulling    37s  kubelet  Pulling image "registry-1.docker.io/bitnami/postgresql:16.2.0-debian-11-r17"
Normal   Pulled     19s  kubelet  Successfully pulled image "registry-1.docker.io/bitnami/postgresql:16.2.0-debian-11-r17" in 17.508s (17.508s including waiting)
Normal   Created    19s  kubelet  Created container postgresql
Normal   Started    19s  kubelet  Started container postgresql
Warning  Unhealthy  7s   kubelet  Readiness probe failed: <IP_ADDRESS>:5432 - no response
Support bundle for troubleshooting
supportbundle_92cb2ea6-db6b-4aa0-b0fe-d70905cd02dd_2024-02-21T08-27-29Z.zip

Environment

Longhorn version: 1.6.0
Impacted volume (PV):
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s - v1.28.6+k3s2

Number of control plane nodes in the cluster: 2
Number of worker nodes in the cluster: 0

Node config

OS type and version: Ubuntu 22.04.3 LTS
Kernel version:  5.15.0-94 and 6.5.0-18-generic
CPU per node: 4
Memory per node: 8

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): VMWARE

Additional context

In longhorn-csi-plugin log, we have 5 PVCs:

b7faa0d9-3e1a-43dc-81cb-fa56121d3467
d4e44af1-8ced-4335-a6f2-0077c4db4623
e38110af-19bb-4d74-a426-52cc0ad3350e
04990e50-5a7c-435a-94b6-498599da70fa
71e4e0f0-19f8-42ab-b105-49ed6ba3078f

Let's take as example the PVC b7faa0d9-3e1a-43dc-81cb-fa56121d3467,  the same is valid for the first 4 PVCs.
We have the following error:
2024-02-21T07:10:43.565093940Z I0221 07:10:43.564859    5598 mount_linux.go:452] `fsck` error fsck from util-linux 2.37.4
2024-02-21T07:10:43.565152157Z /dev/longhorn/pvc-b7faa0d9-3e1a-43dc-81cb-fa56121d3467 is mounted.
2024-02-21T07:10:43.565169464Z e2fsck: Cannot continue, aborting.

The strange thing that I see is that, after having successfully formatted the disk, the format start again and probably this lead to the error above:
2024-02-21T07:05:54.516800958Z time="2024-02-21T07:05:54Z" level=info msg="Formatting device /dev/longhorn/pvc-b7faa0d9-3e1a-43dc-81cb-fa56121d3467 with fsType ext4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/14d3c8622dbd93af988941e6f25b528b29bd6943855b18679120cf77662f96d2/globalmount with mount flags []" func="csi.(*NodeServer).nodeStageMountVolume" file="node_server.go:313" component=csi-node-server function=NodePublishVolume
2024-02-21T07:10:40.372702259Z I0221 07:10:40.372424    5598 mount_linux.go:526] Disk successfully formatted (mkfs): ext4 - /dev/longhorn/pvc-b7faa0d9-3e1a-43dc-81cb-fa56121d3467 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/14d3c8622dbd93af988941e6f25b528b29bd6943855b18679120cf77662f96d2/globalmount
2024-02-21T07:10:42.080259350Z time="2024-02-21T07:10:42Z" level=info msg="Formatting device /dev/longhorn/pvc-b7faa0d9-3e1a-43dc-81cb-fa56121d3467 with fsType ext4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/14d3c8622dbd93af988941e6f25b528b29bd6943855b18679120cf77662f96d2/globalmount with mount flags []" func="csi.(*NodeServer).nodeStageMountVolume" file="node_server.go:313" component=csi-node-server function=NodePublishVolume
2024-02-21T07:10:42.097947101Z time="2024-02-21T07:10:42Z" level=info msg="Formatting device /dev/longhorn/pvc-b7faa0d9-3e1a-43dc-81cb-fa56121d3467 with fsType ext4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/14d3c8622dbd93af988941e6f25b528b29bd6943855b18679120cf77662f96d2/globalmount with mount flags []" func="csi.(*NodeServer).nodeStageMountVolume" file="node_server.go:313" component=csi-node-server function=NodePublishVolume

The PVC 71e4e0f0-19f8-42ab-b105-49ed6ba3078f, which is the only one without errors, instead have only one of this log:
2024-02-21T07:05:58.158608284Z time="2024-02-21T07:05:58Z" level=info msg="Formatting device /dev/longhorn/pvc-71e4e0f0-19f8-42ab-b105-49ed6ba3078f with fsType ext4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/ee3fbed5d0ff84684e62f4f16d31b8ed64590700e4ad87620125e558db6a9803/globalmount with mount flags []" func="csi.(*NodeServer).nodeStageMountVolume" file="node_server.go:313" component=csi-node-server function=NodePublishVolume

Is that issue due to some disk partition ?

@derekbit Can be the error showed above the cause of the user's issue?


Can't create Blazor Client with SDK 3.0.100-rc1-014190

update the command in docs from
dotnet new blazorwasm -o WebApplication1

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c07e57dc-526f-9479-4695-de7d8f06eb9a
Version Independent ID: 8226ed72-69b2-3c93-a931-218bd11cf9c8
Content: Get started with ASP.NET Core Blazor
Content Source: aspnetcore/blazor/get-started.md
Product: aspnet-core
Technology: aspnetcore-blazor
GitHub Login: @guardrex
Microsoft Alias: riande

@smjltd Run the template install command ...
dotnet new -i Microsoft.AspNetCore.Blazor.Templates::3.0.0-preview9.19457.4

It has to be run after the SDK is installed.

you mean 3.0.100-rc1-014190

That's the SDK version ... not the package version that contains the templates.
Execute ...
dotnet new -i Microsoft.AspNetCore.Blazor.Templates::3.0.0-preview9.19457.4

... and try to create the app from the template again. It should work 🤞.

it worked thanks 😘


Add sticky positioning demo

Daily Prebuilt demo that shows how the CSS position:sticky; property can be used to keep the call on screen even when the user scrolls down and the call would otherwise scroll off screen.
Written in HTML/JS/CSS. Intentionally simple (didn't spend a ton of time on the CSS) because I don't want the actual focus (two lines of CSS!) to get lost in a sea of code.
As an addition to make it a little more interesting, the video call also shrinks on scroll so it takes up less space if the user is reading the content below the call.

hey @lazeratops little reminder for a final pr review :)


Added failing test

This will fail:
Failed asserting that 'League\Container\Test\Asset\B...erface' is an instance of class "League\Container\Test\Asset\Bar".

For some reason it returns a string instead of resolving a definition for BarInterface (aliased).
Can you confirm this is not intended?

@hannesvdvreken I wouldn't say it's intended, but was also never intended to be alias -> alias -> concrete, looking at possible recursion but need to have a think about the consequences

Potential use case:
$container->add(SomeContainer::class, fn () => new SomeContainer());
$container->add(InteropContainer::class, SomeContainer::class);
$container->add(PsrContainer::class, InteropContainer::class);

But honestly, I could write:
$container->add(SomeContainer::class, fn () => new SomeContainer());
$container->add(InteropContainer::class, SomeContainer::class);
$container->add(PsrContainer::class, SomeContainer::class);

Something to consider building into v4. For this version, I wouldn't bother.

https://github.com/thephpleague/container/commit/002a94cf83a8af40c0c0cfebab9352631efc6ce1 for 4.x @hannesvdvreken


Dot at the end of error message with directory path is misleading

Please, remove "." from the error messages where it contains the path to the directory.
It's slightly misleading when the user wants to copy and paste the path.
Previous installation of LLVM exist in: /home/rust/.espressif/tools/xtensa-esp32-elf-clang/esp-14.0.0-20220415-x86_64-unknown-linux-gnu.

Thanks for reporting it, I agree it's a bit confusing, do you think the following is clear enough?
Previous installation of LLVM exist in: '/home/rust/.espressif/tools/xtensa-esp32-elf-clang/esp-14.0.0-20220415-x86_64-unknown-linux-gnu'.

On another note, I don't think we should be bailing when there is an LLVM installed, instead, we should warn the user that we are reusing that installation and continue.


Revert "Add a pattern to adapt linalg.generic inputs to outputs."

Reverts google/iree#8440
This PR introduces other memref.alloca op issues. The memref.alloca ops are created even in simple input IR. E.g.,
func @add() -> tensor<33x49xf32> {
  %lhs = util.unfoldable_constant dense<1.0> : tensor<33x49xf32>
  %rhs = util.unfoldable_constant dense<0.5> : tensor<33x49xf32>
  %0 = mhlo.add %lhs, %rhs : tensor<33x49xf32>
  return %0 : tensor<33x49xf32>
}

Since we have https://github.com/google/iree/commit/da06cc91a85b1317a3c5cf6b890386364094fa3a checked in, we can try to vectorize all the operations to avoid the issue.

should be fixed in https://github.com/google/iree/pull/8499


Smooth time

Update Time charwise and create smooth effect

With this update, the seconds don't appear. Could you fix this?


Define API to control the cacheability of artifact transforms

Expected Behavior

capability to control cacheable of artifact transform similar to common task:
tasks.named("").configure {
outputs.cacheIf { false }
}
Current Behavior

Currently unable to control cacheability from artifact transform.
Context

Our android build having this issue: https://issuetracker.google.com/issues/169266011
From chrome trace I can see artifact transform dominating our build and each of them are very large artifact. They also easy to miss and consuming lot of resource.

Sorry for the late reply.
This issue will be closed as a duplicate of

https://github.com/gradle/gradle/issues/27706

Please add your use case and 👍 to that issue.
If you think our analysis is wrong, please provide us with more detailed information explaining why.


Why does line 110 of message-level-store.ts work on main branch

"""const { RESULT: indexResults } = await this.index.QUERY(finalQuery);"""
this is the code on line 110 of the file mentioned however the query function expects a token then an optional options arg
"""QUERY(query: Token, options?: QueryOptions): Promise;"""
so right now my code wont test due to this error:
src/store/message-store-level.ts:111:61

error TS2345: Argument of type '{ NOT: { INCLUDE: { AND: SearchIndexTerm[]; }; EXCLUDE: { AND: SearchIndexTerm[]; }; }; }' is not assignable to parameter of type 'Token'.

111     const { RESULT: indexResults } = await this.index.QUERY(finalQuery);

I think you pulled the source code recently, but did not npm install after, there was an update to type dependencies recently.

ah yes that was the reason my apologies

No worries! It happens to all of us! 😆


Decouple tests from external dependencies

ping @weierophinney @zendframework/community-review-team. Fix stdlib develop branch tests


WIP: Version updated to nightly

Description
quay image tag changed to nightly
GitHub Issues
List the GitHub issues impacted by this PR:

GitHub Issue #

https://github.com/dell/csm/issues/1435

Checklist:

[x] I have performed a self-review of my own code to ensure there are no formatting, vetting, linting, or security issues
[x] I have verified that new and existing unit tests pass locally with my changes
[x] I have not allowed coverage numbers to degenerate
[x] I have maintained at least 90% code coverage
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have maintained backward compatibility

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

@mgandharva Please add TODO comment as to when this night will be changed to v2.12 tag

@mgandharva Please add TODO comment as to when this night will be changed to v2.12 tag

@francis-nijay TODO comments are added. Please review it once.


How to convert dynamic columns or rows?

Here is the table format:

key
values
values-zh

key1
first
第一

key2
second
第二

key3
third
第三

It's generated from Android res folder for multi-translation exporting, and the columns and rows are generated dynamically, I'm writing it to CSV or XLS row by row using Apache POI, how can I do the same thing by this lib? Thanks!

Sorry, I have no idea what you want.

Sorry for this, I just know how to finish it. Thanks!


changed the /roleme command name to /rolemenu

this'll help clarify that the command doesn't give roles itself, but instead spawns a menu that does that

You gotta change the welcome message too.

@conaticus on it

should now have the changed welcome message alongside the new command name


Providing Swagger Documentation in a Separated App?

Hi,
currently when I wrap my app with flasgger I have as a result my app + the Swagger documentation in one app.
Is it possible to separate the Swagger app from the app that needs to be documented? I.e. if I don't want to mingle my production app with other middleware but rather have a separate app for the documentation that only produces the documentation (without providing the actual methods from the production app)?

One solution is passing the full specs in template= argument.
We can think about implementing a way to generate it from a bunch of yaml files names with the method names.


[pkg_tar] Unable to use select in files attribute

This way does not work
pkg_tar(
    name = "tar-binaries",
    files = select({
        "@platforms//cpu:arm64": {
            "@kubectl_linux_arm64-lo//file": "kubectl%s" % KUBECTL_LO,
        },
        "@platforms//cpu:x86_64": {
            "@kubectl_linux_amd64-lo//file": "kubectl%s" % KUBECTL_LO,
        },
    }),
    mode = "0555",
    owner = "1001.1001",
    package_dir = "/opt/kublr",
)

DEBUG: /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_pkg~0.9.1/pkg/private/tar/tar.bzl:318:22: @//images/kubectl:tar-binaries: you provided a non dictionary to the pkg_tar `files` attribute. This attribute was renamed to `srcs`. Consider renaming it in your BUILD file.

But this notation works fine:
pkg_tar(
    name = "tar-binaries",
    files = {
        "@kubectl_linux_amd64-lo//file": "kubectl%s" % KUBECTL_LO,
    },
    mode = "0555",
    owner = "1001.1001",
    package_dir = "/opt/kublr",
)

I suppose this is a bug in this logic
https://github.com/bazelbuild/rules_pkg/blob/main/pkg/private/tar/tar.bzl#L318
Proposed fix:
Change
if not hasattr(kwargs["files"], "items"):
to
if not types.is_dict(kwargs["files"]):

Did you try that? I'll review a PR.

Sorry, I have tested my suggestion and it behaves the same.
So it problem in select()

Closing this issue.


Digital PINs directly connected to MicroController only Segments on Shift Register

not compiled/tested

compiled, issue fixed and verified
documentation is still missing
current testing can be found in
https://github.com/bridystone/SevSegShiftTest


Delete branches on merge.

Hey @geky! Just a project maintenance recommendation,  go under the github settings an enable "Automatically delete head branches." And then maybe go back through and delete all the merged-in branches.
After pull requests are merged, you can have head branches deleted automatically.

[ ] Automatically delete head branches
    Deleted branches will still be able to be restored.

Ah, I'm not sure this will work since littlefs has a number of long-lived branches that occasionally get merged into each other (devel, v2, v2-prefix).
But I can go through and delete the merged branches, and change gears to habitually delete branches on merge.
Out of curiosity, does GitHub document how long deleted branches can be restored for? That's kind of a strange subdescription.


Multilanguage support

I start a 'proof-of-concept' for multilanguage support.
As i described in #179 , the idea is:

create a form in default language (settings.LANGUAGE_CODE)
create a translation for all lables in one of the languages from (settings.LANGUAGES)
render the form will used the translation, if available or fallback to default language

This worked yet for me. But it needs some love and tests. I'm waiting for feedback how this pull request can be accepted and merged ;)

ping

This is already implemented in https://github.com/stephenmcd/mezzanine/tree/master/mezzanine/forms - the implementation here should mimic that.


Use dig in layouts guide ERB

If one uses the suggested layout for the ViewComponent default layout, navigating to localhost:3000/rails/view_components will throw an error due to the deeply nested nature of the Lookbook params hash. Using dig allows for safe access into the hashes without causing problems if any of the keys don't exist.
Hash#dig was introduced in Ruby 2.3, so will be available in all supported versions of Lookbook.
There's a few other instances of params[:some][:deep][:key] in the docs, but I thought this one may be a better target since it could cross the Lookbook boundary to ViewComponent.

Good catch @neanias! Thanks for the PR.
About the docs, it makes sense to improve the syntax there as well, what do you think @allmarkedup?

I'm happy to change the others as part of this PR if that helps the sweep?

@neanias @Spone sorry about the slow reply! This is a good catch, thank you.
@neanias if you would be willing to update the other places that this syntax is used in the docs that would be fantastic, thank you. If you add it to this PR then I'll merge once ready. Appreciate your time on this!

I think that's the main ones. I found page.data[:brand_colors][:red] but I think that one's ok?

I think that's the main ones. I found page.data[:brand_colors][:red] but I think that one's ok?

@neanias yep that should be fine I think. Many thanks for your time with this, will merge now 👍


Improved the rbac settings of yurt-manager component

The RBAC settings have been refactored to split the resource configurations into individual lines for better readability and clarity. This change separates the RBAC rules for different resources, ensuring each resource and its corresponding verbs are clearly defined.
#1812

@MeenuyD please run make manifests to fix ci/verify error.

It seems that another pull request #1829 has been merged, so i will close this pr.


Enabeld: Config from Database

Hi all,
to have a more configurable Module perspective, have you thought about to have the Module itself as a Model with attributes at the database? Of course this entry need to be cached, but I think having a description and the possibility enable/disable at UI makes sense.
I looking in addition into the way to have the enable/disable depending on environment / database setting in a multi tenant application, means I would have depending on domain a module enabled or not.
Even if you say, a config on database level makes no sense for you, it would be great to have the option to overwrite different function by creating a own Service Provider by extending yours and change a config setting. My favorit example for that is Saptie/permission e.g.
 'models' => [
        /*
         * When using the "HasRoles" trait from this package, we need to know which
         * Eloquent model should be used to retrieve your permissions. Of course, it
         * is often just the "Permission" model but you may use whatever you like.
         *
         * The model you want to use as a Permission model needs to implement the
         * `Spatie\Permission\Contracts\Permission` contract.
         */
        'permission' => Spatie\Permission\Models\Permission::class,
        /*
         * When using the "HasRoles" trait from this package, we need to know which
         * Eloquent model should be used to retrieve your roles. Of course, it
         * is often just the "Role" model but you may use whatever you like.
         *
         * The model you want to use as a Role model needs to implement the
         * `Spatie\Permission\Contracts\Role` contract.
         */
        'role' => Spatie\Permission\Models\Role::class,
    ],

What do you think about this?

Hi,
Yes, it has been discussed in multiple issues before, one open still.
And something is under way here... https://github.com/nWidart/laravel-modules/tree/feature/database-modules
Though, it's already possible to have a ui to enable/disable modules and so on, I do this on AsgardCMS.
You can also enable/disable modules based on roles/tenant etc. etc.
I don't understand your service provider thing. Laravel already allows overriding over services providers built in.

"You can also enable/disable modules based on roles/tenant etc. " Yes I see, but I look to have on the same coding tenant1.domain.com a module enabled and under tenant2.domain.com not. So I need a where condition while reading if a module should be enabled or not, and so the caching need to be adjusted too.
Regarding your comment about the provider, more specific; how would I overwrite your function all() and related caching within  laravel-modules/src/FileRepository.php to save the enabled status not in cache or file but in a model which I can extend (because my caching base on model with where('tenant', 'tenantid')?

When you have a look at caffeinated/modules they have two additional parameters in there config:
`/*
|--------------------------------------------------------------------------
| Default Module Driver
|--------------------------------------------------------------------------
|
| This option controls the module storage driver that will be utilized.
| This driver manages the retrieval and management of module properties.
| Setting this to custom allows you to specify your own driver instance.
|
| Supported: "Local" or "Custom"
|
*/
'driver' => 'Local',

/*
 |--------------------------------------------------------------------------
 | Custom Module Driver
 |--------------------------------------------------------------------------
 |
 | Using a custom module driver, the 'driver' value need to be set to 'custom'
 | The path to the driver need to be set in addition at custom_driver.
 |
 | @warn: This value will be only considered if driver is set to custom.
 |
 */

// 'custom_driver' => 'Caffeinated\Modules\Repositories\LocalRepository',`

This need to be read at your  laravel-modules/src/Providers/ContractsServiceProvider.php
Old:
$this->app->bind(RepositoryInterface::class, LaravelFileRepository::class);
New:
`
$driver = ucfirst(config('modules.driver'));
    if (strcasecmp($driver, 'Custom') == 0) {
        $repositoryClass= config('modules.custom_driver');
    } else {
        $repositoryClass= LaravelFileRepository::class;
    }

    $this->app->bind(RepositoryInterface::class, $repositoryClass);

`

I see, that's what the branch I linked is for.
I'll close this in favour for #171


Vulnerability roundup 93: fossil-2.10: 1 advisory [8.8]

search, files

[ ] CVE-2020-24614 CVSSv3=8.8 (nixos-20.03)

Scanned versions: nixos-20.03: 0d0660fde3b.
Cc @maggesi
Cc @viric

20.03 is frozen and EOL.
Closing.


Introduced first-class Permissions protocol

This will replace the entire Permissions interface.
This PR only contains the support of permission life-cycle management through Request, Grant, and Revocation records.
The actual replacing/swapping out the existing logic the depend on Permissions messages will come in the next PR.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.50%. Comparing base (1e9cd85) to head (dd4bef9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #713      +/-   ##
==========================================
+ Coverage   98.46%   98.50%   +0.03%
==========================================
  Files          74       75       +1
  Lines       10195    10459     +264
  Branches     1511     1517       +6
==========================================
+ Hits        10039    10303     +264
  Misses        150      150
  Partials        6        6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Possible for jsmpeg to record from Camera?

Hey,
I was wondering if it is possible to capture video from camera and save it as mpg using the jsmpeg library ? (Not streaming)
Current state of solutions is recrodRTC which uses whammy, but maby it is possible to encode using jsmpeg?

So RecordRTC can recode from camera?

It  uses whammy to encode the images into .webm video file, but the browser just can't handle the data stored in the arrays (due to framerate or amount of data), so the end result would be that you record 40sec video, where 15-20 seconds are fine and the rest of the frame are just bad/unplayable.

@phoboslab any idea on this?

jsmpeg is a decoder, not an encoder. If you're streaming (an encoded mpeg1 video) to jsmpeg, you can create and download a binary Blob of that stream as outlined here: https://github.com/phoboslab/jsmpeg#stream-recording
Never heard of whammy or recordRTC, sry.

@phoboslab yes, i understand that jsmpeg is not an encoder (dang though!!), the question is if it is possible to capture the camera (html5) data and redirect it to the jsmpeg streamer in a way?

Well, no. jsmpeg needs an mpeg1 encoded video and currently no browser supports encoding mpeg1 from getUserMedia.
You could port an mpeg1 encoder to javascript, feed it raw webcam images from getUserMedia and create the mpeg file directly in the browser. That would be quite a bit of work, though. Plus, encoding is way more computationally intensive than decoding, so it probably wont be fast enough for mobile.
What are you trying to do, exactly?

@phoboslab
Well, using your library i can play mpeg files (synched with audio as a separate track), but I am also in need to record audio/video from the camera, but the current state of HTML5 implementation scheme are incomplete, and I am unable to record video & audio for more than 20secs without the browser bailing out with incomplete frames and corruption. How it works (video) is by drawing each frame from the camera onto a canvas, then getting that image from the canvas and then feed it into a library called Whammy, which returns a .webm file. than you export both .webm and .wav file with xhr into your file system (via php for example)
The only only solution (aside from flash) is very bad :( I've tried to reduce the overhead of frames by outputing a .webm and .wav every 5 frames and then concat them together into a single file using ffmpeg, but this way it returns a video file with audio offsets (no sync) and sometimes even currupt video.
I am looking into an HTML5 solution only (since flash is a 3rd party)

What's the target audience? I assume this needs to work on a number of platforms, for (potentially) many clients simultaneously? What kind of quality to do you need? What connection speeds to you assume?
I've had success saving individual PNG frames via XHR to a localhost server and - in a later step - stitching these PNGs frames into a video. I used this to capture high quality video from my HTML5/Canvas games.
If the video quality isn't that important, it might even work on a public server. Make the canvas small-ish (~320x240) use canvas.toDataURL("image/jpeg", 0.3) to encode low quality jpegs, collect 20 per second an send them over in bundles to the server via XHR. You could even fudge an MJPEG header and create an MJPEG file completely clientside.
This is all quite adventurous, though :)

The target video shouldn't be high quality, it's mainly to record a tutorial like, so ~320x240 could be suitable ineed.
Well, mainly browsers, so no mobile anytime soon.
I'll venture into that mjpeg solution and see how it goes!
Do you think that encoding a 320 or even 640 frames into a final mpeg would not be possible in real time even if a port is written?

For tutorials/screencasts you could also use a lower framerate to reduce the size.
A RaspberryPi is able to encode a 25fps 640x480 MPEG1 video in realtime (with native code), so you should definitely be able to do this in Desktop Browsers in JavaScript :)

Yes, that's what I thought so too. I MIGHT check out ffmpeg source code, but I definatelly would love a c encoder source coder that is in-depended and simple to port.

mpeg2enc from the mjpeg tools is reasonably stand-alone: from http://mjpeg.sourceforge.net/
Good luck :)
Btw.: this all sounds really interesting. If this is a commercial project (i.e. you'd be able to pay me), I'd be happy to look into it more.

Thanks! I'll check it out.
It's not really up to me (the paying part) but if it will need to, i'll certainly let you know!

Well, no. jsmpeg needs an mpeg1 encoded video and currently no browser supports encoding mpeg1 from getUserMedia.
You could port an mpeg1 encoder to javascript, feed it raw webcam images from getUserMedia and create the mpeg file directly in the browser. That would be quite a bit of work, though. Plus, encoding is way more computationally intensive than decoding, so it probably wont be fast enough for mobile.
What are you trying to do, exactly?

How about WebAssembly? I found one, this one is very fast:
https://github.com/AlexVestin/WasmVideoEncoder
DEMO link: http://videoncoder.s3-website.eu-central-1.amazonaws.com/


Fix multiple synthesis libraries

Fixes https://github.com/The-OpenROAD-Project/OpenLane/issues/683

@donn
is this PR tested against designs/spm/config.tcl by setting below variable?
set ::env(STD_CELL_LIBRARY) "sky130_fd_sc_hvl"
Still I'm facing error as below
[INFO ODB-0222] Reading LEF file: /openlane/designs/spm/runs/def_hvllib/tmp/merged_unpadded.lef
[INFO ODB-0223]     Created 13 technology layers
[INFO ODB-0224]     Created 25 technology vias
[INFO ODB-0225]     Created 69 library cells
[INFO ODB-0226] Finished LEF file:  /openlane/designs/spm/runs/def_hvllib/tmp/merged_unpadded.lef
Error: sta.tcl, 38 cannot read file  /home/vijayan/OPENLANE_FLOW/Aug15_2021_OpenLane/OpenLane/pdks/sky130A/libs.ref/sky130_fd_sc_hvl/lib/sky130_fd_sc_hvl__tt_025C_3v30.lib /home/vijayan/OPENLANE_FLOW/Aug15_2021_OpenLane/OpenLane/pdks/sky130A/libs.ref/sky130_fd_sc_hvl/lib/sky130_fd_sc_hvl__tt_025C_3v30_lv1v80.lib .
[ERROR]: during executing: "openroad -exit /openlane/scripts/openroad/sta.tcl |& tee >&@stdout /openlane/designs/spm/runs/def_hvllib/logs/synthesis/1-opensta"
[ERROR]: Exit code: 1
[ERROR]: Last 10 lines:
child process exited abnormally

No. If the issue persists please reopen the issue so we can investigate.

If I hack scripts/openroad/sta.tcl in this way, then it works (and crashes later at step 7, uff):
read_liberty "/usr/local/share/pdk/sky130A/libs.ref/sky130_fd_sc_hvl/lib/sky130_fd_sc_hvl__tt_025C_3v30.lib"
read_liberty "/usr/local/share/pdk/sky130A/libs.ref/sky130_fd_sc_hvl/lib/sky130_fd_sc_hvl__tt_025C_3v30_lv1v80.lib"
#read_liberty $::env(LIB_SYNTH_COMPLETE)

So yes, the read_liberty function does not work when multiple libs are passed via LIB_SYNTH_COMPLETE, which happens when using the HVL lib.

and crashes later at step 7, uff

BTW, then throwing this error:
[ERROR STA-0026] /openlane/designs/audiodac/runs/RUN_2021.11.23_15.13.33/tmp/trimmed.lib line 31779, syntax error, unexpected KEYWORD, expecting $end.
Error: replace.tcl, 52 STA-0026
[ERROR]: during executing: "openroad -exit /openlane/scripts/openroad/replace.tcl |& tee >&@stdout /openlane/designs/audiodac/runs/RUN_2021.11.23_15.13.33/logs/placement/7-replace.log"
[ERROR]: Exit code: 1
[ERROR]: Last 10 lines:
child process exited abnormally

[ERROR]: Please check openroad  log file
[ERROR]: Dumping to /openlane/designs/audiodac/runs/RUN_2021.11.23_15.13.33/error.log

Ah, thanks for the diagnostic help.


Unable to read account history over 300 items

This is potentially an issue with the amount the api will let you request without an authorised request token. Is there a way around this or would every user of this script be required to make their own token to use this?
To reproduce: ghstat -u asottile
You only get a value in the high 200s instead of the 900+ that's expected

As per asottile - this should work to refactor to resolve this issue - read the page headers to see if there are multiple pages and parse accordingly. This will also avoid some potential bugs that haven't been run into yet


Caffe integration test fails because it is not able to download files from caffe server

Description
Caffe integration test fails because it is not able to download files from caffe server.
Failed build:
https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/v0.12.0/4/pipeline

If license permits, we should be hosting these files in S3 and downloading from there for the purposes of tests instead of depending on external servers.
The test can also check if the file already exists in the file system to avoid downloading again and again every time.

Environment info (Required)
CI builds.
Steps to reproduce
Build and run integration test.

This is more likely a slave setup problem. The link indicated in the error works fine here. http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel


NEW: Implement accept attribute in FileField (closes #7279)

next time push to your fork ;)

Ugh, I ran a composer update that moved framework into vendor/, which must’ve re-created the git directory with the wrong remotes. Good job I didn’t push to any of the main branches 😛


[libdatachannel] Update, cleanup

Complements https://github.com/microsoft/vcpkg/pull/40335

All features are tested successfully in the following triplet:

x86-windows
x64-windows
x64-windows-static

Tested usage successfully by libdatachannel[*]:x64-windows:
libdatachannel provides CMake targets:

  # this is heuristically generated, and may not be correct
  find_package(LibDataChannel CONFIG REQUIRED)
  target_link_libraries(main PRIVATE LibDataChannel::LibDataChannelStatic)


Asynchronous RLHF: Faster and More Efficient Online DPO

This implements a proposed faster and more efficient paradigm: asynchronous RLHF. See the paper: https://arxiv.org/abs/2410.18252
reasoning
vllm inference is generally faster than hf generate. We want to separate the generation and training so we can use fast generation libraries / utilities
this proposes a simple first solution: run training on n gpus and generation on 1 gpu. This can be extended to run generation on more GPUs but,in practice for >=8 GPU setups with 8B models and less, 1 gpu for generation tends to be fine.
setup
We create an asynchronous trainer for Online DPO that uses vllm for generation. The generation GPU has vllm started on a separate python thread and communication between training and generation is via Queues. The training looks something like this:

the training thread gets the batch of prompts
send data
a. training thread sends batch of prompts and current model weights to generation
b. generation thread sends previous prompts with generated completions to training
parallel training and generation
a. training thread calculates reward, then trains on previous prompts and completions, updates weights
b. generation thread generates completions to each prompt
back to step 2
...
at the end of training the training thread sends None for both prompts and parameters so the generation thread closes itself

Example usage for a 4 GPU setup is
accelerate launch --num_processes 3 examples/scripts/dpo_online_async.py \
    --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft  \
    --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \
    --dataset_name trl-lib/tldr \
    --learning_rate 5.0e-7 \
    --output_dir pythia-1b-tldr-online-dpo \
    --per_device_train_batch_size 8 \
    --gradient_accumulation_steps 16

the generation GPU is by default the accelerate.num_processes + 1 GPU, so GPUs [1,2,3] are for training and GPU 4 is generation with vllm.
notes

online_dpo_trainer currently extends the regular huggingface Trainer and is limited to generating one minibatch of samples and training one step on those samples. As argued in the paper, more training steps on data (num_ppo_epochs) or generating more minibatches (num_mini_batches) can be useful. For this reason, AsyncOnlineDPOTrainer follows the style of RLOOTrainer.

to test functionality, I've added SyncOnlineDPOTrainer that has the exact same structure as Async but is synchronous and uses hf generate. I can remove it for the final submission,

Before submitting / To Do

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,   Pull Request section?
- [ ] Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines.
[ ] Did you write any new necessary tests?

I am working on docs. Do we want tests similar to RLOOTrainer ?
Who can review?
@qgallouedec @lewtun anyone else! comments welcome

What is the primary difference between this PR and #1628 ?

This is an updated and multi-gpu extension of #1628. It is also work between @vwxyzjn and I!
Instead of keeping vllm models on the same GPU, we move them to another. It also uses the more flexible vllm_utils.py written by @vwxyzjn in allenai/open_instruct (https://github.com/allenai/open-instruct/blob/main/open_instruct/vllm_utils.py) which allows using any version of vllm as opposed to the fixed 0.4.2 from #1628.
Finally, this has been tested and verified to match regular Online DPO performance while being faster and more efficient, see our new preprint https://arxiv.org/abs/2410.18252

Hi, I have a question, if I have multiple machines with 8 cards, how would the allocation of gpus for vllm look like? Or is this feature currently supported?

Hi, I have a question, if I have multiple machines with 8 cards, how would the allocation of gpus for vllm look like? Or is this feature currently supported?

I managed to solve this problem. The key point is to replace the part about group_ranks in custom_initialize_model_parallel and init_world_group in vllm_utils.py with group_ranks = [[torch.distributed.get_rank()]]. Maybe you can update it to the new version

Hi, is it possible to use single GPU for both training and inference? Thanks!

Hi, is there any updates? Thanks!


Really? It should look like:

Originally posted by @togenyan in https://github.com/togenyan/Yo-kai_Editor_Getto/issues/12#issuecomment-444453201

The difference between your photo and my editor is that the list and the item are Japanese. My editor has the item in English (excluding the yokai list).


Discord link seems incorrect

Would love to join the chat, but it looks like the discord link is perhaps broken.

Not sure what the issue is, I have updated the link in case it was broken.
It is:
https://discord.gg/jjy2aJ3hj9


How to filter specific user using JWT?

Overview
The JWT token is stored in the headers as follows.
headers:
{
"AUTHORIZATION": "Bearer [JWT]",
}

When a user sends a GET /log request, I want to filter the APILogsModel based on user_id. (i.e. users can only see their own request history)
Pseudo code:
class LogView(generics.ListAPIView):
    serializer_class = LogSerializer

    def get_queryset(self):
        token_str = self.request.META.get("HTTP_AUTHORIZATION", " ").split(" ")[1]
        access_token = AccessToken(token_str)

        ##################################################
        Filter by access_token
        ##################################################

        return queryset

Problem
There is no field for user information in APILogsModel. Therefore, it must be extracted from the JWT stored in headers. However, it seems inefficient to extract it from all headers every time a request comes in.
Conclusion
Is there any other efficient way? (Is it possible to override a model or middleware?)

Hi @LJiHye,
I suggest you to create a separate model (table) to store user-related information that you required (not all) and store the access log there, by subscribing to the API log signals.
Actually, Authorization is supposed to be hidden and not even the developers should know about the API-related user's information.

@vishalanandl177
Thank you for answer. I seem to know a little bit 🙂
However, I'm not familiar with this project yet, so I don't know what to do with it.
So, how can the custom model be set in settings.py? And do I have to override the default middleware APILoggerMiddleware as well?


Coverage

Using tarpaulin for calculating coverage, but it doesn't always work - lots of segfaults. If I can get it to reliably work, then running it on CircleCi will require updating the docker image with the binary (probably want to use an image with it already included so it's not part of build time) and uploading the report to CodeCov.


Cleaning up needed

from @georgemarshall

Annotating function return types would be nice
The mixing f strings and format makes things a bit inconsistent
If you are 3.5+ only, I would consider replacing requests with aiohttp

Fixed the first two things. I'll circle back around and consider aiohttp once I've learned more about it.


No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils (numpy==1.19.1)

Windows 10
Python: 3.11.2
python3: Python 3.10.10
Flask 2.2.3
Hi @fractalego and fact_checking team. So sorry to bug you here. but this error while installing fact_checking comes from numpy=1.19.1.
Project: Trying to build a url fake news app using flask
error: subprocess-exited-with-error

No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils

**libraries mkl_rt not found** in ['C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\\lib', 'C:\\', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\\libs']
	NOT AVAILABLE

**libraries blis not found** in ['C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\\lib', 'C:\\', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\\libs']
        NOT AVAILABLE

**libraries openblas not found** in ['C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\\lib', 'C:\\', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\\libs']

**libraries tatlas not found
libraries satlas not found
libraries ptf77blas,ptcblas,atlas not found
libraries f77blas,cblas,atlas not found
 libraries blas not found
libraries mkl_rt not found
libraries openblas not found
libraries openblas,lapack not found
libraries flame not found**

Hi lexycole, apologies for my late reply. I'm not sure what the issue is. Please let me try to replace it tomorrow

Hi Lexycole, I cannot reproduce your error. However, searching online I found a potential fix: https://stackoverflow.com/questions/74359405/error-while-installing-pycaret-no-module-named-numpy-distutils-msvccompiler
It seems to be related to whether you have visual-cpp studio utils installed. Please try out that solution and let me know if it works.

HI @fractalego, Thanks for the replies. I will be installing visual-cpp studio right now


Extend secrets docs with key to path mapping and automatic updates

Secret volume plugin has been refactored. Currently, AtomicWritter is used to project all keys.
Allowing to automatically refresh projected secrets.
At the same time, operator can specify target file for each secret and thus override the default projection location.
Cherry-picked from #478.

@johndmulhausen

@ingvagabund  In PR #478 you refer to this as being intended for the release-1.3 branch, but this is opened against master?

Note, if this is not documenting 1.3-specific stuff, we can close this PR regardless. The PR in master will do; everything we do in master will end up in release-1.3. LMK!

Note, if this is not documenting 1.3-specific stuff, we can close this PR regardless. The PR in master will do; everything we do in master will end up in release-1.3. LMK!

It is, the secret extension was merge in 1.3. That was the reason I created this PR. Again, you asked for it.

Merging, marking as to-be-iterated in bug #784


ESP32 support?

Hey, I just started using ESP32 in my project and it works great on C and I'm wondering if it is possible to launch Swift on it.
@dragomirecky could you please tell is it possible to launch swift on ESP32 or not?

I believe the original author of this project is no longer active, if you are willing to send me a board, I can try implementing support (just like https://github.com/swift-embedded/swift-embedded/issues/6 )

@demhademha  sounds great, I followed you on twitter let's discuss it there


sqlfluff: 2.1.2 -> 2.2.0

Diff: https://github.com/sqlfluff/sqlfluff/compare/refs/tags/2.1.2...2.2.0
Changelog: https://github.com/sqlfluff/sqlfluff/blob/2.2.0/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 247170 run on x86_64-linux 1

2 packages built:

sqlfluff
sqlfluff.dist

Result of nixpkgs-review pr 247170 run on aarch64-darwin 1

2 packages built:

sqlfluff
sqlfluff.dist


Thank you!

I just want to say thank you for writing this plugin. I was unsatisfied with the built-in parcel image transformer and its heavy dependency chain, and this plugin has simplified my build a lot.
Thanks! 🥇

Hey @agersant! Thank you for such kind words! I'm happy that my code help others :-)


feat: add dart language server

Why
Even though some nice Dart language server configs are present in nvim-lspconfig, the language server itself has not yet been made available in the Mason's core package registry.
What
Add it.

@williamboman, when you get a chance, can you review this PR? Thanks in advance.


Describe the solution you'd like
can you include dockerfile into the project? the image in docker hub seems to be a bit behind

I think you have confused the project... This is just a library and so there will never be a Docker image here ;)
If you meant eufy-security-ws, the new version was released yesterday.


feat: enable jupytext in binder

Should we come up with a Markdown demo, or just let users play about themselves?

Good point -- even a simple demo with the default headers and a dropdown would be helpful!


Update CONTRIBUTING.md w/ detail on opening issues

Again, borrowed heavily from the Redux project.
cc @mattrobenolt @LewisJEllis

This change is

Review status: 0 of 1 files reviewed at latest revision, 2 unresolved discussions.

CONTRIBUTING.md, line 27 [r1] (raw file):

* If you are using the Raven CDN (http://ravenjs.com)

Lastly, it is strongly encouraged to provide a small project reproducing your issue. You can put your code on JSBin or, for bigger projects, on GitHub. Make sure all the necessary dependencies are declared in package.json so anyone can run npm install && npm start and reproduce your issue.

We recommend JSBin here but JSFiddle in the template; let's either pick one or list both in both places?

CONTRIBUTING.md, line 112 [r1] (raw file):

_This is a checklist for core contributors when releasing a new version._

  * [ ] Verify TypeScript [language definition file](https://github.com/getsentry/  * raven-js/blob/master/typescript/raven.d.ts) is up to date

Maybe this is some markdown-fu that I don't know, but what are all these near-end-of-line asterisks for? Looks like maybe a regex gone wild

Comments from Reviewable

Review status: 0 of 1 files reviewed at latest revision, 2 unresolved discussions.

CONTRIBUTING.md, line 27 [r1] (raw file):
Previously, LewisJEllis (Lewis J Ellis) wrote…
> We recommend JSBin here but JSFiddle in the template; let's either pick one or list both in both places?

Switched to JSFiddle.

CONTRIBUTING.md, line 112 [r1] (raw file):
Previously, LewisJEllis (Lewis J Ellis) wrote…
> Maybe this is some markdown-fu that I don't know, but what are all these near-end-of-line asterisks for? Looks like maybe a regex gone wild

This was me not paying attention to line wrapping in my editor. Fixed.

Comments from Reviewable

Reviewed 1 of 1 files at r1, 1 of 1 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


resolve #135

添加中文配置例子文档  docs/zh_cn/example/telecom_unicom.md

Codecov Report

Merging #136 into develop will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #136      +/-   ##
===========================================
+ Coverage    53.13%   53.15%   +0.01%
===========================================
  Files          208      208
  Lines        21108    21108
===========================================
+ Hits         11216    11220       +4
+ Misses        8625     8622       -3
+ Partials      1267     1266       -1

Impacted Files
Coverage Δ

bfe_spdy/server_write_sched.go
78.46% <0%> (-1.54%)
:arrow_down:

bfe_http2/server.go
71.67% <0%> (ø)
:arrow_up:

bfe_spdy/server_conn.go
78.53% <0%> (ø)
:arrow_up:

bfe_websocket/server_conn.go
75% <0%> (+0.75%)
:arrow_up:

...e_modules/mod_trust_clientip/mod_trust_clientip.go
46.15% <0%> (+3.29%)
:arrow_up:

bfe_util/fixed_pool.go
100% <0%> (+16.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff346ed...4d60a14. Read the comment docs.

To be improved  @tianxinheihei


unable to download youtube shorts

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE

Description

WRITE DESCRIPTION HERE

Your issue is bad, and you should feel bad.

https://github.com/ytdl-org/youtube-dl/issues/31166#issuecomment-1213632596 applies.


bug: leverage bar selection

Steps:

select 8x bar
select the top bar of 4x

Expected: the top bar to be selected
Actual: the bottom bar is selected

https://github.com/derivable-labs/leverage-slider-comp/commit/a453fd3f74b786d3c80fd76b17136c8449e815a4


From 1.7.2, upgrade to Orchard CMS 1.9.2 makes the Body Parts missing

Hi,
I upgraded from 1.7.2 to Orchard CMS 1.9.2.
In the Upgrade, I am unable to migrate the body parts for it gets stuck for a record:

And I see that the edit page body is blank for all:

Please help.
Thank you.

I enabled TinyMCE module and it worked.


Kernel can't represent annotations on locals, parameters, or type parameters

Kernel has no way of representing the annotations @Foo() in the following code:
class Foo {
  const Foo();
}

void test(@Foo() x) {
  @Foo()
  var y;

  @Foo()
  void f() {}
}

class C<@Foo() T> {}

Any idea when we can expect this to get fixed?  It came up in https://dart-review.googlesource.com/c/sdk/+/5296

I guess it suffices to add a list of metadata annotations to TypeParameter and VariableDeclaration?  Dima, will you do that?

Yep, I'm on it.

The annotations are added in https://github.com/dart-lang/sdk/commit/e7a29d8dfbf873917096e00d3468bfd08aaca867.


Fix iter typehint

What does this PR do?

Fixes # (issue)
efore submitting

[x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[] Did you make sure to update the documentation with your changes?
[] Did you write any new necessary tests? [not needed for typos/docs]
[x] Did you verify new and existing tests pass locally with your changes?
[ ] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

[x] Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

@kaushikb11 is it valid PR, I see 0 change here :confused:


Long stall when interrupting a long-running compilation with Ctrl+C on MSYS

nightly-x86_64-pc-windows-msvc (default)
rustc 1.47.0-nightly (2d8a3b918 2020-08-26)

mintty 3.3.0 (x86_64-pc-msys)

To reproduce: Open up the default MSYS command line (MinTTY) on Windows 10. Download the ggez/ggez repository, navigate to that repository's root directory, and run cargo run --example 01_super_simple. Wait for the progress bar to reach at least 100/257, and then interrupt compilation by pressing Ctrl+C.
Expected behaviour: Control should return to the command line near-instantaneously. This is the observed behaviour when using cmd.exe.
Observed behaviour: The command line becomes unresponsive for tens of seconds. Throughout this time, Task Manager reports that several instances of rustc.exe are consuming a combined 100% CPU usage. Disk usage and memory usage remain at baseline.

Handling ctrl-c is pretty tricky on Windows. It doesn't work the same way that it does on unix. When you ctrl-c a program at the command line it typically does the equivalent of kill -9 $pid on Unix, meaning the program has no way to clean up after itself. On Windows Cargo uses "job objects" so when Cargo is erroneously torn down it takes down the whole process tree with it. If your cargo.exe in PATH is actually rustup, though, then we also rely on rustup to do that.
Can you try setting CARGO_LOG=cargo::util::job to see if any errors are happening when setting up the job object? There may also be an error on rustup's side but I don'tk now how to debug that.

FWIW, I was unable to reproduce this. I tried a few different environments and terminals and versions of MinTTY.
Is it specific to building that example (01_super_simple)?  Does Ctrl-C need to be pressed while building a specific package? I'm wondering if it is running a build script that is doing something unusual.
Was Rust installed with rustup?

@alexcrichton: If I run CARGO_LOG=cargo::util::job cargo run --example 01_super_simple, and interrupt it with Ctrl+C as described above, nothing unusual is printed to the console.
@ehuss: The problem isn't specific to ggez; it seems to occur for any large, long-running compilation, but only when it has made significant progress. For example, if I start building 01_super_simple and then interrupt it when the progress bar is only at 10/257, the problem doesn't occur.
My environment is an up-to-date MSYS2 installation on Windows 10. I launch the console by selecting the Start Menu item named "MSYS2 MinGW 64-bit". This is a shortcut for the command-line script:
C:\msys64\msys2_shell.cmd -mingw64 -use-full-path

Can you try executing Cargo directly and see if that makes a difference? For example not executing the rustup wrapper named cargo.exe but instead the cargo.exe directly

Interestingly, whereis cargo gives me /c/users/myname/.cargo/bin/cargo.exe, so it looks like I'm might already be bypassing the rustup wrapper?
However, the bug is unchanged when I run any of these commands following a cargo clean:
cargo build --example 01_super_simple
rustup run nightly cargo build --example 01_super_simple
/c/users/myname/.rustup/toolchains/nightly-x86_64-pc-windows-msvc/bin/cargo.exe build --example 01_super_simple

Alas :(
Unfortunately there's not much we can do to help fix this without being able to debug it more, but it's somewhat difficult to know what to debug at this point...


ShowMyCode on object level

The global decision, if to hide AL code or not is not the best choice under all circumstances.
I would love to hide code within certain codeunits only.
Therefore I would like to introduce the idea to set "ShowMyCode" on a per object base, if possible.

I have never really liked the ability to hide code, neither in C/Side or here with AL.
But if we have the option to put it on a single object, instead of the whole extension, then it would not be quite as bad. It would "limit" the damage. So you got my vote for this suggestion!

Well, you need to store your private key for the monetizing access somewhere.
You usually do that inside the code.

I am not so much scared from customers, it is more other NAV partners stealing code and ideas.

Thomas, I wouldn't store such a key hard coded. Instead I would use a web service to retrieve it, upon setting it up. Or import it as data upon installation (if the table is shared by all companies).

I would like to vote this up - it seems to have gone dormant for +two years.
As an example - consider the case of an extension that is primarily designed to support derivative extensions.
If you have a valuable algorithm that you would like to include but not necessarily share in AL code then you are in a tough spot.
You can hide your code, thus reducing the appeal of your solution.
Or you can split out the algorithm into a dependent extension, thus introducing complexity for deployment and maintenance.
A per object decorator that limits visibility of the code in the object would be useful.
I would also request that the feature apply to both runtime packages and standard app packages (if that is possible).
The ability to distribute a standard app package that supports XML documentation, object browsing, etc., but also protects selected IP would be appealing. (e.g., think of the "horizontal toolkit" ecosystem).

Let's channel this feedback on the BCIdea https://experience.dynamics.com/ideas/idea/?ideaid=f7e87a45-fe5a-eb11-8fed-0003ff68c3d2 . It has gathered a large number of votes and we are considering it for the next Wave.

Hi! Please go ahead and post this to our Ideas forum at https://aka.ms/BusinessCentralideas, or vote up the idea if its already there. We're constantly monitoring top Ideas and will consider them for a future release.


cleanup js local eval references

🚀 Deployed on https://6504ed77ba445e430d3ccbd0--cozy-fox-0defec.netlify.app


3.0.0 conflicts with Spring Boot 2.1.1 Release

When adopting 3.0.0 release on a SPRING BOOT 2.1.1 RELEASE based project, it failed during web container start up with an error of - " org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class org.hibernate.validator.internal.engine.valueextraction.ValueExtractorManager
". It seemed like a conflict with hibernate validator while I have no idea why JImageHash would have any side affect of hibernate validator.
When I switched JImageHash version from 3.0.0 to 2.1.1, it all passed. Is it because JImageHash version aligns with SPRING BOOT version?
Thanks.

The library contains a lot of dependencies that are needed for the examples, but not if you only want to use the algorithms.
Try excluding these:
    <exclusions>
        <exclusion>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-graphics</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-swing</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-web</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-fxml</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.json</groupId>
            <artifactId>json</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
        </exclusion>
    </exclusions>


Try out embedding Kotlin 2.0.0-Beta4

This is a discovery PR to validate 2.0.0-Beta4 does not regress.
Not to be merged at this stage.

Superseded by https://github.com/gradle/gradle/pull/28795


add more page user interface widgets

#208 contained a number of things, including my first attempt at this - I'm going to close it, as the UI changes I made didn't work out as well as i'd hoped.
I experience usage impedances when I open more pages than can be seen on my, or my reader's screen.

there is often no visual indication that there are more pages on the left or right - this can be solved either by having a small portion of the offscreen page peeking, a bottom scrollbar, indicating how many pages are open, or icons, or...
when opening a story (a collection of pages) to read/edit/refactor or to send to someone else, I start to want to close some, and re-arrange others into an order that better tells the story. This could be solved with a combination of close and drag&drop pages

These recently closed suggestions #214, #215 and #216 came at the end of wiki's first year when I was showing wiki around my sponsor's headquarters. The problems they site and in some cases solve have remained as other features grow up around them. Looks mattered, especially clean and consistent looks. That made me overly cautious which I regret.