Node Query Raw parts

Problem statement
This is a proposal to add an escape hatch syntax into the sql allowed in DJ node queries. There are many instances where queries could contain some database specific syntax that we either could parse under a very specific dialect or may never even be able to parse.
A trivial example would be a function we don't support but we can parse the query
SELECT myspecialfunc(x) FROM a

when we look to validate this we will not know the function to be able to infer the type. It would be possible to say something like Unknown function `...` specify the type of the expression `myspecialfunc(x)
and then have the means to create nodes (API route) accept specified types. There are more advanced cases though where that would not be possible.
Take for example
> SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2));
 {1:"ax",2:"by"}

from https://spark.apache.org/docs/latest/api/sql/index.html#map_zip_with
we have a combination of some specific functions as well as a lamba. I do not think any sqloxide dialects support this and so we would have to either expand our backends to maybe sqlglot which would introduce multiple extra layers of complexity from selecting default backend to the question of fallbacks... Besides, engines have not shown themselves to be too kind to following standards imo. Also, in the current state if new syntax comes out we automatically do not support it until we can completely parse it.
Proposal
I think what could work is a dedicated syntax that just facilitates marking and escaping from parts of a query that can be typed expressions - essentially anywhere you could use a column or more complicated expression.
Consider an Expression in sql which can ofc be many things (a single Column, Function, Case, a subquery, a primitive - int, str, bool, ...,) and will have a type that when creating a new node from sql we will need to know for validation. The Expression may contain other Expressions but most importantly it may contain Column(s). Columns are relevant because we would like to validate as part of a node creation that columns actually exist so that when the node enters the dag it doesn't invalidate potential parts of the dag that have it as a dependency (child DJ nodes).
To allow raw sql (unparsed) that still allows us type inference and column validation I propose something such as wrapping such expressions in a clear marking such as ${Expression : TYPE} this signifies that this is a part of the sql that we need not worry about only that when we build the query we will dump Expression out and we can assume the type is TYPE. Further, should any sub-expressions exist within Expression that are parseable, they may be escaped with {SubExpression}
This is some pseudo metasyntax I think illustrates this further:
${Expression({Expression})} | ${Expression}
Examples
simple, parsable, unknown function
SELECT ${myspecialfunc({x})} FROM a

special function, constructs, lambda
> SELECT ${map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2)) : DICT};
 {1:"ax",2:"by"}

special function, expression embeds column from a TRANSFORM
SELECT ${apply_func({tfm_node.x}, (x) -> x+1) : INT} FROM tfm_node;

Implementation
I have already created a prototype of this method and the AST is designed with the compile, validate, build in mind. Namely, to facilitate this, I create a new AST Node Raw:
Raw(
    name: str, # the name of the raw element automatically created
    expr_string: str, # the raw expression
    type: ColumnType, # the type defined in the raw expression syntax (: TYPE)
    expressions: List[ast.Expression], # a list of all sub-expressions escaped within the raw expression
    expression_hashes: List[int], # hashes used to recreate the original expression after `expressions` have passed through steps like build
)

parsing the query for this is fairly trivial and can be done with some regex.
Here is a small demo:
from rich import print
from dj.sql.parsing.backends.sqloxide import parse

query = """
SELECT ${apply_func({tfm_node.x}, (x) -> x+1) : INT} FROM tfm_node
"""

print(parse(query))
print("#"*25)
print(str(parse(query)))

OUTPUT:
Query(
    select=Select(
        from_=From(tables=[Table(name=Name(name='tfm_node', quote_style=''), namespace=None)], joins=[]),
        group_by=[],
        having=None,
        projection=[
            Raw(
                name='RAW_N3796958338935914076',
                expr_string='apply_func({EXPR_N7168611668097518469}, (x) -> x+1)',
                type=<ColumnType.INT: 'INT'>,
                expressions=[
                    Column(
                        name=Name(name='x', quote_style=''),
                        namespace=Namespace(names=[Name(name='tfm_node', quote_style='')])
                    )
                ],
                expression_hashes=['EXPR_N7168611668097518469']
            )
        ],
        where=None,
        limit=None,
        distinct=False,
        order_by=[]
    ),
    ctes=[]
)
#########################

#can perfectly recreate the original

SELECT  apply_func(tfm_node.x, (x) -> x+1)
 FROM tfm_node

if the transform node were built and, for example could be materialized from a table my_db.my_tbl, since there is still a column we can trace at compile we would get:
SELECT  apply_func(my_db.my_tbl.x, (x) -> x+1)
 FROM my_db.my_tbl

Take for example
> SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2));
 {1:"ax",2:"by"}

from https://spark.apache.org/docs/latest/api/sql/index.html#map_zip_with
we have a combination of some specific functions as well as a lamba. I do not think any sqloxide dialects support this and so we would have to either expand our backends to maybe sqlglot which would introduce multiple extra layers of complexity from selecting default backend to the question of fallbacks... Besides, engines have not shown themselves to be too kind to following standards imo.

I'm a bit confused here. We shouldn't be parsing DB-specific SQL like this. We should only have to worry about parsing a single SQL dialect, "DJQL", which should be as close as possible to ANSI SQL, and is used to define nodes in a DB-agnostic way. Once we have parsed the SQL we can then translate it to DB-specific SQL.
In your example, we would define the node something like this:
SELECT map_zip_with(
  [map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2)],
  concat,
  [v1, v2]
);

Then in sql/functions.py we'd have (pseudocode):
class MapZipWith(Function):
    @staticmethod
    def get_sqla_function(..., *, dialect: str) -> SqlaFunction:
        if dialect == "spark":
            return sqlalchemy.raw("map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2))")
        raise Exception(f"Function not supported in dialect {dialect}")

I don't remember if the current executor takes into consideration which dialects are supported for the functions when evaluating the best database to run a query (or the possible databases), but it should.
This way, we define nodes independently of the databases. It's the Java ("Write once, run anywhere") of SQL: people write the SQL query once, and they can run it in any database (assuming it supports all the functions and has all the tables).
With this approach, we can implement the map_zip_with function in different dialects. If you look at the defintion of DateTrunc (https://github.com/DataJunction/dj/blob/59c4e3669341feec96e2f549acb2706d2b18e1c2/dj/sql/functions.py#L166), we implement a single DJ function date_trunc than can be computed in many different databases, even ones that don't support the function natively.

@betodealmeida

In your example, we would define the node something like this:
SELECT map_zip_with(
  [map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2)],
  concat,
  [v1, v2]
);

My assumption about DJ overall is that there may be orgs attempting adoption with lots of users who either:

are too pragmatic (lazy? 🥇) to convert a potentially large number of queries to an agnostic form
do not have the knowledge (maybe I'm a poor example but I would fall into this bucket) to convert to an agnostic form

Then in sql/functions.py we'd have (pseudocode):
class MapZipWith(Function):
    @staticmethod
    def get_sqla_function(..., *, dialect: str) -> SqlaFunction:
        if dialect == "spark":
            return sqlalchemy.raw("map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2))")
        raise Exception(f"Function not supported in dialect {dialect}")

My assumption about functions was that there are so many we would have to implement I don't feel like it would be practical would it? I thought we would do our best to cover the most common and still the user has to learn the DJ name of the one they are trying to use. So, if someone brings a query that uses a moderately obscure function we would have to implement it for their query to work or they would have to seek alternate avenues to write their query in an acceptable way?

Ok, I think this issue below is related to this use case, but correct me if I'm wrong.
Let's say that in my org someone has created a custom Spark function that they've deployed onto a given cluster. In that case, we wouldn't want to (nor would we be able to keep up with those changes) implement a SqlaFunction for that function. If we could instead have an escape hatch for that scenario, it would allow them to write that SQL without having to implement support for that function in DJ.

Whoa, what a conversation... I wish I could give you both a winner trophy! But is there a way we can have both?
I get that requiring a user to provide a DJ SQL so we can parse everything and feel awesome is the best option, but is it always possible? We either parse the thing a user will give us or we fail and tell the user what we don't get. From there the user may say:

I don't care about this system, cause it doesn't parse my little query. Se ya later.
Or sees the little button below, that says "Need help parsing the query?"

If they press that button we could present them with some options say:
Hey there, we don't know what you meant by `my_little_udf(...)` but let's see if we can work it out:
(1) Pick this option if you are brave enough to give us a specification of your UDF,
    this would be a fix for everyone later on, and your name would be on that contribution to our Metrics Platform.

(2) Pick this option if you'd like us to wrap the above UDFs in safety-on envelopes, so we can carry it inside the system w/o
    knowing what it does. You can also point us to the referenced columns in order to tell us about your dependencies.
    This option is less time-consuming and you will still be able to use DJ dependency graph. Just don't expect us to get
    too much praise for this.
(3) This is the lazy-option, just click go and we will wrap your query in a flashing-red envelope. This is a way to learn of some
   other DJ features, but know that using this option will not give you all the benefits of this platform. Once you are ready
   to unlock the rest of the DJ awards, come back and help us resolve the parsing issues.

So... can we have it both ways? Or maybe even 3 ways?

I'm also on the side of including this as an escape hatch. @betodealmeida I agree with you that ideally we want all functions implemented and parseable by sqloxide and represented accurately in a DJ AST. For those reasons, I don't think this escape hatch should be highlighted front and center (maybe we don't include it anywhere in the docs).
All that being said, I think this escape hatch is an enormous aide when helping many many people migrate onto the system. When encountering some unrecognized syntax like those pesky scala-like arrow symbols in spark functions, it's nice that we can get that user rolling and then open an issue to get that function implemented. Huge plus that @CircArgs proposal allows for specifying the type of the output which keeps the column inference and node dependency logic all functioning properly. I think this explicit type signaling was something that was floated in a conversation about a very related feature, supporting custom user-defined functions.

I am concerned that if we let users specify types and columns in a DJ SQL then they will "never" go the full way and add that to the system. Why would they? I think finding and instrumenting these "escape hatches" will be on the maintainers but I think I am willing to take that chance if that helps with on-boarding.

I think there's two different topics here:

Allowing users to use DB-specific functions that DJ (still) doesn't know about.
Allowing users to use non-ANSI SQL (lambdas, eg).

For (1), @agorajek had suggested allowing functions to be passed by name when we don't recognize them. SQLAlchemy has support for arbitrary function names (it also passes them by name), but as we know the type inferer is then unable to know the schema of the node.
Ideally, though, we want to implement all the most common functions, because it brings so much more potential to the system. But you might ask:

My assumption about functions was that there are so many we would have to implement I don't feel like it would be practical would it?

Yes, and that's why DJ is the missing piece in the modern data stack — it's hard work. 🙂 It takes a lot foundational work to get it right. But I think that the current approach for defining function is almost as simple as it can be. We all could spend the day doing a marathon and implement the most common 100 functions.
That being said, I understand there are cases where people are in a hurry and want to add a function that is DB-specific, without having it implemented in DJ beforehand. When that happen, I would strongly want for us to keep using ANSI SQL, since it keeps the syntax simple and familiar, and gives us the flexibility of switching parsers, or using a JS parser in the web UI.
There are different ways we could implement this in ANSI SQL. Here we pass the function call as a string, the dialects it works on, and the return type:
SELECT DJ_UDF(
    "my_custom_function_in_spark(1, 2, col)",
    ["spark"],
    "ARRAY[INT]",
);

Or we could use comments for hints, and use the approach @agorajek suggested:
SELECT my_custom_function_in_spark(1, 2, col);  -- returns: ARRAY[INT]

My main concern is the same as @agorajek: as @CircArgs said, people are lazy (and they should be!), if they can write the SQL they know they will write the SQL they know. And the power of DJ comes from having transformations define in ways that can be computed in different engines.

@betodealmeida thanks for separating the issues. I agree we have two here, even though for a user it may look like one.

Allowing users to use DB-specific functions that DJ (still) doesn't know about.

Your comment and the SELECT DJ_UDF(... example gave me an idea. What if we allow a new CREATE UDF syntax to do exactly that, creating new UDFs that DJ does not know about yet, but right there from the DJ SQL prompt? I think this could work just like with any other "database". And in this case we could use @CircArgs's mechanism to add them to the parser on the fly (I think).

Allowing users to use non-ANSI SQL (lambdas, eg).

This is the one that worries me more. On one side we cat just throw an error, and on the other we can say "No idea what your SQL does, but if you give us the output schema (cols + types) we can try to hold that for you as a node.". I think that finding any middle ground here will be tough and will feel icky.

@betodealmeida

SELECT DJ_UDF(
    "my_custom_function_in_spark(1, 2, col)",
    ["spark"],
    "ARRAY[INT]",
);

This looks like a nice compromise. It's basically the essence of my proposal but  introduces dialect. I think there would need to be a reference in this syntax that suggests there is a column used. + to your point it is ANSI

@CircArgs if you allow the user to create the the UDF first with (no content) just the name and input and output types. Then you won't need to make any special column reference syntax, no?

Why don't you just use SQLGlot to parse the SQL? I'm curious what the deficiencies of SQLGlot are that prevent you from going that route?

@agorajek
I was thinking if we had
SELECT dbt.source.jaffle_shop.customers.id,
    first_name,
    last_name
FROM dbt.source.jaffle_shop.customers

after we compile the ast we know where each column came from (if compile succeeds) and so when we build we can correctly reference all columns to the appropriate source whether that be a subquery or a materialized table. In doing so, assuming dbt.source.jaffle_shop.customers is materialized in jaffle_shop.customers in the database, we would potentially build to (assuming that db was chosen)
SELECT jaffle_shop.customers.id,
    jaffle_shop.customers.first_name,
    jaffle_shop.customers.last_name
FROM jaffle_shop.customers

and to do that we need to know how that column is referenced. Assuming the column isn't namespaced in any way we would only really need to know (for this build resolution), to your point and even less (idt we need to know input types), the raw part and the output type.
SELECT DJ_UDF(
    "my_custom_function_in_spark(1, 2, col)", -- raw part
    ["spark"],
    "ARRAY[INT]", -- output type
);

perhaps an alternate signature would be
DJ_UDF(
name: str,
dialects: [str],
return_type: ColumnType,
*args: str|Column
)

where *args would be whatever goes into the function like 1, 2, col above and we can parse it and we get our column as we would otherwise. If you have a lambda or something else you just pass it as a string and we don't care because we won't touch input types.

Why don't you just use SQLGlot to parse the SQL? I'm curious what the deficiencies of SQLGlot are that prevent you from going that route?

We could use SQLGlot but I don't think it solves the problem being discussed in this thread. After parsing the SQL, we convert the AST to a DJ specific AST that carries metadata about the DAG of DJ nodes (sources, transforms, metrics, and dimensions). This includes type inference on the projection based on column metadata from upstream DJ nodes. Even if a custom function is parseable, we need to know two main things-the type of the return value and which columns from upstream nodes are used within the custom function.

SELECT DJ_UDF(
    "my_custom_function_in_spark(1, 2, col)",
    ["spark"],
    "ARRAY[INT]",
);

I also like this syntax.

perhaps an alternate signature would be
DJ_UDF(
name: str,
dialects: [str],
return_type: ColumnType,
*args: str|Column
)

I think what @betodealmeida was getting at is that with his proposed syntax it would still be ANSI SQL compliant. I don't know how interested people will be in the broader vision of having an ANSI SQL only system and the benefits that come with that. I suspect a few people will simply be bothered that they have to rewrite parts of their query (which they'd have to do with either proposed syntax). That being said I say we go with the ANSI SQL proposal that @betodealmeida offered, as @CircArgs said this is a nice compromise.

@CircArgs if you allow the user to create the the UDF first with (no content) just the name and input and output types. Then you won't need to make any special column reference syntax, no?

@agorajek I really like this idea. I'm not trying to take over the world here but...should we maybe allow both? We can include an API for registering custom UDFs in which case after it's registered it's available to use without the additional arguments as you mentioned.
SELECT DJ_UDF(
    "my_pre_registered_custom_function_in_spark(1, 2, col)",
 );

However I think the best escape hatch to have in the moment when someone is migrating an existing query would be to define a temporary UDF within the query as in @CircArgs and @betodealmeida's proposals. If we really want to differentiate these, we could call it DJ_TEMP_UDF.
SELECT DJ_TEMP_UDF(
    "my_custom_function_in_spark(1, 2, col)",
    ["spark"],
    "ARRAY[INT]",
);

gotcha, thanks for the answer. sqlglot has type inference built in where you can override the return types of any function.
https://github.com/tobymao/sqlglot/blob/main/sqlglot/optimizer/annotate_types.py#L35

(3) This is the lazy-option, just click go and we will wrap your query in a flashing-red envelope.
This is a way to learn of some other DJ features, but know that using this option will not give you
all the benefits of this platform. Once you are ready to unlock the rest of the DJ awards, come back
and help us resolve the parsing issues.

@agorajek this is an interesting one. I suspect there will be many circumstances where people are lazy (or lack the time to understand how to define a DJ_UDF), in which case they'll want to fall back to letting the system guess at the input and output types of the function. However, while guessing input is reasonable (we can just look at the types of the columns passed in, guessing output is harder. How would this third option you've described actually work? Would this essentially be a node that's INVALID until this UDF's definition is resolved?

@samredai can you elaborate how my adapted version of @betodealmeida would not meet the same standard?

@shangyian yes, that 3rd option is what I consider an unparsed SQL. I think a bar for specifying the output schema in some cases may be lower than having someone fix all the SQL issues. We could still run some side functionalities of DJ (say materialization) if the user would tell us few things about their SQL rather than spend time fixing all parsing bugs. I don't consider it a flagship feature but rather an easy-onboarding one.

@CircArgs and @samredai , maybe I missing something here, but I just don't see how adjusting a UDF signature inside a query makes things easier or cleaner fot the user than adding a CRATE UDF statement above that query and keeping the query in a "clean" state.

@samredai can you elaborate how my adapted version of @betodealmeida would not meet the same standard?

@CircArgs I just meant that it wouldn't be parseable. The following is parseable by both sqloxide and sqlglot.
SELECT DJ_UDF(
    'my_custom_function_in_spark(1, 2, col)',
    ['spark'],
    'ARRAY[INT]'
) FROM tbl

@CircArgs and @samredai , maybe I missing something here, but I just don't see how adjusting a UDF signature inside a query makes things easier or cleaner for the user than adding a CRATE UDF statement above that query and keeping the query in a "clean" state.

@agorajek I see, I didn't know you meant in the same query definition. Hmm, that also would not parse so we would have to do some pre-processing of the query, similar to @CircArgs's original proposal. Originally I thought you meant someone could separately globally register a function with a signature and then within any query they would only need to put the UDF call (and we lookup the engines and output types in the metadata store of global UDFs). That's something I also think would be a cool thing to eventually add so that users can register the UDF once through an API call and then it's understood everywhere.

Thanks @samredai. I think we agree now that something like the following is parseable
SELECT DJ_UDF('my_custom_function_in_spark', list('spark', 'trino'), 'LIST[INT]', 1, 2, col) FROM table

or
SELECT DJ_UDF('my_custom_function_in_spark', 'spark, trino', 'LIST[INT]', 1, 2, col) FROM table

@CircArgs agreed! Of the two options I like the second one. I know list('spark', 'trino') looks more legit but I like the simplicity of the second option where it's just a single function call with primitive arguments.

That's something I also think would be a cool thing to eventually add so that users can register the UDF once through an API call and then it's understood everywhere.

@samredai yes, that is exactly what I meant. Creating (using CREATE OR REPLACE) a UDF in the same session should mean that it is also available in the system.
@CircArgs are you expecting the users to use the above custom UDF syntax repeatably across their whole query if it happens that such UDF is called multiple times? Please tell me that I am missing something and they only need to do it once, even if they use that UDF many times in the query.

@agorajek

Please tell me that I am missing something and they only need to do it once, even if they use that UDF many times in the query.

As it stands I don't have any better ideas than @betodealmeida suggestion to use something like this syntax to keep it standard sql and have it appear all over. I agree though, it's not nice. I defer to what others think would be the best route would be I think this has been a pretty productive thread for me.
To your earlier point:

Hey there, we don't know what you meant by my_little_udf(...) but let's see if we can work it out:
...

we don't get column, line or any other context beyond the simple error message (e.g. expected ) found...) to what caused a parsing failure with sqloxide so I don't know how we would begin to say "we found this weird thing here what do you expect me to do with it?"

I think we've landed on a final two options:
Option 1
SELECT DJ_UDF('my_custom_function', 'spark, trino', 'LIST[INT]', 1, 2, col) FROM foo

This option is definitely parseable by sqloxide so we can start by immediately parsing the string as given. And then when converting the sqloxide input over to the DJ AST, we would look out for DJ_UDF functions that appear like this in the parse tree:
{
    "projection": [
        {
            "UnnamedExpr": {
                "Function": {
                    "name": [{"value": "DJ_UDF", "quote_style": None}],
                    "args": [
                        {
                            "Unnamed": {
                                "Expr": {
                                    "Value": {
                                        "SingleQuotedString": "my_custom_function"
                                    }
                                }
                            }
                        },
                        {
                            "Unnamed": {
                                "Expr": {
                                    "Value": {"SingleQuotedString": "spark, trino"}
                                }
                            }
                        },
                        {
                            "Unnamed": {
                                "Expr": {"Value": {"SingleQuotedString": "LIST[INT]"}}
                            }
                        },
                        {"Unnamed": {"Expr": {"Value": {"Number": ("1", False)}}}},
                        {"Unnamed": {"Expr": {"Value": {"Number": ("2", False)}}}},
                        {
                            "Unnamed": {
                                "Expr": {
                                    "Identifier": {"value": "col", "quote_style": None}
                                }
                            }
                        },
                    ],
                    "over": None,
                    "distinct": False,
                }
            }
        }
    ]
}

The downside of this is if the function is used 20 times in the same SQL query, the user would have to define it as DJ_UDF('my_custom_function', 'spark, trino', 'LIST[INT]', 1, 2, col) 20 times.
Option 2
CREATE FUNCTION my_custom_function(int, int, column) ENGINES 'spark,trino' RETURNS 'LIST[INT]';
SELECT my_custom_function(1, 2, col) FROM foo

This option is nice in that the user only has to define the engines and return type once at the top, and then if they use the function 20 times in their code, they can simply call the function with the args, my_custom_function(1, 2, col). The downside here is that this entire query string is not directly parseable by SQL so we would have to do some pre-processing to interpret the CREATE FUNCTION call and then strip it out before sending the rest to sqloxide. I don't know how much this overhead is worth it to prevent having to specify the engines and return type every time.

However I want to bring it all the way back to the original thing we're looking for here which is an escape hatch to use when someone brings an unknown/un-parseable function, worst case being a custom UDF where we do not know the return types. Take the example in @CircArgs description for this issue.
SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2));
 {1:"ax",2:"by"}

Even if we provide a mechanism for someone to rewrite this function, we have to go ahead an implement the function that will take DJ_UDF("map_zip_with", "spark", "MAP", "map(1, 'a', 2, 'b')", "map(1, 'x', 2, 'y')", "(k, v1, v2)", "concat(v1, v2)") and produce the original query (including the ->). It doesn't feel like an escape hatch if we have to open, merge, and deploy a PR to utilize it.
I keep finding myself coming back to this proposal in the description of this issue:

special function, expression embeds column from a TRANSFORM
SELECT ${apply_func({tfm_node.x}, (x) -> x+1) : INT} FROM tfm_node;

It requires no PRs to use, keeps both node/column dependencies working properly, and most important of all, it produces the original string exactly as defined. If we can squeeze an array of supported engines into this, I think I'd be completely sold, keeping in mind that this is not a syntax we'd advertise or recommend be used. As we march towards our long term goal of implementing engine specific functions (and finding a way to support user-defined functions that include the lambda expression arrow ->), in success we will never (or very rarely) have to use this feature and can probably remove it.

@samredai For both option 1 and option 2 in your list above, my understanding is that no PRs are required. The UDF can get defined in that session and then be made available for use immediately after. I think option 2 would generally be what users prefer, since you can declare it once and then reuse. Also, the syntax for option 2 is significantly closer to what most users are familiar with when declaring custom functions. I think we should aim to support option 2 in the long run, but it can be punted on for now.
Ultimately I think users should be able to just specify the function name, the input args and types, the output args and types, and have that be enough for the purposes of parsing. They can also implement the function if needed, but if the actual implementation already defined in a custom cluster for an engine, that's unnecessary.

@shangyian with Option 2 are you imagining we just include those pesky lamba functions by treating the entire element as a string? Using the example from the spark docs:
SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2))

With Option 2 maybe we could write something like this?
CREATE FUNCTION map_zip_with(str, str, str) ENGINES 'spark' RETURNS 'MAP';
SELECT map_zip_with("map(1, 'a', 2, 'b')", "map(1, 'x', 2, 'y')", "(k, v1, v2) -> concat(v1, v2)")

Another one from the spark docs is:
SELECT aggregate(some_array_of_arrays_column, 0, (acc, x) -> acc + x, acc -> acc * 10)

which Option 2 would mean
CREATE FUNCTION aggregate(str, str, str, str) ENGINES 'spark' RETURNS 'INT';
SELECT aggregate("an_array_of_arrays_column[0]", "0", "(acc, x) -> acc + x", "acc -> acc * 10")

Maybe that's the solution but it still breaks column/node dependencies. an_array_of_arrays_column above actually comes from the upstream node which we'll need to detect and add that relationship in the metadata. Should we go with Option 2 but also include the curly brace column identifying syntax?
CREATE FUNCTION aggregate(str, str, str, str) ENGINES 'spark' RETURNS 'INT';
SELECT aggregate("{an_array_of_arrays_column}[0]", "0", "(acc, x) -> acc + x", "acc -> acc * 10")

Had a short discussion with @samredai this morning. We agreed that this topic of trying to do something nice for the users, when we really can't parse their UDFs should not be derailing us from the short term goals. But we should have come up with some quick temporary solution that would give us good stage for addressing this issue in the future and here is what we got.

If sqloxide can parse the UDF and DJ does not know about it, then we can keep doing the pass-through.

If sqloxide breaks on parsing a UDF then we can wrap the whole UDF call in a single RAW() function, specify the return type (either as 2nd argument, or using Postgres casting syntax), and optionally reference the columns. This way we are not putting too much burden on the user (or ourselves) while preserving the original code so we can address it and fix it later. See below:

Breaking input: ... SELECT aggregate(array(1, 2, col3), 0, (acc, x) -> acc + x, acc -> acc * 10) FROM node_one ...
Working input:

2nd arg for return type:
... SELECT RAW("aggregate(array(1, 2, {col3}), 0, (acc, x) -> acc + x, acc -> acc * 10)", "INT") FROM node_one ...
Postgres syntax for return type:
... SELECT RAW("aggregate(array(1, 2, {col3}), 0, (acc, x) -> acc + x, acc -> acc * 10)")::INT FROM node_one ...

There is still the open question of what do we do if the outer SQL parsing breaks, but that's a separate issue.

Big +1 here. I think this is an important area of the source code that I wouldn't want to rush to a solution on (or add a ton of code for an intermediate solution that has to be unwinded later). I think the RAW as a very simple blunt escape hatch that gives us some breathing room to accumulate a good number of examples of un-parseable non-ANSI bits so that we can come up with a solid long term solution.


Update nav.yml

removed isOrdered from Main Concepts and Hooks in side navigation bar for uniformity in the side bar
ordering is present:

ordering isn't present:

Hello @ShouryaSrivastava01,
with how the nav is done in the react.dev now, I don't see this PR any longer required.
Let me close this as outdated, but you can create new PRs for other open issues if you would still like to contribute.


204 status code not handled properly

Exception raises when route returns response with 204 No Content status.

INFO:     <IP_ADDRESS>:58657 - "DELETE / HTTP/1.1" 204 No Content
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlite/app.py", line 149, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlite/asgi.py", line 70, in __call__
    await route.handle(scope=scope, receive=receive, send=send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlite/routes.py", line 161, in handle
    await response(scope, receive, send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlette/responses.py", line 167, in __call__
    await send({"type": "http.response.body", "body": self.body})
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 462, in send
    output = self.conn.send(event)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/h11/_connection.py", line 510, in send
    data_list = self.send_with_data_passthrough(event)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/h11/_connection.py", line 543, in send_with_data_passthrough
    writer(event, data_list.append)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/h11/_writers.py", line 65, in __call__
    self.send_data(event.data, write)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/h11/_writers.py", line 91, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlite/app.py", line 151, in __call__
    await self.handle_exception(scope=scope, receive=receive, send=send, exc=e)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlite/app.py", line 165, in handle_exception
    await response(scope=scope, receive=receive, send=send)
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/starlette/responses.py", line 160, in __call__
    await send(
  File "/private/tmp/starlite-fetch-error/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 452, in send
    raise RuntimeError(msg % message_type)
RuntimeError: Expected ASGI message 'http.response.body', but got 'http.response.start'.

This example fails:
from starlite import CORSConfig, Starlite, delete

@delete()
async def route() -> None:
    return None

app = Starlite(route_handlers=[route], debug=True)

This one does fail too. Additionally, Starlite resolved 204, but there is some content (200):
from starlite import CORSConfig, Starlite, delete

@delete()
async def route() -> str:
    return "Hello!"

app = Starlite(route_handlers=[route], debug=True)

Starlette works fine, so the problem is definitely with Starlite:
from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
from starlette.routing import Route

async def route(request):
    return PlainTextResponse("", status_code=204)

app = Starlette(
    debug=True,
    routes=[Route("/", route, methods=["DELETE"])],
)

Starlite resolved 204, but there is some content (200):

There's not really any resolving afaik, if you don't specify different, the status of the response is dynamically set in HTTPRouteHandler.__init__():
        if status_code:
            self.status_code = status_code
        elif isinstance(self.http_method, list):
            self.status_code = HTTP_200_OK
        elif self.http_method == HttpMethod.POST:
            self.status_code = HTTP_201_CREATED
        elif self.http_method == HttpMethod.DELETE:
            self.status_code = HTTP_204_NO_CONTENT

So you'd need @delete(status_code=200).
As far as the empty response goes, I've added a failing test here: https://github.com/starlite-api/starlite/compare/main...issue-154-204-none-response
Inspecting the response in the test shows that response.content is b'null', which just smells to me like the None response from the handler is getting handed straight to orjson.dump(). null is a valid json response, but I think it's pretty safe to assume that if a handler is configured to return a 204 and the handler returns None that the developer wants that to be an empty response.

So you'd need @delete(status_code=200).

Maybe 204 status should be resolved based on SignatureModel? If return type annotation is None, then set 204?

So you'd need @delete(status_code=200).

Maybe 204 status should be resolved based on SignatureModel? If return type annotation is None, then set 204?

No, these are API semantics we can't enforce.

What about defaulting to 200? Let the developer decide if 204 should be returned.

What if we allow the Starlite constructor to receive a mapping of http verbs to response status, and fallback to current behavior if not provided. So you could do:
Starlite(..., default_response_codes={"DELETE": HTTP_204_NO_CONTENT})
We return the deleted object from DELETE as sometimes it can be useful, so I've wrapped starlite.delete locally to override that to be 200 in all cases. Having a way to define those defaults at the application level seems logical.

This is really a different issue though, want to open a new one?

It is not really useful for me. The main issue is handling None with 204

OK well that's already sorted and will be in the next release.


Updating README with Galera guidelines

Signed-off-by: Mateusz Blaszkowski<EMAIL_ADDRESS>

Please change :
High availability is not yet guaranteed for MariaDB, RabbitMQ, Elasticsearch (Searchlight) nor Memcached (Horizon).

@PiotrProkop already done in https://github.com/stackanetes/stackanetes/pull/105


Language/Support for Portuguese

Hey, i'd like to translate the readme file to portuguese, can i? =)

@printf-ana Yes sure you can. Just put the original repository's name(hyperlinked ofc) in your portuguese version repo and tell people to star the our repo for support. Thank you for this wonderful initiative 😄

Hi thanks for your issue.. You can go ahead and create PR. make sure you follow rules mentioned in #79
I m restating the rules :
Translate this repo to multiple languages. Make this knowledge available to all languages

Create a new issue named as Language/Support for xyz. For example, see #78. Some Maintainer will assign you that issue and later you can create a PR
Make sure you are fluent in that language and it is verified by you (Don't simply use google translate)
Create a new branch named as the language. For eg name the branch french if you want to translate the whole repo in french

This issue will be managed by @RajGhugare19 @aditya-shirwatkar
Thanks for making this repo wonderful and useful all around the world!

@printf-ana any update from your side?

Closing this issue due to inactivity. Feel free to reopen for any related contribution.


Can't use most commands with GitHub Enterprise, error message is misleading

I have tried everything below with both 2.2.9 and 2.3.0-pre10
I have been using hub with a GHE instance at work for a very long time and have become very used to git fork, git browse and git pull-request as part of my workflow.  We have switched to a new instance (a centralized instance ran by another team), and those commands no longer work for me.  I can export the GITHUB_HOST variable and git clone <username>/<repo> works fine, so I know it's authenticating properly.  When I use any of the additional hub commands, though, all I get is
Aborted: the origin remote doesn't point to a GitHub repository.

I'm not sure how to troubleshoot further.  I've tried several things, and the repo does point to a GitHub repo because I just cloned it from there.  My basic test was
$ export GITHUB_HOST=<host>
$ git clone <org>/<repo>
$ cd <repo>
$ git fork

The clone succeeded just fine, but the fork command returned the above error.  There should be more information available to determine what the real problem is.

Have you followed the instructions outlined in "GitHub Enterprise" section of the man page? https://hub.github.com/hub.1.html

Regarding the recommended
GITHUB_HOST=my.git.org git clone myproject

I ended up making an alias for that
alias ghe='GITHUB_HOST=my.git.org hub'

and run the clone commands with ghe instead of git because it's too annoying to type out. Not sure if a better solution could be possible at all.

@sinisterstuf: Agreed, would be good to have an integrated solution.  Wondering if hub.host could be used.

To be honest I can't really see a way hub could guess which host you want based only on
git clone myproject

which is why I'm not sure there's going to be a much better solution for this. But as a GHE user I'd be really interested if someone has a solution!

It's not so much about hub guessing from a git clone... that has been working by exporting the environment variable, and I never expected it to work without it.  I too have aliases for git to handle setting the GITHUB_HOST variable (two of them now, since I have to switch between the two).
The variable is needed for "initial" commands like clone or remote add if you're using the abbreviated GHE repo identifiers or new GHE specific commands like fork, but wasn't for commands that work on existing repos (particularly the GitHub specific ones like pull-request or browse) as the appropriate URLs are part of the remote - as long as your remotes are named correctly (github, upstream, etc).
The problem is that I couldn't get hub pull-request and the like to work with the new repo for some reason.
I "fixed" it, though I'm not sure what did it.  I had added a second 'hub.host' entry in my gitconfig for the new GHE instance.  Things worked for the old one, but not the new.
I removed the old one, which made hub pull-request etc. stop working for that host (as it should), but the new one started working.
I then later decided to try adding the old one back... and BOTH worked just fine. I have no idea what was different, other than the order of the entries in my gitconfig.


[CHEC-842] Feature/vertical nav

Added vertical navigation UI component
I was running into issues with this early on as I was trying to use options as props and wasn't able to nail down how to detect the active item correctly. With suggestions from Robbie and John, refactored to use Vue router's router link component which has its native active class
Having trouble trying to figure out alternatively how to programmatically push the routes https://router.vuejs.org/guide/essentials/navigation.html#programmatic-navigation in order to add keyboard events (I have not added keyboard events) <- would love some help on this if this is part of a/cs

Thanks Guy will edit stat!


Test the new preview

Not for merging! Just testing the new workflows merged in #145

The build and preview are working normally!


/usr/local/lib/node_modules/kludgy/node_modules/zombie/lib/eventloop.js:594             throw error;

kludgy "api key redacted" /Users/djbeeker/Desktop/
/usr/local/lib/node_modules/kludgy/node_modules/zombie/lib/eventloop.js:594
throw error;
^
Error: Timeout: did not get to load all resources on this page
at Timeout.timeout (/usr/local/lib/node_modules/kludgy/node_modules/zombie/lib/eventloop.js:601:38)
at ontimeout (timers.js:471:11)
at tryOnTimeout (timers.js:306:5)
at Timer.listOnTimeout (timers.js:266:5)
Error: Must specify panoID with --id option
at Promise (/usr/local/lib/node_modules/kludgy/node_modules/extract-streetview/index.js:42:34)
at parseLocation (/usr/local/lib/node_modules/kludgy/node_modules/extract-streetview/index.js:39:10)
at Object. (/usr/local/lib/node_modules/kludgy/node_modules/extract-streetview/index.js:34:1)
at Module._compile (module.js:413:34)
at Object.devtoolCompileModule [as .js] (/usr/local/lib/node_modules/kludgy/node_modules/devtool/lib/require-hook.js:32:14)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Module.require (module.js:367:17)
at require (internal/module.js:16:19)
at EventEmitter. (/usr/local/lib/node_modules/kludgy/node_modules/devtool/lib/preload.js:78:16)
Error: Format error in file - /Users/djbeeker/Desktop/.jpg
In case this is useful. Probably something to do with my upgrade to High Sierra between now and the last time it was working for me.

Hrmmmm, thanks for the report Beeker! I have not upgraded to the High Sierra yet.
Looks like the panoID is not getting passed down the chain for some reason.
Just tested on Sierra:
$ kludgy "KEY" /Users/mhulse/Desktop/

Did not see error...
I think I will upgrade to High Sierra and see what happens.
Thanks!

Just upgraded to High Sierra... Unfortunately, I could not reproduce. Wondering if it was a network burp?


Support for profiling in PHP 7.0 (XHProf or something else)

There are two options it would seem at the moment. The tideway's solution is more popular ATM, but I haven't tested either of them yet.
https://github.com/tideways/php-profiler-extension
https://github.com/RustJason/xhprof/tree/php7

I'm still happy with XHProf as long as the extension (or a well-supported fork of it) compiles; it's been the de-facto/baseline for a while now, though others like Blackfire are gaining some traction. See related: #526

Also, isn't Tideways for-pay? That's a total non-starter for me... don't want to include any software in Drupal VM that would require a license, for personal and testability reasons.

Yeah, I didn't check that far. I just wanted to get some thoughts down in an issue. Using Blackfire seems like a decent option until the forks for XHProf standardize. I'd be worried that those things are in flux a lot right now.

I've added Blackfire, and it has 'experimental' support for PHP 7.0 (I haven't personally tested except with 5.6 so far). Hopefully one of the XHProf forks will start adding in solid PHP 7.0 support, but I'm not holding my breath right now.
I'm going to close this for now, as I've also updated the docs with information about the three supported ways to profile code built into Drupal VM, and we can maybe open a follow-up or work on the XHProf issue upstream in that geerlingguy.php-xhprof role.


Reconnecting to AP when connection was lost

I experience frequent lost connections when using the the configurator in station mode. I propose this code change to add a reconnect event to mitigate the issue. Works a treat in my own copy running on the Touchdown hardware. Here is my patch.
--- a/ConfigHelper.h
+++ b/ConfigHelper.h
@@ -5,6 +5,15 @@ bool startWifiStation(){
   Serial.printf("[INFO]: Connecting to %s", wificonfig.ssid);
   if (String(WiFi.SSID()) != String(wificonfig.ssid))
   {
+
+       // Attempt reconnect if connection to AP is lost
+      WiFiEventId_t eventID = WiFi.onEvent([](WiFiEvent_t event, WiFiEventInfo_t info){
+        Serial.print("WiFi lost connection. Reason: ");
+        Serial.println(info.disconnected.reason);
+        Serial.println("Reconnecting ...");
+        WiFi.reconnect();
+      }, WiFiEvent_t::SYSTEM_EVENT_STA_DISCONNECTED);
+
       WiFi.mode(WIFI_STA);
       WiFi.begin(wificonfig.ssid, wificonfig.password);
       uint8_t attempts = wificonfig.attempts;

Sorry for the late reply. I'll test this and if this and if it works as it should. I'll implement it! Great for doing this btw, thanks!


README: use Master branch

Seem to work with Master, don't need v2 anymore.

oops, forgot to change the folder name


Incorporate huss

Read in huss, take its mean over the regions/seasons, and incorporate into plots (analogous to mean pr)

huss isn't available for ssp370 for any models on the cloud so we'll have to do it at NCAR


feature: Export as PNG

The Stackoverflow card can be exported in PNG format using the menu at the top-right corner.
Linked Issue: #4 .
Known bug: Image export fails due to CORS policy of imgur profile images.

Hi @DarkFalc0n,
Thank you so much for the submission, but the "Save as PNG" option is failing right now.
You can check the deployment here: https://stackoverflow-card-generator-oasx2p0yo-akshaykhale1992.vercel.app/
This is a very good feature, Please fix the issue.
Gracias!!!


No document in Emoji collection for :simple_smile:

No document in Emoji collection for :simple_smile: unicode is 1f642.

Ahhhh cause not yet supported :(


feat: unable to log in

I purchased year subscription and minted NFT and I downloaded the app it says unable to load purchases ty to login with another device why ?

please email<EMAIL_ADDRESS>from the email you used to sign up


Provide TokenStore and AuthInfoStore in handler

Use jwt token store as TokenStore and DBConn pq implementation as AuthInfoStore.
TODO:

Break down DBConn into multiple interfaces, and only include AuthInfo related API in core package

connect #631

@rickmak removed the middleware


how to make ourself train data in  simulation environment?

thanks
can you give me some infomations?

The script to generate data is provided. Please see our README for more infromation.


Implement options to provide Zoho url

Now it's hardcoded to crm.zoho.com. But in Zoho isset another domains like crm.zoho.eu.
It will be nice to have possibility to setup Zoho domain on instance creation.

I'm extremely confused. Please elaborate. 👍

Sorry. It's wrong project. I add it to wrong repository.
My fault.


[RxSwift] - Implemented mapObjectOptional and mapArrayOptional and updated Demo

Hi...
In this PR, i implemented two new functions, mapObjectOptional and mapArrayOptional for RxSwift version to return nil observables when the mapping fails.
Also, i updated the Demo Project, which now has two tabs, the first one include a example using Moya and the second one is a example using Moya and RxSwift

Hey there,
I'm not sure I understand the need for this as you can always call let result = try? response.mapObject(Repository.self) or response.mapObject(Respository.self).catchErrorJustReturn(nil) for RxSwift counterpart.
Is there a specific use case I'm missing?


Cancelling a Turbolinks visit before is starts fails in IE10/11

Issue
Cancelling a Turbolinks visit before is starts fails in IE10/11 because - I think - IE10/11 .preventDefault doesn't work with "synthetic" events properly.
I have assumed that IE11 at least is still worth supporting at this point as Edge still has a small user base (mind you, I just looked and IE is running at about 5% these days! 😮 )
Expected behaviour:
As described in the readme, if I add an event listener to handle an event of type turbolinks:before-visit then an event.preventDefault() should stop the visit from happening.
Actual behaviour
I've create a minimal example.  If you open that and click the link in a 'modern' browser then the code intercepts the before-visit event, and preventDefault() cancels it.
However, if you open this link in IE10 or 11 (I've not tested any earlier versions) then the event is handled, and the defaultPrevented attribute of the event is set correctly, but Turbolinks fails to prevent the load.
Analysis
I've spent a good few hours with this and from what I can see preventDefault() should, and indeed does, set the defaultPrevented attribute to true. However, on exit from the event handler, defaultPrevented is set back to false!
It's been hard to track down much info on this, but there's a few references where this has come up:

A couple of StackOverflow threads, for example this one
Bizarre workarounds in other people's code, the createEvent method here, for example
This issue in WebComponents (and its fix)

I've tried a few things to try and overcome this - like replicating some of the workarounds mentioned above. But I don't seem to be able to 'override' preventDefault the way other people have and I've been unsuccessful in all my attempts to get this to work.
Note: I'm not a JS Guru - I'd say my JS knowledge is 7/10. I'm not very familiar with coffeescript, but I think I've been able to make sense of it.
The relevant code seems to be:

notifyApplicationBeforeVisitingLocation in controller.coffee which
dispatches the event
applicationAllowsVisitingLocation in controller.coffee which calls notifyApplicationBeforeVisitingLocation and returns not event.defaultPrevented

In my minimal example you'll note that I've used a compiled-but-uncompressed copy of Turbolinks and added some logging.  In IE, my console logging outputs something like:
defaultPrevented: true
in applicationAllowsVisitingLocation. defaultPrevented is false

demonstrating that the defaultPrevented is changed back to false when exiting the before-visit event handler.
I don't know what we do about this. Is there a workaround that other people know of that I don't? Or do we need to create a more explicit mechanism for cancelling the event that works across browsers?  Perhaps setting event.data.cancelVisit = true and then detecting that instead of event.defaultPrevented?  I've provided a non-coffeescript prototype of this that seems to work correctly - the diff of this is:
1719c1719
<           return !event.defaultPrevented;
---
>           return !event.data.cancelVisit;
1735c1735,1736
<               url: location.absoluteURL
---
>               url: location.absoluteURL,
>               cancelVisit: false

I'm sure that wouldn't be hard to translate back into coffeescript. We'd need to make it backwards compatible I guess, using defaultPrevented if the new data attribute wasn't available, and we'd need to update the README too.
Phew. What do you think? Can we fix this?

event.defaultPrevented returned true in IE11 without fix

Send PR for fix that

Sent PR for fix that


make sure that RateLimiter call alert correctly with right duration

Coverage remained the same at 28.6% when pulling 52a036ad95d908889c7aaed823659a4094495c79 on u9520107:bug/rate-alert into 10fa9b196e3d058cd2f126a46f306f75f0c4c320 on ringcentral:master.


Fix sort by absolute value from local importance chart

Fix sort by absolute value from local importance chart

Areas changed

npm packages changed:

[ ] responsibleai/causality
[ ] responsibleai/core-ui
[ ] responsibleai/counterfactuals
[ ] responsibleai/dataset-explorer
[ ] responsibleai/fairness
[x] responsibleai/interpret
[ ] responsibleai/localization
[ ] responsibleai/mlchartlib
[ ] responsibleai/model-assessment

Python packages changed:

[ ] erroranalysis
[ ] raiutils
[ ] raiwidgets
[ ] rai_core_flask
[ ] responsibleai

Tests

[ ] No new tests required.
[ ] New tests for the added feature are part of this PR.
[ ] I validated the changes manually.

Screenshots (if appropriate):
Documentation:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

Codecov Report

Merging #1299 (11b9f30) into main (adbdc74) will increase coverage by 3.40%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1299      +/-   ##
==========================================
+ Coverage   77.02%   80.43%   +3.40%
==========================================
  Files          38       18      -20
  Lines        1541      322    -1219
==========================================
- Hits         1187      259     -928
+ Misses        354       63     -291

Flag
Coverage Δ

unittests
80.43% <ø> (+3.40%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

raiwidgets/raiwidgets/cohort.py

raiwidgets/raiwidgets/explanation_dashboard.py

raiwidgets/raiwidgets/constants.py

...idgets/raiwidgets/responsibleai_dashboard_input.py

raiwidgets/raiwidgets/error_analysis_constants.py

...dgets/raiwidgets/error_analysis_dashboard_input.py

raiwidgets/raiwidgets/__version__.py

raiwidgets/raiwidgets/error_handling.py

raiwidgets/raiwidgets/interfaces.py

...iwidgets/raiwidgets/fairness_metric_calculation.py

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update adbdc74...11b9f30. Read the comment docs.


Improve Travis CI testing

Is your feature request related to a problem? Please describe.
Travis CI currently just does cppcheck and default build on the bundle
Describe the solution you'd like
Parallel test the following:

BUILDTYPE=release OMIT_DEPRECATED=true STRICT_NO_WARNINGS=true
BUILDTYPE=release OMIT_DEPRECATED=false(default) STRICT_NO_WARNINGS=true
BUILDTYPE=debug(default) OMIT_DEPRECATED=true STRICT_NO_WARNINGS=true
BUILDTYPE=debug(default) OMIT_DEPRECATED=false(default) STRICT_NO_WARNINGS=true

And do the following tests:

Add document generation (eventually enforce no warnings)

make doc
make usersguide
make osalguide (dependent on https://github.com/nasa/cFE/pull/446)

Build with ENABLE_UNIT_TESTS=TRUE SIMULATION=native
Run unit tests (eventually enforce coverage)
Execute cFE and pipe output to file
Send reset cmd from cmdutil
Confirm execution output lacks error/warning/fail messages

Describe alternatives you've considered
None
Additional context
Eventually add nightly or triggered local builds for other environments (pc-RTEMS, vxworks on a modern platform)
Eventually add automated build verification (trade CI vs local)
Requester Info
Jacob Hageman - NASA/GSFC

Recommend that this be accomplished by using alternate sample_defs directories (under different names. of course) for the alternate configs.  This should also ideally cover different CFE config options (e.g. TIME server/client) as some TIME options were broken in the past and failed to compile with these options turned on.
This allows the baseline config which is suggested as a starting point for new users to remain simple and straightforward while allowing the CI config(s) to be as complex as it needs to be to cover all these various options.


Test racing conditions in bulkblocks API calls

We should come up with integration tests which will allow to test racing conditions of bulkblocks API. They happen when there are competing (concurrent) calls with bulkblocks API which provides almost identical data (the data only different at block/file level and all other parameters remain the same). In this scenario there are common data such as physics group, dataset access type, processed dataset where we either need to insert or obtain IDs, see https://github.com/dmwm/dbs2go/blob/master/dbs/bulkblocks2.go#L506-L571
I identified that for small tables, like physics group or data tiers, the probability of racing conditions is kind of small, while for larger ones, like processed datasets which contains 149970 entries in ORACLE DB, there is a possibility of racing conditions which may happen if two competing HTTP requests tries to insert/check processed dataset ID.
We need to come up with integration test for this use-case.

@vkuznet While working on this, I discovered that while using sqlite, when I try to insert two bulkblocks simultaneously with goroutines, one succeeds (HTTP 200) while the other fails (HTTP 400). This was because the database was locked. This is a different race condition than what was described. Do you have a suggestion on how to get around this?

Dennis,
we should not experience lock of db if we properly use transactions.
My suggestion is to investigate how sqlite should be initialized that
transactions should be applied to tables rather DB itself.
I think we need to triple check that all statements, rows, are closed
in bulkblocks API. It would be useful to identify on which part of bulkblocks
API the lock occurred, i.e. does the lock appear on specific step, if so
which step it is.
Finally, according to this github issue [1] we may try the following options:
db.SetMaxOpenConns(1)

and journal_mode=WAL which seems to address the issue with database lock.
[1] https://github.com/mattn/go-sqlite3/issues/569
On  0, Dennis Lee @.***> wrote:

@.*** While working on this, I discovered that while using
sqlite, when I try to insert two bulkblocks simultaneously with
goroutines, one succeeds (HTTP 200) while the other fails (HTTP 400).
This was because the database was locked. This is a different race
condition than what was described. Do you have a suggestion on how to
get around this?
—
Reply to this email directly, [2]view it on GitHub, or [3]unsubscribe.
You are receiving this because you were mentioned. Message ID:
@.***>
References

https://github.com/vkuznet
https://github.com/dmwm/dbs2go/issues/30#issuecomment-1267571889
https://github.com/notifications/unsubscribe-auth/AAA6RUQN6VGKXXUUTWJMMNDWBSLBHANCNFSM5TZ327ZA

@vkuznet I implemented the suggestions from the mattn/go-sqlite3 discussions and it seemed to resolve the database locking. I pushed my branch with the current work I have done to try to cause the racing conditions to happen in the bulkblocks API. In my logs, I see this:
[2022-10-05 14:25:16.405983633 +0000 UTC m=+3.899250753] HTTP/1.1 400 POST /dbs-one-writer/bulkblocks [data: 26088 in 1852 out] [remoteAddr: <IP_ADDRESS>:59288] [X-Forwarded-For: ] [X-Forwarded-Host: ] [auth: no-TLS cipher-none "no-auth-cert" no-auth-login no-auth-method] [ref: "-" "Go-http-client/1.1"] [req: 1.1505ms proxy-resp: 0]
[2022-10-05 14:25:16.508029675 +0000 UTC m=+4.001296794] DBSError Code:101 Description:DBS DB error Function:dbs.bulkblocks.InsertBulkBlocksConcurrently Message:Data already exist in DBS Error: Block /unittest_web_primary_ds_name_8268_stepchain/acq_era_8268-v8268/GEN-SIM-RAW#8268 already exists Stacktrace:
goroutine 617 [running]:
github.com/dmwm/dbs2go/dbs.Error({0xc20780?, 0x400055d520?}, 0x65, {0xb31e2c, 0x19}, {0xb424ee, 0x2b})
        /workspaces/dbs2go_worktrees/dbs2go_race/dbs/errors.go:172 +0x84
github.com/dmwm/dbs2go/dbs.(*API).InsertBulkBlocksConcurrently(0x40001ee000)
        /workspaces/dbs2go_worktrees/dbs2go_race/dbs/bulkblocks2.go:673 +0xccc
github.com/dmwm/dbs2go/web.DBSPostHandler({0xc24910, 0x400000e0d8}, 0x4000612300, {0xb251f6, 0xa})
        /workspaces/dbs2go_worktrees/dbs2go_race/web/handlers.go:542 +0x10ac
github.com/dmwm/dbs2go/web.BulkBlocksHandler({0xc24910?, 0x400000e0d8?}, 0x0?)
        /workspaces/dbs2go_worktrees/dbs2go_race/web/handlers.go:957 +0x54
net/http.HandlerFunc.ServeHTTP(0x10?, {0xc24910?, 0x400000e0d8?}, 0xab00fff340a04d4c?)
        /usr/local/go/src/net/http/server.go:2109 +0x38
github.com/dmwm/dbs2go/web.limitMiddleware.func1({0xc24910?, 0x400000e0d8?}, 0x400055cf50?)
        /workspaces/dbs2go_worktrees/dbs2go_race/web/middlewares.go:111 +0x40
net/http.HandlerFunc.ServeHTTP(0xa58d00?, {0xc24910?, 0x400000e0d8

This states that the block already exists in the database. Is this the type of racing conditions we are trying to resolve?

I'm glad that we resolve database lock issue and now can move forward. The error you got is not a racing condition. It clearly states that you try to insert a block which is already in DB.
The racing condition should happen as following:

you have two concurrent HTTP request with bulkblocks API
the JSON payload to bulkblocks API contains plenty of information, e.g. dataset configuration, see this part of the code https://github.com/dmwm/dbs2go/blob/master/dbs/bulkblocks2.go#L506-L571
now, if dataset configuration does not exists yet in DB it should be inserted, therefore we call insertDatasetConfigurations from https://github.com/dmwm/dbs2go/blob/2aad8db22a99f7eb7de7f4c91cfb9420bc809521/dbs/bulkblocks2.go#L38
now if both HTTP requests will do this at the same time such that their request is not yet written to DB then we will get racing condition, i.e. one HTTP will succeed writing to DB but another will fail due to transaction lock. These are two concurrent HTTP calls trying to inject common data which is required for bulkblock API

The question is how to simulate it. I think you need to create multiple JSONs with different blocks but the same common data, like dataset configuration, and then inject them concurrently. The more HTTP requests you'll send concurrently the likely you will hit the racing condition.


Move HostDB inside renter

nothing too fancy here. Mostly trying to move things without changing them significantly -- that can come later.
The most important change here is that the hostdb API calls are now routed through the renter. For the 0.5 release, they should be moved to the /renter/ route instead of /hostdb/. This can be done as part of the renter API overhaul.

build failed

with a billion data races.
if I had a nickel for every "fix tests" commit that failed... :P

is it necessarily a good idea to hide the dataraces by changing the behavior of the program during testing? I think that's bad practice.

This PR actually makes me pretty uneasy. The hostdb showed signs of instability in the past, and I just don't feel like the testing is there to provide security around the idea that this merge was done correctly.
Are there more things we can do to verify that bugs haven't been introduced into the hostdb?
Is there a test that walks through making an announcement, checking the blockchain, and verifying that the host gets into the hostdb? I believe there is.
I'm definitely definitely getting nervous about 0.5.0. We've changed a lot of things since the last release, and even though it hasn't been that many commits it's been a bit of a carpet-bomb.

wrt data races, I think it's fine if a test like TestWeightedList isn't also doing scanning in the background. Testing functions in isolation is good. I think it's a good idea to test that exported functions don't cause data races during scanning, but why go through the effort of adding a bunch of locking to tests like TestWeightedList?
More generally, I think we're going to want to implement this pattern (not always spawning background threads during testing) in other modules, like the gateway. These threads make it a pain to test certain functions in isolation, because they often rely on time.Sleep. That means they can nondeterministically screw up your module's state depending on how slow Travis feels like running that day. So now we have test functions that call time.Sleep as well...it's just gross.
This is sort of related to mocking as well. I think we're going to want a "minimal New" and a "production New" for each module, where the minimal version takes tons of mocks as arguments, and the production version just takes other modules. So the minimal New should instantiate as bare a module as possible, preferably just creating the object and calculating some initial conditions, and leaving it up to the tests (or production New) to spawn background threads, create loggers, subscribe to consensus, etc.

ok

merge conflicts

resolved


Housekeeping

It does feel weird not having travis tests... :)
I wonder if we could use Homu for this project. Catching causes of breakage between five repos might get hairy.

Ah, one more thing should go here -- move the whole gtk up one level.

That's far from easy though, so better merge as is.

Okay, mod gtk has been unwrapped. This probably is the last of major breaking changes.

Importing gtk::* no longer works because it introduces a conflict with fn main(). Importing gtk::traits::* should suffice.

Oh right, I'll re-add travis for this.

@GuillaumeGomez we may need some way to check for cross-repo breakage, e.g. if a change to glib or gdk breaks gtk.

Hum... I don't have a lot of knowledges in that field but yes, we need it. I'll find information about this.

A brute force solution might be to checkout gtk master and try to build examples. Cargo allows overriding dependencies locally so if you're in the gdk repo for example you can
mkdir .cargo
echo 'paths = ["."]' > .cargo/config
git clone https://guthub.com/rust-gnome/gtk _test
cd _test/examples
cargo build

Cargo will find the config and use the checked out gdk tree it references.

I'm afraid this way would take a veeeeeeeeeeeeeeeery long time for each PR.

You might be surprised when we axe the old signals stuff.

Oh that's right, I totally forgot that. Then what about you create a PR for that (it's your idea after all :p) ?

All right, now that travis is hooked up again, I'll look into that.

Thanks !


Delete Matlab

We haven't had any problems so far, and should test on the ISS Friday. Let's go ahead and pull the trigger to delete since this is blocking the ROS2 work.

Note that github thinks we are adding ctl_ros but this is a renamed ctl, since I had foolishly placed the main functionality in gnc_autocode/ctl.cc. So the old gnc_autocode/ctl.cc --> ctl/ctl.cc, old ctl/ctl.cc --> ctl/ctl_ros.cc.


Windows build

We need to build freetype on Windows to avoid pining Windows package to 2.5* and  *nix packages to 2.6*.

Test #2 with matplotlib and pillownow that is got merged..

Fixed in #3


Prevents column width from overflowing the River component

Summary
Closes https://github.com/github/primer/issues/2161
List of notable changes:
Just a simple style update to sizes River grid columns with fr units instead of %
What should reviewers focus on?
How cool and easy CSS grid is.
Steps to test:
Inspect the River component and confirm that neither column overflows the grid parent.
Supporting resources (related issues, external links, etc):
n/a
Contributor checklist:

[ ] All new and existing CI checks pass
[ ] Tests prove that the feature works and covers both happy and unhappy paths
[ ] Any drop in coverage, breaking changes or regressions have been documented above
[ ] New visual snapshots have been generated / updated for any UI changes
[x] All developer debugging and non-functional logging has been removed
[x] Related issues have been referenced in the PR description

Reviewer checklist:

[ ] Check that pull request and proposed changes adhere to our contribution guidelines and code of conduct
[ ] Check that tests prove the feature works and covers both happy and unhappy paths
[ ] Check that there aren't other open Pull Requests for the same update/change

Screenshots:

 Before  After

@rezrah - any idea what this strange CI error is about?
Building Remix app in production mode...
⚠ REMIX FUTURE CHANGE: The `serverModuleFormat` config default option will be changing in v2 from `cjs` to `esm`. You can prepare for this change by explicitly specifying `serverModuleFormat: 'cjs'`. For instructions on making this change see https://remix.run/docs/en/v1.16.0/pages/v2#servermoduleformat
The path "@primer/react-brand/lib/css/main.css?__remix_sideEffect__" is imported in  but "@primer/react-brand/lib/css/main.css?__remix_sideEffect__" was not found in your node_modules. Did you forget to install it?
✘ [ERROR] Could not resolve "./styles/tailwind.css"

    app/root.tsx:5:34:
      5 │ import tailwindStylesheetUrl from './styles/tailwind.css';
        ╵                                   ~~~~~~~~~~~~~~~~~~~~~~~

@rezrah - any idea what this strange CI error is about?

It's from our integration tests, where we test each PR against a CLI installation of popular UI frameworks - in this instance it's Remix. Remix made a CLI change upstream recently, which we need to fix on our end, so you can ignore this failure for now.


crm.activity.add problem on "START_TIME" and "STOP_TIME"

the method call asks me for the START_TIME field.
The date is structured as "2021-04-05T14: 00: 00-00: 00" which is the way indicated by the REST api, however when executing it, the activity arrives but not its start date or its deadline.
Clarify that the date is formatted as str

Can you provide code sample of request?

Can you provide code sample of request?

Thanks, look:

How it looks:

What I am doing is by means of a cycle to go through each record of a dataframe and obtain a field that has a date, to this date I concatenate the letter "T" and add the "-00: 00" at the end, as you indicates rest api in your javascript example.
If you require more feedback, please tell me. I thank you in advance for your prompt response.

According to bitrix documentation it should be a valid datetime (https://training.bitrix24.com/rest_help/crm/rest_activity/crm_activity_fields.php)
In most cases ISO 8601 should work for you, for example
2012-04-23T18:25:43.511Z
However you also can pass datetime as
23.04.2012 18:25:43
Personally I recommend to stick with iso 8601, that should be encoded from the box.

No, not work.
I have tried in various ways.
datetime.datetime.now (). isoformat ()
sfrftime
text string of the form "YYYY-MM-DDTHH-MM: SS"
And nothing works, it is curious that executing the webhook from the url in the browser does it. And the way I put the star_time in the url is: 2021-03-20T13: 00

SORRY!.
It works with isoformat (), I had a parenthesis that came from COMMUNICATIONS that the start_time fields did not work, really an apology and I thank you for all your attention supporting me. Your project is great.


Code coverage comment is not working for Forks based PR

UnhandledPromiseRejectionWarning: HttpError: Resource not accessible by integration
    at /home/runner/work/code-coverage-action/code-coverage-action/dist/index.js:3441:23
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async createComment (/home/runner/work/code-coverage-action/code-coverage-action/dist/index.js:6221:15)
    at async sendSummaryComment (/home/runner/work/code-coverage-action/code-coverage-action/dist/index.js:6050:5)
    at async runFeatures (/home/runner/work/code-coverage-action/code-coverage-action/dist/index.js:6671:3)
    at async runCodeCoverage (/home/runner/work/code-coverage-action/code-coverage-action/dist/index.js:6692:3)
    at async main (/home/runner/work/code-coverage-action/code-coverage-action/dist/index.js:6709:3)
(node:2193) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2193) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

For reference, https://github.com/barecheck/code-coverage-action/pull/55. There is an action that is marked as successfully finished but, there is an error inside, mentioned above.
As result, there is no GitHub code coverage report from the action.

Hi @vitaliimelnychuk ,
I am not sure, if I should create a new bug report as the title somehow fits to a bug, I am currently experiencing.
I am getting the following output from your action.

I am using the following github parameters:
      - name: Generate Code Coverage report
        id: code-coverage
        uses: barecheck/code-coverage-action@v1
        with:
          barecheck-github-app-token: ${{ secrets.BARECHECK_GITHUB_APP_TOKEN }}
          lcov-file: pr-coverage.lcov
          base-lcov-file: ref-coverage.lcov
          send-summary-comment: true
          show-annotations: "warning"

I have also attached the lcov
files for the base and the pr branch.
If you need anything else, please do not hesitate to ask.
ref-coverage.lcov.txt
pr-coverage.lcov.txt

Hi @keyseven123 , thanks for bringing this up.
I imagine the issue here is because the action couldn't get files diff between base and Pull request. Would you be able to also attach logs from Github Action? Wondering if there was any error logged from the execution?
Also, do you see this behaviour only for forked repositories? Would you be able to try opening a Pull request directly in the repository without fork?

Hi,
The only output of the GitHub action is
2024-08-20T15:24:50.3076787Z ##[group]Run barecheck/code-coverage-action@v1
2024-08-20T15:24:50.3076922Z with:
2024-08-20T15:24:50.3077759Z   barecheck-github-app-token: ***
2024-08-20T15:24:50.3077916Z   lcov-file: pr-coverage.lcov
2024-08-20T15:24:50.3078071Z   base-lcov-file: ref-coverage.lcov
2024-08-20T15:24:50.3078219Z   send-summary-comment: true
2024-08-20T15:24:50.3078367Z   show-annotations: warning
2024-08-20T15:24:50.3078489Z ##[endgroup]
2024-08-20T15:24:50.3082422Z ##[command]/usr/bin/docker exec  82ff6cbfc59f11a125a24f522218fdac536e4ba3196441bdb5719c777ef63149 sh -c "cat /etc/*release | grep ^ID"
2024-08-20T15:24:50.5953026Z lcov-file: pr-coverage.lcov
2024-08-20T15:24:50.7370954Z Current code coverage: 25.39%
2024-08-20T15:24:50.7372369Z base-lcov-file: ref-coverage.lcov
2024-08-20T15:24:50.8804575Z Base branch code coverage: 41.89%
2024-08-20T15:24:50.8806011Z Code coverage diff: -16.50%
2024-08-20T15:24:50.8807021Z Code coverage diff: -16.50%
2024-08-20T15:24:51.7473851Z send-summary-comment is enabled for this workflow
2024-08-20T15:24:52.7861553Z minimum-ratio: NaN
2024-08-20T15:24:52.7862643Z minimum-ratio is disabled for this workflow
2024-08-20T15:24:52.7864812Z Show annotations feature enabled

I can also try it directly in the repo to create it. I can do this tomorrow though.
Do you have an idea how I can maybe get more output of the GitHub action?


how can insert a data and find similar images from datasets and if  it not in dataset it will return None

how can insert a data and find similar images from datasets and if  it not in dataset it will return None  just like image search engine

@gautamankul
I was able to achieve it using below code.

`si = SearchImage()
ann = os.path.join('meta-data-files/','image_features_vectors.ann')
pkl = os.path.join('meta-data-files/','image_data_features.pkl')
image_data = pd.read_pickle(pkl)
u = AnnoyIndex(si.f, 'euclidean')
u.load(ann)
#path = '/content/sample_data/marker0_target1.jpeg'
path = '/content/sample_data/dd.jpeg'
query_vector = si.get_query_vector(image_path=path)
index_list = u.get_nns_by_vector(query_vector, 1, search_k=-1, include_distances=True)
#print(index_list)
#print(index_list[1])
distance = index_list[1][0]
index = index_list[0][0]
lt = image_data.iloc[index]['images_paths']
basename = os.path.basename(lt)
print(basename, distance)`

You can exclude the image with the distance threshold. I think 1 means similar to the image and the greater the distance the similarity decreases.
Note: I am no expert, I just wanted to do it for my POC.


dresses dont cover outer player skin layer

title. The players outer skin layer renders over your dress

Hello, can you show a via a screenshot here does it renders under the top skin layer

I also have 3d skin layers so that might be it

Oh, that is it, i've seen that one kinda breaking it, but its kinda minimal tho, probably isn't worth fixing because of the amount of time it would take to change all models

Got around to fixing it, currently fixed in the latest commit


Modular Specification

The OAB has directed the following:
Normative terms particular to the spec up are listed in the front of the document
    Collection, Web API
Other terms in a glossary for terms in an annex
    Conformance class, requirements class
With links internal and external to source spec.

CityGML 3.0 should be updated to comply with this direction.

A glossary annex has been added to the CityGML 3.0 standard
Terms and definitions which are not specific to CityGML have been moved to the glossary
Acronyms which are not specific to CityGML have been moved to the glossary

Discussed and agreed to close by editors


No arm64e in fat layer binary

Closes #2846 (but instead of using a proper arm64e, we're not using any arm64e binary at all).

See https://github.com/metalbear-co/mirrord/issues/2846#issuecomment-2419632439


Inaccurate path to @fastify/pre-commit/hook

Prerequisites

[X] I have written a descriptive issue title
[X] I have searched existing issues to ensure it has not already been reported

Fastify version
0.0.0
Plugin version
2.0.2
Node.js version
10.19.0
Operating system
Linux
Operating system version (i.e. 20.04, 11.3, 10)
ubuntu 20.04
Description
This fixes the installation to correctly find the .git folder to install...when running "npm install" in a subfolder:
https://github.com/fastify/pre-commit/issues/17
.git/
web/
web/node_modules
web/package.json
...

but if you look at the pre-commit it executing:
./node_modules/@fastify/pre-commit/hook
that should be
./web/node_modules/@fastify/pre-commit/hook
Steps to Reproduce

create a folder "web" same level as .git
cd web
npm install --save-dev @fastify/pre-commit
test a commit - errors
cat .git/hooks/pre-commit
see './node_modules/@fastify/pre-commit/hook' instead of './web/node_modules/@fastify/pre-commit/hook"

Expected Behavior
expect  to see './web/node_modules/@fastify/pre-commit/hook' instead of './node_modules/@fastify/pre-commit/hook"

We'd welcome a PR. But I doubt it will be easy. The module is written for plain projects, not monorepos.

@jsumners should just be one line change in install.js line 99
// locate the hook relative from the root of project - support subfolders
let hookRelativeUnixPath = hook.replace(path.resolve(git, '..'), '.')

@jsumners should just be one line change in install.js line 99

// locate the hook relative from the root of project - support subfolders

let hookRelativeUnixPath = hook.replace(path.resolve(git, '..'), '.')

Would you like to send a Pull Request to address this issue? Remember to add unit tests.

Sure I will do it when I have time;

I can just confirm that above change of line 99 helps to fix the path for the hook file execution, but the pre-commit will then fail with :
Error: Cannot find module '@fastify/pre-commit'
I had a further idea, to first go into the subfolder before executing the hook:
#!/usr/bin/env bash
cd ./web
./node_modules/@fastify/pre-commit/hook
...

I get 1 step further but now I have an other error:
pre-commit:
pre-commit: Received an error while parsing or locating the `package.json` file:
pre-commit:
pre-commit:   Cannot find module 'C:\abc\projectX\package.json'

So, we miss the correct place of the package.json now again, this definitely needs more changes to bring it to work with subfolder scenarios...
So on the following lines, you would need to fix the path as well:
try {
    this.json = require(path.join(this.root, 'package.json'))
    this.parse()
  } catch (e) { return this.log(this.format(Hook.log.json, e.message), 0) }

I don't have time to fix this here, but just wanted to share my findings. Cheers!


chore: all modules use new config

Test case for kube. BuildRestConfig will be added after we use terratest.

@gxthrj After this PR is merged, the dev docker image should be updated, also, docs and samples should be changed, since the way we configure ingress apisix is changed.

@gxthrj After this PR is merged, the dev docker image should be updated, also, docs and samples should be changed, since the way we configure ingress apisix is changed.

Should we update the docs in this PR?

@gxthrj After this PR is merged, the dev docker image should be updated, also, docs and samples should be changed, since the way we configure ingress apisix is changed.

Should we update the docs in this PR?

That's OK.

@gxthrj After this PR is merged, the dev docker image should be updated, also, docs and samples should be changed, since the way we configure ingress apisix is changed.

Should we update the docs in this PR?

That's OK.

@gxthrj Added.

Codecov Report

Merging #98 (d9f5234) into master (6eb76b5) will decrease coverage by 0.23%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master      #98      +/-   ##
==========================================
- Coverage   58.43%   58.20%   -0.24%
==========================================
  Files          15       15
  Lines         498      500       +2
==========================================
  Hits          291      291
- Misses        184      185       +1
- Partials       23       24       +1

Impacted Files
Coverage Δ

cmd/ingress/ingress.go
73.97% <66.66%> (-2.09%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6eb76b5...d9f5234. Read the comment docs.


[Feature Request] Display of stat bonuses from Scarves or other items in inventory

In the other PMD games, adjusted stats from scarves or inventory items were displayed in red on the Summary page. Not so much here which makes it hard to tell if a scarf is actually functioning as intended. I am assuming the stats are adjusted during Battle Phase after a small discussion in Discord.
If possible, maybe having a display of +X next to the stat in question? It would look cleaner and be more obvious as to how much you are gaining.

Since items currently affect the damage formula directly, this is a tricky one.  Ideally, the way to do this will be to run a portion of the damage formula that deals with attack multipliers and use the result it spits out for the summary.

That sounds like it can work. I know damage is randomized to a small degree so maybe have it shown as the average?


Multiplayer -- avatar not appearing on other side

Right now the avatar app is not loading consistently both sides. We need to fix this.

Fixing in https://github.com/webaverse/app/pull/3336

Fixed by https://github.com/webaverse/app/pull/3381


Exception when intercepting method with an argument passed by reference

Example of this issue
class Target
{
    public function targetMethod(array &$data)
    {
         // do something with $data
    }
}

The woven class would be like
public function targetMethod(array &$data)
{
    return call_user_func_array(self::$__joinPoints['method']['targetMethod'], [$this, ['data' => $data]]);
}

And get "Argument #1 ($data) must be passed by reference, value given" Exception

Thank you for your contribution :)
Should be good to go with version 1.2.14!


Control SPI chip select

I need to keep the SPI chip select asserted while transferring multiple messages (and doing some processing in between). The necessary low-level interface already exists (_CSpiIocTransfer.cs_change). Can this be made accessible from the high-level Python interface (i.e. SPI.transfer)?

Duplicated issue: https://github.com/vsergeev/python-periphery/issues/37, https://github.com/vsergeev/c-periphery/issues/19
It would be useful for me also. I made a modified def transfer_burst(self, data): method to do it. I could send it in a pull-request.
But pay attention that not all SPI controller dirver can support it, i needed to implement it in spi kernel driver for my SPI controller, too.


chore: change vite conf 2023 replay link

Description
For ViteConf 2023's replay link:
The old one https://viteconf.org/23/replay?utm=vite-sidebar is broken.
Maybe use https://www.youtube.com/playlist?list=PLqGQbXn_GDmkOsHI7-Wrbv1GgAA4tJZhg instead?

Thanks for the heads up, we'll get this fixed on the site.

Closing as the link is fixed.


Correction for raw affiliation INSA Centre Val de Loire, Univ. Francois-Rabelais, Blois, France

Correction needed for raw affiliation INSA Centre Val de Loire, Univ. Francois-Rabelais, Blois, France
raw_affiliation_name: INSA Centre Val de Loire, Univ. Francois-Rabelais, Blois, France
new_rors: 02wwzvj46;03y0qc033
previous_rors: 03ntmtb29;03y0qc033
works_examples: W2007031471
contact: 1abfd20f8135ab4e72cf270a0c6f697f:b981663bceeb0d7232a4a2fe7b112802642252bdef @ recherche.gouv.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-21. The new affiliations should be visible within the next 7 days.


Removed usage of deprecated aliases in generated code

Also denied all warnings in CI and check.sh

bors r+

bors retry

bors retry


feat: change to async based api

Description
Migrates to an asynchronous based api to maintain an interoperable public interface with other bbs-signature libraries in the TS/JS ecosystem

[x] Tests for the changes have been added (for bug fixes / features)
[x] The commit message(s) follow conventional commits
[ ] Documentation has been added / updated (for bug fixes / features)
[ ] Changes follow the contributing document.

Motivation and Context
As alluded to in the description in order for TS/JS based bbs-signature libraries to have a common public interface we are shifting this API towards a promise based API.
Does this PR introduce a breaking change?

[x] Yes
[ ] No

All API's now return a promise wrapped result
Which merge strategy will you use?

[x] Squash
[ ] Rebase (REVIEW COMMITS)

@tplooker I think this is ready to go. Anything else we need to do to merge it?


Update quickstart-v1-ios.md

Updated podfile to install correct ADAL library,

@FrankHu-MSFT : Thanks for your contribution! The author, @rwike77, has been notified to review your proposed change.

Closing here,


Add links to social media popup panel

Make use of the links and other_links fields, and add link buttons to popup panel, especially for Chinese version.

social media icon JSON here
{
    "icon_id": "605077bbd90ba600951cd0e0",
    "img": "https://webstatic.mihoyo.com/upload/operation_location/2021/03/16/40247bcc1f0092a7883949aade6619ac_3149808684055343336.png",
    "tittle": "启动器常见问题FAQ",
    "url": "https://bbs.mihoyo.com/ys/article/4004423",
    "qr_img": "",
    "qr_desc": "",
    "img_hover": "https://webstatic.mihoyo.com/upload/operation_location/2021/03/16/021147ecf6824c3328d6c89b9c1b17e6_3771842119503074595.png",
    "other_links": [
        {
            "title": "启动器意见集中反馈",
            "url": "https://bbs.mihoyo.com/ys/article/5780277"
        }
    ],
    "title": "启动器常见问题FAQ",
    "icon_link": "https://www.miyoushe.com/ys/article/39655595",
    "links": [
        {
            "title": "常用FAQ合集",
            "url": "https://www.miyoushe.com/ys/article/39655595"
        },
        {
            "title": "启动器意见集中反馈",
            "url": "https://bbs.mihoyo.com/ys/article/5780277"
        }
    ],
    "enable_red_dot": false,
    "red_dot_content": ""
}

old
new

Have no complaints with it. LGTM!


Add node_*_fsm_total [JIRA: RIAK-2214]

We need to track how many total gets / put / * FSMs occurred. We already track the point in time metrics, counting the total would allow people to take the derivative themselves.

Are you looking for something other than the node get/put totals?
ubuntu@riak101:/mnt/riak/lib/riak$ sudo riak-admin status | egrep 'node_(put|get)s.*total'
node_gets_counter_total : 0
node_gets_map_total : 0
node_gets_set_total : 0
node_gets_total : 23235117
node_puts_counter_total : 0
node_puts_map_total : 0
node_puts_set_total : 0
node_puts_total : 7381453
vnode_gets_total : 93129314
vnode_puts_total : 23447121


Require Node.js 10

To fix the prototype pollution in the yargs-parser, I bumped meow.
One thing led to another, so I also:

Made the linting / formatting work better with Prettier and VSCode.
Removed Node 8 from the Travis tests. Added 12 and 14.
Bumped all other dependencies as well.
Used require in the tests instead of import, just like in the rest of the code.

Thanks. I reverted the unrelated changes. Not something I want.


Translation IT

This pull request starts the work for Italian 🇮🇹 translations and can be used as a guide.
I did the following in order to start this work:

Added the translations.json file
Added it as a supported option to i18n
Added it as a toggle to the /settings page

See the files changed tab for more details and for clear examples

Please note, this pull request only scrapes the surface of IT translations. It is primarily used as a guide and IT still has a lot of translations to go
related: https://github.com/the-hideout/tarkov-dev/issues/175

.deploy


Python implementation using pybind11 and gobject introspection

Replaces #99

Working plugins here for now: https://github.com/zalox/jucipp-plugins

@milleniumbug Did you manage to test the plugins?

Not yet, I'm trying to. The missing Py_DecodeLocale was the immediate issue that prevented me from compiling this branch.

Ok, after installing necessary dependencies, setting site-packages in config.json to correct path, snippet.py works correctly, but tools.py doesn't seem to load at all.

Are there no errors in the terminal?

None at all. I've verified that it doesn't load by putting some logging statements in tools.py (with jucipp.terminal.println("test"))

Okay, nevermind. Moving snippet.py outside and renaming tools.py to snippet.py loads the plugin, so probably I have forgotten to put it on some list.

@eidheim I'm having trouble persisting my config.json file. Is there any mechanism that overwrites the file or anything in it?
I had the issues in msys2.

As long as you have the same version number in the config file, as in the files.h juCi++ will not touch the config file. However, if the versions are different, changes will be done the the old config file so that it corresponds to the new file specified in files.h.

I had some very strange behaviour then. I'll try to reproduce.

Windows does some file lock if a file is opened by another process? That might have been the case, denying the write to you your config file. Although, I'm not very experienced with Windows.

I've debugged program, and it seems it stops searching for plugins on first hidden file.
/home/milleniumbug/.juci/plugins/snippet.py - full
snippet - stem
/home/milleniumbug/.juci/plugins/.gitignore - full
 - stem
/home/milleniumbug/.juci/plugins/__pycache__ - full
__pycache__ - stem
/home/milleniumbug/.juci/plugins/README.md - full
README - stem
/home/milleniumbug/.juci/plugins/LICENSE - full
LICENSE - stem
/home/milleniumbug/.juci/plugins/tools.py - full
tools - stem
/home/milleniumbug/.juci/plugins/.git - full
 - stem

Changing from
    auto module_name=it->path().stem().string();
    if(module_name.empty())
      break;

to
    auto module_name=it->path().stem().string();
    if(module_name.empty())
      continue;

fixed the problem.

@eidheim Would it be possible to hook up to the auto-complete dialog somehow? While looking through the source I can't see any simple way as is...
Use-case:
Writing for would produce an auto-complete suggestion for a snippet as well as other completion data for for.

It might be possible to do it inClangViewAutocomplete::autocomplete_get_suggestions by asking i.e.Python::Interpreter::get_suggestions. The suggestions can be dynamically produced by clang context in plugins. If the interpreter and clang works in parallel I think it might work without a huge performance drop.
This approach isn't very generic and re-factoring might be needed for a reasonable solution.

@zalox I've been thinking about simpler ways to reuse the autocompletion dialog. I'll prioritize this in the coming weeks.

I updated the docker images with python now. I only updated the packages, so it shouldn't break other builds. https://hub.docker.com/r/cppit/jucipp/builds/

Coincidently, I'll test it on master now:)

https://github.com/cppit/jucipp-docker/commit/9cb5f6e2a5051342fc9b22439372ed9cb84c6c5f


Include go automation integration tests to the drone build process.

Issue:

Need to run go automation integration tests in the drone build process.
Problem

Solution

Added new go automation integration tests in the drone cli build process.

Updated build script to include building the new build-integration-setup bin
Updated test script to run integration tests
Updatde v2/integration to be able to generate a token for the admin user to run integration tests.

Testing

Drone build passes.
Engineering Testing
Manual Testing

Automated Testing

QA Testing Considerations

Regressions Considerations

/backport v2.6.9 release/v2.6

Backport: https://github.com/rancher/rancher/pull/38951


LuaJIT results

There is Lua and LuaJIT. But LuaJIT in this benchmark always run with JIT, without -joff flag, so it is always run with JIT on.
Besides being fast JIT compiler, luaJIT is one of the fastest interpreters (not to confuse with Lua, which also can be run with and without jit).
At least mark it in results table that LuaJIT is with Jit on

dup #406


Why set  the mask_scale=1

An interesting job, but I have some questions. Why set mask_scale=1  with multimask_output=True  instead of mask_scale=0  with multimask_output=False here?
https://github.com/ziqi-jin/finetune-anything/blob/85a0658cb0011a504aa73f2399cf4ab305cc8a65/extend_sam/mask_decoder_heads.py#L192

SAM supports three different scale instance mask outputs for each input, Users could choose the scale mask they want by set the mask_scale, you are very patient, I forgot to add this into the tutorial.