Like operator

Offering implementation of the "LIKE", "NOT LIKE" and "UNLIKE" string matching operators in the Filter grammar, and briefly describing their semantics.
Alternatively, offering implementation of the "MATCH /regexp/" operator.

They say that this is just because they first C-unescape the string, eg to accept \n as a newline. We don't, so we should just say \\ to escape a backslash.
Whoever converts the string to MySQL must then first C-escape whatever its obtained from the query string.

Maybe it's still good to point this out to implementers though, as I'm sure this will be a common error (but anyway if you don't want SQL injection you should encode your string, is the user passes a quote in itts string....)

Is the hope to get this in for v0.10? Because, after having mulled over this a bit since the meeting last Friday, I think this is something we may want to discuss a bit more before merging.
(To me, the point with the several rushed PRs this week is to make sure the spec is in a sane state for a v0.10 release, i.e., internally consistent and preferably without having rough completely backwards-incompatible changes just around the corner--- not so much to get new features in last-minute. I'd be happy to see a release v0.11 with this feature in shortly after v0.10.)
Now, about the content of this PR:
OPTiMaDe filters are designed to be user friendly for our target audience of 'client' end users, right? Hence, even as someone who knows SQL quite well, I'm not super stoked on building in the SQL convention into OPTiMaDe:

This choice has a very non-backend-agnostic "feel". Won't this antagonize those without SQL backends? "So, to implement OPTiMaDe we have to translate wildcard expressions from the SQL convention to our backend?"

I expect most client users of OPTiMaDe to not know SQL and the conventions of its LIKE operator.

I expect those users to find SQL's % and _ unexpected for a wildcard mini-language. Just Google "wildcard" and disregard the hits about SQL; you'll find a wide array of descriptions of built-in functions across various software that very universally uses * for multi-character wildcards, and often ? for single character matches. Two notable examples includes:

Shell glob patterns in Unix shells, Windows powershell, and even MS-DOS.
Microsoft excel, access, etc. as well as all software that mimics the same functions (google sheets, libreoffice) (though, libreoffice apparently also supports regex syntax.)

I suspect users are likely to have come across one of these.

Maybe worst: as coming out from the point @giovannipizzi brought up, I don't think there is a single escape syntax standard for SQL LIKE valid across SQL backends, so this is not a great standard anyway... And, we must make sure escaping in our LIKE does not interfere with string escaping in our string tokens. Reading the specification we cannot say that identifier LIKE "\%" is a search for a literal percent sign, because the right hand side is not a valid string token. A valid string token can only contain \ followed by a " or a \. So, for us it would have to be "\%", and how do we then encode a literal backslash + a wildcard %?

So, my suggestion is for us to skip SQL LIKE entirely, and give the operator a different name, e.g., PATTERN with the conventions:

* for any number of arbitrary characters (including zero).
? for exactly one arbitrary character.
** for a literal *
?? for a literal ?
the use of \ remains as for any string token: \\ for literal \ and \" for literal ".

This seems somewhat user-friendly, and would be trival to search-replace into a corresponding SQL LIKE operator for us with SQL backends.

I see @rartino's point and I tend to agree that, if there is no agreement, it's better not to merge this in 0.10 yet.
And to admit, I also think * and ? are more intuitive (and should be easy to replace for SQL syntax in the backend).
A note on the choice of escaping: I think this is not yet "perfect":

** as a literal *: would then *** mean [a] "a literal * followed by any sequence of characters" or [b] "any sequence of characters followed by a literal *"? Might seem a little detail, but for instance the string ab*cd would match the pattern ab*** in interpretation [a] but not in interpretation [b].
Similar concern for ???
For ?? there is also an additional concern: how then can we specify that we want exactly two characters?

I think the only solution is to have a third character for escaping. I see that using backslash is counterintuitive because one has to escape the string again (indeed to encode \\ one would have to write \\\\ e.g. in JSON) and multiple escaping is the hardest thing ever to get right ;-) )

@giovannipizzi are indeed correct about the quoting scheme not yet being perfect.
(For anyone confused by the discussion about ??, ???, these are based on the first version of my comment; shortly after posting it I noticed the problem with ?? myself and re-edited it to use \*?, thinking no one would notice, sorry. But even with those edits, *** remains broken.)
I have a different suggestion for quoting now: going forward, even with other functionality, I think it may be very useful if we can extend our string tokens to represent "literal characters" aside "normal characters" using a single backslash.
It would require a change in the grammar, so \<character> is allowed for any character in a string token, marking that character as "literal". (Right now it is forbidden by the grammar for <character> to be anything else but a " and \.)
We then need some carefully formulated text in the specification, where we say that for all regular uses of a string token, a literal character \<character> means the same thing as <character>. However, in other situations where some special meaning has been assigned to characters, a literal character refers to that character without that special meaning.
Then in a PATTERN, ? and * are assigned the special meaning as specified above. It then follows from the paragraph above that \? and \* are used to represent the literal ? and *. I think this would be elegant.
Note: this solution allows marking any character in an OPTiMaDe string as "literal" with the following exception: the characters " and \ are always literal, one cannot indicate non-literal versions of these characters. I think that is an OK trade-off. Practically this could mean limitations on what syntax we can allow for other mini-languages embedded in OPTiMaDe strings, e.g., if we want to adopt a standard language for time formatting that uses " and \ with their own special meanings, etc.
We might want to think forward to make sure this allows embedding a reasonable regex format. I think it does.

@rartino wrote:

So, my suggestion is for us to skip SQL LIKE entirely, and give the operator a different name, e.g., PATTERN with the conventions:

for any number of arbitrary characters (including zero).
? for exactly one arbitrary character.
** for a literal *
? for a literal ? (whereas ?? indicates two arbitrary characters, and ? is the standard way to indicate an arbitrary character sequence of at least one character)
the use of \ remains as for any string token: \ for literal \ and " for literal ".

I though about this in the beginning (Issue https://github.com/Materials-Consortia/OPTiMaDe/issues/87), and then I realised that this is a very bad idea. Essentially we put on our shoulders a burden of developing yet another wildcard language, with the explicit goal to be not similar to existing ones, and with no apparent gains. This is a large work; as @giovannipizzi has just pointed out it is very easy to get it wrong; it gives no clear benefit, and adds extra burden on every implementer (because now everyone will have to implement, test and support this notation), and on every OPTiMaDe user (since everyone will have to learn a new wildcard language). I think this is very inefficient and unnecessary, since we will hardly give in the new wildcard syntax any new essential features that one does not have in SQL patterns, globs or regexps.
And then, what is the reason to be not similar to SQL, or any other pattern/regexp standard? One might object to it if it were a MUST feature and one would have to re-implement it in an SQL-unfriendly backend; but since it is supposed to be an optional feature, those backends that find it difficult to implement can simply skip it, and offer their versions of 'MATCH REGEXP' instead. I think sticking to a known standard, if that standard is well documented, has clear benefits: a/ one can find working and tested implementations; b/ the documentation is out there c/ many people will already know the syntax d/ those who learn it anew will be able to use it also in other contexts (thus will have more incentives to learn it)
I therefore suggest the following strategy:

let's settle on several widespread standards with accessible implementations. I suggest picking SQL LIKE patterns, ERE regular expressions, and PCRE regular expressions.

All such matches will be optional. Thus, the implementer can choose to implement those matching operators that are easy to match to their respective backend, and skip all the rest. Nobody should be put at disadvantage by this.

let's have in mind possible extensions of the patterns. This is the reason I suggested (in https://github.com/Materials-Consortia/OPTiMaDe/issues/87) to have regexp type specified in the match expression (e.g. MATCH PCRE); this will allow adding new flavours of regexps would such extensions be needed in the future (I personally doubt that it will be ever needed, but you never know what new "revolutionary" DBs will be rolled out with incompatible and non-standard interfaces ;). One possible backend which we up to now did not discuss much is SparQL; I'm not sure if it uses its own match expressions or some standard ones. Still, it fits into the proposed scheme, either with 'MATCH ERE' if SparQL supports ERE, or with 'MATCH SPARQL' if SparQL has (will have) its own matching syntax, and we decide to support it in the future.

OPTiMaDe in itself is neutral as to the implementation of the matches, but we need to specify what the semantics of the matches will be in case they are implemented. Thus I think we can do it without extension of the existing Filter grammar.

In such setting, which is very simple, and is actually implied in this PR, I think we could make it even into v0.10 :).
Regards,
Saulius

It would require a change in the grammar, so \<character> is allowed for any character in a string token, marking that character as "literal". (Right now it is forbidden by the grammar for <character> to be anything else but a " and \.)

It could be done, but at the moment I think it is totally unnecessary. Filter language itself does not define or handle regular expressions, just strings; strings can encode arbitrary character sequences, including backslash with any characters; we just need to escape the backslash as a double backslash. Thus, an ERE matching arbitrary number of stars after the a character would be a\**, and embedded into a Filter string it will become "a\\**". The Filter language quotes and backslashes are one encapsulation level higher than those of regexps. The situation is similar as with Java regular expressions.

@rartino

* Reading the specification we cannot say that `identifier LIKE "\%"` is a search for a literal percent sign, because the right hand side is not a valid string token. A valid string token can only contain `\` followed by a `"` or a `\`.

True

So, for us the right hand side would have to be "\\%",

True, this is a logical consequence of the current grammar. Java does the same with regexps in strings, awk does it the same way, and I suspect Python has no other option but to behave the same.

and how do we then encode a literal backslash + a wildcard %?

a LIKE "\\\\%" Not the nicest looking but usable (and in fact used in many occasions).
I still need to double-check how SQL escapes %.

* This choice has a very non-backend-agnostic "feel". Won't this antagonize those without SQL backends?

Why should it? As said, if the people find it difficult to implement, they just skip it (the feature is supposed to be optional). Unless they feel bitter envious that SQL has better pattern matching than their beloved super-modern backend, there should be no problems :P. And then again, I suggest also introducing (again optional) match operator syntax that would be easy to implement in alternative, non-SQL backends.

I expect most client users of OPTiMaDe to not know SQL and the conventions of its LIKE operator.

Why? At least currently, most of the OPTiMaDe consrtium members do know about at least something SQL :)

I expect those users to find SQL's % and _ unexpected for a wildcard mini-language.

You can always learn :). You search for "SQL LIKE", and in 5 mins you have the answer. Also, we will include a short description and a couple of examples into the OPTiMaDe spec (I'll update my PR with these features).

Just Google "wildcard" and disregard the hits about SQL; you'll find a wide array of descriptions of built-in functions across various software that very universally uses * for multi-character wildcards, and often ? for single character matches. Two notable examples includes:

Shell glob patterns in Unix shells, Windows powershell, and even MS-DOS.

That is why I suggested also describing the MATCH GLOB operator, with shell glob semantics; I can include it into the PR; I just had impression that we decided to have only LIKE and no more advanced operators when we skyped in Friday.

Microsoft excel, access, etc. as well as all software that mimics the same functions (google sheets, libreoffice) (though, libreoffice apparently also supports regex syntax.)

Again, this would be an argument to include an optional MATCH ERE and MATCH PCRE operator.
What does MongoDB and/or CoachDB support, actually? These would be probably the most likely backends that some of us actually use.

I'll get to the individual points, but I see an important philosophical difference here, so lets start with sorting that out:

let's settle on several widespread standards with accessible implementations. I suggest picking SQL LIKE patterns, ERE regular expressions, and PCRE regular expressions.
All such matches will be optional. Thus, the implementer can choose to implement those matching operators that are easy to match to their respective backend, and skip all the rest. Nobody should be put at disadvantage by this.

When we agreed on the need for optional features in the filtering language, you were very concerned about the fragmentation in support for filters that this would lead to. My argument was that optional features actually helps avoid fragmentation, because instead of every provider inventing their own extension for feature X, we now say in the standard that "you don't have to support feature X, but if you do, this is how you do it." That way, all providers that support feature X do it the same way, and queries are at least universal across all databases that support that feature.
However, to get this effect, we need to be conservative with adding several optional features that are overlapping, i.e. exactly what you propose to do here. If we do, we get back to the same fragmentation we were trying to avoid. Now all databases will support only their own favorite brand of partial text matching operators, and partial text matching queries are not going to be universal even across databases that support partial text matching. Clients will constantly have to rewrite their queries to fit the text matching capabilities of each database. This, to me, is a too high cost for everyone to get their favorite syntax in.
So, I'm very much against supporting e.g. multiple regex flavors, and rather want us to standardize on a flavor that is translatable into the specific ones used by the backends. I realize I put more work on implementors here, but this is necessary to achieve the main goal of a universal api: to provide a common interface for access.

On the specific points:

Essentially we put on our shoulders a burden of developing yet another wildcard language, with the explicit goal to be not similar to existing ones, and with no apparent gains. This is a large work; as @giovannipizzi has just pointed out it is very easy to get it wrong;

I'm with you if we are talking about full regex support, I don't think we want to invent a new flavor of regexes. But, if we talk about patterns with character wildcards, that is trivial apart from how to do escaping. And, unless I am mistaken, there is no SQL standard for that escaping. If we pick one, e.g. MySQL, we still have to do the same amount of work to make sure we embedd it in a way consistent with our string quoting mechanism. This isn't easier than starting from scratch; and for all intents and purposes, we are creating a new pattern language in the eyes of users, because the actual strings they need to input are different in, e.g., how many backslashes are needed.
In fact, what is likely to happen with the SQL approach is that users wondering how to represent a literal % will find a stack overflow question about, e.g., postgree SQL, try that syntax with whatever number of backslashes they use, and then be confused over the exact same string behaving differently in OPTiMaDe.

Filter language itself does not define or handle regular expressions, just strings; strings can encode arbitrary character sequences, including backslash with any characters; we just need to escape the backslash as a double backslash.

Right, but the point in much of the design we do is for the filter language to be user friendly. I know from heavy amount of experience that no one appreciates this "backslash hell" where several quoting mechanisms are stacked to yield unreadable expressions. If we can support single backslash qouting, is it not nice to do so?

Ehhh... why should we disregard SQL?

My point here was that it is only SQL that uses % and _. There are far more pattern mini-language using * and ?. Hence, for users not familiar with SQL LiKE, this is going to be a surprise.

At least currently, most of the OPTiMaDe consrtium members do know about at least something SQL :)

We are all database providers. Who are you designing OPTiMaDe for? I'm having the PhD students at our department in mind who do various projects that would benefit from easy aggregating of data across databases. I think few of them know SQL.

I'm with you if we are talking about full regex support,

ACK. And so, one flavour of matches should include RE support; but we do not make it mandatory and do not want to exclude the use of simpler matching mechanisms, do we? The proposed solution, with 'LIKE', 'MATCH ERE', 'MATCH PCRE' would permit standard implementation of various levels of patterns.

I don't think we want to invent a new flavor of regexes.

I agree, we don't :)

But, if we talk about patterns with character wildcards, that is trivial apart from how to do escaping.

I don't find it trivial; it's not just regexp search/replace, since we will need to deal with escaped characters and character ranges within [] brackets. I feel that a client will have to do a full pattern parsing, and then map it into the backend's patterns/regexps if we want to have specific OPTiMaDe pattern syntax.

And, unless I am mistaken, there is no SQL standard for that escaping.

I think there is; I would be very much surprised if it did not.
I do not have the latest SQL standard at hand (costs 180$ with ISO :/), but open sources on the Web (https://modern-sql.com/standard, http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt) specify that the syntax is column LIKE 'pattern%' or column LIKE '30\%pattern% ESCAPE '\', where ESCAPE allows to specify any escape character one wishes. MySQL defaults ESCAPE to ''; other SQL description do not mention any default, so probably default backslash escape is MySQL specific extension. That does not cause difficulties if we specify backslash as an escape character for OPTIMADE since all SQL databases should support the standard ESCAPE clause.

If we pick one, e.g. MySQL, we still have to do the same amount of work to make sure we embedd it in a way consistent with our string quoting mechanism.

As said, SQL ESCAPE mechanism should allow to specify escape character for any SQL engine.

This isn't easier than starting from scratch; and for all intents and purposes,

I would say it is much easier than starting from scratch, we just take a (subset) of a standard implementation.

we are creating a new pattern language in the eyes of users,

Not if we take an existing standard(s) and reference it/them.

because the actual strings they need to input are different in, e.g., how many backslashes are needed.

We should not confuse defining the meaning of patterns, and defining the way how to encode patterns into strings. Currently the OPTiMaDe string definition allows encoding any character sequence (using escapes); so we have no problem here. If we think that too many backslashes in escaped strings are awkward to read (I agree with this), then we can think about how to improve OPTiMaDe string escaping; this is an issue independent from match-pattern semantics, IMHO.

In fact, what is likely to happen with the SQL approach is that users wondering how to represent a literal % will find a stack overflow question about, e.g., postgree SQL, try that syntax with whatever number of backslashes they use, and then be confused over the exact same string behaving differently in OPTiMaDe.

I don't find this argument convincing. If people take Postgress SQL specific (non-standard) rules and try to apply to something else, be it OPTiMaDe, MySQL, MogoDB or what not, of course they will get nonsense, and here you can not help them in any way – except maybe by writing decent OPTiMaDe documentation (that specified how OPTiMaDe patterns work) and advising them to use '-stackoverflow' in their Google searches :P.

My point here was that it is only SQL that uses % and _. There are far more pattern mini-language using * and ?. Hence, for users not familiar with SQL LiKE, this is going to be a surprise.

SQL seems to be still the most widely used database query language. Its patterns, as I see now after some days of searching on the Web, are very well described and present in all searches. Disregarding this as "only SQL" does not seem wise to me. Nobody will be surprised at SQL pattern languages if they do a short search on the Web.
Yes, there are essentially two match syntaxes – SQL LIKE and PCRE. All the rest is variations and subsets of these two. SQL seems to be a fairly large chunk of DB implementations; probably SQL LIKE is the most frequently supported backend operator, if we take by number of installations, no?
Another is ERE/PCRE. Also fairly popular. That's why I suggest having possibility to use either or both of them in OPTiMaDe queries.

and advising them to use '-stackoverflow' in their Google searches :P.

This won't help.  If we refer to some other standard in our specification, then we have to follow it as closely as possible. It's the "SQL" in "SQLish" that may get the user searching the Web and trying first hits of SQL syntax how-tos.

When we agreed on the need for optional features in the filtering language, you were very concerned about the fragmentation in support for filters that this would lead to. My argument was that optional features actually helps avoid fragmentation, because instead of every provider inventing their own extension for feature X, we now say in the standard that "you don't have to support feature X, but if you do, this is how you do it." That way, all providers that support feature X do it the same way, and queries are at least universal across all databases that support that feature.

I agree, and this argument convinced me that optional features are not bad. They will be even better if supplemented by machine-readable descriptions of "feature sets" (see https://github.com/Materials-Consortia/OPTiMaDe/issues/91, second suggestion).

However, to get this effect, we need to be conservative with adding several optional features that are overlapping,

I agree if the features are completely overlapping, but here we have different features with different trade-offs between search power and implementation complexity.

i.e. exactly what you propose to do here.

This is not true. I propose three well-defined optional features, so that, as you say, "if you do [them], this is how you do it.". Implementers then can choose which ones to implement.

If we do, we get back to the same fragmentation we were trying to avoid.

Not really.

Now all databases will support only their own favorite brand of partial text matching operators,

NO! If compliant to OPTiMaDe, they will support the ones described in the spec.

and partial text matching queries are not going to be universal even across databases that support partial text matching. Clients will constantly have to rewrite their queries to fit the text matching capabilities of each database.

With optional features you will have to adapt your queries to different implementation, no way around it as long as you introduce options. This could only be made somewhat easier if each OPTiMaDe endpoint advertises in a machine-readable form which optional features they support (the above-mentioned "feature sets:").
And then, you do not need to re-write constantly, there will be no constant changes once the specs are out, will there?

This, to me, is a too high cost for everyone to get their favorite syntax in.

Its not about favourite syntax, its about easier implementation. We can not require all implementers say "look, we have invented our own matching semantics, no go and implement it in your backend".

So, I'm very much against supporting e.g. multiple regex flavors, and rather want us to standardize on a flavor that is translatable into the specific ones used by the backends.

At the moment I think that "translatable into the specific ones used by the backends" is too difficult to be reasonably implementable.

I realize I put more work on implementors here, but this is necessary to achieve the main goal of a universal api: to provide a common interface for access.

I think the burden you suggest for implementers is unacceptably high. In that case, we are probably better of with just what is in the specs right now, STARTS/ENDS/CONTAINS. Regexp filtering can then be done at the client side if needed...

and advising them to use '-stackoverflow' in their Google searches :P.

This won't help.  If we refer to some other standard in our specification, then we have to follow it as closely as possible. It's the "SQL" in "SQLish" that may get the user searching the Web and trying first hits of SQL syntax how-tos.

This would be a good reason to make LIKE behaviour as close to the SQL's as possible, this is my main point.

@rartino :

If we do, we get back to the same fragmentation we were trying to avoid.

One more though on avoiding fragmentation.
Lets consider the following scenario. I want to send a query to find entries that match RE 'Saulius.*Grazulis' to multiple OPTiMaDe databases. I want to use pattern match, but I do not know which databases implement which features.
I could compose the following Filter expression: (author LIKE "%Name%Surname%" OR author MATCH GLOB "*[Nn]ame*[Ss]urname*" OR author MATCH REGEXP "[Nn]ame.*[Ss]urname"). Let's assume that  in an OR expression, unimplemented operators return FALSE, and implemented ones the expected values. Then, if a database implements at least one of the queries, I will get the expected answer; and I could send this query to all databases.
The only problem is when an OPTiMaDe endpoint does not implement any of these operators; then I would get no data at all instead of getting the right answer.
Another approach would be, as @rartino suggests, to have the `author MATCH "[Ss]aulius[Gg]ražulis*" operator, with wildcards specified by the OPTiMaDe, which each API implementation will have to translate to the appropriate backend query.
The difference between the two approaches are: in the first case, we put translation burden on the user; in the second case we put the translation burden on the API implementers. Arguably, it is easier to compose a query in different languages, sincne you know what you are looking for, than to parse a formal definition and translate it in all cases to another language. So, assuming qualification of both users and implementers is the same, composing multiple queries is easier than translating. It is only if we target very inexperienced users that composing queries becomes an issue.

I could compose the following Filter expression: (author LIKE "%Name%Surname%" OR author MATCH GLOB "*[Nn]ame*[Ss]urname*" OR author MATCH REGEXP "[Nn]ame.*[Ss]urname"). Let's assume that in an OR expression, unimplemented operators return FALSE, and implemented ones the expected values. Then, if a database implements at least one of the queries, I will get the expected answer; and I could send this query to all databases.

So, in effect, you propose that the syntax in OPTiMaDe for partial string matching is to give manually translated versions of your pattern to cover ALL relevant flavors of pattern languages? You don't think this is overly cumbersome to use, even for experienced users? To me, your example shows precisely why supporting all these overlapping options all as individually optional is a bad idea.
The translator approach is not as bad as you suggest. If we are talking about patterns (and not regexes) it is going to be a more or less straightforward string transform. And as soon as someone implements these, the code can be converted to other languages and distributed as support libraries. Whereas if we put all of the burden on the user as you propose, you can bet every second query is going to have one of the redundant patterns translated in error, and users will act on incomplete search results without knowing it.

However, to get this effect, we need to be conservative with adding several optional features that are overlapping,

I agree if the features are completely overlapping, [...]

If you agree with that statement; can we perhaps discuss a "ladder" of complexity levels? This is my view:

Basic operations: STARTS WITH, ENDS WITH, CONTAINS: required
Basic pattern with arbitrary character string (e.g., * or %), arbitrary single character (e.g., ? or _): optional
Full regex matching support: optional

What I am trying to say is that it would be optimal to only provide ONE facility for text matching on each level. E.g., SQL LIKE or GLOB-like or something else for level 2. But, I agree that Level 3 is trickier. Maybe we can postpone the discussion of REGEX flavors for a while, because I'd like to think more on the question "translating" in this context. And this PR is about level 2.
With reference to this ladder of complexity, I propose that if an API implementer supports a higher level, they MUST also support all lower levels. (This should be ok, as the lower levels are translatable into higher level expressions.) This means that we can recommend a user to use the lowest level at which a query is expressible, and it will then be handled by the highest possible number of databases.
Now, specifically about whether the SQL LIKE standard is a good idea for OPTiMaDe level 2:

if we talk about patterns with character wildcards, that is trivial apart from how to do escaping.

I don't find it trivial; it's not just regexp search/replace, since we will need to deal with escaped characters and character ranges within [] brackets. I feel that a client will have to do a full pattern parsing, and then map it into the backend's patterns/regexps if we want to have specific OPTiMaDe pattern syntax.

Why are you talking about character ranges? I'm talking about LIKE-equivalent (level 2) patterns here as you have described it in your PR, so one arbitrary length wildcard, and one single-character wildcard + escaping rules for those. And the mapping I am talking about then is from our "own" syntax (? and *) into the LIKE syntax you want (% and _), not into the backends regex format. You are certainly allowed to implement these patterns via SQL LIKE, and those who don't will have an equally hard time from this point as with an SQL LIKE-based syntax.

I do not have the latest SQL standard at hand (costs 180$ with ISO :/),

Isn't this a major problem in itself that not even we have access to the standard you want to base the OPTiMaDe pattern operator on? Especially since we conclude the need to firmly point users to it, since, there are a LOT of helpful sources on the Internet that cover database-specific flavors that isn't exactly implementing the standard. (E.g., MS SQL Server supports [ and ] for ranges, whereas it seems few others do.)
I'm already running into this issue, as for my response above, I tried to figure out if the SQL standard LIKE actually supports character ranges with [ and ], but I find no place I can confirm of deny it with certainty. At least in the 1992 draft you linked, they are not supported.

the syntax is column LIKE 'pattern%' or column LIKE '30\%pattern% ESCAPE '\', where ESCAPE allows to specify any escape character one wishes. MySQL defaults ESCAPE to ''; other SQL description do not mention any default, so probably default backslash escape is MySQL specific extension. That does not cause difficulties if we specify backslash as an escape character for OPTIMADE since all SQL databases should support the standard ESCAPE clause.
As said, SQL ESCAPE mechanism should allow to specify escape character for any SQL engine.

After researching this, I'll accept that LIKE with ESCAPE behaves fairly standard across SQL backends, aside that some add their own pattern extensions... Without ESCAPE, not so much. In my reading of the 1992 draft, it seems the default should be no escape symbol at all. This is apparently what Oracle uses (according to a help page for v10.1 I found online). But then many other implementations (MySQL, postgresql) takes backslash as the default, which leads to confusing "help" webpages.
What is the concrete proposal for OPTiMaDe? That we also include the ESCAPE keyword? I.e., to search for a literal backslash + wildcard one have these options for filter: identifier LIKE "\\\\%" ESCAPE "\\" or, e.g., identifier LIKE "#%" ESCAPE "#"? What is our default escape char?
My end conclusion is that I suppose it wouldn't be a catastrophe to standardize on LIKE + ESCAPE, but I'm far from convinced it is the best choice, since:

There is no openly accessible standard to point users to.
Many actual SQL implementations do not follow the standard, which makes it confusing for end users who seek help (which is worsened by point 1)
For those who want to support LIKE without having an SQL backend, the construct with arbitrary escape character adds extra work with not very much benefit.
End users with little experience in SQL will find % and _ unusual as wildcards, and the syntax with LIKE + ESCAPE bulky.

And what is the gain compared to the proposed alternative?:
identifier MATCHES <PATTERN> where in <PATTERN>:

* means any number of arbitrary characters (including zero).
? means exactly one arbitrary character.
\* is a literal *
\? is a literal ?
\\ is a literal \

which you can translate to an SQL LIKE query for your backend this way:

Make these substitutions in <PATTERN>:

% -> \%
_ -> \_
? -> _
* -> %

Query your SQL backend with identifier LIKE <PATTERN> ESCAPE "\"

My end conclusion is that I suppose it wouldn't be a catastrophe to standardize on LIKE + ESCAPE, but I'm far from convinced it is the best choice, since:

I agree. I also think we both agree that ESCAPE would be too bulky and unnecessary for OPTiMaDe.

1. There is no openly accessible standard to point users to.

Well, not quite true, one is (SQL-92)[http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt] that describes the LIKE predicate, and from the documents that we discussed privately we can see that SQL-99 did not change that definition significantly.

2. Many actual SQL implementations do not follow the standard, which makes it confusing for end users who seek help (which is worsened by point 1)

True, but those implementations only extend the standard, so 'column LIKE pattern ESCAPE ''` should be portable across SQL implementations.

3. For those who want to support LIKE without having an SQL backend, the construct with arbitrary escape character adds extra work with not very much benefit.

I think we all agree that OPTiMaDe should specify one fixed escape character (or one fixed escape mechanism, such as character doubling).

4. End users with little experience in SQL will find `%` and `_` unusual as wildcards, and the syntax with LIKE + ESCAPE bulky.

After all the research that we have done on the Web, I strongly disagree that "% and _ [are] unusual as wildcards". Rather, I would say, * and ? are limited to the shell glob community... :)

And what is the gain compared to the proposed alternative?:
identifier MATCHES <PATTERN> where in <PATTERN>:
* `*` means any number of arbitrary characters (including zero).

* `?` means exactly one arbitrary character.

* `\*` is a literal `*`

* `\?` is a literal `?`

* `\\` is a literal `\`

So essentially we arrive at the same SQL LIKE convention, with the backslash ('\') as an escape character :). I'm not against that, since, in addition to being a reworded SQL LIKE, this is superficially similar to shell GLOBS, but I want to point out that '*' and '?' have no clear advantage over '%' and '_'; as said, I strongly disagree that '%' are "unusual"; moreover, '?' is confusing since REs use '.' instead, and ? means something slightly different there (I myself was confused by this in my young days ;).

which you can translate to an SQL LIKE query for your backend this way:
1. Make these substitutions in `<PATTERN>`:

   * `%` -> `\%`
   * `_` -> `\_`
   * `?` -> `_`
   * `*` -> `%`

This is not that straighforward as you write, since \* does not need to be translated to \%, it translates to *, and \\* would not translate to \*, what is meant presumably 'backslash-any_character' which would be \\%', whereas \ais 'backslash-a' which translates to\a` in LIKE...
In other words, doable but requires extra work to get it right. My original instinct is that I want to avoid that work if possisble.

If you agree with that statement; can we perhaps discuss a "ladder" of complexity levels? This is my view:
1. Basic operations: STARTS WITH, ENDS WITH, CONTAINS: _required_

2. Basic pattern with arbitrary character string (e.g., * or %), arbitrary single character (e.g., ? or _): _optional_

3. Full regex matching support: _optional_

Yes, I agree on this complexity ladder, this is what I meant. We can settle on this :)

After researching this, I'll accept that LIKE with ESCAPE behaves fairly standard across SQL backends,

I also find it so.

aside that some add their own pattern extensions...

I have the same impression.

Without ESCAPE, not so much.

Probably yes...

In my reading of the 1992 draft, it seems the default should be no escape symbol at all.

True, I think.

This is apparently what Oracle uses (according to a help page for v10.1 I found online).

I thought Oracle has a (non-standard?) feature SET ESCAPE ... that changes the behaviour, bu we do not need to rely on that.

But then many other implementations (MySQL, postgresql) takes backslash as the default,

True

which leads to confusing "help" webpages.

OPTiMaDe should be the only Help :)

What is the concrete proposal for OPTiMaDe? That we also include the ESCAPE keyword?

property LIKE pattern with default backslash ('\') as an escape character, translatable to SQL as:
property_column LIKE pattern ESCAPE '\'

with pattern taken more-or-less verbatim after resolving OPTiMaDe string double-quote and backslash escapes). More-or-less dependes on whether we take % and _ as wildcards (then it will be verbatim), or * and ? as wildcards, then it will need translation with good chances that people will get it incorrectly :)

Why are you talking about character ranges? I'm talking about LIKE-equivalent (level 2) patterns here as you have described it in your PR

I meant the most general case, level 3, with regexps.
This is the whole point. Do we design syntax only for the level 2 patterns (LIKE), or do we want it to be general enough to be usable for level 3 as well? If it is only for LIKE, then regexps will have different metacharacters and different escape rules anyway (that is why I mentioned '[]' brackets). This would imply different syntax constructs for different complexity levels, as I suggest now. And since SQL LIKE seems to be the only existing standardised implementation giving the level 2 complexity, I see no reason why we should deviate from it.
If we want just a single OPTiMaDe operator, MATCH, that covers all complexity levels (essentially, 2 and 3), then we need to think how we convey them in the syntax. Should brackets '[]' be escaped by a back-slash even at the level 2? Should bare [] be reported at level 2 as errors even though they are not used for matching?  Wil it be the same at level 2 and level 3? We need to decside all these issues before we move on.
My suggestion is essentially to segregate level 2 and level 3 syntactically, by having predicated LIKE pattern (for level 2) and predicate MATCH ERE pattern (for level 3). True, this will require the user to know all these different pattern encodings; but a serious computer user will have to know them anyway, IMHO, and indicating syntactically which complexity level we mean will simplify job for specification writers (meaning us ;), for implementers, and arguably also for users (there will be no confusion what ? means in p LIKE 'xa?' and p MATCH ERE xa?`)
We can, of course, say that level 3 is not supported by OPTiMaDe at all, but that would be a pity, since some backends allow to implement RE search easily, and people will start doing this in their own incompatible ways, which we wanted to avoid by describing optional (but standardised) features.
As a side note: SQL (SQL-99) has SIMLAR TO predicate that supports bracketed character ranges in a standard way. Not all backends seem to implement it, though (MySQL that we have does not). I am not suggesting it for OPTiMaDe :)

After talking with Andrius over the lunch, we came up with an alternative (compromise?) solution:
Let's leave just to "complexity levels":

STARTS/ENDS/CONTAINS operators (required);

ERE MATCH operator (optional).

That is, we skip LIKE with all its "unusual" wildcards altogether.
The REs used could be a common subset of expressions supported by ERE and PCRE.
The RE strings can be encoded as OPTiMaDe strings (in that case, the RE backslash will be encoded as two back-slashes in a string), or we can provide a /-delimited specialised RE syntax – in that case, we can use single physical backslash to escape RE meta-characters, improving readability at the expense of extra grammar rules in the Filter language.
Rationale:

not supporting LIKE only makes problems for backend engines that implement SQL LIKE but do not implement RE matches. Arguably, there are not many of such engines;

MATCH, if implemented, covers all functionality of LIKE, and more;

when a backend supports (PC|E)REs, the implementation will be simple – just pass the RE to the backend RE engine;

since MATCH is optional, anyone who encounters difficulties implementing RE match can simply skip it;

the mandatory STARTS/ENDS/CONTAINS operators offer some of the LIKE functionality;

since we have only one optional feature, there will be minimal fragmentation and no room for confusion;

setting on EREs allows to reuse the existing ERE standard (POSIX), with increased likelihood that it will be implemented in backends in a compatible way.

A Regexp survey:

MySQL uses the extended version to support regular expression pattern-matching operations in SQL statements
(https://dev.mysql.com/doc/refman/5.7/en/regexp.html)

MongoDB uses Perl compatible regular expressions (i.e. “PCRE” ) version 8.41 with UTF-8 support.
(https://docs.mongodb.com/manual/reference/operator/query/regex/)

PostgreSQL supports the same three regular expressions flavors: Tcl Advanced Regular Expressions, POSIX Extended Regular Expressions and POSIX Basic Regular Expressions.
(https://www.postgresql.org/docs/9.3/functions-matching.html)

A word of caution:

While most regular-expression searches can be executed very quickly, regular expressions can be contrived that take arbitrary amounts of time and memory to process. Be wary of accepting regular-expression search patterns from hostile sources. If you must do so, it is advisable to impose a statement timeout.

LIKE searches, being much simpler than the other two options, are safer to use with possibly-hostile pattern sources.
(https://www.postgresql.org/docs/9.3/functions-matching.html)

I like your latest suggestions for how to support regexes. I see no issue with implementing regex support first and leave a hypothetical level 2 until later.
But, can you clarify a homework-not-done question: is PCRE a strict superset of functionality of EREs so we can simply say that OPTiMaDe uses the POSIX ERE format, and databases that uses PCRE can then simply plug those ERE expressions into their PCRE engines without issue? Or, do we need to say that we only support the common subset of features supported by both EREs and PCREs? Because for the latter, we will have to do some work to define the precise syntax.
Furthermore, I'm in support (but it is not an overly strong opinion) of reducing the number of backslashes via your suggestion of a regex token on format /... regex pattern.../, OR, alternatively picking up my idea further up in this thread to extend our string tokens to carry meta information about per-character literal-ity. (Which is conceptually a bit more tricky, but would resolve this issue more generally for all kinds of embedded mini-languages.)
Now onto the basic pattern level (level 2):
I'm not quite convinced of the idea of completely dropping the basic pattern level. Two reasons from the top of my mind:

OPTiMaDe support for basic patterns allows backends that do not support regexes (or, at least not, exactly EREs/PCREs), but which can do other decent partial text matching, to provide at least that level of support. (As far as I can find online, an example of such a backend is MS Access.)
Novice users may find the basic pattern syntax easier than the full regex syntax.

I agree with most of your latest comments about level 2 support, so we may be converging on a design here. In particular we seem to agree that:

level 2 and level 3 partial text matching should use segregated syntax.
OPTiMaDe basic pattern support should work equivalently to standard SQL LIKE with ESCAPE \ (but possibly with other wildcard characters.)

So, perhaps then all that is left is to decide is between these alternatives?:

SQL LIKE-based basic patterns with wildcard characters % and _ as it is in that standard.
SQL-LIKE-like basic patterns with wildcard characters * and ?.
Skip basic pattern support, have only regex.

I see good and bad things with each of these options, so I propose to bring this up for discussion at the next meeting.

@rartino :
just a quick answer to the ERE<->PCRE compatibility question:

But, can you clarify a homework-not-done question: is PCRE a strict superset of functionality of EREs so we can simply say that OPTiMaDe uses the POSIX ERE format, and databases that uses PCRE can then simply plug those ERE expressions into their PCRE engines without issue? Or, do we need to say that we only support the common subset of features supported by both EREs and PCREs? Because for the latter, we will have to do some work to define the precise syntax.

I have DuckDucked yesterday on this subject and could not find a definitive answer, but it seems that ERE and PCRE are not in the subset-superset relation. The demonstration can be found in character set ([a-z]) behaviour:

in EREs, "An escape inside a character class declaration shall match itself: in other words the escape character is not "special" inside a character class declaration; so [\^] will match either a literal '\' or a '^' " (https://www.boost.org/doc/libs/1_64_0/libs/regex/doc/html/boost_regex/syntax/basic_extended.html);

in Perl flavour of PCER, 'Characters that may carry a special meaning inside a character class are: "\", "^", "-", "[" and "]" /.../  The sequences "\a", "\c", "\e", "\f", "\n", "\N{NAME}", "\N{U+hex char}", "\r", "\t", and "\x" are also special and have the same meanings as they do outside a bracketed character class' (man perlrecharclass), which in my understanding is incompatible with the ERE definition.

This is attested by the behaviour of GNU grep:
saulius@koala ~/ $ echo -e 'one\ntwo\none and a half\n1\\2\nTAB\tSEPARATED\nthree' | grep --color -E '[ \t]'
two
one and a half
1\2
three

saulius@koala ~/ $ echo -e 'one\ntwo\none and a half\n1\\2\nTAB\tSEPARATED\nthree' | grep --color -P '[ \t]'
one and a half
TAB	SEPARATED

That is, with the -E (ERE) option, the GNU grep treats \ in square brackets as an ordinary character (so both \ and t are matched), while in the -P (PCRE) mode the \t in the bracketed character set is treated as a code for the TAB character, and is matched accordingly.
So yes, we will have to define a common subset...
(More on you remaining comments later; I think we are converging :)

I like your latest suggestions for how to support regexes. I see no issue with implementing regex support first and leave a hypothetical level 2 until later.

I agree.

Furthermore, I'm in support (but it is not an overly strong opinion) of reducing the number of backslashes via your suggestion of a regex token on format /... regex pattern.../, OR,

Fine, I will add more tests to check the current grammar more thoroughly.
We also need to decide on two approaches:

do we want the Filter grammar to accept any RE character and any RE escape int the /.../ construct

or do we want it to specify the syntax of the supported REs in full.

The second approach is more flexible – to add a RE extension in the future, you do no need to change the Filter grammar, just amend the documentation. The downside is that now Filter language will permit any sequence of characters in the REs, even if they do not make sense as REs, and rely on (informal) description for the back-end implementers to detect these;
The approach is more strict – we will have a definite grammar to specify what a filter RE is; implementations accepting this (or compatible) grammars will reject incorrect REs at parsing stage, and all will do it in the same way; and we can attach more formal meaning to RE constructs. However, future extensions would need extension to Filter grammar, an may potentially conflict with the current parsing engine capabilities.

alternatively picking up my idea further up in this thread to extend our string tokens to carry meta information about per-character literal-ity. (Which is conceptually a bit more tricky, but would resolve this issue more generally for all kinds of embedded mini-languages.)

I am not quote sure what you mean by literal-ity here, and I am afraid that this approach already contains conflicting purposes (conflicting == not implementable at the same time in the same syntax).
The current filter strings currently can encode arbitrary printable character sequence (which is probably enough for search language). To make it fully transparent, we can add \n, \t, \012 and \x1BCD style escapes, to specify arbitrary character sequences (both printable and non-printable).
Regular expressions are encoded by those character strings. The strings serve only as a wrapper layer for REs. This means, for example, that \s will not have any special meaning in strings (since it does not stand for any specific Unicode character); but in PCREs the two characters \ and s for a special wildcard \s matching arbitrary space. We could say that \ and a letter s in strings mean just those two characters if it is not a special escape, but what will happen when at some stage one will need \b to denote back-tab control character and it will already be taken by a \ and b character sequence?
We can of course say that in Filter strings any combination of a backslash and a character is allowed, both these characters continue a string even if the second character is a double-quote, and we do not prescribe any interpretation to these characters; i.e. in "some \strange string" the \s pair might mean "arbitrary space" in REs and "literal s character" in strings... I am not sure if this is a good approach since we leave implementations to interpret the meaning of the escapes.
/.../

I see good and bad things with each of these options, so I propose to bring this up for discussion at the next meeting.

OK.

Now onto the basic pattern level (level 2):
I'm not quite convinced of the idea of completely dropping the basic pattern level.

I was originally in favour of this idea, and so am I now :)

Two reasons from the top of my mind:
* OPTiMaDe support for basic patterns allows backends that do not support regexes

(or, at least not, exactly EREs/PCREs), but which can do other decent partial text matching,
to provide at least that level of support. (As far as I can find online, an example of such a backend is MS Access.)

* Novice users may find the basic pattern syntax easier

than the full regex syntax.
In addition to these, the security warning mentioned by Postgresql people should be considered as well.
Also, simpler patterns allow easier and more uniform implementation.

I agree with most of your latest comments about level 2 support, so we may be converging on a design here. In particular we seem to agree that:
* level 2 and level 3 partial text matching should use segregated syntax.

Agreed.

* OPTiMaDe basic pattern support should work equivalently to standard SQL LIKE with ESCAPE `\` (but possibly with other wildcard characters.)

I agree on the first part, but I do not see a reason to change wildcard characters.
Moreover, if we say that OPTiMaDe LIKE is designed and should work in the same way as the SQL LIKE, changing wildcards will introduce more confusion, not less. As an experiment, I have search in DuckDuckGo for "LIKE SQL patterns", and the whole first page points to SQL tutorials and manuals than plainly mention '%' and '_' as the wildcards, with no mention of '' and '?'. Google will probably give the same result. Imagine how confused and irritated the users will be if the documented SQL '%' wildcard does not work, and instead they find out that OPTiMaDe has changed them to '' for... clarity!

So, perhaps then all that is left is to decide is between these alternatives?:
* (1) SQL LIKE-based basic patterns with wildcard characters `%` and `_` as it is in that standard.

If we implement LIKE and explicitly say that it should behave like in SQL, then I'm very much in favour of this option;

* (2) SQL-LIKE-like basic patterns with wildcard characters `*` and `?`.

I rather strongly opinionated against this, since here we invent yet another match pattern that needs to be translated into SQL (extra work, extra bugs), needs to be learned and remembered by end users (extra confusion), and gives no clear benefit.
The shell globs that this is modelled did enter computer folklore, for sure (in form of "*nix" systems ;), but this does not mean that inexperienced users will understand them better than SQL patterns, and for experienced users like us, I think, uniformity with existing standards and tools is more important.

* (3) Skip basic pattern support, have only regex.

This could be an option if we want to bet on simplicity of the OPTiMaDe specs, non-proliferation of features and if we want to avoid fragmentation of implementations. For this to work, we need to very carefully describe what we mean by regexps and how they are intended to work, so that implementations are simple enough and the searches are powerful enough. I see a common subset of ERE and PCRE as a possibility, but we need to look of this does not exclude too many useful patterns (as I have mentioned before, even [\t\n] character ranges pose a problem). We need to be careful not to reduce the resulting REs to the SQL LIKE functionality :)
In my perception, options would be grouped from most favourable to least favourable as (1), (3), (2), with (1) and (3) pretty close in my view (if we do them right). An yes, I am strongly against (2).
Let's discuss during the Skype meeting. It would be interesting also to have opinions of @giovannipizzi , @merkys , @fawzi and others.

This has been sitting for a while, perhaps long enough for others to solve part of the problem. We discovered in #436 that the JSON Schema standard now may have defined a reasonable "common" regex subset here:
https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-01#name-regular-expressions
This PR includes adding "MATCH" to the filter language with a regex expression. If the PR is updated to also document "MATCH" with info on that the regex syntax is the SHOULD-level limited one documented in the JSON Schema text I think that part of this PR could be merged. (The limited regex syntax could either be spelled out in our spec or referenced; however, I'm a bit wary of referencing a draft specification; maybe including it as a quote is the best solution?).
I suppose we also just should agree on the LIKE syntax. I suggest we send out an email to the full OPTIMADE list to get a bit of broader feedback from the community, and then take a decision on the next web meet based on the resulting discussion (if any).

This has been sitting for a while, perhaps long enough for others to solve part of the problem. We discovered in #436 that the JSON Schema standard now may have defined a reasonable "common" regex subset here:
https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-01#name-regular-expressions
This PR includes adding "MATCH" to the filter language with a regex expression. If the PR is updated to also document "MATCH" with info on that the regex syntax is the SHOULD-level limited one documented in the JSON Schema text I think that part of this PR could be merged. (The limited regex syntax could either be spelled out in our spec or referenced; however, I'm a bit wary of referencing a draft specification; maybe including it as a quote is the best solution?).

The syntax in JSON Schema draft seems to me to be the very core of many regular expression syntaxes I am aware of, thus I support this proposal. On the other hand, I would prefer something more stable than draft specification, but spelling the syntax out in our spec would suffice.
Alternatively we may select one of the other syntaxes, preferably stable and having wider support. For example PCRE2 which has implementations at least in Perl, Python and JavaScript.

Alternatively we may select one of the other syntaxes, preferably stable and having wider support. For example PCRE2 which has implementations at least in Perl, Python and JavaScript.

The issue is not what is supported by programming languages, but across database backends, since it is your database that needs to be able to search efficiently for matches of the regex.

Alternatively we may select one of the other syntaxes, preferably stable and having wider support. For example PCRE2 which has implementations at least in Perl, Python and JavaScript.

The issue is not what is supported by programming languages, but across database backends, since it is your database that needs to be able to search efficiently for matches of the regex.

Right. Just checked MariaDB (backend for COD) and Mongo, both seem to support PCRE2 (although COD would have to update MariaDB to v10.5.1 to have PCRE2 instead of PCRE). It seems to me that JSON Schema draft is a subset of PCRE2.

Just checked MariaDB (backend for COD) and Mongo, both seem to support PCRE2 (although COD would have to update MariaDB to v10.5.1 to have PCRE2 instead of PCRE). It seems to me that JSON Schema draft is a subset of PCRE2.

Just the fact that one of the most active OPTIMADE participants would need to update their backend is perhaps itself an argument against, e.g., PCRE2.
Nevertheless, also consider, e.g.:

https://learn.microsoft.com/en-us/sql/ssms/scripting/search-text-with-regular-expressions?view=sql-server-ver16
https://docs.oracle.com/cd/B13789_01/appdev.101/b10795/adfns_re.htm
https://www.ibm.com/docs/en/db2/11.5?topic=sql-regular-expression-control-characters
https://www.elastic.co/guide/en/elasticsearch/reference/current/regexp-syntax.html
https://docs.snowflake.com/en/sql-reference/functions-regexp.html
https://docs.splunk.com/Documentation/Splunk/9.0.3/SearchReference/Regex (PRCE1 I think?)
https://duckdb.org/docs/sql/functions/patternmatching.html#regular-expressions

etc., etc.
(To be honest, I didn't explicitly check that none of the above doesn't happen to exactly be equal to PCRE2, just that they don't explicitly say so.)

Just the fact that one of the most active OPTIMADE participants would need to update their backend is perhaps itself an argument against, e.g., PCRE2.

Makes sense. However I have to check to make sure JSON Schema regex is really a subset of the old PCRE that our backend currently uses.

This has been sitting for a while, perhaps long enough for others to solve part of the problem. We discovered in #436 that the JSON Schema standard now may have defined a reasonable "common" regex subset here:
https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-01#name-regular-expressions

After a quick look at the JSON Schema REs, I have the question similar to that one we had with the LIKE operator strings: how do we escape special characters? E.e. what is the regular expression according the the JSON schema variety to match "(", ")", "|" and similar characters?
Maybe I overlooked in the https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-01#name-keyword-independence document, but I could not found the escape methods there...

@sauliusg, according to the regular expression examples given in https://json-schema.org/understanding-json-schema/reference/regular_expressions.html, special characters can be escaped using a double backslash.

@sauliusg @vaitkus:
@sauliusg makes a good point about the escapes. The text in the json schema standard points to the syntax standardized in ECMA-262, section 21.2.1 which says the syntax is "modeled" after Perl 5. However, the json schema standard then says - on SHOULD level - that the expressions should be "limit[ed] [...] to the following regular expression tokens", which doesn't actually include any description of the escape token.
This is likely an oversight. In the ECMA standard it is handled in the grammar where an Atom can be \ plus an AtomEscape, which is then further discussed in <IP_ADDRESS>.1.
If we end up quoting this subset of ECMA into our standard (as we discussed above that we may want to do), we probably want to clarify this.
Furthermore, @sauliusg did question whether the lazy wildcards was widely supported; so we may want to consider to also exclude those, making our subset even more stricter than the one described by json schema.

@sauliusg, according to the regular expression examples given in https://json-schema.org/understanding-json-schema/reference/regular_expressions.html, special characters can be escaped using a double backslash.

Thanks, @vaitkus for finding this document! AFAIR, this is the same "random page" that I tried to refer to during the last meeting, or at least it looks very similar :).
Under these circumstances I have no more objections to include a subset of these regexps, or the subset of regexps listed in https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-01#name-regular-expressions, in OPTIMADE.
The only point I'd like to stress rather strongly:
I would not reference any JSON document as a source of the description. I would rather explicitly list the permissible regexps in the OPTIMADE standards; we can (and should) reference the rationale, where these definitions come from – PCRE, JSON Schema, JSON Schema draft, etc., but the definitive list should be in the OPTIMADE standard.
All RE construct listed in the mentioned documents seem to be modeled after the PCRE, and PCRE is probably the most convenient form of REs, so I suggest we go for it.
That would leave us with one 'string MATCH "re"' or 'string MATCH /re/' operator, RECOMMENDED, to implement RE matches.

I would not reference any JSON document as a source of the description. I would rather explicitly list the permissible regexps in the OPTIMADE standards; we can (and should) reference the rationale, where these definitions come from – PCRE, JSON Schema, JSON Schema draft, etc., but the definitive list should be in the OPTIMADE standard.

Yes, also from prior discussion above I think we are generally in agreement on that the rules need to be stated in our standard.
However - this a technicality - it is relevant for us to say that what we end up with is not just based on/inspired by the PCRE regex subset in the JSON Schema draft, but is equal to, or a subset (whichever is true) of the JSON Schema draft standard. The reason is that this matters for the other place we want to invoke this regex standard, namely when specifying RE string validations in property definitions.
Question: what should a server do if a client uses REGEX that are valid PCRE2 but NOT constrained as specified? Since we've concluded that the ECMA standard is based on Perl 5 (i.e., PCRE2 if I'm not mistaken) we may as well say that supporting the subset is a MUST, but full support of PCRE2 is allowed, and on MAY level?

That would leave us with one 'string MATCH "re"' or 'string MATCH /re/' operator, RECOMMENDED, to implement RE matches.

Right, about that - the PR right now implements them as "/"-delimited. I was initially surprised by that decision, but realized I had no reason to oppose it. Does anyone see things differently?

I agree with most of your latest comments about level 2 support, so we may be converging on a design here. In particular we seem to agree that:
* level 2 and level 3 partial text matching should use segregated syntax.

* OPTiMaDe basic pattern support should work equivalently to standard SQL LIKE with ESCAPE `\` (but possibly with other wildcard characters.)

So, perhaps then all that is left is to decide is between these alternatives?:
* SQL LIKE-based basic patterns with wildcard characters `%` and `_` as it is in that standard.

* SQL-LIKE-like basic patterns with wildcard characters `*` and `?`.

* Skip basic pattern support, have only regex.

So to make things converge – is there a consensus, or at least not a strong opposition, to the following OPTIONAL features:

The LIKE operator: string LIKE "%s_". Characters of the SQL standard are used (% for any sequence of characters, _ for any single character; alternatively – * for any sequence of characters, ? for any single character, which is essentially a shell glob); the UNLIKE operator MUST be a synonym of NOT ... LIKE (i.e. NOT (string LIKE "*s?") <=> string UNLIKE "*s?"). Escape character is backslash (''), which escapes itself ("\\" → a string with a single character ''`);

The MATCH operator that uses a subset of PCRE (based on [1] and [2]); string MATCH /abc.*xyz[0-9]{3,7}/; I'll check if slashes '/' can be easily accommodated into out Filter language grammar – otherwise we use the standard Filter strings. Escape character is backslash ('\'), which escapes itself ("\" → a string with a single character '\').

Shall we go for it?

That would leave us with one 'string MATCH "re"' or 'string MATCH /re/' operator, RECOMMENDED, to implement RE matches.

Right, about that - the PR right now implements them as "/"-delimited. I was initially surprised by that decision, but realized I had no reason to oppose it. Does anyone see things differently?

You are right, I start forgetting what is in the PR :).
The /re/ syntax should minimize the escape backslashes that are needed.

Question: what should a server do if a client uses REGEX that are valid PCRE2 but NOT constrained as specified? Since we've concluded that the ECMA standard is based on Perl 5 (i.e., PCRE2 if I'm not mistaken) we may as well say that supporting the subset is a MUST, but full support of PCRE2 is allowed, and on MAY level?

If we do so, OPTIMADE RE will have to stay a subset of PCRE2 at least until the next major release. I am not against it, just stating that all additions/changes will have to be PCRE2 compatible. In addition, PCRE2 standard has life of its own, so if we tie OPTIMADE RE to PCRE2, we have to tie to some specific version.

If we do so, OPTIMADE RE will have to stay a subset of PCRE2 at least until the next major release. I am not against it, just stating that all additions/changes will have to be PCRE2 compatible. In addition, PCRE2 standard has life of its own, so if we tie OPTIMADE RE to PCRE2, we have to tie to some specific version.

This is all true. And, arguably (but I suspect this will be argued...) our reference should be ECMA-262 v11 just like JSON Schema rather than a specific version of the PCRE2 library.
Nevertheless, I feels strange about forcing those with full PRCE2 implementations in their backends to put in the extra work to lock away useful features. I suspect people will be reluctant to do that in practice.
We could say (not saying it is a good idea) something along the lines of: "For non-compliant REGEXs servers MAY behave in any undefined way, including serving matches according to the PCRE2 standard (or any other standard). Non-compliant REGEXs SHOULD trigger a warning. Servers are free to document their behavior for non-compliant REGEXs."
Such a statement does work against general compatibility, but will make implementing MATCH for various underlying REGEX standards a lot easier. Also for those backends where one need to translate REGEXs it would be nice to only have to worry about what the translator is doing for compliant REGEXs.

The MATCH operator that uses a subset of PCRE (based on [1] and [2]); string MATCH /abc.*xyz[0-9]{3,7}/; I'll check if slashes '/' can be easily accommodated into out Filter language grammar – otherwise we use the standard Filter strings. Escape character is backslash ('\'), which escapes itself ("" → a string with a single character '\').

Using / in grammar might clash with math operator / if we want to introduce math operators in future. SQL seems to support math operators.

I think the fact that the RE slash follows the MATCH keyword and REs never participate in arithmetic expressions should resolve the ambiguity...

The MATCH operator that uses a subset of PCRE (based on [1] and [2]); string MATCH /abc.*xyz[0-9]{3,7}/; I'll check if slashes '/' can be easily accommodated into out Filter language grammar – otherwise we use the standard Filter strings. Escape character is backslash (''), which escapes itself ("" → a string with a single character '').

Doesn't the grammar change in the PR already do this with "/"?

Yes it does; I forgot it (had to review the PR again ;)

Using / in grammar might clash with math operator / if we want to introduce math operators in future. SQL seems to support math operators.

I think the fact that the RE slash follows the MATCH keyword and REs never participate in arithmetic expressions should resolve the ambiguity...

Indeed. I suppose our LALR parsers will be able to handle that.

I suggest to skip LIKE entirely and just add MATCH. For simple matching we already provide BEGINS WITH, ENDS WITH, and CONTAINS which are user friendly in their definitions (i.e., no need to remember any particular wildcard syntax). Those with more sophisticated needs for general wildcard matching can use MATCH. Lets try to keep down the implementation work for servers.

Agreed. So we are converging on:

The MATCH operator that uses a subset of PCRE (based on [1] and [2]); string MATCH /abc.*xyz[0-9]{3,7}/; I'll check if slashes '/' can be easily accommodated into out Filter language grammar – otherwise we use the standard Filter strings. Escape character is backslash ('\'), which escapes itself ("\\" → a string with a single character '\').

The /re/ syntax should minimize the escape backslashes that are needed.

This is an excellent point. This is why we have to use / as the delimiter. Otherwise users would be exposed to figuring out our own double-escaping rules for regexs inside strings. As someone who has occasionally been exposed to preparing regexs through >= 4 layers of escaping, extra escape layers is something that absolutely should be avoided.

To test for the possible grammar conflicts with putative / operator I made the following change (which I think reflects a possible use-case):
diff --git a/optimade.md b/optimade.md
index 030be71..a364197 100644
--- a/optimade.md
+++ b/optimade.md
@@ -2164,7 +2164,7 @@ Filter = [Spaces], Expression ;

 Constant = String | Number ;

-Value = String | Number | Property ;
+Value = String | Number | Property | Number, '/', Number ;
 (* Note: support for Property in Value is OPTIONAL *)

 ValueList = [ Operator ], Value, { Comma, [ Operator ], Value } ;

With it, every grammar test fails with the following output:
> Error: in tests/generated/Filter.g: line 110:
>     inherent ambiguity in production 'Value' starting with tokens "+",
>     "-", "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", and "."
>
> Value = String | Number | Property | Number '/' Number;
>

Thus I assume / operator and current RE syntax cannot co-exist. Which is not a show-stopper, unless we foresee introduction of math operators in OPTIMADE.

Thus I assume / operator and current RE syntax cannot co-exist. Which is not a show-stopper, unless we foresee introduction of math operators in OPTIMADE

This strikes me as strange, and may be a consequence on the details of how /regex/ is introduced in the grammar, because it should be possible to formulate the grammar so that a division / can ONLY follow a <number> whereas <non-number>/ puts you in a state where the following <anything>/ completes the regexp, and thus cannot be ambiguous with a division.

Thus I assume / operator and current RE syntax cannot co-exist. Which is not a show-stopper, unless we foresee introduction of math operators in OPTIMADE

This strikes me as strange, and may be a consequence on the details of how /regex/ is introduced in the grammar, because it should be possible to formulate the grammar so that a division / can ONLY follow a <number> whereas <non-number>/ puts you in a state where the following <anything>/ completes the regexp, and thus cannot be ambiguous with a division.
edit Actually, looking at your error message, isn't that just because it is impossible to distinguish Number and Number '/' Number? What if you rewrite the rule as:
Value = String | Number, ['/', Number] | Property

This sounds strange to me as well, since I thing we tried things similar to REs, I guess. I wold like to look closer into the grammar and see if we can transform it in such a way that the language stays the same (with REs) and the ambiguity goes away.

@sauliusg Why did you close this? If we get #490 merged, this PR can be updated to just reference that format for the MATCH operator and we will be done?


libri animalia crashes the game

Hi,
when I click on a animal with the libri I get a new topic in the book. But when I try to read the topic, the game just crashes.

do you have a crash message?

Sorry I forgot. Here it is:
AsyncErr: Lua: Runtime error from mod 'animalia' in callback on_playerReceiveFields(): ...\minetest-5.8.0-win64\bin..\mods\animalia/api/libri.lua:348: attempt to concatenate field 'text' (a nil value)
stack traceback:
...\minetest-5.8.0-win64\bin..\mods\animalia/api/libri.lua:348: in function 'render_element'
...\minetest-5.8.0-win64\bin..\mods\animalia/api/libri.lua:364: in function 'get_page'
...\minetest-5.8.0-win64\bin..\mods\animalia/api/libri.lua:490: in function <...\minetest-5.8.0-win64\bin..\mods\animalia/api/libri.lua:481>
...\minetest-5.8.0-win64\bin..\builtin\common\register.lua:26: in function <...\minetest-5.8.0-win64\bin..\builtin\common\register.lua:12>

Should be resolved. Reopen if it persists after the latest commit.


Bug: MockInstance omits type check when returning desired shape

Description of the bug
maybe I'm missing something, but in case if I need to mock some class instance,
I would expect MockInstance to give me an error if in desired shape I specify the property that doesn't exist
class Project {
  name = 'Project name'
}

MockInstance(Project, () => ({
  names: 'this property does not exist'
}))

It works with single value, though
MockInstance(Project, 'names', 'this property does not exist') // error ;

Hi there,
actually it should. Could you try your code on stackblitz or codesandbox playgrounds?
I'll check later on today. Greetings from the Alps.

There is our coverage https://github.com/ike18t/ng-mocks/blob/master/tests-failures/mock-instance-service.ts

Looks like something is wrong in your project, errors are displayed correctly: https://codesandbox.io/s/infallible-hertz-o6nnm7?file=/src/test.spec.ts
I'm going to close the issue, feel free to reopen with steps how to reproduce it, or a min repo which demonstrates the issue.

Sorry, bad example from my side. The real issue seems to be more strange.
Errors are not thrown if the mock has at least one correct property specified. Then any extra properties are allowed
If no prop is specified correctly, invalid properties trigger error.
class Project {
	name = 'Project name';
}
MockInstance(Project, () => ({
	name: 'this property does exist', // works
	names: 'this property does not exist', // should throw the error, but doesn't
}));

Here's a sandbox that demonstrates this
https://codesandbox.io/s/nice-frog-tri7p5?file=/src/test.spec.ts

Ah, I see.
I think, what you are requesting is a typescript feature, and should be changed there:
https://www.typescriptlang.org/play?strict=false&noImplicitAny=false&strictNullChecks=false&strictFunctionTypes=false&strictPropertyInitialization=false&strictBindCallApply=false&noImplicitThis=false&noImplicitReturns=false&alwaysStrict=false&esModuleInterop=false&declaration=false&experimentalDecorators=false&emitDecoratorMetadata=false#code/JYOwLgpgTgZghgYwgAgApQPYCsILMgbwChlTkQ4BbCALmQGcwpQBzAbiIF8iiEMRGyMAAY6ACgCUyALwA+NHChhgcADYAedNlxh505JJnyxxMuSq1kAIkiMrXCR179BYAIzipchUpUatOHh6Bl7GpmQU1HQ2EHYANCQRFvTRtmD0Vgmcjjx8AvhgAEyeRj7KapqYgboyIaUmiaSRsamx6ZkOTnmuAMwl3qiK5f5VOsGG3g3ZbEA

It works if you specify the return type on the function itself like this
const t4 = (): Partial<Project> => ({     name: "test",     names: "tests", });
I updated your example

Sorry. I didn't get your point. Could you share playground?

I mean that it is possible to check the return value of the function in the way that we want by specifying the return type on the function declaration rather then on variable like you did in your example (strange that is doesn't work, though).
playground
Also, there's a similar package (not so full-featured as yours) called omnimock, that also allows to automatically get the mock by providing the class to mockInstance() function, and type-checking works correctly there. So I think it's possible to implememnt

Looks like we talk about different things. If you check mockInstance from omnimock it has a different signature: https://github.com/hmil/omnimock/blob/master/src/static-api.ts#L96-L100
it's static, and doesn't allow you to utilize injector when a complex mock needed.
I understand your what you want to have, but doesn't looks like it works in typescript without specifying types.
Your playground, doesn't solve the issue, it is the same as:
MockInstance(Project, (): Partial<Project> => ({
  name: 'this property does not exist',
  names: 'this property does not exist',
}));

As an option you can rely on the passed instance:
    MockInstance(Project, (instance) => {
      ngMocks.stub(instance, {
        name: 'this property does not exist',
        names: 'this property does not exist',
      });
    });

then it fails.

OK, I think using MockInstance(Project, (): Partial<Project> => ({ ... }) is an acceptable workaround, though specifying Partial<> return type seems not very DRY.
Thanks for the help.
It seems that MockInstance serves the different purpose, but what would then be the closest ng-mock analog to omnimock's 'mockInstance' for mocking the data types like Project?

OK, answering my own question after digging some more, I think it would be MockService, despite he name. Though it doesn't report 'un-backed' properties access (just returning undefined), I think it has the same purpose
Probably we can close this)

Ah, indeed, I forgot about MockServer, right, it might be a solution, props are undefined, because they are mock.
However, I would still suggest to use MockInstance(Class, 'method', 'implementation');, and in case of removed / renamed members, it will fail as expected.
Thank you for the investigation, closing it.


Not Working

*Will not allow me to login after all credentials are inserted: **

Logs:
11/13/2023, 3:32:43 PMWyze Smart Home BridgeWyzeERRORRequest failed: AxiosError: Request failed with status code 403
11/13/2023, 3:32:43 PMWyze Smart Home BridgeWyzeERRORResponse PerformRequest ([object Object]): {"URL":"https://support.wyze.com/hc/en-us/articles/16129834216731-Creating-an-API-Key"}
11/13/2023, 3:32:43 PMWyze Smart Home BridgeWyzeERRORError getting devices: AxiosError: Request failed with status code 403
11/13/2023, 3:32:46 PMmyQ BridgemyQERRORmyQ API error: Forbidden API call. This error is typically due to an offline or unavailable myQ device
Show the Homebridge logs here, remove any sensitive information.

Plugin Config:
Show your Homebridge config.json here, remove any sensitive information.

Screenshots:

Environment:
Hoobs

Plugin Version:
0.5.37
Homebridge Version:
Node.js Version:
NPM Version:
Operating System:

Can you paste your config?

Starting a couple weeks ago, I was getting a similar error.   Initially, I hadn't added an API key to my config (worked for months without one) so I tried that, and it fixed it.  I did have to remove/re-add the Wyze accessories to Apple Home.

@Peltier36 has this been resolved?

Should be resolved in latest release, and stale.


<script confuses Bikeshed.

Bikeshed became quite confused after I added an unclosed HTML tag inside backticks. That is, with the raw text:
  ... the string "`<script`" ...

various processing things broke down. For instance, https://www.w3.org/TR/2016/WD-CSP3-20160901/#iana-registry didn't get formatted correctly, and many sections after the unclosed element are missing paragraphs and other niceties.
I've worked around it for the moment by switching to <code><script, so not a huge deal, but it confused me for a bit.

Right. In order to prevent various text-mangling things from running inside of scripts or styles, I preemptively look for a handful of "raw" elements (script, style, xmp, pre, anything you've manually marked as raw) and turn off processing until I see the appropriate closing tag.
I could potentially only look for those elements at the start of a line, but I chose to go for the conservative solution and recognize them anywhere on the line.
That said, though, my actual regex is r"<({0})[ >]" - that is, it looks for either a space or a closing angle-bracket following the word. It shouldn't match a snippet like you provided. Can you point me to some source that is causing the problem?

Ah, I see, it was specifically line 3595 that was causing it, because that has a larger prefix of a <script> tag, enough to trip the regex.

Yeah ok, I'm not happy that this is happening. I'll see what I can do.

Hah, this is biting me hard trying to port the Bikeshed documentation to Bikeshed, since I use markdown-backticks to escape a bunch of <pre> and <script> elements.
I'm just gonna only recognizing them at the start of the line, there's no reason to format your code such that a <script> has significant content preceding it on the line.

Okay, should hopefully be fixed, but brain no work good right now. Tests pass, at least.


Security Service: Verify must be an Observable

Description
Some times the authentication must be evaluted with external resources and to improve this mechanism the actual behavior must be changed to observable.
Acceptance Criteria

[x] The authentication verification must be an observable and customizable.

The return of the Authentication can be a Promise, Observable or a not async type


[stable/fluentd] add support to deployment with scale

What this PR does / why we need it:
We would like to deploy the Fluentd as a Deployment and not as Statefulset
Since flush_at_shutdown param is supported, there is no risk of losing data in scaling down in the stateless deployment
Which issue this PR fixes
in the current state, fluentd will be stateful if one of the following has been set
.Values.autoscaling.enabled  OR Values.useStatefulSet
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x ] DCO signed
[x ] Chart Version bumped
[x ] Variables are documented in the README.md
[x ] Title of the PR starts with chart name (e.g. [stable/mychartname])

/assign @miouge1

/assign @hectorj2f

/assign @miouge1

/assign @hectorj2f

/ok-to-test

@sagile1 can you describe more in details your scenario?

@sagile1 can you describe more in details your scenario?

Hi @hectorj2f hectorj2f,
We would like to deploy the fluentd to K8S as deployment and not statefullset
In the current state, there is a condition the deploy the chart as statefullset if autoscale is enable.
we don't see any risk to use deployment + autoscale since fluentd has the ability to flush the buffer to s3 once it scale down (flush_at_shutdown param). so there is no data loss in this case. you can find more details in https://docs.fluentd.org/configuration/buffer-section

/assign @hectorj2f


@font-face declarations breaking

Using the new @tailwindcss/jit compiler seems to break font-face declarations added to base styles.
Using code like this in your CSS entry point:
@layer base {
  @font-face {
    font-family: "test";
    src: url("/fonts/test.woff2") format("woff2");
    font-weight: 400;
    font-style: normal;
    font-display: swap;
  }
  @font-face {
    font-family: "test";
    src: url("/fonts/test-italic.woff2") format("woff2");
    font-weight: 400;
    font-style: italic;
    font-display: swap;
  }
}

Results in the following CSS during dev and build:
  @font-face {
    font-family: "test";
    src: url("/fonts/test.woff2") format("woff2");
    font-weight: 400;
    font-style: normal;
    font-display: swap;
    font-family: "test";
    src: url("/fonts/test-italic.woff2") format("woff2");
    font-weight: 400;
    font-style: italic;
    font-display: swap;
  }

Without JIT, the font-face declarations are compiled to individual declarations and work as usual.
Versions:
"@tailwindcss/jit": "^0.1.0",
"tailwindcss": "^2.0.3",

Hitting the same thing where my css file is almost identical to the heroicons repo: https://github.com/tailwindlabs/heroicons.com/blob/master/src/css/main.css

Same here

It seems like @layer base isnt working. I had the same problem  but if I remove @layer base, it starts working again (but might add scoping issues...).

This is an issue for me as well

Same for me

Should be fixed in v0.1.1, thanks for reporting!


Typescript Support?

Is there any way to get typescript support? I've used the serverless-plugin-typescript before but it doesn't seem to work with serverless-kubeless.
https://github.com/prismagraphql/serverless-plugin-typescript
Anyone using this plugin with typescript or have recommendations?
Thanks,
James

Are you able to build before running sls deploy?
Then you can include the dist/ or whatever your compilerOptions.outDir is, and everything should work since it's javascript.


VM creation window collapses and console error appears sometimes when page refresh

Description
There is the supposition that sometimes when you open VM creation window something in code downloads faster or slower then it should.
The problem may be somewhere in this piece of code:
    VmCreationComponent.prototype.isSubmitButtonDisabled = function (isFormValid) {
        return (!isFormValid ||
            this.nameIsTaken() ||
            !this.vmCreationState.template ||
            !this.vmCreationState.serviceOffering.isAvailableByResources);
    };

Steps to reproduce
This bug is hard to catch. Frequent page refresh or slow internet may work
Actual result

Expected result
VM creation window opens with all its contents, no errors in the console
Connected feature ID
vm_creation

To discuss with PO: this problem occurs when a user sets SO in config which ID is incorrect.
We suggest to treat this case the following way:

VM creation dialog opens
An error message (red color) in SO section: "ID of SO that you set in config is invalid. Please change your configurations"
Create button is disabled

The previous comment was discussed with PO.
This issue covers the following requirements:
[REQ_VM_CREATE_024] If an invalid ID is specified in the default compute offering configuration, then it should be missing when creating a VM

Test to:
REQ_VM_CREATE_024


terraform destroy fails when IP is used for source NAT

Hi there,
Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.
Terraform Version
Terraform v0.11.11

provider.cloudstack v0.1.5

cloudstack_firewall.ip-01-fw: Destroying... (ID: 923fb5a0-c029-484b-b7c0-cf468494bf6a)
cloudstack_instance.vm-01: Destroying... (ID: b9249dca-3865-4c22-a9a4-bbae72a2839b)
cloudstack_instance.vm-02: Destroying... (ID: a1299ed4-e140-42e9-9f27-1cafc2d85a59)
cloudstack_egress_firewall.default: Destroying... (ID: 56c51f1b-cd01-43c5-b3a3-bf23816ae683)
cloudstack_firewall.ip-01-fw: Destruction complete after 7s
cloudstack_ipaddress.ip-01: Destroying... (ID: 923fb5a0-c029-484b-b7c0-cf468494bf6a)
cloudstack_instance.vm-01: Still destroying... (ID: b9249dca-3865-4c22-a9a4-bbae72a2839b, 10s elapsed)
cloudstack_egress_firewall.default: Still destroying... (ID: 56c51f1b-cd01-43c5-b3a3-bf23816ae683, 10s elapsed)
cloudstack_instance.vm-02: Still destroying... (ID: a1299ed4-e140-42e9-9f27-1cafc2d85a59, 10s elapsed)
cloudstack_egress_firewall.default: Destruction complete after 17s
cloudstack_instance.vm-02: Still destroying... (ID: a1299ed4-e140-42e9-9f27-1cafc2d85a59, 20s elapsed)
cloudstack_instance.vm-01: Still destroying... (ID: b9249dca-3865-4c22-a9a4-bbae72a2839b, 20s elapsed)
cloudstack_instance.vm-01: Still destroying... (ID: b9249dca-3865-4c22-a9a4-bbae72a2839b, 30s elapsed)
cloudstack_instance.vm-02: Still destroying... (ID: a1299ed4-e140-42e9-9f27-1cafc2d85a59, 30s elapsed)
cloudstack_instance.vm-02: Destruction complete after 37s
cloudstack_instance.vm-01: Still destroying... (ID: b9249dca-3865-4c22-a9a4-bbae72a2839b, 40s elapsed)
cloudstack_instance.vm-01: Destruction complete after 46s

Error: Error applying plan:

1 error(s) occurred:

* cloudstack_ipaddress.ip-01 (destroy): 1 error(s) occurred:

* cloudstack_ipaddress.ip-01: Error disassociating IP address 923fb5a0-c029-484b-b7c0-cf468494bf6a: Undefined error: {"errorcode":530,"errortext":"ip address is used for source nat purposes and can not be disassociated."}

The network offering uses the first IP that is created as source NAT. This seems to affect the destruction of this IP. Can this be resolved?

Hi @gurubert, thanks for opening this issue! I think the big question is how this is solved by CloudStack itself? So are you able to release/destroy the IP using the web UI or any API calls? As it seems to be a limitation of the current API aka CloudStack itself.
If I'm correct, with older versions of CloudStack a network with the source NAT service would always get an IP for the NAT service when you created the network. It looks like that is now changed which creates this problem.

Cloudstack seems to create this dependency internally.
I am only able to delete the whole network instance via the web UI.
I have no idea how to solve this for "terraform destroy".

Thanks for the update. I also thought about this and have an idea how we can solve this issue.
I’m working on updating some other bits as well (as a holiday project) so will add this to my list of things to fix/update.
I expect to be able to release a new version of the provider somewhere during the holidays.


Remove voaac deps

see http://ffmpeg.org/index.html#removing_external_aac_encoders

Requires FFmpeg v3

First FFmpeg 3 build on linux:
src/lib_media/encode/libav_encode.cpp: In constructor ‘Modules::Encode::LibavEncode::LibavEncode(Modules::Encode::LibavEncode::Type, const Modules::Encode::LibavEncodeParams&)’:
src/lib_media/encode/libav_encode.cpp:97:24: error: ‘PIX_FMT_YUV420P’ was not declared in this scope
    codecCtx->pix_fmt = PIX_FMT_YUV420P;
                        ^
src/lib_media/encode/libav_encode.cpp:99:24: error: ‘PIX_FMT_YUVJ420P’ was not declared in this scope
    codecCtx->pix_fmt = PIX_FMT_YUVJ420P;

First step in 2c6000d4eb30fea37d58e0863d09ae3900343b5d


suggestions about pull request

As previously discussed in https://github.com/oceanbase/oceanbase/discussions/60, we'd better use squash and merge to merge pull requests on GitHub.
Besides, the CLA check is required item and it's recommended to add this restriction to repository settings.

Thanks for your suggestion, we will following pull requests in the 'squash and merge' way.


Add support for marking repositories as important

This is for use in conjunction with the "filterusers" parameter, to make
PRs on some repositories always be shown, even if not opened by one of
the specified users.
This allows us to have a wall which lists any PRs on repositories
"owned" by our team, together with any PRs on other repositories made by
team members.

Thanks @edds!

:+1: this covers some more use cases.
The whole set of filter/important params are a bit hacky, and would be better set on a per-repo basis - but thats tricky to do via query params.


fix incorrect usage of Ordering::Relaxed

Motivation
Ordering is important. While Relaxed is atomic on its own, it can't sync cache between threads.
This PR fixes such incorrect usage

@JakkuSakura thanks again for the contribution, really appreciate it! Just a heads up, I've updated the PR with a commit to adjust the formatting.

gm


Add trailing newline to generated 15update-stamp

The apt_update resource generates a file at /etc/apt/apt.conf.d/15update-stamp that conflicts with the system-provided update-notifier-common package on Ubuntu. Because the file dropped by Chef differs from the system-provided file, apt-get requires user interaction to install update-notifier-common. This creates a problem if you're trying to install update-notifier-common via a Chef package resource... Adding a trailing newline makes the file identical to the one provided by update-notifier-common, and allows the package install to proceed without interaction.
A debatable shortcoming of my fix is that if someone installs update-notifier-common and then later removes it, /etc/apt/apt.conf.d/15update-stamp will be removed. However, within the apt_update update interval, the file will be re-created by Chef.

I see that the Travis build failed for my PR, but after looking at the build logs, I am unsure of why my change would cause it to fail. If someone can point me in the right direction, I'm happy to make any necessary changes to the PR.

The failure is actually unrelated. I kicked off that test again to see if it goes green now

@tas50 Is there anything else you guys need from me to initiate a code review, or will that happen in due time?

It just need a review from core maintainers in order to get merged. I'll see about making sure that happens tomorrow.


Crash when logging into Client - Missing Permission: MESSAGE_WRITE

I've recently placed this mod on my testout Minecraft server.  it completely crashes the server every time I log into it from client, and every time I check it gives me some missing permission: message_write in the logs and the terminal.  I've already given the bot writing permissions on Discord.  I'm stumped on what to to tbh.
crash-2019-09-18_14.35.01-server.txt

As say in the log your bot is missing permission. Perhaps have you enabled the wrong one ?
You can try to give the bot admin permission and try again. If you still have crash there is a problem with the mod. If you have no more issue you need to recheck your bot permission in your guild.
Denis

As say in the log your bot is missing permission. Perhaps have you enabled the wrong one ?
You can try to give the bot admin permission and try again. If you still have crash there is a problem with the mod. If you have no more issue you need to recheck your bot permission in your guild.
Denis

It now works after giving the bot admin privileges.  Before it never sent a message to the Discord server, now it does.    Though I am worried about giving any bot admin privileges, is there any specific ones that it needs?  Also, do I have to run multiple bots if I have multiple servers that needs to be integrated, or can I just run the plugin with the same credentials for the bot and use different channels to output?
Currently testing some more things out while the bot is working, thanks.

The bot need the read message, send/write messages and manage webhooks. It's the second one who is missing on the crash you send. You can add many server on the same bot but if you send message on the same channel the message send from the bot account by the first server will not be send to the second one and vice versa. While it's just an account you can connect multiple time at once. So there is no problep with different channels.


Add query parameters to the url

Fixes quarkusio/quarkusio.github.io#1917

Thanks! I didn't test, but I suppose you did, and this looks good to me. I'll let @ia3andy chime in if he wants to have a look.

Silent minority won't get a vote :) Merging.


Gladys products

Made changes to render products by tag:
The following routes now work:
products/women
products/men
products/kids
products/dogs

*** This pull request was meant for another repo!


fix broken bxnotes-couse links

missing "/" at the end of links, which gave 404

Thanks, and welcome to free-programming-books!

Thank you!