WIP: Proper highlighting for the Wolfram Language (Mathematica)

This is work-in-progress and not ready for merging, but I wanted to start a discussion about what is feasable/wanted for highlight.js. The Wolfram Language (WL) is one of those languages that are very different in some points. Most notably, it has close to 7000 built-in symbols and none of the usual "keywords" that other languages have. I rewrote the entire Mathematica (it's the same as "WL", don't ask, it's confusing) highlighter and here are the major changes:

The many ways how a number can be specified is now handled correctly
All built-in functions and so-called named-characters (a way to represent e.g. Alpha or Arrows) are matched correctly and updated for the most recent version of Mathematica
Instead of using lists of keywords for the built-in symbols, I created a trie to compress the regex. This basically shaves 30kB from the regex of the built-in symbols.

I'll attach an image of a more realistic test-case at the end, but first I'll have some questions:

Is it OK to use such a trie in favor of a human readable list of keywords?
Does anyone have experience regarding the performance of such a trie? In theory, it should be faster but I was never really able to measure this. On the other hand, I'm a JS noob and I'd like to keep it that way :)
Am I doing this right by using begin: and specifying the regex there? I mean, it works but what do I know?
Are there formatting guide-lines? Specifically, can I leave these long regex on one line?

Any tips are appreciated.

It seems Travis fails because now many languages are detected as Mathematica. Is there a way I can provide further meta-data that defines how to detect it correctly?

Instead of using lists of keywords for the built-in symbols, I created a trie to compress the regex. This basically shaves 30kB from the regex of the built-in symbols.

30kb measured how?  Raw size is irrelevant.  What matters (some) is Brotoli or gzip size.

Is it OK to use such a trie in favor of a human readable list of keywords?

Off the top of my head: no.  This is pretty much unmaintainable - unless there was a whole build system internal to Highlight.js that took a human readable list, built the trie by hand, etc... and I don't think adding such a complex beast to core (just to support a single large language) doesn't make sense.   Out of curiosity what does that code to build the regex look like?
Although if you wanted to maintain your own 3rd party grammar module then you could do it however you wanted.

Does anyone have experience regarding the performance of such a trie? In theory, it should be faster but I was never really able to measure this. On the other hand, I'm a JS noob and I'd like to keep it that way :)

No idea, that's something we'd likely want some real numbers on before even considering something like this for core (if we were ever going to).

Am I doing this right by using begin: and specifying the regex there? I mean, it works but what do I know?
Are there formatting guide-lines? Specifically, can I leave these long regex on one line?

begin works if there is a single thing to match... but I dropped a comment on NUMBER already... generally if a regex can be split into smaller pieces (for readability/maintainable) it should be... so if there are 8 variants of numbers they each should really have their own regex using variants rather than one complex | that's impossible to read.
There are helpers in regex.js for some common things...

It seems Travis fails because now many languages are detected as Mathematica. Is there a way I can provide further meta-data that defines how to detect it correctly?

You have look at what it's matching and tone down the match count or the relevance... relevance: 10 is probably wrong...

@joshgoebel Thanks for your very thorough replies. Let me answer some of your comments:

30kb measured how? Raw size is irrelevant. What matters (some) is Brotoli or gzip size.

Gzipping the trie regex results in 43kB while the list of keywords would be 47kB. Not sure if this difference justifies the usage of a trie.

Off the top of my head: no. This is pretty much unmaintainable - unless there was a whole build system internal to Highlight.js that took a human readable list, built the trie by hand, etc...

The human-readable list is not going to be maintainable either. No one is editing  6659 keywords by hand and for all highlighters I've worked on, the list was always created automatically through Mathematica. Everything else required for creating the regex is small, open-source and well documented. I've put the exact commands used in the comments. It only relies on a package of mine that I use for the Wolfram Language IntelliJ plugin, where the code for creating the trie is written in Kotlin.
The gist is that you will need Mathematica either way to maintain the list of keywords, and the rest is open-source.

Although if you wanted to maintain your own 3rd party grammar module then you could do it however you wanted.

The reason I'm working on it is StackOverflow/StackExchange :)
I've written the google-prettify support for Mathematica that was used there but now we're moving to highlight.js.
The goal is bring the Mathematica highlighter to a reasonable state to get it eventually merged to StackOverflow.

but I dropped a comment on NUMBER already... generally if a regex can be split into smaller pieces (for readability/maintainable) it should be

I understand that. However, for Mathematica we're not really having different "variants". It's more like that you can prepend or append different things, e.g. if you have 123 and you can say it should be in base 8 and you get 8^^123. My major point however is that I linked a document in the comment above the NUMBER regex that goes into great detail about numbers. Everyone who's considering to work on this needs to read it anyway and it carefully lays out why and how things are matched. But I could try to split the regex in more readable chunks.

You have look at what it's matching and tone down the match count or the relevance... relevance: 10 is probably wrong...

Alright, I'll check the API.
Question:
How should I move on regarding the large list of keywords? Should I use a simple list or the trie regex? I'd also try bring some more people to this PR to get different opinions because in the end, it should be nice for the users. @CarlQLange already agreed to discuss matters.

Gzipping the trie regex results in 43kB while the list of keywords would be 47kB. Not sure if this difference justifies the usage of a trie.

Yeah for such a tiny difference we definitely want a clean inline list of keywords.

The human-readable list is not going to be maintainable either. No one is editing 6659 keywords by hand and for all highlighters I've worked on, the list was always created automatically through Mathematica.

Typically for longer lists (that are easily generated by the host language or script) we'll inline that code in the grammar itself (as a comment - usually these are only a few short lines of code)... so that the keywords are still listed and readable in the source, but maintainers can easily run the snippet themselves, generate a new list, copy and paste that into the file.  Then we have clear readable diffs for what has changed from one release to another.

The reason I'm working on it is StackOverflow/StackExchange :)

Nothing prevents them from using 3rd party grammar modules. :-)

The goal is bring the Mathematica highlighter to a reasonable state to get it eventually merged to StackOverflow.

I don't know what this means... doesn't it automatically get included with the switch to Highlight.js... just it's being "loaded as needed" rather than bundled - which makes sense if most sites aren't going to use it (only the Mathematica Stack). What am I missing here?  I mean I obviously get the desire to improve it... but you lost me with "merged".

I understand that. However, for Mathematica we're not really having different "variants". ... But I could try to split the regex in more readable chunks.

If there aren't truly variants the yes we'd split it up into more easy to read (and maintain) chunks so that reading it at a glance it's obvious what the component parts are, etc... rather than just a wall of regex as it stands now.

How should I move on regarding the large list of keywords? Should I use a simple list or the trie regex? I'd also try bring some more people to this PR to get different opinions because in the end, it should be nice for the users. @CarlQLange already agreed to discuss matters.

Simple list, and it should originally be an array, one word per line (for maintainability/git diffs/etc)... then as necessarily you can String#join it... it's going to be long but should be easier to see what changes in the future.  Hopefully the process to generate the list can be documented in the comments as done with other languages and mentioned above.

Am I doing this right by using begin: and specifying the regex there? I mean, it works but what do I know?
Are there formatting guide-lines? Specifically, can I leave these long regex on one line?

And once we fall back to using a simple list you should be able to use the built in keyword functionality (as the grammar uses now) instead of needing to use a complex regex at all....
There is an eslint file...

Great! Then I'll know how to move and hope I'll find some time.

I don't know what this means... doesn't it automatically get included with the switch to Highlight.js

StackOverflow is still in the process of deciding how regularly they will update their version of highlight.js. The most upvoted user-wish on the official announcement however is that they do this more regularly. So I guess they will pull new versions of highlight.js to their side. I don't know specifics but here is the comment of the mod regarding the question "can the highlighter be updated more often?":

This is a good question. I'll bring it up internally to formally decide as to what team will be responsible for keeping this up to date as well as how we're going to ensure that we're alerted when a new version comes out. As it currently stands, I have a few OSS side projects relying on highlightjs that get weekly notifications when dependency updates hit npm, so at the very least, I'll be aware of them. Thanks for bringing this to my top of mind. This will be very important moving forward, because, as you mentioned, highlightjs is being actively maintained.

The only thing I do know is that SO will not provide Mathematica highlighting for the whole network but only for our StackExchange site because even in google-prettiy days, our highlighter was huge compared to others. Here is the semi-official statement about this.

And once we fall back to using a simple list you should be able to use the built in keyword functionality

Yep, I got that. Use an array. Each keyword a line for diffing and joining it to a string in the code.
Thanks again, I really appreciate your input.

@joshgoebel Let me try to explain the high-level problem that led to the awkward begin matching rules you commented on. In Mathematica, we have two types of symbols

Usual symbol names: Start with [a-zA-Z$] and can be followed by [a-zA-Z0-9$]. Note the missing _ in the regex which is an operator in Mathematica. Therefore, the usual word boundary \b cannot be used since it, AFAIK, allows _ to be part of the word.
Special characters like α which are represented as \[Alpha]. So it starts with a \[ and ends with a ] and in between, you have the symbols from 1. Examples are \[Gamma], \[VeryThinSpace], \[ScriptCapitalA], .. you get the idea.

These two forms can be mixed. So you can have a variable myAwesome\[Gamma] and it's one single symbol. Some of the forms from 2. are actually operators and e.g. a\[ImplicitPlus]b is really just a sum of a and b. But that doesn't concern the highlighter which can't solve the problem anyway without parsing.
So what I want to have is:

All symbols that are possibly a mix of form 1 and 2 should be highlighted as symbol/variable
All symbols that exactly match one of the keywords should be highlighted as such.

I implemented all your suggestions on a separate branch and, for the love of God, it works. So in particular, I put the whole regex in the begin section.
One thing I don't understand: I still need the $pattern here and the reason is that otherwise my keywords that start with a $ like $RecursionLimit would not be highlighted like here:

Do you have an idea, why this happens? I know that $ is a special char in regexes but even escaping them (several times) in the list of keywords did not help yesterday.
Your other question was concerning the usage of strong. This is for such a case:

So the parameter x1 has a "type" of Integer which is a keyword and it should be highlighted as such. When I use the keyword class, it would also set the color which is not what we want in this situation. Do you have a better way of expressing this without introducing another style class?

I implemented all your suggestions on a separate branch and, for the love of God, it works.

Glad to hear it. :)

One thing I don't understand: I still need the $pattern here and the reason is that otherwise my keywords that start with a $ like $RecursionLimit would not be highlighted like here:

The default keyword pattern doesn't include $ I don't think and pattern isn't magically recursive... so anywhere you have a keyword list you're going to need to pass $pattern also.  Keywords are found by FIRST executing a match with $pattern and THEN seeing if any of those matches are in the keyword list.  So that $pattern is correct very important for languages with special characters in their keywords.

I know that $ is a special char in regexes but even escaping them (several times) in the list of keywords did not help yesterday.

Lists of keywords (as passed to keywords) are plain text.. the only "magic" is relevancy: a|0 meaning keyword a has a relevancy of 0.  There is no regex going on in normal keyword lists (other than the special $pattern key).

So the parameter x1 has a "type" of Integer which is a keyword and it should be highlighted as such. When I use the keyword class, it would also set the color which is not what we want in this situation. Do you have a better way of expressing this without introducing another style class?

I know what you're getting at but I have a very hard time even parsing this - because the parsing step is VERY different from the highlighting.

has a "type" of Integer which is a keyword and it should be highlighted as such.

Right - and it would be, and if a theme says "keywords" are orange, then it would be orange.  That means some themes MIGHT be friendlier to some languages than other themes... simply because of language semantics.  Since we much support an arbitrary number of themes (including user defined ones) the only way we can solve this properly is with proper semantic tags - and then themes being aware of them.
It's possible what's needed here semantically is a ".arguments.keyword" class (we don't support that yet)... but themes can already do this via the simple nesting that we already provide within match blocks.  I think perhaps Stack Overflow wants to add a rule like .hljs-mathematica .arguments .keyword (or something) to their custom theme that applies font-weight:bold.
Ie, this is a theme problem, not a grammar problem.

The closest comparison (I can think of) would be to Fortran where we have highlighting like:
6.666666666666666_DBL

The type simply being part of the number, not trying to a apply a different style to it.

I think perhaps Stack Overflow wants to add a rule like .hljs-mathematica .arguments .keyword (or something) to their custom theme that applies font-weight:bold.

Of course that only solves your problem with Stack Overflow... if the general parsing looks "ugly/wrong" for most themes, then that's a separate issue and probably means the grammar may have to be artificially crippled to better fit with the larger world of themes.
I'd suggest trying:

Don't highlight these keywords at all (let the whole argument be green - in your example)
Don't highlight the _ at all (to provide separation) and then just let the name and the type be highlighted as the theme desires.

OK, you convinced me. I'm opting for this solution

Don't highlight these keywords at all (let the whole argument be green - in your example)

because the underscore must be green. You know how users are.

OK, you convinced me. I'm opting for this solution

Don't highlight these keywords at all (let the whole argument be green - in your example)

because the underscore must be green. You know how users are.

This is the option I would go for, personally. I'd also note that inside the front-end the pattern spec (e.g. the Integer) is displayed the same for known symbols as for unassigned ones.

There might also need to be a larger discussion here if you're inventing new css classes that are not documented in css-classes-reference... generally that's not something core grammars do.  For example (upon taking a closer look) I have no idea what brace and pattern are... I'm not opposed to figuring something out here, but we can't just add them randomly without looking at the bigger picture (whether using an existing css class would not be preferable to support existing themes).
We may have to fall back to generics and then look at expanding them along with the other issue relating to more complex grammar support.
To start could you make a list of what any custom "classes" you have are, what they are (short explanation) and the closest thing that they would match with in our existing set of semantic classes?

because the underscore must be green.

Even this type of talking/thinking is problematic.  :-) For Stack overflow it may be green (if your ultimately in charge of the theming)... but it will definitely not be green as a general rule... it'll be whatever color the theme decides arguments are, etc.

Don't highlight these keywords at all (let the whole argument be green - in your example)

I'm fine with that for now.  Though it's possible someone raises an issue in the future and it gets changed (and I would probably support that)... that's why if a site wants a very specific look it's best that the grammar handles the semantics and the site handles the theming.  Ie, it might be more "future proof" for the grammar to flag it as what it is and let site owners decide how they want to theme it...  but I'm happy revisiting that on a different day. :)
I just tried it as "keyword" with several themes and it looked a bit strange, but not terrible... but also hard to say since I already find the code very strange and weird to begin with. :-)  If I was more familiar with Mathematica I might be a stronger opinion here. :-)

because the underscore must be green.

Even this type of talking/thinking is problematic. :-) For Stack overflow it may be green (if your ultimately in charge of the theming)... but it will definitely not be green as a general rule... it'll be whatever color the theme decides arguments are, etc.

We understand that, of course, but 90% of Mathematica users don't. It's a weird programming community composed of the usual SWE types, but also a majority of people who use it as a fancy graphing program and who only ever see it in Wolfram's front-end. This means that to the best of our ability we want to get Stack Overflow to display it the way those users would see it so that we can have a maximally useful forum for that.
If someone somewhere else decides to change the theme, who cares, but we want to make it possible for it to look right if we can convince SO of the utility.

FYI: I type a lot because I'm verbose and try to be thorough, it isn't an attempt to overwhelm with more words than you. :-)  I've been told sometimes it comes off that way.

This means that to the best of our ability we want to get Stack Overflow to display it the way those users would see it so that we can have a maximally useful forum for that.

Sure, I think I understand the GOAL. :-)  It just doesn't fit with HLJS as a generic highlighter for 200+ languages... to get what you want really involves controlling the grammar and theme tightly.

If someone somewhere else decides to change the theme, who cares, but we want to make it possible for it to look right if we can convince SO of the utility.

In this case perhaps you should consider #2 then:

The grammar is precise.
StackOverflow's rendering is "spot on" (via your control of the theme)

And if someone finds that it looks "weird" with other themes, then we revisit THAT in the future. :)

Don't highlight these keywords at all (let the whole argument be green - in your example)

I'm fine with that for now. Though it's possible someone raises an issue in the future and it gets changed (and I would probably support that)... that's why if a site wants a very specific look it's best that the grammar handles the semantics and the site handles the theming. Ie, it might be more "future proof" for the grammar to flag it as what it is and let site owners decide how they want to theme it... but I'm happy revisiting that on a different day. :)
I just tried it as "keyword" with several themes and it looked a bit strange, but not terrible... but also hard to say since I already find the code very strange and weird to begin with. :-) If I was more familiar with Mathematica I might be a stronger opinion here. :-)

BTW from a pure semantic perspective when we have an expression like
myFunc[i_, int_Integer, r_Real, q_?(Internal`RealValuedNumericQ)]:=...;

that int_Integer actually parses out like
Pattern[int, Blank[Integer]]

but and so the _ should be at minimum attached to  the Integer argument, i.e. they should be a semantic unit. Most Mathematica users, though, would likely argue that the entire expression should be a semantic unit named pattern, since _Integer, even though it's a semantic unit, is never styled.

and so the _ should be at minimum attached to the Integer argument, i.e. they should be a semantic unit. Most Mathematica users, though, would likely argue that the entire expression should be a semantic unit named pattern, since _Integer, even though it's a semantic unit, is never styled.

Your lost me here. I don't know enough to say if the _ should be attached to the name or the type... (if they were going to be grouped separately)... so I'd probably just defer to you.

and so the _ should be at minimum attached to the Integer argument, i.e. they should be a semantic unit. Most Mathematica users, though, would likely argue that the entire expression should be a semantic unit named pattern, since _Integer, even though it's a semantic unit, is never styled.

Your lost me here. I don't know enough to say if the _ should be attached to the name or the type... (if they were going to be grouped separately)... so I'd probably just defer to you.

This is one of the issues with using a not-formally-specified language... Thanks for all of the help, by the way. Once this thing is merged in I'm also going look into moving to using highlight.js to highlight my own blog content, and so that is definitely a case where I'd like to be able to make things look "right".

hence in the spirit of specifying a grammar should be contained in a single pattern class.

Are we mixing discussions now?  The problem with "pattern" (explicitly) is that it's not a CSS style we have... it will get no highlighting... and hence serve no purpose (we don't/shouldn't care about semantics UNLESS they affect styling).  So hence the grammar containing "pattern" serves no purpose.
If the name only exists for debugging then we should find a different way of encoding it...
We need to find a semantically equivalent existing CSS style, or simply not use a class at all.

@joshgoebel

For example (upon taking a closer look) I have no idea what brace and pattern are...

They are my testing classes. I will replace them with the ones I find in default.css. In the past we handled this as follows: Let's say we want the xxx_YYY to be displayed differently, then I just use, e.g. selector-attr which is the same as variable and symbol in the default style. Then it looks reasonable (though not fancy) and StackOverflow can just adjust the CSS with selector-attr defined differently to provide the exact coloring.
Btw, this is also the reason why I have matched operators and braces separately. Not because I want them styled differently for highlight.js, but to give StackOverflow and other users the ability to style them. So here is the list of context information for the separate classes:

PATTERNS_AND_SLOTS: Patterns are things like "let x be an integer" (x_Integer) or "xs are several real numbers" (xs__Real). They appear in the arguments of function definitions and should as a fallback be handled like normal variables. Slots are the positional arguments of anonymous functions. They look like # or ##[3]. Again, their fallback could be normal symbols.
BRACES are just normal code, however, since Mathematica is so heavy on braces, they probably will be styled differently
OPERATORS are just the normal operators and in most cases they are just styled as usual code... in "most" cases

The rest of the rules fit perfectly into the usual number, keyword, string, ... scheme.

hence in the spirit of specifying a grammar should be contained in a single pattern class.

Are we mixing discussions now? The problem with "pattern" (explicitly) is that it's not a CSS style we have... it will get no highlighting... and hence serve no purpose (we don't/shouldn't care about semantics UNLESS they affect styling). So hence the grammar containing "pattern" (or "brace") serves no purpose.
If the name only exists for debugging then we should find a different way of encoding it...
We need to find a semantically equivalent existing CSS style, or simply not use a class at all.

Where's the list of existing CSS styles so that we can look through them to try to specify one? I'd say the semantically closest thing is some kind of arguments class. My semantic/styling breakdown of this would be
int_Integer (* The entire object should be highlighted as a function argument. A more flexible system would tag the subparts, but I think highlight.js is supposed to be fast-over-flexible? *)
int:(_Integer).. (* Only the `int` part should be highlighted as a function argument as the latter parts are a pure pattern spec *)
int_Integer?(GreaterThan[3]) (* The `int_Integer` should be highlighted as a function argument, but highlighting should stop at the `?` *)

Let us know which tag is closest to a function argument tag.

https://highlightjs.readthedocs.io/en/latest/css-classes-reference.html

@joshgoebel in that list when you have stuff like the ReasonML subsection are all of those names free to use (e.g. operator and pattern-match?)

which is the same as variable and symbol in the default style. Then it looks reasonable

This thinking scares me... please stop thinking about it visually.  :-)   Perhaps I'm reading too much into what your saying but look at the CSS (any CSS) to see what things are colored like other things is often the path to the dark side.  Most languages should not be using selector-attr as that has a very specific meaning... it's not just an extra name laying around for a grammar to "borrow" because "it's the right color usually". :-)
For most languages this should be done entirely semantically.  Ie, you should be able to write a good grammar WITHOUT any CSS (or visual rendering) at all.  Working backwards ("this is how i want it to look") usually results in poor results in the aggregate - but really good results for a single theme.  Of course this approach isn't perfect either - and the best results can only really be achieved by pairing themes/languages tighter than we do - but we do OK.

OPERATORS - thankfully have operator though it's seldom used (I'm happy to see that change).  They problem (as usual) is theme support. I feel like this is the easy one, just use operator
PATTERNS_AND_SLOTS - we have symbol and variable.  And of course we have the wrapping block params... in some languages individual params don't even need to be identified, just the whole area falls under params)...
BRACES - So what would an equivalent be in another language?  Javascript has "braces" for code blocks, but usually we don't highlight them....

in that list when you have stuff like the ReasonML subsection are all of those names free to use (e.g. operator and pattern-match?)

I should make operator generic (and yes please use it)... but I want to start borrowing names just cause they're "similar" (or similar colors)... does Mathematica have "patterns" or "pattern matches"?
I really need to review the ReasonML stuff and see what the heck those things are - that's before my time. :)

in that list when you have stuff like the ReasonML subsection are all of those names free to use (e.g. operator and pattern-match?)

I should make operator generic (and yes please use it)... but I want to start borrowing names just cause they're "similar" (or similar colors)... does Mathematica have "patterns" or "pattern matches"?
I really need to review the ReasonML stuff and see what the heck those things are - that's before my time. :)

We've got both. Basically _Integer is a pattern specification, i.e. the thing to match. On the other hand a_Integer is a full Pattern where something that matches _Integer gets bound to a. I've been looking into other languages, since many, many functional languages use pattern matching, and it seems like there's no agreement on how even other languages opt to highlight their patterns. So absent any kind of standard, we'd like to be able to highlight Mathematica code (not make it visually identical to the front-end mind you, just highlight it) in a way that makes it easiest for people who actually use the language to read it.
Personally, I think from a semantic perspective there should be support in the highlight.js classes for things like pattern, pattern-spec, and pattern-match. I am no Rust developer, but I think that adding good support for these kinds of things would allow for better highlighting of stuff like the code samples here: https://doc.rust-lang.org/book/ch18-03-pattern-syntax.html
And in the absence of that, I'd say the semantically closest thing to the int in int:_Integer is a pattern-match.

(at least based on the highlighting here: https://reasonml.github.io/docs/en/pattern-matching)

Which is using HLJS, lol.

I really need to review the ReasonML stuff and see what the heck those things are - that's before my time

Yes, I just saw that while everything should be generic for no particular language in mind, there are general-purpose classes that are missing. Why is there no general "operator" and it has to be borrowed from an obscure ReaonML language?

Javascript has "braces" for code blocks, but usually we don't highlight them....

Well, why do you think of JavaScript when it comes to highlighting? :smile:
It's just one of many languages. So while JS usually don't style braces, other languages might very well do it.

Most languages should not be using selector-attr as that has a very specific meaning..

Yes, for exactly one particular language: CSS. The whole table of possible classes is pretty much written with JS, XML, CSS, and ReasonML in mind. Besides borrowing classes for the moment, I don't see a good way to solve this. It would be great if there were additional general-purpose classes:

operator, punctuation, parenthesis which should default to normal code
pattern

Would it be OK if I use

operator from ReasonML for the operators
pattern-match from ReasonML for what I call patterns and slots
variable from Tags, ... for all symbols
built_in for built-in symbols

That leaves only the problem with the braces. I could simply use a class parenthesis or brace and it would simply not be highlighted for the moment except with an adjusted CSS. Would that one exception be possible?

Which is using HLJS, lol. ReasonML basically slipped in a bunch of things we don't really support (though perhaps should)...

I think there are likely many functional languages for which highlight.js likely doesn't provide phenomenal coverage, since it looks like it was designed with mainly C-flavored languages in mind? I'm looking at the ones under the demo: https://highlightjs.org/static/demo/ and the functional languages seem to be mostly un-highlighted, which is maybe fine for CS types, but would cause huge headaches for your average scientist who just wants to be able to normally read the Mathematica code that someone posted online.
What is the chance that highlight.js provides better semantic coverage in its classes for pattern matching or sub-expressions following operators like :: or .? In Mathematica we've got Name::msg where msg is semantically very, very different from Name (it's much more like a string) and obviously C++ has std::vector and friends. My IDE provides different highlighting for the std and vector in that case.

Why is there no general "operator" and it has to be borrowed from an obscure ReaonML language?
So while JS usually don't style braces, other languages might very well do it.

Because 'by default' we don't highlight operators at all (on purpose).  In the past it's been more of a "policy" but that's no longer true, and I'd like to see that change...  Braces would fall under the previous "we do not highlight operators/punctuation" policy, so it's not really been considered before.

Yes, for exactly one particular language: CSS.

HAHAHA.  CSS is a pretty "particular" type of language, but of course you could apply that to anything...

The whole table of possible classes is pretty much written with JS, XML, CSS, and ReasonML in mind.

Yet MOSTLY it works surprisingly well for general support for 200+ languages...  Most of the "common" things apply to many many object oriented languages - not just Javascript.

It would be great if there were additional general-purpose classes:

operator : done
punctuation: What would this be if not operators?
parenthesis: hmmmm...
pattern: How would you describe this in the CSS docs?

And now we're starting to get into a more complex issue in that we purposely have tried to do a certain 'level' of highlighting complexity... we dot NOT want to be a full parser for many languages.  So that's influenced what we highlight.  Highlighting operators correctly in many languages is annoying the way we currently do it... and highlighting some of these others things can start to get into things we don't want to actually encourage.
We strive to be more complex than say Prism, but we have our limits... at the point I can no longer read and maintain a grammar easily it's become too complex... that's why some of our grammars are quite "fuzzy" instead of precise... because that last 10-20% of work would be HARD AND make the grammars very hard to maintain.  Just at a glance the ReasonML stuff is very complex compared to a lot of our other grammars.

operator from ReasonML for the operators - I already have a PR to move this into the generic list.
pattern-match from ReasonML for what I call patterns and slots: This needs more thought.
meta-string for usage messages.  - Not sure what a "usage" method is but meta or meta-string sounds right OTTOMH.
variable from Tags - Yes, I'm also moving this as we support variables as a generic item already.
built_in for built-in symbols - Sure that's what it's for.

OK, let's tackle this tomorrow. It's already 4:30am here and I need to be up and running in a few hours.

What is the chance that highlight.js provides better semantic coverage in its classes for pattern matching or sub-expressions following operators like :: or .? In Mathematica we've got Name::msg where msg is semantically very, very different from Name (it's much more like a string) and obviously C++ has std::vector and friends. My IDE provides different highlighting for the std and vector in that case.

This can get dangerous close into parsing though... I'd have to see the rules to decide if they are "too complex"... it's no good if some genius provides us an amazing (and unreadable/unmaintainable) grammar that is spectacular but then moves onto something else and people file bugs against it that no one is capable of fixing... we'd rather just say "we don't do that, sorry" rather than over promising and under delivering.
See #2521 and #2500.  I'm open to stuff like this, but someone needs to show me it can be done in a maintainable fashion without adding too much complexity to the grammars.

We've got both. Basically _Integer is a pattern specification

Sure.

On the other hand a_Integer is a full Pattern where something that matches _Integer gets bound to a.

To me this also sounds very much like static type declaration. :-)  As soon as you say "well i can have different signatures of the function that take different types" is that pattern matching?  IE, Nim does this very well, but I don't think anyone calls it pattern matching there... the compiler is just finding the correct method signature at compile time based on the types involved.

I've been looking into other languages, since many, many functional languages use pattern matching, and it seems like there's no agreement on how even other languages opt to highlight their patterns.

Yeah, I think this is the problem... we say "ok, we added pattern-match as a supported CSS name" and then 50 languages go and define it differently which defeats the purpose of trying to style it consistently across multiple languages. Having to measure everything we do against almost 200 languages and 100+ themes adds a lot of complexity you have to take into account for things like this.
If there can be no consistency then I would probably opt to not highlight - rather than highlight poorly.

myFunc[i_, int_Integer, r_Real, q_?(Internal`RealValuedNumericQ)]:=...;

So to me (at a glance this looks like).

A function dispatch (or definition?)
A list of arguments/params with types (not sure what the last one is doing)

So this might mean:

function
params
type, built_in, keyword (for the types)

So something like "int_Integer" would get split into name:int, type:Integer pieces (irregardless of what we name them).
When I think of pattern matching I usually thikn of elixir and something like:
defmodule Speak do
  def greet({:formal, first_name, last_name}) do
    IO.puts("Greetings #{first_name} #{last_name}!")
  end

  def greet({:informal, first_name, last_name}) do
    IO.puts("What’s up #{first_name} #{last_name}!")
  end
end

Now there is pattern matching happening there (to decide which function to dispatch), but to me it seems "irrelevant" in the highlighting of said function... what I see are:

def keyword
function name title
function params params
tuple in {} (unhighlighted)
and a list of argument names (which typically we wouldn't highlight other than params - although you easily could)

This seems more relevant in something like ReasonML:
  switch (x) {
  | A => "a"
  | B(0) => "b_zero"
  | B(k) => "b_" ++ string_of_int(k)
  };

Though again, which part is the "pattern matching", and is that really a "thing" in an of itself or just a "grouping"... to me it looks like the A/B are "classes/modules/type" of some sort and the 0 and k perhaps values?  I'm not sure we need to add the explicit "pattern-match" designation to anything here.
So again, the semantics are important, but only towards the service of the ultimately goal of highlighting/colorizing.

We've got both. Basically _Integer is a pattern specification

Sure.

On the other hand a_Integer is a full Pattern where something that matches _Integer gets bound to a.

To me this also sounds very much like static type declaration. :-) As soon as you say "well i can have different signatures of the function that take different types" is that pattern matching? IE, Nim does this very well, but I don't think anyone calls it pattern matching there... the compiler is just finding the correct method signature at compile time based on the types involved.

It's a lot like one...and yet a static type declaration can't check that its argument is also greater than 3, nor is it messed up if you accidentally pass it an object that looks like Intarger[3] (which wouldn't be an error in Mathematica). For better or for worse, Mathematica isn't typed, really.  All of that is just to say this really is a pattern, not a type declaration. We could piggy back off of what Nim calls their type declarations to make the highlight.js code base more consistent, but that'd be conceptually wrong and people in the future would look back and be like "WTF were they thinking?".

If there can be no consistency then I would probably opt to not highlight - rather than highlight poorly.

Cool. But we're not advocating for slipshod highlighting. We're trying to add the appropriate semantic tag to a semantic structure in a language...like these are patterns. The pattern matcher that powers the language literally sees them as a structure that looks like Pattern[name, spec]. Calling them a type declaration is wrong and has all sorts of connotations that the language doesn't support. Other non-object-oriented, casually-typed languages surely share some of these features, so why not give people the semantically appropriate classes to use? (e.g. pattern or pattern-spec to go with pattern-match). If you have the classes available, people will use them. If not, people will have to try to hack around that limitation like we clearly have to.

Yeah, I think this is the problem... we say "ok, we added pattern-match as a supported CSS name" and then 50 languages go and define it differently which defeats the purpose of trying to style it consistently across multiple languages. Having to measure everything we do against almost 200 languages and 100+ themes adds a lot of complexity you have to take into account for things like this.

I don't quite see why you think it is possible to style stuff perfectly consistently across languages? There's no Fundamental Theorem of Programming Languages that states that you can style everything consistently. If you don't style the key features of a language, you're effectively styling it inconsistently with the languages that you initially thought of when writing your code... OTOH, I do think that many languages have similar flavors of pattern matching and that can be supported in an internally consistent manner. Again, I'm not a CSE prof so I can't sit down and formally prove that for you, but based on my brief survey of pattern matching in languages it seems to be the case.

So now to bring this full circle when you have a rule like PATTERNS_AND_SLOTS, I'm not sure you need pattern or pattern-match... just break down the components...

name
type (to match)

Calling the whole this grouping pattern confuses me personally since it seems like type is the "pattern" part... "makes sure you find a match with Integer, store that match in int. Int is just the symbol/variable name.  Perhaps I'm missing something.
I should have looked more closely before pointing you to CSS Class reference (though at the time I didn't know you'd latch onto the ReasonML stuff)... using "pattern-match" solves nothing because none of the ReasonML "classes" are actually supported by any of our actual themes...  so essentially all those ReasonML classes (other than "operator") are "off limits" because none of our themes support them.  ReasonML shouldn't have been allowed to use them either without first solving the larger issues here.

It would be great if there were additional general-purpose classes:

I do agree, but the 100 existing themes is the problem...   We can't just add new major classes.  We have to either upgrade all 100 themes for every new added class, or provide some "fallback" option (aliasing, etc).  We can sneak in more "minor" classes like "operator" because we've NEVER highlighted operators in the past... so just starting to support them sort of "breaks nothing"... Before: No one saw operators highlighted.  After: 99% of people don't see operators highlighted, unless they're using a new theme.  I'm not 100% happy with that either, but it's less problematic.
It's much harder though when you say:

well, in my language it's not a "variable" it's a "placeholder/slot"

That's great, but our themes have NO idea what a placeholder/slot is... so we're force to just say it's variable - since that's the closest semantic match.  And at a high level this type of "consistency" also helps the core team [attempt to] maintain 190 different languages.
I feel like one of the issues here might be something like:

Stack Overflow wants to highlight brackets {} in Mathematica even if NO other Highlight.js users will ever benefit from this functionality.

I see two routes to handle that scenario:

Stack Overflow uses a 3rd party grammar module that can do whatever it wants (ie full control of grammar + style)
We find a way to make sure all Highlight.js users can benefit from the functionality - which is why issues like #2500 and #2521 exist.

Can't keep going on this and I'll let @halirutan decide what he thinks is best and obviously we don't have any control over what actually happens. I just want to say that maybe giving you the int_Integer example is confusing things? So here are four common cases where we use patterns:
fun[vec:{(_?(Internal`RealValuedNumericQ))..}]:=...; (* function def *)
MatchQ[<|a -> 1, b -> 2, c -> 3|>, KeyValuePattern[b -> 2]] (* pattern check *)
someXML /. XMLElement[__, a_]:>Hash[a] (* subpart replacement  *)
Replace[list, {a_, b_}:> a+b ] (* classic(?) destructuring *)
Cases[Plot[Sin[x], {x, 0, 2 Pi}], l_Line?(Length[#]>100&):>l, Infinity] (* finding all things matching the pattern *)

You'll also note that GitHub does a pretty terrible job of highlighting all of that. We're hopeful we can get something at least a little bit better.

You'll also note that GitHub does a pretty terrible job of highlighting all of that.

Might be helpful to post the "canonical" highlighting from the software if you can.

Sorry lots of other things happening.  Regarding our discussion about in or outside of core I don't think that can actually be decided at a single point in time.

If you want the freedom to do whatever you want with the CSS/naming/theming, then make this a 3rd party module... but if while doing that you add a bunch of things that aren't compatible with our existing themes I don't think we could "replace" it with the 3rd party module (ie remove it from core and say "use this 3rd party module instead")... since the outcome would be very poor for many uses with many themes.
And if you make it fully compatible with the core library and existing themes, then there is no reason it really needs to be external at all - unless you just had very, very different ideas about code style, maintenance, etc...  ie, it'd be easiest to leave it in core and let everyone have it "for free" wherever the core library is used.

So it seems you just need to decide which direction you want to go.

I have since decided we definitely should add aliases... so for very unusual languages I want to allow things like saying slot => variable... so then you could write "slot" everywhere and only specify the mapping once in a table, and then the output would be "variable" (not slot), making existing themes "just work".
I've also decided that we should allow 2nd level expansion of the top-level namespaces by any grammars IF it seems clear that expansion is relevant/correct/obvious (at a glance).  IE, (entirely made up example) if a certain language has a KEY feature of uppercase vs lowercase strings... and that was HUGELY important somehow then we'd be open to string.upper and string.lower for that language, and those could be themed separately - although existing themes would not automatically pick up on that nuance... they'd just apply the string rule.

So I think we're almost back to where we were before... if we want to continue this as a PR for the core library then we need to figure out what the classes are (and what the aliases are)... the only new option would be if you could persuade me that there are any true 2nd level classes to be added here - but again those are going to pick up the styling from the 1st level since they aren't supported by themes.
It sounds like long-term we may need a pattern-match class, but seeing as we don't have one now if you feel strongly that's what these are they I'd suggest just using that as an alias and pointing to perhaps type or another legacy class.
Does that help at all?

So it seems a remaining concern here if you want to be in core but ALSO more in control of theming, i.e. perhaps you really want "pattern-match" NOT just as a alias (limiting your 3rd party website theming options) but as it's OWN individual addressable CSS class - despite whatever the core library has to think on the topic. (and despite our issues with legacy themes)
In that case wouldn't just hacking the alias table solve the problem?
let lang = hljs.getLanguage("mathematica"); // fetch the core bundled language
// rewrite the alias into a non-alias...
lang.themeAliases["pattern-match"] = "pattern-match"
// no 3rd step

IE, internally we choose to alias pattern-match to type (or whatever)... any website that wants to "disagree" with us merely rewrites the alias table to simply make the alias a literal instead... boom, everyone is happy.
Would that solve the issues we were having about different ideas regarding theming?

Possibly of interest: https://github.com/joshgoebel/se_highlightjs

@joshgoebel Thank you for the very detailed description of the possible ways to move forward. I would like it if we could keep the Mathematica highlighting in the core of Highlight.js. The alias idea looks good to me as well and I think it shouldn't be too much work for anyone who would like to highlight Mathematica code.
Right now, I removed most of the things that I'd probably want to see styled in the future. I hope this is a good temporary compromise until we sorted out the things with the Aliases. The only thing that's still included is the pattern-match class that we use for, well, pattern matches. If this is not styled in most themes, it still looks great.
You might wonder, why I removed the variable highlighting for normal symbols and named-characters. The reason is that most language, like Java, also don't highlight normal code. Also, one thing I haven't told you is that I don't match e.g. greek unicode characters, which are also valid input. However, I'd need to assemble and verify the correct utf8 ranges for these characters first and it's probably a good thing to just leave it out for now. It's rarely used anyway, but when someone does, it probably looks shitty if these things were coloured differently.
The se_highlightjs repo is a nice thing, and, in fact, we had something similar in the past to let users test new schemes. However, for the common user on StackExchange, that's not an option. Either they get good highlighting out of the box or they just don't care.

One thing I was wondering when looking over the added stackoverflow-light.css is why a built_in is regarded semantically equal to a literal:
.hljs-name,
.hljs-type,
.hljs-number,
.hljs-selector-id,
.hljs-quote,
.hljs-template-tag,
.hljs-built_in,
.hljs-title,
.hljs-literal {
  color: #b75501;
}

E.g. for C/C++ this would mean that struct is styled like numbers. In my case, it means that "built-in" functions get the same styling as numbers. That is odd.

Right now, I removed most of the things that I'd probably want to see styled in the future. I hope this is a good temporary compromise until we sorted out the things with the Aliases.

I wish you hadn't done that. :)  I'd prefer if you'd just added a cssAliases key (and added the aliases) and then threw it back to me to actually make it work. :-)  The next minor release really isn't until Dec anyways and I think we can figure this out before then.  Could we do that?  So I'm talking about something simple like:
cssAliases: {
  "somefunkymathematicathingy": "type",
  "pattern-match" : ...,
  ...
}

Left hand side is completely up to you... whatever concepts exist in the Mathematica world.

The only thing that's still included is the pattern-match class that we use for, well, pattern matches. If this is not styled in most themes, it still looks great.

Ha, just not going to let that one die... I think aliases solve this anyways... I can decide what it should alias to for most users or I'm forced so conclude there are 0 good choices then we promote it. :-)  So I think it will solve itself.

Also, one thing I haven't told you is that I don't match e.g. greek unicode characters, which are also valid input. However, I'd need to assemble and verify the correct utf8 ranges for these characters first and it's probably a good thing to just leave it out for now.

I'd say that's good for now.  This may also be relevant: https://github.com/highlightjs/highlight.js/issues/2756

The se_highlightjs repo is a nice thing, and, in fact, we had something similar in the past to let users test new schemes. However, for the common user on StackExchange, that's not an option. Either they get good highlighting out of the box or they just don't care.

Sure, I just thought if you wanted a chance to see your work in play before SE gets around to updating (who knows when) that it might be of interest.

why a built_in is regarded semantically equal to a literal:

You'd have to ask SE unless you're saying we simply made a mistake in our translation of the theme from their UI to our repo. :-)

So do you want to go and add the aliases table (and improved naming back) and then give me motivation to build that out, or should I just review this as is and see if we can get it merged sooner?

So do you want to go and add the aliases table (and improved naming back) and then give me motivation to build that out, or should I just review this as is and see if we can get it merged sooner?

Sounds like a plan. What if I need something like this?
cssAliases: {
  "variable": "plain/none/nostyle"
}

That would be important for, e.g. braces that should not get any styling if it's not explicitly provided by a style-class.

That would be important for, e.g. braces that should not get any styling if it's not explicitly provided by a style-class.

I'm not sure I follow the question.  This has no awareness of the actual CSS style in use... This is just mapping the grammars OWN choice of phraseology/terms to OUR overall naming scheme (for purposes of themes "just working").  If you truly have a thing that no one would ever choose to highlight, then it simply would not have a className (and hence not need an alias).
Or maybe you're talking about things you'd like to allow highlighting for but don't think we support at all now (things you think can't be alised?)... I doubt that's truly the case, but you can leave them blank if you want and I'll sort it out.  But taking brace as an example I would map it to punctuation... I think for MANY users of the library that's a great choice and consistent with other grammars (and other highlighters).  If a specific user/site of the library disagrees then as I said they could solve it just be remapping the alias (with a single line of code).  Or adding a scoped CSS rule like: .mathematica .punctuation, etc...

@joshgoebel Alright, I made the necessary changes to push the Alias-table idea forward :)
First off, I think there should be a general style class for variable. It's currently buried in "Tags, ..." section but I'm sure that there are not only a few languages that would make use of this. I mean, how else would you tag user-defined variables except not styling them altogether which is done by most languages.
Unfortunately, I'm also uncertain what a good/better name for theming could be. Basically, we "tag the parsed tokens" which, as you already mentioned, is not specifically bound the theming. However, for the general user, a thing like "theming" is probably easy to understand while something with "tokening" is not.
I'm unsure about the things I currently aliased to "variable". I'd be perfectly fine if they're not styled at all for the themes available. One major reason to do something like that is that I want to parse the named-characters as one. That is, the whole \[Gamma] thing should look like a "variable".

@joshgoebel Alright, I made the necessary changes to push the Alias-table idea forward :)

Awesome.

First off, I think there should be a general style class for variable. It's currently buried in "Tags, ..."

I feel like we're recovering some of the same territory but it's ok, this has been a long thread.  There is already a PR to rework this list... variable IS generic and should be moved to the top, yes - as well as some other things.  The ReasonML stuff is still "bad" for now. :-)  But you can use variable to your hearts content.

Unfortunately, I'm also uncertain what a good/better name for theming could be. Basically, we "tag the parsed tokens" which, as you already mentioned, is not specifically bound the theming. However, for the general user, a thing like "theming" is probably easy to understand while something with "tokening" is not.

I'll give this some thought... Details like this matter because of how strict we are with "no breaking changes"... once something gets released we're stuck with it until a major release...

I'm unsure about the things I currently aliased to "variable". I'd be perfectly fine if they're not styled at all for the themes available. One major reason to do something like that is that I want to parse the named-characters as one. That is, the whole [Gamma] thing should look like a "variable".

Gibberish, I'll have to boot it up and look at some samples... speaking of that is there a great place I can easily find some real pieces of Mathematica code to try out the highlighter with?  I'll of course first try and find the mathematica SE site...

Finally, you will see that I chose the "symbol" class for what you call "variables" and aliased it to "variable". The reason is that these things are really called "Symbol" in Mathematica and if your approach allows it to alias an existing class "symbol" to an existing class "variable", I'd prefer to keep it like that. Semantically, this makes more sense for Mathematica.

So they are called symbols but semantically they are more like variables? LOL. Something like that?
    theming: {
      aliases: {
        brace: 'punctuation',
        pattern: 'type',
        slot: 'type',
        symbol: 'variable',
        'named-character': 'variable',
        'builtin-symbol': 'keyword',
        'message-name': 'string'
      }

Oh yeah, we should have done this from the beginning. :-)  Let me pull the branch local.

Mathematica is VERY colorful right now. :)  I find I have little thoughts on the mappings (likely my lack of Mathematica foo) other than my curiosity about mapping symbol to variable.  I had planned to use a quick look-up table for this so that kind of thing should be possible, but does seems strange. :-)
It does feel much easier to just give you a mapping table though rather than endlessly debate what is what... @egor-rogov You had some thoughts before I think, any thoughts on the mapping table?
Sorry for the zillion comments, but I think we're making some real progress now. :-)
FYI: The general rule about relevance is you shouldn't be claiming relevance for something that is so generic it could appear in almost ANY source code at any time (other than some common things we already have rules/guidance for like strings, etc).  And you can't claim relevance for things that commonly exist in other languages but they don't generally claim relevance - ie that's why you don't get any points for operators or braces.  Every language has operators but no one gets relevance for them (because of legacy).

Let me know when you reach a stable point again and I'll push the aliasing code to you. :)

About the "symbol": If we make this whole alias-thing, I think we can go the whole way

normal user symbols that have not built-in meaning are tagged as "symbol" and default to "variable"
named-characters which are just fancy looking symbols are tagged as "named-character" and default to "variable"
symbols with built-in meaning (like defined system functions) are tagged as "builtin-symbol" and default to "keyword"

You asked about real Mathematica code for testing. The first half of the demo-page I added is real code that shows most of the features. The second half is code to test specific features like if all number-forms are correctly highlighted. I guess mathematica.stackexchange has the best real-world use cases. If you want some larger test code, maybe look at this:

Rubi.m
IntelliJ Tools

The first half of the demo-page I added is real code that shows most of the features.

Yes, it's quite helpful. :-)

About the "symbol": If we make this whole alias-thing, I think we can go the whole way

Ok, so I get there are 3 cases... 3 "symbol like" thingies... yet we already have a symbol CSS class that we consider general purpose... what is the reasoning for going out of your way here to avoid using that to represent at least one/some of the symbols - rather than instead falling directly to variable or keyword?

This is a huge thread and obvious I haven't read every message, but I noticed the concerns about tagging operators.
The original highlighting on SE bolded brackets, as well as several other operators. This made a big difference to readability (because Mathematica code is usually dense with both brackets and operators). It's one of the reason I can't read code as fast on SE with the current highlighting.

The original highlighting on SE bolded brackets, as well as several other operators. This made a big difference to readability (because Mathematica code is usually dense with both brackets and operators). It's one of the reason I can't read code as fast on SE with the current highlighting.

As mentioned somewhere above this is now becomes an implementor concern.   The core library has it's own stance on these things for legacy reason (few of our themes support operators and punctuation yet and it seems you can argue if brackets are truly "punctuation" or something else entirely)... but if SE desires a VERY specific look they can get it easily with only custom CSS and a single line of JS to tweak/reset the CSS alias table.
For example you could even alias "brace" to "strong" though I think that's a terrible semantic idea. :-)

what is the reasoning for going out of your way here to avoid using that to represent at least one/some of the symbols - rather than instead falling directly to variable or keyword?

In Mathematica, variables are symbols. The things that you use in the same way as variables or keywords in other languages are symbols in Mathematica. The styles that a general theme/stylesheet might use for variables / keywords are likely to look good if they are used for symbols in Mathematica. It is also a reasonable match semantically.
I hope this is a helpful explanation, and will save @halirutan typing an answer :-) If not, I'll stay out of the already very long thread.

According to the highlight.js reference, "symbol" is for "symbolic constant, interned string, goto label". That is an absolutely bad match semantically. Just because it's called the same thing doesn't mean it's really the same.

True.  I can live with that explanation.  :-)

According to the highlight.js reference, "symbol" is for "symbolic constant, interned string, goto label". That is an absolutely bad match semantically. Just because it's called the same thing doesn't mean it's really the same.

True. I can live with that explanation. :-)

Exactly that was the reason for not using the general symbol class. I remembered we had a long discussion about that we should not abuse classes that have a specific semantic meaning. So the general symbol is indeed a very bad match for our symbols semantically and that's why I thought aliasing it to variable was one reasonable way to solve this.

disableAutodetect: true,

So I know you've explained why you don't think the auto-detect will be great... but is that a reason to turn it off completely?   The HUGE list of built-in keywords would tend to argue against this I think.

It does seem perhaps builtin-symbol should alias to built_in though, no? (the fact that you named it "built-in symbol" is a pretty large clue for me)... they seem at a glance to be built-in functions/classes/objects/somethings at a glance much more than they seem to be language keywords.

Overall looks great and most seems to work. To answer your questions:

So I know you've explained why you don't think the auto-detect will be great...

Then let's turn it back on. In the beginning, I didn't know about all the relevance stuff. Now that we fixed it, it seems reasonable to so if it works.

It does seem perhaps builtin-symbol should alias to built_in though, no?

Well, this is the point, where the "theme reality" hits. Many of the themes have built_in in the same style class as number which makes absolutely no sense in our case. So while I agree that built_in should fit better here, in reality, using keyword will give much better results in many themes.
About the escaped closing ] in the regexp: You haven't really told me why you think the escape is necessary. In the commit message, it states something about compatibility with UTF-8 but I don't understand this. Can you explain or give a concrete example, why this should break something?
Can we make the linter happy here? The anonymous function is no type annotation and therefore pops up as a warning in my IDE. I'd suggest using something like this:

However, there is a caveat: Since there are built-in Response types other than the one in ./lib/response.js we'd need to import this so that the response variable points to the right type. So locally here, I have
/* eslint-disable no-unused-vars */
...
// noinspection ES6UnusedImports
import Response from '../lib/response.js';
...
  const SYMBOLS = {
    variants: [
      {
        className: 'builtin-symbol',
        begin: SYMBOL_RE,
        /**
         * For performance out of fear of regex.either(...Mathematica.SYSTEM_SYMBOLS)
         * @param {Array<String>} match
         * @param {Response} response
         */
        "on:begin":
          (match, response) => {
          if (!SYSTEM_SYMBOLS_SET.has(match[0])) response.ignoreMatch();
        }
      },
...

Would that be OK with you? I really like my source files "green" without warnings because right now I know what we did and that suppressing the warnings was OK. The might not be true in some months.
Finally, I made all the changes explained above and fixed up lib/regex.js. I built the highlight.js and ran all the tests and except the 2 failing ones for ReasonML, that we also have on this branch, everything seems to work. I would not include the improvements of regex.js in this branch. However, do you think that the passing tests are a good indicator that my changes work?

compatibility with UTF-8 but I don't understand this. Can you explain or give a concrete example, why this should break something?

Add /u and watch it blow up without the escape.  In my experience non-UTF8 mode is more permissive in general though I'm not sure which is correct - I'm calling the UTF8 more strict parsing "correct" as in many cases it seems to be.

Can we make the linter happy here?

Sure, but we'd want to just tell it the type of the data (the whole variable is a Mode), it doesn't need to be annotated on-site.  Resolved.  Does this resolve on your side also?  I've been a lot stricter with the type annotations with the source in the past than the grammars themselves.

and fixed up lib/regex.js

Wait, I think I lost the thread about what we're talking bout here. :)

Well, this is the point, where the "theme reality" hits. Many of the themes have built_in in the same style class as number which makes absolutely no sense in our case.

By my count:

Themes where they are the same: 40
Themes where they are different 51

Without looking it's easy to find themes that blend type and keyword also (15 at a quick count)... many themes are "quirky".  I personally find many unattractive - but that's just me - there is no accounting for taste.  Since we currently don't have any objective standards on what makes a "good" theme (or what themes we include)... my bent leans heavily towards theme autonomy - letting themes determine their own look based on the semantic meaning of the terms - rather than allowing grammar one at a time to "cheat" themes because they disagree with theme choices.
I expect most sites/implementors typically use a single theme (or two) and if they have issues/disagreements with that theme then they can customize it. (using CSS or not aliases)

So while I agree that built_in should fit better here

This kind of seals the deal for me... these are indeed more built-ins than they are keywords.  Your reasoning from a purely visual perspective.

using keyword will give much better results in many themes.

I think this is very dependent on how you define "better results".  When your results become "the opposite of what the theme author intended" that is not better.  I get that your heart is in the right place here, but I feel this is short-term thinking... perhaps it does make your theme slightly better today but it costs the whole library tomorrow in terms of muddier semantics, theme authors not being certain of what things mean what in which grammars, etc...
...now a theme author who already clearly expressed "really i want numbers and built-ins to look the same" (for whatever reason) has to add a special case CSS to handle Mathematica because you pulled the rug right out from underneath them by calling your built-ins keywords.
If there is a true issue with our themes here then we need to work with the theme authors (or a visual designer) to actually fix the themes, not just let grammars one at a time redefine the semantic meaning of the terms in order to achieve a certain "look".  This is the road to insanity.  This probably also touches on the broader long-term initiative of higher fidelity grammars.
We could surely use some help here, but the solution isn't grammars just going rogue like this.  I will spin this off into a new issue.

@halirutan Lets focus on the remaining technical issues here (if any).  Are they any?  I'm still not sure what your reference was to regex.js but if you don't think it belongs in this PR anyways then maybe it's simply not relevance here. :-)
If you want to respond regarding the theming (built_in vs keyword) please continue that discussion over in the new thread I started.

@halirutan Lets focus on the remaining technical issues here (if any). Are they any?

@joshgoebel I think it's good to go now.

@joshgoebel Ah, didn't you want to turn on autodetect? I tested it locally already and it seems that it doesn't break anything.

@halirutan Thanks for all the great work on this!!! :-)  Hopefully it serves the SE community (and others) well.

This was before our new PR template... could you update the changelog (this might be worthy of several bullets if you wanted, you decide)... And if you wanted to provide me a short summary for the squashed commit that'd be great also.

This was before our new PR template... could you update the changelog (this might be worthy of several bullets if you wanted, you decide)... And if you wanted to provide me with a short summary for the squashed commit that'd be great also.

@joshgoebel Would something like this work
- enh(mathematica) Rework entire implementation [Patrick Scheibe][]
    - Correct matching of the many variations of Mathematica's numbers
    - Matching of named-characters aka special symbols like `\[Gamma]`
    - Updated list of version 12.1 built-in symbols
    - Matching of patterns, slots, message-names and braces

How does the linking of author name to their GitHub page work? I guess it requires that the PR is merged first and then it automatically retrieves it from people who have committed in the repo? Never seen this before.
For squashed commit, you could use something like this which details the technical points more:
Fix several issues and implement additional features for the Wolfram Language (Mathematica)

- Include an up-to-date list of built-in symbols in a separate `lib/mathematica.js` file. It's one keyword per line and more easy to maintain.
- Fix regexp to identify symbols/variables which requires special treatment and does not follow the common `IDENT_RE` matching.
- Replace generic `C_NUMBER_MODE` matching with dedicated regular expressions for all possible numbers in Mathematica.
- Include named-characters in the matching of symbols.
- Allow for dedicated styling of
    - pattern-like forms, e.g. `par_String`
    - slots of anonymous functions, e.g. `##3`
    - message names, e.g. `myFunc::usage`
    - braces, curly braces and brackets
- Introduce `classNameAliases` to map specific styles to general styles used by all themes. This allows for using built-in themes and writing sophisticated Mathematica themes.

How does the linking of author name to their GitHub page work?

No magic, it's just a footnote link... see lines 35-41, etc...  I'm gonna noodle on this a bit more to see if I come up with a better name for the alias stuff (or anyone comments).  This should get merged in the next day or two though! :-)

@halirutan  🚀

Huge thanks to both of you!

@joshgoebel Thanks a bunch and thanks for guiding me so well along the way. I really enjoyed our discussions!

Are there any common patterns from other languages that are ILLEGAL in Mathematica so we can add an illegal key?  Things that would let us know for sure that a given snippet is NOT mathematica?  Or just any normal characters that are simply entirely disallowed in the code?
# ruby code
// JS comment
/* C++ comment */

Often times comment patterns can be good for this...


Update bootcode4.bin

Update this binary to the latest.
Also rebuilt the windows installer

@timg236 Is this ok?


feat: Add support for custom annotations and labels for runners and starter

Resolves #23 and allows for adding of custom annotations and labels for runners and the starter.
Reasonably straight forward PR.
I did have a question about the imports however, it appears there are duplicate imports in runner.go and starter.go (k6 is imported in both of them but doesn't need to be, and is done so under 2 different names. As is core api from k8s). I can open another PR to create a helper.go that has newLabels, and the imports in it if desired. I found it a little confusing for a moment and is probably unnecessary.

Mistakenly merged #26 prior to this one, but since all commits are present there as well I'll just go ahead and close this one. Thanks!


Feature/update block position #129

Description
Updated the UI to send position changes back to the server.
Also updated the nav menu to use the names from the layout attribute
Testing instructions
Add a set up instructions describing how the reviewer should test the code

Review code
Check Travis build
Review changes to test coverage

Agile board tracking
connect to #129
connect to #127

Codecov Report

Merging #151 into version1 will decrease coverage by 2.04%.
The diff coverage is 46%.

@@             Coverage Diff              @@
##           version1     #151      +/-   ##
============================================
- Coverage     88.41%   86.36%   -2.05%
============================================
  Files            47       47
  Lines           941      983      +42
  Branches        142      150       +8
============================================
+ Hits            832      849      +17
- Misses           93      114      +21
- Partials         16       20       +4

Impacted Files
Coverage Δ

src/layout/block/blockWidget.component.js
92.3% <ø> (ø)
:arrow_up:

src/navbar/navcontrol.component.js
73.33% <ø> (ø)
:arrow_up:

src/layout/block/block.model.js
83.33% <ø> (+6.41%)
:arrow_up:

src/malcolm/reducer/navigation.reducer.js
82.05% <58.82%> (-17.95%)
:arrow_down:

src/malcolm/malcolmActionCreators.js
48.27% <6.25%> (-51.73%)
:arrow_down:

src/mainMalcolmView/middlePanel.container.js
72.72% <62.5%> (-7.28%)
:arrow_down:

src/malcolm/reducer/malcolmReducer.js
87.37% <77.77%> (-0.13%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6677e11...1def33d. Read the comment docs.

Reviewed myself, I am aware there is a technical debt issue around increasing test coverage again.


Make max_model_len configurable

Allows the user to override derived max_model_len if they so desire.
We can also warn the user if the max_model_len is set above what vLLM has derived - lmk if you think that's a good idea!

@zhuohan123 added warning


Add id_token refresh to Google provider

Token refresh does not update id_token in session when using Google provider.
Description
Token refresh does not update id_token in session, so JWT token passed to the backend application is expired while the access_token is updated.
Motivation and Context
Add id_token refresh to Google provider because the JWT token provided by set-authorization-header is outdated after a token refresh.
How Has This Been Tested?
It is tested manually requesting the bearer token from the backend application.

Thank you for your review. I have updated the changelog.


x/tools/gopls: saving is very slow with the latest update

What version of Go are you using (go version)?

$ go version
go version go1.16.2 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/evan/Library/Caches/go-build"
GOENV="/Users/evan/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/evan/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/evan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/evan/dev/esbuild/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j6/np400cw17sz0n5ljd67byrzw0000gn/T/go-build1589752302=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I use the Go extension in Visual Studio Code (v0.23.2) which uses gopls (v0.6.7) on a project with a single dependency.
What did you expect to see?
The Go extension formats code and automatically inserts missing imports whenever I save. I expect this to feel instant, and it has been instant in the past.
What did you see instead?
Something changed recently. Now saving a Go file takes a really long time (e.g. 7 seconds) and causes Visual Studio Code to show a Saving 'resolver.go': Getting code actions from ''Go'' (configure). message in the corner. This is really breaking my flow because it means I constantly have to wait in between adjusting code and trying it out.
I would love for this to be fixed of course, but I would also love to have a workaround in the meantime. For example, I assume this is due to automatic import insertion and not due to code formatting. Is there a way I can disable the slow part but just keep code formatting on save? Maybe there is some cache somewhere that has gotten filled up and can be cleared to make this go faster?

Here's a log capture: gopls-slow-save.log. This is the result of clearing the log, changing the indentation of a single line, re-saving the file, and waiting for the file to be saved. This particular save operation took around around 23 seconds to complete.
I do have a lot of files under the top-level project directory in case that's what's causing this. The count from find . | wc -l in the top-level project directory is 351,777 files and directories. This is because I'm writing a JavaScript bundler and there are some projects I use for testing with node_modules directories. These test projects are all under .gitignore however.

Also experiencing the same issue with a large codebase[1]

https://github.com/juju/juju

Thank you for these reports. We were able to debug this issue using the juju repository, and we'll have a patch release out to fix this soon. You can try it out now by downloading gopls at master: GO111MODULE=on go get golang.org/x/tools/gopls@master golang.org/x/tools@master. Please let us know if the issue is resolved at master.

I just updated and it appears to be fixed! Thanks so much for the fix.


TalkerView doesn't user overall theme colors

I'm using a custom theme for my app. While my app and the Talker Monitor take the colors from my theme, TalkerView doesn't.
Screenshots
My app :-)

Talker Monitor :-)

TalkerView :-(

I guess lines 33, 34 and 52 in talker_view_appbar.dart cause the problem.
Instead of using the talkerTheme, they should use the app's theme.
I tried to adjust the TalkerScreenTheme but that doesn't work in my case as the white text in the filter buttons is hardly not visible and I'd prefer a white background for the log data.

It would be great, if this could be fixed. :-)

Just found PR https://github.com/Frezyx/talker/pull/242 by XanderD99.
I guess it will cover this problem.

Hi @Vera-Spoettl
That pr of me would indeed fix it as it would use your apps theme instead.
I however already made a previous PR that fixed some small ui inconsistencies and added a factory method to the TalkerScreenTheme. This I think should fix your issues as well.
If I recall correctly the AppBar should then use talkerTheme.backgroundColor.
  factory TalkerScreenTheme.fromTheme(ThemeData theme, [LogColors? logColors]) {
    return TalkerScreenTheme(
      backgroundColor: theme.colorScheme.surface,
      textColor: theme.colorScheme.onSurface,
      cardColor: theme.colorScheme.surface,
      logColors: logColors,
    );
  }


[stable/stolon] Upgrade Stolon image to v0.13.0-pg11

What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/assign @jlegrone

lgtm

@lachie83 @davidkarlsen Hi guys. Can any of you please advise what needs to be done here to move this PR forward? This is my first PR in helm repo and I am not sure what the next steps are. Is anything else needed for me.
Please note, I've added the OWNERS file with the current 2 maintainers as per #13869. Thank you.

@viglesiasce @prydonius Can one of you guys please review this PR and let me know if anything is preventing it from being merged? Any other feedback that would help to maintain this helm chart fresh would be also appreciated. I am pretty new to this. Thank you.

/verify-owners

@demisx
Could you bump the chart to latest please? Thank you.

@lwolf
I suggest splitting update OWNER file and image to different PR.

@zanhsieh thank you, will do

@lwolf
Would you mind to post the link of separate PR for owners changes?

@zanhsieh yes, sorry for the delay - https://github.com/helm/charts/pull/18175

@zanhsieh I've updated this PR to include the latest chart version. Please let me know if anything else is needed.

/ok-to-test

@demisx
I think it would be better if we take off OWNER modification as separate PR; otherwise this would always have "do-not-merge/invalid-owner-file" tag on.

@zanhsieh I've removed OWNERS file from this PR and will create a separate one as requested.

/retest

/retest

/verify-owners

Guys, is there anything else I can do to move this PR forward?

@demisx sorry about the delay, I tried updating your branch but got hit with the DCO bot. Please rebase from master and I'll approve again.


♻ SQLModel EmailStr support

Thanks @efemiydi! :coffee:
This was handled in another PR, so I'm gonna pass on this one, but thanks for the effort! :cake:


[MSG] Change inquiry statusbar background to white

Fixes https://github.com/artsy/collector-experience/issues/754

Fails

:no_entry_sign:

No CHANGELOG added.

  Generated by :no_entry_sign: dangerJS

🔥


Update tomcat-catalina, tomcat-coyote to 9.0.21

Updates

org.apache.tomcat:tomcat-catalina
org.apache.tomcat:tomcat-coyote

from 9.0.20 to 9.0.21.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a nice day!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [{ groupId = "org.apache.tomcat" }]

labels: semver-patch

Retargeted in #2630.


✨ Support .NET 6

Description
AutoFixture ought to keep up with the times and add support for .NET 6.
Tasks

[ ] Get all solutions to build on the .NET 6 SDK
[ ] Update the build on GitHub Actions
[ ] Add support for DateOnly and TimeOnly (see #1304)

Feel free to add more tasks as you see fit.

.NET 8 is coming next month. Is this project still being maintained?

@aivascu how about using github packages for the previews? at least we could push to nuget previews and not artifacts from every build.

@Kralizek, it's a good idea in the sense that there's no storage limit for oss packages, unlike MyGet. However, I wasn't planning on publishing packages on every build, only tagged releases.
The reason for moving preview packages to NuGet is that no one bothers to add the MyGet feed to use the available previews.
Perhaps we could get some use out of GitHub packages with branch releases?

I was thinking about non tagged packages for GHP as well.
So tagged previews and release packages go to nuget, everything else to GHP.

Support for .NET 6 and DateOnly, TimeOnly structs are now available as part of v5.0-preview011.
Additional requests related to .NET 6 will be implemented as separate requests/issues.


Fabric: support for multiple simulatenous graves

Hi there, not sure if there's a config for this I'm missing but if you die trying to retrieve your grave, a new grave is generated as expected but breaks the old grave.
Steps to replicate:

In creative mode give yourself some gear and inventory items
Spawn a few creepers and switch to Survival and let them kill you
Switch back to creative upon respawn, give yourself some more items and repeat the entire process - preferrably close to the first grave that spawned
Now there's 2 graves as expected but only the second one works, the first one doesn't do anything anymore

Thanks!

This is not intended, and likely a bug. I would recommend checking what your "maxBackupsPerPerson" config is set to. If this is 1, it would explain at least half the issue (only half as that should delete the old grave too).
Does this happen with only YiGD and its dependencies? Or can you find some mod in particular that causes the issue?
I'm assuming you're playing on 1.20(.1) using the latest version of YiGD, but answering these other questions would help, as I might not have time for testing for a few days myself

Thanks for the reply!  I'm playing on 1.20.1 with Travelers Backpack, Trinkets and Elytra Trinked - which was the combo I'd been specifically testing because most other grave mods don't play well with them...
I'll try and replicate on a cleaner instance and also check that config and post back in a bit..

Any updates? I'm sorry for the long response time, but I was not able to reproduce the issue myself


Code completion does not work on MySQL and version 3.6.6

Title says it all. Works fine after installing version 3.6.5.

Can't reproduce.
Check Error Log (and debug log) for any related errors.

I have checked error log, there were some old entries so I deleted them. Then restarted DBeaver and reproduced the problem. Nothing appeared there.
I also started DBeaver from command line and nothing signaling any issue was written to console either.
How do I provide debug log?

https://github.com/serge-rider/dbeaver/wiki/Log-files
BTW what do you actually mean under code completion?
What happens when you type "select * from <Ctrl+Space>"?

I have attached debug log, but there seems to be nothing related to the issue.
dbeaver-debug.log.zip
What happens is: I log in to the database. Type "select * from " and I get list of suggested tables. Then I select a table and give it an alias. Using this alias, I try to add fields that I am interested in ->
select  t.
from table_t t

It is a bug in the recent versions.
Thanks for report.

This issue is back again in 3.6.8

I guess this is duplicate of #467

I can not reproduce #467 in 3.6.7 version on my machine. That works as expected. Do you want me to try it again in 3.6.8 anyway?

#467 is a bug in 3.6.8. And it differs this bug (auto-compelete doesn't work only if you use table alis and empty column name).

Ok, I misread the bug report and thought that the bug is reported against 3.6.7.
I went to reinstall 3.6.8 and tested it again and it seems to behave as in #467. It works when no alias is used, but I have not noticed that as I always use aliases. It also works after typing at least one letter.

Ok. So this is a duplicate. Will be fixed in 3.6.9


MacOS Build

In order to successfully build piper on MacOS devices (Intel or Apple Silicon), the follwing pull requests have to be merged:

espeak-ng/espeak-ng#1811
rhasspy/piper-phonemize#11

When trying to run the freshly built executable on a Mac, please keep in mind that the executable will not find the phontab file since it is not located where expected. In order to test the successful compilation, run the following command after downloading the required models:
echo 'Welcome to the Machine!' | build/piper -m en_US-lessac-medium.onnx -f welcome.wav  --espeak_data
 ./install/espeak-ng-data

Issue was posted in the wrong project - See rhasspy/piper#227


[CryptomeBridge] Fix pageformat

Bridge was broken, I fix with the new format

Perhaps NatoNiapcBridge.php should not be included in this PR?

Also please fix linting

New commit with actons requested

I tested bridge locally and it works fine now.

Fixed in https://github.com/RSS-Bridge/rss-bridge/commit/943a5e3e8b6a3ba0c4a339db4e3c98864117e18f


Updated README.md - small typo edit

Line 487 says "...similar to example 4..", but I think it should say "...similar to example 3...". Could confuse a newcomer.

Thanks!


Fix debug action missing kubectl info when using keystone

The debug action failed to collect kubectl info when keystone was in use:
Unable to connect to the server: getting credentials: exec: fork/exec /snap/bin/client-keystone-auth: no such file or directory

This should hopefully fix it - don't bother using the kubeconfig at /home/ubuntu/config, which is meant for end users. Instead, use the kubeconfig from /root/.kube/config (default path) which is what the rest of the charm uses. It should work.

@battlemidget


Prefixes don't work

Hi! i have a server and i want to make prefixes, i already wrote them, but the don't work. probably blocked by other plugin, the problem is i don't know what plugin is it. so can someone help me?
here's our pex permissions.yml file : http://pastebin.com/qZn37jSu
and this is out plugins :
CoreProtect
DoubleJump (custom plugin by me XD)
Essentials-2.x-SNAPSHOT
HolographicDisplays
Modifyworld
Multiverse-Core
Multiverse-Inventories
PerWorldPlugins
PermissionsEx (Of course XD)
VanishNoPacket
WorldEdit

Anyone?

i seem to have the same issue on my server too? worked while my server was 1.7.10 but now i upgrade my jar to 1.8 and now running 1.8.7 spigot. Prefix doesnt work. i tryed removing a plugins just leaving PermissionsEx on but still nothing.

solved it, essentials blocked it


Upgrade sbt to 0.13.8

It finishes a move from sbt 0.13.6 to 0.13.8. This transition does not
include the change of deprecated AggressiveCompile to recommended IC or
MixedAnalyzingCompiler. This part will be covered with separate ticket.
resolves #1002481

This also fixes #1002493.

LGTM

Superseded by #986, since the commit message was not perfect. Assembla can't understand the resolve ticket mention.


Merge Transformer-NO with newest library code, convert wrapper and custom embedding to DataProcessor and GridEmbedding

This is an update of #293 that addresses feedback from @JeanKossaifi and @juliusberner . The main difference is an update in the script train_darcy_transformer.py that gets the code more in line with the new library's API.
Specifically, instead of creating a custom ModelWrapper class and creating custom grid embedding tensors (as well as using an unneeded multigrid-patching callback), I wrote a custom TransformerNODataProcessor that does the preprocessing necessary to convert darcy-flow inputs to the form expected by the TransformerNO model.
This model works on irregular meshes, so I also verified that the model's outputs are invariant to the ordering of the input coordinates, and changed the variable names that refer to grid to refer to mesh instead to disambiguate.

Closed #363 to transfer branch to my fork to save space in git clone. Will re-open.


editor issues

icons not showing on the completions list, etc., 404 on codicons.ttf
hovers not showing at all, this includes docs and also errors
minimap not showing, not a big deal, most likely related to 2, seems like some parts of editor are not loaded

I removed the minimap.

Pushed fix for worker URL to #298


Get-CsAutoAttendant, CsOnlineAudioFile filename is empty

When I try to retrieve the CsOnlineAudioFile associated with an auto-attendant I am not able to, I can only get the ID of the CsOnlineAudioFile object but I cannot retrieve the filename with this information
This must be an issue with powershell as the filename is clearly visible from the Teams Admin Center
How can I retrieve the Filename for the CsOnlineAudioFile that is associated with an Auto-Attendant?
The same issue is not present on the CsCallQueue object, just with the CsAutoAttendant
Can you help?
AudioFileID is present:
($CsAutoAttendantList |Where {$_.Name -eq 'AA-TestRGS-0'}).DefaultCallFlow.Greetings.AudioFilePrompt.Id
AudioFileName is not present:
($CsAutoAttendantList |Where {$_.Name -eq 'AA-TestRGS-0'}).DefaultCallFlow.Greetings.AudioFilePrompt.FileName
AudioFileID is present:
($CsAutoAttendantList |Where {$_.Name -eq 'AA-TestRGS-0'}).DefaultCallFlow.Menu.Prompts.AudioFilePrompt.Id
AudioFileName is not present:
($CsAutoAttendantList |Where {$_.Name -eq 'AA-TestRGS-0'}).DefaultCallFlow.Menu.Prompts.AudioFilePrompt.FileName
AudioFileName is present:
($CsCallQueuesList |Where {$_.Name -eq 'CQ-Test'}).WelcomeMusicFileName
AudioFileName is present:
($CsCallQueuesList |Where {$_.Name -eq 'CQ-Test'}).MusicOnHoldFileName

Hi @darwincr, thank you for your feedback.
Did you try with Export-CsOnlineAudioFile?

@dariomws Thank you very much for the contribution and sharing this explanation. @darwincr Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


On failure of a datasource command, log output of stderr

Currently outputs stdout only.

Fixed in #142.


URL on "Customizable" filter is not right

Describe the bug
if you look at the URL, it says: https://www.ensmerchshop.xyz/search/new-arrivals

To Reproduce
Steps to reproduce the behavior:

Go to https://www.ensmerchshop.xyz/search/products
Click on Customizable
See error

Expected behavior
The URL should say: https://www.ensmerchshop.xyz/search/customizable
I'm not sure if this is a shopify issue that @ahmadabugosh can fix, or if it needs a dev @jainkrati

@jainkrati I don't see an option to change the URL on shopify, I think this is something we need to do on the frontend side on Vercel

@ahmadabugosh You can set it by changing the "handle" field in "Search engine listing" section.
I've fixed it so no change is needed now.

@maryjaf to test and verify


Oracle Test fix - p2

This PR is meant to fix #757
Tests in error:

RawPaginatorSpec.shouldLoadOnlyRequestedColumns:134
JsonLog4jLayoutSpec.shouldPrintJsonLogValuesAndParams:102

As always, if you have any question, feedback or suggestion, please do not hesitate to say so :)
Cheers

@jfcabral thanks for the fix!


Clarify Blazor ParameterView remark

Fixes #19597
Thanks @pstricks-fans! :rocket:

I think this will need to say "for the component" because there's nothing stopping a dev for substituting a different ParameterView in the overridden method. A ParameterView passed to the base method didn't come from the parent in that scenario. Best to leave the source of the actual values for assignment a bit loose.

@Rick-Anderson @scottaddie @wadepickett ... looks like our whole topic versioning is blowing up the suggestions/warnings now (new build system checks?) because we duplicate headings ...
https://opbuildstorageprod.blob.core.windows.net/report/2020\8\20\332f35cd-658d-4914-0f85-907e40ee89f0\PullRequest\202008202214594937-19598\workflow_report.html?sv=2016-05-31&sr=b&sig=fxEiybLyquVWqVZ%2BMKe6UanyyI1UdVwM%2B0BjHjALcfw%3D&st=2020-08-20T22%3A10%3A56Z&se=2020-09-20T22%3A15%3A56Z&sp=r

I see some descriptions that I need to fill in there. I'll put up a PR for that.


archive this repo?

Thanks for your work! It seems to be continued in another repo now by @ryanabx so I think it would be best to archive this one?

I am currently still using this one myself, though I would recommend @ryanabx fork. I will archive this eventually and link to that one.

I don't think this needs to be archived, unless drakulix would like to of course