TeX input "splitting up" \mathXX{foo}

As per discussion with @zorkow, filing this here.
@davidmjones recently pointed out to me that (from a TeX perspective), things like \mathrm{foo} and \mathbf{foo} and \mathsf{foo}, etc., shouldn't be split up into individual characters.
Avoiding this would be a nice improvement for accessibility (and situations with missing glyphs and their shaping maybe as well).

I'm not sure I understand.  In LaTeX, \mathrm{x+y} is displayed with the proper spacing around the +, for example, and that can't be done unless the characters are interpreted as math.  My understanding is that \mathbf changes some fo the math fonts, but does not otherwise alter the way the math is processed.  Of course \textrm{x+y} is treated as a single text item, and the plus will not have extra space around it.  Perhaps that is what you want?
I also don't understand the comment about missing glyphs and their shaping.  Sorry!

In my experience, your example is unrepresentative.  (In my opinion, it's also a perverse, but my argument doesn't depend on that.)  In a typical use of \mathrm, \mathit, \mathbf, \mathsf, etc., it's clear that the argument to the macro is meant to be treated as a single token.  If I write
\mathrm{area} = \mathrm{length} \times \mathrm{width}

I expect it to be vocalized as "area equals length times width", not as "ay ar ee ay equals...".  If I wanted that second reading, I would write
\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{a} = ....

I can provide more examples from real documents if that would be helpful.

OK, @davidmjones, thanks for the additional information.  I understand your viewpoint, and I believe you that many people use the macros in this way.  I did some Googling and while the vast majority of uses are for a single letter where it doesn't matter, I do see this usage in the wild.  But I also do see other usage, such as \mathbf {^{208}Pb}, \mathbf {Au+Au}, and \mathbf{v\times u} along with more reasonable things like \mathbf{\Gamma} and \mathbf{\hat x}.  These all require processing the contents as math (as the name \mathbf indicates it will do).  And while you may consider most of these to be "perverse", I can't think of another reasonable way to obtain \mathbf{\hat x} so that the hat is the proper bold one.
My suggestion, then, would be to make these macros check the contents, and if it is a string of letters (where what counts as a letter is configurable), then enclose it in an <mi>, otherwise process it as math as is currently done.  That should mean that existing math will not break, but your use would proceed the desired output.
In the meantime, you could use
\newcommand{\mathMi}[1]{\mmlToken{mi}[mathvariant=#1]}
\renewcommand{\mathrm}{\mathMi{normal}}
\renewcommand{\mathbf}{\mathMi{bold}}
\renewcommand{\mathsf}{\mathMi{sans-serif}}
...

or the equivalent in the macros list of the tex block in your MathJax configuration (or 'Macros' and 'TeX' in v2).

OK, @davidmjones, thanks for the additional information.  I understand your viewpoint, and I believe you that many people use the macros in this way.  I did some Googling and while the vast majority of uses are for a single letter where it doesn't matter, I do see this usage in the wild.  But I also do see other usage, such as \mathbf {^{208}Pb}, \mathbf {Au+Au}, and \mathbf{v\times u} along with more reasonable things like \mathbf{\Gamma} and \mathbf{\hat x}.  These all require processing the contents as math (as the name \mathbf indicates it will do).  And while you may consider most of these to be "perverse", I can't think of another reasonable way to obtain \mathbf{\hat x} so that the hat is the proper bold one.
My suggestion, then, would be to make these macros check the contents, and if it is a string of letters (where what counts as a letter is configurable), then enclose it in an <mi>, otherwise process it as math as is currently done.  That should mean that existing math will not break, but your use would proceed the desired output.
In the meantime, you could use
\newcommand{\mathMi}[1]{\mmlToken{mi}[mathvariant=#1]}
\renewcommand{\mathrm}{\mathMi{normal}}
\renewcommand{\mathbf}{\mathMi{bold}}
\renewcommand{\mathsf}{\mathMi{sans-serif}}
...

or the equivalent in the macros list of the tex block in your MathJax configuration (or 'Macros' and 'TeX' in v2).

To my dismay, it's extremely hard to distill the point I want to make into a simple statement without getting bogged down in endless exceptions and special cases.  So, in case it's not obvious already, let me state up front that I know there is no perfect solution when you try to assign semantics based on visual markup.
Nevertheless, I think there is a clear pattern of using these macros to encode identifiers based on natural language into, and I think it's important to try to preserve those semantics where possible for the benefit of the text-to-speech engine.  I also think there are some relatively simple heuristics that would capture most of those patterns without breaking anything.

These all require processing the contents as math (as the name \mathbf
indicates it will do).

Right.  I never meant to imply otherwise.  In TeX, these macros only affect letters and digits, so those are the only characters that require special handling.  [Technically, it applies to any character whose math class is 7, which also includes Greek symbols like \alpha, \Gamma, etc., but I don't think those should be treated as letters.]
BTW, that's why something like \mathrm{x + y} strikes me as perverse: It has no affect on the +.  An author might write \mathbf{x + y} expecting the + to become bold, but they would be disappointed.

My suggestion, then, would be to make these macros check the contents, and if it is a string of letters (where what counts as a letter is configurable), then enclose it in an , otherwise process it as math as is currently done.

That would be a huge improvement, but my suggestion would be to combine into a single <mi> element any sequence of math Ord atoms with the following properties:

They are contained in the argument of single \mathrm, \mathit, \mathsf, \mathtt, or \mathbf.
The nucleus of each Ord atom consists of a Unicode letter.
The subscript and superscript fields are empty.

This doesn't apply to things like \mathcal, \mathfrak or \mathbb since those are explicitly used to access distinct Mathematical Alphanumeric Symbols (to use the Unicode terminology), not a specific text font face.  I left out \mathbfit because in the sample I looked at it's too rare to draw any conclusions about.

To my dismay, it's extremely hard to distill the point I want to make into a simple statement without getting bogged down in endless exceptions and special cases.  So, in case it's not obvious already, let me state up front that I know there is no perfect solution when you try to assign semantics based on visual markup.
Nevertheless, I think there is a clear pattern of using these macros to encode identifiers based on natural language into, and I think it's important to try to preserve those semantics where possible for the benefit of the text-to-speech engine.  I also think there are some relatively simple heuristics that would capture most of those patterns without breaking anything.

These all require processing the contents as math (as the name \mathbf
indicates it will do).

Right.  I never meant to imply otherwise.  In TeX, these macros only affect letters and digits, so those are the only characters that require special handling.  [Technically, it applies to any character whose math class is 7, which also includes Greek symbols like \alpha, \Gamma, etc., but I don't think those should be treated as letters.]
BTW, that's why something like \mathrm{x + y} strikes me as perverse: It has no affect on the +.  An author might write \mathbf{x + y} expecting the + to become bold, but they would be disappointed.

My suggestion, then, would be to make these macros check the contents, and if it is a string of letters (where what counts as a letter is configurable), then enclose it in an , otherwise process it as math as is currently done.

That would be a huge improvement, but my suggestion would be to combine into a single <mi> element any sequence of math Ord atoms with the following properties:

They are contained in the argument of single \mathrm, \mathit, \mathsf, \mathtt, or \mathbf.
The nucleus of each Ord atom consists of a Unicode letter.
The subscript and superscript fields are empty.

This doesn't apply to things like \mathcal, \mathfrak or \mathbb since those are explicitly used to access distinct Mathematical Alphanumeric Symbols (to use the Unicode terminology), not a specific text font face.  I left out \mathbfit because in the sample I looked at it's too rare to draw any conclusions about.

Here is a configuration that implements the proposal I made above:
MathJax = {
  tex: {packages: {'[+]': ['math-fonts']}},
  startup: {
    ready() {
      //
      //  These would be replaced by import commands if you wanted to make
      //  a proper extension.
      //
      const {Configuration} = MathJax._.input.tex.Configuration;
      const {CommandMap} = MathJax._.input.tex.SymbolMap;
      const BaseMethods = MathJax._.input.tex.base.BaseMethods.default;
      const TexParser = MathJax._.input.tex.TexParser.default;

      //
      //  Remap \mathrm, etc. to be able to create single <mi> elements
      //
      new CommandMap('math-fonts', {
        mathrm: ['MathFont', 'normal'],
        mathbf: ['MathFont', 'bold'],
        mathit: ['MathFont', '-tex-mathit'],  // internal variant for text italic font
        mathsf: ['MathFont', 'sans-serif'],
        mathtt: ['MathFont', 'monospace']
      }, {
        MathFont(parser, name, variant) {
          const text = parser.GetArgument(name);
          //
          //  Check if the argument is a string of letters only
          //     Make a single <mi> of them if so, otherwise
          //     Parse the argument as normal.
          //
          if (text.match(/^[a-z]+$/i)) {
            parser.Push(parser.create('token', 'mi', {mathvariant: variant}, text));
          } else {
            let mml = new TexParser(text, {...parser.stack.env, font: variant}, parser.configuration).mml();
            if (mml.isKind('inferredMrow')) {
              mml = parser.create('node', 'mrow', mml.childNodes);
            }
            parser.Push(mml);
          }
        }
      });
      Configuration.create('math-fonts', {
        handler: {macro: ['math-fonts']}
      });

      MathJax.startup.defaultReady();
    }
  }
}

in case you want to try that out.

Technically, it applies to any character whose math class is 7, which also includes Greek symbols like \alpha, \Gamma, etc.,

[Actually, I don't think it doesn't apply to the lower-case Greek letters, only the upper-case ones.  Because I knew that that is how \mathbf works, and I know that + is not class 7, it makes sense to me to use \mathbf{x + y} and not expect the + to be in bold.  But I understand that not everyone knows those details.]

my suggestion would be to combine into a single <mi> element any sequence of math Ord atoms with the following properties:...

MathJax's parsing does not produce TeX math lists, and so this characterization is not natural within MathJax.  It would require significant changes to the parser to be able to accomplish it, and while one might suggest trying to combine nodes returned in the mml variable above, that would be a rather fragile approach, as there is no indication of where the nodes came from, or if there was any spacing, etc.  So \mathbf{x y} would produce <mi>xy</mi> which seems inappropriate.
Can you give an example of where your algorithm would be needed (in place of the one I give above)?

This doesn't apply to things like \mathcal, \mathfrak or \mathbb ...

I'm a bit concerned about the inconsistency of having some of these macros combine characters into one <mi> and other not.  I'm not sure I buy the argument about the Math Alphanumerics block, because MathML doesn't have separate text and math fonts.  That is, <mi mathvariant="bold">A</mi> is supposed to be treated identically to <mi>𝐀</mi>, and so when \mathbf{ABC} produces <mi mathvariant="bold">ABC</mi>, it is also producing values in the Math Alphanumeric block.  Why should that be different for any other characters in that block.  Why shouldn't \mathbb{ABC} produce <mi mathvariant="double-struck">ABC</mi> which is equivalent to <mi>𝔸𝔹ℂ</mi> (since double-struck C is in the Letterlike Symbols block, not the Math Alphanumerics)?
TeX doesn't have a separate "math bold" and "text bold" font (they are labeled "text fonts" in Appendix F of the TeXbook, so I guess Knuth considered them text fonts); the only distinction is between math italics (cmmi) and text italics (cmit).  While Unicode does have a distinction (the text font being in the usual ASCII range and the math font in the Math Alphanumerics block), MathML doesn't give a natural means of accessing the text versions (as I describe above).  So while TeX thinks of bold as a text font, MathML thinks of it as a math font.
Similarly, MathJax doesn't have separate text and math fonts, except for italics, where it uses a special internal math variant to handle the text italics.  So when you use \mathbf{ABC} you will be getting the Math Alphanumeric versions.

Here is a configuration that implements the proposal I made above:
MathJax = {
  tex: {packages: {'[+]': ['math-fonts']}},
  startup: {
    ready() {
      //
      //  These would be replaced by import commands if you wanted to make
      //  a proper extension.
      //
      const {Configuration} = MathJax._.input.tex.Configuration;
      const {CommandMap} = MathJax._.input.tex.SymbolMap;
      const BaseMethods = MathJax._.input.tex.base.BaseMethods.default;
      const TexParser = MathJax._.input.tex.TexParser.default;

      //
      //  Remap \mathrm, etc. to be able to create single <mi> elements
      //
      new CommandMap('math-fonts', {
        mathrm: ['MathFont', 'normal'],
        mathbf: ['MathFont', 'bold'],
        mathit: ['MathFont', '-tex-mathit'],  // internal variant for text italic font
        mathsf: ['MathFont', 'sans-serif'],
        mathtt: ['MathFont', 'monospace']
      }, {
        MathFont(parser, name, variant) {
          const text = parser.GetArgument(name);
          //
          //  Check if the argument is a string of letters only
          //     Make a single <mi> of them if so, otherwise
          //     Parse the argument as normal.
          //
          if (text.match(/^[a-z]+$/i)) {
            parser.Push(parser.create('token', 'mi', {mathvariant: variant}, text));
          } else {
            let mml = new TexParser(text, {...parser.stack.env, font: variant}, parser.configuration).mml();
            if (mml.isKind('inferredMrow')) {
              mml = parser.create('node', 'mrow', mml.childNodes);
            }
            parser.Push(mml);
          }
        }
      });
      Configuration.create('math-fonts', {
        handler: {macro: ['math-fonts']}
      });

      MathJax.startup.defaultReady();
    }
  }
}

in case you want to try that out.

Technically, it applies to any character whose math class is 7, which also includes Greek symbols like \alpha, \Gamma, etc.,

[Actually, I don't think it doesn't apply to the lower-case Greek letters, only the upper-case ones.  Because I knew that that is how \mathbf works, and I know that + is not class 7, it makes sense to me to use \mathbf{x + y} and not expect the + to be in bold.  But I understand that not everyone knows those details.]

my suggestion would be to combine into a single <mi> element any sequence of math Ord atoms with the following properties:...

MathJax's parsing does not produce TeX math lists, and so this characterization is not natural within MathJax.  It would require significant changes to the parser to be able to accomplish it, and while one might suggest trying to combine nodes returned in the mml variable above, that would be a rather fragile approach, as there is no indication of where the nodes came from, or if there was any spacing, etc.  So \mathbf{x y} would produce <mi>xy</mi> which seems inappropriate.
Can you give an example of where your algorithm would be needed (in place of the one I give above)?

This doesn't apply to things like \mathcal, \mathfrak or \mathbb ...

I'm a bit concerned about the inconsistency of having some of these macros combine characters into one <mi> and other not.  I'm not sure I buy the argument about the Math Alphanumerics block, because MathML doesn't have separate text and math fonts.  That is, <mi mathvariant="bold">A</mi> is supposed to be treated identically to <mi>𝐀</mi>, and so when \mathbf{ABC} produces <mi mathvariant="bold">ABC</mi>, it is also producing values in the Math Alphanumeric block.  Why should that be different for any other characters in that block.  Why shouldn't \mathbb{ABC} produce <mi mathvariant="double-struck">ABC</mi> which is equivalent to <mi>𝔸𝔹ℂ</mi> (since double-struck C is in the Letterlike Symbols block, not the Math Alphanumerics)?
TeX doesn't have a separate "math bold" and "text bold" font (they are labeled "text fonts" in Appendix F of the TeXbook, so I guess Knuth considered them text fonts); the only distinction is between math italics (cmmi) and text italics (cmit).  While Unicode does have a distinction (the text font being in the usual ASCII range and the math font in the Math Alphanumerics block), MathML doesn't give a natural means of accessing the text versions (as I describe above).  So while TeX thinks of bold as a text font, MathML thinks of it as a math font.
Similarly, MathJax doesn't have separate text and math fonts, except for italics, where it uses a special internal math variant to handle the text italics.  So when you use \mathbf{ABC} you will be getting the Math Alphanumeric versions.

Coincidentally, I've been rereading the unicode-math documentation, which I had forgotten spends quite a bit of space discussing exactly these issues.  See especially sections 3.1 and 4.4, but also parts of section 5 in http://mirrors.ctan.org/macros/unicodetex/latex/unicode-math/unicode-math.pdf.

Technically, it applies to any character whose math class is 7, which also includes Greek symbols like \alpha, \Gamma, etc.,

[Actually, I don't think it doesn't apply to the lower-case Greek letters, only the upper-case ones.

Yes, listing \alpha was a mistake.

Can you give an example of where your algorithm would be needed (in place of the one I give above)?

Needed?  No, as long as the user knows what they are doing and uses the commands carefully, I think your solution is probably sufficient.

This doesn't apply to things like \mathcal, \mathfrak or \mathbb ...

I'm a bit concerned about the inconsistency of having some of these macros combine characters into one <mi> and other not.

Yup.  It's a mess.  Damn Knuth for not anticipating Unicode when he designed TeX and the Computer Modern fonts in the mid 70s. :)

Similarly, MathJax doesn't have separate text and math fonts, except for italics, where it uses a special internal math variant to handle the text italics. So when you use \mathbf{ABC} you will be getting the Math Alphanumeric versions.

Interesting.  I don't think I knew that.  FWIW, here's what various alphabets give you by default:
\documentclass{article}

\usepackage{unicode-math}
\setmainfont{STIX Two Text}
\setmathfont{STIX Two Math}

\loggingoutput

\begin{document}

\textbf{a}          % STIXTwoText U+0061

$a$                 % STIXTwoMath U+1D44E

$\mathrm{a}$        % STIXTwoMath U+0061

$\mathbf{a}$        % cmbx10 U+0061 (surely a bug)

$\symbf{a}$         % STIXTwoMath U+1D41A

\end{document}

Like I said, it's a mess.

Coincidentally, I've been rereading the unicode-math documentation, which I had forgotten spends quite a bit of space discussing exactly these issues.  See especially sections 3.1 and 4.4, but also parts of section 5 in http://mirrors.ctan.org/macros/unicodetex/latex/unicode-math/unicode-math.pdf.

Technically, it applies to any character whose math class is 7, which also includes Greek symbols like \alpha, \Gamma, etc.,

[Actually, I don't think it doesn't apply to the lower-case Greek letters, only the upper-case ones.

Yes, listing \alpha was a mistake.

Can you give an example of where your algorithm would be needed (in place of the one I give above)?

Needed?  No, as long as the user knows what they are doing and uses the commands carefully, I think your solution is probably sufficient.

This doesn't apply to things like \mathcal, \mathfrak or \mathbb ...

I'm a bit concerned about the inconsistency of having some of these macros combine characters into one <mi> and other not.

Yup.  It's a mess.  Damn Knuth for not anticipating Unicode when he designed TeX and the Computer Modern fonts in the mid 70s. :)

Similarly, MathJax doesn't have separate text and math fonts, except for italics, where it uses a special internal math variant to handle the text italics. So when you use \mathbf{ABC} you will be getting the Math Alphanumeric versions.

Interesting.  I don't think I knew that.  FWIW, here's what various alphabets give you by default:
\documentclass{article}

\usepackage{unicode-math}
\setmainfont{STIX Two Text}
\setmathfont{STIX Two Math}

\loggingoutput

\begin{document}

\textbf{a}          % STIXTwoText U+0061

$a$                 % STIXTwoMath U+1D44E

$\mathrm{a}$        % STIXTwoMath U+0061

$\mathbf{a}$        % cmbx10 U+0061 (surely a bug)

$\symbf{a}$         % STIXTwoMath U+1D41A

\end{document}

Like I said, it's a mess.

$\mathbf{a}$        % cmbx10 U+0061 (surely a bug)

Of course, that was the one I really wanted to see.  :-)

I've been rereading the unicode-math documentation

That link was very useful, thank you.  I'm thinking about how best to incorporate that information into MathJax.

I can imagine someone wanting to using an identifier with letters outside of the ASCII range

Absolutely.  I hard coded the pattern in the example above, but it would be a configurable value if it is to be included in Mathjax itself, so those using other languages could include the characters they need.

$\mathbf{a}$        % cmbx10 U+0061 (surely a bug)

Of course, that was the one I really wanted to see.  :-)

I've been rereading the unicode-math documentation

That link was very useful, thank you.  I'm thinking about how best to incorporate that information into MathJax.

I can imagine someone wanting to using an identifier with letters outside of the ASCII range

Absolutely.  I hard coded the pattern in the example above, but it would be a configurable value if it is to be included in Mathjax itself, so those using other languages could include the characters they need.

$\mathbf{a}$        % cmbx10 U+0061 (surely a bug)

Of course, that was the one I really wanted to see. :-)

To be clear, the bug is that it was using cmbx10, not that it was mapping the character to U+0061; that's the expected behaviour from the documentation.
It did inspire me to make a more thorough catalog of the math alphabets supported by the unicode-math package, though.  Here's the result: mathalpha.pdf.  It makes for interesting if somewhat maddening reading.

I've been rereading the unicode-math documentation

That link was very useful, thank you. I'm thinking about how best to incorporate that information into MathJax.

@pkra and I have been working on a MathJax extension to support the unicode-math package.  It's in a private repo at the moment, but we hope to make a beta version public soon.  Maybe that would be a good place to experiment with the math alphabet support?

$\mathbf{a}$        % cmbx10 U+0061 (surely a bug)

Of course, that was the one I really wanted to see. :-)

To be clear, the bug is that it was using cmbx10, not that it was mapping the character to U+0061; that's the expected behaviour from the documentation.
It did inspire me to make a more thorough catalog of the math alphabets supported by the unicode-math package, though.  Here's the result: mathalpha.pdf.  It makes for interesting if somewhat maddening reading.

I've been rereading the unicode-math documentation

That link was very useful, thank you. I'm thinking about how best to incorporate that information into MathJax.

@pkra and I have been working on a MathJax extension to support the unicode-math package.  It's in a private repo at the moment, but we hope to make a beta version public soon.  Maybe that would be a good place to experiment with the math alphabet support?

Volker pointed out to me a suggestion for how to do something more like what you have suggested in terms of grouping multiple letters together.  Here is an implementation for that:
MathJax = {
  tex: {packages: {'[+]': ['math-fonts']}},
  startup: {
    ready() {
      //
      //  These would be replaced by import commands if you wanted to make
      //  a proper extension.
      //
      const {Configuration} = MathJax._.input.tex.Configuration;
      const {CommandMap, RegExpMap} = MathJax._.input.tex.SymbolMap;
      const TexParser = MathJax._.input.tex.TexParser.default;
      const ParseMethods = MathJax._.input.tex.ParseMethods.default;

      new RegExpMap('multi-letter', function (parser, c) {
        if (parser.stack.env.multiLetterIdentifiers) {
          c = parser.string.substr(parser.i-1).match(/^[a-z]+/i)[0];
        }
        ParseMethods.variable(parser, c);
        parser.i += c.length - 1;
      }, /[a-z]/i);

      new CommandMap('math-fonts', {
        mathrm: ['MathFont', 'normal'],
        mathbf: ['MathFont', 'bold'],
        mathit: ['MathFont', '-tex-mathit'],  // internal variant for text italic font
        mathsf: ['MathFont', 'sans-serif'],
        mathtt: ['MathFont', 'monospace']
      }, {
        MathFont(parser, name, variant) {
          const text = parser.GetArgument(name);
          const old = parser.stack.env.multiLetterIdentifiers;
          parser.stack.env.multiLetterIdentifiers = true;
          let mml = new TexParser(text, {...parser.stack.env, font: variant}, parser.configuration).mml();
          if (!old) {
            delete parser.stack.env.multiLetterIdentifiers;
          }
          if (mml.isKind('inferredMrow')) {
            mml = parser.create('node', 'mrow', mml.childNodes);
          }
          parser.Push(mml);
        }
      });
      const mathFonts = Configuration.create('math-fonts', {
        handler: {
          character: ['multi-letter'],
          macro: ['math-fonts'],
        }
      });

      MathJax.startup.defaultReady();
    }
  }
}

This adds a character map that (conditionally) turns on multi-character identifiers, so that within \mathbf{} and the others, multiple letters will be combined into a single identifier, while still processing everything else as normal.  So \mathrm{area = length \times width} would produce
<mi mathvariant="normal">area</mi>
<mo mathvariant="normal">=</mo>
<mi mathvariant="normal">length</mi>
<mo mathvariant="normal">×</mo>
<mi mathvariant="normal">width</mi>

This is slightly different from what you suggest, in that \mathrm{inch^3} will produce
<msup>
  <mi mathvariant="normal">inch</mi>
  <mn>3</mn>
</msup>

rather than the
<mi mathvariant="normal">inc</mi>
<msup>
  <mi mathvariant="normal">h</mi>
  <mn>3</mn>
</msup>

that your algorithm would produce, and something like
\newcommand{\a}{a}
\mathbf{a\a}

will produce
<mi mathvariant="normal">a</mi>
<mi mathvariant="normal">a</mi>

rather than
<mi mathvariant="normal">aa</mi>

that your approach would produce.
Anyway, it turns out that your area example can be handled reasonably.

Volker pointed out to me a suggestion for how to do something more like what you have suggested in terms of grouping multiple letters together.  Here is an implementation for that:
MathJax = {
  tex: {packages: {'[+]': ['math-fonts']}},
  startup: {
    ready() {
      //
      //  These would be replaced by import commands if you wanted to make
      //  a proper extension.
      //
      const {Configuration} = MathJax._.input.tex.Configuration;
      const {CommandMap, RegExpMap} = MathJax._.input.tex.SymbolMap;
      const TexParser = MathJax._.input.tex.TexParser.default;
      const ParseMethods = MathJax._.input.tex.ParseMethods.default;

      new RegExpMap('multi-letter', function (parser, c) {
        if (parser.stack.env.multiLetterIdentifiers) {
          c = parser.string.substr(parser.i-1).match(/^[a-z]+/i)[0];
        }
        ParseMethods.variable(parser, c);
        parser.i += c.length - 1;
      }, /[a-z]/i);

      new CommandMap('math-fonts', {
        mathrm: ['MathFont', 'normal'],
        mathbf: ['MathFont', 'bold'],
        mathit: ['MathFont', '-tex-mathit'],  // internal variant for text italic font
        mathsf: ['MathFont', 'sans-serif'],
        mathtt: ['MathFont', 'monospace']
      }, {
        MathFont(parser, name, variant) {
          const text = parser.GetArgument(name);
          const old = parser.stack.env.multiLetterIdentifiers;
          parser.stack.env.multiLetterIdentifiers = true;
          let mml = new TexParser(text, {...parser.stack.env, font: variant}, parser.configuration).mml();
          if (!old) {
            delete parser.stack.env.multiLetterIdentifiers;
          }
          if (mml.isKind('inferredMrow')) {
            mml = parser.create('node', 'mrow', mml.childNodes);
          }
          parser.Push(mml);
        }
      });
      const mathFonts = Configuration.create('math-fonts', {
        handler: {
          character: ['multi-letter'],
          macro: ['math-fonts'],
        }
      });

      MathJax.startup.defaultReady();
    }
  }
}

This adds a character map that (conditionally) turns on multi-character identifiers, so that within \mathbf{} and the others, multiple letters will be combined into a single identifier, while still processing everything else as normal.  So \mathrm{area = length \times width} would produce
<mi mathvariant="normal">area</mi>
<mo mathvariant="normal">=</mo>
<mi mathvariant="normal">length</mi>
<mo mathvariant="normal">×</mo>
<mi mathvariant="normal">width</mi>

This is slightly different from what you suggest, in that \mathrm{inch^3} will produce
<msup>
  <mi mathvariant="normal">inch</mi>
  <mn>3</mn>
</msup>

rather than the
<mi mathvariant="normal">inc</mi>
<msup>
  <mi mathvariant="normal">h</mi>
  <mn>3</mn>
</msup>

that your algorithm would produce, and something like
\newcommand{\a}{a}
\mathbf{a\a}

will produce
<mi mathvariant="normal">a</mi>
<mi mathvariant="normal">a</mi>

rather than
<mi mathvariant="normal">aa</mi>

that your approach would produce.
Anyway, it turns out that your area example can be handled reasonably.

Thank you for this.  I haven't had a chance to take a close look at it or try it out yet, but I wanted to comment on this part:

This is slightly different from what you suggest, in that \mathrm{inch^3} will produce
<msup>
  <mi mathvariant="normal">inch</mi>
  <mn>3</mn>
</msup>

rather than the
<mi mathvariant="normal">inc</mi>
<msup>
  <mi mathvariant="normal">h</mi>
  <mn>3</mn>
</msup>

that your algorithm would produce,

If that's what my algorithm would produce, my algorithm was clearly wrong.

and something like
\newcommand{\a}{a}
\mathbf{a\a}

will produce
<mi mathvariant="normal">a</mi>
<mi mathvariant="normal">a</mi>

rather than
<mi mathvariant="normal">aa</mi>

that your approach would produce.

Fair enough.  That's weird enough that I'm not too worried about how it comes out.

I've made a PR to implement the solution above, and added the remaining \math* and the \sym* macros.  This allows easy access from TeX to all the MathML variants, which was not the case before.


ipam: Add exponential backoff when pool maintanance fails

When pool maintenance fails, the pool maintenance trigger is triggered
such that the logic is executed again. However, if maintenance fails for
example because of external reasons, the default retry interval of 10
milliseconds is way to short. Especially if the cloud provider API is
overloaded, multiple nodes can be stuck in a 10 millisecond retry loop,
which will make the situation even worse.
Therefore, this commit introduces an exponential backoff if the pool
maintenance function fails with an error. The minimum trigger interval
remains 10 milliseconds to allow for other trigger reasons (e.g.
because of a resync) to not be delayed as long as the node is healthy.

/test

Failing CI pipelines are not required. Marking ready-to-merge.


Improving pre-SMT reasoning capabilities

In the recent proofs, I have often observed the following pattern of reasoning:

We need C1: X <Int pow256 for a rule/simplification to fire;
We have C2: X <Int pow96, say, in the path constraints, so C1 holds;
Our simplifications aren't powerful enough to understand that C1 holds, so rule/simplification does not get applied in Booster and then (I think?) we either fall back to Kore which uses Z3 (if we're dealing with a rule) or we do nothing (if we're dealing with a simplification).

I believe that we could/should take care of these cases before they reach the SMT solver, and I would like to understand/discuss how this could be done modularly and where it should be done.
One way of doing this for this specific case would be to write simplifications in which I could tell the engine to syntactically match the path condition, rather than semantically, along the lines of:
rule A <Int B => true
  requires A <Int C
   andBool C <=Int B
  [simplification, concrete(B, C), match(C)]

where match(C) would mean 'match the path constraints syntactically to learn C, possibly also specifying which conjunct of the requires clause to use for the match (match(C, 1), for instance). In this example, we couldn't use C <=Int B anyway because both B and C are concrete, so that constraint would be fully resolved.

I think we should have C actually be ?C, because it's existentially quantified here. "If there exists a ?C such that A <Int ?C andBool ?C <=Int B, then A <Int B => true".
Because of that, I think we shouldn't need the match(C) attribute either. The backend can see that there is an existential variable in the requires clause, and so it can know that it must:

Process the side-conditions in the requires clause in order,
Attempt to find a value for ?C by looking for syntactic exact matches of the first clause (possibly coming up with multiple options),
Filter the options by seeing if later clauses are also satisfied by those matches.

This can be a very specific reasoning module for just this case, only when dealing with side-conditions.

Yes, ?C would work, I was trying to avoid it because I think that would require front-end parsing changes, since currently ?C is not parseable on the LHS of a rule.

That's fine, we can change the frontend to allow it for simplification rules. We need to make sure not to send it to the backend.
More important that the rule reads correctly. We don't need another way to say Exists, the ? already does that.

We have had an discussion of this proposal in yesterdays Kontrol<>HB meeting, here's the summary of what I recall:

@PetarMax suggest this feature to enable interval analysis for integer inequalities. His hypothesis is that if we implement this, than Z3 would have an easier job or would not be needed a all in some cases.
I think that this feature would not actually help much, as we rely on the SMT solver for this sort of things. If we see that the solver struggles in some cases, we need to isolate the exact case where it does and debug the SMT transalation of this specific predicate in Booster and Kore.
Only when we are absolutely sure that the SMT translation is not the problem and the solver indeed struggles should we implement ad-hoc reasoning modules.

The reason I am somewhat hostile to the idea of adding ad-hoc reasoning procedures as part of the requires clause check (or anywhere else) is that these are very difficult to maintain. We know that Kore has a number of such procedures that are essentially a black box for us as the backend developers. Deciphering and perhaps refactoring them is certainly possible, but would require significant resources.
tl;dr
We have the SMT solver for this sort of thing, both in Booster and in Kore. If we see examples where either Booster or Kore fails to apply simplifications due to unknown predicates, we should drill into why these are unknwon. There may be holes in Booster's SMT interface and hopes+historical baggage in Kore's. We should identify and fix those first, and only then extend K with additional rule types that hook into the backend.
@goodlyrottenapple please add any points from the discussion that I have forgotten to mention.

I understand the concern about ad-hoc reasoning modules, especially in the context of Kore history. I do believe that will not have the same issues if we document precisely what is being done and implement reasoning modules modularly, with the ability to be switched on and off.
I think the question I would like to have an answer to is: is Z3 called in Booster when trying to determine if a condition of a requires clause holds? If yes, then I'm surprised there's a fallback to Kore for transitivity of <Int. If not, why not?

This is not ad-hoc, it's very specific, and has come up as a soft ask several times over the years. Basically, what we want is existential binding against things that already exist in the side-conditions. It's easy to see how this reasoning works, and also easy for the backend to see when it should apply this reasoning (are there simplification rules with existentials in the requires? Then try applying this when evaluating side-conditions).
I disagree fully with this being ad-hoc. Improving SMT translation is just making the backend better at "magic", but not giving the user more power here.

The proposal has been refined and a new issue in the K repo has been opened. Let us use this issue to track the implementation of this feature in Booster.
I have a question regarding what to do when we have multiple matches.
Let's have a look at a happy-path scenario when we have two matches, but only one is actually non-False:
rule [test1-init]: <k> test1Init() => test1State1() ... </k>
                   <int-x> _         => ?X              </int-x>
                   <int-y> _         => ?Y              </int-y>
                   ensures  ?X <Int 100
                    andBool ?Y <Int 50
                    andBool ?X <Int ?Y

rule [test1-1-2]: <k> test1State1() => test1State2() ... </k>
                  <int-x> X                              </int-x>
                  requires X <Int 50

rule [test1-simplify]:
    X <Int Y => true
    requires X <Int Z
     andBool Z <Int Y
     // [simplification, syntactic(1)]
     [simplification]

In order to apply rule test1-1-2 to the current pattern
[booster][execute][term 4dea6a7][rewrite 0bf6c1a][success][term 40a330c][kore-term]
<generatedTop>(<k>(kseq(test1State1()_SYNTACTIC-SIMPLIFICATION_State(), dotk())), <int-x>(Var?X0:SortInt{}), <int-y>(Var?Y0:SortInt{}), <generatedCounter>("0"))
/\ _<Int_(Var?X0:SortInt{}, "100")
/\ _<Int_(Var?X0:SortInt{}, Var?Y0:SortInt{})
/\ _<Int_(Var?Y0:SortInt{}, "50")

we need to prove
[booster][execute][term 40a330c][rewrite 18e3545][constraint][term fde6171][kore-term] _<Int_(Var?X0:SortInt{}, "50")

we match with the simplification test1-simplify and now we need to instantiate the free variable Z in its requires clause:
[booster][execute][term 40a330c][rewrite 18e3545][constraint][term fde6171][simplification c9b0e63][failure][continue]
Aborting syntactic simplification after instantiating the condition to
_<Int_(Var?X0:SortInt{}, Eq#VarZ:SortInt{}) , _<Int_(Eq#VarZ:SortInt{}, "50")

The question is: how do we pick the instantiation? In this particular case, it's relatively easy, as we only have two options and only one of them is valid:
We can substitute Eq#VarZ for "100"
_<Int_(Var?X0:SortInt{}, "100") , _<Int_("100", "50")

which the LLVM backbend will discharge as false, no Z3 needed.
Or we can  substitute Eq#VarZ for Var?Y0
_<Int_(Var?X0:SortInt{}, Var?Y0:SortInt{}) , _<Int_(Var?Y0:SortInt{}, "50")

in this case, we get two clauses that are syntactically present in the constraints — we can apply test1-simplify and rewrite with test1-1-2.
The amount of matches will however grow exponentially with the amount of constraints, and we will likely start getting multiple non-False instantiations.
How would we choose which one to use?

I would apply first successful and do not cache.


while creating delta pkg .forceignore not getting copied to working directory

Create Delta Package based on two commits task is failing with below error while delta package creation
##[error]ENOENT: no such file or directory, open 'portal_src_delta/.forceignore'

@ramesh-karanji  This issues is triaged and is an issue with sfpowerkit, I will raise an issue in that repo

@manisfdcsfdx  fyi


Error Matcher module

Extract the task catcher code into a module.
Generalize exceptions per https://states-language.net/#errors
Discussion points:

Do we want to use a boolean for all so we don't need to error_equals.include?("States.ALL")?
Do we want to error_equals.add("States.HeartbeatTimeout") if error_equals.include?("States.Timeout"). This will remove that special case.

I think your branch name and PR title are wrong?

update:

fixed note
fixed specs (was not properly comparing equals_value)

un-wip: tests passing

update:

added error_equals validation

update:

moved error_equals up to parent - no longer the Note stuff
added "States.ALL" special case
generalized Heartbeat special case
specs for ErrorMatcher

update:

rewrote tests to go through workflow
added tests for validation
now checks that retrier (and catch) are non-empty

update:

fixed tap in tests

update:

fix code complexity via validate_array_field!

update:

compressed _catch_all case to convert guard into just a boolean ||

update:

rebased
depend upon common error handling

update:

rebase (fixed conflict)
rubocop: fixed spacing

update:

based upon #209
Greatly simplified the logic in here.

update:

remove 209 - trimmed it down

update:

rubocop

update:

extracted #212 (few pushes back) to make this as simple as I can

update:

rebase master

@agrare @Fryguy this has been greatly simplified. Is this good to go now?


Rbac Engine matchers

This PR is part of gRPC authorization effort. It does the following -

Implements matchers for RBAC based authorization engine.
Adds required getters to EvaluateArgs, and updates existing endpoint getters to handle IPv6 format as well.
Renames existing Cel engine implementation, to avoid conflict with Rbac engine in future PRs.

This change is

@ZhenLian @jiangtaoli2016 @markdroth Friendly ping!:)

@markdroth @ZhenLian PTAL. Thank you!

@markdroth @ZhenLian Friendly Ping!:)

Thanks for the review, Jiangtao!:)


sensu-install error on CentOS 7

Trying to follow docs and set up Sensu on two VMs, a server/client, and a client. In both instances, they're minimal CentOS 7 installs, set up as per the Sensu docs. However, when trying to set up checks, and install a plugin via sensu-install, the download of the plugin fails with:
[SENSU-INSTALL] installing Sensu plugins ...
[SENSU-INSTALL] determining if Sensu gem 'sensu-plugins-process-checks' is already installed ...
false
[SENSU-INSTALL] Sensu plugin gems to be installed: ["sensu-plugins-process-checks"]
[SENSU-INSTALL] installing Sensu gem 'sensu-plugins-process-checks'
ERROR:  While executing gem ... (ArgumentError)
    IPv4 address expects 4 bytes but 0 bytes
[SENSU-INSTALL] failed to install Sensu gem 'sensu-plugins-process-checks'
[SENSU-INSTALL] you can run the sensu-install command again with --verbose for more info
[SENSU-INSTALL] please take note of any failure messages above
[SENSU-INSTALL] make sure you have build tools installed (e.g. gcc)
[SENSU-INSTALL] trying to determine the Sensu plugin homepage for sensu-plugins-process-checks ...
ERROR:  While executing gem ... (ArgumentError)
    IPv4 address expects 4 bytes but 0 bytes

This happens on both machines; couldn't find any similar bug-posting. Based on reading https://github.com/rubygems/rubygems/issues/1274, I applied a similar fix to the embedded ruby, by editing host = uri.host to host = uri in /opt/sensu/embedded/lib/ruby/site_ruby/2.3.0/rubygems/remote_fetcher.rb . This seems to fix the problem, at least until an RPM update which wipes the change out (which I noticed today). For testing, I did switch SELinux off, but it seemed to have no effect.
Expected Behavior
Plugins install without issue via sensu-install.
Current Behavior
Currently, the installs bomb out with the noted error message above.
Possible Solution
Not so much a solution, but the workaround above seems to resolve the issue until an update.
Steps to Reproduce (for bugs)

Do a default install of CentOS 7, fully patch/reboot.
Install and config Sensu based on docs.
Use sensu-install -p {PLUGIN NAME}, hit error

Context
As mentioned in the summary, trying to get a testing/dev environment off the ground for eval purposes.
Your Environment

Operating System and version (e.g. Ubuntu 14.04): CentOS Linux release 7.2.1511 (Core)
Transport and runtime versions (e.g. RabbitMQ 3.6.2 on Erlang 18.2): rabbitmq-server-3.6.5-1 on erlang-R16B-03.17

Sensu 0.26.5-2

Hi @adam820, thanks for opening this issue. I've opened https://github.com/sensu/sensu-omnibus/issues/44 to track this in the repo where we integrate Ruby, Rubygems and so on. Once there's an upstream fix we'll be sure to pull it in.


Lambda soes not support making

Hello,
Thanks for sharing you experience on this subject.
I've got an issue running insurance_qa_eval.py. Here's the full stacktrace  :
Traceback (most recent call last):   File "insurance_qa_eval.py", line 274, in <module>     model.compile(optimizer=optimizer)   File "/home/myuser/tests/insurance_qna/keras-language-modeling/keras_models.py", line 114, in compile     qa_model = self.get_qa_model()   File "/home/myuser/tests/insurance_qna/keras-language-modeling/keras_models.py", line 101, in get_qa_model     self._models = self.build()   File "/home/myuser/tests/insurance_qna/keras-language-modeling/keras_models.py", line 279, in build     question_pool = merge([maxpool(question_f_dropout), maxpool(question_b_dropout)], mode='concat', concat_axis=-1)   File "/usr/local/lib/python3.4/dist-packages/keras/engine/topology.py", line 485, in __call__     self.add_inbound_node(inbound_layers, node_indices, tensor_indices)   File "/usr/local/lib/python3.4/dist-packages/keras/engine/topology.py", line 543, in add_inbound_node     Node.create_node(self, inbound_layers, node_indices, tensor_indices)   File "/usr/local/lib/python3.4/dist-packages/keras/engine/topology.py", line 149, in create_node     output_masks = to_list(outbound_layer.compute_mask(input_tensors[0], input_masks[0]))   File "/usr/local/lib/python3.4/dist-packages/keras/engine/topology.py", line 578, in compute_mask     'but was passed an input_mask: ' + str(input_mask)) Exception: Layer lambda_1 does not support masking, but was passed an input_mask: Elemwise{neq,no_inplace}.0
I'm using

Keras 1.0.2
Thenao 0.8.2

Thanks

It is a small change in the Keras source code (set the supports_masking class variable in the Lambda layer to True instead of False). Otherwise there isn't a way to do this. Masking isn't really necessary though.

You could also adapt the attribute of the Lambda Layer, by adding the line
maxpool.__setattr__('supports_masking',True)
But both changes raise a new error
Exception: Merge does not support masking, but was passed an input mask
caused by the line
 question_pool = merge([maxpool(question_f_dropout), maxpool(question_b_dropout)], mode='concat', concat_axis=-1)

ah yeah merge doesn't support masking yet, you can use this pull request -> https://github.com/fchollet/keras/pull/2413

Ok thanks :)

Hi @codekansas,
Thanks for this promising package. I look forward to testing it.
Even once I use the pull request you mention above, i get an error. Any ideas?
I typed the following command:
python insurance_qa_eval.py
I get this output:
Epoch 1 :: 2016-05-13 04:23:07 :: Train on 18540 samples, validate on 3708 samples
Epoch 1/1
Traceback (most recent call last):
File "insurance_qa_eval.py", line 289, in
evaluator.train(model)
File "insurance_qa_eval.py", line 127, in train
hist = model.fit([questions, good_answers, bad_answers], nb_epoch=1, batch_size=batch_size, validation_split=split)
File "/Users/username/Documents/keras-language-modeling/keras_models.py", line 132, in fit
return self.training_model.fit(x, y, **kwargs)
File "/Users/username/anaconda/lib/python2.7/site-packages/keras/engine/training.py", line 1011, in fit
callback_metrics=callback_metrics)
File "/Users/username/anaconda/lib/python2.7/site-packages/keras/engine/training.py", line 749, in _fit_loop
outs = f(ins_batch)
File "/Users/username/anaconda/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 513, in call
return self.function(*inputs)
File "/Users/username/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 786, in call
allow_downcast=s.allow_downcast)
File "/Users/username/anaconda/lib/python2.7/site-packages/theano/tensor/type.py", line 177, in filter
data.shape))
TypeError: ('Bad input argument to theano function with name "/Users/username/anaconda/lib/python2.7/site-packages/keras/backend/theano_backend.py:509"  at index 3(0-based)', 'Wrong number of dimensions: expected 1, got 2 with shape (128, 1).')
Thanks for your help!

Oh I'm not sure. To be honest, I've been messing around with stuff a lot and it's far from cohesive. I can't tell much from the error report... You might try cloning again. I'm not going to be able to work on this much until June, unfortunately.

Got it. Thanks for your quick response! I'll read the pull request comments a bit more carefully to see if i can make it work. Good luck with your projects!

@migueljette Any update? I have met the same error with the pull request. I am now trying to make it work as well.

@eshijia Unfortunately, no. I had to put this aside for a few weeks as well. I was hoping to test it out quickly, and I didn't have any time to debug it at all. Do you have ideas for debugging?

@migueljette Yes! After git reset with a few specific commits, I found some problems. In the LanguageModel class (keras_model.py), @codekansas had changed the line 107 from
qa_model = merge([question_output, answer_output], mode=similarity, output_shape=lambda x: x[:-1])
to
qa_model = merge([question_output, answer_output], mode=similarity, output_shape=lambda x: x[0][:-1]).
If you don't change that, you will make the insurance_qa_eval.py work.
I am still reading and running the code to verify some results. Maybe @codekansas can give more explanations when he works on this again.

Hi @eshijia ! Thanks for the reply! That totally works!! Now I have to test the model and framework with my own data. Ha ha! Thanks again. I hope i have a bit more time soon to play with this.

@migueljette Cool! In my experiments with my own data, the basic 'cosine' similarity metric bring the best result. In addition, I haven't reproduced the results with Insurance_QA  as the blog mentioned. Hope @codekansas can share the model parameter settings of his results.

I didn't reproduce the results from the paper, I'm not sure how they got the results they did. I wondered if they just let it train for a really long time.

Thanks for the reply! I mean to reproduce the results of your own experiments which mentioned in the blog, not the results from the paper.

Oh sure! Hmm... The major thing was using a bunch of dimensions, I found 1000 embedding dimensions worked well (no word2vec pretraining). I think this commit has the right parameters (cosine similarity, margin of 0.009 maybe?) I found that varying the margin from 0.009 - 0.2 was a good range, changing similarity might help... It is pretty fast to train on my GPU, so you can try different sets of parameters, but I think those parameters will give you similar results to what I got after ~100 epochs maybe (unfortunately I don't have my desktop with me right now so I can't do it myself)

OK, Got it! I am training the model on my GPU. I will share my results in the future! Thanks again!

Hi, I have changed my keras installation as stated in: https://github.com/fchollet/keras/pull/2413
after installing keras 1.03.
And then I change the following line in keras_model.py:
qa_model = merge([question_output, answer_output], mode=similarity, output_shape=lambda x: x[0][:-1])
back to:
qa_model = merge([question_output, answer_output], mode=similarity, output_shape=lambda x: x[:-1])
Now I am getting this error:
Epoch 1 :: 2016-06-06 08:27:43 :: Traceback (most recent call last):
File "insurance_qa_eval.py", line 291, in
evaluator.train(model)
File "insurance_qa_eval.py", line 130, in train
hist = model.fit([questions, good_answers, bad_answers], nb_epoch=1, batch_size=batch_size, validation_split=split)
File "/home/ubuntu/keras-language-modeling/keras_models.py", line 132, in fit
return self.training_model.fit(x, y, **kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/Keras-1.0.3-py2.7.egg/keras/engine/training.py", line 994, in fit
batch_size=batch_size)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/Keras-1.0.3-py2.7.egg/keras/engine/training.py", line 925, in _standardize_user_data
exception_prefix='model target')
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/Keras-1.0.3-py2.7.egg/keras/engine/training.py", line 104, in standardize_input_data
str(array.shape))
Exception: Error when checking model target: expected merge_4 to have shape (None, 282) but got array with shape (18540, 1)
What have I done wrong? I notice the change in Keras source involves changes in the test folder. Do I need to replace the merge by the merge in the test file?
Thanks.

@wailoktam I think you should uninstall your keras installation, and re-install it by cloning the Keras repo of codekansas and running python setup.py install. You can create a virtual environment to do that.

Hi, I replace my Keras installation with what is found here:
https://github.com/codekansas/keras
Now it is yielding this warning:
/home/ubuntu/anaconda2/lib/python2.7/site-packages/Keras-1.0.1-py2.7.egg/keras/backend/theano_backend.py:509: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 3 is not part of the computational graph needed to compute the outputs: merge_4_target.
To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'.
**kwargs)
I have change the following line in keras_model.py:
qa_model = merge([question_output, answer_output], mode=similarity, output_shape=lambda x: x[0][:-1])
back to:
qa_model = merge([question_output, answer_output], mode=similarity, output_shape=lambda x: x[:-1])
following eshijia.
Can any body help?

Yes, I also meet this warning. But it has no effect. The code still works. I think the warning is due to the change of the pull request.

I get the error too, I'm not sure why it happens (has happened in other Keras projects). I will change the master branch back (changed it originally without testing it)

Hi, thanks for you guy's prompt response. I have not yet start checking the cause of the error. I once get such error for my own code because I define a custom objective function not in the way theano likes it. The error is always pointing to the last layer, no matter what it is. Once I change it to a default objective function, the warning disappears. It may be irrelevant this time though.

Hi, I am trying to reduce the size of the training set and reducing the training epochs such that I can speed up debugging. (I am looking for the cause of the warning) However, I find some puzzling codes.
In line 117, I find the following line, which I believe is for creating negative training samples:
 bad_answers = self.pada(random.sample(self.answers.values(), len(good_answers)))
I think there are some chance that the random sampling would actually get the correct answer.
In line 126, I find the parameters good_answers and bad_answers puzzling:
        ```hist = model.fit([questions, good_answers, bad_answers], nb_epoch=1, batch_size=batch_size, validation_split=split)```

It does not quite match the usage described in the Keras documentation given below:
fit(self, x, y, batch_size=32, nb_epoch=10, verbose=1, callbacks=[], validation_split=0.0, validation_data=None, shuffle=True, class_weight=None, sample_weight=None)
x: Numpy array of training data, or list of Numpy arrays if the model has multiple inputs. If all inputs in the model are named, you can also pass a dictionary mapping input names to Numpy arrays.
y: Numpy array of target data, or list of Numpy arrays if the model has multiple outputs. If all outputs in the model are named, you can also pass a dictionary mapping output names to Numpy arrays
I am not sue what bad_answers correspond to in this line.
Can anyone explain what is going on here? Many thanks.

Hmm... Line 117 is actually for creating negative training samples, and may get the correct answer, but it will not result in a bad training result after some epochs. (Of course, maybe we can find a better way to create negative samples)
As for line 126, I think you should understand more about the functional API of keras, not just the sequential API.

Thanks for your prompt response. I think the x in fit is the training sample whereas the y in fit is the label (1 for correct label and 0 for incorrect label) and x and y must have the same number of elements? The document does say either x or y can be a list. But now we get one list. I can see that the list correspond to x. Where is the array of labels for training?

I figure out how to do away with the warning about unused input:
Changing the following line:
self.training_model.compile(loss=lambda y_true, y_pred: y_pred, optimizer=optimizer, **kwargs)
to:
self.training_model.compile(loss=lambda y_true, y_pred: y_pred + y_true - y_true, optimizer=optimizer, **kwargs)
However, I don't quite get what exactly the first line is trying to do. I can see that the author is moving the calculation of the loss to a previous layer and this line is supposed to just pass the output of that layer.
If the author can explain the details, I would be extremely grateful. I am also looking for the codes that calculate the result given in results.note but they are not in the repository.

The loss function is a hinge loss, i.e. loss = margin - cos(question, good_answer) + cos(question, bad_answer) where cos is the cosine similarity between the two things. We're trying to minimize this value. Cosine similarity between two layers can be done with a merge operation. To fit this idea within the schema of Keras, we can do another merge to get the desired loss and call that value y_pred, which becomes the value we want to minimize.
We basically have two models, one for the question and one for the answer, and we're learning them at the same time to be close to each other rather than close to some predicted value. This isn't really what Keras was set up to do, since the usual way you characterize this is with a single model which learns to be close to a specified output.
Thanks for pointing out the bit about the unused inputs part, it makes sense now. The way I set it up has y_true always be zero (I think, I haven't looked at it in a while), so you could change it to just y_pred + y_true.

@eshijia Can you share the results you promised? :-)
I've trained the embedding model out of the box. I've got the following numbers, which are way below what is indicated in these notes:
Top-1 Precision: [0.18611111111111112, 0.18722222222222223, 0.184]
MRR: [0.2722904433042418, 0.26672897883551283, 0.26999670775205087]

Each model is highly dependent on the parameters you pass it and how long you train it for. I've messed around with them a lot and the parameters in the project aren't the ones I got the best performance for. For the Embedding model, I got the best performance by using around 1000 embedding dimensions and not pretraining the weights with Word2Vec. The model should be trained to minimized error on a validation set. The results you pasted are about what got after using a 100-dimension Embedding model I think (it's been a while since I looked at it).

I've retrained the embedding model with 1000 dimensions without loading the pertained w2v model. The numbers are only slightly better than what I had mentioned with the default setup:
Top-1 Precision: [0.18722222222222223, 0.1922222222222222, 0.206]
MRR: [0.28435607158749127, 0.2773795710911621, 0.2948951984190565]

Are the models sensitive to parameters tuning (e.g., dim = 100 vs 1000) more than to the architecture (e.g., Embedding vs Embedding + CNN)?
If someone can share their setup on how to get some reasonable top-1 accuracy (e.g., > 0.40), that would be really appreciated.

You can try this:
conf = {
    'question_len': 40,
    'answer_len': 40,
    'n_words': 22353, # len(vocabulary) + 1
    'margin': 0.05,

    'training_params': {
        'save_every': 1,
        'batch_size': 128,
        'nb_epoch': 1000,
        'validation_split': 0.2,
        'optimizer': 'adam',

        'evaluate_all_threshold': {
            'mode': 'all',
            'top1': 0.4,
        },
    },

    'model_params': {
        'n_embed_dims': 1000,
    }

    'similarity_params': {
        'mode': 'cosine',
    }
}

evaluator = Evaluator(conf)

model = EmbeddingModel(conf)
optimizer = conf.get('training_params', dict()).get('optimizer', 'adam')
model.compile(optimizer=optimizer)

evaluator.train(model)

Try tweaking the margin as well (usually 0.009 - 0.2). You can play around with it to try to get better results (e.g. the similarity type). evaluate_all_threshold triggers when to evaluate on the test set; in this case, if the validation accuracy for all the validation sets is above 0.4, then it will evaluate on the entire testing set.

Man, that helped. Thanks a lot!
----- test1 -----
Top-1 Precision: 0.518889
MRR: 0.632561
----- test2 -----
Top-1 Precision: 0.493889
MRR: 0.607853
----- dev -----
Top-1 Precision: 0.513000
MRR: 0.627165


Hotspot not working on 5 Ghz STA

Hi,
thank you very much for this script!
I tried running it a few times and adjusting it every time I did because I just could not get it to work.
In the end the problem was that before running the script I was connected to a 5 Ghz network on channl 48. It seems that either the script or the Pi does not support AP+STA with thi setup.
Is this a known issue or is there a fix?
Thank you for your great work!
Running on a Raspberry Pi 4 Model B with 8 GB.

Not wanting to mention the obvious, but did you try to set hw_mode=ag in the script?? See below:
# Use the 2.4GHz band (I think you can use in ag mode to get the 5GHz band as well, but I have not tested this yet)
#hw_mode=g
hw_mode=ag # commented the default g mode and enabled ag mode for 5 GHz WiFi

This might not work as, like the dev said, the feature has not been tested at all. Please let us know your mileage.

Hi
Not sure if this is issue is still unsolved.
I had a similar problem that my AP was not detectable after running the setup-network.sh after being connected to a 5 GHz network. I tried the solution proposed by sebascarra.

Setting "hw_mode=ag" did not work and resulted in an error during boot.
Setting "hw_mode=a" worked, though.


feat: save COGs with dask to azure

This PR introduces Azure Blob Storage support for saving COGs with Dask, complementing the existing AWS S3 support.

Implements a MultiPartUpload class in _az.py to handle Azure-specific multipart upload operations, including block staging, finalization, and cancellation.
Adds MultiPartUploadBase in _multipart.py to provide a shared interface for multipart uploads, applicable to both Azure and S3 implementations.
Updates save_cog_with_dask to recognize Azure URLs and integrate the Azure multipart upload workflow.
Adds unit tests in tests/test_az.py using mocked Azure SDK to verify functionality specific to Azure Blob Storage.

Here’s a simple test I ran to verify the Azure implementation:
import dask.distributed
import planetary_computer as pc
from odc.geo.cog._tifffile import save_cog_with_dask
from odc.stac import configure_rio, stac_load
from pystac_client import Client

# Setup Dask client for efficient processing
client = dask.distributed.Client()
configure_rio(cloud_defaults=True, client=client)

# Query STAC API for Sentinel-2 data
catalog = Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")
query = catalog.search(
    collections=["sentinel-2-l2a"],
    datetime="2019-06",
    query={"s2:mgrs_tile": dict(eq="06VVN")},
)
items = list(query.items())
print(f"Found {len(items)} datasets")

# Load Sentinel-2 data with specific bands
resolution = 40
xx = stac_load(
    items,
    bands=["SCL"],
    resolution=resolution,
    chunks={"x": 2048, "y": 2048},
    patch_url=pc.sign,
    dtype="uint16",
    nodata=0,
)

# Select a single time slice for saving
to_save = xx.SCL.isel(time=3)

# Azure Blob Storage details
blob_name = f"SCL-{to_save.time.dt.strftime('%Y%m%d').item()}.tif"
azure_url = f"az://{container}/{blob_name}"

result = save_cog_with_dask(
    to_save,
    dst=f"az://{container}/{blob_name}",
    compression="DEFLATE",
    predictor=2,
    overview_resampling="nearest",
    azure={
        "account_url": f"https://{storage_account}.blob.core.windows.net",
        "credential": sas_token,  # can be blob specific or container wide
    },
    blocksize=[512, 512],
)

# Compute the result to trigger the actual upload
result.compute()
print(f"COG successfully uploaded to: {azure_url}")

While implementing this, I noticed a couple of quirks in the original code:

Spelling Inconsistencies: Words like finalise vs. finalize were used. I  used ise rather than ize.
Typing Updates: The code inconsistently uses Dict/List from typing vs. built-ins (dict/list). I've changed it to Dict and List since the dev-env.yml specifies python=3.8 (but I hadn't used python 3.8 in ages).

As this is my first PR here, I look forward to feedback and suggestions for improvement!

@wietzesuijker thanks for the PR.
Can I ask you to undo dict -> Dict, list -> List changes, we no longer aim to support 3.8 going forward as most of our dependencies do not support it either. This also blows up review effort. When making changes like this please try to separate "clean up changes" from "actual changes" and use separate commits for each separate logical unit of work, example:

fix spelling
mypy harmonization
add doc comments
implement azure back-end

I'll need to find some time to review this properly and to remember what was the original plan for extending to other blob stores. At this point I'm not convinced we need a base class, but my memory of this code is hazy. In the meantime please

undo mypy changes
separate changes to doc comments from the "real change"
make sure to run black formatter, or do the pre-commit thing
push new change history into this PR with --force-with-lease option

Optional Dependency Handling
Example:
https://github.com/opendatacube/odc-geo/blob/62b7c5ebe41a3c7fc19f267daf06246e437ff830/odc/geo/_xr_interop.py#L55-L61
The have object is here:
https://github.com/opendatacube/odc-geo/blob/62b7c5ebe41a3c7fc19f267daf06246e437ff830/odc/geo/_interop.py#L22-L28
Add azure to that, then use it instead of try imports.
Tests
We use pytest for tests, here is example of skipping whole file when needed dependencies are not present
https://github.com/opendatacube/odc-geo/blob/62b7c5ebe41a3c7fc19f267daf06246e437ff830/tests/test_converters.py#L6-L9
Packaging
Add optional azure feature here:
https://github.com/opendatacube/odc-geo/blob/62b7c5ebe41a3c7fc19f267daf06246e437ff830/setup.cfg#L54-L60
Run checks locally
mypy and pylint checks are failing, run them locally or setup actions in your fork, actions on the PR require manual approval  from maintainers.
Simplifying Review process
There is now quite a bit of cleanup changes here, let's have those in a separate PR that I can merge, then we can focus on actual change without getting bogged down by all the mypy harmonization changes.

Added a PR for the cleanup. I'll give the rest another push tomorrow (UK time). Hopefully, that will give us enough time to review before the holiday season; if not, we can review again in 2025 :).

@wietzesuijker other PR is merged, and some other changes, please rebase this against main.

Thanks @Kirill888! I didn't get as much done as I'd liked. I updated the test for az and started using have.azure and have.s3. Could you review the code to see if I'm on the right track? I might need to git rebase and rewrite the history, but I pushed that issue into tomorrow as I was working on this when it wasn't merged yet.
Thanks again.


⚠ ClamAV has degraded performance

In 406e4ab, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 473 ms

Resolved: ClamAV performance has improved in bab4816.


Allow setting ClientOptions with Options API

Which issue does this PR close?

Relates to https://github.com/apache/arrow-rs/issues/4047
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

happy to contribuite some of the legwork to define all the client configs

That would be great :+1:


Limit the pattern matching to comments

Is there a way to do the pattern matching only in the comment without adding it in the regex? An option in the setting could be nice, this could use the language syntax to detect comment section and perform the regex matching.
If there is already a solution I didn't find one that could work on all language.

This is a good idea, @mrvkino. I've added this as a suggested enhancement.
(I'm not the original author, and I don't really know Javascript, so I'm not going to attempt it, but I'm happy to test PRs and release new versions to the marketplace.)
If you only work in languages that between them have a few different comment styles, it takes very little time to add multiple regex that cover this.


Release mobilenetwork microsoft.mobile network 2024 04 01

ARM (Control Plane) API Specification Update Pull Request

[!TIP]
Overwhelmed by all this guidance? See the Getting help section at the bottom of this PR description.

PR review workflow diagram
Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

Purpose of this PR
What's the purpose of this PR? Check the specific option that applies. This is mandatory!

[ ] New resource provider.
[ ] New API version for an existing resource provider. (If API spec is not defined in TypeSpec, the PR should have been created in adherence to OpenAPI specs PR creation guidance).
[ ] Update existing version for a new feature. (This is applicable only when you are revising a private preview API version.)
[ ] Update existing version to fix OpenAPI spec quality issues in S360.
[ ] Other, please clarify:

edit this with your clarification

Due diligence checklist
To merge this PR, you must go through the following checklist and confirm you understood
and followed the instructions by checking all the boxes:

[ ] I confirm this PR is modifying Azure Resource Manager (ARM) related specifications, and not data plane related specifications.
[ ] I have reviewed following Resource Provider guidelines, including
ARM resource provider contract and
REST guidelines (estimated time: 4 hours).
I understand this is required before I can proceed to the diagram Step 2, "ARM API changes review", for this PR.

Additional information

 Viewing API changes
For convenient view of the API changes made by this PR, refer to the URLs provided in the table
in the Generated ApiView comment added to this PR. You can use ApiView to show API versions diff.

Suppressing failures
If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
suppressions guide to get approval.

Getting help

First, please carefully read through this PR description, from top to bottom. Please fill out the Purpose of this PR and Due diligence checklist.
To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
For help with ARM review (PR workflow diagram Step 2), see https://aka.ms/azsdk/pr-arm-review.
If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

This PR copies over the changes from:
https://github.com/Azure/azure-rest-api-specs-pr/pull/17894

Signing off because its already signed off in private repo.
You still need to fix the remaining lintDiff issues before merging.

Please also complete the PR template

Signing off because its already signed off in private repo.
You still need to fix the remaining lintDiff issues before merging.

@TimLovellSmith We have 1 must fix Lint error. Error is add pageable annotation for the endpoint which is a GET API endpoint, not a LIST endpoint. We have a backlog item to add new LIST API endpoint for the left out resources.
This has been reviewed and approved in RPSaaSMaster and previous API versions.

/azp run

/azp run

/azp run


Can't get Random Custom_URL_Selector To Work

Honestly not sure what is wrong here. I've tried iterations of "[{random}]", "['{random}']", [{random}], ['{random}']. Removing the quotes causes some transparent breakage (mostly background sayagata-400px.png has an odd appearance probably caused by the missing quotes throwing errors).
Assuming the use of "[{random}]" from README.md, what I see in my browser console for headers on ajax_get_image.php looks like this:
{"success":1,"url":null}
So the ajax_get_image.php file call is not able to get the url of images for me when using {random}. Changing from {random} to any of the selectors in the file (ex 'bga') will cause that one image to load (interestingly the message remains success:1 with url no longer null but an actual address), but I would like to use the random feature properly and cycle through several images that are listed in my custom_url json file.
I looked at the function from ajax_get_image.php on line 24 but I'll be honest, code reading/writing is not my forte. I have a general idea of what should be happening, but no idea how to debug further.
On a couple of tangents, I also noticed that:

Custom_URL doesn't seem to be able to accept {{cur}} variable. Nor does it seem to accept a relational url. If I want Custom_URL to work, I have to enter the entire URL, even if my JSON file is located on the same server / path as the rest of the web files.
The credits feature also doesn't seem to work when using custom_urls. I get Picture by / Unsplash in the credit section, and looking at the code I think the user's name is supposed to appear before that / (line 70, column 38-72 of index.php file). The best option if possible is to get it so that the credit section checks for protected being empty, and if not it uses the json file from our protected section to dynamically create values. I'm not that familiar with json so not sure how the arrays have to be structured. If possible, could we define a url to the image, the artist's name, and the artist's portfolio link in our custom json file and have it generates the image credits matched to the url? Also not sure if the picture refresh actually reloads the entire page or not (I think that or another ajax would be required to update the credit section, no?) so if not possible to do that just have the Picture by / Unsplash message hidden when using the protected section.

Thanks for the detailed breakdown and for using homepage, @armouredking !

I checked and am able to properly use [{random}] for custom URL's. For example, if your custom_url is set to an Unsplash API page like so: https://api.unsplash.com/photos?page=1&client_id={client id}, then your custom_url_selector could be set to [{random}]['urls']['raw'], which will pull in a random image from the Unsplash API.
You're right in that the custom_url value itself did not support the {{cur}} parameter for relative URL's. That has now been resolved. So you should be able to do something like {{cur}}/images.json for the custom_url parameter.
The Unsplash link back modal was never supposed to have been shown when using a custom URL. This has now been corrected.

Hope this helps!

Closing since I haven't heard an update. Comment again if you're still having issues.


Relation caching after create

This is a bit of a weird bug. The behavior is somewhat unexpected, but understandable after spending time debugging.
Given this model:
class Section extends \Eloquent
{
    protected $table = "ibf_games";
    protected $primaryKey = "ga_id";

    public function league()
    {
        return $this->hasOne(League::class, 'section_id', $this->primaryKey);
    }

    public function enableLeague()
    {
        if (!is_null($this->league)) {
            throw new Exceptions\LeagueAlreadyEnabledException;
        }

        return $this->league()->create([]);
    }
}

If you then call the two methods in this order:
$section->enableLeague();
dd($section->league);

You will get null for the value of $section->league, even though you just created (and yes, the record does get created)
What has happened is you did an initial check for the existence of the league, which was null. Eloquent then automatically cached this result, so the subsequent call to $section->league will have the cached value, which is null. This makes sense.
However, it is expected that $this->league()->create([]) on a one-to-one relationship would purge the relation cache since now a relation has been created, and therefore the relation status has changed.
The work around to this presently is to call $this->load(['league']) immediately after the create so as to replace the cache with a new query of the freshly created object.

Sorry, this is actually the expected behaviour, and you do need to call load, or manually set the relation.

It's strange expected behavior for the database representation of a model to be "ahead" of the in-memory representation, when the database change was made through the model whose in-memory state is stale.

I don't think it is, sorry. It's very consistent behaviour, and makes total sense.

Consistent with the way eloquent works that is.


distributive check failing after upgrade to 1.1 (resolv.conf)

We have an outdated distributive check that fails when upgrading a 1.0 cluster to 1.1. See https://github.com/asteris-llc/mantl-packaging/pull/95

Ansible version (ansible --version): n/a
Python version (python --version): n/a
Git commit hash or branch: 1.0 -> 1.1
Cloud Environment: n/a
Terraform version (terraform version): n/a

hm, I cannot reproduce this problem. Can you confirm that you are upgrading a 1.0.3 cluster (built with a 1.0.3 security|mantl.yml) and following all of the steps in https://github.com/CiscoCloud/mantl/blob/master/docs/upgrading/index.rst#upgrading-from-103-to-11?

Couldn't reproduce it myself!


Allowing php8.3-mcrypt

Adding a ent folder
Adding the php8.3-mcrypt lib install

Added the image to the mulit-arch builds


Delete static/uploads/resume.pdf

👷 Deploy Preview for dat4lab processing.

Name
Link

🔨 Latest commit
244180de34fb4a26bb2f4da446d210b7aea7730d

🔍 Latest deploy log
https://app.netlify.com/sites/dat4lab/deploys/6536bacd9a71e900082135d5


Provide the means for custom embedders to refresh /etc/localtime

https://b.corp.google.com/issues/145541732.

cc @chinmaygarde @cbracken

The proposed fix was rejected after the discussion in https://github.com/flutter/engine/pull/14518#pullrequestreview-332975057.

Internal bug 145541732 has been closed.


Can you make this plugin compatable with the Clear Light Bulbs Wiz Offers?

Can you make this plugin compatible with the Clear Light Bulbs Wiz Offers?
The clear light bulbs dont use the RGB and only use the Kelvin Temperature and dimming functions.

Hi, @Vic-Firth could you please provide some more information?
Do you mean a specific product or the Clear Light Bulbs by WIZ in general?
Do you use a non EU version? Because I am based in the EU and therefore can't test my implementation with non-EU versions of this light bulbs.
Which internal product ID you wish to control?
Thanks for your feedback in advance.
P.S. maybe you can give me a link to your version on the light bulbs you wish to control, if you're not capable of providing the information I asked for.

Thank you so much for the fast response. It turns out it only the On/Off works for the colored versions of the bulbs too. Basically it seems that the Colorchaning and dimming is broken for the Phillips versions. It's too bad you are in the EU, but thank you so much for taking time and being interested in helping.
These are the Phillips Branded Wiz Bulbs. While made by the exact same company, it seems they must have changed something inside the bulb in these versions to make only the On/Off Compatiable with this plugin.
https://www.homedepot.com/b/Smart-Home-Smart-Lighting-Smart-Light-Bulbs/Philips/N-5yc1vZc7chZ15b

Hi @Vic-Firth
sorry for the inconveniences. I still want to help you as far as it is possible for me. I am unable to open your link, but I figured, if you use a homedepo-link your are probably from the US or Canada, so I searched the Philips website for customers based in North America and couldn’t find the homedepo number :(
But I still won’t give up, so maybe you can direct me to your product. I think if we find your product here:
https://www.usa.lighting.philips.com/consumer/choose-a-bulb/products/bulb#filters=SMARTLIGHTING_BULB_SU&sliders=&support=&price=&priceBoxes=&page=&layout=12.subcategory.p-grid-icon
Maybe I can try to emulate your setting with a EU-Version of your product and all I have to figure out with you is the Internat ID of your product type, to add the support you wish to see implemented. I don‘t think Philips is using a different firmware, I think the light bulbs of Philips simply have a few different values for controlling the light temperature.


Support for DPDK 18.02 or higher

Hello,
do you plan to add support for DPDK 18.02 or higher in the near future?
Thank you,
Adrian Vladu

Latest ans support dpdk-18.11 LTS


lunr index generation needs attention

it seems that the file is being placed in the /js/ folder and mislabeled as lunr-index.json when the lunr-ui.js wants it to be root and labeled search-index.json.

I've been playing around with where the lunr files should be generated by default and how much control users should have about that. I'm open to thoughts, but am at the moment refactoring the lunr task within wax_tasks to have the UI file use liquid so it can find the appropriate index dynamically by referencing the site's _config.

I also think the UI should show '0 results' when typing by default, so it doesn't look like the search isn't working when in fact there are just no results.
I just created another issue #37 for UI updates. @elotroalex: can you add ideas / issues to it and I'll turn them into action items?

(Closing in favor of issue #37)


[WIP]: Expand swap capability and reorganize package

In trying to apply the XSwap method to other networks, I have discovered that this implementation fails for several publicly-available networks. A few improvements address most of the underlying issues.
First, I added a preprocessing file to process files containing graphs. The underlying C++ functionality deals with edges represented as int*s. Since permutable graphs may be in files as integers or strings, (eg: node_a,node_b\nnode_c,node_d\n...), it is helpful to be able to process such graphs to a form that is usable by the fast XSwap implementation. Moreover, assigning new ids to the nodes allows us to ensure that nodes are indexed 1, 2, ..., num_edges. This kind of indexing allows larger graphs to be memory-feasible using the fast Cantor pair hash table.
Second, I added an additional hash table implementation using the C++ standard library std::unordered_set. While this is significantly slower than the Cantor pair hash table, it will work for networks with too many potential edges for the original hash table to fit in memory. (I have set the cutoff at 4 GB, though this was a totally arbitrary choice.) To ensure a uniform API, I added a final hash table wrapper, the HashTable class, which can call both the original EdgeHashTable and the new BigHashTable.

As an example, using Zachary's karate club network with multiplier=10000, we find that it does not affect the swap outcomes, but significantly affects computation time.
Using slow hash table
43.5897 % unchanged
5.205 seconds
Using fast hash table
43.5897 % unchanged
0.134 seconds

In the previous comment, does "fast hash table" refer to the "Cantor pair hash table" and "slow hash table" refer to the std::unordered_set hash table?

That is basically correct, though I have found that std::set actually gives better performance than std::unordered_set. I am not sure why, but std::unordered_set had been giving insertion performance much closer to the worst case (O(n)) than to the best case (constant). Since std::set uses a self-balancing binary search tree (O(log(n))), log(n) is proving to be faster than the worst-case hash table. I am looking into the possibility that a better, custom-specified hash function would improve performance on the hash table.
Here's a quick rundown of the differences in implementations between set and unordered_set:
https://www.geeksforgeeks.org/set-vs-unordered_set-c-stl/


Input freezes after changing decimal value symbol twice

Describe the bug
If we enter decimal value and change symbol twice using minus key, input freezes until we change our cursor position or select whole value. It's not possible to add or delete numbers.
Add a link to a codepen, jsfiddle or other example page which shows the problem
https://robinherbots.github.io/Inputmask/#/demo (decimal)
OS: Ubuntu 22.04.2 LTS
Browser: Chrome 115.0.5790.110
Inputmask version: v5.0.8

@RobinHerbots Can I ask for a response here? This is very problematic for our project and also occurs in the latest beta

@embargiel ,
I will look into it.

Can you provide some more info of how to reproduce.

@RobinHerbots

Go to https://robinherbots.github.io/Inputmask/#/demo
In "Decimal" input type , 5 - - -. It should be -0,5
Try to change value

https://github.com/RobinHerbots/Inputmask/assets/70658299/2f40b946-278a-4a14-9570-b3633910fbf9

@RobinHerbots Hi, I also noticed that when entering a large number of digits after the decimal point, the text field starts to lag terribly

@skrasnoyarov,
The lag should be fixed with the latest beta

@Pomierski ,
The latest beta 71 should fix the issue.


Android: Grid View

Android is missing the grid-style story list mode that prioritizes thumbnails.  We should modernize the ItemsList activity to use a RecyclerView that will allow us to support more novel layouts than the vertical lists of old.

Completed in #1095.

can this be ported to ios client ? (please)

Yep, this is now #1270.


[textdiffuser-2] where to set the loss type during training?

hello, I am trying to use textdiffuser-2. When I am in the stage of M1's layout planning training, I found that there is no flag for setting the type of loss, whereas in the paper the author said such training requires loss type being cross-entropy. How come to set this configuration?  and by the way, how can I monitor the training loss curve like the one in readme.md?

Thanks for your attention to TextDiffuser-2. Please dive into the Transformer Trainer code. The default loss is the cross entropy.

Thanks for your attention to TextDiffuser-2. Please dive into the Transformer Trainer code. The default loss is the cross entropy.

thanks for you tips. i thought i missed some cricial argumental code haha.


Include tag aerialway for foot/bike?

Example

Personally I would prefer to be routed in my hiking/cycling route just on paths.

Sure, you can customize this to your needs. But here we need to support general 'by-bike' or 'by-foot'

Personally I would prefer to be routed in my hiking/cycling route just on paths.

And some people would prefer being routed by car just on roads but not ferries. Yet they have to be supported for reaching certain regions.
In my opinion aerialways should be supported by adding a reasonable penalty. If the user still doesn't like to get routed via an aerialway he/she still can set a corresponding via-point to avoid it.

There is no alternative route to aerial ways in example region.

I have to agree that aerialways be supported with a sensible penalty.  In many regions they are the only way to cross rivers.  This is true in the mountains of India, Pakistan, and Nepal -- but I suspect other developing countries too. Here is an example.  In these regions simple cable cars pulled by hand are often built as a less expensive option to bridges.  As my example, failure to support these aerialways results in serious misrouting.

Yes, please provide a pull request which we can review and discuss. I agree that aerialway should be added but I would need time to dig into it and make it properly. Also I do not have a realworld example in mind which I could check if afterwards everything is reasonable :)

Example in France, the Téléo of Toulouse (osm).

This also came up in our team lately. The option as such would be nice.
But I thing cheking OSM-Data would be needed? I found some aerialways that don't  have a proper connection to the rest of the road-network
Like this one here for example: https://www.openstreetmap.org/#map=19/47.66605/11.89004
But as I understand it, Graphhopper needs that, right?

I found some aerialways that don't have a proper connection to the rest of the road-network Like this one here for example: https://www.openstreetmap.org/#map=19/47.66605/11.89004 But as I understand it, Graphhopper needs that, right?

Yes, a proper connection is needed for routing. Please add one (maybe highway=footway + footway=link?). To start with, the geometry does not have to match reality exactly, as it can be improved later (but of course you should try to add to it in a meaningful way).

Ok... I noticed that on quite a few aerialways and just wanted to point it out as it might be problematic for Graphhopper. But yes.. I will fix some in my region... But I guess there will be a ton of those in OSM...