arXiv identifiers in BibLaTeX export

When downloading from arxiv automatically into zotero and exporting the database with better bibtex I have the following issue: The citation should read arXiv:1501.03850 [physics.chem-ph]. Zotero places this in the "Publication" field. However, when the entry is exported, all colons, brackets and dashes are lost and furthermore the field content is written not in \tt as it should according to the arxive and also most journal guidelines. One would somehow actively have to recognize that this an arxive entry or alternatively have zotero treat the importing differently, I guess. Is there any way around that?
I sent a bug report for an example with the number JGNRQ3ZM.

You mean you'd want the publication title preserved even when the arxiv data has been added as eprinttype/eprint?

No. It is about a paper that has not yet been published where else than in arxive. The issue is that I do not get the correct bib entry in such a case.

Ah. Hmm, how would I recognize this to be the case? Alternative I could just keep the publication title even if it does say arxiv:..., I currently actively remove the publication title if it looks like that.

But if you actively remove the title, then the entire information is not there any more in the bib file or at least not any more in fields that will be printed in most journal formats (I this particular case achemso).

I only remove it when it starts with arxiv:; this was prompted by #170; and the discussion that happened here. I don't mind either way (all that needs to be changed is to remove a single line of code that scrubs the publicationTitle) but I'd very much like to know how the behavior differs between this issue and #170.

If I don't export the journal title, I currently get this:

Kastenholz, M. A. and Philippe H. Hunenberger (2006). "Computation of methodology-independent ionic solvation free energies from molecular simulations.I. The electrostatic potential in molecular liquids". In: J. Chem. Phys.124,124106. doi:10.1063/1.2172593.1

(source: https://www.sharelatex.com/project/54feca38f58d781e0c982eeb)
Is this not how it should be? The Kastenholz article was published at JCP (http://scitation.aip.org/content/aip/journal/jcp/124/12/10.1063/1.2172593), not at arXiv.
I'm looking at https://arxiv.org/help/faq/references and that does suggest one should cite arXiv. That strikes me as odd, for the general case; I thought arXiv was like more like JSTOR than like Elsevier.
If a paper is only published in arXiv that would be a different matter of course. I don't yet know how I should detect that. Should it perhaps be a manuscript instead of a journal article if it is only published at arXiv? arXiV isn't really a journal.

also this: http://arxiv.org/hypertex/bibstyles/

Ah, https://arxiv.org/help/faq/references does not say "you must use \tt", but "if you use \tt, don't put it within the arXiV ID"

Ugh, so many conflicting answers:

http://tex.stackexchange.com/questions/3833/how-to-cite-an-article-from-arxiv-using-bibtex
http://arxiv.org/hypertex/bibstyles/ (which omits the journal/journaltitle field)

Hi! Seems to be 170 seems to be a similar issue. However, solving it with eprint is not the best option in my eyes. At least achemso package simply ignores those fields. What actually works for me and gives a correct output is the following:
@article{Shulenburger.2015.chemical,   title = {Beyond chemical accuracy: {{The}} pseudopotential approximation in diffusion {{Monte Carlo}} calculations of the {{HCP}} to {{BCC}} phase transition in beryllium},   journal = {},   author = {{Shulenburger}, Luke and {Mattsson}, Thomas R. and {Desjarlais}, M. P.},   month = jan,   year = {2015},   pages = {arXiv: 150103850 [Physics:chem-Ph]}, }
i.e. leave the journal empty and put the reference in the pages section, where it will also not be printed in italics - which is correct. In order to detect such a case from zotero, it would be sufficient to check whether the title begins with "arxiv:". If so: leave the journal section empty, to not remove ":, [, ]" and put it into the pages section. This is an ugly hack, but I guess it will work in most cases. You could try what happens for example in revtex, but I am pretty sure the hack will work, although revtex supports the eprint field since version 4.1 or so.
I think the largest problem here is finding a portable solution. Maybe one could add an output option to how to treat things with "arxiv:" in the title...

Is leaving the journal field empty to accomplish this documented somewhere? It does seem like a bit of a hack, and one that directly contradicts what arXiv itself says should be done. I could leave the publicationTitle in place, which would mean the arXiv:... would end up in the journal field... which is what arXiv seems to recommend, but by your description, that doesn't seem like it would get the desired output.
I don't want to put the arXiv bits in the pages field. For one, a paper could actually have something there, and this would overwrite it. It also just doesn't belong there -- it'd be better even to put it in the url field (after all, the arXiv bits are sort of pointers, not page numbers) or a note. One major reason for this is that different styles could treat this very differently; where possible, I want to stay close to spec so the output behavior is somewhat predictable across styles.
I'm terribly conflicted about this one. On the one hand I want BBT output to "just work". But I'd need to know that this would be the acceptable behavior for the majority of styles out there, and I can't tell right now. @DenizSt, you submitted #170, any thoughts?
In the mean time, there's a development build at https://s3.eu-central-1.amazonaws.com/zotplus-builds/zotero-better-bibtex-1.6.37-travis-2467.xpi that would allow the following postscript:
if (this.item.arXiv) {
  this.add({ name: 'journaltitle', bibtex: '{}' });
  this.add({ name: 'pages', value: this.item.arXiv });
}

which should do what you want. It won't work with the current release of BBT as that doesn't have the item.arXiv field

... doesn't have the item.arXiv field yet. The next release will have it.

putting it in the title would probably be OK if it is not in italics. However, in many output formats, this will produce weird output, since the journal entry expects a page number, which it does not get (no arxiv file has a page number. It's "page number" is actually it's identifier, ie.e the arXiv:... bit, so it is really not that weird to put it there (all phys revs for example also do not really give a true page number but rather an identifier).
Another possibility would be to put the arxiv into a @misc and put the identifyer into the url. I guess that would work too. (Haven't tried though)

I guess some of the problems come from the fact that there are too may bibtex standards around. Some allowing eprint, some not,...

If that's the case (the arXiv ID is sort of its page number), why not put that in the page number field in Zotero? That would make a lot more sense. That way it could be unequivocal whether a paper is only published at arXiv (arXiv ID in pages, no journal title) or elsewhere (in the case of Computation of methodology-independent ionic solvation free energies from molecular simulations. I. The electrostatic potential in molecular liquids -- published at JCP, available at arXiv)
Still unclear on how one would indicate (in Zotero or in BibLaTeX) "published at journal X, issue Y, pages E-F, available at arXiv:..."

Here, I would say it is pretty clear how one should proceed: If the author references to the arxiv, then he really references to the arxiv and that reference should be used. So if the citation is downloaded from the arxiv to zotero, it should stay the arxiv citation. It is the authors responsibility to check whether this has been published and then download and cite that reference. This is the only clean thing to do. The publication MAY change between the arxiv version and the published one, so one absolutely HAS to be clear about which one one is citing!

In terms of putting the citation into the page slot in zotero: does it then not loose all its :, [, ] when exporting? (Did not check for now). If not, I guess it may be a question of discussing with zotero then how this should be done... (at the moment if getting the citation automatically, this is not where the entry is put.)

It's not that I'd doubt the author, but Zotero's standard behavior. If I import something from JSTOR, I don't expect to see JSTOR as the journal. If Zotero's default behavior would be to add the arXiv ID to the pages field, and the author moves it to the journal field, that's clearly intentional.
I could in principle go the other way: assume that the arXiv ID in the journal title means you want it there, and if you don't, you could move it to the pages field, where I'd then pick it up as a fallback.
That would leave, in either case, the problem of the empty journaltitle field. I recognize it's a hack to make the : show up, but if we're in hackish territory anyhow, I could just add the : to the pages. I don't like generating empty fields exactly because it causes punctuation to show up where you wouldn't normally expect it.
As for the pages field, BBT exports that as-is if it doesn't recognize it as NNNN or NNNN-MMMM.

I'd rather not make all references that happen to have an arxiv id into a @misc. That would be throwing away too much information.

So, indeed, Zotero does re-reference to the Journal (if possible) if the entry is generated automatically, only leaving the information that it came from zotero in the library catalog and the extra information.
I do not like this behaviour of Zotero, but other people may like it, so lets not try to change that. Instead, one could detect the library catalog field and if that is one of the known eprint servers, do the hack to put the info in the pages field and leave the journal empty. That sounds most straight forward to me, and as argued above is not quite such a "workaround" as it sounds in the beginning.

I'm still not entirely convinced that putting things into the pages field is the appropriate general solution. Is there anything on the arxiv site that specifies how biblatex entries should look for them? I've only seen them talk about the output, but the output they talk about must be style specific, and they don't even say what style they have generated that with. Biblatex is a subtle and fickle beast - if I'm going to deviate from the original meaning of fields (and pages certainly wasn't intended for this - it just happens to generate the desired output for those cases we happen to know about) I'd like to know if there is any consensus about this appropriation. I understand your need, but I want to prevent putting something out that conflicts with a known case (#170), and if possible, prevent adding another preference to the already overloaded prefs pane.

Could you perhaps tickle arXiv on this and see if someone could chime in?

Good idea, I will try. So far I tried to contact ACS, but they were not very
helpful. I'll thus try to contact ACS.
On March 20, 2016 at 5:13 PM retorquere<EMAIL_ADDRESS>wrote:

Could you perhaps tickle arXiv on this and see if someone could chime in?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/ZotPlus/zotero-better-bibtex/issues/460#issuecomment-198956059

Dr. Katharina Doblhoff-Dier
Leiden University
Faculty of Science
Leiden Institute of Chemistry
<EMAIL_ADDRESS>

Any news?

no :-(
On 04/07/2016 03:58 PM, retorquere wrote:

Any news?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/retorquere/zotero-better-bibtex/issues/460#issuecomment-206919944

So, what to do? If arXiv has no active interest to resolve this, I don't honestly know how to proceed.

Hi, I am just chiming in because of this issue had some impact on how I will like to have my citation style (Not for any journal, but a project report). In Physics, the APS and AIP style are quite popular, and they recommend that arxiv paper be cited as (say) ,
A. Kitaev, and C. Laumann, “Topological phases and quantum computation”, arXiv:0904.2771 [cond-mat, physics:quant-ph] (2009).

The RevTeX style .bst file assumes that the arXiv id (In this case, arXiv:0904.2771 [cond-mat, physics:quant-ph] ) is put in the 'journal' field. This works fine for Better-BibTeX translator.
For better BibLaTeX, the export file (correctly) leaves the journal field black and populated the appropriate eprint fields. Unfortunately, this breaks the BibLaTeX-Phys style. I will say that this is more of a bug in the style file (where it should publish arXiv field when journal field is empty), but conventions are conventions.
In any case, I have this fixed by adding this post-script,
if (this.item.arXiv && Translator.BetterBibLaTeX) {
    this.add({ name: 'journal', bibtex: '{' + this.item.arXiv + '}'});
}

This works well enough for me. I think the current defaults are good enough. Perhaps a link in README.md file referencing the post-script page will make everyone happy.

I dunno; the export page references it, and the postscript stuff is not really mainstream; not really something that any new user needs to take in.
Out of curiosity, why bibtex: rather than value:?

BTW, you are the first user that I know of that wrote his own postscript 👍

Out of curiosity, why bibtex: rather than value:?

Ha-ha, I don't know even a least bit of Javascript. I just copy pasted it from the example page and modified it slightly to make sense. :smile:

Ah. Well, kudos to you still! Is there any other facility I could be offering instead of javascript to make things easier?
bibtex: assumes you know what you're doing; it will pass whatever you hand it straight out to the output. value: has various encoding rules (the default if you specify none being latex) which make sure your value is translated to proper LaTeX; it will also include the braces. There are others, but only latex and verbatim are going to be interesting to most cases. The arXiv sample uses bibtex: because it's the only way to specify an empty field; if you hand BBT and empty value, no field output is generated.

Ah thanks! You should write this down somewhere! (I kid! )
Well you can certainly add another tick box where one can export arXiv identifiers to journals field. But I am not sure, there are infinite variation of things that people might wish to have. One good way will be to make the post-script option more discoverable. Add a preference box in options page which takes a JS file as input, and a link to Wiki there. Then, people can copy-paste examples from the Wiki to their taste. You can also add a repository like thing which pulls options in terms of small snippets of JS and appends them to main JS file. That might make you job easier in future for every little formatting request.
Here, I am assuming that (most) people who use LaTeX have knowledge of at least some programming language. I had no clue about JS, but programming in C++ gave me enough idea to look up and customize something to my taste.
As for this particular bug report, I think the default should be to stick with putting the information in e-print field. This is exactly what they are for, seeing that arXiv is really not a journal. The problem lies with various style files as BibLaTeX explicitly provides for eprint fields and the style files should use them properly. In BibTeX world, it makes sense to put arXiv data in journal field, because by default, BibTeX has no idea that eprints exist.

Well it is written down, but I suppose the "somewhere" aspect is the issue. Findability is not great. As a very old-school software engineer, I get all warm and fuzzy about writing code, but much less so about great UI or docs. A sizeable chunk of my users have no programming interest, so it should be more of a priority I suppose... but life's short.
There is currently a box where you can edit your postscript in the Advanced tab of the BBT prefs, but a file is problematic, as the translator has no access to the file system. It can read the preference I store the postscript in. Copy-pasting should work OK though. I noticed I hadn't updated the wiki for that; that's now done. A snippet repository... I'd rather not; copy-paste should work, and the Wiki (which is where I'd likely host them) is freely editable. I wouldn't want people to accidentally pull in code.

After searching various ideas on internet, I have found that the "best" solution is to use the fields "archivePrefix" (for arxiv), "eprint" (the number) and "primaryClass" (for the category). Modern bibtex styles support these fields, and also it is (somewhat) easy to change the presentation using biblatex. If the style used does not support those then I would say "too bad": it is not possible to be backward compatible with everything. Also not that the problem could happen with other preprint directories, and so all of them should used the above fields.
In practice I have a script that I am using after every export to reformat the bib file, and in particular I am creating the above fields from the URL, and I am deleting the publication field.
I am not sure putting arxiv in "pages" is a good idea, but this may be less worse than others: it is clear that Zotero and Bibtex are missing an "ID" field (in particular some journals use ID and not pages anymore). So using pages for ID is maybe not that bad, due to the circumstances. The main problem would be that most of the styles are not waiting for such behaviour, while on the other hand, as said above, the more modern styles will naturally handle the eprint fields.
Maybe a good solution is to just indicate "ArXiV" in the publication, and then use the URL to reconstruct the eprint fields.

Still unclear on how one would indicate (in Zotero or in BibLaTeX) "published at journal X, issue Y, pages E-F, available at arXiv:..."

In my case I just add "arxiv:xxxx" below the DOI (itself inserted below the main citations). In my field (theoretical physics) most of the people cite like this.

Here, I would say it is pretty clear how one should proceed: If the author references to the arxiv, then he really references to the arxiv and that reference should be used. So if the citation is downloaded from the arxiv to zotero, it should stay the arxiv citation. It is the authors responsibility to check whether this has been published and then download and cite that reference. This is the only clean thing to do. The publication MAY change between the arxiv version and the published one, so one absolutely HAS to be clear about which one one is citing!

I totally disagree with this point of view. In theoretical physics we basically never read the published version and always use the arxiv (also most authors are updating the arxiv preprint to match the published version). But in any case there is one paper, the fact that you have several versions in several places does not change this fact. Otherwise this is just chaos.
Publishing in journals is useful for having the peer-review, but not more. And in citing it is important to give both the journal reference (to now who has made the peer-review) and the arxiv (for practical reading). The reader should check if the versions differ or not (in my case I always assume it does not differ: since most of the authors are updating on arxiv I don't want to look for the few cases where they did not).
In my opinion the formatting using \tt should be done using a bibtex style and should not be present in the export.

Well, you'll have more experience with arXiv than I do. I really don't like putting stuff in the pages fields that aren't pages, but if that's the norm for arXiv... can you get me some test cases for arXiv? You could have some references in Zotero, right-click them and select "Report Better BibTeX error", and post the resulting ID here -- that will give me the input references. If you could then attach the desired BibTeX/BibLaTeX records here (please specify which is which), I'll do my best.
Much of the post-processing script should in principle be doable in a BBT postscript, but I can't assess how complex your postprocessing is.

but if that's the norm for arXiv

In fact this is not the norm, and this is why I believe one should not do it. This is what is done for some journals, but not arxiv: as I was saying earlier, the styles currently are using the eprint fields. So if the export put arxiv in pages this may break the configuration of many people. For this reason I would recommend the solutions with using only "archivePrefix" and "eprint" (one problem being that the category to be translated into "primaryClass" is not saved in Zotero with the current arxiv translators). Here is a report: 2M6D6V96, and the assoctaed file: arxiv.bib.zip.
Indeed I guess that much of my postprocessing could be done through BBT, the point is that most of it is inherited from the time I was not using BBT, and I did not want to break anything. The main problems I am fixing are the formulas inside titles (BBT makes strange things with them), brackets around title (simpler to have {{title}} than brackets around every words with capitals) and conditional processing of url/doi (the last version of the script is at the bottom of this post). For the latter this is due to the fact that I need to export both url and doi for arxiv entries in order to get the "eprint" field from the url (and then I can delete the url). The problem is that I need to delete all urls for non-arxiv also if doi is present. This may be complicated because now the arxiv number is exported in an "arxiv" field, which I can use (and this is basically what I do because of the problem #466). But ok all this processing is mildly related to the current post, so don't worry about this.

Is this sample for BibTeX or for BibLaTeX? I see there's a "month" field in there, which indicates BibTeX, but I need to be sure.

This is BibTeX indeed.

But http://arxiv.org/hypertex/bibstyles/ says this requires modified bibstyles. Are these in wide enough use to be considered the default? If so, why should that page refer to them as "modified"?

@KatharinaDD , why do you include the arXiv ID in both the extra field and the journal title?

@melsophos, this is what would be exported from your reference (ignore the citation key and the timestamp):
@article{Sen.2016.BV,
  abstract = {We construct the quantum BV master action for heterotic and type II string field theories.},
  annote = {27/03/16 - overview - studying the papers of Ashoke
This paper is of great importance because it constructs a fully consistent quantum action for (covariant) superstring field theory, including the Ramond sector. The idea is to introduce extra dynamical fields which are free (they don't even couple to gravity) and so they can be consistently decoupled.},
  archivePrefix = {arXiv},
  author = {Sen, Ashoke},
  doi = {10.1007/JHEP02(2016)087},
  eprint = {1508.05387},
  eprinttype = {arxiv},
  issn = {1029-8479},
  journal = {Journal of High Energy Physics},
  keywords = {Batalin-Vilkoviski formalism,_tablet,covariant string field theory,futur: 3,note: 4,printed,superstring field theory},
  month = feb,
  number = {2},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {{{BV Master Action}} for {{Heterotic}} and {{Type II String Field Theories}}},
  urldate = {2016-04-17},
  volume = {2016},
  year = {2016}
}

@melsophos what kind of math strangeness?

And you can avoid the brackets by either sentence-casing your titles (which is what zotero recommends) or setting the language to non-English.

I just had it in twice, because I copied it to the new field. Most fields do not print in the end.
The problem with using eprint (while I agree that this would be the nicest solution) is that it is still not supported by several styles, and especially some journals. But I guess, if that is what you are now heading for, it is at least a very reasonable solution.

@KatharinaDD I've tried to come up with other solutions, but putting things in the pages field always is too risky an option for me. It relies on side-effects of particular styles, and I simply don't know what this would do across the board, and my nightmare scenario is that I put out a release with a given behavior that people start to rely on, only to find out that others need a different side-effect for their styles, and then I'd have to break the behavior for the former group. If arXiv would be so kind to go on record that the pages field is the right approach I'd be happy to oblige, but their recommendations only mention the eprint stuff.
For your case, after I remove the arxiv id in the extra, and apply the following postscript (which supports both an arXiv journaltitle, and having a "real" journal title in addition to an arxiv field in extra):
if (this.item.arXiv) {
  this.add({ name: 'pages', value: this.item.arXiv });
  if (!this.has.journaltitle) { this.add({ name: 'journaltitle', bibtex: '{}' }); }
}

or even just this if you always only use the title as the arXiv id:
if (this.item.arXiv) {
  this.add({ name: 'pages', value: this.item.arXiv });
  this.add({ name: 'journaltitle', bibtex: '{}' });
}

I get
@article{Shulenburger.2015.chemical,
  abstract = {Motivated by the disagreement between recent diffusion Monte Carlo calculations and experiments on the phase transition pressure between the ambient and beta-Sn phases of silicon, we present a study of the HCP to BCC phase transition in beryllium. This lighter element provides an oppor- tunity for directly testing many of the approximations required for calculations on silicon and may suggest a path towards increasing the practical accuracy of diffusion Monte Carlo calculations of solids in general. We demonstrate that the single largest approximation in these calculations is the pseudopotential approximation. After removing this we find excellent agreement with experiment for the ambient HCP phase and results similar to careful calculations using density functional theory for the phase transition pressure.},
  annotation = {Comment: 5 pages, 3 figures},
  archivePrefix = {arXiv},
  author = {Shulenburger, Luke and Mattsson, Thomas R. and Desjarlais, M. P.},
  date = {2015-01-15},
  eprint = {1501.03850},
  eprinttype = {arxiv},
  journaltitle = {},
  keywords = {Condensed Matter - Materials Science,Physics - Chemical Physics},
  pages = {arXiv:1501.03850 [physics.chem-ph]},
  primaryClass = {physics.chem-ph},
  shorttitle = {Beyond chemical accuracy},
  timestamp = {2015-02-24 12:14:36 +0100},
  title = {Beyond chemical accuracy: {{The}} pseudopotential approximation in diffusion {{Monte Carlo}} calculations of the {{HCP}} to {{BCC}} phase transition in beryllium},
  url = {http://arxiv.org/abs/1501.03850},
  urldate = {2015-09-11}
}

Plus, if you really want the eprint fields gone (although the should do no harm, Bib(La)TeX styles generally ignore fields they don't understand), you could either add those to the skipfields preference, or if you want it arXiv-specific, go
if (this.item.arXiv) {
  this.add({ name: 'pages', value: this.item.arXiv });
  this.add({ name: 'journaltitle', bibtex: '{}' });
  this.remove('archivePrefix');
  this.remove('eprint');
  this.remove('eprinttype');
  this.remove('primaryClass');
}

@anuragm the current implementation has the side effect that if you have the arXiv ID in both the journal title and the extra field, it will use the one in the extra field and will leave the journal title as-is.
@KatharinaDD: #85, #86, #152 and #170 had different desires when it comes to arXiv references, and putting the ID in pages would have broken all of those. Even the change proposed by @melsophos which I thought was rather innocuous breaks those cases. Not sure what to about that yet.

@melsophos, http://arxiv.org/hypertex/bibstyles/ says this requires modified bibstyles. Are these in wide enough use to be considered the default? If so, why should that page refer to them as "modified"?

I don't know how widely spread it is, but I would say that one should encourage this norm. Since arxiv is explaining how to work with that I would expect that anyone in a field using mainly arxiv should be able to manage this.
This is annoying if this breaks other points. On the other hand I am not sure that a solution that works in all instances exist, and as such I would better choose the one advised by arxiv, which should be the reference for people using it.

@melsophos, this is what would be exported from your reference (ignore the citation key and the timestamp): (…)

This looks good to me. I found the duplicate of eprinttype and archivePrefix a bit strange (since arxiv is not mentioning the first one), but I guess that this is more flexible and prudent.

The eprinttype is there for compatibility with earlier testcases. Should do o harm, and no surprises for existing users.
I agree we should all stick to the norms, but many users are bound to mandatory styles out of their control. I can't be pedantic on their behalf. I'm currently toying with a pref that allows turning on "the right thing" but that doesn't assume I can move about user-entered data such as the journal title without asking.

@retorquere I am fine with the current ways of things. I do applaud your pragmatic view. arXiv is a mess as different journal styles treat it differently.
Speaking of pragmatism, I don't think there are any journals out there that are using BibLaTeX, at least none do in Physics and Quantum Information. In that case, it boils down to what to do with BibTeX styles, because real world editors couldn't care less about BibLaTeX. As of now, arXiv IDs go to the journal field in BibTeX translator, and this works fine for most of the BibStyles.
In particular, the biggest Physics publisher (APS) suggests using the fields as shown on arXiv's own page. (See page 15 of APS Author's guide). So if you want to make the physicists happy, just do to arXiv field as shown on arXiv's own guide, like so,
     archivePrefix = "arXiv",
     eprint        = "0707.3168",
     primaryClass  = "hep-th",

This will keep the RevTeX 4-1 style happy. Coincidently, BibLaTeX export is done with this way, and if some one uses the option eprint=true while loading BibLaTeX pacakge, the arXiv identifiers are printed in the proper way (with \tt and all that jazz). It's just that usually I like to display arXiv number only when it is the primary source (that is, hasn't been published any where else). In that case, putting arXiv ID in Journal field is a hack, but one that satisfies me, and I think such hacks are exactly what post-scripts are made for.
If I have to make a suggestion, you should do by default what the good people at arXiv suggest should be done. This will make a big chunk of journals happy too, as they have adopted arXiv's recommendation. For other poor souls, you should add an option like it is done currently for the url field. But this is all for BibTeX export. I think BibLaTeX export is quite correct, and when the day comes that journals start supporting BibLaTeX, I am sure they will heed to current arXiv and BibLaTeX's recommendation about how to format .bib database for arXiv papers.
(P.S: This bug discussion is only about BibLaTeX, right? )

@retorquere The option might look like
Put arXiv data in :
( ) arXiv's suggested format ( ) In journal field ( ) In pages field.

with first option as the default option.

No, we're discussing both BibTeX and BibLaTeX. BibLaTeX currently always moves stuff to the fields arXiv recommends. For BibTeX, it's currently enabled by an option.
I'm leaning against implementing the pages field (outside what can be done in the postscript) unless arXiv weighs in. I can't assess what other styles will do with it, and I don't want to have to revert behavior later.
I don't understand what "When Publisher field is some other journal" means to do?

Yeah that's confusing. What I meant is whether Publisher field in Zotero says arXiv:xxxx or not.
If the paper is only on arXiv, the Zotero's arXiv translator fills Publication field with arXiv:xxx , otherwise if DOI is mentioned on arXiv page, the actual journal's detail is filled from the DOI and arXiv data goes to URL and Extra. At least that's how I think it works.

Can you get me samples (input: export with "BetterBibTeX JSON", output: create the Bib(La)Tex you want) together with the combination of settings that would create the one from the other? I work best from samples.

@retorquere So I went ahead and created a new Firefox profile using latest availible release of Zotero and Zotero Better BibTeX. I imported there two paper from arXiv as an example.

This paper is purely a paper on arXiv which hasn't been published in any peer-reviewed journal.

This paper is a pre-print on arXiv, where the final version was published in Science. Notice how a DOI of the peer-reviewed article is contained in arXiv abstract page.

I imported both these papers to Zotero's library using the arXiv translator. The entry for paper (1) has arXiv:xxxx as its Publication field in Zotero, while the entry for paper (2) has Science as its Publication field in Zotero. Both contain arxiv:xxxx in the Extra field.
Next, I exported my library with Better BibTeX as JSON here, and as BibLaTeX Bib file here. I did not change any configuration of either Zotero or Better BibTeX.
Here is the BibLaTeX output that I will prefer. Note how the entry for (1) now has a journal field with arXiv:xxxxx as its content.
Hope this helps!

Here is the BibTeX output that is produced by Better BibTeX, and this the output I will like to have. Notice how I have added fields that arXiv recommends on its page, which kept the journal field of (1) and other default fields in order to maintain support with other legacy BibTeX styles

As for combination of settings, the script I mentioned in a comment above fixes one issue with BibLaTeX output, that is, it adds the journal = {arXiv:xxxx} field for cases like (1) where the paper is only on arXiv, and leaves the peer-reviewed cases like (2) alone. I haven't figured out how to add primaryClass = {xxx} field with a script to cases like (2).

I am guessing a regex like \[.*\] on this.item.arXiv followed by removing the braces and adding the resultant string to primaryClass might do the trick.

By the mean time, I think I lost track of all different desires and suggestions, but I like the idea by anuragm to have a clickable option. I think the problem mainly arises because different fields have different methods to treat this. Having two or three options may make most people happy.

@anuragm Zotero doesn't know anything about arXiv in the extra field, it's just text as far as it's concerned. It's BBT that interprets that data. this.item.arXiv is set by BBT, Zotero doesn't know about it.
Just keeping in mind your sketch of the preferences, which would be set how to produce your "want"?
@KatharinaDD the problem is it never stays with 2 or 3 options. The more I can sensibly automate the better.

@anuragm my main concern in this is that the arXiv data has no dedicated place in Zotero, so I need to make decisions on

How do I decide this is an arXiv paper, and what status does it have there? Currently, it is taken either from the extra field or the publication title, but I haven't look at "status" before.
Having decided that, what to do with the field I took it from? Should I record it and then proceed as if the field I took it from was empty? Keep it there?
Having done all that, what output should it generate.

Preferably I'd have fixed rules for all 3. When this cannot be done, I add preferences, but BBT already has an overwhelming number of preferences, so I try to go for the minimum.

@anuragm For each reference, I first determine:

Does it have an arXiv identifier in extra? If yes, remember it as arXiv.extra, and remove it from the extra field
Does it have an arXiv identifier in the journal title? If yes, remember it as arXiv.title, but do not remove the journal title.
Remember arXiv.title or arXiv.extra in item.arXiv, title taking precedence over extra (this is only for the postscript)

Looking at your "want" sample, it sounds like what you are looking for is the following (the first matching rule is performed, no further rules are examined):

If arXiv.title is set, ignore arXiv.extra (they could be different), leave the journal title as-is, set eprint/eprinttyp/primaryClass/archivePrefix from arXiv.title
If arXiv.extra is set, leave the journal title as-is, set eprint/eprinttyp/primaryClass/archivePrefix from arXiv.extra

That's it? What would all the preferences be for then?
If you want the pages implementation, you could still do that by the existing postscript. I've contacted<EMAIL_ADDRESS>to see if they would weigh in. @KatharinaDD, do you have any of those journal recommendations? I'd like to include that in the discussion I hope to have with arXiv.

@anuragm, the docs at https://arxiv.org/hypertex/ don't mention keeping the "arXiv:..." in the journal title, and keeping it there currently conflicts with #170.
@KatharinaDD, arXiv has gotten back to me, but they consider the pages way to be a bit hacky
I've put up a test build at http://tempsend.com/F2EB73F9B3 (can't publish at the regular place because the test case for #170 currently fails) which does what I explained in https://github.com/retorquere/zotero-better-bibtex/issues/460#issuecomment-211872386 with one additional change: if you put the arXiv: id in the pages field, it will add an empty journaltitle, which should do what @KatharinaDD wants, without any preferences. I'm not committed to that workaround yet (certainly not with arXiv not being enthusiastic about it), but I'm curious what you think.

@KatharinaDD I just now noticed a comment in your first message in this thread: BBT has no influence on how arXiv is imported into Zotero, for that you'd have to contact the Zotero team

@anuragm looking at your "test-biblatex-want.bib" sample, this is BibLaTeX, right? Shouldn't it then be journaltitle instead of journal fields?

@anuragm how does Ronnow get primaryClass = {quant-ph},? Copy-paste issue?

@retorquere

Looking at your "want" sample, it sounds like what you are looking for is the following (the first matching rule is performed, no further rules are examined):

If arXiv.title is set, ignore arXiv.extra (they could be different), leave the journal title as-is, set eprint/eprinttyp/primaryClass/archivePrefix from arXiv.title
If arXiv.extra is set, leave the journal title as-is, set eprint/eprinttyp/primaryClass/archivePrefix from arXiv.extra
That's it?

Yes, this is what I will like to be done by default.

looking at your "test-biblatex-want.bib" sample, this is BibLaTeX, right? Shouldn't it then be journaltitle instead of journal fields?

Yes, I should modify my post-script to use journaltitle instead of journal.

how does Ronnow get primaryClass = {quant-ph}? Copy-paste issue?

That is what I will like to see, but there is no way to do this since the extra field contains no information about the primaryClass.

What would all the preferences be for then?

This is where the pragmatic part comes in. There are quite a few BibTeX styles which don't read from eprint/eprinttype/primaryClass/archivePrefix. In that case, we will need options. So the options might read as
Better BibTeX adds arXiv information in recommended places by default. Also add information in
( ) pages field ( ) blah blah

In any case, I don't see need for options since all this can be done by appropriate post-script, but that is up to your discretion.

You can add the quant-ph info in the extra field and bbt will pick it up.
If you want it everywhere, that would be a postscript. But I can't pull the
information out of thin air, right?
Wrt the options, I'd need a little more info on the "blah blah" part. I do
not want to add an option to accommodate every individual combination of
styles out there - if there's "pages" and "blah blah" you can bet that
there will be users that need one paper to be "pages" and the other "blah
blah",leading to more and more "if you have x and Y but not Z, then"
options. That way madness lies. Bbt looks to do what is right in the
majority of cases, preferably backed by a spec. Neither pages nor blah blah
do that.
More curious what you do about arxiv: in the journal title wrt #170 tbh

As far as BibTeX goes, if you don't populate journal field with arXiv:xxxx [yyyy-yy] for papers which are arXiv only, it will break a lot of styles out there, and arguably break work flow of lot of people. Remember there are no standard for arXiv in BibTeX (as BibTeX was finalized before arXiv was popular), and just because people at arXiv want journals to use certain non-standard BibTeX fields will not make them do so.
As far as BibLaTeX goes, arXiv information should not go into journaltitle since arXiv is not a journal. Personally I don't give two hoots about standards if my BibLaTeX style file gives me a good looking output when I put arXiv data in journaltitle. Moreover, BibLaTeX standards are for pre-prints, which is true for good chunck of arXiv papers, but not for all. There are many papers on arXiv (well cited) whose authors couldn't care less about submitting them any where. In those case, arXiv is the journal. Such "papers" are routinely cited in published research as well.
In any case, as I said, I will be happy if you had no option at all, and just confirmed to standards for BibLaTeX, which is to put arXiv data in eprint, etc and leave journaltitle blank. I am happy to use a post-script to serve my purpose.
As for options, well, I guess you will have to ask @KatharinaDD and others who want them.

I do have to care about (de jure or de facto) standards because the alternative is chaos. For highly individual choices, there's postscript.
Yes, if you have text between brackets, it will be picked up as the primaryClass.

CircleCI currently has problems, but I'm hitting all green locally on transforming these source references to this BibTeX and this BibLaTeX without any options. I'll know how well I'm doing across all of my tests when Circle is back up.

All tests run green, including existing cases, so the transformation from https://github.com/retorquere/zotero-better-bibtex/issues/460#issuecomment-212291737 could in principle be merged. @anuragm, @melsophos, how say you?
@KatharinaDD would you be helped by having the pages hack implemented as I described? It'd mean you'd have to move the arXiv ID to the pages field. If not, the postscript would work.

(to be clear, the current implementation does what https://github.com/retorquere/zotero-better-bibtex/issues/460#issuecomment-212270638 says)

For me it looks good.

I guess that is a clear way to treat it. I can hack in if needed :-)

And this would have your preference over a postscript?

arXiv confirms the "arXiv" id in the journal title for BibTeX, but says people using the pages field are pretty much on their own.
I'm going to merge and release this, except for the pages hack. It's easily accomplished with a postscript, and it's really out of spec. I can't include everyone's postscript as special cases -- sorry about that, but the postscript will always work, and it only has to be applied once.

@retorquere Everything looks good to me! Now I will go lobby Zotero to add primary class to Extra field. :smile:


FormatException - element WorkfloRulesToInject

I was using RulesEngine 2.0.0 and when I updated my application, update RulesEngine to 4.0.0.
When I run my application, it throws the exception below
And I got this error:
Message=An error occurred while deserializing the Rules property of the class RulesPackage.Model.RulesInformation: Element 'WorkflowRulesToInject' does not match any field or property of class RulesEngine.Models.Workflow.
FormatException: Element 'WorkflowRulesToInject' does not match any field or property of class RulesEngine.Models.Workflow.
So I downloaded the RulesEngine to check and I noticed that the class Workflow was updated, the element WorkflowRulesToInject was renamed to WorkflowsToInject.
But in this update, the Getter from WorkflowRulesToInject was removed, and because of that, I am getting this error.
This is occurring cause my App read the info from a mongo database collection, and the json has the old element WorkflowRulesToInject.
Is it possible to add a getter again on this property WorkflowRulesToInject to avoid this error?

Hi @ronaldoperes , the property WorkflowRulesToInject has been renamed to WorkflowsToInject.
It has been mentioned in https://github.com/microsoft/RulesEngine/blob/main/CHANGELOG.md#400
You can either rename the references in your collection or do a custom mapping.
You can also use version 3.x in case you need time to migrate.


Documentation: Add libncurses-dev ubuntu install requirement to kernel build instructions

Windows Version
Microsoft Windows [Version 10.0.22631.4387]
WSL Version
<IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[x] WSL 2
[ ] WSL 1

Kernel Version
<IP_ADDRESS>-microsoft-standard-WSL2
Distro Version
Ubuntu 24.10
Other Software
No response
Repro Steps
Follow kernel build instructions at https://github.com/microsoft/WSL2-Linux-Kernel and  and one gets this message:
make menuconfig KCONFIG_CONFIG=Microsoft/config-wsl
  HOSTCC  scripts/basic/fixdep
*
* Unable to find the ncurses package.
* Install ncurses (ncurses-devel or libncurses-dev
* depending on your distribution).
*
* You may also need to install pkg-config to find the
* ncurses installed in a non-default location.
*
make[2]: *** [scripts/kconfig/Makefile:224: scripts/kconfig/mconf-libs] Error 1

One cannot run make menuconfig without having libncurses-dev installed.
Please add it to the list of packages that need to be installed in ubuntu containers to build a kernel.
Expected Behavior
As above...
Actual Behavior
As above...
Diagnostic Logs
No response

/question


potions are black

using a 1.8.8 server, when a potion is thrown using a 1.12 client the particle effects in the world are black, not colored

I'm unable to replicate this issue on any 1.8.8 server. Ensure your potion ids are correct for the version? Maybe it's to do with custom potions?
Otherwise reopen with info on how we can replicate, potion ids etc :)


Include a sample Pod configuration yaml

What content needs to be created or modified?
I created pod deployment with the correct dapr annotations. After deployment, it worked correctly without the 'aadpodidbinding' label. When I restarted the pod, the connection to the KeyVault started to fail.
Describe the solution you'd like
After talking to MS Support for AKS, we identified the pod deployment template has to have the 'aadpodidbinding'.
Where should the new material be placed?
Include a statement at the end of the document or attach a sample yaml file for pod deployment.
Additional context
I just found a sample from a google search:
(https://github.com/RicardoNiepel/dapr-docs/blob/master/howto/setup-secret-store/azure-keyvault-managed-identity.md) [move to the end of the page]
Another concern is since we are independent of the backing services, should we attach the label  'aadpodidbinding' to the application container? It should be a part of the Dapr component yaml right??

@berndverst @ItalyPaleAle aadpodidbinding is now deprecated, right?

It has been replaced with workload identity that similarly needs to be annotated with azure.workload.identity/use: "true"
See docs here: https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#deploy-your-application


Fix playground

This PR gets rid of hostname as

hostname should not be required by playground, because it is served always from the same server the proxy is being saved so using just endpoint option to koa playground middleare is enough
the only place protocol was used was exactly in the playground. So, not required anymore
hostname causes issies while listening using http. For some reason node tries to replace the name passed as argument by the resolved ip. Setting <IP_ADDRESS> is not possible because it is treated as a hostname thus causing the need for CORS in case you access via localhost or any other name for example.

I find the best option is to get rid of anything related to hostname.

Thanks again @paulosuzart for the fix and the PR!


Support LuaRocks dependencies

Situation
Project-keeper is useful in Lua projects too. Lua does not come with a standard build environment and at least in Exasol's case integration tests with the database are best written in Java.
Dependency management is a problem though, because currently PK only knows dependencies through the POM file. Since LuaRocks (the standard Lua package format) don't show up in the POM, PK has no idea those dependencies exist.
This has different consequences:

PK can't add them to the auto-generated file dependencies.md
PK can't update the change log
PK complains when builds contain luarocks calls required to install the Lua dependencies

See also #459
Acceptance Criteria

PK can read the dependencies from a file which is also used by the build.
PK takes rocks into account when creating the dependency list.
PK accepts rocks installation in the CI build files
PK generates config files if they don't exist:

Create a default config.ld for lua-doc (must be filled with project specific content)
Create a default .busted file (can be adapted with project specific configuration?)
Create a fixed .lua-format file (ensures consistent formatting)

Shelved.


⚠ ClamAV has degraded performance

In e70fe79, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 526 ms

Resolved: ClamAV performance has improved in e2db32f.


Ubuntu 14.04 Broken

Cookbook version
2.0.0
Chef-client version
12.14.77
Platform Details
Ubuntu 14.04
Scenario:
Run default recipe
Steps to Reproduce:
Run default recipe on Ubuntu 14.04
Expected Result:
Recipe completes without error
Actual Result:
`  * template[/etc/squid3/squid.conf] action create (up to date)

execute[initialize squid cache dir] action run

Error executing action run on resource 'execute[initialize squid cache dir]'
Mixlib::ShellOut::ShellCommandFailed
Expected process to exit with [0], but received '1'
---- Begin output of squid3 -Nz ----
STDOUT:
STDERR: 2016/09/20 15:21:23| aclParseAclLine: ACL 'manager' already exists with different type.
FATAL: Bungled /etc/squid3/squid.conf line 7: acl manager proto cache_object
Squid Cache (Version 3.3.8): Terminated abnormally.
CPU Usage: 0.004 seconds = 0.004 user + 0.000 sys
Maximum Resident Size: 373328 KB
Page faults with physical i/o: 0
---- End output of squid3 -Nz ----
Ran squid3 -Nz returned 1
Cookbook Trace:
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:41:in `run_action'
Resource Declaration:
In /var/chef/cache/cookbooks/squid/recipes/default.rb
93: execute 'initialize squid cache dir' do
94:   command "#{node['squid']['package']} -Nz"
95:   action :run
96:   creates ::File.join(node['squid']['cache_dir'], '00')
97: end
98:
Compiled Resource:
Declared in /var/chef/cache/cookbooks/squid/recipes/default.rb:93:in `from_file'
execute("initialize squid cache dir") do
action [:run]
retries 0
retry_delay 2
default_guard_interpreter :execute
command "squid3 -Nz"
backup 5
creates "/var/spool/squid3/00"
returns 0
declared_type :execute
cookbook_name "squid"
recipe_name "default"
end
Platform:
x86_64-linux

`
This is the commit that broke it:
https://github.com/chef-cookbooks/squid/commit/b1bf42c4430e1491e8a0e69d5f1de41e0cf201a4
Ubuntu 14.04 uses squid 3.3 not 3.1

I am experiencing this as well.
ChefDK version 0.17.17.
Ubuntu 14.04.
Downgrading/Pinning to 1.1.1 seems to be a good work around until this is resolved.

Same issue.  Ubuntu 14.04 uses 3.3.8 (currently) by default,  this causes the template to be incompatible

Note that I raised a PR some time ago to deal with Ubuntu vs Debian some time ago: https://github.com/chef-cookbooks/squid/pull/60
Travis tests failed before I opened that PR and I've not dug further, nor have I had any feedback on the PR.

As a workaround until the PR gets merged, I've added the following to the attributes.rb file in a wrapper cookbook:
if node['platform'] == 'ubuntu'
  if node['platform_version'] == '12.04'
    override['squid']['version'] = '3.1'
  elsif node['platform_version'] == '14.04'
    override['squid']['version'] = '3.3'
  elsif node['platform_version'] == '16.04'
    override['squid']['version'] = '3.5'
  end
end

Fixed by #50


remove unused reqwest dependency

I was investigating to use this library for migrating a mid-sized iron app to axum.
While digging through source etc I found some issues, which I created as issue.
Here a short PR for a simple additional thing I saw.
( I'm planning on fixing the other issues too, piece by piece, depending on if you think these should be solved here or not )

Thanks for the PR!


Green color on Map for all positives cases recovered from that state/UT

We will be seeing 100% recovered in Ladakh, few NE states and the Andaman region in the coming days. Highlighting those areas with green color will give some positivity amidst this chaos in the pandemic.

IMO not required. We will end up with places like Nagaland that have no cases yet being white, places with recovery being green and then lots of shades of red. Although the intention seems noble, it creates confusion and the legend becomes complex.


Fix: Server does not stop

Problem: Server doesn't stop when using MinecraftServer#stopCleanly inside a command
Resolves #2096
Why this bug happens ?
During the server lifecycle commands are executed in connection().tick(msTime) (ServerProcessImpl.TickerImpl class)

When you use the /shutdown command, MinecraftServer#stopCleanly method is called which stops all TickThread
After all commands are executed, serverTick(msTime) is called which start (stopped TickThread) and wait until threads are finished
Solution: set a flag to true when stop method is called and check this flag at end of tick

i concur, when i made my /stop command i had to just put a System.exit(0) at the end.

Why don't you just schedule the call to stopCleanly?

I agree, this would be very helpful. In fact, recently, the reason for this strange behavior was being discussed on discord.

Yeah. And you don't necessarily have to do it within stopCleanly, but there should be an API note that you can't call this method from within a method that's called during server ticks without scheduling.

Yeah. And you don't necessarily have to do it within stopCleanly, but there should be an API note that you can't call this method from within a method that's called during server ticks without scheduling.

I'm sorry, but if a method that should shut down the server doesn't shut it down, that's a bug, not a feature.

I understand this, but you don't have to schedule the call to stopCleanly for every call. It therefore doesn't make sense to change the stopCleanly method to make this implementation.

Fixed in #2445


How can I listen to receive transaction or get history of transactions?

Hi
How can I listen to receive transaction or get history of transactions?

by using indexers, near cli does not support this


Add TypeScript definition

If anyone wants to work on this, see the initial attempt and feedback in https://github.com/sindresorhus/tasklist/pull/24.

any update?


Vmware expanded virtual hardware features

What does this PR do?
What issues does this PR fix or reference?
Previous Behavior
Remove this section if not relevant
New Behavior
Remove this section if not relevant
Tests written?
[NOTICE] Bug fixes or features added to Salt require tests.
Please review the test documentation for details on how to implement tests into Salt's test suite.
Yes/No
Commits signed with GPG?
Yes/No
Please review Salt's Contributing Guide for best practices.
See GitHub's page on GPG signing for more information about signing commits with GPG.

Closing as it's been merged into Salt directly


Add optional cover image captions

Give the option to provide a caption for the cover image on a page.
If provided, this caption is also used as an alternate text, increasing the accessibility for any user with a screen reader.

Thanks for your PR.
According to the MDN documentation, "The title attribute is not an acceptable substitute for the alt attribute. Additionally, avoid duplicating the alt attribute's value in a title attribute declared on the same image. Doing so may cause some screen readers to announce the same text twice, creating a confusing experience.".
So, I think the title attribute should not be set, and the frontmatter key should be cover.alt.


Mix 5.0.5: Causing UnhandledPromiseRejectionWarning error

Laravel Mix Version: 5.0.5 (npm list --depth=0)
Node Version (node -v): v12.12.0
NPM Version (npm -v): 6.11.3
OS: MacOS

Description:
Running npm run dev no longer works once Laravel mix is upgraded to 5.0.5. It's throwing an UnhandledPromiseRejectionWarning about file/directory not existing, but the files in question exist. This error prevents mix from, well, mixing.
The issue is that these files exist. And if I roll back to Mix 5.0.4, everything works swimmingly.
Full error output:
(node:26595) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '/usr/local/var/www/exedra/resources/assets/js/*.js'
(node:26595) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)

Steps To Reproduce:
package.json file:
{
    "private": true,
    "scripts": {
        "dev": "npm run development",
        "development": "cross-env NODE_ENV=development node_modules/webpack/bin/webpack.js --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
        "watch": "cross-env NODE_ENV=development node_modules/webpack/bin/webpack.js --watch --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
        "watch-poll": "npm run watch -- --watch-poll",
        "hot": "cross-env NODE_ENV=development node_modules/webpack-dev-server/bin/webpack-dev-server.js --inline --hot --config=node_modules/laravel-mix/setup/webpack.config.js",
        "prod": "npm run production",
        "production": "cross-env NODE_ENV=production node_modules/webpack/bin/webpack.js --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js"
    },
    "devDependencies": {
        "cross-env": "^7.0.2",
        "laravel-mix": "^5.0.5",
        "resolve-url-loader": "^3.1.1",
        "sass": "^1.26.10",
        "sass-loader": "^7.1.0"
    }
}

webpack.mix.js file:
let mix = require('laravel-mix');

mix.disableNotifications();

// establish known paths for sanity
let vendorPath     = 'node_modules/',
    assetPath      = 'resources/assets/',
    libPath        = assetPath + 'lib/',
    sassPath       = assetPath + 'sass/',
    sassVarPath    = sassPath + 'variables/',
    cssPath        = assetPath + 'css/',
    jsPath         = assetPath + 'js/',
    cvmPath        = 'modules/CompositeViews/',
    cvmAssetPath   = cvmPath + 'Resources/assets/',
    cvmSassPath    = cvmAssetPath + 'sass/',
    cvmSassVarPath = cvmSassPath + 'variables/',
    bvmPath        = 'modules/BinViews/',
    bvmAssetPath   = bvmPath + 'Resources/assets/',
    bvmSassPath    = bvmAssetPath + 'sass/',
    bvmSassVarPath = bvmSassPath + 'variables/';

// compile all SASS styles from modules and primary resources
mix.sass(bvmSassPath + 'module.scss', 'assets/css/compiled-bvm.css').setPublicPath('resources');
mix.sass(cvmSassPath + 'module.scss', 'assets/css/compiled-cvm.css').setPublicPath('resources');
mix.sass(sassPath + 'app.scss', 'assets/css/compiled-sass.css').setPublicPath('resources');

// combine javascript files
mix.scripts([
    assetPath + 'descriptor.js',
    jsPath + '*.js'
], 'public/js/app.js');

// combine app styles into one CSS file
mix.styles([
    cssPath + '*.css',
], 'public/css/app.css');

With Laravel Mix 5.0.5 installed, run the following command:
npm run dev
Downgrade to Laravel Mix 5.0.4 and run the same command and no error appears. The mix completes without issue.

I'm getting the same issue. I can probably share logs if needed.

Instead of using an asterisk for all files under specified folder (*.js, *.css etc), use the full file path.
For example:
mix.scripts([     'resources/js/utility/add_me_to_scripts.js',     'resources/js/filters/you_dont_wanna_forget_me.js' ], 'public/js/app.js');

I appreciate the workaround as an option, but, as far as I know, the wildcard is a valid option from Laravel Mix for both CSS and JS files. Unless that's slated to change? (I hope not.)
Using the full file path of every single file isn't a great solution when you have many, many files... not to mention that you'd have to update the mix file every time you add/remove a file.

I am also experiencing this issue.
(node:5334) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '/application/resources/assets/js/pages/*.js'
Been using this for ages:
mix.scripts('resources/assets/js/pages/*.js', 'public/js/pages.js');
Now have to list each separatly:
mix.scripts([
    'resources/assets/js/pages/_global.js',
...
    'resources/assets/js/pages/guide.js'
    ]
    , 'public/js/pages.js');

I am having the same issue as described above by OP, and the workaround is a bit of a maintenance nightmare.

i am having the same issue as described above,

I resolved this issue by changing the version from 5.0.5 to 5.0.4.

same issue here, any temp fix ? maybe previous version ?

@neoteknic
in the previous version, issue was resolved.

Ok simple temp fix :
"laravel-mix": "5.0.4", in package json and npm install

Got the same issue on the following code running<EMAIL_ADDRESS>reverting to 5.0.4 fixes the warning.
{
			apply: (compiler) => {
                compiler.hooks.done.tap('SpritemapFileRemover', (compilation) => {
                    // After the compiler is done remove the spritemap.js file created by SVGSpritemapPlugin.
                    // This workaround is needed due to the versioning plugin requiring all created files to be present
                    //  for versioning.
                    const fs = require('fs');
                    const spritemapPath = mix.config.publicPath + '/spritemap.js';
                    if (fs.existsSync(spritemapPath)) {
                        fs.unlinkSync(spritemapPath);
                    }
                });
            },
		}

any 5.0.6 to fix ?

Can someone give me a basic example to reproduce this?

Something as simple as this seems to break on my solution under ("webpack.mix.js")
mix.scripts([
'resources/js/vendor/.js',
'resources/js/security/.js'
], 'public/js/security.js');

Thank you. Fixed.

Just curious because I'm new to this ... will this end up in a 5.0.6 release or what's the process? Thank you.

(node:92562) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open 'resources/assets/css/vendor/*.css'
same package.json file with webpack-mix version 5.0.5 have this error and not render my files .
please help me solve this problem  .

@jafarchoupan updating to version 5.0.6 or 5.0.7 should fix this issue.

@jafarchoupan updating to version 5.0.7 should fix this issue.
steps :
rm -rf node_modules
rm package-lock.json
npm cache clean --force
Both version check 5.0.6 and 5.0.7 , I still have the problem .

Hmm I'm sorry, I saw that the version was bumped to 5.0.7 and was expecting that it features the fix for this problem. The only solution for now is locking Laravel Mix add version 5.0.4 :(
"laravel-mix": "5.0.4"

@tsharms
i used version 5.0.4 by the way and get this Error .
any body don't know this problem ?

@tsharms
i used version 5.0.4 by the way and get this Error .
any body don't know this problem ?

This is still pending being merged into a release. @JeffreyWay mentioned it may be back ported once v6 is finalized, hopefully.

@reidkornman
So should I run until then? which one version must be run my project ?

@reidkornman
So should I run until then? which one version must be run my project ?

I'm running 5.0.4 without an issue currently. That's what I would advise.

5.0.9, not fixed yet.

It looks like the focus is on 6.0 being released. Any plans to fix this in the 5.x branch @JeffreyWay ? It looks like 5.0.9 is out but not marked as "latest" (5.0.5 is currently marked as latest). I haven't seen anything on this since Jeff's comment 2 months ago.


Fun Menu & Remote Michrophone and Remote camera please

I want you to add remote michrophone, remote camera and fun menu please.
I want to prank people :D

remote microphone and remote camera were both removed due to security concerns (see issue #899)
From the README: "Quasar is a fast and light-weight remote administration tool coded in C#. The usage ranges from user support through day-to-day administrative work to employee monitoring." I don't think you're supposed to prank employees.


Improve index creation from scratch

Great repo! I was able to get up and running within a few hours.
I've added few changes and minor improvements. These are really focused around decreasing the adoption curve. It also might help with #21 and progress on #6.

Added a quickstart guide. Initially I found myself a bit lost trying to figure out what was needed to start and which components needed what data. I was shooting for functional docker build/images without running it natively on my host.
Added a docker-compose.yaml which stitches everything together.
Changed airmail_import_osm allowing for --pelias-url to be specified, removing the dark magic of docker container creation within the rust code and hard edges where files are expected to be located. This was very broken in my slightly cursed environment.
A few minor clippy suggestions/changes which showed up in the code I was working on.
A few minor formatting changes so I could better read the code.

Do you have a docker hub repo if you were interested in pushing the images?

Great work! If we're going to take a dependency on docker-compose/podman-compose for indexing I think we should break the dependency on bollard and remove all of the logic to start the spatial server docker container from the indexer. It shouldn't be too hard to rip it all out, but if you'd like I can do that as a follow-up because these changes do look cohesive and good overall. Removing bollard means less complexity but it also means no more docker-in-docker nonsense which has been a problem because the docker/podman socket isn't always in the same location.
If you do break the dependency on bollard, you'll be able to remove the docker socket bind mount in the compose files too.
Minor nit-pick: Pelias is a different geocoder, what we're using from the project is an experimental system that they built a while ago called the spatial server which is different from pelias itself so I don't want to call its base url pelias_url.

Perfect! I'll make the suggested changes and thanks for the clarification on pelias, I thought it didn't quite fit but now it makes sense.
I'll include a separate change with bollard and the dind capability removed - happy to do it.
Sounds good re ghcr. It was more for better container naming in the compose. I'd imagine most use-cases someone would create an index and be done with it, whereas larger users would likely have some kind of custom pipeline to integrate anyway.
I came across your repo from a Reddit post months ago which I starred and planned to come back to. Now I'm looking to include both forward and reverse geocoding in my stack (pure rust) and evaluating options. I like where this is at and what you're looking to do, so happy to contribute.

I'd love to add reverse geocoding at some point. People do seem to want it. It should be as simple as just looking up a set of s2 cells in the index then doing post-processing on the returned docs to sort them exactly by distance.

Awesome!! It would be neat. I might take a look at it later in the week(?).
7b81765 removes the dind function. I've left in the CLI parm for specifying the path to the wof database, as this could also be incorporated into the indexing component at some stage?

I have created a separate branch on my fork spatial which replaces the spatial container/server with rust native wof sqlite queries.
Performance wise, it's pretty much the same (docker makes this 5-10x slower, similar to the spatial container version). Indexing is ~45-50k POIs/s. I tested a few implementations based on futures, without async, parallel readers and a connection pool. They're pretty similar but the current connection pool approach appears to be the most stable CPU wise. According to some dirty flamegraphs the slowest points seem to centre around indexing, loading and cache lookups now. Interestingly CPU load decreases over time as does the parsing speed.
The implementation utilises a r2d2 connection manager and rusqlite. One downside I don't like but it seems to be the normal approach is it requires the libsqlite3-mod-spatialite package/sqlite extension. But you can pretty much run a few binaries and you have a functional geocoder :)

Very cool. I'm ok taking a hard dependency on a spatialite module if there's a docker image for those on weird distros without it packaged, and Windows users and stuff.
Currently indexing of ways is very slow because we calculate the centroid. I've been meaning to quantize all of the points in the way to s2 cells then iff they're all quantized to the same cell do a lookup for just that cell, otherwise fall back on the centroid calculation. From the profiling I did I suspect this would speed things up.
I'm going to review this PR soon.  Very excited about these contributions!
Another thing to look at once the spatialite changes are upstreamed is doing spatialite queries in the server itself. I'd absolutely love to be able to return shapes for admin area queries instead of just a center point. Center point isn't useful if I type "Australia" into a geocoder. Even a bounding box isn't really what I want for that query. Having spatialite on the server is also useful for coarse reverse geocoding points in the middle of nowhere. There won't be any indexed features in the middle of nowhere but it would be nice to at least provide the administrative area hierarchy.
Quick question: Are you planning on running from block storage or over an HTTP connection?

Awesome. I've merged in the native wof code.
Haha sounds good re the review and Australian/British english vs American. It's tricky hey! I try to catch myself in international codebases but forget in docs most of the time.
So, I realised I messed up the profiling and why the rust wof code was virtually the same speed as the node implementation (I expected some change), it was based on pre-cached data. Uncached is around 10-15k/pois/s, not 50k. I did have a look at the centroid calculation as well and what you suggested could help. I tried a channel approach to offload/parallelise the calculation and see what effect it had, and overall execution time was pretty similar. There's a small improvement in switching from malloc to mimalloc as my profiles showed quite a lot of alloc calls - I think this is really just a side effect of loading all of the POIs etc. from osmx.
Exactly my thinking for the bbox approach and reverse geocoding. It's those little engineering quirks which help make it useful!
My usecase is one needing to do both forward and reverse geocoding. For a given point or polygon, determine towns/regions close by/nearest. And forward is the typical usecase. I'll likely host the index on Cloudflare R2 or S3 on k8s and query with a remote index (depends on perf - R2 is a bit slow here I find for too many range queries...).
Currently I'm checking to see if I can improve some of the wof lookups as I don't like the node library is faster at the moment.

Sounds good. I'm not sure if you caught it from the website but HTTP indices are very slow. The demo is down so you can't see for yourself but I'm talking generally under 10s latency, 1-2s latency common if the cache is hot. I'd generally recommend hosting the index with a web server on a VPS that can pull the data directly from a local disk. That'll be way faster than object storage and you can still scale sub-linearly because lots of airmail instances can share one index host. R2 is the very slow (but very cheap) option. I don't have numbers on latency characteristics of lower-latency HTTP indices, so that's something you'll have to test out.
Another consideration is that Airmail enables the quickwit feature on tantivy for remote indices which switches the termdict to be sstable-based instead of fst-based. High level this makes access patterns sequential instead of random (extremely helpful if you're fetching 2mb chunks every time you get a fault a 4kb page), but it essentially rules out any possibility of performant typo-tolerant queries.
The reason this is on my mind is because taking an optional dependency on spatialite in airmail_service means allocating a nontrivial amount of local storage per instance just for the WOF database (35gb as of today). I don't have a planet index on my workstation currently but if I remember right it's in the low hundreds of gigabytes, so about one order of magnitude larger than the WOF database.
When the local disk space requirements were more like 500mb vs 300gb (approx) it was compelling to offload onto object storage for use-cases where poor latency doesn't matter. 35gb vs 300gb is a lot less compelling.

I've had a really mixed time with HTTP ranged requests querying remote indices/files. Even with optimised cog geotiff, which was kind of built for the purpose. It breaks down really quickly, one user requesting 16 256x256 tiles for an area, and each tile requires 5-6 requests to fetch the tile data is a bad user experience. But with other formats it's been amazing.
It's why geo stuff is kind of interesting right, it's the worst of all worlds. I guess including sqlite + spatialite for indexing is a slightly different problem, as you need the local space to build the index anyway. But incorporating it into the core part of the geocoder can be pretty beneficial. I guess one approach is essentially having features on/off depending on user requirements? Then users can opt-in.
Another option potentially worth considering (depending how deep you want to go...), is to rip the useful information out of the sqlite database (polygons, places and properties - at the moment), and store it directly in novel geo structures to query at runtime. For instance, pull out the polygon data, store it in a R-Tree, extract associated data etc. This might hit the small (fits on a node easily), performant, but more complexity - this doesn't solve 'search' more broadly haha.

Some more updates... I'm thinking a bit about my deployment and how it'll all fit together. Apart from forward and reverse geo, it would be neat to be able to maintain a separate index for stuff interesting to my own services/users. This separate index would need to be updated/re-created every few minutes - pretty straight forward, but I'm keen to ensure the import process is rock-solid.
I hope you don't mind that I've added these into this branch/PR. But hopefully they'll help to progress the safety, performance and reliability of airmail too. The last thing I'm trying to crack for this PR is improving the performance of the PIP code.
Updates:

Standardise error handling with anyhow and thiserror to avoid unexpected panics. Removing most unwraps and expects in indexing and related code. I will look at the service code too, just need to ensure there aren't any unexpected panics when running in prod... :)
Reduce some async executor starvation.
Switch to sqlx from rusqlite, for perf.
Merge import and index project files, as the concerns are dependent. This might be contentious (so I'm sorry if it is!) but I struggled constantly to find the indexer/import main code.). Currently there's now the airmail lib, indexer related code, and the service.
Cleanly terminate import process, terminating all child tasks. e.g. ctrl+c and it dies.

Rough perf notes:
Using R2D2 SQLite, sync default, journal default, pool size = max physical, main importer worker count = all
[2024-07-24T22:50:20Z INFO airmail_indexer::importer] 1780000 POIs parsed in 316 seconds, 5627.248985045054 per second.
Using R2D2 SQLite, sync off, journal wal, pool size = max physical, importer worker count = all
[2024-07-24T22:56:41Z INFO airmail_indexer::importer] 1770000 POIs parsed in 260 seconds, 6793.537725653552 per second
Using R2D2 SQLite, sync off, journal wal, pool size = all cpus, importer worker count = all
[2024-07-24T23:02:11Z INFO airmail_indexer::importer] 1770000 POIs parsed in 305 seconds, 5786.769726327416 per second.
Using R2D2 SQLite, sync off, journal wal, pool size = physical, importer worker count = physical
[2024-07-24T23:12:25Z INFO airmail_indexer::importer] 1770000 POIs parsed in 243 seconds, 7254.268044496302 per second.
Using SQLx, pool = physical, worker = physical
[2024-07-24T23:19:44Z INFO airmail_indexer::importer] 1770000 POIs parsed in 267 seconds, 6628.422913329843 per second.
WTF - Flamegraph runs faster
[2024-07-25T00:04:08Z INFO airmail_indexer::importer] 1780000 POIs parsed in 232 seconds, 7672.089567886623 per second.
Using R2D2, rusqlite, release
[2024-07-25T00:24:27Z INFO airmail_indexer::importer] 1780000 POIs parsed in 236 seconds, 7536.300938360011 per second.
With SQLx, rw database debug = 1 in release? This makes no sense.
[2024-07-25T00:29:45Z INFO airmail_indexer::importer] 1770000 POIs parsed in 203 seconds, 8707.955521146809 per second.
I have no idea why, but when running with flamegraph, it's even faster. what on earth.
[2024-07-25T04:32:42Z INFO airmail_indexer::importer] 1770000 POIs parsed in 189 seconds, 9338.917903036392 per second
Flamegraph is roughtly 60-70% mod_spatialite (point-in-polygon) processing. The load queue is kept full, indexing and caching are empty. I'm thinking about a better pip implementation...

Is this ready for review? At some point where you're at a good stopping point I'll make time to review and merge. I noticed that the source branch for this PR is main which might cause fun problems post-merge due to divergence, but we can cross that bridge when we get there. As long as it's not substantially slower than before I'm fine with being bottlenecked on PIP queries. Indexing in airmail is already crazy fast compared to a lot of other geocoders so I don't think it's a huge issue.

Ok awesome. I've renamed the branch so it'll make merging easier (I didn't realise it'd close this PR). What I might do is finish off this tranche I'm working on and submit it as a new PR when it's ready. It's definitely a bit bigger than I thought originally... But hopefully it helps the project overall.
I've managed to improve/fix the pip query speed, with a an optional r-tree index. The tradeoff is the pip index takes ~5 minutes to build and 10GB of memory, but it's ~2x faster than the pip sqlite queries.
I'm just running a planet import to see how it goes. I think it's in the ways loading step as it seems single thread CPU constrained. Otherwise the Australia region takes 75 seconds :) Planet is 28820000 POIs parsed in 19492 seconds, 1478.5547875774112 per second.

That makes sense, just keep me posted. I've spot-checked your work and it's solid, I'll review it even if it's a huge PR though I may not get to it right away. There may be some back-and-forth but I don't have any major objections so far. Like what am I going to do, ask you to put the unwraps back in? 🤣
I noticed you removed the separation between the indexer crate and the OSM importer, which seems fine to me but I want to make sure you don't do anything that will make it harder to index OpenAddresses and (eventually) TIGER data.

Haha I mean, I could put them back in. I think most things I've worked on are dual purpose, if it's 11/10 stable then confidence improves, but it also makes it easier to maintain and improve on. And selfishly, it also makes it easier to integrate with as well. I was also thinking about why the demo might be dead too...
I kind of hope that might be the most controversial change. I noticed the deps were pretty much identical and thought most future use cases would consist of indexing + loading one source or another, but you wouldn't ordinarily index or load in isolation. So adding a new source would likely be adding some new args, dedupe code and reusing everything - and hopefully it's a bit simpler and easier to manage too.
I'm super relaxed at reviews either way 😀 please take your time.


How to use with classic desktop app and Radial Controller

Firstly thanks for the tremendous work in putting this together.  Some awesome engineering in here.
Watched the CppCon video with a lot of respect.
So, I have an interop issue...   I'm using the Radial Controller in a classic desktop application and have it mostly working in the C++/WRL Raw style like that long Hello World in the CppCon video.  Would love to just do the whole thing with cppwinrt but there's some snags.
This journey started when I wanted to remove some default menu items.  There's an api for it, SetDefaultMenuItems and in C# it's trivial to give it a list of the enums for the menu items you want.  Not so in classic desktop.  I have tried many approaches to bridge the gap.  Thought I had it with this project.
One of the issues is that there's a separate api for getting a Radial Controller for an HWND.  This is exposed by IRadialControllerInterop which you can get like this:
RETURN_IF_FAILED(Windows::Foundation::GetActivationFactory(
HStringReference(RuntimeClass_Windows_UI_Input_RadialController).Get(),
&_controllerInterop));
It's defined by the header in C:\Program Files (x86)\Windows Kits\10\Include\10.0.15063.0\UM\RadialControllerInterop.h
I need to call a method IRadialControllerInterop::CreateForWindow but this is not wrapped within cppwinrt, only a RadialController::CreateForCurrentView which takes a CoreWindow.
I have the same problem with IRadialControllerConfigurationInterop::GetForWindow which returns an IRadialControllerConfiguration.  It's this interface I need to use in order to change the default menu.  There's only a RadialControllerConfiguration::GetForCurrentView that also takes a CoreWindow
IRadialControllerConfiguration::SetDefaultMenuItems takes an IIterable.
Tried collections.h but that only works for C++/Cx and consuming the extensions.  Thus far haven't found a good way to get from std::vector to something it wants.  Of course I'd like to just pass the vector but unless I can use the projected version, I'm a bit stuck.
Thanks again!

Firstly, you need to make it very clear that my management needs to give me a Surface Studio so I can really help you. 😉
Seriously, it should not be too hard, although I don't actually have a radial controller so I can't verify. Since there's no metadata for interop APIs, you need to write a helper function of some kind to bridge the gap. Here's what I'd do. You can use a function like this to create a RadialController object for a given HWND:
RadialController GetRadialController(HWND window)
{
    auto factory = get_activation_factory<RadialController>();
    auto interop = factory.as<IRadialControllerInterop>();

    RadialController controller = nullptr;

    check_hresult(interop->CreateForWindow(window,
                                           guid_of<RadialController>(),
                                           reinterpret_cast<void**>(put_abi(controller))));

    return controller;
}

You can then stick to the C++/WinRT language projection from there on out:
HWND window = GetWindow();

RadialController controller = GetRadialController(window);
printf("degrees: %.2f", controller.RotationResolutionInDegrees());

You can do much the same for creating a RadialControllerConfiguration object:
RadialControllerConfiguration GetRadialControllerConfiguration(HWND window)
{
    auto factory = get_activation_factory<RadialControllerConfiguration>();
    auto interop = factory.as<IRadialControllerConfigurationInterop>();

    RadialControllerConfiguration config = nullptr;

    check_hresult(interop->GetForWindow(window,
                                        guid_of<RadialControllerConfiguration>(),
                                        reinterpret_cast<void**>(put_abi(config))));

    return config;
}

You can then stick to C++/WinRT to use it. The language projection will automatically bind to a variety of C++ containers to make the SetDefaultMenuItems call work. Here I'm using an initializer list:
    RadialControllerConfiguration config = GetRadialControllerConfiguration(window);

    config.SetDefaultMenuItems({ RadialControllerSystemMenuItemKind::Scroll,
                                 RadialControllerSystemMenuItemKind::Zoom });

You can also use a std::vector if you wish:
RadialControllerConfiguration config = GetRadialControllerConfiguration(window);

std::vector<RadialControllerSystemMenuItemKind> items
{
    RadialControllerSystemMenuItemKind::Scroll,
    RadialControllerSystemMenuItemKind::Zoom
};

config.SetDefaultMenuItems(items);

Hope that helps. And please send me a Surface Studio!

Indeed it does! Awesome thanks!  It works :)
Note I don't have Surface Studio either but the Dial works with a Surface or a Dell Canvas or pretty much anything that has Win10 Creators Edition on it as long as it will pair with the Dial over Bluetooth.   It's a cool accessory.
Yes, Microsoft, please get Kenny a Surface Studio for all the great contributions he's made to the C++ community over the years.
Now in porting the raw stuff to cppwinrt, sometimes I don't feel 'with it'  Especially when the compiler spews what seems to be hundreds of With statements.  That's a slight drag to working with templates but I've run into the same sometimes with STL. It is surprising how much spew a few misteps can make.
Thanks again for all that you do!
Keith
(Jer 29.11)


disconnectFromLocalNetwork: reject the promise there is an unhandled exception

It's pretty easy to get "NetworkCallback was not registered" error when disconnecting from a local network which is unhandled one. The PR handles it and rejects returned promise if it has been thrown.

Thank you for this pull request. I think the disconnect func will always has an error when there's NetworkCallback registered so it's better to just resolve it silently in catch instead of throwing an error. What do you think?

Need to investigate why it throws the error. I guess, that's because unregisterNetworkCallback is called once the app connects. Why do we do so instead of bindProcessToNetwork? I understand that there's a good reason for that since bindProcessToNetwork was used previously.

Need to investigate why it throws the error. I guess, that's because unregisterNetworkCallback is called once the app connects. Why do we do so instead of bindProcessToNetwork? I understand that there's a good reason for that since bindProcessToNetwork was used previously.

Sorry I meant when you don't have a NetworkCallback registered it will always throws an error


Efficient training of streaming Transformer encoder

The previous implementation adopts reshaping-based encoding to shorten the input length for efficient training.
Therefore, there is some redundant input encoding.
Although this is effective, the model consumes additional memories.
As a result, I had to reduce the batch size.
In this new version, contextual information is controlled with masking only.
Thus there is no redundant frame and the same batch size as in offline models can be used.

Codecov Report

Merging #102 into master will increase coverage by 0.32%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #102      +/-   ##
==========================================
+ Coverage   38.68%   39.00%   +0.32%
==========================================
  Files          83       83
  Lines        9873     9922      +49
  Branches     2063     2077      +14
==========================================
+ Hits         3819     3870      +51
  Misses       5849     5849
+ Partials      205      203       -2

Impacted Files
Coverage Δ

neural_sp/models/seq2seq/encoders/build.py
0.00% <ø> (ø)

neural_sp/models/seq2seq/encoders/transformer.py
73.11% <63.88%> (+0.76%)
:arrow_up:

neural_sp/models/seq2seq/encoders/conformer.py
79.28% <68.88%> (+3.57%)
:arrow_up:

neural_sp/models/modules/cif.py
87.34% <0.00%> (-1.27%)
:arrow_down:

neural_sp/models/seq2seq/decoders/las.py
50.61% <0.00%> (+0.11%)
:arrow_up:

neural_sp/models/modules/multihead_attention.py
96.55% <0.00%> (+1.14%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update adbdade...9066d0f. Read the comment docs.


ipinfo man page

Includes the man page written in roff markup, parsing the man page to print formatted --help output on the terminal and handling the installation of the man page to the user's local machine.

@UmanShahzad

This is how -h or --help for ipinfo looks like:

Currently, git --help also prints something on the terminal instead of opening the man page:

This is what git -h looks like:

Upon further research into how Git maintains documentation for its git command, it appears that the source file is a plain text file written in AsciiDoc format—a more human-readable option. This text file contains the same sections as the man page, including NAME, SYNOPSIS, DESCRIPTION, COMMANDS, OPTIONS, EXAMPLES, etc. This file serves as the source for both the man page (often converted from AsciiDoc format) and the --help output. In either case, we would need to custom-parse the source file to extract the desired sections for printing the --help output. Instead of parsing the potentially cumbersome man page format, we can parse this more human-readable file.

@Taimoor-12 let's try an approach that doesn't involve any actual man file, but acts like one.
The main benefit I personally find from a man page is that it's very detailed and searchable using commands you find in pagers like less.
So can we find a way where -h (short help) is the same as today, but --help (long help) ends up printing a different help string for the same command that's the "full detailed" one and that runs in a pager? The key part I don't know myself is how would a CLI open something in a pager. I've seen e.g. psql open up a pager when the results returned from making a query are too long, for example, so it must be possible.

@UmanShahzad If I understand correctly, we want to put all the information about the ipinfo command and its subcommands, including examples and options, into a simple text file. When --help is passed, we will pipe the contents of that file to less. For -h, we can continue using the current hard-coded string version. Is that accurate?

Even for --help it can be a hardcoded string inside Go (or embedded via Go embed), but just paged instead of printed, so we can freely write pages and pages of details and examples.

What kind of a structure should we follow for the hardcoded string that will be paged with less when --help is used?

@Taimoor-12 we can follow our own structure at this point - anything that just allows us to communicate a lot more examples than currently. It could be the same structure but with just more details, explaining all edge cases, etc.

The output for --help after the new commit is this:

I think yes. I encountered a CLI called pgcli and went through their docs on the website. They mentioned that the default pager for displaying long result sets is less. In their config file, users can explicitly specify the pager value, and the specified pager would be executed to display result sets (or simply less if nothing is specified). That's what I understood
Through the use of flags and when --help is invoked, I directly executed less on the long string.

Yes the pager can be set through an env var called PAGER which is a common name to set the pager tool. We can read its value and use that pager for the string. If the user has set no PAGER value, we can then use less as a pager.

Alright let's incorporate that!


Fix Windows Hardcoding

Compiles on Linux & co. now
And also consider getting rid of the partial busy loop as a default.
It makes 0 visual difference and just eats up CPU.
I think it has LESS input lag when it's off, but maybe that's just me, and also shows how little difference it makes that i don't even really know.
I know that's not part of this PR, but something to consider!

@mr-sihc thanks for the review. Note that the busy wait loop could be disabled with a flag is desired.

Actually, #include "SDL.h" is the official SDL-recommended way to do it.
I think you should review how you build raylib yourself, this change is not a good idea.

Its how its installed on Arch and everywhere else

That's not true
pkg-config --cflags sdl2 returns -I /usr/include/SDL2

then the Makefile does it wrong, which i used, and the cmake file has outdated doesnt even list SDL as an option

@mr-sihc @Peter0x44 Just reverted this cahnge, checked the official SDL examples and they use #include "SDL.h". Following the same pattern as the creators of SDL.

@mr-sihc Sorry, you're just wrong.
There's no other way to put it.
SDL2/SDL.h only works by accident, because your compiler automatically searches in /usr/include.
Both pkg-config and sdl2-config indicate that #include "SDL.h" is the intended and endorsed way to include the header.
It has nothing to do with "subprojects" or not, in fact it actually accommodates "subprojects" better to do it this way.

i like this, are you going to include this failsafe in the makefile?
I'm not sure. I do think it's a good idea, but I don't know if Ray would approve. It could work on Windows too, but it needs a posix-sh type shell, not cmd.exe.
A better idea might be to put it behind an:

ifeq ($(PLATFORM_OS), $(filter $(PLATFORM_OS), LINUX BSD OSX))
# ...
endif

I'm unlikely to have enough time to formalize a PR for this, but I'm happy to review any such PR if you choose to submit it.

i like this, are you going to include this failsafe in the makefile?

I'm not sure. I do think it's a good idea, but I don't know if Ray would approve. It could work on Windows too, but it needs a posix-sh type shell, not cmd.exe. A better idea might be to put it behind an:
ifeq ($(PLATFORM_OS), $(filter $(PLATFORM_OS), LINUX BSD OSX))
# ...
endif

I'm unlikely to have enough time to formalize a PR for this, but I'm happy to review any such PR if you choose to submit it.

Hmm, i'll see tomorrow if i could make it work.
But no promises i dont know much about makefiles xD!
If Ray even approves... Would this be a nice thing in the makefile?


should be able to pull changes for a single observer

 Issue by jmesserly
Originally opened as dart-lang/sdk#14090

currently, Observable.deliverChanges delivers to all observers, this is not ideal. see:
https://github.com/dart-lang/bleeding_edge/blob/29051a999ea1fd3fdb7eece67af9f7f3a9fa13bf/dart/pkg/observe/lib/src/observable.dart#L54

 Comment by jmesserly

I made some progress on this, but it's not going to make M8

Removed this from the M8 milestone.
Added this to the Later milestone.

 Comment by clayberg

Removed this from the M9 milestone.
Added this to the 1.1 milestone.

 Comment by sigmundch

Removed this from the 1.1 milestone.
Added this to the 1.2 milestone.

 Comment by jmesserly

ideally this API change would be done for M8, but I don't know if that's realistic. It is a fairly simple change on the package:observe side, but I didn't see any way to implement it without copying or forwarding most of dart:async APIs.

Added this to the M8 milestone.

 Comment by sigmundch

Removed this from the Later milestone.
Added this to the M9 milestone.

 Comment by anders-sandholm

Removed Library-Observe label.
Added Pkg-Observe label.

 Comment by sigmundch

Added Polymer-P-2 label.

 Comment by kasperl

Removed this from the 1.2 milestone.
Added this to the 1.6 milestone.

 Comment by sigmundch

Removed this from the 1.6 milestone.
Added this to the Later milestone.
Removed Priority-Unassigned label.
Added Priority-Medium label.

 Comment by jmesserly

Removed Area-Polymer label.
Added Area-Pkg label.

 Comment by sigmundch

Removed Polymer-P-2 label.
Added Polymer-Milestone-Later label.

 Comment by jmesserly

https://github.com/dart-lang/observe/issues/5

Added MovedToGithub label.

 Comment by sigmundch

Removed this from the Later milestone.

 Comment by sigmundch

Removed Polymer-Milestone-Later label.
Added PolymerMilestone-Later label.


Global attribute-Event-List is not implemented in chip-tool.

Problem
As per Spec (7.13. Global Elements) chip-tool need to be implemented Mandatory global attribute event-list.
Below mentioned cluster needs an EventList attribute implementation

[ ] Low Power Cluster
[ ] Keypad Input Cluster
[ ] Application Launcher Cluster
[ ] Media Input Cluster
[ ] Wake on LAN Cluster
[ ] Channel Cluster
[ ] Media Playback Cluster
[ ] Audio Output Cluster
[ ] Target Navigator Cluster
[ ] Application Basic Cluster
[ ] Content Launcher Cluster
[ ] Account Login Cluster

Platform Version(s)
Chip-tool - RPI-4, 8GB RAM
Commit-ID: https://github.com/project-chip/connectedhomeip/commit/25e241ebcbf11b1f63dbe25546b1f10219866ad0
Dut application: tv-app
PFA Spec screenshot below:-

PFA Chip-tool screenshot below:-

Duplicate of https://github.com/project-chip/connectedhomeip/issues/15011


Add electronic cutting height support for old automower identifiers

The 320 and 330 have the same hardware as 420/430 and could be updated to the same firmware.
Fix for #367

Okay, now I get it. But that would enable the support also for all other 320 and 330 who haven't the 4xx firmware?

As far as I know to install the connect modul you need a firmware update to the 420/430 version.
Without the module you can not use this integration.

Is there something I need to do to get this merged?

Sorry, I was quite busy. Think this could work.