Suggest Prefix encoding instead of a version number

Bogdan suggested creating a pull request for feedback on the spec.   I wanted to confine myself to only those things we NEED to agree on.    The first version Bogdan only described the two level ID format (presumably we would have another version number for the hierarchical format.  However this 'version at the top level' approach has the problem that we don't capture the fact that both formats agree on the TraceID and having a format that allowed the Trace-id to be parsed before 'diverging', would be useful.
My counter proposal is what I can a 'prefix encoding.   Instead of having a version number up front, each component of the ID has a unique prefix (effectively format code) that defines the meaning and syntax for that part.   This allows us to define both the Span-id component (used by Bogdan) and the hierarchical-id (used by Microsoft), while BOTH share the definition of Trace-id (and in fact the flags, but that is not as important).
This scheme also makes for a very nice versioning story.  We can add new components WITHOUT BREAKING existing parsers (they ignore the extra information).   This also allows a transition where you might CHANGE a components meaning by first providing BOTH (so it works with both old and new), and then when all relevant parsers have been updated, you can send just the new format.
The format also happens to be a bit shorter.
Note that I happen to pick a set of characters for the prefixes, but I don't really care too much.
Note that the result is that from Bogdan's point of viewpoint, instead of
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
he generates
*4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7~
Which is pretty much the same.  There is no requirement that he support anything else, but it would be nice if he saw
*4bf92f3577b34da6a3ce929d0e0e4736.342.23.ad.34~
His parser would at least recognize the TraceId before bailing
possible variations
In the proposal I made the flags a single bit that we have defined (the ~ prefix), but we can easily define true flags (e.g. we could define that the | prefix is followed by two hexadecimal digits which are flags where we define the individual bits).   I did not do this because it seems likely that we will have very few flags, and so we can simply define prefix characters for each of them.
Note  that if we feel like we are running out of punctuation to use as prefix characters, we can always create two character prefixes.
Finally if desired we could eliminate the leading *, it just means that the trace-id is REQUIRED (we can never replace or change it).
quick link
The diff is not really very useful, you may wish to simply read the new spec at https://github.com/vancem/tracecontext-spec/blob/master/HTTP_HEADER_FORMAT.md.
@bogdandrutu @lmolkova @jacpull @adriancole @SergeyKanzhelev

Let me start with some high-level feedback about the design:

Overall looks very good.
I would like to keep 1-byte for the trace options / trace suggestion and define maybe few more bytes. Some vendors use 2-3 properties. One of the useful property is "deferred sampling" which can be used for example in a LB to ask the service to maybe send back the sampling decision for this trace. I know this will force us to define a response header, etc., but I would like to have that option to control more bits in the options.
We should have some "default actions" defined when things are missing now that we have everything optional.
Is more like a question. Do we want to have trace-id required? I think it is by far the ID that we all want to use.

There are minor other minor things but I will address them after we fix the high-level things.

What happens when one cant read or understand one of the encodings, yet
needs to trace downstream? This seems to be one of the problems of trying
to do switch statements in header encoding.. wouldn't the next node clobber
the incoming data? Or are you expecting to send multiple trace-context
header values in this case?

@adriancole
When encountering an unrecognized encoding, simply ignore it and proceed with parsing the remnant and leave as is for downstream partners. This would be analogous to encountering an unknown header and simply passing it forward. And unlike the use of a separate header, using the prefix design on the same header helps simplify the propagation, especially if transport needs to be explicitly handled by non-infrastructure code.

@bogdandrutu

I would like to keep 1-byte for the trace options / trace suggestion and define maybe few more bytes. Some vendors use 2-3 properties. One of the useful property is "deferred sampling" which can be used for example in a LB to ask the service to maybe send back the sampling decision for this trace. I know this will force us to define a response header, etc., but I would like to have that option to control more bits in the options.

I think this is fine.   have ~ (or some other character, be a prefix that is followed by a hexadecimal digit (or two, if you real thing more than 4 will be defined that will be common), is fine by me.

Is more like a question. Do we want to have trace-id required? I think it is by far the ID that we all want to use.

I am OK with this as well (as mentioned, it just means we can't play with compressed formats later, but I think that is probably OK).  If we do that we can remove the * prefix.   If we don't remove the prefix the difference between saying its requires and not pretty much disappears (if it is not there you pretty much have to ignore everything, which is what you would do regardless of whether we said it was required or not).

We should have some "default actions" defined when things are missing now that we have everything optional.

Yes, however the main default action is that you can always simply give up (ignore everything completely), and you are encouraged to at least parse what you can  and use that (thus if you find the trace-id you can do so)
It sounds like we are close enough that what we should do is accept this PR, and then make the deltas from there (although if you want I can take a wack at incorporating your feedback, but I suspect you know better than I exactly what you want).

@adriancole - I think your scenarios are definitely interesting.   What I would like to do however, is to put the analysis of these kinds of things in a separate section because it otherwise makes the spec more complicated than it is.   Ultimately we need to insure that any things parsers have to do are put back into this first section, but the rationale /scenario analysis can be later (for those who care).
Note that we do have a problem with respect to 'pass through', because right now new prefixes can define ANY mechanism for deciding when the data associated with the prefix ends.   To make it so you can 'pass along' we need to have some sort of uniform 'stop' mechanism (for example defining the set of allowable prefixes (probably we just pick a set of punctuation that is big enough), and indicate that data sections CANNOT use those characters in their data (they would have to escape them somehow).  That allows parsers to skip unknown prefixes.
I think this is a good idea.
My recommendation, however is that if we are happen enough with the general prefix architecture, we should merge that, and then make more PRs for suggestions from there.

I dont think we can punt this part honestly. Before there was one encoding
and no chance of a collision. This change tunnels a completely different ID
scheme into the same header. If it were a different header we could punt. I
would expect if we merged this that the next hop would actually restart the
trace as it cant reasonably opt out of tracing just because someone sent a
hierarchical header.
Iotw, this adds significant debt which is worth avoiding. It isnt simply
interesting, this would happen in all tracing except yours, right?

@adriancole hierarchical postfix is a good way for proxy to indicate retries. Proxy can add .1, .2, etc. instead of generating new span identifiers. Which I expect can be quite a common use case.
W.r.t. loggers implementing it - generally loggers that send data to separate storages do not "trust" each other completely. However still wants to provide a link to the incoming request identity. So one may decide to put all unrecognized characters into the property bag associated with the span. This may be a suggestion we can put into the additional section on how to handle such cases later as @vancem suggests.

I dont think we can punt this part honestly

@adriancole I am not suggesting we punt it.   I am just suggesting a structure for the document.

If it were a different header we could punt

While I am not saying we should punt, I do think that punting is an option (in the short term).   Implementations are free to think of any the hierarchical data as being in a separate header and completely ignore it.   That has interoperability ramifications, so ideally we give guidance, but it certainly is an option.
I view this specification as best done in stages.   The first stage is to define some basic format with enough flexibility that all logging systems can

Express their information so that they can make a homogeneous system work.
Standardize at least the parts that are likely to be common (in our case the Trace-id) and define its semantics.

This allows a certain amount of interoperability.   It is unclear that this level of interoperability is useful, but it seems like it might be (at least you can gather together all information for a given trace, which is a HUGE step forward).
We should nail down this stage one first.    At the least, this allows us to both implement homogeneous .solutions in the short term with the POTENTIAL for interoperability.
We can also of course start thinking about stage two (which is what your questions are about), and writing guidance (in this doc) for that.     My expectation is that mostly this additional analysis/guidance will NOT effect format (it is mostly about rules of how to interpret and update the ID), but even if it does affect format, the format is sufficiently flexible/versionable that it should not be a problem.
I just want progress.  Stage two is much less defined than stage one.  Lets get the merge done for stage one and get it stable as we can get it,  (we can also work on the other stage concurrently, I just don't want it to block stabilizing the stage one spec (which I think we are close to agreement on).

@bogdandrutu - how do you want to handle this?   I suggested merging this PR and modifying it, but I am also happy to make modifications first.   What do you want done?

I think this issue basically changes the charter of this spec to carry
unlike and incompatible formats, which is why I feel a bit concerned about
the several requests to quickly merge it.
I get a sense that from the submitters' POV (microsoft in this case), you
feel the trace-context header is firstly not for interop, rather it has a
more primary goal which is tunneling. I am not sure everyone or even a
majority agree with this. If it were something to agree on, probably a
top-level issue on the charter is a better way to frame vs a sidecar on
format encoding.
I asked Mark N who helped lead http/2 and other specs and he isnt aware of
any prior art for "tunneling headers" ex passing incompatible data through
the same name. Probably worth at least talking with people familiar with
http rfcs as a part of expanding charter along these lines as it is a bit
odd so worth it.
If we were to have and define multiple formats, it would seem rule of three
would at least apply here. While there are multiple tracing systems who can
use the format previously maintained, it is unclear if any non microsoft
system is or would use the hierarchical format. If not, how would we make
sense of defining a top level format as such? What would happen if say
toyota asked for a different format (trust me there are at least 10!)..
would we simply accept and add to registry each deviation? How would that
work?
Personally, I do see value in what the hierarchical id is trying to
accomplish, it isnt a technical detail more that we are discussing what
some besides me believe to be an interop spec. If that is the case, we
really ought to do prioritize dilligence over decisions as say we just
merged fast as requested.. would that be a great idea? We would have taken
a spec that can be implemented in multiple systems and broken that without
a fix.

Taking that last comment a step further I think this PR should be limited to just the format discussion. The hierarchical ID part should be moved to a separate PR where it can be better explained and discussed.

had a chat w/ @basvanbeek I think @tylerbenson's comment about understanding formats is the biggest thing we should start with. For example, bas mentioned maybe we have too much focus on convergence without understanding the landscape fully. It may be easy for existing dapper-like systems to use the format already in place, but very hard for others, and we should know more about the impact.
Let's make a trace-contexts.md file or similar which describes known formats (regardless of whether they are interested in converging on trace-context or not)? This should include truncation impact. With some of this in mind, we could know for example, if some IDs are contributory (Ex append data via path additions etc), incremental (one link incrementing a number from a prior node), or opaque (ex traditional dapper IDs)?
Maybe through this, we can help run through the problems and what's worth solving by propagated correlation IDs (ex imprecise within a trace) vs trace IDs (ex putting something precisely at a place in a trace).
I'll offer to start that file (ps suggestion is file vs #4 as the latter is presumptuous about adoption, when the first goal is understanding)

@adriancole this is correct. We have a problem that is hard to solve with the trace-span combination and we can discuss it. For the sake of this comment - let's assume we agree that we need a way to restore span relationships in high-loss environment and we agree that the hierarchy is a way to go.
The way we are going to use the proposed format is to use all defined pieces of it. We are going to use <trace-id> to indicate the common identifier across the trace, we are going to use <span-id> as a base for the hierarchy and than every layer is free to just append `.X' to the end or reset the base of hierarchy. So this proposal is not an attempt to tunnel two different formats - it's a way to extend the format for our scenario.
The reason we propose to make <span-id> optional is that for mobile devices initiated the trace and just generated a 128bit number we want to save space and not generate another 64bit. So the first few spans which may still be on the device the correlation "string" will be shorter.
That's said in the discussion with @bogdandrutu we decided that since all pieces except <trace-id> are optional we may want to support extensibility this way.

Do you think we need to add the requirement as a section in the document in this PR?

@adriancole  I have no problem having a document as part of this that addresses the interop scenario.
Indeed a first stab that we (Microsoft) made internally at trying to make a ID standard tried to do this.
https://github.com/Microsoft/ApplicationInsights-Home/blob/1a5deefda4e26afee796bd7675e77b2f40391597/Correlation/Experimental/HttpRequestCorrelationSpec.md
By the way the first section of this doc does try to present goals for the standard as a whole.
Basically the idea was to make the ID 'successively more refined'.  At the top level it was just a string and had no structure.   At the next level if you put a - in the string it was assumed that the first part was the TraceID and the second was the span.     It then addressed what to do if you did not want variable-sized ID (you hash), but frankly the hash was basically really just a character->binary converter (which you needed anyway).  Thus fixed-sized ID systems would 'just work'  with their own ID (the hash would leave them unchanged), but would do something sensible (hash) on anything else (thus there would be no illegal IDs).
The scheme above I think has merit, because it keeps the mandates to a minimum (it allows lots of ID formats, but they all can still interoperate to some degree).    However we knew that since most of the logging world seemed to be 2-level, we decided that we just wanted the hierarchical format to  'not be excluded'.     What this meant was that since both schemes had the notion of a TraceID we wanted to be able to participate in that at least.    This is what the proposal above does.
Note that any 'technical debt' (not saying exactly what the propagation logic should be), ONLY has a negative for he hierarchical case, but that is STRICTLY better than the original proposal (which prevents systems with hierarchical IDs from even sharing their TraceID).
Really all I wanted in the short term was to allow that the format as FLEXIBLE (it had chunks below of which parts could be used (and new parts added in the future).
What I really want, however is to see PROGRESS (lets agree on what we can agree on).   It seems like one thing we can agree on is that the SYNTAX for the ID should be such that we can VERSION things and that we can SHARE parts that are the same (e.g. TraceID).     That part is hopefully relatively uncontroversial, and we should at least nail that down.
To that end, I am OK MOVING the hierarchical format to another document/section.   Just let me know if you want it ripped out and I will rip it out.   PROGRESS is the key thing here...

I have a few concerns with this (and sorry for the wall of text):
I think this change makes it noticeably more difficult for implementers. I think it has a higher cognitive load to understand all the possibilities and options, figure out how they relate to your tracing system, figure out how you can/want to handle formats that come from different systems, and actually implement those concretely for your tracing system.
I think it's a bit hand-wavy to say implementers can just ignore the prefixes that don't apply to them. Many many devs don't want to or have the time to understand the full implications of their implementation of a spec, especially in regards to interop. They usually stop at the point of "yeah this seems to work for my use case" and call it good. In practice different people will create their parsers in different ways given the ambiguities and intentional openness of the prefix-based format, and the reality of their decisions will cause a wide range of strange bugs and broken tracing in the wild.
We can mitigate this a bit with more documentation around recommended ways to parse and fulfill interop in the prefix-based format, but that increases the cognitive load even more, leading to more potential for bugs and less desire to participate in and implement the spec. From a personal anecdote perspective, I find the existing spec to be short, easy to understand, and easy to implement. Low cognitive load and high testability. I had to spend significant time reading through the diffs and comments here for this pull request to really get what was happening and how I might fulfill the new spec in my tracing system.
Again, it's easy to say "we can't fix other people's apathy or incompetence - the spec has the information in it necessary to do the job - if you can't hack it that's your problem", but if the ultimate goal of this spec is successful interop between as many tracing systems as possible (even if it's just partial or best-effort interop) then I think this new format is likely to work against that goal (sorry for the bold - I don't want this point to be lost in the wall of text).
I'm also not a fan of the idea that if I don't understand an incoming format then I should just pass it downstream unmodified. As a tracing tool implementer my tool has a job to do - if I see a header I don't understand, I'm much more likely to assume a bug in the caller (which unfortunately happens much too often) and clobber it so that the downstream systems will have a tracing context that I know works. As an HTTP service creator and maintainer that's the behavior I'd want to see too. I'll likely be able to extract at least the trace ID and keep that going, but will that always hold true in the future for such an open ended format? What about other implementers that may not be as diligent and careful with their parsers? Again, the higher cognitive load and openness of the format means higher chance of tool A not understanding tool B's format, raising the chance of their parser completely blowing up due to bad exception handling or naive parsing implementation.
I don't want to completely lock out formats that don't fit into the precise current definition of 16 byte trace ID and 8 byte span ID though. I agree with some others on here that we should start with an attempt to understanding existing formats. I imagine we'll start to see patterns that are conceptually similar and can be grouped together. For example there's the current spec's version 00:
• <Version>
• <Trace ID> (32 char lowercase hex string)
• <Span ID> (16 char lowercase hex string)
• <TraceOptions> (optional)
• Previous bits separated by a dash

The hierarchical ID format described in this pull request seems to share a lot conceptually, except the "span" in that case is a more free-form hierarchical ID. So why not a version 01 specified like this?
• <Version>
• <Trace ID> (32 char lowercase hex string)
• <Span ID> (unbounded string that cannot contain a dash but is otherwise unconstrained)
• <TraceOptions> (optional)
• Previous bits separated by a dash

And as a hint to help with interop for hierarchical ID systems, maybe one of the trace options for version 01 could indicate that implementers should append a new span ID onto the old one rather than throwing the old one away and creating an entirely new one? That seems like it could potentially cover a lot of tracing systems that don't fall into the strict 16 byte trace ID and 8 byte span ID bucket.
Maybe after doing an audit of the tracing landscape we find that there are only a small handful of these kinds of patterns, and we could have a few well-defined and well-constrained (and similar-feeling) versions that reduce the cognitive load and possibility for interop bugs, but don't cut anybody out. Maybe that's overly optimistic but I think there are significant real-world benefits to the versioned explicitly-defined formats vs. an open-ended prefix format that help with adoption and interop, and I'd like to see the analysis done before abandoning the versioned formats. And as mentioned the analysis could have other benefits even if the prefix format is ultimately adopted so I don't think it's a wasted effort regardless of the outcome.
If anyone's bothered to read all this I salute you! 🖖  :)

I have updated the pull request to pull the Hierarchical (now call structural) IDs into their own document.
This is meant to emphasize that for those who like the simplicity of the specification before this PR, you can have this.
@nicmunroe - I like to think that the cognitive load is really not significantly higher in this proposal than the original.  You have two numbers and some options.
But there is real value in doing things this way rather than the original.  Before this PR, if you need to change the layout of the ID FOR ANY REASON, then you need to update the version number.   However all existing systems will NOT RECOGNISE this new number, and thus are very unlikely to do something useful.   In all likelihood this will prevent version 2 from every being created (since you would need to insure that all parts of the system understand Version 2 before any part of the system can start using it).   More likely new systems will simply use another HTTP header (e.g. Trace-Context-2).
The goal of the tagged values is to allow INCREMENTAL improvements (or divergence), in the IDs.   Thus you can add  NEW tagged values and the old parsers will still at least parse what they know.   This is an important property of the system, and can be achieved with the VERY SIMPLE proposal here.
I understand that there are MANY issues associated with interoperability if we allow variations of what the structure of the ID is.     However I do not wish to block getting WHAT WE EASILY CAN AGREE ON, on resolving those questions (we can do so, but lets do it as a separate thread/PR).
I would like to believe the tagging idea expressed here is pretty uncontroversial, as it costs basically nothing (it is smaller than the original, and just as easy to parse)  and has obvious benefits (you can make additions to the ID structure WHILE ALLOWING existing parsers to parse what IS unchanged about version 2 (it also allow you to encode both the new and old formats SIMULTANEOUSLY, which is very useful turn the transition from Version 1 to version 2).    Finally it ALLOWS the POTENTIAL for MORE SYSTEMS to participate in INTEROPERABILITY.
I would like ask if there are still any vetos of this PR given the change I made?
Thus the

FYI each spec change/addition requires careful thought and review, and just
to let you know there are folks who haven't weighed in on anything at all,
yet. Basically, let's bear in mind that Nov 1 workshop was set aside for
folks who can't dedicate daily or weekly time to review PR comments. A
major change like this, however sensible, needs as much focus on support as
it does lack of vetos, IMHO.

I still think the parsing in the prefix model is more complex and requires more careful reading of the spec, and therefore more prone to misinterpretation and bugs than the current versioned model. I also acknowledge that it's a bit subjective and others will have differing (and still valid) opinions depending on language, experience, or preferences.
I see your point about better likelihood of supporting some parts of the tracing context even as more prefixes are added (e.g. a properly coded parser can always parse trace ID from the * prefix no matter what new prefixes are added down the road), but I disagree that the prefix model is the only way to accomplish that goal.
To my mind, the prefix model can be thought of as versioning each component of the tracing context separately with the ability to add more separately-versioned components as time goes on. i.e. * is the "version 00" for trace IDs, - is the "version 00" for span ID, . is the "version 00" for hierarchical/appended span ID, and ~ is the "version 00" for options.
Maybe the prefixes are the best way to do such a separately-versioned-components tracing context, maybe not. I think there are some significant benefits to versioning the components separately as you've described - especially for those components like trace ID that are largely compatible across tracing systems and won't need many versions (maybe just one), but I think it necessarily comes with some more complexity both conceptually and practically, and the tradeoff needs to be considered.
After switching my perspective on this PR to thinking about it as a way to separately version the components I think I'm more ok with it, but now new questions have popped up. On one hand, this means if I receive a tracing context with a . version of the "span ID component" and my tracing system only understands the - version, I can add a new - span ID to the context while leaving the . one unmodified. This could potentially improve interop. On the other hand if my tracing system understands both, and I receive both, what should I do about it? Adjust the one I prefer and leave the other unchanged? Adjust both since I understand both (extra code to maintain)? What about option-component-versions, or trace-id-component-versions? The interactions between all these prefixes/component-versions and what's recommended for one set of them vs. the other could get really lengthy and complex. This is partly what I meant by higher cognitive load and more possibility for bugs, different interpretations, and confusion.
This is all just my opinion as a relatively new tracing system maintainer. I don't feel like I have the breadth of experience in this space to hold a veto, and even if I had one I wouldn't necessarily use it here since I can see some benefits to both models.  But I believe I am part of the intended audience for this spec; I will be implementing it at some point and I don't think I'm the only one who will have these concerns.

I think at this point, prefix encoding or version make s the spec too complex for implementation. Version way is the easiest way, maybe not the best way.  We should focus on our first release, and give the implementation more time to support it.

Note in reply to @adriancole  @nicmunroe and @wu-sheng.
For those that think the prefix notion is too complex, please just take a look at it   Here is the resulting spec.
https://github.com/vancem/tracecontext-spec/blob/793c4012f0f78cf1c8371287d6a1db7d8e598b08/HTTP_HEADER_FORMAT.md
It is literally 2 pages in length, and the meat of it is only 1 page (the rest is examples and clarification).  It is just not that complex.   It pretty much just says that the ID has parts, and this is how you parse each of the parts.     Those parts are EXACTLY the three parts that existed in the original spec.    Even the actual characters that are being sent (see examples) have barely changed.
The goal of this PR is to try to make it possible to version INCREMENTALLY.  That is if you wanted to add something to the ID you could do so WITHOUT having to modify all the existing clients (with a classic version number, when existing code sees version 2, it knows NOTHING about how to parse it and has to give up (unnecessarily). That is at REAL problem).   By explicitly decoupling the PARSING of the parts of the ID from each other, it allows for more stuff to be added while the existing implementations can parse what they always have.
Just work through how you would ever move the version number from 1 to 2 using the old approach, and then ask yourself how you would do it if you can simply add the new information at the end of the ID with a new prefix FOR THAT PART (thus retaining compatibility with all existing parsers).  Is that not worth something?
Sure you might be able to achieve this versionability other ways, but this way is here, now, and is VERY SIMPLE.  It is hard to believe you can make something significantly simpler.   The risk is thus very low, and you CAN say that it is BETTER than the original 'version-number-for-the-whole-id' approach.  Given that is it not better to at least move on to that (and someone can submit a PR for something even better, if they can find it).
Sure when you add a new feature (e.g. support for hierarchical IDs), there are all sorts of questions that arise.  We are NOT trying to answer those as we are only worrying about the format that we need and the versioning scheme we need so we CAN MAKE THESE CHANGES LATER.
If we are to have any hope of a coherent standard, we need to find those things that we can agree upon.  One such principle is that we should not NEEDLESSLY limit flexibility (since we would like many existing systems to interoperate with this standard to the degree they can).   This format helps with that as well.
If we are to succeed in agreeing to a standard, we should all be flexible, and accept small things that really don't interfere with your important scenarios if they ARE important to other members of the standards group.   In that light, is there REALLY pushback to proposal?   Exactly what is the problem?

@vancem how is the versioning achieved in the prefix format, by using different prefix characters? If so, it means the protocol will keep increasing the number of reserved characters, which actually breaks backwards compatibility.

@vancem how is the versioning achieved in the prefix format, by using different prefix characters? If so, it means the protocol will keep increasing the number of reserved characters, which actually breaks backwards compatibility.

Yes, when you add new things you create new prefix characters (but you also put the new stuff at the end).   Yes, you 'use up' characters as you go, but that is really not a big problem.   The number of expected new version things is probably < 10.  We have a couple dozen punctuation characters that we can use.   Moreover, we can ALWAYS make some of the new prefixes take two characters instead of one giving us 100s of more prefixes.   We really can't run out.

which actually breaks backwards compatibility.

I don't understand this comment.  How do new prefix characters at the end break existing parsers?   When these parsers see the unexpected prefix, they give up (although we encourage them to save the whole thing so 'smart' back ends may be able to do something).   Still they have already parsed what they can, and will work as well as they did if the new information was not there.   That seems like the best you can do...
The main point is that this is STRICTLY better than the original (where a new version number again forces the parser to give up, but there, without parsing ANYTHING (and in particular the TraceID).

which actually breaks backwards compatibility.
I don't understand this comment.

e.g. if in the future versions you declare a certain character as special when previously it was allowed in the values.

e.g. if in the future versions you declare a certain character as special when previously it was allowed in the values.

Indeed, each prefix must define the mechanism by which it decides what it consumes in its parse.   Typically it is either a fixed number of characters or that that the character is outside a set (e.g. Hex digits).   It might define a special terminating character.    The current spec makes these choices in a sensible way leaving plenty of room for new prefixes.


polygon error

one error in polygon class, pls help...
line 431 in polygon.py file, error: "Undefined name ‘total’ (pyflakes E)"

@harrywang2018 That's a bug in the error message; should be fixed in the latest commit. Thanks!


Pip install fails with type error

Platform: CentOS Linux release 7.5.1804
Python:  Python 2.7.5
When installing with pip install xmlsec the setup.py fails. Below is the output:
`# pip install xmlsec
Collecting xmlsec
Using cached https://files.pythonhosted.org/packages/35/42/d7cd323c91d4706f3cc32ffe7d5f851ab8ef9898ccb350f6ba593dd8b89a/xmlsec-1.3.3.tar.gz
Requirement already satisfied: pkgconfig in /usr/lib/python2.7/site-packages (from xmlsec) (1.1.0)
Requirement already satisfied: lxml>=3.0 in /usr/lib64/python2.7/site-packages (from xmlsec) (4.2.4)
Installing collected packages: xmlsec
Running setup.py install for xmlsec ... error
Complete output from command /usr/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-install-_uAooh/xmlsec/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-Sj4ENM/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-_uAooh/xmlsec/setup.py", line 60, in
'Topic :: Text Processing :: Markup :: XML'
File "/usr/lib/python2.7/site-packages/setuptools/init.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib64/python2.7/distutils/command/install.py", line 563, in run
self.run_command('build')
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
self.run_command(cmd_name)
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-install-_uAooh/xmlsec/setup.py", line 11, in run
self.patch_options()
File "/tmp/pip-install-_uAooh/xmlsec/setup.py", line 16, in patch_options
ext.define_macros.extend(xmlsec_setupinfo.define_macros())
File "xmlsec_setupinfo.py", line 48, in define_macros
macros.extend(xmlsec_config()['define_macros'])
File "xmlsec_setupinfo.py", line 111, in xmlsec_config
__XMLSEC_CONFIG = load_xmlsec1_config()
File "xmlsec_setupinfo.py", line 178, in load_xmlsec1_config
macros[i] = ('XMLSEC_CRYPTO', '"{0}"'.format(v[1]))
TypeError: 'set' object does not support item assignment
----------------------------------------

Command "/usr/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-install-_uAooh/xmlsec/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-Sj4ENM/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-_uAooh/xmlsec/`
All dependancies are installed as per documentation. Please advise.

More info:
pkg-config --cflags xmlsec1
-DXMLSEC_CRYPTO="openssl" -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=FUNCTION -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_XKMS=1 -DXMLSEC_DL_LIBLTDL=1 -I/usr/include/xmlsec1 -I/usr/include/libxml2

Make new release 1.3.4 with fix

Thanks, however now I recieve the following error:
"Installing collected packages: xmlsec
Running setup.py install for xmlsec ... error
Complete output from command /usr/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-req-build-CEh_Gx/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-RHg8KR/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-req-build-CEh_Gx/setup.py", line 60, in
'Topic :: Text Processing :: Markup :: XML'
File "/usr/lib/python2.7/site-packages/setuptools/init.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib64/python2.7/distutils/command/install.py", line 563, in run
self.run_command('build')
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
self.run_command(cmd_name)
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-req-build-CEh_Gx/setup.py", line 11, in run
self.patch_options()
File "/tmp/pip-req-build-CEh_Gx/setup.py", line 17, in patch_options
ext.include_dirs.extend(xmlsec_setupinfo.include_dirs())
File "xmlsec_setupinfo.py", line 77, in include_dirs
dirs.extend(lxml.get_include())
AttributeError: 'set' object has no attribute 'extend'"

I did a small conversion at:
76     dirs = list(xmlsec_config()['include_dirs'])
77     dirs.extend(lxml.get_include())
It installs now, not sure if anything broke though

Cool, could you please create a PR ?

I have fixed the installation issue, but I cannot publish new release to pip since there is some privileges issues on travis. only  @mehcode   can resolve this.

Going to look at this in a few hours

This shouldn't be an issue anymore since the releases are rolling again.


Dataset modality missing and total storage space for memory

Thanks for your great work! Based on the dataset links you've shared, it appears that the RGB modality is missing for all four datasets. Could you kindly confirm if this is the case? Additionally, could you provide an estimate of the total storage space required to download all three modalities (RGB, optical flow and hand pose heatmap) for all four datasets?

We didn't upload the RGB frames to avoid the violence of dataset usage. (For instance, Ego4D requires access tokens for each individual).
However, RGB video can be downloaded from the original websites.
As for the amount of storage required, it is approximately ~2TB.

Thank you very much for your reply!


[Theme] Handle async errors

WHY are these changes introduced?
Related to #4576 and #4598
A proper request retry + token refresh is still needed but this PR tries to handle async errors more gracefully.

WHAT is this pull request doing?
Ensure we print async errors properly with a red banner, and avoid process exit in some situations:

On file delete / upload error in the remote theme, we show the error but the process continues. This is because the local development can continue working with local files in most cases, and is up to the developer to halt the process after seeing the error.
On polling errors, wait for 5 errors before exiting the process. This is because it might recover from one or two failed attempts since it still compares local checksums with latest.

For example, when you lose your internet connection while polling:

During the initial sync, if there are too many delete errors or a single upload error:

How to test your changes?
Create fake errors around the modified code or directly disconnect your internet + reconnect it within 15 seconds.
Post-release steps

Measuring impact
How do we know this change was effective? Please choose one:

[ ] n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
[ ] Existing analytics will cater for this addition
[ ] PR includes analytics changes to measure impact

Checklist

[ ] I've considered possible cross-platform impacts (Mac, Linux, Windows)
[ ] I've considered possible documentation changes

@app-inner-loop, Could you please take a look at this PR? It seems like the merge is blocked by that requirement.

@shopify/app-inner-loop, Could you please take a look at this PR? It seems like the merge is blocked by that requirement.


Add require_password_in_authentication_commands driver flag to allow skipping password check in authentication commands

Issue: #927

Please cherrypick it to (23.2), 24.1 and 24.2, if possible

There are multiple errors:
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:40: [E302] expected 2 blank lines, found 1
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:92: [F541] f-string is missing placeholders
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:94: [F541] f-string is missing placeholders
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:104: [E303] too many blank lines (3)
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:137: [F541] f-string is missing placeholders
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:139: [F541] f-string is missing placeholders

contrib/python/requests/py3/requests/models.py:434: in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
contrib/python/urllib3/py3/urllib3/util/url.py:397: in parse_url
    return six.raise_from(LocationParseError(source_url), None)
E   urllib3.exceptions.LocationParseError: Failed to parse: http://localhost:24268ytct-69db-360354809c55a39f22100f678f77b547

During handling of the above exception, another exception occurred:
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:73: in test_revoke_token
    self._check_allow(t)  # Activate cache.
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:30: in _check_allow
    rsp = self._make_request(path, token)
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:25: in _make_request
    return requests.get(
contrib/python/requests/py3/requests/api.py:73: in get
    return request("get", url, params=params, **kwargs)
contrib/python/requests/py3/requests/api.py:59: in request
    return session.request(method=method, url=url, **kwargs)
contrib/python/requests/py3/requests/sessions.py:575: in request
    prep = self.prepare_request(req)
contrib/python/requests/py3/requests/sessions.py:486: in prepare_request
    p.prepare(
contrib/python/requests/py3/requests/models.py:368: in prepare
    self.prepare_url(url, params)
contrib/python/requests/py3/requests/models.py:436: in prepare_url
    raise InvalidURL(*e.args)
E   requests.exceptions.InvalidURL: Failed to parse: http://localhost:24268ytct-69db-360354809c55a39f22100f678f77b547

yt/yt/tests/integration/proxies/test_cypress_token_auth.py:58: in test_simple
    assert get("//sys/cypress_tokens/@count") == 1
E   AssertionError: assert 4 == 1
E    +  where 4 = get('//sys/cypress_tokens/@count')

Fixed failing tests

Shipped!

Test test_cypress_token_auth.py::TestCypressTokenAuth test_simple (gritukan) fails:
yt/yt/tests/integration/proxies/test_cypress_token_auth.py:59: in test_simple
    assert get("//sys/cypress_tokens/@count") == 1
E   AssertionError: assert 3 == 1
E    +  where 3 = get('//sys/cypress_tokens/@count')

Strange. I don't get the same error locally
It is probably because tests in the same class reuse the same cluster state, so they should clean up after themselves
test_simple adds a token to user u and checks tokens count, expecting 1
test_authentication_commands adds 2 tokens and finishes.
And if test_authentication_commands was executed first, test_simple fails with 3 tokens instead of 1
I changed tests to use different users for their checks

Shipped!

Shipped!

Are tests ok now?


syntax error in antsRegistrationSyN.sh ????

I am registering a patient´s T1 to an MNI template via the procedure you suggested recently here:
https://github.com/stnava/ANTs/issues/251#issuecomment-159251798
The procedure seems to work great, however when I look into the nohup log I see the report pasted below.
Can this be safely ignored?

Can this be safely ignored?

I am using bash by calling explicitly:
nohup bash antsRegistrationSyN.sh -d 3 -f ../Data/sMRI/mni.nii  -m  ../Data/sMRI/${sub}/reg_t1_${sub}_a.nii -o test.nii

here is the code:
https://github.com/stnava/ANTs/blob/master/Scripts/antsRegistrationSyN.sh#L367
maybe you can figure this out for yourself.
brian
On Wed, Nov 25, 2015 at 10:54 AM, Metasoliton<EMAIL_ADDRESS>wrote:

I am using bash by calling explicitly:
nohup bash antsRegistrationSyN.sh -d 3 -f ../Data/sMRI/mni.nii -m
../Data/sMRI/${sub}/reg_t1_${sub}_a.nii -o test.nii
—
Reply to this email directly or view it on GitHub
https://github.com/stnava/ANTs/issues/256#issuecomment-159649932.

What about nohup sh antsRegistrationSyN.sh...?

$  sh antsRegistrationSyN.sh -d 3 -f ../Data/sMRI/mni.nii  -m  ../Data/sMRI/${sub}/reg_t1_${sub}_a.nii -o test
trap: SIGINT: bad trap
antsRegistrationSyN.sh: 8: antsRegistrationSyN.sh: function: not found

Error locating ANTS
It seems that the ANTSPATH environment variable is not set. Please add the ANTSPATH
variable. This can be achieved by editing the .bash_profile in the home directory.
NOTE:
I have pasted ANTSPATH=/usr/lib/ants
at the end of both
.profile   and .bashrc
and because that was not effective, I define again ANTSPATH=/usr/lib/ants at the top of antsRegistrationSyN.sh
(I am using Ubuntu 12.04.5 LTS Precise Pangolin)

@ stnava
I always try to figure things out myself before posting here and really appreciate your time and support.
On a sidenote, in https://github.com/stnava/ANTs/wiki/simple-case:-T1-to-MNI-mapping-for-both-structure-and-pre-registered-fMRI
the suggested commands only work properly if the structural and functional images have the same resolution, which is typically not the case, e.g. 1x1x1 and 3x3x3 in my case.
The ANTs documentation suggests that  the -s and -f arguments of antsRegistration  can be used to account for differences in resolution but when using antsRegistrationSyN -s and -f refer to different parameters!
Do I just have to add "-s 3x3x3" at the end of line 50 in antsRegistrationSyN.sh:
line 50: basename $0 -d ImageDimension -f FixedImage -m MovingImage -o OutputPrefix

if the fmri is indeed registered to the T1 ( in the ITK definition ), then
you should have to trouble using antsApplyTransforms.  that program does
not care about the input image resolution - only that it is in the same
space as the T1 as per the ITK / ANTs definition of this.   we have several
comments throughout that suggest mixing FSL and ITK / ANTs can lead to
unexpected issues (often caused by FSL's idiosyncratic use of NIFTI) that
we do not support.
brian
On Wed, Nov 25, 2015 at 11:31 AM, Metasoliton<EMAIL_ADDRESS>wrote:

@ stnava
I always try to figure things out myself before posting here and really
appreciate your time and support.
On a sidenote, in
https://github.com/stnava/ANTs/wiki/simple-case:-T1-to-MNI-mapping-for-both-structure-and-pre-registered-fMRI
the suggested commands only work properly if the structural and functional
images have the same resolution, which is typically not the case, e.g.
1x1x1 and 3x3x3 in my case.
The ANTs documentation suggests that the -s and -f arguments of
antsRegistration can be used to account for differences in resolution but
when using antsRegistrationSyN -s and -f refer to different parameters!
Do I just have to add "-s 3x3x3" at the end of line 50 in
antsRegistrationSyN.sh:
line 50: basename $0 -d ImageDimension -f FixedImage -m MovingImage -o
OutputPrefix
—
Reply to this email directly or view it on GitHub
https://github.com/stnava/ANTs/issues/256#issuecomment-159663715.

ooops - should read " should have no trouble using "
brian
On Wed, Nov 25, 2015 at 12:40 PM, brian avants<EMAIL_ADDRESS>wrote:

if the fmri is indeed registered to the T1 ( in the ITK definition ), then
you should have to trouble using antsApplyTransforms.  that program does
not care about the input image resolution - only that it is in the same
space as the T1 as per the ITK / ANTs definition of this.   we have several
comments throughout that suggest mixing FSL and ITK / ANTs can lead to
unexpected issues (often caused by FSL's idiosyncratic use of NIFTI) that
we do not support.
brian
On Wed, Nov 25, 2015 at 11:31 AM, Metasoliton<EMAIL_ADDRESS>wrote:

@ stnava
I always try to figure things out myself before posting here and really
appreciate your time and support.
On a sidenote, in
https://github.com/stnava/ANTs/wiki/simple-case:-T1-to-MNI-mapping-for-both-structure-and-pre-registered-fMRI
the suggested commands only work properly if the structural and
functional images have the same resolution, which is typically not the
case, e.g. 1x1x1 and 3x3x3 in my case.
The ANTs documentation suggests that the -s and -f arguments of
antsRegistration can be used to account for differences in resolution but
when using antsRegistrationSyN -s and -f refer to different parameters!
Do I just have to add "-s 3x3x3" at the end of line 50 in
antsRegistrationSyN.sh:
line 50: basename $0 -d ImageDimension -f FixedImage -m MovingImage -o
OutputPrefix
—
Reply to this email directly or view it on GitHub
https://github.com/stnava/ANTs/issues/256#issuecomment-159663715.

I actually used LIPSIA´s preregistration, which is also based on ITK.
Regardless, I would be happy to do the whole registration using ANTs, however, it seems that the moving image always attains the xyz dimensions of the reference image.
Isn´t there a parameter that prevents this or could you please modify https://github.com/stnava/ANTs/wiki/simple-case:-T1-to-MNI-mapping-for-both-structure-and-pre-registered-fMRI to include the pre-registration of T1 to fMRI?
I have looked through all the examples yet it seems that this was too simple a case to include!?

share some data. make it very clear.
brian
On Wed, Nov 25, 2015 at 12:49 PM, Metasoliton<EMAIL_ADDRESS>wrote:

I actually used LIPSIA´s preregistration, which is also based on ITK.
Regardless, I would be happy to do the whole registration using ANTs,
however, it seems that the moving image always attains the xyz dimensions
of the reference image.
Isn´t there a parameter that prevents this or could you please modify
https://github.com/stnava/ANTs/wiki/simple-case:-T1-to-MNI-mapping-for-both-structure-and-pre-registered-fMRI
to include the pre-registration of T1 to fMRI?
I have looked through all the examples yet it seems that this was too
simple a case to include!?
—
Reply to this email directly or view it on GitHub
https://github.com/stnava/ANTs/issues/256#issuecomment-159686095.

Dear Brian,
Thank you very much in advance for looking into this...
So, I am sharing the
T1 (T1.nii) and fMRI (fMRI.nii) from the most troublesome subject (even SyN failed to normalize the orbitofrontal cortex)
plus the MNI template I am using (mni.nii),
through this link:  http://we.tl/pdeLWR1uq0
The T1 and fMRI have been preregistered using the ITK-based functions of LIPSIA (https://www.cbs.mpg.de/institute/software/lipsia/index.html)
Issue A:
The T1 is in 1x1x1 and the fMRI is in 3x3x3 mm voxel resolution.
If you were to update the ANTs wiki page: https://github.com/stnava/ANTs/wiki/simple-case:-T1-to-MNI-mapping-for-both-structure-and-pre-registered-fMRI
to account for this common mismatch in voxel resolution
and/or
include the step that pre-registers the T1 to the fMRI data,
I (as well as many other novice ANTs users) would be very grateful.
(sidenote: I have noticed that you recommend normalizing the zmaps from the 1st level instead, yet this deviates somewhat from the most commonly accepted procedure. I would have trouble defending this approach - are there any papers comparing these two approaches?)
Issue B:
Does the procedure suggested on the wikipage produce a better result on your systems?
In mine it produces the attached SyN_T1.nii which is quite poor; look at the "shortage" of the orbitofrontal cortex specifically!
Are there any other parameters that can be further optimized in the implementation of the wikipage?
Issue C:
Even when antsRegistrationSyN.sh (as called in the wiki page above) works well, these errors appear in the report file:

Mapping parameters
ANTSPATH is /usr/lib/ants
Dimensionality:           3
Output name prefix:       g219_
Fixed images:             ../Data/sMRI/mni.nii
Moving images:            ../Data/sMRI/g219/reg_t1_g219_a.nii
Number of threads:        1
Spline distance:          26
Transform type:           s
CC radius:                4
Precision:                d
Use histogram matching    0
antsRegistrationSyN.sh: line 367: [1,: syntax error: operand expected (error token is "[1,")
antsRegistrationSyN.sh: line 367: 1,: syntax error: operand expected (error token is ",")
antsRegistrationSyN.sh: line 367: 1]: syntax error: invalid arithmetic operator (error token is "]")
antsRegistrationSyN.sh: line 367: [90,: syntax error: operand expected (error token is "[90,")
antsRegistrationSyN.sh: line 367: -92,: syntax error: operand expected (error token is ",")
antsRegistrationSyN.sh: line 367: 110]: syntax error: invalid arithmetic operator (error token is "]")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: [180,: syntax error: operand expected (error token is "[180,")
antsRegistrationSyN.sh: line 367: 218,: syntax error: operand expected (error token is ",")
antsRegistrationSyN.sh: line 367: 182]: syntax error: invalid arithmetic operator (error token is "]")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: {[90: syntax error: operand expected (error token is "{[90")
antsRegistrationSyN.sh: line 367: 110],: syntax error: invalid arithmetic operator (error token is "],")
antsRegistrationSyN.sh: line 367: [270: syntax error: operand expected (error token is "[270")
antsRegistrationSyN.sh: line 367: 292]}: syntax error: invalid arithmetic operator (error token is "]}")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: [1,: syntax error: operand expected (error token is "[1,")
antsRegistrationSyN.sh: line 367: 1,: syntax error: operand expected (error token is ",")
antsRegistrationSyN.sh: line 367: 1]: syntax error: invalid arithmetic operator (error token is "]")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: [0,: syntax error: operand expected (error token is "[0,")
antsRegistrationSyN.sh: line 367: 9999]: syntax error: invalid arithmetic operator (error token is "]")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: 2758.21: syntax error: invalid arithmetic operator (error token is ".21")
antsRegistrationSyN.sh: line 367: .: syntax error: operand expected (error token is ".")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: el->RAS: syntax error: operand expected (error token is ">RAS")
antsRegistrationSyN.sh: line 367: :: syntax error: operand expected (error token is ":")
antsRegistrationSyN.sh: line 367: Metadata:: syntax error in expression (error token is ":")
antsRegistrationSyN.sh: line 367: =: syntax error: operand expected (error token is "=")
antsRegistrationSyN.sh: line 367: =: syntax error: operand expected (error token is "=")
antsRegistration call:
/usr/lib/ants/antsRegistration --verbose 1 --dimensionality 3 --float 0 --output [g219_,g219_Warped.nii.gz] --interpolation Linear --winsorize-image-intensities [0.005,0.995] --use-histogram-matching 0 --initial-moving-transform [../Data/sMRI/mni.nii,../Data/sMRI/g219/reg_t1_g219_a.nii,1] --transform Rigid[0.1] --metric MI[../Data/sMRI/mni.nii,../Data/sMRI/g219/reg_t1_g219_a.nii,1,32,Regular,0.25] --convergence [1000x500x250x100,1e-6,10] --shrink-factors 12x8x4x2 --smoothing-sigmas 4x3x2x1vox --transform Affine[0.1] --metric MI[../Data/sMRI/mni.nii,../Data/sMRI/g219/reg_t1_g219_a.nii,1,32,Regular,0.25] --convergence [1000x500x250x100,1e-6,10] --shrink-factors 12x8x4x2 --smoothing-sigmas 4x3x2x1vox --transform SyN[0.1,3,0] --metric CC[../Data/sMRI/mni.nii,../Data/sMRI/g219/reg_t1_g219_a.nii,1,4] --convergence [100x100x70x50x20,1e-6,10] --shrink-factors 10x6x4x2x1 --smoothing-sigmas 5x3x2x1x0vox
=============================================================================
The composite transform is comprised of the following transforms (in order):

Center of mass alignment using fixed image: ../Data/sMRI/mni.nii and moving image: ../Data/sMRI/g219/reg_t1_g219_a.nii (type = TranslationTransform)
=============================================================================

(I am using Ubuntu 12.04)
All the best,
Stav

hi stav
thanks for sharing the data.  this makes it much easier to respond.
issue A:  the wiki example is correct.  you may be using some old code that
had a bug in the antsApplyTransforms -e 3  option.  try the source
version of ANTs.
sub-issue A:  mni is a highly distorted space and, if you do statistics in
that space, you are embedding substantial distortion and resampling-induced
issues into your study. however, if you do not care about individual level
fidelity, then you can ignore the subject space and just work in group
space.  personally, i like to see individual level activation in a
subject-specific template space that is distorted only by the
machine/session, not by additional transformations that make already
challenging data even more difficult to analyze and interpret.  but i am
interested in, also, having high fidelity maps between structure and
function whereas most studies ignore the structural contributions to their
activation/connection maps.
Issue B:  the wiki example also suggests:
" 2 stages of registration: T1 to customTemplate to MNI and then
concatenating these via antsApplyTransforms"
long ago, we recommended not trying to go directly to MNI and we still
maintain this position ... mapping whole head images direct to MNI is a
particularly unreliable strategy.  better to perform brain extraction.
see the ants cortical thickness paper for more motivation / explanation.

issue C: i cant help with your shell.


Integration test between broker and api server

[#133153945]

This is incomplete.
You have 2 tests that barely cover bind.
Why aren't there tests for provision and deprovision?
Why aren't there tests when the API server is down?
Use the ginkgomon helpers to Interrupt or Kill.
I don't see how this can ever run in parallel given one DB.


Multiple Instances

Is there anyway that we can run multiple instances of BackgroundCheck on the same page?
I tried assigning BackgroundCheck to different variables but to no avail.

If your elements are generated, or too abundant to manually write them down, you can use this script (it does what kennethcachia says, be it automatically):
var targets = [];
var images = [];
$('.element-with-bg-IMAGE-to-check-on_REPLACE-ME-with-yours').each(function (index) {
    // Collecting ID
    var image = '#' + $(this).attr('id');
    var target = image + ' .TARGET-with-backgroundcheck-style_REPLACE-ME-with-yours';
    // We need to check if there is bg image on target, otherwise JS will break.
    if ($(this).css('background-image') != 'none') {
        targets.push(target);
        images.push(image);
    }
});

// BackgroundCheck needs a comma separated string to work.
targets = targets.join(', ');
images = images.join(', ');

BackgroundCheck.init({
    targets: targets,
    images: images
});


Thoughts about ViT register and anchor token

Hello :)
Thank you for the brilliant work; it brings me a lot of thoughts!
Here, I have some questions:

In the final experiment regarding the ViT register, "replacing all register features at the output of every layer with their means" did not cause a performance drop. Does this imply that the role of registers is limited to providing fixed attention biases rather than collecting global image information? This seems to differ significantly from previous conclusions.
Recently, many attention-based observations have noted that certain tokens have exceptionally high attention values, such as "Label Words are Anchors." It was commonly believed that these tokens summarized information at various stages. However, from your article's perspective, it seems they might merely serve as biases without actual semantic information. One possible scenario is that both bias tokens and anchor tokens exist. I look forward to your thoughts on this.

I really appreciate your work and look forward to your response.

Thanks for your interest in our work.

The answer is yes. This experiment shows that registers are simply learned biases in ViTs, thus acting equivalently as the attention bias formulation proposed in our paper.

Briefly going through the paper, it seems that label words are different than the tokens of massive activations. In Figure 2 of the paper "Label Words are Anchors.", label words refer to the labels of in-context examplars. Also their paper uses the saliency score to show the importance of label words in self-attention, while our work simply check the basic attention probabilities. This could be the cause for difference.

Let me know if you have further questions.

Thank you very much for your prompt response. I have benefited greatly from it. Here are some additional thoughts on the ViT register that I would like to discuss with you:

Based on the supplementary experiment in Appendix D.2 of the ViT register paper, as shown in the below figure, the register used for classification accuracy is significantly higher than the patch token. This indicates that the register indeed contains a lot of global information.
According to the conclusions in the paper, even if the register does not contain any global information, it does not affect the classification performance of the [CLS] token. This may be because the [CLS] token itself contains sufficient global information.

From these two conclusions, we can infer that the register simultaneously possesses the functions of collecting global information and attention biases. However, the role of collecting global information seems to be unimportant (at least for classification tasks), whereas the role of attention biases is indispensable.
Looking forward to further discussing this with you.

This looks interesting. This is how i would interpret this result. My hypothesis is that the non-massive activations in register features may store global input information but they are not important to the computation of CLS token and patch token features. In other words, registers could store global information but what makes them important are the massive activations in them and the role of attention biases.


Better Timer and General Improvements

The timer now is a node from the start, instead of creating and deleting the node in code (which is inefficient)
also some minor adjustments here and there to the rest of the code following your coding style
and deleted some unreachable code

oof, conflict. I'll take care of that

Use godot3 branch

alright, I changed it

Thanks!


[Addition]: Different wallets, online on the same day, cannot witness other miners.

Hotspot b58 Addresses
11Kds72AmT3TWV7wLKWuwtGDHngqDfSm4zaSj9rnb3WbpRxpXWV
11SN9Eab5WDkLhq9aFYqwRPmrdxDrLwNbre8Nt4D85SM7WzgG1q
1126qpnaJAnLnm5731V49NpqZ67FXzPRB26tZq3KnMg2YXSPURH1
11SydSdtxu4TxfAytsBgSPxjKFsafTpQNvhywd1EaTVkb9VWUkz
1129iZTVjtrepw4QwysoPnrNaRjugovQAA46xNtBEX6WwwwXKZQ5
1123TrP7w7G1dycMzEZ2NyfghYDMm3BHmResE9Jax1hhsp17t7hS
11dDwSFHhN9ikvtvF9Pc9WgmDrf3ZKUccMxuSm8v3Axxk8mubqm
1122U5NRZioziTTyutrqzhGMsweJ2MtiWqQPKJU98svAELdu8jUN
Discord Handle
No response
Reason(s)
Similar IP address:   <IP_ADDRESS>-188
Similar list of witnesses
Online same day
High rewards
Different wallets, online on the same day, cannot witness other miners.

This cheating machine group has added a new mining machine:
114PLQWLMiWmXpG866aLSeXCxjnQuP7mAgpUvrzQ3cnz9UaZXZX

This cheating machine group has added a new mining machine:
1164r97VybFCizewBjYtT2kdF3JDctVpPtnAMXTh9UYVpREncGr


1320 dps enforcement - max speed check

Hi Tobit,
I am testing this SDK with a RMD-X6-S
This motor can in theory achive speeds of 6000RPM (36000dps)
I think it is not related but it comes with a newer version of the protocol V4.01
I am trying to achive high speeds with it.
Is there a reason for the speed check in the code
?

Hi Dani,
There is not, just adjust the check to a value that is reasonable for your actuator for now. I will have to rethink some of these limits and make them use values in the actuator constants. Just leave this issue open and I will close it in one of the following weeks.
I have also updated the code quite significantly over the last two weeks so you might want to pull the latest version as well...


RFE: “When to use this” in documentation

I am submitting a …

[ ] bug report
[x] feature request

Description
It would be nice to have a section of the main documentation that outlines the intended uses of this module versus ssl. mbedTLS is quite a bit slower than OpenSSL, and on any device that needs mbedTLS’s lightness it’s probably advantageous to use a compiled language rather than python. The notion of “alternative” TLS libraries intrigues me, but the performance hit makes it seem hard to justify.
The number of GitHub forks & stars here, though, suggests that someone finds it of use, so I’m curious what those use cases are. Having them in the docs would probably help potential users to know “sooner-rather-than-later” that python-mbedtls is right for them.
Thanks for reading!

Hi Felipe,
Thank you for your interest. This is a strange question I cannot answer better than you. I do not know why people use this library or star it. All I know about its users are the stars, forks, and issues you can find here as well. This is FOSS! 😊
I think some people just read the code in order to understand (upstream/C) mbedTLS. Then, some users need DTLS and a few other features this library proposes and that are not commonly available in alternative Python libraries, or at least they were not the last time I checked.
You are very welcome to write a blog post and publish benchmarks elsewhere. The same goes with patches that would increase the performance of this library.
Do you know why Python is such a popular programming language when C++ and other compiled languages are so much faster? 😉
Best,
Mathias

@Synss Python is, of course, popular for the same reason as node.js: it’s easier than C++ et al. Python ssl versus this library doesn’t seem to have the same distinction, AFAICT, though I would agree that OpenSSL’s enormous C API is more imposing than mbedTLS’s.
I’d love to make your library faster, but mbedTLS itself seems to be the bottleneck.
I’m sorry if this was an awkward question/RFE. There are other language bindings to mbedTLS around (e.g., Lua), so clearly a critical mass of folks out there does find this useful; I really am just wondering what those use cases are.
Anyhow, thanks.


Use https version of the Vimeo API endpoint

Looks like it mirrors the scheme used there now.

I see #45 merged, any feedback on this one?

Oh, shoot! Sorry that I missed this one in yesterday's release & thanks for the bump. I'll keep this browser tab open & try to get a new tiny release out with this pull in the next day or three.

Alrighty @jhass, the https Vimeo endpoint is now part of v0.8.14. Thanks for the contribution!!

Thanks!


Linter is something weird

I got some linter messages whenever i loaded Modules.
it is actually installed already but i dont know why.
example:
1:6   	Pyright (reportMissingImports)	error     	Import "selenium" could not be resolved
2:6   	Pyright (reportMissingImports)	error     	Import "selenium.webdriver.common.keys" could not be resolved
3:6   	Pyright (reportMissingImports)	error     	Import "selenium.webdriver.common.by" could not be resolved
4:6   	Pyright (reportMissingImports)	error     	Import "selenium.webdriver.support.ui" could not be resolved
5:6   	Pyright (reportMissingImports)	error     	Import "selenium.webdriver.support" could not be resolved
7:8   	Pyright (reportMissingImports)	error     	Import "pandas" could not be resolved
68:32  	Pyright (reportUnboundVariable)	error     	"titles" is possibly unbound
68:40  	Pyright (reportUnboundVariable)	error     	"prices" is possibly unbound

Are you using a virtualenv?

Are you using a virtualenv?

Yes i do.

If you're using a virtual env then you can create a .sublime-project and set up the VIRTUAL_ENV environment variable.
Click on Project > Save Project As... and save your project somewhere.
Then click on Project > Edit Project, and paste the following:
// my-python-project.sublime-project
{
  "folders": [
    {
      "path": "/path/to/my-python-project"
    }
  ],
  "settings": {
    // Project-specific overrides for LSP.
    "LSP": {
      // Project-specific overrides for the LSP-pyright language server.
      "LSP-pyright": {
        // When starting the pyright process, override these
        // environment variables.
        "env": {
          // Inform pyright that we're using a virtual environment.
          "VIRTUAL_ENV": "${folder}/venv"
        }
      }
    }
  }
}

Will it be good if the following as the default for ST 4 since it's a convention?
        "env": {
          // Inform pyright that we're using a virtual environment.
          "VIRTUAL_ENV": "${folder}/venv"
        }

If you're using a virtual env then you can create a .sublime-project and set up the VIRTUAL_ENV environment variable.
Click on Project > Save Project As... and save your project somewhere.
Then click on Project > Edit Project, and paste the following:
// my-python-project.sublime-project
{
  "folders": [
    {
      "path": "/path/to/my-python-project"
    }
  ],
  "settings": {
    // Project-specific overrides for LSP.
    "LSP": {
      // Project-specific overrides for the LSP-pyright language server.
      "LSP-pyright": {
        // When starting the pyright process, override these
        // environment variables.
        "env": {
          // Inform pyright that we're using a virtual environment.
          "VIRTUAL_ENV": "${folder}/venv"
        }
      }
    }
  }
}

Since you're using ST3, you might need to change "${folder}/venv" into an absolute path (without the template string), but unsure if that's really the case.

// my-python-project.sublime-project
{
"folders": [
{
"path": "/home/hun/바탕화면/PythonFiles"
}
],
"settings": {
// Project-specific overrides for LSP.
"LSP": {
// Project-specific overrides for the LSP-pyright language server.
"LSP-pyright": {
// When starting the pyright process, override these
// environment variables.
"env": {
// Inform pyright that we're using a virtual environment.
"VIRTUAL_ENV": "/home/hun/anaconda3/envs/fizzbuzz"
}
}
}
}
}
I'm not sure i revise it correctly. thank you for replaying!

If you're using a virtual env then you can create a .sublime-project and set up the VIRTUAL_ENV environment variable.
Click on Project > Save Project As... and save your project somewhere.
Then click on Project > Edit Project, and paste the following:
// my-python-project.sublime-project
{
  "folders": [
    {
      "path": "/path/to/my-python-project"
    }
  ],
  "settings": {
    // Project-specific overrides for LSP.
    "LSP": {
      // Project-specific overrides for the LSP-pyright language server.
      "LSP-pyright": {
        // When starting the pyright process, override these
        // environment variables.
        "env": {
          // Inform pyright that we're using a virtual environment.
          "VIRTUAL_ENV": "${folder}/venv"
        }
      }
    }
  }
}

Since you're using ST3, you might need to change "${folder}/venv" into an absolute path (without the template string), but unsure if that's really the case.

Thank you sir. Solved Everything, I edited LSP-Pyright's setting and your setting too. all problems are solved!

Hi!
I've setted up my virtualenv, as @rwols recommended and it works, BUT, with one thing it is not working... See print n the link below.
https://prnt.sc/KzYDPHVXiguh
The python line of code is  :
venue_list = Venue.objects.all() - It is django ORM
but it shows an error: Cannot access member "objects" for type "Type[Venue]"
I tried just to shut it up with:
"reportFunctionMemberAccess":"false", - in the pyright.config.json,
but it doesn't stop showing this error...

For those framework specific issues, please do search them in pyright's repository as you are unlikely the first person who encountered them.

Search django objects => https://github.com/microsoft/pyright/issues?q=django+objects
Found https://github.com/microsoft/pyright/issues/3542
pip install django-types
Done.


no docid if using set()

Looks like a bug with the set() method:
 let uid = admin.firestore().collection("users").add(userData);
console.log(uid.id); //prints id

 let uid = admin.firestore().collection("users").doc().set(userData, {merge: true});
console.log(uid.id); //prints undefined

set only returns a WriteResult, which returns the time of the write but not the DocumentReference. Since add creates a new DocumentReference (which is not known beforehand and thus deemed more important), it returns the DocumentReference instead of the WriteResult.


Can you please add node red to IBM BPM connector?

Hi,
Thanks for producing many excellent IBM nodes, yet I would definitely like to fire off business processes based on node red flows. Can you make a node red wrapper for the IBM BPM service?

please use the mailing list or slack channels for suggestions and discussions.
We don't know anything about the BPM service other than it's existence so will need to leave that to others who are closer to the topic than ourselves.
Does this help ? https://developer.ibm.com/bpm/videos/node-red-and-bpm/


How to use this with a image.network

I need the zoomable function for a image I pull from the network, any ideas how to do this?

Nevermind, I missed NetworkImage as a provider

How to use with CachedNetworkImage?


[Question] Cannot login to GitHub

I'm using github for testing, in the frame pane, i cannot signin to my GitHub. It responed the following message:
Cookies must be enabled to use GitHub.

Hmm, it doesn't look like I can do anything about this from my end without overriding @lishid's new header policies in Obsidian. Stay tuned though, we might be able to figure something out!

This issue will be fixed in the next preview version of Obsidian thanks to @lishid! Until then, you'll have to access GitHub manually, unfortunately.


[BUG] Movielens not working in python 3.7 due to Pandera library

Description

In python 3.7, movielens doesn't load due to a conflict with Pandera library:
from recommenders.datasets import movielens
/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/recommenders/datasets/movielens.py:[36](https://github.com/LinkedInLearning/recommendation-systems-a-practical-introduction-2703578/actions/runs/7327519315/job/19954500824?pr=24#step:7:37): in <module>
    import pandera as pa
/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/pandera/__init__.py:4: in <module>
    import pandera.backends
/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/pandera/backends/__init__.py:6: in <module>
    import pandera.backends.pandas
/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/pandera/backends/pandas/__init__.py:5: in <module>
    import pandera.typing
/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/pandera/typing/__init__.py:9: in <module>
    from pandera.typing import (
/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/pandera/typing/geopandas.py:5: in <module>
    from typing import (  # type: ignore[attr-defined]
E   ImportError: cannot import name 'get_args' from 'typing' (/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/typing.py)

In which platform does it happen?

How do we replicate the issue?

Expected behavior (i.e. solution)

Other Comments

@loomlike it seems we only use pandera when creating a fake movielens dataset, do you know why we are using that dependency and if we can drop it?

This also affects Python 3.8.

FYI @SimonYansenZhao @anargyri more problems with dependencies

This is strange because they say that pandera requires Python >=3.7

Also here

But maybe the typing dependency is the problem.

@anargyri the error @miguelgfierro is sharing is from Python 3.7.17 so technically it is > 3.7.
Yes, tracing the error goes back to the typing thing.
there is a stalk overflow post about two solutions to this problem. https://stackoverflow.com/questions/77247446/cannot-import-name-self-from-typing

Thanks, so pandera needs to handle this. We need to remove the dependency then.

This is the original commit that introduced pandera btw https://github.com/recommenders-team/recommenders/commit/fd33efe2d6fc1f736809665c8d5657476374de32

from recommenders.datasets import movielens

I could only replicate the issue in 3.7, not in 3.8. @john0isaac what machine did you use? mine is a Standard_E4ds_v4 (4 cores, 32 GB RAM, 150 GB disk). What did you do to get this error? I tried
conda create -n foo python=3.8
conda activate foo
pip install recommenders
python
Python 3.8.18 (default, Sep 11 2023, 13:40:15)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from recommenders.datasets import movielens
>>>

Thanks @john0isaac I don't have quota for this type of machine, I am afraid. I should make a commit for Python 3.8 too. What happens if you try with higher Python versions on that machine?

@anargyri It's serverless doesn't consume lots of money and enables the spark jobs don't worry about quota.
As for your question as this is a preconfigured compute there are only two options python 3.8 or python 3.10 and python 3.10 is not supported by the recommenders package so, you need to fix this in order for this environment to work on Azure ML.

I am fixing this in a new PR. Until we release the fix on PyPI, one way to bypass the issue is after you have done pip install recommenders you do a pip install pandera[strategies]>=0.6.5,<0.18
This should uninstall the latest version of panderas (which is causing the error) and install the previous one.

Thanks


Add danish translation of five

I don't see adds to the module. What you are trying to achieve is already possible by using   five.norwegian, which is you should be using anyway.
It's noble for you to contribute, but I'd rather not import a module that is bogged down with useless features.


Contextmenu in blockeditor doesn't disappear or stays in front after selecting an item in the main menu.

Describe the bug
If you select the contextmenu it will not disappear or go to the back if you for instance click on the menu item to start to build an apk.

Affects

[ ] Designer
[x] Blocks editor
[ ] Companion
[ ] Compiled apps
[ ] Buildserver
[ ] Other... (please describe)

Expected behavior
When selecting a item in the main menu the context menu should disappear or go to the back.

Steps to reproduce
Go to the blockeditor and right click on the white background. Click on the menu item to build an apk. The apk is being build but the dialogform is behind the context menu.

I believe this is the dreaded https://github.com/google/blockly/issues/2066 Both alshmiedt (https://github.com/google/blockly/pull/2359) and I (https://github.com/google/blockly/pull/2286) have tried to fix this before to no avail.
I'd like to take a look and see if there's a way to fix this for App Inventor specifically (and if it is indeed the issue I think it is). But my instinct is that this is probably unfixable with the current state of Blockly's event handling :/

I wonder if it is possible for us to install a mouse up event handler such that if the event doesn’t occur on an element with a parent of either the Blockly widget div or a Blockly SVG workspace root calls Blockly.hideChaff(). @ellelili2025 also worked on making sure we hide the Blockly chaff more frequently and may have insights to offer.
Sent from my iPhone

On Mar 5, 2020, at 16:32, Beka Westberg<EMAIL_ADDRESS>wrote:

I believe this is the dreaded google/blockly#2066 Both alshmiedt (google/blockly#2359) and I (google/blockly#2286) have tried to fix this before to no avail.
I'd like to take a look and see if there's a way to fix this for App Inventor specifically (and if it is indeed the issue I think it is). But my instinct is that this is probably unfixable with the current state of Blockly's event handling :/
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

@ewpatton @ellelili2025 Idea: What about adding a listener to the TopToolbar? or to the TopToolbar dropdowns? I think that should pretty much cover any time page navigation happens, or popups (like the apk building) are created. But I'm not familiar enough with App Inventor to be sure.
I couldn't figure out how to attach a listener to the TopToolbar, but I did get one working on the projects dropdown:


add time resolution to create_grouped_edge_dataframe

Currently, the time resolution 'year' is hardcoded. This could be passed as an argument to allow for month/day/hour?

Oh that could be. My internet connection was really poor (train) so I couldn’t see everything 🙈. Sorry in that case. On 23. Jan 2024, at 18:57, Timo Diepers @.***> wrote:
Is this a duplicate of #5 or am I missing anything?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>


Update scispacy version on streamlit demo

I am getting different results for the same input text when I use the streamlit demo vs. when I run the code locally. The text in question:
text = "The structural unit of the secretory Na+-K+-2Cl- cotransporter (NKCC1) is a homodimer."

NER results using "en_ner_jnlpba_md" on streamlit demo

Then, running things locally:
import spacy

model = "en_ner_jnlpba_md"
nlp = spacy.load(model)

doc = nlp("The structural unit of the secretory Na+-K+-2Cl- cotransporter (NKCC1) is a homodimer.")

for ent in doc.ents:
    print(f"{ent.text}\t{ent.label_}")

"""
The above prints:

NKCC1	DNA
homodimer	PROTEIN
"""

My pyproject.toml has the following dependencies:
scispacy = "^0.4.0"
en_ner_jnlpba_md = {url = "https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.4.0/en_ner_jnlpba_md-0.4.0.tar.gz"}

Any idea what might be causing this? I consider the streamlit demo response to be more correct, and am interesting in getting the same result locally!

Also, I am only showing one example here, but I found I could quickly come up with other examples where the streamlit demo specialized NER results were better (IMO) than the results I got locally. A second example is:
text = "Fourteen residues of the U1 snRNP-specific U1A protein are required for homodimerization, cooperative RNA binding, and inhibition of polyadenylation."

The streamlit seems to be running scispacy_lg version 0.2.4 and the one you can currently download is 0.4.0, that might be the issue (I ran into this a couple of times when dealing with incompatibilities with spacy 2.X):
"lang":"en"
"name":"core_sci_lg"
"version":"0.2.4"
"spacy_version":">=2.2.1"

You not referring to the "specialized NER model" here though, right?
I can see that the streamlit loads two spacy models, spacy_model (here) and ner (here). My question is about ner, which should be unaffected by spacy_model. You can see that the text is processed with ner independently of spacy_model here.

You're totally right. I couldn't find which version of en_ner_jnlpba_md they are using on streamlit demo, but given that en_core_sci_lg was older, it wouldn't surprise me if the en_ner_jnlpba_md was too.

@MichalMalyska Yeah you are totally right. When I load the 0.3.0 version of model and repo, I get results that closely match the streamlit demo for both of my examples. Weird, because at least for these examples, the output of the 0.4.0 version model/code is worse (IMO).

@danielkingai2 I guess the bigger underlying problem is why are the 0.4.0 models so much worse than the older versions.

I think this could be one reason:
https://github.com/explosion/spaCy/issues/8138

I did my best to match everything to the old versions, and our reported accuracy didn't drop much I don't think, but there are a bunch of hyperparams that we haven't really done any search over, just tried to use whatever spacy is using. If you wanted to play around with retraining with different hyperparameters or something, all the training scripts should be clear from project.yml

As an original author of explosion/spaCy#8138 (which has been closed), I still keep trying to figure out what has changed.
I have a case where the 'accuracy' in the downstream application has dropped over 20%, despite Spacy training validation scores dropping less than 5%. There is a clear, consistent case where for my triplet of entities such as:
JOHN BROWN and JANE BROWN as trustees of JOHN AND JANE FAMILY TRUST
the Spacy-2 correctly predicts all 3 entities above
whereas Spacy-3 only predicts the first one (JANE BROWN) in 200 out of 1000 test documents.
Honnibal suggested there was some change in 'dropping entities' that can not be predicted, and perhaps that change is doing more than envisioned. I am trying to see if I can reproduce the same behavior using other data sets.


Updated ion authorization url to the current url

I noticed that the authorization URL used in cesium-native is an old one that's doing a redirect last week. I checked with @mramato and confirmed that we should be using https://ion.cesium.com/oauth. This is just a PR to make this update.
P.S. I also added .idea to the .gitignore to prevent git from trying to scoop up the Clion prefs on my machine.

Thanks @weegeekps!


Certain enemies not spawning when customizing moon configs

Describe the bug
Changing spawn weights for enemies on moons who's default spawn chance is 0 doesn't spawn them and throws out an error when the game tries to spawn them in while still adding the enemy's power to the moons max power count. (e.g. trying to spawn coil-heads on Experimentation or the ghost girl on Offense when by default their spawn weight is 0.) This will eventually lead to the moon hitting it's spawn cap without anything actually spawning in.
To Reproduce
Steps to reproduce the behavior:

Go to Moons in the config that pops up before starting a game.
Click on Experimentation.
Scroll down to the Inside enemies list Section.
Check the box for Spring and change the value to 9999.
Start the game and land on Experimentation.
The game will try to spawn in coil-heads but nothing will spawn in and the indoor power will raise by 1.

Expected behavior
The enemy in question should spawn in a vent somewhere in the facility. If an enemy isn't allowed to be spawned in for whatever reason it would be preferable if it's settings were ignored instead of trying to spawn it and taking away both a spawn chance of another enemy and adding to the total power.
Additional context
This also applies to trying to give a spawn weight to the outside enemies that populate the inside list in the configs. While I don't expect things like giants and eyeless dogs to actually spawn inside if someone where to configure them to try and spawn they would run into the same problem.
Logfiles
Player.log
Add a list of all of your mods
BepInEx-BepInExPack-5.4.2100
Rune580-LethalCompany_InputUtils-0.6.1
PotatoePet-AdvancedCompany-1.0.143

I think this might also be a bug with setting up custom enemy spawns on the map Infernis.

Has been fixed in 146 or so. Try 147


Deferred commands delayed for one word?

First: I love your emulator. I spend all my 2 weeks of vacation playing with it.
Today I started to optimize my first program using the "minimum acces coding" method.
The time of execution went only from 24 seconds to 22 seconds.
I then checked with the JS debugger to see why.
With instructions like "L10 T11 N12 C0 S20 D28", "drum.waitFor(107) was called...
I think the call "this.drum.waitFor(1)" in case of deferred command in "processor.readCommand", line 2353, should be removed.
I tried to remove it, and the execution of my optimized program went to 16 seconds. The kind of improvement I was expecting.
The basic functions of the PPR seem to work fine.
Thanks again for your great work!

Thanks for submitting this issue and your kind words about the emulator. You raise a very important question.
That one word-time wait for deferred commands exists for a reason. A G-15 command goes through potentially four states:
RC -- Read Command (loads the command word into registers).
WTR -- Wait for Transfer (waits for the word designated by T to come around on the drum, only applies to DEFERRED commands).
TR -- Transfer (execute the command; takes one word time for DEFERRED commands and T-L-1 word times for IMMEDIATE commands).
WRC -- Wait to Read Command (waits for the word designated by N to come around on the drum).
And then the cycle repeats. For IMMEDIATE commands, WTR is skipped, and the processor goes directly from RC state to TR state at L+1. TR state then lasts up to but not including T.
The important thing to note about WTR is that the end of that state is detected during word time T-1 and always takes at least one word-time. If T=L+1, then T-1 is the command word time (L), but WTR doesn't start until L+1, which is too late. Therefore, WTR needs to wait for 107 word times until it can detect T-1 and allow TR to start at T.
The G-15 coding manuals (they're more like "coding cookbooks" than reference manuals) are not very explicit about DEFERRED commands requiring at least one word time between L and T. The best resource I've found for timing details like this is the Theory of Operation manual:
http://bitsavers.org/pdf/bendix/g-15/60121600_G15_Theory_Of_Operation_Nov64.pdf
The most relevant portions are the discussion on Transfer Timing (p.28) and the discussions on Relative Timing Numbers, RC-WTR-TR-WRC Control, and Deferred Commands (p.44-47). Note especially the sentence that starts at the bottom of p.45:

"This accounts for the programming restriction that any DEFERRED command will involve at least one word time in WTR, hence if T=L+1, 108 word times will be spent in the WTR state."

The way that memory addressing and command timing works with the L, T, and N numbers and the hidden COMMAND register is very clever, but also difficult to understand. It makes my head spin.
The solution to your timing problem is really simple, though. Just make your command IMMEDIATE rather than DEFERRED and change T to L+2. That will guarantee that the TR state will start on L+1 and last for one word time.
I'm going to close this issue, but please feel free to continue commenting against it, especially if you have further questions.

OK, but it's strange that in the 3 programming books there are a lot of "Ln Tn+1 Nn+2", especially when they are teaching "minimum access coding".
Executed on your emulator, this is the worst case: an instruction that should take 1 word-time takes 109 word-times.
Even in their test routines there are a lot of these cases (pages 208 - 240 in the book "theory of operation".

Yes, but are you sure those commands are actually coded as DEFERRED? The coding examples and the coding sheets for the BXTST diagnostic tests are usually written in "PPR notation."
PPR assumes that, unless you override it with a "w" or "u" prefix, any command with D=31 should be assembled as immediate and all others should be assembled as deferred, except if T=L+1, in which case it will be assembled as immediate and T will be increased by one. See the Coding Manual for the Bendix G-15, under the heading "A Note Concerning the Program Preparation Routine" (p.22, PDF p.24) at http://bitsavers.org/pdf/bendix/g-15/G15_CodingManual.pdf.
I wrote a very basic disassembler to decode paper tape images and help me to undestand the BXTST diagnostics when I was debugging the emulator. That script is available in the repo at https://github.com/pkimpel/retro-g15/blob/master/software/tools/Paper-Tape-Disassembler.html.
I modified that this morning to test your assertion about the number of "Ln Tn+1 Nn+2" instances in those programs. I added three columns to the output, WTR, WRC, and TT, for the wait-for-transfer, wait-to-read-command, and total instructions times for the commands. If the command has T=L+1, the TT column entry has a "+" suffix.
I only found three cases where commands having T=L+1 were deferred. These were all in block #7 of the tape at locations 41, 67, and 70 (see PDF pages 221 and 223 in the Theory of Operation manual). They all involve one or two word registers, so the max additional rotational delay for them would only be one word time.
I did this change quickly, so the timing reported may not be correct in all cases (especially when shorter lines are involved). I've attached a unified diff of the changes and a copy of the output if you are interested in looking at it.
BXTST-Disassembly-20240108.zip

PPR assumes that, unless you override it with a "w" or "u" prefix, any command with D=31 should be assembled as immediate and all others should be assembled as deferred, except if T=L+1, in which case it will be assembled as immediate and T will be increased by one.

"T will be increased by one" Now that makes sense! And I observed it a few times when I forgot the "w", without understanding.
I was fighting with PPR to force deferred command all that time.
Sorry to have waste your time with this.

It wasn't a waste of my time. I am always happy to help someone with one of my emulators, and as usual, I learned something when researching this issue, and I'm pleased that we got your mystery cleared up.


JSON files not generted

Steps to reproduce

Install Linked Data vocabularies
Install Linked Data helper
Download LCSH -> https://id.loc.gov/download/ -> latest file "LC Subject Headings (LCSH) NEW Pilot (MADS/RDF and SKOS/RDF)" in .ndjson
Unzip
Set the path in Linked Data helper to the unzipped folder-> C:\Users\User\Downloads\LCSH
Leave "Folder for JSON files" empty
Start conversion
Message displays that conversion started

Expected Results
Files to be generated or maybe a message if/when it finished processing
Actual results
Nothing happens. Files not generated and no other notification. Tried with different file location of the unzipped LCSH. Maybe I'm doing something wrong?

Set the path in Linked Data helper to the unzipped folder-> C:\Users\User\Downloads\LCSH

The path needs to be set to the file, not the folder.

The path needs to be set to the file, not the folder.

Thanks! That fixed it.
It did create a folder tilted "linked-data-vocabularies" under my attachments folder. I also got the confirmation in Obsidian that the json files have been created. Unfortunately when I open the new folder its empty..
Tried to extract with different folder locations and same result. The "conversion" took about 30 seconds, and I did see obsidian take up some memory during that time, so it looks like it kinda did something but not quite there

Did you open it in Obsidian or with your OS file explorer? By default, these files will not show up in Obsidian's file explorer because they are JSON files, unless you activate show all files.

It was with my OS (Win10), I have hidden files enabled and the folder is empty. I tested on two machine, a laptop and desktop, both on windows. When running the plugin, pointed to the new folder it says "JSON could not be read"

Are you using the latest plugin version and seeing any errors in the console when you start the parsing?

And what is your Obsidian installer version? Please make sure it is up-to-date.

So I did some more testing and it somehow worked out finally.
My installer is 0.14.5 and the plugin is up to date. I deleted the folder and ran the conversion again which created the empty folder (no errors in the console). Then with the folder open in my explorer I ran the conversion once more and voila, the 3 files got populated in the folder...weird.
Anyways its working now so I will be playing with this. Thanks for the help!


Add success/failure messages after inserting a question

I think redirection to the display question page is enough.