Roadmap

Here I'd like present my suggestions for what I believe could be done to improve quality of JSMN. Everyone should feel free to comment on this and give own suggestions. Also, I will need opinion from @zserge on this.

Split strict and non-strict parsing code
Idea behind it is that this would make it easier to maintain both versions without having to overcomplicate the code and to eliminate risk of breaking strict version when trying to improve non-strict one

Make strict mode the default
Considering that JSMN is mostly aiming at parsing JSON strings and not it's abbreviations the library should do it by default.

Enable parent links by default
Parent links feature enables a great speed boost at a quite low memory cost. I believe a lot of people aren't using this library on really memory constrained platforms. But if there's a need for it the user should just disable parent links. It could be probably nice to rename the define to something like JSMN_LOW_MEMORY.

Strict mode should follow RFC-8259
There are a few standards with slight differences for JSON. I believe it should be explicitly said that we would follow RFC-8259 as THE standard for our JSON parsing.

Maybe create a package for Arduino
Considering the fact JSMN aim at platforms that are similar to what Arduino offers I believe it would be nice to either offer this library as is to Arduino community or separately write a wrapper to make access to this library more C++ like.

Bit flags for token types #108
Small change that would make it easier to write parsing code in certain places.

Utilities

Token array traversing
Using JSMN in a safe manner can be difficult. Because of that I think it would be nice to write an optional header file with utilities to easily and safely traverse JSMN's output. These should work at least for the strict mode.

Values decoding
At this point JSMN made the decision of not parsing the values. It could be left this way but then I believe it would be great to optionally offer utilities that could take for example a string from a token and decode it into UTF-8 or a widechar string.

I've been using JSMN in a somewhat memory constrained application. The 16-bit micro has 64kB internal flash, but due to other required libraries, I didn't have a lot of room to spare. So keeping a low-memory footprint in mind is important to me. JSMN_LOW_MEMORY sounds great.

Utilities

Values decoding
...

Some platforms I've used don't have the non-standard itoa() functions. So when I'm generating JSON replies, I've had to create a local_itoa() sort of function, optimized to only support bases 2 - 16. This is more of a values encoding idea, but maybe something to discuss?

I also use this parser and faced a troubles while dealing with invalid jsons. So, first of all I'd like you to thank for such nicely done piece of free software. I really appreciated of it.
Ok, since you're planning to make significant changes to this project, I'd like to post here some thoughts about this:
At this point JSMN made the decision of not parsing the values
For my humble opinion you should at least check if this is even printable character in case of regular strings. If in double quotes pair suddenly happens binary garbage - that should not be parsed anyway, welp.
From other point of view, this project is not JSON validator, but JSON parser. Maybe it's better to throw away any validation since current validation is not even close to what should be and focus on performance and low memory usage. Then, recommend users to use separate modules for JSON validation, like this: http://www.json.org/JSON_checker/JSON_checker.c

Well, for the JSON to be parsable it has to be valid so as long as the parser is never wrong then if we are able to parse the thing then we are basically validating it. There's no way out of it.
In JSMN values aren't really fully parsed and so they are not validated.
How I think it could work is that the utilities for parsing JSON values as seen by JSMN will have to adhere to RFC-8259 anyway and so they will be able to tell you whether there is a problem with value itself. For example if token is of type string but it contains for example a byte of value 0x05, then the value parser should return an error as this is invalid according to the standard.

Also I'm feeling the a lot of pressure here as it seems like the original creator of this project either abandoned it or is too busy/tired to work on it and beside him I am the only person that can do anything with this repository. I have ambition of keeping JSMN as great as it is when it comes to parsing speed and memory usage but make it better at actually parsing JSON. Right now in my opinion there are quite a bit parsing bugs in JSMN which mostly come from the fact of how it's built.
Right now I have an idea of how JSMN could be slightly modified in a sense of idea of how the parsing happens but it would mean somewhat of a large code changes. And so I thought that why not as well just think of what else could be changed and just make JSMN the best thing out there that's easy to use.

In JSMN values aren't really fully parsed and so they are not validated.

Currently, if \n happens in the middle of " " pair, JSMN still will parse it. BTW it could be fixed, so I'd like to ask: are you accepting PR's or you're already started developing an update for this project so there is not reason to even open PR?

Also I'm feeling the a lot of pressure here

Dunno. It's free software, people are free to modify it for their own needs.
For my humble opinion, from human philosophy PoV: If nothing was given then nothing should be expected. This thought partially described here: https://github.com/do-not-theme/do-not-theme.github.io/issues/3#issuecomment-496958706

I have ambition of keeping JSMN as great as it is when it comes to parsing speed and memory usage but make it better at actually parsing JSON.`

Hmmm... It reminds me an old joke about having a fast, reliable, strong, and cheap vehicle at same time. I have no idea if you can keep the same performance with all required validation.

Haven't really done any work myself yet. So far it's all planning and getting the courage to do any changes to this repository. After all I'm not @zserge and this is basically official repository of JSMN. I was given responsibility I haven't exactly asked for :p
Feel free to create PR. Although I think the way library works should be discussed to make it as close to perfect without weird trade-offs. But you are right, there are limits to what can be achieved here.

@pt300 I really like the suggested list of changes.
Strict mode by default, compatibility with RFC8259, Arduino package and bitflags - all sounds good.
But I want to share my vision of the 'non-strict mode'. Basically, it's not a feature of JSMN, but rather a side-effect of "what would happen if we #undef some of the parser code". This eliminates certain sanity checks, and parses invalid but nicely looking JSON-ish text, with unquoted strings and the absence of the root object. This has never been a priority for JSMN (in my mind), and I don't think it deserves a separate implementation. If, by undef-ing some code form the RFC-compatible strict implementation we can achieve some syntax sugar for JSON - fine, if not - I'd rather forget about non-strict mode at all.

Additionally, I would rather invest into advanced documented supersets of JSON, like JSON5, rather than in the undocumented, barely known non-strict mode.

What about the ability of estimation of token array length? Considering it works only for specific cases I think it's not a really good feature to keep around, especially that it seems to add more complexity to the code than it's worth.

In my case I need to open a JSON object from an embedded file system based storage that may potentially be too large (up to ~64kb) to fit into the available RAM (~4 - 8kB), pull certain configuration values from it, then I'm done. I'm not particularly concerned about speed. Allocating RAM space for the token array is doable.
I currently have JSMN implemented and working well, and have been getting by though minimising the size of the JSON string (remove whitespace, ensure that only required JSON elements are present, etc) and maximising the heap space available to fit larger objects; however we're starting to feel the pinch as other areas of the code grow and start to demand more RAM.
I have been toying with the idea of abstracting the access to the underlying original JSON string so that it can still be backed by a raw array of bytes as it currently is, or alternatively a paged based solution that will only require a portion of the JSON string to be in RAM at once.
I'm unsure if this is even possible with the structure of JSMN, and even if it is, it is probably too much of a change to be considered based the 'light weight and fast' ethos (which I very much appreciate).
If I do get around to implementing it, I will certainly make the changes available.

Make strict mode the default / RFC-8259

Non-strict mode is important to the extent that JSON implementations differ in the wild (or, at least, to the extent that the "few standards" differ.)

a paged based solution that will only require a portion of the JSON string to be in RAM at once... if this is even possible with the structure of JSMN

JSMN is simple enough to already be "complete" --- so you should consider forking and modifying as you need, manually merging bugfixes if/when bugs are ever discovered.

Something I'd like to see is much clearer documentation of the functions, expectations in error cases, etc. Currently this requires code inspection & experimentation.

Thank you for your work on jsmn.  On my private copy I've added some functionality for easier traversal, in particular all tokens have a "first_child" and a "next_sibling".  I added a very simple function to populate these values after parsing (speed is not important to me).  And then I can iterate over children of a node by:
for (int child = tokbuf[node].first_child; child != -1; child = tokbuf[child].next_sibling) {
   ...
}

I am not asking anything from the jsmn project but if you would like to see the code I would be happy to share (not that it's difficult).

Strict mode should follow RFC-8259
There are a few standards with slight differences for JSON. I believe it should be explicitly said that we would follow RFC-8259 as THE standard for our JSON parsing.

We needed this for a project I'm on. I went ahead and implemented it. See PR #194. The downside is there are some sacrifices made to non-strict mode (see the PR for details on that).

But I want to share my vision of the 'non-strict mode'. Basically, it's not a feature of JSMN, but rather a side-effect of "what would happen if we #undef some of the parser code". This eliminates certain sanity checks, and parses invalid but nicely looking JSON-ish text, with unquoted strings and the absence of the root object. This has never been a priority for JSMN (in my mind), and I don't think it deserves a separate implementation. If, by undef-ing some code form the RFC-compatible strict implementation we can achieve some syntax sugar for JSON - fine, if not - I'd rather forget about non-strict mode at all.

Additionally, I would rather invest into advanced documented supersets of JSON, like JSON5, rather than in the undocumented, barely known non-strict mode.

@zserge This seems reasonable to me. (And is why I did not worry about sacrificing some non-strict functionality for RFC 8259 compliance in the PR above.)

The rest of the proposed changes sound very nice. If we do add extra utilities, it would be nice to see them in either a separate header or able to be enabled/disabled with a #define.
I'd also like to propose a few additional improvements:

Single-object mode
Currently, the parser consumes the entire input string. It parses multiple objects if necessary, returning JSMN_ERROR_PART if the last object is incomplete, even if there are complete objects before it. Sometimes, it would be useful to parse one object at a time. For example, if parsing a stream of JSON objects, there would be no need for the input buffer to end on object boundaries, as long as at least one complete object is present.
A new macro, JSMN_SINGLE, could be created that would cause jsmn to parse one object at a time and ignore the rest of the input.

More granularity for non-strict macros
Some applications may have use for certain non-strict features but won't want all of them. For example, someone might want non-strict primitive validation for performance reasons (especially if PR #194 is accepted), but not non-string keys.
Separate macros could enable the various non-strict features, and they would all be enabled by default unless JSMN_STRICT is defined.

Improved docs
The README and jsmn website are vague about a lot of the details. For example, the following are not mentioned at all:

size for an object refers to key/value pairs, not the total number of tokens
Object keys have size 1 and object values have the key as the parent, not the object
The end field in a token is the index of the first character after the token, not the last character in the token
If there is more than one object in the input, they all will be parsed.

These all make sense, but aren't documented. Some well-explained examples would be useful too, especially for traversing arrays and objects.

I plan to work on these in the near future and will create additional pull requests as they are ready, in case there is interest.

I would have preferred single-parsing to be the default as well, but I am hesitant to change this behavior since it would break backward compatibility.

My thought is that since we are changing the default functionality from non-strict to strict that we were breaking backwards compatibility already. If we are trying to follow RFC 8259 as strictly as possible (pun slightly intended), I think we should go all out and follow it as closely as we can. If the JSMN_NON_STRICT flag enables the permissive functionality that people are currently use to, this would just be one of the functions that they lose by default. JSMN_MULTIPLE_JSON would be one of the modular macros that enables functionality beyond the default.
As for test_issue27, I could argue that single primitives weren't intended for strict parsing either.
int test_simple_json(void) {
  check(parse("\"Hello world!\"", 1, 1, JSMN_STRING, "Hello world!", 0));
  check(parse("42", 1, 1, JSMN_PRIMITIVE, "42"));
  check(parse("true", 1, 1, JSMN_PRIMITIVE, "true"));
  return 0;
}

Both the "42" and the "true" return JSMN_ERROR_PART in strict mode. I think that the ability to parse multiple json strings was an unintended "feature" that ended up become the default mode.
For JSMN_NON_STRICT vs JSMN_PERMISSIVE, both work equally well, I agree. Personal preference that really isn't set it stone.

First I propose changing size to children. ...

I agree that this behavior is a good idea (see discussion under "Arrays and objects as keys" on PR #194), my point was just that it should be documented. Changing the name feels like a step too far though--see notes above on backward compatibility.

You're probably right on this one. The other option would be to start with something like the following to have the possibility to phase it out.
typedef struct jsmntok {
  jsmntype_t type;
  int start;
  int end;
  union {
    int size __attribute__((deprecated("use children instead")));
    int children;
  }
#ifdef JSMN_PARENT_LINKS
  int parent;
#endif
} jsmntok_t;

For non-strict mode: 1) We should define non-strict mode precisely, i.e. describe each deviation from RFC 8259. We can't hope to preserve compatibility when we don't really know what is "compatible." 2) Breaking backward compatibility to get there is okay. 3) Once we define the exact behavior, those features should be protected from future backward incompatible changes.
This would mean that non-strict deviations should be few, limited in scope, and be useful, since they would become fully-supported features. (After all, if users of the library can't rely on a feature to continue working in future versions, it loses most of its utility. Why even bother?)

Absolutely. Let us get into specifics over in issue #154.

I would have preferred single-parsing to be the default as well, but I am hesitant to change this behavior since it would break backward compatibility.

My thought is that since we are changing the default functionality from non-strict to strict that we were breaking backwards compatibility already.

That is a fair point.

If we are trying to follow RFC 8259 as strictly as possible (pun slightly intended), I think we should go all out and follow it as closely as we can. If the JSMN_NON_STRICT flag enables the permissive functionality that people are currently use to, this would just be one of the functions that they lose by default. JSMN_MULTIPLE_JSON would be one of the modular macros that enables functionality beyond the default.
As for test_issue27, I could argue that single primitives weren't intended for strict parsing either.
int test_simple_json(void) {
  check(parse("\"Hello world!\"", 1, 1, JSMN_STRING, "Hello world!", 0));
  check(parse("42", 1, 1, JSMN_PRIMITIVE, "42"));
  check(parse("true", 1, 1, JSMN_PRIMITIVE, "true"));
  return 0;
}

Both the "42" and the "true" return JSMN_ERROR_PART in strict mode. I think that the ability to parse multiple json strings was an unintended "feature" that ended up become the default mode.

Okay, I think there are two issues here: 1) Whether multiple-parsing violates RFC 8259 and 2) if it should be the default in jsmn.
For (1): RFC 8259 ultimately just specifies a grammar. Something like {"a": 1, "b": 2} [1, 2, 3] does not follow the grammar, thus it is not a JSON object. But jsmn doesn't parse it as a JSON object. It parses it as two JSON objects, each of which conform to the RFC. While that is perhaps surprising (It's certainly not documented, and I was surprised to learn it behaved this way), it does not violate RFC 8259. The RFC doesn't say JSON objects must always be in a string by themselves, or JSON objects must not touch, or a list of JSON objects must look like <X>. This behavior should be documented, but there is no conflict with RFC 8259.
For (2): My original argument was that, as a behavior RFC 8259 is neutral on, we should preserve it in the interest of backward compatibility. As you point out, backward compatibility is right out the window when we make strict mode the default. As long as we are changing the defaults for some configuration macros, there's little reason not to make single-parsing the default if that is more useful. So, that becomes the question: is single-parsing a more useful default? I would prefer to have wider input than just @themobiusproject and me on that, but I haven't seen much other activity in a while, so that might be wishful thinking.
In fact, I see three options for this:

Multiple parsing. This is the current behavior. If multiple objects are present in the input string, all will be parsed. If the last object is incomplete, JSMN_ERROR_PART is returned for the whole thing.

Exact single parsing. The input string must contain precisely one RFC 8259 JSON object. If there is anything more present, the result is JSMN_ERROR_INVAL. If I understand, this is the default you are advocating for.

Liberal single parsing. The parser consumes input, returning successfully once it consumes a complete JSON object. Everything left in the string is ignored. If we are changing the default, this is the default I would advocate for: it allows for parsing either single objects or a stream of objects. If you check if there is any unconsumed input after the parse, you can ensure the input was precisely one object. Otherwise, you can get stream semantics: if jsmn_parse() returns successful, simply call it again to get the next object. (This is very close to the way I implemented single parsing in PR #196, although as writing this, I realize I should make a few tweaks to make it the most flexible.)

I am biased toward (3) being the default because the flexibility seems very useful (in fact, for my use case, I have a stream of JSON-formatted messages that need to be processed), but I can see the argument for (2): it's perhaps the least "surprising" way for jsmn to behave.
Also, while one of them has to be the default, I think it is reasonable to have all three be selectable options.

First I propose changing size to children. ...

I agree that this behavior is a good idea (see discussion under "Arrays and objects as keys" on PR #194), my point was just that it should be documented. Changing the name feels like a step too far though--see notes above on backward compatibility.

You're probably right on this one. The other option would be to start with something like the following to have the possibility to phase it out.
typedef struct jsmntok {
  jsmntype_t type;
  int start;
  int end;
  union {
    int size __attribute__((deprecated("use children instead")));
    int children;
  }
#ifdef JSMN_PARENT_LINKS
  int parent;
#endif
} jsmntok_t;

Unfortunately, anonymous unions aren't allowed under C89, or even C99 :( That said, I still don't think it's a major issue as long as size is well-documented.

Token array traversing

This was the second reason behind my original fork; I needed a faster way to get from one sibling to the next. This, combined with parent, makes going to the next and previous sibling very efficient.

Adding a field in jsmntok_t seems reasonable, only extra comment I have is this should probably be one of the exclusions for JSMN_LOW_MEMORY.

I want to add a little more follow up on this: Parent links nearly universally improve parsing speed, so if memory constraints aren't a concern, you basically always want them. Sibling links aren't as universal of a benefit: they don't improve parsing at all and only benefit applications with the right use case (needing to traverse arrays or objects but not needing to visit every token within).
I bring this up because sibling links seem to be a common request, but I think it's partly due to perceived difficulty of implementing array/object traversal. Without arrays/objects as keys and with the current semantics for size, it's actually fairly straightforward:
/* Assume token is a pointer to the current jsmntok_t */
size_t remaining;
for (remaining = 1; remaining > 0; remaining--) {
    remaining += token->size;
    token++;
}
/* token now points to the next sibling */

Of course, the performance benefits are not to be ignored, and I suspect the majority of applications will benefit from them, so sibling links should likely be the default. I just wanted to mention that there are non-memory-constrained applications that would have reason to turn off sibling links.

In fact, I see three options for this:

Multiple parsing. This is the current behavior. If multiple objects are present in the input string, all will be parsed. If the last object is incomplete, JSMN_ERROR_PART is returned for the whole thing.

I think this point is what I disliked about multiple parsing by default; if a later json object fails then everything fails.

Exact single parsing. The input string must contain precisely one RFC 8259 JSON object. If there is anything more present, the result is JSMN_ERROR_INVAL. If I understand, this is the default you are advocating for.

Advocating for? Pretty close to that. If given the choice between 1 and 2, I would choose 2.

Liberal single parsing. The parser consumes input, returning successfully once it consumes a complete JSON object. Everything left in the string is ignored. If we are changing the default, this is the default I would advocate for: it allows for parsing either single objects or a stream of objects. If you check if there is any unconsumed input after the parse, you can ensure the input was precisely one object. Otherwise, you can get stream semantics: if jsmn_parse() returns successful, simply call it again to get the next object. (This is very close to the way I implemented single parsing in PR #196, although as writing this, I realize I should make a few tweaks to make it the most flexible.)

But then you add option 3...
Figuring out if there is more to the buffer that was passed in is simple (comparing the jsmn_parse len argument against tokens[0].end). I think my only aversion to this is the idea that in a stream two primitives next to each other require whitespace between each other where nothing else has that requirement. This isn't a coding issue, just a personal ocd type issue. That personal issue notwithstanding, option 3 would certainly be my preference over options 1 and 2.

Unfortunately, anonymous unions aren't allowed under C89, or even C99 :( That said, I still don't think it's a major issue as long as size is well-documented.

I recognize that anonymous unions are an extension of compilers rather than part of a standard, but I had to throw in one more option before allowing children to pass into the nether.

/* Assume token is a pointer to the current jsmntok_t */
size_t remaining;
for (remaining = 1; remaining > 0; remaining--) {
    remaining += token->size;
    token++;
}
/* token now points to the next sibling */

My goodness, that is much more elegant solution than I came up with. (Heading back to my code to make some changes.)
I would still want to include a next_sibling option (whether default or not though I would likely recommend it), but beyond that I think we should, in a separate optional file, include functions to help use the tokens that jsmn_parse provides. The examples that are currently given are lacking in that department.
To this point, I have an explode function, which I mentioned earlier, that visually helps people understand the data in the tokens and how to utilize them. I wrote it for a coworker to show how to use the tokens. A short example of the output of this (which uses the supplied library.json) is as follows:
    Token |      Type |    Start |      End |   Length | Children |   Parent |  Sibling |     |
----------+-----------+----------+----------+----------+----------+----------+----------+-----+-
        0 |    Object |        0 |      367 |      367 |        8 |       -1 |       -1 | VAL |  {
        1 |    String |        5 |        9 |        4 |        1 |        0 |        3 | KEY |    "name" :
        2 |    String |       13 |       17 |        4 |        0 |        1 |       -1 | VAL |      "jsmn",
        3 |    String |       23 |       31 |        8 |        1 |        0 |        5 | KEY |    "keywords" :
        4 |    String |       35 |       39 |        4 |        0 |        3 |       -1 | VAL |      "json",
        5 |    String |       45 |       56 |       11 |        1 |        0 |        7 | KEY |    "description" :
        6 |    String |       60 |      171 |      111 |        0 |        5 |       -1 | VAL |      "Minimalistic JSON parser/tokenizer in C. It can be easily integrated into resource-limited or embedded projects",
        7 |    String |      177 |      187 |       10 |        1 |        0 |       13 | KEY |    "repository" :
        8 |    Object |      192 |      264 |       72 |        2 |        7 |       -1 | VAL |      {
        9 |    String |      199 |      203 |        4 |        1 |        8 |       11 | KEY |        "type" :
       10 |    String |      207 |      210 |        3 |        0 |        9 |       -1 | VAL |          "git",
       11 |    String |      218 |      221 |        3 |        1 |        8 |       -1 | KEY |        "url" :
       12 |    String |      225 |      259 |       34 |        0 |       11 |       -1 | VAL |          "https://github.com/zserge/jsmn.git"
          |           |          |          |          |          |          |          |     |      },
       13 |    String |      269 |      279 |       10 |        1 |        0 |       15 | KEY |    "frameworks" :
       14 |    String |      283 |      284 |        1 |        0 |       13 |       -1 | VAL |      "*",
       15 |    String |      290 |      299 |        9 |        1 |        0 |       17 | KEY |    "platforms" :
       16 |    String |      303 |      304 |        1 |        0 |       15 |       -1 | VAL |      "*",
       17 |    String |      310 |      318 |        8 |        1 |        0 |       20 | KEY |    "examples" :
       18 |     Array |      321 |      344 |       23 |        1 |       17 |       -1 | VAL |      [
       19 |    String |      328 |      339 |       11 |        0 |       18 |       -1 | VAL |        "example/*.c"
          |           |          |          |          |          |          |          |     |      ],
       20 |    String |      349 |      356 |        7 |        1 |        0 |       -1 | KEY |    "exclude" :
       21 |    String |      360 |      364 |        4 |        0 |       20 |       -1 | VAL |      "test"
          |           |          |          |          |          |          |          |     |  }

This would be part of the better documentation that is sorely needed.

Multiple parsing. This is the current behavior. If multiple objects are present in the input string, all will be parsed. If the last object is incomplete, JSMN_ERROR_PART is returned for the whole thing.

I think this point is what I disliked about multiple parsing by default; if a later json object fails then everything fails.

I agree, I found this to be difficult to work with, even if there are multiple objects to be parsed.

Liberal single parsing. The parser consumes input, returning successfully once it consumes a complete JSON object. Everything left in the string is ignored. If we are changing the default, this is the default I would advocate for: it allows for parsing either single objects or a stream of objects. If you check if there is any unconsumed input after the parse, you can ensure the input was precisely one object. Otherwise, you can get stream semantics: if jsmn_parse() returns successful, simply call it again to get the next object. (This is very close to the way I implemented single parsing in PR #196, although as writing this, I realize I should make a few tweaks to make it the most flexible.)

But then you add option 3...
Figuring out if there is more to the buffer that was passed in is simple (comparing the jsmn_parse len argument against tokens[0].end).

It's almost that simple...if there is trailing whitespace, this won't work. I was thinking about this while I wrote that last night and I thought of two ways to implement it:

The parser consumes all trailing whitespace. The check then becomes parser->pos == len ||  js[parser->pos] == '\0'. But to prevent applications from having to go into the jsmn_parser struct, it would be better to have a function for this check, maybe call it jsmn_eof() or jsmn_end().

The parser ignores trailing whitespace, but the jsmn_eof() function would consume it before doing the check from (1). I like this way because I think it will lead toward a cleaner implementation. With no need to consume trailing whitespace, jsmn_parse() can return as soon as it closes an array or object with no parent or parses a string or primitive with no parent.

I think my only aversion to this is the idea that in a stream two primitives next to each other require whitespace between each other where nothing else has that requirement. This isn't a coding issue, just a personal ocd type issue. That personal issue notwithstanding, option 3 would certainly be my preference over options 1 and 2.

It's worth mentioning, this isn't the only sticking point for unenclosed primitives. They are also troublesome for incremental parsing. Suppose we have the following two JSON entities:
"abcdefgh"
1.23456789

If they happen to get split when an application is doing incremental parsing:
"abcde
1.2345

the first results in JSMN_ERROR_PART, but the second returns successfully with a truncated primitive. Not ideal, but I don't think there's a great way around the issue. The best I can come up with is simply making sure that the next call to `jsmn_parse() recognizes that it's a split primitive and fixes the bad token. (That being said, I think this is not likely to be a problem in practice. While primitives can be unlimited length, they typically are fairly short, so they should generally fit into any input buffer entirely. But this should be documented.)

/* Assume token is a pointer to the current jsmntok_t */
size_t remaining;
for (remaining = 1; remaining > 0; remaining--) {
    remaining += token->size;
    token++;
}
/* token now points to the next sibling */

My goodness, that is much more elegant solution than I came up with. (Heading back to my code to make some changes.)
I would still want to include a next_sibling option (whether default or not though I would likely recommend it),

I think making it the default is reasonable. I think a lot of applications will benefit from the performance boost. And as straightforward as the above is, it's still not as easy as token = token->next_sibling.
Also worth mentioning: traversing backward is not so easy. It's not as common a need, but should there also be token->prev_sibling?

but beyond that I think we should, in a separate optional file, include functions to help use the tokens that jsmn_parse provides. The examples that are currently given are lacking in that department.

I agree. (This is also one of the things originally proposed at the top of this thread, of course.) That snippet above is straightforward, but it's not immediately obvious (though it might be more obvious with better documentation for size).
I'd like to see the following in a jsmn_utils.c/.h:

Next sibling
Key lookup in object
De-escape string
Determine if primitive token is int, double, true, false, or null
Primitive token to int (What about other sizes of int and unsigned?)
Primitive token to double

To this point, I have an explode function, which I mentioned earlier, that visually helps people understand the data in the tokens and how to utilize them. I wrote it for a coworker to show how to use the tokens. A short example of the output of this (which uses the supplied library.json) is as follows:
    Token |      Type |    Start |      End |   Length | Children |   Parent |  Sibling |     |
----------+-----------+----------+----------+----------+----------+----------+----------+-----+-
        0 |    Object |        0 |      367 |      367 |        8 |       -1 |       -1 | VAL |  {
        1 |    String |        5 |        9 |        4 |        1 |        0 |        3 | KEY |    "name" :
        2 |    String |       13 |       17 |        4 |        0 |        1 |       -1 | VAL |      "jsmn",
        3 |    String |       23 |       31 |        8 |        1 |        0 |        5 | KEY |    "keywords" :
        4 |    String |       35 |       39 |        4 |        0 |        3 |       -1 | VAL |      "json",
        5 |    String |       45 |       56 |       11 |        1 |        0 |        7 | KEY |    "description" :
        6 |    String |       60 |      171 |      111 |        0 |        5 |       -1 | VAL |      "Minimalistic JSON parser/tokenizer in C. It can be easily integrated into resource-limited or embedded projects",
        7 |    String |      177 |      187 |       10 |        1 |        0 |       13 | KEY |    "repository" :
        8 |    Object |      192 |      264 |       72 |        2 |        7 |       -1 | VAL |      {
        9 |    String |      199 |      203 |        4 |        1 |        8 |       11 | KEY |        "type" :
       10 |    String |      207 |      210 |        3 |        0 |        9 |       -1 | VAL |          "git",
       11 |    String |      218 |      221 |        3 |        1 |        8 |       -1 | KEY |        "url" :
       12 |    String |      225 |      259 |       34 |        0 |       11 |       -1 | VAL |          "https://github.com/zserge/jsmn.git"
          |           |          |          |          |          |          |          |     |      },
       13 |    String |      269 |      279 |       10 |        1 |        0 |       15 | KEY |    "frameworks" :
       14 |    String |      283 |      284 |        1 |        0 |       13 |       -1 | VAL |      "*",
       15 |    String |      290 |      299 |        9 |        1 |        0 |       17 | KEY |    "platforms" :
       16 |    String |      303 |      304 |        1 |        0 |       15 |       -1 | VAL |      "*",
       17 |    String |      310 |      318 |        8 |        1 |        0 |       20 | KEY |    "examples" :
       18 |     Array |      321 |      344 |       23 |        1 |       17 |       -1 | VAL |      [
       19 |    String |      328 |      339 |       11 |        0 |       18 |       -1 | VAL |        "example/*.c"
          |           |          |          |          |          |          |          |     |      ],
       20 |    String |      349 |      356 |        7 |        1 |        0 |       -1 | KEY |    "exclude" :
       21 |    String |      360 |      364 |        4 |        0 |       20 |       -1 | VAL |      "test"
          |           |          |          |          |          |          |          |     |  }

This would be part of the better documentation that is sorely needed.

That is incredibly useful. That should absolutely be part of the documentation.

It's almost that simple...if there is trailing whitespace, this won't work. I was thinking about this while I wrote that last night and I thought of two ways to implement it:

Leading whitespace shouldn't be an issue either. I was planning on implementing it so when count > 0 and it reached the root level again jsmn_parse returns. If there is whitespace afterwards then it just becomes leading whitespace for the next object which passes.

Also worth mentioning: traversing backward is not so easy. It's not as common a need, but should there also be token->prev_sibling?

For the previous sibling I go to the toksuper's first child and follow token->next_sibling until the next sibling is the token I started at. This is essentially what I do when I am figuring out next sibling anyways.

I'd like to see the following in a jsmn_utils.c/.h:

Next sibling
Key lookup in object
De-escape string
Determine if primitive token is int, double, true, false, or null
Primitive token to int (What about other sizes of int and unsigned?)
Primitive token to double

I have a good key lookup function that I would be happy to share and compare with what you have. I am getting utf8 implemented to also go from the escaped \u format to utf{8,16,32} and back again.
My parser flags tokens as JSMN_PRI_LITERAL for true, false, and null, JSMN_PRI_MINUS if it is a negative number, JSMN_PRI_DECIMAL if it has a decimal, and JSMN_PRI_EXPONENT if it is an exponent. This is beyond its default JSMN_PRIMITIVE flag. This is also why I love jsmntype_t being bit flags.

That is incredibly useful. That should absolutely be part of the documentation.

I'm glad you like it. It has also really helped me debug jsmn as I have been modifying it.

It's almost that simple...if there is trailing whitespace, this won't work. I was thinking about this while I wrote that last night and I thought of two ways to implement it:

Leading whitespace shouldn't be an issue either. I was planning on implementing it so when count > 0 and it reached the root level again jsmn_parse returns. If there is whitespace afterwards then it just becomes leading whitespace for the next object which passes.

Thinking about it more I see your point; if you reach the end of valid json object but there is only whitespace left in the buffer, then you could misinform an application saying that there is more data to be parsed.

Ack on some single object mode!  We're abusing it currently by setting toks[0].type = JSMN_UNDEFINED;  then checking toks[0].type != JSMN_UNDEFINED && toks[0].end != -1) if we get JSMN_ERROR_PART.
This "works" at cost of wasted cycles if there is more than one object, since we discard the buffer covered by the first token and try again (this is a JSON command stream).


Relay Groups

Better to implement here https://github.com/BlowaterNostr/blowater/issues/30

Did some experiment and it didn't feel correct to implement relay groups at this layer.


Hide group selection when needed

This pull request adds the ->groupSelectionVisible method to exclude the Group Selection from the table output. This makes sense especially when only one group or ->defaultGroup() is used.
Example:
use Filament\Tables\Table;

public function table(Table $table): Table
{
    return $table
	->defaultGroup('status');
        ->groupSelectionVisible(false);
}

Screenshot:

Shouldn't this be done automatically instead of adding a new API?

I don't think so. Even if there's just one group, someone might want to offer the user the option to choose between ascending and descending order. Don't you think?

Yeah right, forgot about the sort direction.

Thanks, Dan!!


adding known deployments page

Adds a page where we can include known deployments that we hear of

Opened #540 for licensing topic. I think the text from that should be in our READ/deployment guide so merging this now.


Update/change macro name

This PR aims at renaming the interface macro to the interface_entry_point macro

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: CyberHoward:x: KayanskiYou have signed the CLA already but the status is still pending? Let us recheck it.


remove compiled files for distribution

https://developer.github.com/v3/repos/releases/

ref commits: a0dcb7c79749fea520ea46e1a6736b657ecabe83, 08777a780461880b87909625d4c646288f221860, 93754e371d302a76fbd7b3ae2672b9d8e8dc62ed

related documents:  GitHub Releases Uploading


Enabling the Packer vars to be set

Not able to customize packer build
There is an issue which stops customisation of packer variables via options in Makefile.local.
If you run virtualbox/osx1010-desktop with a simple Makefile.local  Something like:
    CM:=puppet

Packer will not receive these arguments and will continue to use those defined inside the target .json file.
What have I done?
I spotted that the PACKER_VARS variable could never be set unless CM_VERSION was defined.
This pull request adjusts to logic of the ifdef CM_VERSION statement so that PACKER_VARS is always set

This fixes github #8 which I just filed/opened. I almost duplicated the PR before finding this one. Can you please merge this?

Thanks.

No, thank you! Running packer by hand would be tedious. I appreciate boxcutter-osx and your help.


import {Foo as Bar} from 'baz'; comes out wrong

TypeScript:
import {View as ViewAnnotation} from 'angular2/src/core/annotations_impl/view';

Dart:
import "package:angular2/src/core/annotations_impl/view.dart" show View ViewAnnotation ;

Which causes a parse error.

+1


reset permissions

This thing already exists, but I want you to change Expressions from remove all to reset, for example
change it from remove all (your permissions) from permissions of event-member in event-channel
to reset (your permissions) from permissions of event-member in event-channel
and add the Expressions in https://disky.me/docs/docs/expressions/?h=permission
and add new thing doing remove member or role from permission list in channel
like clear member/role from permissions of event-channel

The reset mode does not exist in Skript, as the expression is directly used by Skript's changer. It would require creating a whole new change effect with just one new pattern.

The reset mode does not exist in Skript, as the expression is directly used by Skript's changer. It would require creating a whole new change effect with just one new pattern.

Sorry a little confused here, RESET does indeed exist
https://github.com/SkriptLang/Skript/blob/b0052d25d6314c0cf7279d089aa4201bad23f713/src/main/java/ch/njol/skript/classes/Changer.java#L39-L41


Plotting pruned networks

Hi Marcus,
Thank you very much for developing a package to visualise NNs.
I noticed it does not allow to visualise pruned networks however.
Recently I approached the developer of RSNNS to extend the capacity of the package to introduce pruning capacity - a very useful feature indeed which is now available as standard in RSNNS.
Would it be possible please to look into extending the functionality of NeuralNetTools to allow plotting pruned networks as well?
Thank you very much in advance.
Regards,
Max

Max, try the v1.3.3 release.  I tested plotnet with a simple pruned network and it worked as expected.  The other functions should work though I'm getting screwy results with lekprofile.
-Marcus


Correct ordering of transform coefficients

Components with a xyScale and yxScale are currently broken. Noto Sans Balinese before this patch:

After:

Comment explains why.

Tests are failing but how much do you trust your tests? ;-)

The eternal fight between xy vs yx in 2D graphics...
Can you please update the tests as well?
this reminds me of a discussion we had when drafting the COLR spec:
https://github.com/googlefonts/colr-gradients-spec/pull/85

After re-reading the old long thread at https://github.com/googlefonts/colr-gradients-spec/pull/85 and I think the flip you're doing here is not at the right place, it might be that it's the ufo2fontir that is setting the kurbo Transform incorrectly.
The labels xy and yx or b or c are always confusing which is why one needs to look at the meaning of those values, i.e. interpreting them as x or y components of the î or ĵ basis vectors, and also take into account whether the values of the matrix are stored in column-first or row-first order.
Please let me double check

@simoncozens can you please help me to reproduce the issue you've shown in the screenshots above?
You compiled https://github.com/notofonts/balinese using fontc? Which glyph/character is that?
I think the test results that you have modified to make the test pass following your change, were actually correct. And in fact if I build the same test font resources/testdata/glyphs2/Component.glyphs using fontmake 3.8.1 I get exactly the same glyf table.
If I inspect the composite glyphs that contain some 2x2, like 'comma' or 'non_uniform_transform' (encoded as digit 0), they render as expected.

You compiled https://github.com/notofonts/balinese using fontc? Which glyph/character is that?

I did fontmake -o ufo -g sources/NotoSansBalinese.glyphs; fontc master_ufo/NotoSansBalinese-Bold.ufo; shape build/font.ttf -u 25cc.

After applying your patch the "non_uniform_scale" and "comma" glyphs look like this when dumped with ttx and diff'ed. fontc is using your PR branch, fontmake is v3.8.1:
$ diff -u build/default/fontc.glyf.ttx build/default/fontmake.glyf.ttx | colordiff | less
--- build/default/fontc.glyf.ttx        2024-02-14 13:09:27
+++ build/default/fontmake.glyf.ttx     2024-02-14 13:09:27
@@ -19,12 +19,12 @@
       <instructions/>
     </TTGlyph>

-    <TTGlyph name="comma" xMin="241" yMin="-481" xMax="508" yMax="-253">
-      <component glyphName="period" x="-233" y="-129" scalex="1.6655" scale01="-1.1611" scale10="1.1611" scaley="1.6655" flags="0x4"/>
+    <TTGlyph name="comma" xMin="67" yMin="245" xMax="334" yMax="473">
+      <component glyphName="period" x="-233" y="-129" scalex="1.6655" scale01="1.1611" scale10="-1.1611" scaley="1.6655" flags="0x4"/>
     </TTGlyph>

-    <TTGlyph name="non_uniform_scale" xMin="8" yMin="-481" xMax="143" yMax="-253">
-      <component glyphName="period" x="-233" y="-129" scalex="0.8452" scale01="-1.1611" scale10="0.58923" scaley="1.6655" flags="0x4"/>
+    <TTGlyph name="non_uniform_scale" xMin="-138" yMin="102" xMax="26" yMax="259">
+      <component glyphName="period" x="-233" y="-129" scalex="0.8452" scale01="0.58923" scale10="-1.1611" scaley="1.6655" flags="0x4"/>
     </TTGlyph>

     <TTGlyph name="period" xMin="250" yMin="50" xMax="375" yMax="100">
\ No newline at end of file

Basicaly you swapped "scale01" and "scale10" component attributes. But the glyphs look wrong with this patch.

ah so you compiled from a UFO, not directly from the NotoSansBalinese.glyphs source. Ok so I need to look somewhere else for the cuplrit. But it definitely this is not the right fix, because both kurbo::Affine and the fontations Transform structs use a column-first order for representing the matrix values

Ah, OK, so it's probably ufo2fontir

can you please try with:
diff --git a/ufo2fontir/src/toir.rs b/ufo2fontir/src/toir.rs
index 566b67ff..5fdfdef2 100644
--- a/ufo2fontir/src/toir.rs
+++ b/ufo2fontir/src/toir.rs
@@ -60,8 +60,8 @@ fn to_ir_component(component: &norad::Component) -> ir::Component {
         base: component.base.as_str().into(),
         transform: Affine::new([
             component.transform.x_scale,
-            component.transform.yx_scale,
             component.transform.xy_scale,
+            component.transform.yx_scale,
             component.transform.y_scale,
             component.transform.x_offset,
             component.transform.y_offset,

I'm off to lunch and can take a closer look later

Yes, that works on my fonts.

Ahaha I still wish transforms more directly spoke to basis vectors.
Good catch. Sounds like a hole in our tests, quite likely by me :)
Please add a new test to prove it was broken (e.g. test fails on the current code) and is now fixed.

Sounds like a hole in our tests, quite likely by me :)

After #716 and #719, we're now in a position where we can use diffenator2 to compare fontc/fontmake outputs of UFO builds, which will make isolated regression testing a lot easier. I'm hoping to be able to oxidise some of the Noto production build (just the instance statics), using this to find differences and failures, and hopefully iterate fontc closer to correctness.

Please add a new test to prove it was broken

Yep, will do.

After https://github.com/googlefonts/fontc/pull/716 and https://github.com/googlefonts/fontc/pull/719, we're now in a position where we can use diffenator2 to compare fontc/fontmake outputs of UFO builds, which will make isolated regression testing a lot easier. I'm hoping to be able to oxidise some of the Noto production build (just the instance statics), using this to find differences and failures, and hopefully iterate fontc closer to correctness.

Great news!

So all the confusion was due to norad's AffineTransform using the same terminology as fonttools' Transform where it refers to the "y component of the î basis vector" as xy and to the "x component of the ĵ basis vector" as yx.
https://github.com/linebender/norad/blob/e58e32009786e767066b9af2cba17bcff23c9ae3/src/glyph/mod.rs#L443-L456
On the other hand, read-fonts' 2x2 Transform (also used by write-fonts) names the same values with the opposite labels: yx is "y component of the î basis vector" and xy is "x component of the ĵ basis vector". This follows a naming convention also used in FreeType and Cairo.
Note however that the values of the matrix are all listed in column-first order, the same that kurbo Affine uses, or that the OpenType glyf Component's "scale01" and "scale10" values are (the first index is the column, the second is the row).
So it's not like we are swapping places, we're simply calling the same things differently.

It would help enormously if documentation shows the matrix itself using the local terms. (Until today I thought glyf’s scale10 and scale01 were the other way around!)
For example, in glyf:
\begin{bmatrix}
    xscale & scale10 \\
    scale01 & yscale
\end{bmatrix}

Thus for a 90º CCW rotation…
\begin{bmatrix}
    0 & -1 \\
    1 & 0
\end{bmatrix}

… we have:
xscale = 0
scale01 = 1
scale10 = -1
yscale = 0


Notebook Insert Link paths have a / when editing

Create text cell in Notebook
Insert a link to a local file
Select the link and click the Insert Link button to edit it

The path is displayed with a prefixed / on windows
/C:/Users/chgagnon/AppData/Local/Programs/Azure%20Data%20Studio%20-%20RC1/resources/app/out/vs/code/electron-browser/workbench/Desktop/SQLQuery_1.sql

I had a look at this. While it's harmless for now, I think it will cause confusion.
I noticed that with the "/" it seems to still be working OK after clicking Insert or Cancel.
Also when adding the link again (without the "/") works too.
Can we please get this fix soon please?


x/mobile: bind command fails for iOS with Go 1.15

$ go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/user/Library/Caches/go-build"
GOENV="/Users/user/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/path/to/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/path/to/go"
GOPRIVATE=""
GOROOT="/usr/local/Cellar/go/1.15/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/path/to/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/05/01234/T/go-build111673491=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
gomobile bind -v -ldflags '-s -w' -target=ios -o=MyLib.framework github.com/user/mylib-go/...

What did you expect to see?
Normal binding output.
What did you see instead?
gomobile: go [-e -json -compiled=true -test=false -export=false -deps=false -find=true -tags=ios -- github.com/user/mylib-go/...]: exit status 2: cmd/go: unsupported GOOS/GOARCH pair darwin/arm

Is this related to https://golang.org/doc/go1.15#darwin?

@hajimehoshi @dmitshur just curious how gomobile detects the system is darwin/arm when the system says otherwise:
$ go env GOOS GOARCH
darwin
amd64

I'm temporarily downgrading to<EMAIL_ADDRESS>

Is this issue different from #39575, which was fixed?
What version of gomobile are you using?

No longer a problem, my gomobile was outdated


Add Hungarian translation

I've added a Hungarian translation, please review and merge if it is acceptable.

Thank you @nXu :+


[Add] Haraka SMTP server

Issue description
https://github.com/haraka/Haraka

@MrSorcus Is this still relevant?

@MrSorcus Is this still relevant?

Hi. I don't know. I switched to Postfix.

@MrSorcus Please close this PR.

@MrSorcus Please close this PR.

This an issue, not a PR. And what reason for closing?

Yes, i meant to say issue. Are you still motivated to switch from Postfix to Haraka? If so, you can help by providing derivations for Haraka itself, its plugins, a NixOS module etc.
If you are not interested in Haraka anymore, the issue seems no longer relevant and in case someone else is seeking the inclusion in NixOS he/she may open a new issue.


⚠ ZV800.com has degraded performance

In e763e6e, ZV800.com (https://www.zv800.com//) experienced degraded performance:

HTTP code: 200
Response time: 4317 ms

Resolved: ZV800.com performance has improved in c8035ea after 14 minutes.


Pinging notifies all subscribers one at a time before returning

When switching to async/await I seem to have bottlenecked myself by notifying each subscriber one at a time before the ping responds. One test took almost 9 minutes to complete. Need to make sure this works in parallel and perhaps working out of a queue.

Fixed in 001f39d2b291e343ddf3304debeae788e80f8af7


[fix] once errorBuilder is called Image widget stops loading images

Description
Image widget does not show a valid image after loading a bad image. The issue is that the exception thrown by the previous loading is not being reset (set to null) when a new image is loaded successfully.
Related Issues
Fixes https://github.com/flutter/flutter/issues/64752
Tests
TBA (to be added)
Checklist
Before you create this PR, confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I signed the [CLA].
[x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[x]  I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read [Handling breaking changes].

[x] No, no existing tests failed, so this is not a breaking change.

/cc @dnfield
This PR is also missing tests. Can you add some?

/cc @dnfield
This PR is also missing tests. Can you add some?

Working on that!

Change seems fine, let me know if you'd like any help writing a test.

Change seems fine, let me know if you'd like any help writing a test.

I definitely need some help. Can't find a way of testing this!

@pedromassango - if you take a look at the tests in test/widgets/image_test.dart, you can see how we create fake ImageStreams and ImageStreamCompleters.
You should be able to mimic one of those tests and basically just drive a fake ImageStreamCompleter such that you call reportError and then afterwards call setImage and then pump the tester and make sure the right widget(s) are displaying.

@pedromassango - if you take a look at the tests in test/widgets/image_test.dart, you can see how we create fake ImageStreams and ImageStreamCompleters.
You should be able to mimic one of those tests and basically just drive a fake ImageStreamCompleter such that you call reportError and then afterwards call setImage and then pump the tester and make sure the right widget(s) are displaying.

I tried exactly this approach yesterday, but when I call streamCompleter.reportError() the test stops and throws the exception provided in reportError(). Need to find a way to send the exception to the widget without throwing and stopping the test.

You'll want to set FlutterError.onError to something else, or use tester.takeException.

onError is not working, it always show me this error:
'package:flutter_test/src/binding.dart': Failed assertion: line 754 pos 14: '_pendingExceptionDetails != null': A test overrode FlutterError.onError but either failed to return it to its original state, or had unexpected additional errors that it could not handle. Typically, this is caused by using expect() before restoring FlutterError.onError.
dart:core                                      _AssertionError._throwNew
package:flutter_test/src/binding.dart 754:14   TestWidgetsFlutterBinding._runTest.handleUncaughtError
package:flutter_test/src/binding.dart 759:28   TestWidgetsFlutterBinding._runTest.<fn>
dart:async                                     Future._asyncCompleteWithValue.<fn>
package:fake_async/fake_async.dart 193:32      FakeAsync.flushMicrotasks
package:flutter_test/src/binding.dart 1192:19  AutomatedTestWidgetsFlutterBinding.runTest.<fn>

✖ Load a good image after a bad image was loaded
Exited (1)

May be similar to https://github.com/flutter/flutter/issues/34499

I have the test bellow but the issue is that the errorBuilder is not being called and it is the only way to verify that the issue was fixed. Tried using imageProvider.fail() but it fails the test case.
  testWidgets('Load a good image after a bad image was loaded', (WidgetTester tester) async {
    final UniqueKey errorKey = UniqueKey();
    Object caughtException;
    final ui.Image image = await tester.runAsync(createTestImage);
    final TestImageStreamCompleter streamCompleter = TestImageStreamCompleter();
    final TestImageProvider imageProvider =
        TestImageProvider(streamCompleter: streamCompleter);

    Widget app() {
      return Image(
        image: imageProvider,
        errorBuilder: (BuildContext context, Object error, StackTrace stackTrace) {
          caughtException = error;
          return SizedBox.expand(key: errorKey);
        },
      );
    }

    await tester.pumpWidget(app());
    expect(find.byKey(errorKey), findsNothing);

    streamCompleter.reportError(exception: 'throw', stack: StackTrace.empty);
    await tester.pump();

    expect(caughtException, isNotNull);
    expect(find.byType(SizedBox), findsOneWidget);

    streamCompleter.setData(
        imageInfo: ImageInfo(
      image: image,
      scale: 1.0,
      debugLabel: 'test.png',
    ));

    await tester.pump();
    expect(caughtException, isNull);
    expect(find.byType(SizedBox), findsNothing);
  });

I think you'll want to create your own custom ImageProvider fake that throws an exception (say in load) during the first pump and then doesn't throw an exception in the second pump.

@dnfield This is ready for another look!

Web error appears to be a flake. Can you just push a new empty commit to kick CI?

Something int he test isn't quite right - analysis is complaining.

Yes, looks like some API has changed


Update KinkedRconsumerType.ipynb

Made typo and LaTeX environment changes as usual. Link to the paper detailing the underlying theory was broken, so I linked the AER and NBER websites hosting the published version and its expanded edition instead. If there is some alternate link on the JHU website, which you would want to use instead, I can make the change.
CC: @llorracc @alanlujan91

@sidd3888 the text "\verb" seems to show up in the notebook preview, is this a bug?

@alanlujan91 Yes, I have changed that on my local repo... I'll push the changes here in just a bit. By the way, where can I see the preview, as I am currently just relying on compatibility across multiple IDEs

@alanlujan91 is the \verb still visible on the preview? I looked at the notebook on all possible platforms, barring the preview you mentioned. If it is still the case, please let me know where I can see this preview? Thanks.


NimBLEScan results are incomplete!

Add NimBLEAdvertisedDevice copy constructor and use it in NimBLEScanResults::getDevice(), because otherwise the std::string class member variables of the advertisedDevice is not returned correctly by NimBLEScan::getResults() and NimBLEScan::start(). Only the data from the BLEAdvertisedDeviceCallbacks (so on a per device basis) is correct.
I think (but I am not a 100% sure) the root cause is because std::map<std::string, NimBLEAdvertisedDevice*> m_advertisedDevicesMap in NimBLEScanResults is pointer based, and thus the NimBLEAdvertisedDevice* pointers are copied and not constructed.
The uint8_t *payload pointer may still be dangling!

Good catch, can't say I've used that method much. Any reason not to just return a pointer/reference there instead of copying?

Well I did not design the API! ;)
I added NimBLEAdvertisedDevice *getDevice(const NimBLEAddress &address) in PR #46, and this works correctly without the copy constructor. However, I think it is not correct to return non-const pointers internal structures that should not be modified by the program. But non-const pointers are returned everywhere in this library! If we make them const, as suggested in PR #42, the user program should update all pointer variables to be const also, so this is not without impact.
If we return a reference we should come up with a solution to a getDevice() out-of-bounds. Now a nullptr is returned, then some kind of empty device should be returned.
Speaking of returning references, I think it would be cool to return references everywhere instead of returning pointers. But then we should introduce also 'empty objects' for (intermediate) non-existent services or characteristics. Also some kind of 'check breadcrumb trail' should be introduced in each leaf of the service.characteristic.descriptor tree, for the user to check, if the retrieval of a value fails, until what level the path is valid. This heavily changes the API!

I did some additional tests and I think the copy constructor is not necessary. Closing for now.

Yes ideally we a const reference should be returned wherever internal structures are returned but that's a pretty big ask. I don't mind things as they are so much since it would be the API user that messes up the data if they change things, that;s on them 😃. The only other alternative is to return copies of the data, which I don't like, but is necessary sometimes.


_unequal_var_ttest_denom causes ZeroDivisionError in early samples

I am using python2.7 on CentOS 7, the latest version of scipy from pip.
def _unequal_var_ttest_denom(v1, n1, v2, n2):
When v2 (standard deviation from sample two passed from ttest_ind_from_stats) is 0, the context manager has no effect, and the code halts on a ZeroDivisionError exception. I've worked around it by modifying the code to catch the exception and assign df to np.nan, but not sure what should really be happening here.

Can you add a self-contained example which reproduces the problem please?

Sure, here you go:
Python 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy import stats
>>> stats.ttest_ind_from_stats(0.0,0.0,6.0,0.0,0.0,6.0,False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/scipy/stats/stats.py", line 3754, in ttest_ind_from_stats
    std2**2, nobs2)
  File "/usr/lib64/python2.7/site-packages/scipy/stats/stats.py", line 3680, in _unequal_var_ttest_denom
    df = (vn1 + vn2)**2 / (vn1**2 / (n1 - 1) + vn2**2 / (n2 - 1))
ZeroDivisionError: float division by zero

Thanks.
What would you expect the function to do given this input?

Based on the code and scipy documentation, I'm assuming that since the errstate context manager is used with divide='ignore', raising a ZeroDivisionError exception isn't intended.  Also, df is tested with isnan() in the next step, so it appears things were meant to continue under zero division conditions.
To answer your question more directly, I would expect the function to return with df=1.

A fix for this is in https://github.com/scipy/scipy/pull/13366.

A fix for this is in https://github.com/scipy/scipy/pull/13366.


Advise user that a variable is currently used in the demo before deletion

Considering that variables are not optional element, when deleting a variable, instead of blocking the action we should inform the user before confirmation

added ! to mark endpoints and variables being used in the demo. Can not find a more suitable icon


Module name on Documentation

Hi, thank you for publishing carousel library! I like your demo.
 import {IvyCarouselModule} from 'angular-responsive-carousel';

  @NgModule({
      imports: [IvyCarouselModule]
  })

I installed the latest version this morning and I could not import module like above in README.
I guess followings are correct.
 import {CarouselModule} from 'angular-responsive-carousel';

  @NgModule({
      imports: [CarouselModule]
  })

(Sorry, I cannot submit PRs to the repository without license due to my company's rule)

@renamoo Thank you for informing! I will fix it soon.

I have the same problem. But both modules (IvyCarouselModule and CarouselModule) are not found.
Error message:
./src/app/app.module.ts
Module not found: Error: Can't resolve 'angular-responsive-carousel' in '/code/src/app'

@FrancescoPaiola which version of Angular are you using?


Migrate Walden datasets from DAG to Snapshots

This would involve migrating all ingest scripts from Walden and rerunning them to get them into DVC. Though it would be nice to get rid of Walden dependency and be consistent, it looks like a lot of work for no practical reason. Some Walden datasets have been uploaded manually and some source URLs might have expired (or changed), so trying to rerun them again could be harder than initially thought.
An alternative is to write ingest scripts that download data from Walden and re-upload them as Snapshots. We could also link to the original Walden ingest script in case someone ever wants to replicate it (or rewrite it to use Snapshots with the new version). This would at least get rid of Walden as a dependency.

@larsyencken do you have an opinion on this one? I'm more and more leaning towards the alternative where we only copy commented code from walden ingest script. That shouldn't be much work.

I'd be inclined to automatically migrate the contents of Walden to Snapshots somehow, and then to remove walden as a dependency altogether if that's possible.
You could ask the data team if there are any frequently run or high-value ingest scripts, and port the most important two or three, but in general it's probably fine to copy them across commented out.

Migrating all datasets is probably not worth it due to potential errors in manual refactoring. We'd have to do thorough verification of all datasets (using compare tool). This might be worth doing after we have an automatic way of doing that (i.e. having something like SVG tester for ETL).

Not worth it. Better to let it die slowly.


Support for TMTpro/TMTpro-18plex

Hi all,
I'm planning to use MetaMorpheus to search a TMTpro 18 dataset.
For this, I was planning to add the modifications to the tmt.txt file.
While inspecting the file, I saw that e.g. TMT11 lists 12 diagnostic ions.
Is the 12th DI the full label?
Before I lookup all the DI, is it already possible to search TMTpro/TMTpro 18 with MetaMorpheus?
Best,
Manuel

Hello Manuel,
It's been a while since I coded this. I don't recall if there are or were different versions available. I see we have three reporters for 117. Here are them all below from unit tests.

If you come up w/ the formulas and so forth, maybe you could post them here. Then we can update MM so they are native in the next release.

I'll definitely post them here, once I got them and verified they are correct!
Regarding the masses above, afaik the TMT reporter masses range from 126 to 135, so I'm a bit confused what these masses actually are (see here http://www.unimod.org/modifications_view.php?editid1=2016)

These should be the right reporter ions masses and formulas for TMTpro and TMTpro 18
Label,Mass,Formula
126,126.12773,C(8)H(15)N(1)
127N,127.12476,C(8)H(15)15N(1)
127C,127.13108,13C(1)C(7)H(15)N(1)
128N,128.12812,13C(1)C(7)H(15)15N(1)
128C,128.13444,13C(2)C(6)H(15)N(1)
129N,129.13147,13C(2)C(6)H(15)15N(1)
129C,129.13779,13C(3)C(5)H(15)N(1)
130N,130.13483,13C(3)C(5)H(15)15N(1)
130C,130.14115,13C(4)C(4)H(15)N(1)
131N,131.13818,13C(4)C(4)H(15)15N(1)
131C,131.14450,13C(5)C(3)H(15)N(1)
132N,132.14154,13C(5)C(3)H(15)15N(1)
132C,132.14786,13C(6)C(2)H(15)N(1)
133N,133.14489,13C(6)C(2)H(15)15N(1)
133C,133.15121,13C(7)C(1)H(15)N(1)
134N,134.14825,13C(7)C(1)H(15)15N(1)
134C,134.15457,13C(8)H(15)N(1)
135N,135.15160,13C(8)H(15)15N(1)

I verified formulas have the right masses using this python script:
In [1]: import numpy as np

In [2]: from chemical_composition.chemical_composition_kb import PROTON

In [3]: from chemical_composition import ChemicalComposition

In [4]: with open("TMTpro_formulas.csv") as fin:
   ...:     next(fin) # skip header
   ...:     cc = ChemicalComposition()
   ...:     for line in fin:
   ...:         formula = line.split(',')[2]
   ...:         mass = float(line.split(',')[1])
   ...:         cc.use(formula=formula)
   ...:         theo_mass = cc.mass() + PROTON
   ...:         print(bool(np.isclose(mass, theo_mass)) is True)
   ...:
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True


Added RequestCollapser metrics to Yammer Metrics Publisher

Fixes #609

Hystrix-pull-requests #264 FAILURE
Looks like there's a problem with this pull request

NetflixOSS » Hystrix » Hystrix-pull-requests #25 SUCCESS
This pull request looks good


Adds doc example for Listview and pageview

Description
Added example code that creates a ListView or Pageview with SliverChildBuilderDelegate that supports children reordering.
Related Issues
https://github.com/flutter/flutter/issues/21023
Tests
N/A
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[ ] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] I signed the CLA.
[ ] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] All existing and new tests are passing.
[ ] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[ ] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require Flutter developers to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (Please read Handling breaking changes). Replace this with a link to the e-mail where you asked for input on this proposed change.
[ ] No, this is not a breaking change.

@dnfield can you take another look at this pr?


[SPARK-34771]  Support UDT for Pandas/Spark conversion with Arrow support Enabled

What changes were proposed in this pull request?
Fix a bug of Pandas/Spark conversion with Arrow support enabled.
Support DataType with UDT:

[x] UDT
[x] ArrayType(UDT)
[x] StructType(..., UDT, ...), postponed

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", True)
from pyspark.testing.sqlutils  import ExamplePoint, ExamplePointUDT
from pyspark.sql.types import StructType, StructField
import pandas as pd
schema = StructType([StructField('point', ExamplePointUDT(), False)])
pdf = pd.DataFrame({'point': pd.Series([ExamplePoint(1, 1), ExamplePoint(2, 2)])})
df = spark.createDataFrame(pdf, schema)
print(df.toPandas())

before (because it fallbacks to arrow disabled):
       point
0  (0.0,0.0)
1  (0.0,0.0)

after:
       point
0  (1.0,1.0)
1  (2.0,2.0)

Why are the changes needed?
Fix a bug.
For UDT without Arrow support, I will create another PR.
Does this PR introduce any user-facing change?
Support Python UDT in PySpark now. No user-facing changes.
How was this patch tested?
python/run-tests --testnames pyspark.sql.tests.test_arrow

Test build #136813 has started for PR 32026 at commit ecf0b14.

Can one of the admins verify this patch?

Test build #136820 has started for PR 32026 at commit cce66e4.

Test build #136813 has finished for PR 32026 at commit ecf0b14.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #136820 has finished for PR 32026 at commit cce66e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136820/

Can one of the admins verify this patch?

Test build #136834 has started for PR 32026 at commit 92f3829.

Test build #136834 has finished for PR 32026 at commit 92f3829.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136834/

@HyukjinKwon @maropu
First PR for SPARK-34600, please review.

Test build #136844 has started for PR 32026 at commit 1c84781.

Test build #136844 has finished for PR 32026 at commit 1c84781.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136844/

Test build #137878 has started for PR 32026 at commit 5845ee2.

CI failed because of branch name, as a result, I created another PR:
https://github.com/apache/spark/pull/32321

Test build #137878 has finished for PR 32026 at commit 5845ee2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137878/


Documentation

The document should be hosted on ReadTheDocs before 1.0 release.

In progress...
https://readthedocs.org/dashboard/sphinx-intl/

Done!


pick support explain delete from clause (#36782)

Proposed changes
support explain like:
explain delete from T where A=1
Issue Number: close #xxx

(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)
Proposed changes
Issue Number: close #xxx

run buildall

run p0

run buildall

run external

run p0

run buildall


BLD: fix issue with capitalisation when meson detects Accelerate

Should fix #21738. However, it's not clear to me what changed in upstream meson to cause this issue in the first place.

Hmm, I'll look into this. The form used in dev.py (lower-case, -Dblas=accelerate) is the desired one in principle. It looks like it didn't break in NumPy, so I think we're missing the custom lower-case accelerate dependency here and were inadvertently relying on the detecting via the 'framework' option to dependency().
Thanks for fixing the breakage!


My PR don't have the "approved-2" label

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the FastAPI documentation, with the integrated search.
[X] I already searched in Google "How to X in FastAPI" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to FastAPI but to Pydantic.
[X] I already checked if it is not related to FastAPI but to Swagger UI.
[X] I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
N/A

Description
I make a PR : https://github.com/tiangolo/fastapi/pull/5477
And 2 persons already approve it : emmrichard and rjNemo
but the bot didn't add the approved-2 label.
Futhermore, some PR have this label and aren't merge to the master branch : https://github.com/tiangolo/fastapi/pulls?q=is%3Aopen+is%3Apr+label%3Aapproved-2
I'm pretty sure that I forgot something to do, but I don't know what.
Thank you for your advice.
Operating System
Linux
Operating System Details
N/A
FastAPI Version
0.85.1
Python Version
3.10
Additional Context
N/A

You can just comment this at that PR itself, instead of creating another issue. We already have many issues, I would suggest you to close and discuss PR related stuff within the PR.

@iudeen : I understand, but If I add a comment in my PR, who will see it ? Only people who already aprove my PR.
I just want to know how the bot works to add the "approved-2" label.

Hey there! Yep, the label is added only to PRs that have the labels for "lang-all" and "awaiting review". You did nothing wrong, I just hadn't added those labels.
Thanks for help @iudeen !
🍰☕

Thanks!

Thanks for coming back to close the issue! ☕


trace.grpcUnaryInterceptor: fatal error: concurrent map writes

When calling several grpc service concurrently from the same context which contains some grpc metadata, we get a fatal error: concurrent map writes.
The issue is at trace/grpc.go:49:
md[grpcMetadataKey] = []string{header}
since md is a map shared between the various call. it should probably be cloned before been modified.

Sent out https://code-review.googlesource.com/#/c/12550/.

Thanks, I see you fixed #605 at the same time


check:data Last question: Reconciliation is NOT TESTED???

other question is passed ,but Reconciliation is NOT TESTED.
what add "xxx":"xxx" for config.json.
How can i to do ?
please tell me ,thank you very much!!!

Can you elaborate the following things:

What use case you are trying to test?
How do you test with reproducible steps?
What the expected results vs actual results?

Which are the api ports that must be implemented, I want to pass data api
and consturction api of rosetta-cli .but I don't want to implement them all。
Jingfu Wang @.***> 于2022年9月15日周四 23:40写道：

Can you elaborate the following things:

What use case you are trying to test?
How do you test with reproducible steps?
What the expected results vs actual results?

—
Reply to this email directly, view it on GitHub
https://github.com/coinbase/rosetta-cli/issues/358#issuecomment-1248278104,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AYMCDKS2WGEW2PDZFXQ4TNTV6M7PDANCNFSM6AAAAAAQNHOOH4
.
You are receiving this because you authored the thread.Message ID:
@.***>

hello my friend @GeekArthur Jingfu Wang .I want to pass data api and construction api of rosetta-cli.
Now, data api have last question ==> Reconciliation is NOT TESTED.
I don't know if there is something missing in my config.json =>
"data": { 	  "active_reconciliation_concurrency":16, 	  "inactive_reconciliation_concurrency":4, 	  "inactive_reconciliation_frequency":250, 	  "ignore_reconciliation_error":false, 	  "bootstrap_balances":"bootstrap_balances.json",       "historical_balance_disabled":true, 	  "interesting_accounts":"interesting_accounts.json",       "reconciliation_disabled":false, 	  "reconciliation_drain_disabled":true,       "balance_tracking_disabled":false,       "coin_tracking_disabled": true, 	  "exempt_accounts":"exampt_accounts.json",       "end_conditions": {         "duration": 20       } 	}
I just want the simplest use case to make sure that the coinbase exchange can be met.
can you tell me how to do? thank you very much!

Can you elaborate the following things:

What use case you are trying to test?
How do you test with reproducible steps?
What the expected results vs actual results?

*first, I want my blockchain to pass the test and meet the requirements of the coinbase exchange

*I have completed the development of my blockchain api based on rosetta-cli, but now check:data and the last one it does not detect, check:construction tells me that /construction/preprocess needs to be implemented, I have not implemented all of them, so I Wondering what needs to be done

*The test result is definitely not passed, so I am asking for your help, the reason has been mentioned in the first two

You should follow Getting Started for what APIs you need to implement and How to Write a Configuration File for rosetta-cli Testing for how to test your implementation properly
Also which blockchain you are working on?

I am AELF blockchain's Technical Engineer,Currently responsible for developing rosetta api, in order to connect with coinbase


How to install Bioconductor packages?

Hi Jinhwan Kim,
Thanks a lot for the detailed tutorial.
Do you know how to install Bioconductor packages for the shiny app? Maybe it is better to use remotes or renv to install all the required R packages rather than using automagic ?
Best,
Dong

Sorry for the late reply.
First of all, I have not yet succeeded in including packages from github outside of CRAN (bioconductor is the same).
So I used the functions from the github package in a rather strange way, by declaring them one by one inside.
Since I don't know exactly what automagic does, I might consider using remotes or renv as you mentioned.
I was hoping you could recommend a lightweight example package to practice embedding a package from bioconductor inside.
Thank you.
Kim.

Dear Jinhwan,
Thanks for your kind help.
Here is one simple shiny app using Bioconductor package ComplexHeatmap to plot Heapmap.
library(shiny)
library(ComplexHeatmap)

ui <- fluidPage(
  titlePanel("ComplexHeatmap Demo"),
  mainPanel(plotOutput("heatmapPlot", height = "800px"))
)

server <- function(input, output) {
  output$heatmapPlot <- renderPlot({
    heatmap(as.matrix(mtcars), show_row_names = TRUE, show_column_names = TRUE)
  })
}

shinyApp(ui = ui, server = server)

In the above example, I used only a single Bioconductor package. However ComplexHeatmap relies on 30 direct dependencies and 126 indirect dependencies. Manually declaring and installing all these dependencies can be a cumbersome and error-prone. It will be great that can be installed automatically in a similar way as well make shinyapp as docker images.

Best,
Dong

I'll check and let you know. (may in July~August)

Hi Jinhwan Kim,
I would like to give you a quick update on the installation of Bioconductor packages.
Currently, I copy the entire libraries from my original shiny project into the r-mac folder. This method allows me to install all the necessary packages, including the Bioconductor packages, without the need to run Rscript add-cran-binary-pkgs.R.
It works very well. The only disadvantage is the inclusion of some unnecessary packages, and the resulting electron app is large. I think this issue can be easily solved with renv.
Best regards,
Dong

Hi, @YonghuiDong
Based on the code you provided, I installed the bioconductor package and succeeded in running the shiny electron app.
and below is code for installing bioconductor's package and their dependency.

Important Note

I couldn't find a function to provide links to the binary files for the bioconductor packages.
so I had to manually paste the links to the binary source files in the documentation for each package. (as links)
# Copyright (c) 2023 Jinhwan Kim

# !/usr/bin/env Rscript

# Script to find dependencies of a pkg list, download binaries and put them
# In the standalone R library.

library(BiocPkgTools)

# get dependent Bioconductor Package

get_bioc_pkgs <- function(biocpkgs = 'ComplexHeatmap'){
  res <- biocpkgs
  while(length(biocpkgs)>0){
    dep <- unname(unlist(BiocPkgTools::pkgBiocDeps(biocpkgs[1])))
    if(length(dep)>0){
      res <- union(res, dep)
      biocpkgs <- biocpkgs[-1]
      for(i in 1:length(dep)){
        if(!dep[i] %in% biocpkgs){
          biocpkgs <- c(biocpkgs, dep[i])
        }
      }
    } else{
      biocpkgs <- biocpkgs[-1]
    }
  }

  return(res)
}

get_cran_pkgs <- function(biocPkgs){
  res <- c()
  for(i in 1:length(biocPkgs)){
    CRANdep <- unname(unlist(BiocPkgTools::pkgBiocDeps(biocPkgs[i], only.bioc = FALSE)))
    bioCdep <- unname(unlist(BiocPkgTools::pkgBiocDeps(biocPkgs[i])))
    res <- union(res, setdiff(CRANdep, bioCdep))
  }
  return(res)
}

# ComplexHeatmap - IRanges - BiocGenerics - S4Vectors

links <- c('https://bioconductor.org/packages/release/bioc/bin/macosx/big-sur-arm64/contrib/4.3/ComplexHeatmap_2.16.0.tgz',
           'https://bioconductor.org/packages/release/bioc/bin/macosx/big-sur-arm64/contrib/4.3/IRanges_2.34.1.tgz',
           'https://bioconductor.org/packages/release/bioc/bin/macosx/big-sur-arm64/contrib/4.3/BiocGenerics_0.46.0.tgz',
           'https://bioconductor.org/packages/release/bioc/bin/macosx/big-sur-arm64/contrib/4.3/S4Vectors_0.38.1.tgz')

td <- tempdir()
decompress <- untar

# download tgz file to temporary directory

for( i in 1:length(links)){
  downloaded <- download.file(links[i], destfile = paste0(td, '/package.tgz'))
  decompress(paste0(td, '/package.tgz'), exdir = file.path('r-mac', 'library')) # unzip to r-mac library
  unlink(paste0(td, '/package.tgz')) # delete temporary downloaded file
}

# delete unnecessary files
remove_dirs = c("help", "doc", "tests", "html", "include", "unitTests", file.path("libs", "*dSYM"))

z <- lapply(
  list.dirs(file.path('r-mac', 'library'), full.names = TRUE, recursive = FALSE),
  function(x) {
    unlink(file.path(x, remove_dirs), force = TRUE, recursive = TRUE) # delete
  }
)
invisible(NULL)

############## SAME code to install bioconductor-cran dependent packages
## install cran packages

library(automagic)

options(repos = "https://cloud.r-project.org")

cran_pkgs <- setdiff(unique(c("shiny", automagic::get_dependent_packages("shiny"))), "automagic")
cran_pkgs <- union(cran_pkgs, get_cran_pkgs(biocPkgs))

install_bins <- function(
    cran_pkgs, library_path, type, decompress,
    remove_dirs = c(
      "help", "doc", "tests", "html",
      "include", "unitTests",
      file.path("libs", "*dSYM")
    )) {
  installed <- list.files(library_path) # check installed packages

  cran_to_install <- sort(setdiff(
    unique(unlist(
      c(cran_pkgs,
        tools::package_dependencies(cran_pkgs,
                                    recursive = TRUE,
                                    which = c("Depends", "Imports", "LinkingTo")
        ))
    )),
    installed
  ))

  if (!length(cran_to_install)) {
    message("No packages to install")
  } else {
    td <- tempdir()
    downloaded <- download.packages(cran_to_install, destdir = td, type = type)
    apply(downloaded, 1, function(x) decompress(x[2], exdir = library_path))
    unlink(downloaded[, 2])
  }

  z <- lapply(
    list.dirs(library_path, full.names = TRUE, recursive = FALSE),
    function(x) {
      unlink(file.path(x, remove_dirs), force = TRUE, recursive = TRUE)
    }
  )
  invisible(NULL)
}

if (dir.exists("r-mac")) {
  install_bins(
    cran_pkgs = cran_pkgs, library_path = file.path("r-mac", "library"),
    type = "mac.binary.big-sur-arm64", decompress = untar
  )
}

I checked your comment before (copy and paste all packages) and it may work.
and i think it's not elegant way but still fine if it worked.
only you can try is remove unnecessary files for each packages like Doc / Help.
and it can be done with below's code
library_path = file.path("r-mac", "library")
  z <- lapply(
    list.dirs(library_path, full.names = TRUE, recursive = FALSE),
    function(x) {
      unlink(file.path(x, remove_dirs), force = TRUE, recursive = TRUE)
    }
  )
  invisible(NULL)

and finally
This is bad news.
Although I verified execution with electron-forge start command,
the creating the standalone app with electron-forge make was not possible due to an unknown error.
after I checked error log some font-config file in R (4.3.1 version) made this issue.
so I removed every file in r-mac/fontconfig/fonts/conf.d directory and it built (but not run)
but I think you may solve your main problem: Install bioconductor packages.
Thanks.
Kim

Dear Kim,
Thanks a lot for your help. I will test it and update you soon.
Best regards,
Dong

I closed this issue, but if problem occurs again, you're welcome to re-open or create new issue.


Refactor fileserver into sidecar with progress reporting

Note: This PR currently targets a throw-away branch. This PR should be rebased onto master after PR #1403 is merged, and the throw-away branch should be deleted.
Changes proposed in this PR

Renames the current "fileserver" to the more generic "sidecar"
Adds progress-reporting functionality to the sidecar (copied and adapted from the Cook Executor)
Enables the existing progress reporting integration tests to also run on Kubernetes.

Why are we making these changes?
Better parity of our Kuberentes scheduler with our Mesos scheduler with regard to job progress reporting.

@shamsimam - Opened PR #1419 to split the rename of "fileserver" to "sidecar" into a separate patch.

The fileserver module renaming PR (#1419) has been merged, and this PR has been rebased on top of that new master commit.

Awaiting green build.

Currently failing because we're not doing this in the job's container:
https://github.com/twosigma/Cook/blob/master/executor/cook/executor.py#L284


unsloth, bitsandbytes fail with triton installed, giving OSError: could not get source code

When importing unsloth executable is created, but when running it, It throws:
OSError: could not get source code
and exits.
Full log:
Traceback (most recent call last):
  File "D:\Project\ui.dist\program.py", line 2, in load_model
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\unsloth\__init__.py", line 97, in <module unsloth>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\bitsandbytes\__init__.py", line 16, in <module bitsandbytes>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\bitsandbytes\nn\__init__.py", line 17, in <module bitsandbytes.nn>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\bitsandbytes\nn\triton_based_modules.py", line 7, in <module bitsandbytes.nn.triton_based_modules>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\bitsandbytes\triton\int8_matmul_mixed_dequantize.py", line 9, in <module bitsandbytes.triton.int8_matmul_mixed_dequantize>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\triton\__init__.py", line 8, in <module triton>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\triton\runtime\__init__.py", line 1, in <module triton.runtime>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\triton\runtime\autotuner.py", line 7, in <module triton.runtime.autotuner>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\triton\testing.py", line 8, in <module triton.testing>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\triton\language\__init__.py", line 6, in <module triton.language>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "D:\Project\ui.dist\triton\language\standard.py", line 11, in <module triton.language.standard>
  File "D:\Project\ui.dist\triton\runtime\jit.py", line 714, in jit
  File "D:\Project\ui.dist\triton\runtime\jit.py", line 705, in decorator
  File "D:\Project\ui.dist\triton\runtime\jit.py", line 584, in __init__
  File "D:\Project\ui.dist\inspect.py", line 1240, in getsourcelines
  File "D:\Project\ui.dist\inspect.py", line 1077, in findsource
OSError: could not get source code

Nuitka version, full Python version, flavor, OS, etc. as output by this exact command.

python -m nuitka --version
2.4.7
Commercial: None
Python: 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
Flavor: CPython Official
Executable: C:\Users\Deepak\AppData\Local\Programs\Python\Python311\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 11
Nuitka-Scons:WARNING: Windows SDK must be installed in Visual Studio for it to be usable with Nuitka. Use the Visual
Nuitka-Scons:WARNING: Studio installer for adding it.
Version C compiler: ~\AppData\Local\Nuitka\Nuitka\Cache\DOWNLO~1\gcc\x86_64\13.2.0-16.0.6-11.0.1-msvcrt-r1\mingw64\bin\gcc.exe (gcc 13.2.0).

How did you install Nuitka and Python
I used pip install nuitka

The specific PyPI names and versions
xformers == 0.0.25.post1, unsloth==2024.8, trl== 0.8.6, triton== 2.1.0, transformers==4.43.4,
torch==2.2.2+cu121, torchaudio==2.2.2+cu121, torchvision== 0.17.2+cu121, tokenizers==0.19.1,
sentencepiece==0.2.0, safetensors==0.4.4, rich==13.7.1, deepspeed==0.13.1, bitsandbytes== 0.43.0.dev0,  decord == 0.6.0,
accelerate== 0.33.0.

In case you are using windows, please add llvm's bin and lib folder to your path (You can get it here : https://github.com/wkpark/triton/releases/tag/llvm-build-for-win)

Also supply a Short, Self Contained, Correct, Example

def load_model(path="unsloth/Meta-Llama-3.1-8B"):
    from unsloth import FastLanguageModel

    max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
    dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
    load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.

    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name = path,
        max_seq_length = max_seq_length,
        dtype = dtype,
        load_in_4bit = load_in_4bit,
        device_map = 'auto'
        # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
    )

    model = FastLanguageModel.get_peft_model(
        model,
        r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
        target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                        "gate_proj", "up_proj", "down_proj",],
        lora_alpha = 16,
        lora_dropout = 0, # Supports any, but = 0 is optimized
        bias = "none",    # Supports any, but = "none" is optimized
        # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
        use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
        random_state = 3407,
        use_rslora = False,  # We support rank stabilized LoRA
        loftq_config = None, # And LoftQ
    )

    return model, tokenizer

model, tokenizer = load_model()

Just FYI, the source can be found here : https://colab.research.google.com/drive/1Ys44kVvmeZtnICzWz0xgpRnrIOjZAuxp?usp=sharing

Provide in your issue the Nuitka options used

python -m nuitka --standalone --include-package='unsloth' --include-package='triton'  --module-parameter=torch-disable-jit=no --module-parameter=numba-disable-jit=no --follow-imports .\program.py

Another JIT to conquer, we will aim at 2.6 for this kind of operations to have plugin supports.


SLIM-844_Replace_Use_Of_SoapUI_For_SmartMeterMonitoring

This pull request makes the build fail.

This pull request makes the build fail.

retest this please

This pull request makes the build fail.

This pull request makes the build fail.

All is well.