Trackers: add stress-testing modules

To facilitate stress-testing for trackers we can add modules providing boilerplate code for each tracker allowing us to:

check how long does it take to send big queue of events
check if there's any errors happening due sending big volumes of events
compare performance of trackers

Though, I don't see it as unit-test, but rather as manual approach with manual deployment of collector.
Here's Node.js example:
// npm install node-uuid

// Random generator
var uuid = require('node-uuid');

// Initialization
var snowplow = require('snowplow-tracker');
var emitter = snowplow.emitter;
var tracker = snowplow.tracker;
var errors = [],
    successes = [],
    unknowns = [];

// Sent event counter
var sentEventCounter = 0;

function trackCallback(error, body, response) { // Callback called for each request
    sentEventCounter++;
    if (error) {
        errors.push({'error': error, 'sentEventCounter': sentEventCounter});
    } else if (body.statusCode === 200){
        successes.push(sentEventCounter);
    } else {
        unknowns.push({
            'error': error,
            'body': body,
            'response': response,
            'sentEventCounter': sentEventCounter
        });
    }
}

var e = emitter('collectorUri', 'http', 80, 'POST', 0, cb);
var t = tracker([e], 'myTracker', 'myApp', false);

// Getters
function getSuccesses() { return successes; }
function getErrors() { return errors; }
function getUnknowns() { return unknowns; }

function generateEvent() {
    var rnd = uuid.v4();
    var pageView = {
        'pageUrl': 'http://example.com/' + rnd,
        'pageTitle': rnd,
        'referrer': rnd
    };
    t.trackPageView(pageView.pageUrl, pageView.pageTitle, pageView.referrer);
}

function start() {
    for (var i = 0; i < 10000; i++) {
        generateEvent();
    }
}

module.exports = {
    generateOneEvent: generateEvent,
    getErrors: getErrors,
    getSuccesses: getSuccesses,
    getUnknowns: getUnknowns,
    start: start
};

Love this idea! It's kind of like "avalanche-client"...

I can say so far is that Python tracker's async emitter performance is extremely low. With 4 threads it loads collector with 2 req/sec.

Hey QA-workgroup (@alexanderdean @BenFradet @jbeemster @ihortom)!
TL;DR I propose a TSV file format that trackers can read to get events and then we can compare Redshift content and these files.
I implemented small PoC of "QuickCheck for Snowplow" in Scala Tracker. It basically consist of:

Random event generator
Logic to save  objects to TSV files with format similar to following:pv	chuwy.me/page.html	Main page	http://google.com		ttm:12312313253 where first pv refers to page view and subsequent columns are just tracking functions argument. Having that most of our trackers use same positional argument, I believe this should be quite ubiquitous
Logic to create emitter and tracker, read these TSV files and flood collector with these events from TSV.

What do I propose here? As I said we can re-use this format to see if there's any data-loss on Tracker -> Collector part of pipeline, compare tracker's performance and discover problems.

I like it! The only thing I am not super-hot on is the TSV format. It feels a bit artisanal / YAP (yet another protocol). Why not:

Use a Thrift or Protobuf
Use standard naming for each field based on the Snowplow Tracker Protocol

Sure, why not. If this can be implemented on all our platforms.

Move to general tracker (protocol) repo.


curl/wget (need help)

I'm trying to PR a working curl / wget (using /dev/tcp)
set -ex

function __curl() {
  read proto server path <<<$(echo ${1//// })
  DOC=/${path// //}
  HOST=${server//:*}
  PORT=${server//*:}
  [[ x"${HOST}" == x"${PORT}" ]] && PORT=80

  exec 3<>/dev/tcp/${HOST}/$PORT
  echo -en "GET ${DOC} HTTP/1.0\r\nHost: ${HOST}\r\n\r\n" >&3
  (while read line; do
   [[ "$line" == $'\r' ]] && break
  done && cat) <&3
  exec 3>&-
}

__curl http://www.google.com/favicon.ico > mine.ico
md5sum mine.ico

Yet 'im stuck on the && cat to handle the file body (binary)
I'm sure i can use a new file descriptor && echo , but my bash skill ends here  😞
I can link & use a pure bash [1], yet i'm sure there is something more elegant to do here.
[1] https://unix.stackexchange.com/questions/83926/how-to-download-a-file-using-just-bash-and-nothing-else-no-curl-wget-perl-et

I got it working. It's a little slow as it requires two while loops. I'm going to work on making this even faster but for now it's an example. Usage is script url > file.
Example script:
#!/usr/bin/env bash
#
# Download a file in pure bash.

download() {
    IFS=/ read -r _ _ host query <<< "$1"

    # Send the HTTP request.
    exec 3<"/dev/tcp/${host}/80"; {
        printf '%s\r\n%s\r\n\r\n' \
               "GET /${query} HTTP/1.0" \
               "Host: $host"
    } >&3

    # Strip the HTTP headers.
    while IFS= read -r line; do
        [[ "$line" == $'\r' ]] && break
    done <&3

    # Output the file.
    nul='\0'
    while IFS= read -d '' -r line || { nul=""; [[ -n "$line" ]]; }; do
        printf "%s%b" "$line" "$nul"
    done <&3

    exec 3>&-
}

download "$1"
``

The fist loop is reasonably slow as it will just drop a sane amount of headers, i can't understand how the 2nd loop (a simple cat !!) can be so complicated (hence, slow i guess)

Bash is slow at file IO and it doesn't handle binary data very well. I'm sure it can be optimized but I have some doubts as to whether or not this will ever be faster than wget/curl.

According to the "bash bible" - yours :p   a simple cat alternative might be
file_data="$(<"file")"

Yet i cannot make this work with my design, but i do not understand why

cat handles binary data correctly iirc, bash doesn't. What causes a larger problem is that bash handles binary data and null bytes differently depending on which version you're using (In 4.4+ null bytes are skipped and never reach the variable).

Enjoyed every bit of your blog post. Really looking forward to read more. Fantastic.
SEO Service Packages In Sri Lanka

All the other examples here make sense and can often be faster than invoking another program. However, in the case of networking, I think it makes sense to depend on the binaries, both for useability and performance.
In the case of wget / curl replacements, all of these only work when you have a HTTP endpoint. This code is not going to work for HTTPS.


Error describing ; it is not the same as <!ww>\s*

On Regexes > Predefined Regexes the page says

<ws> | no | Whitespace, same as: <!ww>\s*

This is not quite true.  As nemokosch explains, \s can backtrack, but <ws> cannot.  Here's an example to demonstrate the difference:
say "a\n" ~~ / 'a' <!ww>\s* \n /   #Output ｢a
                                   #       ｣
say "a\n" ~~ / 'a' <.ws> \n /      #Output Nil

I think the ws rule should be described as a token. So it's the same as:
:r <!ww> \s* :!r

Hi @deoac,
I'm wondering why you haven't responded. I worry that I wasn't clear enough so will be even more explicit below. Plus I have another thing to mention that's technically a red herring relative to this issue but might be confusing some folk so I'm going to write another comment about that too.

"<ws> is not the same as <!ww> \s*.

That's incorrect. It is exactly the same.

As I tried to demonstrate/explain in my first comment, the key doc improvement that would technically fully address this issue per its stated substance would be to explicitly classify <ws> as a token, though I recommend broadening it to cover all the "rules" in the table in the doc.

The red herring that I chose not to mention before (precisely because it's a red herring relative to the substance of this issue), but may cause some folk confusion (because it is confusing -- as far as I can tell it directly contradicts statements in the design docs, the user docs, and @Larry discussions) is that :sigspace / rules enable backtracking for the atom preceding any inserted <ws>. This is my current best formulation of the surprise discussed in Rakudo issue https://github.com/rakudo/rakudo/issues/3726.

Hi raiph!
Your explanation that <ws> is a token, and hence doesn't backtrack, was very clear and helpful.
Thanks,
Shimon

Well, "rules" also don't backtrack, if we can trust the documentation with this: https://docs.raku.org/language/grammars#Rules
Actually, I couldn't tell by only what we see whether they are implemented as tokens or rules. Perhaps this one?
Either way, I think the most apparent problem is not what we call it but the fact that it is not the same as the snippet given as "the same". I think it should be checked if there is a similar snippet that is indeed the same, and if it starts to get complex, it's better to just describe the behavior.

Well, "rules" also don't backtrack, if we can trust the documentation with this: https://docs.raku.org/language/grammars#Rules

Yeah. Unfortunately you can't trust it. And, more to the point, it's wrong.
rule has backtracked in Rakudos since 2014.01.
I keep forgetting that I haven't yet widely reported/discussed that.
See https://github.com/rakudo/rakudo/issues/3726.

Actually, I couldn't tell by only what we see whether they are implemented as tokens or rules.

Imo it's not important whether they're implemented as tokens or rules, just what their behavior is best described as.
And while the word "token" and keyword token have unfortunate ambiguities, they aren't pertinent in any way I can see here, and the unfortunate ambiguities that "rule" and rule have, and especially the current Rakudo behavior of rules, make "rules" an especially poor pick for describing what these rules (tokens) do.

Perhaps this one?

No. That's the _ws that's called by the ws that's part of the grammar for parsing Raku code.
The ws we want to document for end users is the one that gets invoked by writing <ws> or writing significant space when :sigspace is in effect in a user defined grammar.
And, as explained in my related detailed write up of this stuff, that's in NQP, presumably here.

Either way, I think the most apparent problem is not what we call it but the fact that it is not the same as the snippet given as "the same".

I think that, as far as documenting it for an ordinary user is concerned, it should be referred to as the same as the snippet given as "the same", and the key thing that needs to change is to refer to it as a "token" or token.
I have gone down the rabbit hole of analyzing the code for ws like I did for ident in the SO I linked above, but need to call it quits on that for about a week, other than the following summary.
The ws I've linked has this comment:
# skip over any whitespace, fail if between two word chars

Because the whitespace matching ("skip") consumes characters and the other ("between") doesn't, the fact that the comment mentions them in the opposite order from the snippet is technically immaterial, even though it's disconcerting. That is to say, these are logically the same:

Match any amount of whitespace, including none. Then test if not between (both before and after) word characters.

Test if not between (both before and after) word characters. If not, then match any amount of whitespace, including none.

Imo it's easier to understand written as <!ww> \s*. But YMMV.
Beyond the comment I've explored around the code a bit, but have concluded it's beyond my paygrade to draw any conclusions beyond that I get the impression the code does what the comment says it does, and that that matches the snippet.

I think it should be checked if there is a similar snippet that is indeed the same, and if it starts to get complex, it's better to just describe the behavior.

I think it's best to do both, but that what matters for closing this ticket is just to refer to the "rules" as "tokens" instead.
For the snippet, I think the two options are:
<!ww> \s*
\s* <!ww>

and I favor the first.

I've got some heavy RL commitments until next week so probably won't be able to follow up further till then.

Yeah. Unfortunately you can't trust it. And, more to the point, it's wrong.
rule has backtracked in Rakudos since 2014.01.
I keep forgetting that I haven't yet widely reported/discussed that.
See rakudo/rakudo#3726.

Well, to me, the linked issue, and the linked discussions in that issue, rather sound like this is simply a bug, moreover a bug that doesn't make rules consistently backtrack, only under certain circumstances. So rather than "fixing" the documentation, what should be noted, if at all, is that Rakudo is buggy with rules. I don't think that has any implications on the terminology.

No. That's the _ws that's called by the ws that's part of the grammar for parsing Raku code.
The ws we want to document for end users is the one that gets invoked by writing <ws> or writing significant space when :sigspace is in effect in a user defined grammar.
And, as explained in my related detailed write up of this stuff, that's in NQP, presumably here.

Given my subjective impression on Rakudo's architecture, even being aware that the ws defined here is for parsing Raku itself, it wouldn't have surprised me if the named tokens simply slipped through from an arbitrary Raku-related grammar Rakudo has... I wouldn't assume common sense with these things, to be honest.
But anyway, thank you for clarification. I checked several NQP source files but didn't think of that one.

I think it's best to do both, but that what matters for closing this ticket is just to refer to the "rules" as "tokens" instead.

Well, what can I say. You haven't convinced me at all about the second part of the sentence. If someone just changed "rules" to "tokens" and called it a day, I would genuinely open another issue because it doesn't address the problem that is even stated in this one.
<ws> is not the same as <!ww> \s*, period. One may say something like "<ws> could be defined as token ws { <!ww> \s* }", that would be accurate at least, and the user could at least get the idea that the word "token" bears special meaning and significance here; that it's not "this simple".
In general, it would be great to set up less traps for users in the documentation as well. If somebody already opened an issue for two pieces of syntax inside a regex not being the same, and they are indeed not the same, it's better to not claim that they are the same, with an overly subtle disclaimer. I'm not saying this is deliberate but I think there are loads of communication patterns in the explanations users get from the Raku community that are most similar to gaslighting. Let's be more open in the communication! Let's show the connection, let's maybe offer a pseudo-implementation and let's emphasize the significance of the word, whether that word is "rule" or "token". (I'm fine with "token", especially if "rules" are broken.)

I just opened Raku/problem-solving#390 to help resolve the uncertainty around how rules should behave.
Please note that, as I mention in that issue, the problem is not actually with rules specifically but rather with :sigspace more generally.

@raiph wrote:

All of which leads me back to my original conclusion: make the tiny judicious edit: s/other rules/C<tokens>/. Job done.

@2colours wrote:
<ws> is not the same as <!ww> \s*
I agree that describing <ws> as "the same as" <!ww> \s* is misleading – or at least potentially confusing.  When $thing is described as "the same as" $other-thing, I usually think that they can be freely substituted, which isn't the case here.
However, with a tiny extension to @raiph's judicious edit, I think we can avoid the confusion: I propose replacing "Whitespace, same as: <!ww>\s*" with "Whitespace, same as: [:ratchet <!ww>\s*]". (Plus a parallel edit for ident).
Would that solve this issue?

I've been semi-following this issue in the background, but it feels like it's gone all wonky.
My suggestion is to just say that the default <ws> is the same as  <!ww>\s*:  and leave it at that.  Then it doesn't matter if the reader is thinking rule or token or regex.  The behavior is "correct" in all of those contexts.  You don't need to get into the vagaries of :ratchet in the middle of regex slang to figure out what it does.
FWIW, I think it's recursive/self-referencing (and incorrect) to think of <ws> as rule ws { <!ww> \s* } , so let's avoid  that from now on.
My apologies if this suggestion misses some key point that was made earlier in the thread.
Pm

Also, I would agree with @raiph that whenever <ws> is identified, it's better to call it a "token" than a "rule".  It doesn't need to be pedantically done throughout the documentation, but anytime the docs or someone can say "token" instead of "rule" and still have the correct concept come through it's a level of precision that will greatly improve the documentation, imo.
I also think understand some of the confusion.  The common idiom for formal grammars has historically been that they consist of "rules" as the fundamental element, this idiom existed long before Raku was a thing.  Even Wikipedia says "A formal grammar is a set of rules..."  In Raku, grammars are really classes with methods written using regex syntax, some of which methods are "tokens" and most of which are often "rules" in a ratchet/sigspace sense.  When describing Raku to someone supposedly having a background in formal grammars, we'll want to use "rule" in its more generic "it describes a syntax" sense rather than the Raku "a regex with ratchet and sigspace enabled" sense.
IMO one of Raku's greatest innovations was recognizing that what everyone traditionally called "rules" and "tokens" in the formal grammar world could actually be specified as regular expressions with certain features enabled by default.  Prior to my encounters with Raku (or "Perl 6" as it was known then), almost all parser tools and language descriptions I worked with used one syntax/system to describe tokenization, another syntax/system to describe grammar rules, and handwaved whitespace issues by English descriptions or custom code to handle the special cases.  Raku provided a common syntax and description language that cleanly encompassed all three.
To me, Raku's meaning for rule most closely approximates what we used to call "rules" in Backus-Naur form and other formal language syntax notations, so we shouldn't try to change Raku terminology, or even consider it a mistake in the original language design.  Coming from a Perl 5 world, it seemed natural to treat regexes as fundamental and look at token/rule as specializations of that.  But for and audience of people coming from a formal grammar background, it might've been better to conceptualize it the other way -- a "token" is a rule where whitespace in the syntax isn't significant, and a "regex" is a token where backtracking is enabled by default.   (I do think Raku is correct by treating regex as fundamental.)
Even if it had been different, we'd still have the issue that some "rules" (regexes) in a grammar have special features, and whether we choose "rule", "regex", or "method" as the term for its fundamental building block, we still have to handle describing to a new user that something like <ws> is a "token" and not strictly either a "rule" or a "regex".
Thanks for letting me add a (potentially very wrong) perspective to the discussion.
Pm
We defined Raku rule in terms of token and regex... perhaps it would have been better to conceptualize it the other way around:   A token is a rule where whitespace in the syntax isn't significant, and a regex is a token with backtracking enabled by default.  But our Perl 5 roots were pretty strong, and everyone loved regexes and so treating those as the base element felt like the right thing at the time.
Pm

Lastly, and apologies for the third post:
I'm the one that coined <ww>, and it was explicitly coined to make default <ws> behavior easier to describe by using the existing regex syntax.   Before <ww> existed it was always kind of difficult to explain how <ws> actually worked.
But behind the scenes <ws> never actually used the <ww> subrule directly, and I suspect it still doesn't.  For efficiency reasons <ws> was a hard coded method that directly checked word characters and consumed whitespace as needed, it didn't make an actual call to <!wb>.  When parsing something the <.ws> subrule tends to get called a lot (like, oh, when parsing Raku source programs), so you want that to be subrule call to be as absolutely efficient as you can possibly make it.
I spent a long time trying to describe <ws> behavior in regex syntax using the existing built-in constructs at the time, and I never found a satisfactory answer.   Once I came up with <ww> as a new pseudo-builtin it was easy though, and Larry and the rest of the design team ultimately accepted and adopted it.  <!ww>\s* describes exactly what we want -- "Fail if between two word characters, otherwise consume any optional whitespace".  (It turns out the prose description is also much easier if you think of "between two word characters" as a built-in of some sort... all of the prior English descriptions we had for "optional whitespace except where it isn't" were also either long or convoluted or handwavy.)
If anyone comes up with a definition for <ws> that doesn't use something like a <ww> subrule, perhaps that'd be a better description for documenting it now.  Back then we didn't have the << or >> word boundary built-ins in the language yet, so maybe defining <ws> can be easier now with current built-ins than with what we had defined then.  I haven't thought it through deeply.
Regardless, I ultimately believe the default <ws> needs to be a ratcheting expression (a token) and not backtrack by default.
Lastly, even though <ws> didn't actually use <ww> in its implementation due to performance reasons, I did go ahead and implement a built-in <ww> subrule anyway.  I did this because (1) it's much easier to do that than to explain that <ww> was primarily intended to succinctly explain what <ws> did without actually using it, and (2) someone creating a custom :sigspace token might want to have a built-in like <ww> readily available, since it's apparently relatively hard to create it using just basic regex syntax.
Pm

But it seemed to me you thought the doc was right, and I thought it appropriate to let you know it wasn't, and might need to be changed, but definitely needed to not be trusted on this point until we resolve the issue.

I still think the doc is right, and Rakudo is wrong. I'm not going to delve into Roast again, neither do I demand you to do that but as long as it's not proven to be specified that rules were meant to backtrack, under pretty much unspecified circumstances, I won't deem it a doc issue. Why would it be different than any of the Rakudo bugs?

I haven't attempted an analysis of code in the wild, but my guess is that more than 90% of all grammars that have ever been written, including one line grammars, contain at least one instance of this, and more than 10% contain hundreds of instances.

The irony is that I picked up this habit to call Raku a "90% language" whenever I'm criticizing the utterly non-exhaustive design of it. I can only assume this mentality is coming straight from Perl. Regardless, I have never seen a language where it was okay if something worked 90% of time and whenever users made to the remaining 10%, there was even a magic phrase to invalidate the use case (DIHWIDT, sometimes used abusively).
I think this mentality should go, and after that, eventually the language should move away from hardcoded defaults that are cute 90% of the time and completely ruin the remaining 10%. So please, don't normalize something working one way an undefined 90% of time and another way the remaining times. If it should backtrack, it should backtrack always, and we should document that. If it shouldn't backtrack, then we are back to a serious bug.

(...) And sorting out those problems may take years.

It doesn't matter. Severity and consequences doesn't make a bug a feature. If it takes long, that's all the more reason to state that it's bugy.

But you said something to the effect of trusting the doc about something it was wrong about.

Frankly, I'm gonna repeat it as many times as you make that reference: the doc was not wrong about it. Rakudo is wrong about it. In an ideal world where Rakudo doesn't have monopoly over the project, it shouldn't even matter. And I don't know why you thought it added to the discussion in the first place if you even knew that this was a Rakudo bug all along.

What does have an implication on the terminology relevant to this issue is that ws is a token in terms of its behavior. And calling it a "rule" is unfortunate, because the word is ambiguous, and that ambiguity was central to @deoac filing this issue.

I don't think the docs read with "rule" as a term in the sentence, rather than just rule as in a rule of thumb or a rule to follow. Moreover, it seemed to me that you proposed to use the word "token" under similar vague conditions but that's beside the point.

The fact is, there is no error in describing <ws>. The fact is it is the same as <!ww> \s*. And @deoac has (presumably) agreed.

I think this is so far-fetched that it's basically untrue. Somebody, who uses the documentation to get to understand things, could understand the behavior and was happy about that. This is a perfectly normal reaction for a user: they rather want an explanation than a fix for the documentation. Hell, actually, I was the one who proposed opening an issue! If it matters so much for the essence of the problem, I'm going to just open a new one.
And I don't know, I frankly cannot get around your usage of "the same". As @codesections said, if they are "the same", it's the least to be expected that they are substitutable for each other. <!ww> \s* is not a token, for starters. Of course they are not substitutable for each other when they aren't even the same category of things.
It seems to me that you expect the user to resolve this apparent contradiction routinely in favor of the word "token", like the users should automatically go from "hm, it cannot be the same and a token at once" to "okay, I get it, it's as if you defined a token with that regex". Why are you so sure this happens? Why do you think their eyes won't catch "the same as (...)" more than the disclaimer somewhere on top about being a token, or won't trust this (wrong) statement more?

it would have surprised the heck out of me because those named tokens were designed before their implementation, and the Raku ws is very clearly a custom user defined ws.

Point is, existing methods slip through grammars and can interfere with tokens you define there. I couldn't be very surprised if there was some global mishmash in the core as well.

We disagree about whether they are the same. And I provided no disclaimer. I merely suggested changing "other rules" to "Cs". The doc had been fine for years, with no one being "trapped" by what it said. When @deoac posted to say there was an issue, I accepted that, and drilled down to the key change that would have headed off filing an issue about it.

It's really hard this way, basically every sentence is a serious obstacle to what I'm trying to get across.
Whether two things are the same or not shouldn't be a matter of opinion; I cannot accept your standpoint on that. I think it's factually proven at this point, back and forth, that they are not the same.
I would seriously disadvise drawing positive (as in, actively stating) conclusions from lack of feedback. We don't know how many people have read that, with what background, and what their reaction was. It could even be that they also thought of free substitution (what "the same" really implies) and they just haven't hit a case where it matters. It could be that they had a similar bad opinion about the docs as you have, and ignored the whole stuff.
And as I already said, if it makes things clearer: treat this issue as my issue. A user was baffled about behavior based on what the docs said and it was probably nighttime so I asked them to open a doc issue, rather than doing a PR right away. I'm thankful for catching this issue which I think stays with us, whether the OP cares about it or not.

But imo making ws complicated is gaslighting. It's really dead simple. If the current regex engine is between two word characters, then it doesn't match. If it does match, it consumes all contiguous whitespace. And it doesn't backtrack -- it's a token. End of story.

Excellent. This could even be written down instead of "it's the same as (...)" something that isn't even a token.

My own proposal would be still something like:

let's refer to them as tokens, preferably with a link to what a token is
instead of saying "it's the same of ", let's indeed say "it's defined like token { <this piece of regex> }, which is consistent with the hinted "tokenness"

My suggestion is to just say that the default <ws> is the same as <!ww>\s*: and leave it at that.

@pmichaud
I had to read all your messages, then go back to these snippets, confirm they all contain the colon and even mention it... Aaand it is even documented, apparently... https://docs.raku.org/language/regexes#Preventing_backtracking:_:
I have never seen this over years, neither in code, nor in the documentation, and the biggest problem is probably the usual one: I would have never imagined something like this exists, let alone with a colon. I have the vague impression that this never got proposed because the others also never thought of this, or worst case, genuinely weren't aware of it.
I think it is a great idea actually to use it here but at the moment this colon seems so arcane that we should perhaps deliberately direct some attention to it for the time being, even if by a lame didactic "please mind the colon modifier of the quantifiers".

By the way, https://docs.raku.org/language/grammars#ws even states the wrong thing explicitly: with regex, instead of token...

I have never seen this over years, neither in code, nor in the documentation, and the biggest problem is probably the usual one: I would have never imagined something like this exists, let alone with a colon. I have the vague impression that this never got proposed because the others also never thought of this, or worst case, genuinely weren't aware of it.
I think it is a great idea actually to use it here but at the moment this colon seems so arcane that we should perhaps deliberately direct some attention to it for the time being, even if by a lame didactic "please mind the colon modifier of the quantifiers".

I agree.  The :, ::, and ::: colon modifiers for backtracking control have been present from the earliest versions of the regex language, even (I think) predating "token" and "rule".    The fact that we no longer encounter them frequently in actual code perhaps testifies to just how effective "token", "rule", ":ratchet", and ":sigspace" have been.  Earlier versions of grammars we wrote would make frequent use of the colon modifiers to try to DTRT, which led to finding ways to make backtracking control more automatic and not have to sprinkle them everywhere.
The colon modifiers just make sense -- they give a way to control backtracking at an atomic level, for those cases where you want to enforce/prevent backtracking for a specific quantifier or backtracking point and not as the default used by the entire regex/token/rule.
: came first, the :! and :? forms came much later as we were coming up with a generic pattern for understanding backtracking control.   Things we use all the time like .*? are really a shorthand notation for .*:?.
Thanks to whoever added #4370 to get these (highly useful in key contexts) modifiers documented, since their existence had been lost to the sands of time.....
Pm

For those who like history, I even found the original March 2009 thread where <ws> was clarified and <ww> got introduced.
https://www.nntp.perl.org/group/perl.perl6.language/2009/03/msg31119.html
Larry's comment three messages later still warms my heart to this day:

From: Larry Wall
Date: March 9, 2009 10:53
I have wanted <!ww> a number of times, particularly after generic
tokens that might or might end in \w.  So feel free to spec it.

Pm


[spark] Set DMatrix enable_categorical to True if user provide feature_types param

If user provide feature_types param, we should set enable_categorical to True when constructing DMatrix.

@trivialfis Is my understanding correct ? I will file PR for it once you confirmed it.

Correct. Please submit a PR.

@trivialfis Thank you!


Allow changing font size and keyboard shortcuts

Two core configurations I would love to see implemented (even if just in a config file).

Allow changing the font and size.
Allow changing keyboard shortcuts.

BTW: I am loving some of the enhancements over Crunchyfrog. Tab complete is awesome! Loading the last editor state on startup is awesome, and the simplistic view is nice.

Thanks for the feedback! I really appreciate it.
I'll add some configuration options for the next release. But I've just committed a change that uses the default monospace font of your system (when using the GNOME desktop).
Change font/font size and keyboard shortcuts are both very high on my list.

Font size can now be changed using the preference dialog.

Forgive my ignorance. I just pulled the latest code, and don't see any button or menu item to open the preference dialog.

The drop-down menu on the application name in Gnome's app bar:

Weird, I don't have that option. I see the following on branch master..

New SQL Editor (which is dimmed out)
Close SQL Editor
Quit

Maybe you'll have to run make compile-resources before launching the app.
This recompiles the UI stuff.
Michael Shepanski<EMAIL_ADDRESS>schrieb am Di., 24. Jan. 2017
um 15:06 Uhr:

Weird, I don't have that option. I see the following on branch master..

New SQL Editor (which is dimmed out)
Close SQL Editor
Quit

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/andialbrecht/runsqlrun/issues/15#issuecomment-274811198,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABkIACEys_f16GlpZxpO9Vzs_12Z620ks5rVgVogaJpZM4GnRyF
.

That was it.  :+1:
I need to get used to compiling python code I suppose!

That's actually a GTK thing. I create parts of the UI using Glade (
https://glade.gnome.org/). The UI definitions are stored in the data/*.ui
files. Those (XML) files need to be compiled to some binary stuff, that can
be loaded into the GTK framework.
Michael Shepanski<EMAIL_ADDRESS>schrieb am Di., 24. Jan. 2017
um 15:10 Uhr:

That was it. 👍
I need to get used to compiling python code I suppose!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/andialbrecht/runsqlrun/issues/15#issuecomment-274811993,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABkIFOQyhbXrBf3wHAX6GM4CAlh0CoTks5rVgYzgaJpZM4GnRyF
.


Fetch as Google & images Temporarily unreachable

Hi everyone,
I have a problem with Google that indicates that all images optimized with pagespeed are "Temporarily unreachable".
This is a screenshot 👇

👉 This is a picture: picture
and this is the response header:
HTTP/1.1 200 OK
Server: nginx
Content-Type: image/jpeg
Content-Length: 162041
Connection: close
Link: <https://img.petyoo.it/blog/post/gatti/cibo_fatto_in_casa/carne_macinata_uova.jpg>; rel="canonical"
Last-Modified: Wed, 25 Apr 2018 13:47:47 GMT
Date: Wed, 20 Jun 2018 14:53:51 GMT
Expires: Thu, 20 Jun 2019 14:53:51 GMT
Cache-Control: max-age=31536000
ETag: W/"0"
X-Original-Content-Length: 162041
X-Page-Speed: Powered by ngx_pagespeed

What can it be? I have seen that other sites with pagespeed also have this problem. You can also test with the Google Mobile-Friendly tool Test

UPDATE: I forgot to mention the version I used: nginx/1.13.9 and ngx_pagespeed v<IP_ADDRESS>-stable

Hi
I don´t know why google bot had problem to fecht you resources, but I think is not pagespeed related.
I you try this no files are pagespeed rewrited and the error remains.
Not only images, css and js files too.
Have you tried to change the robots.txt?
I see in your host https://img.petyoo.it/ the robost.txt contain an Allow:/ directive, try to change it to Disallow:

Hi @Lofesa ,
I thank you for your reply but the directive in the robots.txt file is correct.

Hi @ivanionut
I know that the Allow directive is valid...... but is not a standar and some bots don´t understand it.
In goole terms Allows is "....This is used to override Disallow to allow crawling of a subdirectory or page in a disallowed directory...."
And, well, google has a strange way to do thinks. My last experience is that the json+ld schema.org code for Blog (After moths of be valid) is not valid anymore for amp pages.
Images are not crawled by bots but yes for browser user agent or Pagespeed Insigths user agent.... seems the problem is only related to bots.

@Lofesa,
in my opinion you're a bit confused...
User-agent: *
Disallow: /

The "Disallow: /" tells the robot that it should not visit any pages on the site.
Look, I also performed a test of robots.txt directly from the GSC (google search console):

I know, I know..... but is not "Disallow: / ", I talk about "Disallow:" w/o the slash, as you can see in the references I posted

ok, I understand. but robots.txt is ok as it is.

I also have temporarily unreachable errors in "fetch as google" and I have disallow: in my robots.txt. The robots.txt is no the reason for that.


Visual Glitch in iOS 10

Hi guys
After migrating my project to Xcode 8 and Swift 2.3 (not 3.0)
I have visual glitches when scrolling. Like overlaps in several cells

Hi Frank,
I assume you are using the 1.1.1 version, am I right?

Yep!

So, should I definitely move to swift 3.0 version? :/

@frankgumeta Hi Frank, I'm terribly sorry, but I didn't have time to look on this issue but moving to swift 3.0 is reasonable :)

I updated to Swift 3.0, just to leave the solution here. Is not related to Swift Version. Is related to UICollectionView Prefetch!!!!
If someone experience this issue just needs to disable prefetching in the UICollectionView
See you later!


Provide aliases for patchES300

Current approach is verbose and error prone
ctx.pipeline({
   vert: parser.patchES300("..."),
   frag: parser.patchES300("...", "fragment")
})

The alternative approach is much more readable
ctx.pipeline({
   vert: parser.patchVert("..."),
   frag: parser.patchFrag("...")
})

Downsides:

assumes glsl (could be wglsl one day)
function name doesn't imply that it is for webgl2/ES300

But for comparision i don't think this is better
ctx.pipeline({
   vert: parser.patchVertES300("..."),
   frag: parser.patchFragES300("...")
})

Maybe it should be the parser that is typed?
import { parser } from "pex-shaders";

to
import { es3000Parser as parser } from "pex-shaders";

then you could switch language by switching import (in my dreams..)

One could as why do we even have to bother and why pex-context is not auto-upgrading it for us?

Another option build a online tool when we can upgrade our code by copy paste and stop using parser.

I think parser.patchES300 is purposely verbose as it should be a relic from the past by now and we are willing to remove it entirely.
The current solution (as of Jan 2024) is:

provide a shader written in GLSL 1.00 (WebGL 1 only) with output.vert/frag chunks that allows the shader to be used seamlessly with GLSL version 3 (WebGL2) https://github.com/pex-gl/pex-shaders/blob/6e2cd8fe97fbda9331e0e8f1b2e6ef6275468f45/shaders/chunks/output.glsl.js#L7-L13
use parser.build(ctx, src, defines = [], extensions = {}) to add #version, defines and extension

That works beautifully in pex-renderer v4 alpha and has the following pros:

one source of truth, write shader once
familiarity of attribute/varying
no string replace except adding defines/extension in parser.build (which means greater performance improvement when diffing shader code): what you write is what you get
MRT (WebGL2 multiple color attachments) made easy with define (again perf improvement by reducing GPU memory pressure) https://github.com/pex-gl/pex-shaders/blob/6e2cd8fe97fbda9331e0e8f1b2e6ef6275468f45/shaders/chunks/output.glsl.js#L56-L58
easy to see what code is there for #if (VERSION >= 300) and check new https://github.com/pex-gl/pex-shaders/blob/6e2cd8fe97fbda9331e0e8f1b2e6ef6275468f45/shaders/chunks/output.glsl.js#L27-L36

And cons:

written in "old school" language (although so many more snippets out there are written in 1 and not 3), not all features (3d textures, transform feedback etc.)
shaders code slightly bigger (although we use template strings so not a biggy)
WGSL doesn't support #define (at least last time I checked)

That's for current status.
We reached the conclusion that –now that Safari caught up– WebGL 2 has enough worldwide support so we should entirely drop WebGL1. In a soonish future that means:

ditching all the tricks above (parser output chunks, VERSION checks...)
rewrite shaders to use in/out instead of attribute/varying and only use that syntax
rely on non WebGL fallback if any device still doesn't support WebGL2
enjoy

PS: I still don't see any obvious GLSL -> WGSL bridge without DSL extracting shader variables and providing different body like I did in dgel:
https://github.com/dmnsgn/dgel/blob/b3a213d636a3327eb80a89a93344a4f852c25db2/examples/cube.js#L159-L194

Ok. I missed that defines SOTA. It kind of brings back the question of Shift from framework to library #9 as those defines are in final shaders that could be in pex-renderer instead of here.
Re. glsl/wgsl we could got further into thi.ng/umbrella and copy ThreeJS approach with DSL in JS/TS.  https://github.com/thi-ng/umbrella/tree/develop/packages/shader-ast. There is no WGSL front end yet, only glsl.
const sinc = (x: FloatTerm, k: FloatTerm) =>
    div(sin(mul(x,k)), mul(x, k));

Not sure how does it feel to write whole PBR shader in this though... Seems like 6 months project.

Another approach for WGSL, without DSL, that is gaining traction is WESL (for imports, conditional, etc.).
Good summary of challenges and approaches here.


(Not a request) I READ THE CHANGELOGS!

I don't want to make a Twitch account so I don't know a better means of providing positive feedback: Just need to make sure you know at least one person reads and very much appreciates changelogs (and finds it incredibly annoying when mods don't provide them, such as TeamCoFH mods...).
Just started using this mod last night and individually activated almost every feature, thanks!

I've received at least 4 other messages that people read the changelogs, so you're definitely not alone. :P
Thanks, I'm glad you like the mod!

I've received at least 4 other messages that people read the changelogs, so you're definitely not alone. :P
Thanks, I'm glad you like the mod!


Add Prosit RT dataset

Make the Prosit retention time dataset into a tensorflow dataset, available in aiproteomics.rt.datasets

We are no longer using tfds, and will start from .msp format (instead of the raw model training set)


semantic_fields_for + :time bug

When using f.semantic_fields_for and a :time attribute there is no name attibute in that input.
Example:
<%= f.semantic_fields_for :schedules do |schedule| %>
    <%= render :partial => 'schedule_fields',  :locals => { :f => schedule} %>
<% end %>

in my partial:
<%= f.input :start ,       :as => :time %>
the result is a input with no name

This should be okay now. Closing it.


[VS Code] Joining collaboration session: The specified value isn’t a valid Live Share URL. Please check the link provided by the host and try again.

%0A%0A%23%23 Error:%0AJoining collaboration session: The specified value isn’t a valid Live Share URL. Please check the link provided by the host and try again.%0A%0A%23%23 Steps to Reproduce:%0A1.%0A2.%0A%0A||Version Data|%0A|-:|:-|%0A|**extensionName**|VSLS|%0A|**extensionVersion**|1.0.337|%0A|**protocolVersion**|2.2|%0A|**applicationName**|VSCode|%0A|**applicationVersion**|1.35.0|%0A|**platformName**|Windows|%0A|**platformVersion**|10.0.17134|

Hi @ceencee ,  thanks for reporting this. Would you give us some more info for investigation. Did you think this was a copy paste failure, did you joined the session when you tried again ? and how the host send you the link? ( via e-mail, messenger ect.) Also , your logs may provide us with helpful information. You can access them via the "Live Share: Export Logs" command from the command palette (ctrl + shift + p) and attach them to the issue. I am consolidating the issue with #2179 . Please attach your logs there in #2179 and help us to investigate.


cluster: fix double "/"

It’s “addr//v1” right now. Not sure how it works. But it should be
fixed.

lgtm

@etcd-bot retest this please


Install Rancher in air-gap env by using helm install

What kind of request is this (question/bug/enhancement/feature request):
enhancement
Helm 3 supports installing a local package chart: https://helm.sh/docs/helm/helm_install
We need to validate that we can use helm install to install Rancher in the air-gap environment, and update the documentation if it works.

This was updated elsewhere.


Allow user to signin with username or email

I thought this was already resolved by one of the pull requests. Remember to always reference the issue id.


[Bug] wave 0.19.0 theme color unexpected changes upon client reload

0.19.0
Actual behavior
theme color is inconsistent. The yellow/orange turns into black when the client is reloaded. Noticed this with ui.separator(), ui.progress_bar()
Sometimes, textbox color and labels for other ui elements also become black.
Expected behavior
yellow/orange temporarily changes to black, which is erroneous
Steps To Reproduce

make a progress bar, use theme as h2o-dark for example
when the page is loaded, the progress bar may be black
(but when navigating through the app, not a reload, the color will be back to normal)

@azim-b Could you please provide an isolated min repro code?
Tried with this one, but it seems to work fine:
from h2o_wave import main, app, Q, ui

@app('/demo')
async def serve(q: Q):
    q.page['meta'] = ui.meta_card(box='', theme='h2o-dark')
    q.page['example'] = ui.form_card(
        box='1 1 4 10',
        items=[
            ui.progress(label='Indeterminate Progress', caption='Goes on forever'),
            ui.progress(label='Standard Progress', caption='Downloading the interwebs...', value=0.25),
        ]
    )
    await q.page.save()

Hi @mturoci ,
I tried the same code snippet and still see the issue. It seems to happen only in Safari though. I checked on Firefox and Chrome and the colors look correct.
Here's a screen capture of the app on Safari for the code snippet above:
https://user-images.githubusercontent.com/64787868/141656806-742dca39-2067-4f49-961a-c66b7e4cc5c3.mov

Thanks @azim-b, what Safari version do you use?

Version 14.0.3 (166<IP_ADDRESS>.7)

Hi @azim-b, I just tested on Safari, but on a newer version and it seems to work.

@aalencar did you have to reload? Reloading the client (browser) seems to fix the problem for me. This only occurs when the client is first initialized.

@azim-b I didn't have to reload for it to work. Be aware that I'm running a newer version of safari though.

I tested it on Safari 12. Couldn't repro my minimal example code. I could repro it within your app @azim-b back when you first reported the bug, but I cannot reproduce it anymore.
@azim-b could you please open the browser console in your Safari and check if there are any errors when you reproduce?

Closing as unreproducible, feel free to reopen if it appears again.


Exercise 5.21

https://github.com/frank67/Cpp-Primer/blob/master/ch05/ex5_21.cpp
You wrote:
    while ( cin >> read )
        if (read == tmp && isupper(read[0])) break;
        else tmp = read;

well this isn't properly a bug, but in this exercize the continue statement is supposed to be used but is missing in your code, therefore I suggest to apply this change:
        while ( cin >> read )   {
                if (read == tmp)        {
                        if(isupper(read[0])) break;
                        else continue;
                }
        tmp = read;
        }

otherwise you could consider to rewrite it without the tricky cin.eof() usage:
#include <iostream>
#include <string>

using std::cout; using std::cin; using std::flush; using std::endl; using std::string;

int main()
{
        string sword, sprev;
        cout << "Please enter some strings: " << flush;
        while(cin >> sword)     {
                if(sprev == sword)      {
                        if(isupper(sword[0]))   {
                                cout << "The word " << sword << " appears two times consecutively." << endl;
                                sprev="";
                                break;
                        } else
                                continue;
                }
                sprev=sword;
        }
        if(!sprev.empty())
                cout << "There aren't words that appear two times consecutively." << endl;

        return 0;
}

HTH
Best regards

@frank67
Agree with you. Only one little problem: when sword is empty, sword[0] is UB.'' I have improved your code as following:
#include <iostream>
using std::cin; using std::cout; using std::endl;
#include <string>
using std::string;

int main()
{
    string str_read, str_prev;
    while ( cin >> str_read && !str_read.empty() ) {
        if ( !isupper(str_read[0]) ) {
            str_read = "";
            continue;
        }
        else if ( str_prev == str_read ) {
            cout << str_read << " occurs twice in succession." << endl;
            str_read = "";
            break;
        }
        else str_prev = str_read;
    }
    if ( str_prev == str_read )
        cout << "no word was repeated." << endl;
}

Any question?

Yes, I'm not able to catch the possible UB, I tried this code:
        while(cin >> sword)
                if(sword.empty())
                        cout << "empty" << endl;

"empty" was never printed, how can I do that?
Thanks for any answer!
For the rest your code had trouble with this input S s it gives no output, I suggest to rewrite it this way:
#include <iostream>
using std::cin; using std::cout; using std::endl;
#include <string>
using std::string;

int main()
{
        string str_read, str_prev;
        while ( cin >> str_read && !str_read.empty() ) {
                if ( !isupper(str_read[0]) )
                        continue;
                else if ( str_prev == str_read ) {
                        cout << str_read << " occurs twice in succession." << endl;
                        str_read="";
                        break;
                        }
                        else str_prev = str_read;
        }
        if( !str_read.empty() )
                cout << "no word was repeated." << endl;

        return 0;
}

Best regards

"empty" was never printed, how can I do that?

Enter EOF(Ctrl + Z in windows) directly.

For the rest your code had trouble with this input S s it gives no output,

What? Impossible...

Input: S s then EOF.
Output: There aren't words that appear two times consecutively.

Could you please check again?

Looking at the output I guess you'compiled my code ROFL, test your code.

@frank67
:flushed: I solemnly apologize...

"empty" was never printed, how can I do that?

You are exactly right.  && !str_read.empty() is here superfluous...

@frank67
Could you please send a pull request? Or I just use your code in place of the origin.


reduce sidebar for more space for samples table

@sadelokiki @cyrielo isn't this a repro of 866

Yes! Closing this because its a duplicate of #866 and the corresponding PR is #977


Add error-handling to generated code.

Prerequisite to Issue #7 .

Generally addressed in this commit and the one preceding it.

Verified with unit tests in this commit.


Music files with very large cover art cause OutOfMemoryError exception

If a music file has very large cover art, (in my test case, 7.8MB), calls to BitmapFactory.decodeStream() throw a java.lang.OutOfMemoryError exception. Current implementation catches this exception, but halts playback.
(working on fixing this)

Fixed by specifically catching the OutOfMemoryError exception when calling decodeStream() and in that case setting the Cover to null instead of halting playback.


Replace Airbnb linting with prettier

Airbnb code guide is great, but developer's job is to focus on code, not on code styling. Prettier "just works" and produces nicely formatted code without forcing you to think about formatting.

@arvigeus I think Airbnb's linter provides more than just styling rules, it also checks for possible mistakes. Prettier in the other hand, only cares about how your code looks.
So IMO they can both live together, maybe some of the eslint rules for code-style could be disabled then.

I agree with @lucasmotta. I think there's a huge benefit to enforcing linting during the code process, not just afterwards. In bigger teams, it keeps everyone on the same page. In solo teams, it can pick up on obvious goofs like missing commas or uninitialised vars. The eslinter provided also dynamically checks propTypes on components and highlights if something is declared but not used, or vice versa. IMO, these are all invaluable coding aids that I would like to keep.
As for prettier integration, where in the workflow do you think it should fit in? Since prettier outputs a uniform format, it's just as easy (and possibly more useful) to enable it in Atom or another IDE and get prettify-on-save functionality. I could add it as a step to Webpack to prettify on bundle, but then that only happens when you run a corresponding npm... command -- arguably, the job of the editor you're writing it in. And there's no sense prettifying bundled code, because that's all minified for production anyway, so bolting it on to the webpack flow probably isn't that useful.
I'll keep this open to thoughts for now.
Somewhat related to #19.

Airbnb and Prettier disagree on some styling rules - for example 'string' vs "string" and can't be used together (or at least not in their current form)
As for fixing possible mistakes, I think prettier does that. Not totally sure, @lucasmotta, can you provide and example? This could be somehow related to #19, because CRA uses prettier by default and probably eslint-config-react-app contains those checks as well.
As for where prettier could fit, maybe as a pre-commit step.

There's this that might be useful.
Picking up code goofs and conforming code to a style would be perfect.

I want to create a 'slim' version of the repo shortly that would not have any opinions as far as linting and stylistic choices, so it'd be up to the developer to add in things like prettier.
I'll close this for now.


freqz: suboptimal performance for worN=2**n+1, include_nyquist=True and real coefficients

Describe your issue.
freqz can be way faster, when the coefficients are real, worN=2**n+1 and include_nyquist=True. In this case rfft can be used directly.
The attached code illustrates this, I hope.
Thanks for the great work!
Reproducing Code Example
import numpy as np
from scipy.signal import freqz
import timeit

N = 128

for n in range(7):
    worN = 2**n*N//2+1
    print('worN = {}'.format(worN))

    h = np.random.rand(N)*2-1

    f0 = lambda: freqz(h, worN=worN, include_nyquist=True)[1]
    f1 = lambda: np.fft.rfft(h, n=2*(worN-1))

    H0 = f0()
    H1 = f1()

    err = np.sum(np.abs(H0-H1)**2)/np.sum(np.abs(H0)**2)
    print('error = {}'.format(err))

    t0 = timeit.timeit(f0, number=10000)
    t1 = timeit.timeit(f1, number=10000)
    print('speedup = {}\n'.format(t0/t1))

Error message
worN = 65
error = 7.582436736470968e-29
speedup =<PHONE_NUMBER>968968

worN = 129
error = 1.2033665898383146e-28
speedup = 62.96460648165139

worN = 257
error = 6.748460212611348e-29
speedup = 54.55989853561076

worN = 513
error = 6.378075125203393e-29
speedup = 47.368120583063835

worN = 1025
error = 8.559451508895459e-29
speedup = 38.34191925764777

worN = 2049
error = 7.356556838100076e-29
speedup = 29.080013414218744

worN = 4097
error = 8.396001469974994e-29
speedup = 22.058802358213786

SciPy/NumPy/Python version information
1.7.3 1.21.4 sys.version_info(major=3, minor=10, micro=1, releaselevel='final', serial=0)

Hi @drestebon, thanks for reporting. Looking at the documentation, there are some rules to observe so that (R)FFT is used. From what I see in your example, worN's shape is not good. I did not check yet the code as to why it would be needed. Did you have a look? Are you suggesting something else?

Hi @tupui, thanks for your reply.
It seems like the rules implemented to use (R)FFT do not reflect the case I'm referring to.
I made an attempt at this. You can see it here:
https://github.com/drestebon/scipy/blob/f4c9c947b2220f52415fff349a79147ec6afc4ab/scipy/signal/_filter_design.py#L445
My feeling is that the way I did it is a bit hackish and clumsy, but it shows what is required: n_fft for this case is 2*(N-1).
It still looks like numpy's fft does a better job though.
worN = 65
error = 5.63413763959627e-29
speedup =<PHONE_NUMBER>9708475

worN = 129
error = 4.657502702331312e-32
speedup = 11.959434175189896

worN = 257
error = 4.2029261309019205e-32
speedup = 9.685627976708963

worN = 513
error = 4.568295653285404e-32
speedup = 7.261198177906964

worN = 1025
error = 3.702060680434987e-32
speedup = 4.869401531950861

worN = 2049
error = 3.6579413212381833e-32
speedup = 3.173570125199443

worN = 4097
error = 4.7836664035146534e-32
speedup = 2.060847729547769

When worN is small, it is faster, but the error looks big. We should be careful for backward compatibility.
worN = 3
error = 1.9374009635386986
speedup = 21.30843158223846

worN = 5
error = 0.18741190054056306
speedup = 16.77478100125656

worN = 9
error = 8.612108265284007e-32
speedup = 15.351992009731502

worN = 17
error = 5.22888645839279e-31
speedup = 13.55208721451381

worN = 33
error = 0.10942613312814237
speedup = 12.463569226773217

worN = 65
error = 3.79794227671082e-31
speedup = 14.018722794359956

If we use rfft with 2*(worN-1) < b.shape[0], it means we are truncating the impulse response, which causes the error. My guess would be, that the user wouldn't want freqz to make any approximations.
In my branch I included the condition 2*(N-1) >= b.shape[0] to reflect this.

I ran a bit more exhaustive benchmarks and it seems the rfft approach is faster than the polyval for any value of worN.
For these simulations I introduced a force_poly argument, to skip the fft-related optimizations in freqz.

The code:
import matplotlib.pyplot as plt
import numpy as np
import scipy
from scipy.signal import freqz
import timeit

from scipy import fft as sp_fft

print(scipy.__file__)

fig = plt.figure(1, figsize=(19, 12))
fig.clf()
ax = fig.subplots(3,4)

Ntop = 20

# for Ntc, N in enumerate([64, 128, 256, 512]):
for Ntc, N in enumerate([4, 8, 16, 32]):

    NN = np.arange(N//2+1, Ntop*N//2+1)
    speedup = np.zeros((len(NN),))
    error   = np.zeros((len(NN),))
    t0   = np.zeros((len(NN),))
    t1   = np.zeros((len(NN),))

    for tc, n in enumerate(NN):
        # worN = 2**n*N//2+1
        worN = n

        h = np.random.rand(N)*2-1

        f0 = lambda: freqz(h, worN=worN, include_nyquist=True, force_poly=True)[1]
        f1 = lambda: sp_fft.rfft(h, n=2*(worN-1), axis=0)
        # f1 = lambda: np.fft.rfft(h, n=2*(worN-1))

        H0 = f0()
        H1 = f1()

        error[tc] = np.sum(np.abs(H0-H1)**2)/np.sum(np.abs(H0)**2)

        t0[tc] = timeit.timeit(f0, number=200)
        t1[tc] = timeit.timeit(f1, number=200)
        speedup[tc] = t0[tc]/t1[tc]

        print('N = {}\tworN = {:5d}\terror = {:.10e}\tspeedup = {:.5f}'.format(N, worN, error[tc], speedup[tc]))

    ax[0,Ntc].semilogy(NN, error)
    ax[1,Ntc].semilogy(NN, speedup)
    ax[1,Ntc].hlines(1, NN[0], NN[-1], color='k', ls=':')
    ax[2,Ntc].semilogy(NN, t0, label='freqz{polyval}')
    ax[2,Ntc].semilogy(NN, t1, label='rfft')
    ax[0,Ntc].set_xlabel('worN')
    ax[1,Ntc].set_xlabel('worN')
    ax[2,Ntc].set_xlabel('worN')
    ax[0,Ntc].set_ylabel('error')
    ax[1,Ntc].set_ylabel('speedup')
    ax[2,Ntc].set_ylabel('time')
    ax[2,Ntc].legend()
    ax[0,Ntc].set_title('sig.freqz{{polyval}} v/s scipy.fft.rfft\nlen(b)={}'.format(N))

fig.savefig('rfft_speedup.png')

Thanks you for the benchmark @drestebon. It looks like we could benefit for another condition to benefit from this. @ilayn does this look right to you?

People,
I was trying to get fairer benchmark, that included the fixed costs of sig.freqz and noted a much more concerning issue. For the new benchmark, the first thing I wanted to do was to check that, when freqz uses rfft, it does the same operation I think it should be doing (an fft of a given length), so that I can substract the rfft-calculation-time and get the overhead in freqz. To my surprise, freqz is not doing the rfft at the length I was expecting.
I think the way freqz is currently using the rfft is wrong!!!
In the example below I am calculating the frequency response of H(z) = z^-8 on an evenly spaced grid of 64 points between DC and Nyquist (0 and pi included). For rfft to produce the correct grid, I need to apply it on a window of 126 samples, i.e. 2*(worN-1). freqz, however, uses a window of 128 samples, which produces an evenly spaced grid of 65 points!!
This error should be very easy to catch, since using the wrong fft length will give a result of a different length than what we are asking for (worN). Alas! in freqz we are truncating the result of the fft here: h = fft_func(b, n=n_fft, axis=0)[:N], masking the error.
For sanity, in the example I am calculating the frequency response analytically, using freqz, with an rfft and forcing freqz to use polyval. As you can see in the plot, the two methods currently implemented in freqz give very different results! The polyval being the correct one (to the best of my knowledge).
I think this is a separate issue from what we have been discussing in this thread. Is the use to open another issue, or should we open the scope of this issue?

Script for reproducing the plot:
import numpy as np
import matplotlib.pyplot as plt

import scipy
import scipy.signal as sig
from scipy import fft as sp_fft

print(scipy.__file__)

has_freqz_debug = False

worN = 64
N = 8

# impulse response for H(z) = z**-7
h = np.zeros(N)
h[-1] = 1

fig = plt.figure(1, figsize=[21,  6])
plt.clf()

if has_freqz_debug:
    # default behaviour
    foo = {'debug' : True}
    w, H = sig.freqz(h, worN=worN, include_nyquist=True, freqz_debug=foo)

    plt.plot(np.real(H), label='freqz')
    plt.plot(np.imag(H), label='freqz')

    print(foo)

    # Forced poly
    w, Hp = sig.freqz(h, worN=worN, include_nyquist=True, force_poly=True)

    plt.plot(np.real(Hp), '.', lw=3, label='freqz force_poly=True re')
    plt.plot(np.imag(Hp), '.', lw=3, label='freqz force_poly=True im')
else:
    # default behaviour
    w, H = sig.freqz(h, worN=worN, include_nyquist=True)

    plt.plot(np.real(H), label='freqz')
    plt.plot(np.imag(H), label='freqz')

# Analyic
zm1 = np.exp(-1j * w)
Ha = zm1**(N-1)

plt.plot(np.real(Ha), lw=1, label='analytic re')
plt.plot(np.imag(Ha), lw=1, label='analytic im')

# rfft
nrfft = 2*(worN-1)
Hrfft = sp_fft.rfft(h, n=nrfft, axis=0)
plt.plot(np.real(Hrfft),  '--', lw=2, label='rfft re')
plt.plot(np.imag(Hrfft),  '--', lw=2, label='rfft im')
print('nrfft = {}'.format(nrfft))

# error
plt.plot(np.real(Ha - H),  lw=1, label='Error: analytic - freqz re')
plt.plot(np.imag(Ha - H),  lw=1, label='Error: analytic - freqz im')

plt.legend()

plt.savefig('freqz rfft grid error.png')

plt.show()

Output of the script (when I set has_freqz_debug=True):
/home/e/src/scipy/scipy/__init__.py
{'debug': True, 'n_fft': 128, 'is_fft': True}
nrfft = 126

Sorry for yet another very long comment.
I did some benchmarking using the fix in the commit above.
I'm measuring the execution time of the poly and (r)fft implementations. The plots below show the error (difference between both approaches) and the speed improvement of the later as a function of worN and of n_fft. The column on the left corresponds to randomized impulse responses of length n_fft. The column on the right corresponds to impulse responses where len(b) << n_fft, which means the fft will do some zero-padding. They also include an allpass component. The purpose of this IR is to show the limitation of the current implementation (mentioned earlier).
Things to note:

The fix removes the spikes in the error that appear on the right column
The fix improves the execution time across the board, except for small values of n_fft. An ideal solution would include a threshold on n_fft. I tried the fix on two very different machines (see below) and the results look equivalent. A conservative threshold would be, for example, that the (r)fft is applied only for n_fft >= 32.
The point on the n_fft axis where the (r)fft improves the execution time increases as the amount of zero-padding increases: for a small number of coefficients, the poly approach is faster than (r)fft, and improvements only appear for very long values of n_fft. An ideal solution would have a threshold on n_fft that is a function of n_fft / len(b). This would require a bit more benchmarking that I'd be happy to do.

So far, the fix boils down to correctly calculating n_fft for include_nyquist=True and removing the constraint sp_fft.next_fast_len(N) == N (see the commit mentioned above).
Benchmark for the current implementation:

Benchmark for the fix:

Both these benchmarks were carried out on this computer:
model name	: Intel(R) Core(TM) i5-7260U CPU @ 2.20GHz
MemTotal:       32763848 kB

Run on a bigger computer
Further results were obtained on this computer:
model name      : Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
MemTotal:       197564584 kB

Benchmark for the current implementation

Benchmark for the fix

script

Click to expand!
import sys

if len(sys.argv) < 2:
    print('Hey! give a name to the run with an argument')
    sys.exit()

import matplotlib.pyplot as plt

import numpy as np
import numpy.random

import scipy
from scipy.signal import freqz
from scipy import fft as sp_fft

import timeit

print(scipy.__file__)

fig = plt.figure(1, figsize=(12, 12))
fig.clf()
ax = fig.subplots(3,2)

attempts = 256

NN      = np.arange(4, 256)
speedup = np.zeros((len(NN),))
error   = np.zeros((len(NN),))
t0      = np.zeros((len(NN),))
t1      = np.zeros((len(NN),))

rng = np.random.RandomState(0)

for zp in [False, True]:
    for cplx in [False, True]:
        for whole in [False, True]:
            for include_nyquist in [False, True]:
                for tc, n in enumerate(NN):
                    worN = n

                    N = n if whole else 2*(n - 1) if include_nyquist else 2*n

                    # Bad things happen with zero padding (len(impulse response) << n_fft)
                    if zp:
                        N = N//2

                    h = rng.randn(N)
                    if cplx:
                        h = h + 1.0j*rng.randn(N)

                    # Bad things happen with a wide band response
                    if zp:
                        k = min(4, N-1)
                        h[k] = h[k]+10.0 # mala leche

                    ww, Hf = freqz(h, worN=worN, include_nyquist=include_nyquist, whole=whole)
                    f0 = lambda: freqz(h, worN=ww)[1] # poly
                    f1 = lambda: freqz(h, worN=worN, include_nyquist=include_nyquist, whole=whole)[1]

                    H0 = f0()
                    H1 = f1()

                    error[tc] = np.sum(np.abs(H0-H1)**2)/np.sum(np.abs(H0)**2)

                    t0[tc] = timeit.timeit(f0, number=attempts)
                    t1[tc] = timeit.timeit(f1, number=attempts)
                    speedup[tc] = t0[tc]/t1[tc]

                    print('{}{}{}{} N = {}\tworN = {:5d}\terror = {:.10e}\tspeedup = {:.5f}'.format(
                           int(zp), int(cplx), int(whole), int(include_nyquist), N, worN, error[tc], speedup[tc]))

                lbl = []
                if cplx:
                    lbl.append('c')
                if whole:
                    lbl.append('wh')
                if include_nyquist:
                    lbl.append('ny')
                if not lbl:
                    lbl.append('-')

                Nfft = NN if whole else 2*(NN - 1) if include_nyquist else 2*NN
                ax[0, int(zp)].semilogy(NN , error   , label=' '.join(lbl))
                ax[1, int(zp)].semilogy(NN , speedup , label=' '.join(lbl))
                ax[2, int(zp)].semilogy(Nfft , speedup , label=' '.join(lbl))

title = ['sig.freqz[poly] v/s sig.freqz[(r)fft]',
         '$\\bf{c}$plx in [False, True]',
         '$\\bf{wh}$ole in [False, True]',
         'include_$\\bf{ny}$quist in [False, True]']

for k in range(2):
    ax[0, k].set_title('\n'.join(title+['zero_padding='+str(bool(k))]))
    ax[0, k].set_ylabel('error')
    ax[0, k].set_xlabel('worN')
    ax[0, k].legend()
    ax[1, k].hlines(1, NN[0], NN[-1], color='k', ls=':')
    ax[1, k].set_xlabel('worN')
    ax[1, k].set_ylabel('speedup')
    ax[1, k].legend()
    ax[2, k].hlines(1, NN[0], 2*NN[-1], color='k', ls=':')
    ax[2, k].set_xlabel('n_fft')
    ax[2, k].set_ylabel('speedup')
    ax[2, k].legend()

fig.savefig('freq_fft_benchmark_{}.png'.format(sys.argv[-1]))

plt.show()

import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)


Some consumption show empty but not show consumption fuel

Hi. i Have boler EcoCompact VSC206 4-5 90 and now i see in your integration that for my boiler no show fule/gas consuption and all consuption is show for energy. But some show as 0 , and two show for heating( 48,125 Wh and 3100 Wh) but other  which show 0 is Domestic Hot Water. Someting is no good and no correct.

Does the myVAILLANT app show the same thing, or does it look different?

Yesterday in myVaillant app show that i must wait 48hours to collect data, but today this info is not show and show only temp power zone and pressure and show info about my boiler.

and now in HA show more

look
How my consuption look:

Seems fine? These are daily totals that increase from 0 to whatever you use on that day. Add it to your energy dashboard, if you'd like to see totals over weeks or months.


Function Test: Graph formatting and corner case test

Creating the log to document the progress of writing test functions.

The test function has been added. It is open for edit and there may be updates in the future

Looks good as per the merge good stuff, looks like it's as good as a plot test can be. Good stuff! Maybe in the future it would be cool to look at that method Tiff mentioned about pixel comparisons.


Additional Status (Braiins / Bosminer) Tuner & Pool

Describe the solution you'd like
It would be nice to see in Home Assistant along side each of the respectively hash cards other details like the Tuner Status. perhaps Pool status
Describe alternatives you've considered
I believe the option is available in pyasic
new versions of BraiinsOS+ (post-gRPC)
Source code  pyasic/miners/backends/braiins_os.py
if rpc_tunerstatus is not None:
            errors = []
            try:
                chain_status = rpc_tunerstatus["TUNERSTATUS"][0]["TunerChainStatus"]
                if chain_status and len(chain_status) > 0:
                    offset = (
                        6 if int(chain_status[0]["HashchainIndex"]) in [6, 7, 8] else 0
                    )

                    for board in chain_status:
                        _id = board["HashchainIndex"] - offset
                        if board["Status"] not in [
                            "Stable",
                            "Testing performance profile",
                            "Tuning individual chips",
                        ]:
                            _error = board["Status"].split(" {")[0]
                            _error = _error[0].lower() + _error[1:]
                            errors.append(BraiinsOSError(f"Slot {_id} {_error}"))
                return errors

old versions of BraiinsOS+ (pre-gRPC)
Source code in pyasic/miners/backends/braiins_os.py
if rpc_tunerstatus is not None:
            errors = []
            try:
                chain_status = rpc_tunerstatus["TUNERSTATUS"][0]["TunerChainStatus"]
                if chain_status and len(chain_status) > 0:
                    offset = (
                        6 if int(chain_status[0]["HashchainIndex"]) in [6, 7, 8] else 0
                    )

                    for board in chain_status:
                        _id = board["HashchainIndex"] - offset
                        if board["Status"] not in [
                            "Stable",
                            "Testing performance profile",
                            "Tuning individual chips",
                        ]:
                            _error = board["Status"].split(" {")[0]
                            _error = _error[0].lower() + _error[1:]
                            errors.append(BraiinsOSError(f"Slot {_id} {_error}"))
                return errors

linux cli.
echo -n "stats" | nc IP_OF_MINER_GOES_HERE 4028
and
echo -n "pools" | nc IP_OF_MINER_GOES_HERE 4028
Additional context
Thanks. Really enjoy being able to automate with the currently available triggers. Great granular options would be enabled with some additional visibility.

Yep, errors are available via `pyasic' by default, and are in the call that we are using.  Here's the list of available data for reference - https://docs.pyasic.org/en/latest/data/miner_data/
The biggest issue with this is that I'm not sure the best way to display text based data on the HASS interface.  You can imagine that it could get clunky, and it's not feasible to create separate entities for each "error" as they happen.  I would love to understand why you want the error codes in this case, as I believe it would be much easier to automate something based off board hashrate which is already available, but there may be a use case I'm missing here.


Error when running lighthouse

This is what happens. I'm not sure if I did anything incorrectly
$ lighthouse | sh
Couldn't execute file.
Failed to spawn piped process.
Error in spawned cmd.

That means you probably didn't run make config or lighthouse doesn't have permissions to run the file refered to by cmd in your lighthouserc.  If you believe the config file exists (~/.config/lighthouse/lighthouserc), try chmod +x ~/.config/lighthouse/cmd

Ah yes. cmd had to be executable, thanks!


Contributing a high-level PAC function

Hey folks - I've got a PR right now in the MNE-sandbox repo, which adds a high-level function to do a bunch of PAC stuff between signals. It lets you:

Give an input that is shape (n_signals x time), and specify pairs of indices that you want to calculate PAC between.
Give a list of phase frequencies and amplitude frequencies to calculate PAC between.

It will then pre-filter the proper channels only one time per pair of low/high frequencies, so that we don't duplicate filtering etc.
You can also:

Give a list of indices corresponding to events
Give a list of tmin / tmax pairs either relative to event onsets, or to the whole dataset

It will then calculate PAC using the metric of choice for each pair of channel indices, phase/amp frequencies, and tmin/tmax window. With the option of either calculating this per epoch, or concatenating across epochs first.
So this way, you could make PAC plots with this function like this:

Or like this:

Or do a connectivity plot w/ lots of pairs of electrodes.
In the PR we've been discussing whether it would make more sense to contribute this to PacPy, since there's a hidden function that's basically designed to not be specific for MNE objects, but instead operate on arrays. Would you all be interested in this? I'd be willing to take some time porting over the tests and the visualization functions (or whatever subset of that you're interested in). However I know that you guys generally want to keep this repo restricted to just the PAC metrics themselves.
LMKWYT. Pinging @srcole and @parenthetical-e in case you guys have thoughts ;)

I think it'd be great to add to pacpy! This new function would definitely be an improvement to the current 'comodulogram' function that currently repeats filtering and also doesn't have these nice visualizations.

The visualizations are pretty simple to implement, and I don't think they'd need to be part of a PR. The benefit of the function is that it can output PAC as an array of shape (n_pac_metrics, n_epochs, n_channel_pairs, n_frequency_pairs, n_time_windows) so that you can then slice and dice the output however you like :)

True. Yeah, I think this will be good to provide that functionality for people analyzing trial data sets with many simultaneous recordings.

@parenthetical-e what do you think? I believe you were the most hesitant to start adding extra functionality to pacpy (which I think has reasonable arguments behind it). IMO this is a close-enough extension to the functionality that's already there, so doesn't deviate too much from the overall goal of pacpy. What do you think?

I am all for adding the ability to work with trials and 2-d arrays of data, but am very much against the making a monolithic
phase_amplitude_coupling(..., method='plv')

Something like,
from pacpy.pac import plv
trials(lo, hi, f_lo, f_hi, trial_index, pacfn=plv)
channels(X f_lo, f_hi, lo_index, high_index, pacfn=plv) # where X is 2d

(Sorry. Had no intent to close this. I've thumb hands.)

@parenthetical-e I think I'm +1 to that, but let's chat about implementation etc. The reason that I've got all those flags is because I'm assuming that the input to the function is always the same type (channels x time) and it always does the filtering on that data first, before it does any slicing and dicing. That's why I don't have, e.g., an extra function for trials-shaped data. (and why this one is kind of a mammoth)

if you wanna skype and chat about the function etc, then I'm down for that. Only catch is that I'll be out of communication from Thursday to ~Tuesday (I'll be in Yellowstone hanging out with elks and stuff :D )

Let's just chat when you get back. Thursday the 9th? 10:30 AM?

wellllll, I might be leaving for knight lab camping on Thursday the 9th, so
wednesday would be better for me if that works for you.
On Tue, May 31, 2016 at 3:57 PM, Erik<EMAIL_ADDRESS>wrote:

Let's just chat when you get back. Thursday the 9th? 10:30 AM?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/voytekresearch/pacpy/issues/45#issuecomment-222846081,
or mute the thread
https://github.com/notifications/unsubscribe/ABwSHfeb3ZaHWdfOlnuGnFdMYKWitKgWks5qHLzVgaJpZM4Iq3iY
.

Sure. 10 AM ok?

Sounds good to me, Wed June 8th, 10AM.
On Tue, May 31, 2016 at 3:58 PM, Erik<EMAIL_ADDRESS>wrote:

Sure. 10 AM ok?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/voytekresearch/pacpy/issues/45#issuecomment-222846322,
or mute the thread
https://github.com/notifications/unsubscribe/ABwSHbpRcGI128I--E-lJkhHwyMNQaYcks5qHL0sgaJpZM4Iq3iY
.

Works for me too

Cools.

On May 31, 2016, at 4:01 PM, Chris Holdgraf<EMAIL_ADDRESS>wrote:
Sounds good to me, Wed June 8th, 10AM.
On Tue, May 31, 2016 at 3:58 PM, Erik<EMAIL_ADDRESS>wrote:

Sure. 10 AM ok?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/voytekresearch/pacpy/issues/45#issuecomment-222846322,
or mute the thread
https://github.com/notifications/unsubscribe/ABwSHbpRcGI128I--E-lJkhHwyMNQaYcks5qHL0sgaJpZM4Iq3iY
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/voytekresearch/pacpy/issues/45#issuecomment-222846692, or mute the thread https://github.com/notifications/unsubscribe/ABIxh2dqXRSk2r55OMLHJcoyJBxwzjM-ks5qHL20gaJpZM4Iq3iY.

Hey guys - I just realized that I had already opened this. Let's discuss integrating PAC efficiency etc into pacpy here.


[suggestion] Nitrogen chemistry for fertiliser and explosives

Chemical production lines are cool. Here are a couple of suggestions for some new ones.
Fertiliser:
For the source of nitrogen. I tried to think of how to do this without a new machine, but failed.
Step 1: "Gas seperator" converts air into Nitrogen & oxygen
Step 2: Nitrogen + hydrogen in a chemical infuser -> Ammonia gas
Step 3: Ammonia gas in a rotary condensentrator -> Liquid Ammonia
Urea (a common fertiliser) is created by reacting ammonia with carbon dioxide. I think this can be finessed to work with the Pressurized Reaction Chamber.
Step 4: Liquid Ammonia + coal (carbon) + oxygen in a PRC -> "Chemical fertiliser" + water vapour.
"Chemical fertiliser" obviously works like bonemeal for crops, but can't be used for dyeing.
Explosives:
Same as before up to ammonia gas:
Step 3: Ammonia gas + oxygen in a chemical infuser -> Nitric oxide
Step 4: Nitric oxide + oxygen in a chemical infuser -> Nitrogen dioxide
The next step is really two steps in one, creating nitric acid with water and then nitrating an organic compound.
Step 5: Nitrogen dioxide + Water + Substrate in PRC -> explosive pellet + nitric oxide
explosives could be oredict'd with gunpowder, or used for later recipes.

I'm actually trying to begin work on a mod that adds nitrogen related chemistry (and depends on Mekanism) but because of this issue, I cannot: https://github.com/aidancbrady/Mekanism/issues/3896


Update charts and configurations for OTel-Arrow v0.25 with OTel-Contrib v0.107 components

Description
ServiceNow Cloud Observability is recommending OTel-Arrow components and exporters for OpenTelemetry traces data.
This updates the tracesCollector deployment to unconditionally use the OTel-Arrow collector built in this repository instead of the contrib collector.  This means changing to the concurrent batch processor and disabling the sending queue.
Recommended resource values have been updated based on SNCO internal production testing.
How Has This Been Tested?
WIP

R/CC: @AnaMMedina21 @jaronoff97

Tested on a DigitalOcean K8s cluster
✅  otel-cloud-stack
✅  collector-k8s
✅  kube-otel-stack


Basic submission notification

This will email the default admin email when a new form submission comes in, giving the id of the submission, and an estimate of completeness.

:disappointed: those are only failing on Travis. ugh.

Coverage increased (+0.6%) to 80.975% when pulling 680bb1b88760768bd3f24d583913df65b48cc43d on basic_submission_notification into 1f7d486546d08b228fbed90f162c1eb757fdff2a on master.


MakeHuman: Add version 1.1.1

MakeHuman is an open source tool for making 3D characters.
Official homepage of MakeHuman: http://www.makehumancommunity.org/
note: MakeHuman has an official Github repo, but the "release files" in the official Github repo does not contain binary releases (.exe)
Please let me know if there are any problems. Thanks.

There is a checksum file here
"hash": {
    "url": "$baseurl/md5sums.txt"
}

There is a checksum file here
"hash": {
    "url": "$baseurl/md5sums.txt"
}

Thanks!


[Typescript-Resolvers] Add custom DataSources type for apollo resolvers

I'm trying to create resolvers and keep them strictly typed, but I get an error when trying to access the dataSources that are added automatically to the context by Apollo.
I would like to add an option to typescript-resolvers plugin to enhance the Resolvers types by adding a custom DataSource type.
Before:
export type Resolvers<ContextType = ApplicationContext> = ResolversObject<{
  Mutation: MutationResolvers<ContextType>;
  Query: QueryResolvers<ContextType>;
}>;

After:
export type ResolversContext = ApplicationContext & {
  dataSources: ApplicationDataSources
};

export type Resolvers<ContextType = ResolversContext> = ResolversObject<{
  Mutation: MutationResolvers<ContextType>;
  Query: QueryResolvers<ContextType>;
}>;

The ApplicationDataSources type would be provided by the user in the same way that the custom application context is provided.
I will try to put an PR together for this feature.

I am not sure if we need a feature for that specific case. Why do you think contextType is not enough? dataSources is an object that contains some other objects. So this can be easily added into your custom contextType.

Make sense I could send a context type with the dataSources added to it, I'm just worried if it could affect other types/plugins since AFAIU Apollo only add the data sources for the resolvers context.

@victor-accarini It should be fine to use contextType because this option is only used for resolvers. So there is nothing to worry about in my opinion.

Thks, I will close the issue


Deploy a RHEL 8.3 VM on Azure with IHS, Plugin and WCT installed

Description
This PR is to address the following issues:

resolve #28
resolve #29
resolve #20
resolve #26

Change summary

Implement an ARM template to install a RHEL 8.3 VM on Azure along with all other required resources
Implement post scripts to install IHS, Plugin and WCT after VM is provisioned
Implement cloud-init script to do entitlement check and upgrade
Add pipeline to verify ihs VM and generate artifact to create ihs virtual machine offer
Update Azure CLI version to resolve Issue 20
Add additional step in pipeline to verify the installation directory's existence based on difference account entitlement status
Add entitlement check in install.sh to fail fast if the user account is unentitled

Test cases
The following cases have already been verified before sending out the PR:

Successfully deployed a RHEL 8.3 VM on Azure along with other required resource;
IHS, Plugin and WCT are automatically installed after deployment is completed;
Successfully created a private image based on the above VM;
Successfully deployed a RHEL 8.3 VM with IHS, Plugin and WCT pre-installed using the above private image, for an entitled IBMid;
Successfully deployed a RHEL 8.3 VM without IHS, Plugin and WCT pre-installed using the above private image, for a non-entitled IBMid;
The updated and new added pipelines run successfully

Happy pass: https://github.com/zhengchang907/azure.websphere-traditional.image/actions/runs/944969497 & https://github.com/zhengchang907/azure.websphere-traditional.image/actions/runs/944969638. Note: please ignore the failure in step 'Summary', it won't work in local repo since its logic queries GitHub workflow jobs of the product repo. This needs further verification after the PR is in.

Signed-off-by: Jianguo Ma<EMAIL_ADDRESS>

Hello @git4rk @edburns @zhengchang907 Could you pls review the PR? Thanks.

@git4rk  I have addressed your comments in commit 81bd04d.


Field record renaming breaks interfacing of extracted code with OCaml code

Consider the following example:
module A
type u = { f : unit }

Previously, the projector of the record field f of u could always be accessed as A.Mk_u.f, both as a user notation and internally. However, this breaks the invariant that the namespace part (here A.Mk_u) must be a valid module name. This invariant must be true not only for user notation (dependencies) but also internally, to avoid clashes with any existing A.Mk_u module.
Thus the user notation for the projector was changed into A.Mk_u?.f , and internally A.__proj__Mk_u__item__f.
The latter internal notation is also used internally for record fields and is also used for extracted OCaml code, which then breaks when linked with OCaml code that uses field f of type u.
We need to restore the original field names in extracted OCaml code (and, ideally also, anywhere else in the implementation of F*)

OCaml extracted examples now have original record field names, no longer renamed.
Merged into master.


Ability to remove built-in reference counting from a implementation type

Currently every WinRT implementation type inheriting from implements automatically gets reference counting implemented. While you could override AddRef and Release, you still get the reference count field which cannot be removed. In some scenarios, you might want to eliminate the reference count field and do it all manually. These might include:

Allocating many objects from a shared memory pool, which means the lifetime of the object needs to be tied to the pool and you may want to a single reference count for all of them.
A singleton or class factory with static lifetime where you could make AddRef and Release not do anything at all.

I propose adding a new marker type, winrt::no_ref_count which will remove the ref count field, as well as the default implementations of AddRef and Release.

I've tagged a few project maintainers. If one of them is available to offer guidance and mentorship for this issue, they will reach out according to the contributing guide to discuss and agree on an approach.
@microsoft/cppwinrt-maintainers
FWIW the implements class template is terribly complicated already. I would recommend you simply use your own fork for such dramatic customizations.

Overriding AddRef and Release are not sufficient because m_references is also used for weak reference support.
Even static lifetime objects need to implement AddRef and Release so they can tear down at COM uninitialization. Otherwise, you get bugs where the DLL is unloaded (DllCanUnloadNow) because it lost track of its static lifetime objects.

Otherwise, you get bugs where the DLL is unloaded (DllCanUnloadNow) because it lost track of its static lifetime object
Well I can prevent unload and have it has a pinned DLL.

Currently I have complex cyclic object relationship where both objects must be alive at simultaneously. Weak references won't in my case because once get destructed while the other is alive. Having this feature would be really helpful so I can use a shared ref count for the two objects. Without it, I'm forced implement the ABI interfaces (or inherit from impl::produce_t), without the help of winrt::implements.

The WindowsAudioSession sample had a similar issue, which it solved by using embedded objects, although in the WindowsAudioSession case, it was clear which object was "primary" and which was "ancillary".
Another solution is to make the two objects explicitly one object under the hood, so their lifetimes are naturally coupled.
struct WidgetState
{
 // Widget business logic goes here
};

struct DoodadState
{
 // Doodad business logic goes here
};

struct CommonState
{
    WidgetState widgetState;
    DoodadState doodadState;
};

struct Widget : WidgetT<Widget>
{
    std::shared_ptr<CommonState> const m_state;
    Widget(std::shared_ptr<CommonState> const& state) : m_state(state) {}

    void WidgetMethod1() { return m_state->widgetState.WidgetMethod1(); }
    void WidgetMethod2() { return m_state->widgetState.WidgetMethod2(); }
};

struct Doodad : DoodadT<Doodad>
{
    std::shared_ptr<CommonState> const m_state;
    Doodad(std::shared_ptr<CommonState> const& state) : m_state(state) {}

    void DoodadMethod1() { return m_state->doodadState.DoodadMethod1(); }
    void DoodadMethod2() { return m_state->doodadState.DoodadMethod2(); }
};

std::pair<winrt::Widget, winrt::Doodad> MakePairedWidgetAndDoodad(Args args)
{
    auto state = std::make_shared<CommonState>(args);
    return { winrt::make<Widget>(state), winrt::make<Doodad>(state) };
}

Now you have a Widget and a Doodad whose lifetimes are coupled. Even if the app releases the last Widget, the WidgetState will remain alive until they also release the last Doodad.

@oldnewthing I don't want to have state separately; I want the sub-object itself to be alive. The embedded object solution is what I'm trying to do but implementing ABI interfaces directly is tedious, which is why I want winrt::implements to allow customisation of reference counting.


ISPN-13604 Should use IntSet.set whenever possible instead of IntSet.add

https://issues.redhat.com/browse/ISPN-13604

Needs rebase

merged! thanks @wburns !


[BUG] self-profiler hangs on spark-submit when trying to profile an entire job

Describe the bug
I would like to profile a pyspark job with self-profiler, here's the command I use:
spark-submit --master local[*] --jars ${SPARK_RAPIDS_PLUGIN_JAR} --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.rapids.sql.enabled=true --conf spark.rapids.sql.explain=ALL --conf spark.rapids.profile.pathPrefix=file:///home/haoyangl/rapids-nsys --conf spark.rapids.profile.executors=0,driver --conf spark.rapids.profile.compression=zstd --conf spark.executor.extraJavaOptions="-Dai.rapids.cudf.nvtx.enabled=true" --conf spark.driver.extraJavaOptions="-Dai.rapids.cudf.nvtx.enabled=true" test_profiler.py

Here is the pyspark job, but I think any kind of job will meet this issue:
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Test").getOrCreate()
df = spark.createDataFrame([("a",)], ["a"])

df.write.parquet("TEST")

df = spark.read.parquet("TEST")

df.selectExpr("ascii(a) AS ascii_value").show()

but the spark-submit hangs on
24/12/13 16:45:59 INFO DAGScheduler: Job 2 finished: showString at NativeMethodAccessorImpl.java:0, took 0.081151 s
24/12/13 16:45:59 INFO CodeGenerator: Code generated in 13.786791 ms
+-----------+
|ascii_value|
+-----------+
|         97|
+-----------+

Then control-C:
^C24/12/13 16:46:22 INFO RapidsBufferCatalog: Closing storage
24/12/13 16:46:24 WARN ProfileWriter: Profiling completed, output written to<EMAIL_ADDRESS>24/12/13 16:46:24 WARN ProfilerOnDriver: Profiling: Executor driver ended profiling, profile written to<EMAIL_ADDRESS>24/12/13 16:46:24 INFO AwsStorageExecutorPlugin: Shutting down S3 Plugin ...
24/12/13 16:46:24 INFO SparkContext: Invoking stop() from shutdown hook
24/12/13 16:46:24 INFO SparkContext: SparkContext is stopping with exitCode 0.
24/12/13 16:46:24 INFO SparkUI: Stopped Spark web UI at http://spark-haoyang:4040

and everything works fine in the next steps.
However, when adding a stage limit, it does not hang:
spark-submit --master local[*] --jars ${SPARK_RAPIDS_PLUGIN_JAR} --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.rapids.sql.enabled=true --conf spark.rapids.sql.explain=ALL --conf spark.rapids.profile.pathPrefix=file:///home/haoyangl/rapids-nsys --conf spark.rapids.profile.executors=0,driver --conf spark.rapids.profile.stages=1,2,3,4 --conf spark.rapids.profile.compression=zstd --conf spark.executor.extraJavaOptions="-Dai.rapids.cudf.nvtx.enabled=true" --conf spark.driver.extraJavaOptions="-Dai.rapids.cudf.nvtx.enabled=true" test_profiler.py

Have you tried reverting the PR related to stage limits to see if the problem is not reproducible?  https://github.com/NVIDIA/spark-rapids/pull/11708

Have you tried reverting the PR related to stage limits to see if the problem is not reproducible? #11708

Yes it can be reproduced without 11708. I will look into it.


Refactor ResolveAuthConfig to remove the builder dependency on cli code

registry.ResolveAuthConfig() only needs the AuthConfigs from the ConfigFile, so this changes it to accept just the AuthConfigs.
This is just a first step to removing any dependency on cliconfig. The full change is pretty large, so I've split this up into a smaller first step.
cc @tonistiigi, @anusha-ragunathan, @calavera

LGTM

LGTM 🐸

Windows failure not related, merging :wink:


fixes cncf-automation

Pushed last config.yaml change to main in error with an error, this should fix that.
This adds a cncf-automation team and repo.

My inbox thanks you <3


implement a task for pruning old image/release images

Description
Implements a task to prune old image/release built images, retains the most recent 5 images/releases for a given application.
This allows the docker registry garbage collection to remove the associated blobs to keep our object storage usage from growing indefinitely.

Merging this without actually setting to to schedule so it can be supervised (at least initially).


Mushop free tier Terraform error

Hi,
I'm getting the following errors when applying the Mushop basic resource manager stack:
2 errors occurred:
* oci_core_nat_gateway.mushopNat: 1 error occurred:
* oci_core_nat_gateway.mushopNat: Service error:LimitExceeded. NAT gateway limit per VCN reached. http status code: 400. Opc request id: ae77a4d41495cbd3de5a52e8eb8337bc/A3BA31EBE6075E6E67C818DF7E40B170/A4D3C0DCF86690B80178F16FA443F0C4
* oci_core_service_gateway.mushopSGW: 1 error occurred:
* oci_core_service_gateway.mushopSGW: Service error:InvalidParameter. The maximum limit of 0 for Service gateway per VCN has been exceeded in VCN ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaawoynhtclagxyz2zxgnru636v7gy3tpj4yyzeoc6ljxoyial5orxq.. http status code: 400. Opc request id: b3acb66ed4a7e06a30fd72a004ec768b/A439D2CEDCCC9A499DFF0085E605C3C6/5ADAAEEF00ED5CF651D25EB1EEE81BE1

This can happen if the trial period has expired and the account has not been upgraded. We are working on a solution to this problem

@mkratky can you please try to use the new stack? mushop-basic-stack-v1.0.3.zip
This version does not require NAT gateway, and you will be able to use the pure Always Free resources, even if the trial have been expired.

Closing - this is fixed in release 1.0.3 as @junior mentioned. See #36 for more details.


Name cannot have spacebar

Issue description
When there is a spacebar in the name it will only use the first part.
Steps to reproduce the issue

Compile and run code (via qt)
fill in name

What's the expected result?

2 names split with spacebar

What's the actual result?

only first part of the names

Merged


Stabilize the behavior of ConstructorAnalyzer

Description
Removed vistiedConstructors from ConstructorAnalyzer.analyze() method. The intention of that container was to prevent from infinite recursion and to increase performance when analyzing chain of constructors. However, it lead to possible errors if the outer method was called more then once. Also, infinite recursion isn't actually possible since recursive constructor calls are forbidden both in Java and Kotlin. Performance changes seems to be miserable.
Type of Change

Minor bug fix (non-breaking small changes)

How Has This Been Tested?
Manual Scenario
Tested on the following class -- previously incorrect behavior of ConstructAnalyzer lead to unnecessary reflection used for expected models, after these changes no reflection is used.
class NullabilityIssuesExample(val x: Int) {
    fun copyWithX(new_x: Int?): NullabilityIssuesExample? {
        if (new_x == null)
            return null
        return NullabilityIssuesExample(x + new_x)
    }
}

Checklist (remove irrelevant options):
This is the author self-check list

[x] The change followed the style guidelines of the UTBot project
[x] Self-review of the code is passed
[x] The change contains enough commentaries, particularly in hard-to-understand areas
[x] New documentation is provided or existed one is altered
[x] No new warnings
[ ] New tests have been added
[ ] All tests pass locally with my changes

Doesn't work and needs smarter approach -- see #1353


Use DEBUG log level when listing link children message

In the noetic-devel branch, this message is printed using ROS_DEBUG. This changes the ROS2 version to use the equivalent RCUTILS_LOG_DEBUG_NAMED macro.

Pinging @clalancette since this PR is related to #73

CI:

Linux
Linux-aarch64
Windows