Sonify

Implements #82

Note: pitch sonification is broken because mir_eval doesn't support overlapping intervals (or explicit durations).

Question for the crew: do we want to stall this until we can arrive at a good solution for pitch sonification?  Or just merge as is?
If the latter, can someone please CR this? (Looking at you, @justinsalamon :))
It shouldn't take long, as it's mainly just glue code that maps namespaces to mir_eval functions.

@bmcfee I wouldn't merge as is. If we remove the pitch_midi method and the mapping from SONIFY_MAPPING we could merge the rest, we seems fine to me.

Based on how the time_frequency sonification method is written in mir_eval, a solution for sonifying pitch_midi would be to generate a gram with a higher temporal resolution and fill in activations for each note based its start time and duration. Could perhaps try and avoid unnecessary sparsity by rounding all start/end times to match a manageable hop size (e.g. 10ms?)

@bmcfee I wouldn't merge as is. If we remove the pitch_midi method and the mapping from SONIFY_MAPPING we could merge the rest, which seems fine to me.

That was my feeling as well.
Alternately, if someone wants to PR mir_eval to fix the pitch rendering, we could just stall merging this one until the next revision.
@craffel thoughts?

What exactly is your gripe with time_frequency?  Can you make a PR?

What exactly is your gripe with time_frequency? Can you make a PR?

Essentially that intervals are implicit, rather than explicit, so you can't easily specify duration.
I can do the PR if nobody else wants to.

I'm not sure what you mean, so maybe it'd be better if you did it.

Am I wrong in understanding that time_frequency in mir_eval was written with spectrograms in mind, not symbolic representations? Perhaps it's just not the right target method for sonifying MIDI-style note sequences?

Am I wrong in understanding that time_frequency in mir_eval was written with spectrograms in mind, not symbolic representations? Perhaps it's just not the right target method for sonifying MIDI-style note sequences?

It should work fine either way.

I'm not sure what you mean, so maybe it'd be better if you did it.

ok.

Am I wrong in understanding that time_frequency in mir_eval was written with spectrograms in mind, not symbolic representations? Perhaps it's just not the right target method for sonifying MIDI-style note sequences?

I think it's more that time_frequency is meant for sonifying a single pitch trajectory, since the duration of one frequency is implicitly coded by the start time of the next.
It's pretty trivial to fix this so that it accepts a [start, end] array instead of onset times, and does the right thing with overlap and add.

It's pretty trivial to fix this so that it accepts a [start, end] array instead of onset times, and does the right thing with overlap and add.

Can you do it without breaking current functionality?  We can discuss in the hypothetical PR.

Tests are done and this should be ready to merge as soon as the next mir_eval release is out.

@justinsalamon care to sign off on this one as well?

@justinsalamon I'll trade you a mir_eval CR for a jams CR here....

@bmcfee sounds like a fair trade :) I'll give it a look this afternoon

@bmcfee I've added a bunch of comments. Don't hate me for asking, but wouldn't it make more sense for all the sonification code to live natively in jams rather than wrapping around mir_eval? The annotation viz module is a parallel case: does it make sense for it to live in mir_eval or jams?
On the one hand I'd argue that mir_eval is for evaluation whilst jams is for annotation generation/manipulation/exploration, and hence both sonification and visualization should be a native component of jams. On the other hand I can understand one would want to support sonification in mir_eval independently of jams, given that most people (still) don't use jams and would be left without sonification otherwise. I guess there are arguments in favor of both options...

Don't hate me for asking, but wouldn't it make more sense for all the sonification code to live natively in jams rather than wrapping around mir_eval?

If it wasn't already 90% done in mir_eval, it would make sense to do it in jams.  But, since we already have mir_eval as a dependency, I'd prefer to not duplicate code.

The annotation viz module is a parallel case: does it make sense for it to live in mir_eval or jams?

I think for mir_eval, that would constitute bloat, since its main purpose is evaluation.  (To channel Colin, sonification falls under "evaluation by inspection", hence its inclusion.)  Viz could also fit in mir_eval, but since mir_eval has such limited scope in terms of what kinds of annotations it supports, I think it makes more sense to do it in jams.  Or as a separate project altogether.

Coverage decreased (-0.09%) to 99.549% when pulling 0f261b814ab3ac28d33541a6ae28ac02dcdb34ab on sonify into b8e9a58037adb86aa006604259403c500425574e on master.

Coverage decreased (-0.04%) to 99.605% when pulling 7621eb4725e8043b9d9494098c1601b2563e5580 on sonify into 74bed816918736dbac1f38099e9f424e775a69e9 on master.

Coverage increased (+0.02%) to 99.663% when pulling 01f541e7af5b902e9f82ce6b4a13e193b08e830d on sonify into 74bed816918736dbac1f38099e9f424e775a69e9 on master.

It seems like the last remaining issue on this is how to handle 0-duration intervals in pitch_hz sonification.
What do folks think about the following logic?
intervals, pitches = annotation.data.to_interval_values()

if np.all(intervals[:, 0] == intervals[:, 1]):
    intervals[:, :-1] = np.diff(intervals[:, 0])
    intervals[-1, -1] = annotation.duration

# proceed

That is, only impute durations if all observations have 0 duration.  The last observation is inferred to span until the end of the annotation's valid timing interval (which need not be the entire track).  Does this seem sane?

Implemented the above -- it works, but it sounds abysmal due to heavy modulation from the imputed intervals.  I can't think of a good solution here, other than writing a special case synthesizer for continuous contours.  The problem with that being that there's no good way to tell when to deploy that instead of the more general time_frequency synth.

Implemented the above -- it works, but it sounds abysmal due to heavy modulation from the imputed intervals. I can't think of a good solution here, other than writing a special case synthesizer for continuous contours. The problem with that being that there's no good way to tell when to deploy that instead of the more general time_frequency synth.

Just gave it a spin, unfortunately the generated audio is indeed unusable. It also took about 80 seconds to process a 35 second pitch_hz annotation, which I is probably not reasonable. It looks like we do indeed require two different pitch sonification methods, one for "sparse" data and one for "dense" data. For dense data we could, for example, base it on melosynth.
The problem is, as you've noted, to determine which sonification function to use given an annotation. It's not a great solution, but perhaps we need to provide some optional parameter to allow the user to specify (something) to help disambiguate these cases?

The problem is, as you've noted, to determine which sonification function to use given an annotation. It's not a great solution, but perhaps we need to provide some optional parameter to allow the user to specify (something) to help disambiguate these cases?

I think this is getting at a deeper problem.  We're trying to overload "pitch_hz" to do too many things: dense vs sparse, continuous (monophonic) vs overlapping/multi-f0, etc.
Maybe we just need to split the namespace, eg, pitch_hz_mono and pitch_hz_multi?

Maybe we just need to split the namespace, eg, pitch_hz_mono and pitch_hz_multi?

I'm not sure that solves the problem: both mono and multi can be represented as sparse (note onset + duration) or dense (continuous f0) annotations. If anything it would have to be pitch_hz_sparse and pitch_hz_dense, or maybe pitch_hz_notes and pitch_hz_contour (the former always being sprase and the latter always dense).
An alternative option (for sonification) would be to change strategy and synthesize both sparse and dense pitch annotations by first sampling them using e.g. intervals_to_samples and then using a melosynth-like approach, instead of the gram approach.

If anything it would have to be pitch_hz_sparse and pitch_hz_dense, or maybe pitch_hz_notes and pitch_hz_contour (the former always being sprase and the latter always dense).

I like that last idea.

An alternative option (for sonification) would be to change strategy and synthesize both sparse and dense pitch annotations by first sampling them using e.g. intervals_to_samples and then using a melosynth-like approach, instead of the gram approach.

I don't understand what you mean.  Melosynth wouldn't handle overlapping notes, right?

I like that last idea.

I'm happy to go with this - I don't think it makes sense to overload the same namespace with note and continuous f0 annotations, they're fundamentally different types of annotation, so splitting into 2 namespaces (well, 4, we'd have to split pitch_midi_note and pitch_midi_contour too). Perhaps the names should be pitch_contour_[hz/midi] and pitch_note_[hz/midi] (i.e. first specify note/contour and then hz/midi, seems like that would make automatic conversion easier to implement?

I don't understand what you mean. Melosynth wouldn't handle overlapping notes, right?

No, but you could separate overlapping notes into individual tracks, synthesize each track separately and then mix together.

Perhaps the names should be pitch_contour_[hz/midi] and pitch_note_[hz/midi] (i.e. first specify note/contour and then hz/midi, seems like that would make automatic conversion easier to implement?

I don't think we need contour_midi -- if you've already quantized the frequencies to midi scale, grouping them into coherent intervals is easy/expected, right?
This leaves the question of what to do with the current pitch_hz namespace.  Any suggestions?
At any rate, forking the pitch_hz namespace should be a separate PR.  @justinsalamon care to start that thread?

No, but you could separate overlapping notes into individual tracks, synthesize each track separately and then mix together.

Sounds messy. :(

I don't think we need contour_midi -- if you've already quantized the frequencies to midi scale, grouping them into coherent intervals is easy/expected, right?

midi doesn't imply quantization, it's only the scale used to represent pitch (i.e. you can have continuous midi), the only different is whether you represent pitch in hz or in midi note numbers, but otherwise the two are equivalent. I think it makes sense to first split on note/contour, and then split on hz/midi, so I'd advocate for 4 namespaces. It also simplifies conversion: pitch_note_[hz/midi] is bi-directional and pitch_contour_[hz/midi] is bi-directional, and you can convert pitch_note_* to pitch_contour_* but not in the other direction.

This leaves the question of what to do with the current pitch_hz namespace. Any suggestions?

What do you mean? If we're splitting it up into a several namespaces, shouldn't we just eliminate it?

At any rate, forking the pitch_hz namespace should be a separate PR. @justinsalamon care to start that thread?

Happy too, but I think i'm running out of cycles. Once I have the cluster spinning and am waiting for results I could look into it.

Sounds messy. :(

Yeah, it could be.

I think it makes sense to first split on note/contour, and then split on hz/midi, so I'd advocate for 4 namespaces. It also simplifies conversion: pitch_note_[hz/midi] is bi-directional and pitch_contour_[hz/midi] is bi-directional, and you can convert pitch_note_* to pitch_contour_* but not in the other direction.

In that case, why not drop "pitch" from the name and just call them "contour_" or "notes_"?  Alternately, why not just keep "pitch" for dense data and use "notes" for sparse data?

What do you mean? If we're splitting it up into a several namespaces, shouldn't we just eliminate it?

Happy to, but I think i'm running out of cycles. Once I have the cluster spinning and am waiting for results I could look into it.

I'm not asking for code, just start the issue and write down your thoughts -- seeing as you're the expert here, it would make more sense than my spin on things.

In that case, why not drop "pitch" from the name and just call them "contour_" or "notes_"? Alternately, why not just keep "pitch" for dense data and use "notes" for sparse data?

Yes, we could just have notes_hz/midi and contour_hz/midi, though "contour" is a tad more ambiguous than "pitch_contour". I have no strong preference.

True, but someone could always checkout an older version (i.e. the current version) if they wanted support for this namespace. I'm not sure it makes sense to support backwards compatibility at this early stage where things are still morphing quite often, especially of our initial namespace was broken?

I'm not asking for code, just start the issue and write down your thoughts -- seeing as you're the expert here, it would make more sense than my spin on things.

Ah sure, on it.

Yes, we could just have notes_hz/midi and contour_hz/midi, though "contour" is a tad more ambiguous than "pitch_contour". I have no strong preference.

I'd vote for pitch_* and notes_* then.  I think that's pretty straightforward without being terribly confusing.

True, but someone could always checkout an older version (i.e. the current version) if they wanted support for this namespace. I'm not sure it makes sense to support backwards compatibility at this early stage where things are still morphing quite often, especially of our initial namespace was broken?

Forcing people to use old versions is always a bad policy, if only for keeping folks up to date with bugfixes.
I'm fine with deprecating things, but we need to give folks a runway to transition their data within a functioning environment.  My usual policy is to deprecate features within minor revisions -- with warnings to that effect -- and remove them across major revisions.

I'd vote for pitch_* and notes_* then. I think that's pretty straightforward without being terribly confusing.

No no, you can't have pitch and note as two separate things... a note is a representation of pitch! The options I'm proposing are either note_* and contour_* or pitch_note_* and pitch_contour_*.

I'm fine with deprecating things, but we need to give folks a runway to transition their data within a functioning environment. My usual policy is to deprecate features within minor revisions -- with warnings to that effect -- and remove them across major revisions.

That sounds fine to me, as long as we eventually kill pitch_hz altogether :)

Circling back on this, since #111 suggests that we're going to deprecate pitch_* in favor of more specific namespaces.
I think the piano roll sonification that we currently have will work fine for note_* annotations, but not contour_* annotations.  As @justinsalamon mentioned previously, we could roll our own sonifier for continuous pitch annotations (and/or steal from melosynth).  The question I have is: does this constitute bloat?  I suspect it might make better sense to PR melosynth into mir_eval, assuming that @craffel would find it useful.
Implementing it within jams seems a little to special-case to me, and borders on bloat.  What do y'all think @justinsalamon ?

I guess it a philosophical question: where should sonification and visualization code live - mir_eval or JAMS?
If you ask me I actually think both should be part of JAMS, since it's designed for annotation generation/manipulation/exploration, whereas mir_eval is focused on annotation comparison. Of course one could argue that evaluation by audition/visualization is relevant to mir_eva. I can also imagine other valid arguments for including this functionality in mir_eval, an obvious one being that if everything lives in JAMS people using mir_eval with different annotation formats can't take advantage of these tools.
For historical reasons right now visualization lives inside JAMS and sonification lives inside mir_eval, though I think that's pretty arbitrary.

Of course one could argue that evaluation by audition/visualization is relevant to mir_eval.

That's why it was added.

I can also imagine other valid arguments for including this functionality in mir_eval, an obvious one being that if everything lives in JAMS people using mir_eval with different annotation formats can't take advantage of these tools.

This is important.  Whatever the visualization/sonification tool is, it should (IMO) be storage format-independent.

For historical reasons right now visualization lives inside JAMS and sonification lives inside mir_eval, though I think that's pretty arbitrary.

Visualization was always intended to be part of mir_eval, but no one has gotten around to adding it.

Visualization was always intended to be part of mir_eval, but no one has gotten around to adding it.

Yes, but that was before JAMS was a thing (I think). Now you have 2 separate libraries, one for annotation wrangling and another for annotation comparison. Both visualization and sonification fit naturally into either.
Come to think of it I'm actually not sure whether viz was always part of the plan for mir_eval: I actually wrote basic viz for melody extraction eval into the original mir_eval melody module but it was canned.

It's been part of the plan for 2 years, at least: https://github.com/craffel/mir_eval/issues/28

It's been part of the plan for 2 years, at least: craffel/mir_eval#28

Haha ok cool. Still, we need to reach consensus as to where sonification and visualization go. I don't think it makes sense to have either spread across both libraries.

Still, we need to reach consensus as to where sonification and visualization go. I don't think it makes sense to have either spread across both libraries.

In both cases, I think it makes sense to have core routines in mir_eval that do the heavy lifting, and wrappers in jams that handle conversion and maybe metadata decoration (in the case of viz).
For the purposes of this discussion though, @craffel : how do you feel about adding a continuous pitch sonification routine alongside piano roll and clicks?  It would make our lives easier over here.

Haha ok cool. Still, we need to reach consensus as to where sonification and visualization go. I don't think it makes sense to have either spread across both libraries.

Well, like I said, as long as the visualizers are not tied to a specific data storage format, it's fine either way.

In both cases, I think it makes sense to have core routines in mir_eval that do the heavy lifting, and wrappers in jams that handle conversion and maybe metadata decoration (in the case of viz).

This seems to make sense to me.

For the purposes of this discussion though, @craffel : how do you feel about adding a continuous pitch sonification routine alongside piano roll and clicks? It would make our lives easier over here.

Sure, that would be great to have.  Realistically I won't do it anytime in the foreseeable future.

Realistically I won't do it anytime in the foreseeable future.

understood; I'll try to find time for it soonish.

Note to self: this PR is stalled by the following:

pitch namespace revisions
mir_eval 0.4 release

Holding off on moving this guy forward until after display merges.  There's redundant changes there for ns conversion on the new pitch namespaces that we'll need here.

Okay, I think this one's good to go now.
NB: the semantics for pitch are only, say, 99% obvious.
We support two styles of pitch sonification: piano-roll and contour.  Piano roll has higher priority, so if you have an annotation in the old pitch contour namespace(s), it will be sonified by piano roll.  This is because all pitch annotations can be coerced into a contour set, but not all can be coerced into a piano roll (eg, note_hz).
Practically, this means that only annotations of type pitch_contour will be sonified using contours.  I think this is the expected behavior, but I wanted to make it 100% clear in the comments here.

Reviewed 1 of 4 files at r6.
Review status: 1 of 5 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

This is basically done, modulo two things:

beat_position -> beats+downbeat sonification with different click frequencies
multi_segment -> similarly

We could do this, but because of how mir_eval.sonify.clicks works, we'd have to synthesize our own click samples for different frequencies.  Is this worth the effort?

downbeats (beat_position) and multi_segment are now in.
I think this covers everything except for the ones that don't make sense to sonify: key, tag_*, tempo, etc.
Anyone care for a final CR?

Reviewed 1 of 2 files at r1, 1 of 4 files at r6, 2 of 2 files at r7.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


[Pianoroll graph] replace note widgets with painted elements.

Fixed, still more optimizations to go but the pianoroll graph now just paints notes on the 'OnPaint' event, culls them on the 'RecalcGrid' method, it seems it works for up to 40k notes or so.


"pthread_create failed: Resource temporarily unavailable" during Grafana dashboard load

tl;dr
The Zed queries that populate a Grafana dashboard using an S3-based Zed lake service can cause a crash. The opening text of the error dump:
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort

Details
Repro is with Zed commit f497079.
This one is kind of complex to reproduce since it's effectively a production use case with many layers involved:

Grafana
The grafana-zed-datasource
The zed-js client
The Zed lake service

However, since it all ends in a zed serve crash that I wasn't trying to trigger, I'll start by describing the repro and sharing the dump details, then if necessary I could try to craft a simpler test that exhibits the behaviors to trigger the crash.
The repro involves our "Autoperf" metrics for tracking performance. The metric data is ultimately stored in an S3 bucket, and I run a Zed service locally on my Macbook with the bucket as its lake directory.
$ zed -version
Version: v1.16.0-6-gf4970799

$ zed -lake s3://brimdataio-lakes-us-east-2/shasta.lake.brimdata.io serve

The config for my Zed datasource plugin for Grafana remains at its default setting of http://localhost:9867.
As shown in the attached video, when I access the dashboard, Grafana begins filling in the panels by executing the Zed queries that populate them. Grafana is in some ways a black box, but I've found that if I "scroll down" in the large dashboard, it seems to trigger loading many panels at once (which is to say, if I stayed near the top, it might be a gentler test by only loading the first few panels, and now that I've found the bug I'm intentionally being more tortuous). Because of the WAN latency from my SF location to the S3 bucket in Ohio, I'm accustomed to it taking a long time for the panels to load. What I've observed is that sometimes many of the latter panels will fail to load and be shown with a red "X". As I expect a user would likely do, at this point I refresh to try loading the dashboard again hoping it will work on the second try. However, as shown here, after another long delay a large error dump appears on the zed serve console as it exits.
https://github.com/brimdata/zed/assets/5934157/669d6c6d-1416-4855-932f-9d4f6d02b46c
The complete error dump is large, so I've attached it here: zed-serve-log.txt
This issue does not seem to repro if I copy the lake contents to my Macbook and serve the lake off my local SSD instead of S3.
While I can't make sense of the error dump, given what I know of the setup, I have some theories. My first time seeing this crash happened to be right after I added three additional panels to plot metrics for a newly added test. Given what I know of the high latency between my Macbook and the S3 bucket, I would guess that perhaps there are too many query requests "in flight" and it's exhausting some resources.
I respect that this setup is arguably abnormal. That said, if it's possible that I'm correct about it being a problem with too much concurrent work, perhaps there's still something legitimate to be learned, e.g., maybe the same kind of crash could happen if many users were converging on a lake service and each simultaneously happened to be initiating a lesser number of queries in parallel?

tl;dr
I've attached a shell script repro.sh.gz that reliably triggers the repro for me with GA Zed tagged v1.17.0.
Before running, start:
zed -lake s3://brimdataio-lakes-us-east-2/shasta.lake.brimdata.io serve

Details
I made this script by using Wireshark to sniff the queries to the Zed lake API that were being issued by Grafana when loading the dashboard. There's actually more queries in the pcap than I transcribed (i.e., I only made it as big as I needed to trigger the problem) so if this doesn't work for others on the first try (geographic locations that see lower latency to the S3 bucket in Ohio might fare better than me) just say the word and I can add more.

I happened to just now notice some slow performance when accessing that Shasta S3 bucket. On closer inspection it appears compaction has not run successfully on the Autoperf pool since April, even though the zed manage process has been running this whole time. That makes me think that the extra-slow performance puling so many tiny objects may go a long way toward aggravating this problem. If we end up fixing whatever's wrong there such that it compacts again, this crash might magically go away, but if so I could probably create the repro conditions again by manually creating a fragmented, non-compacted pool.

Per the previous comment, the Zed services on Shasta have been updated to GA v1.17.0, compaction ran successfully, and indeed the problem no longer reproduces when the repro script is run against the pool data in this compacted form. However, anticipating this, I saved off the pool beforehand in its fragmented state, so to repro the problem, now do:
zed -lake s3://brimdataio-lakes-us-east-2/shasta-fragmented/shasta.lake.brimdata.io serve

...and the repro script still triggers the crash just as before. I just confirmed this with Zed commit 54a5a15.


Add support for casting between all supported datatypes on Metal

This PR aims to remove the discrepancy of casting DTypes for Metal. I've gone ahead and refactored the tests as well to group them appropriately.
One area of the change that I'm uncertain of is the version support, there was a callout to metal version 220 in the code which I preserved, but I could not find any documentation from Apple on these versions and which casts are supported in what release... Guidance on this would be great!

Thanks, I don't have a good reference for this version so let's keep this for now.


Use named subgroups exclusively

To make sure our template spec files are maintainable, we force the use of named subgroups (eg (?P<mygroup>re) rather than numbered subgroups (eg (re)). Correspondingly we force the replacement strings to use named subgroup expansion (eg ${mygroup}) rather than numbered (eg ${1}).
Also rename the "subgroup" field to "subgroup_to_replace" to make it more obvious how it works.

Fixes #45


Update NFC sample driver for Windows Driver Compliance

Install driver to driver store.
Include reference to wudfrd.inf

@jasonknichel Do you know why GitHub treats this INF file as binary file? I see the diff of INF change in some other PR can show on the web page, but this one doesn't.

l Do you know why GitHub treats this INF file as binary file? I see the diff of INF change in some other PR can show on the web page, but this one doesn't.

@tenglu83 I have seen this before while editing INFs. I believe I was able to fix this by opening the INF in notepad and saving it with the encoding set to UTF-8.

l Do you know why GitHub treats this INF file as binary file? I see the diff of INF change in some other PR can show on the web page, but this one doesn't.

@tenglu83 I have seen this before while editing INFs. I believe I was able to fix this by opening the INF in notepad and saving it with the encoding set to UTF-8.

Thanks for helping. I tried your method but it seems that the file diff still doesn't show.

l Do you know why GitHub treats this INF file as binary file? I see the diff of INF change in some other PR can show on the web page, but this one doesn't.

@tenglu83 I have seen this before while editing INFs. I believe I was able to fix this by opening the INF in notepad and saving it with the encoding set to UTF-8.

Thanks for helping. I tried your method but it seems that the file diff still doesn't show.

The diff starts to show on the next commits after I saved it as UTF-8. I think it worked. Unfortunately it cannot compare with the original file. Thanks again!

l Do you know why GitHub treats this INF file as binary file? I see the diff of INF change in some other PR can show on the web page, but this one doesn't.

@tenglu83 I have seen this before while editing INFs. I believe I was able to fix this by opening the INF in notepad and saving it with the encoding set to UTF-8.

Thanks for helping. I tried your method but it seems that the file diff still doesn't show.

The diff starts to show on the next commits after I saved it as UTF-8. I think it worked. Unfortunately it cannot compare with the original file. Thanks again!

You're welcome. I'm glad that worked. I think the way you described it is the expected behavior.

l Do you know why GitHub treats this INF file as binary file? I see the diff of INF change in some other PR can show on the web page, but this one doesn't.

@tenglu83 I have seen this before while editing INFs. I believe I was able to fix this by opening the INF in notepad and saving it with the encoding set to UTF-8.

Thanks for helping. I tried your method but it seems that the file diff still doesn't show.

The diff starts to show on the next commits after I saved it as UTF-8. I think it worked. Unfortunately it cannot compare with the original file. Thanks again!

You're welcome. I'm glad that worked. I think the way you described it is the expected behavior.

Please don't convert the file to UTF-8.  If you do, we will undo that at a future time.
The PnP platform requires that INF files be ANSI or UTF-16.  Git in general doesn't like UTF-16 and thinks it is a binary file and git prefers things to be ANSI or UTF-8.  So the INF files that github will not diff (and says are binary files) are UTF-16 and the INF files that github will diff are either ANSI or UTF-8.  Since the PNP platform requires that INF files be ANSI or UTF-16, at some point in the future, we will likely convert all INFs in this sample repo to be UTF-16 since that is preferred over ANSI since UTF-16 allows for more proper localization of the INF for those who choose to localize their INF.
We do not know of a way to get github to do a friendly diff of UTF-16 files and have reached out to some github folks to see if it is possible.

Can you please remove these from the INF sample?  They shouldn't be needed from Win8 and later, IIRC.
[MyDevice_Install.NT.CoInstallers]
AddReg=CoInstallers_AddReg
[CoInstallers_AddReg]
HKR,,CoInstallers32,0x00010000,"WUDFCoinstaller.dll"

See comments about making sure you leave the INF file as UTF-16 and removing the coinstallers part.

@jasonknichel  Shall I use UTF-16 BE or LE?

See comments about making sure you leave the INF file as UTF-16 and removing the coinstallers part.

Addressed comments. Use UTF-16 LE coding.

The PnP platform requires that INF files be ANSI or UTF-16.

This assertion doesn't appear to be true (anymore?). We have UTF-8 encoded driver INFs that are operating normally.

The PnP platform requires that INF files be ANSI or UTF-16.

This assertion doesn't appear to be true (anymore?). We have UTF-8 encoded driver INFs that are operating normally.

@riverar - that assertion continues to be true.  If you have UTF-8 INF files that 'work', you are probably getting lucky and only using characters that are present in ASCII.  I believe that the UTF-8 representation of those characters is backwards compatible with ASCII/ANSI so if you only use ASCII characters in a UTF-8 file, from the INF parsing point of view, it would parse fine as an ASCII/ANSI file.  But as soon as you used a UTF-8 character that wasn't backwards compatible with ASCII, it wouldn't get interpreted correctly.
This unfortunately isn't well documented in the INF documentation, but I did find that https://learn.microsoft.com/en-us/windows-hardware/drivers/install/general-guidelines-for-inf-files does say "You can create or modify an INF file by using any text editor in which you can control the insertion of line breaks. If your INF contains non-ASCII characters, save the file as a Unicode (UTF-16) file."
If you are using https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/infverif to verify the INF and you have saved the file as UTF-8 with a BOM (byte order mark), then infverif will tell you that UTF-8 is not supported:

infverif.exe .\test-utf-8.inf
ERROR test-utf-8.inf: INF has invalid or unknown file encoding, must be ANSI or UTF-16.

@jasonknichel Makes sense, thanks. That means this documentation needs correction too https://learn.microsoft.com/windows-hardware/drivers/display/general-unicode-requirement.

@jasonknichel Makes sense, thanks. That means this documentation needs correction too https://learn.microsoft.com/windows-hardware/drivers/display/general-unicode-requirement.

Thanks for a pointer to that page.  That page might say an INF file should not be ANSI because the INF file being ANSI prevents proper localization of the strings in the INF to languages that cannot be represented in ANSI.  So while the INF parsing platform may support the INF being ANSI, I can see why it would be actively discouraged.  However, notepad has evolved since the time those screenshots were taken, so that page probably should be update to provide new screenshots and clarified to indicate that the INF files should be saved as "UTF-16 LE".


@cartogram/remove selected variant add to cart button

Description
This PR adds fallbacks in the <AddToCartButton /> component that allow us to remove the <SelectedVariantAddToCartButton />.
Additional context
More context in this issue https://github.com/Shopify/hydrogen/issues/592
Before submitting the PR, please make sure you do the following:

[ ] Add your change under the Unreleased heading in the package's CHANGELOG.md
[ ] Read the Contributing Guidelines
[ ] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123)
[ ] Update docs in this repository for your change, if needed

@cartogram Just FYI -- I whipped up a commit on the Shopify.dev side to correspond with these changes: https://github.com/Shopify/shopify-dev/pull/16194/commits/54407716a2fe085218827b81d50da858aae7f0df.


Demo scenario for conference demos / minimal E2E experience

Define a demo scenario that highlights how users benefit from using odo when doing iterative development.

There should be 2 scenarios:
1- Start from scratch
$ cd my_dir_with_code
$ odo create java   # Or springboot, or wildfly, or ...
$ odo push
# Change code
$ odo push
# Watch for code changes
$ odo watch
# Make more code changes
# I'm ok. Push code to git
$ git commit -A -m "Code is working" && git push
# If user wants to change his DC to rely on git
$ odo update --git=URL_FOR _THE_REPO
# Else
$ odo delete -f

2- Work on existing code
$ cd my_dir_with_code
$ git fetch # or pull
$ odo create java   # Or springboot, or wildfly, or ...
$ odo push
# Change code
$ odo push
# Watch for code changes
$ odo watch
# Make more code changes
# I'm ok. Push code to git
$ git commit -A -m "Code is working" && git push
# If user wants to change his DC to rely on git
$ odo update --git=URL_FOR _THE_REPO
# Else
$ odo delete -f

there is an e2e-scenerio test currently being run as part of the test suite. So AC of this issues seems to have been accomplished

Closing as per my above comment


merge package from arlac77/npm-package-template

package.json

chore(package<EMAIL_ADDRESS>

Coverage remained the same at 83.983% when pulling 84bc29c1ef3195b8b779b3c7ea9ddf3a7173cf2a on template-sync-1 into 129cbb4ffc6da31d8dc2f0a19f7c3ba22a31d824 on master.


MFAOptions: Email delivery?

Aws::CognitoIdentityProvider::Model::MFAOptionType has a GetDeliveryMedium() method (with corresponding Set and With methods), which returns a DeliveryMediumType value of NOT_SET, SMS, or EMAIL.
Won't the delivery medium always be SMS, and won't EMAIL always be an invalid setting?
Is there any reason this setting exists?

Looking over the CIP documentation, both values appear to be valid (http://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_GetUserAttributeVerificationCode.html for example).  Why do you think email is potentially invalid?

It's my understanding that MFA codes are only delivered by SMS.
"When MFA is required on a user pool, the phone number attribute is automatically required, and phone number verification is automatically turned on."
https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-email-phone-verification.html?icmpid=docs_cognito_console#user-pool-settings-mfa

The next section seems to indicate that both SMS and email are supported as options.  It sounds like SMS is just the default once enabled.

Closing for now.  Please reopen if you have further questions or issues.


Complete version

In this milestone

create recipe index page that shows the list of all recipe created by the logged in user
create a button that leads to a form to add to the list of recipe
create a show page, show the detail of the recipe
A button that allow the user to delete it.
display a toggle button that allows for a recipe to be made public or private

Thank you @MarkoKermi


Add test for TestBot in Functional Tests project

Proposed Changes

Enable echo controller in Microsoft.Bot.Builder.TestBot project by adding an environment variable with the name of the bot we want to use.

Add new DirectLineClientTests to FunctionalTests project to test sending a message to the TestBot deployed in Azure with ARM Templates and check its response.

Steps to configure the Pipeline:
1- Define five environment variables: AppId, AppSecret, BotName, BotGroup and CHOSENBOT and mark the first two as Secrets.

2- Add a Dotnet Task to publish the Test Bot project and leave it as a zip in a folder named PublishedBot.

In Arguments we need to set:
--output $(System.DefaultWorkingDirectory)\tests\Microsoft.Bot.Builder.TestBot\PublishedBot

3- Configure one Azure CLI Task to:

Run az deployment create command to create the resources in Azure.
Run az webapp deployment command to deploy the TestBot sample.
Run az bot direcline create command to connect the bot to the DirectLine channel

The output goes into a json file from where we will recover the channel key later.

The script is:
call az deployment create --name "$(BotGroup)" --template-file "$(System.DefaultWorkingDirectory)\FunctionalTests\ExportedTemplate\template.json" --location "westus" --parameters appId=$(AppId) appSecret="$(AppSecret)" botId="$(BotName)" botSku=F0 newAppServicePlanName="$(BotName)" newWebAppName="$(BotName)" groupName="$(BotGroup)" groupLocation="westus" newAppServicePlanLocation="westus"

call az webapp deployment source config-zip --resource-group "$(BotGroup)" --name "$(BotName)" --src "$(System.DefaultWorkingDirectory)\tests\Microsoft.Bot.Builder.TestBot\PublishedBot\PublishedBot.zip"

call az bot directline create -n "$(BotName)" -g "$(BotGroup)" > "$(System.DefaultWorkingDirectory)\DirectLineCreate.json"

4- Add a new PowerShell Script Task to read the json file generated in the previous step and get the secret key to connect to the bot. It also sets an environment variable with this value.

The script is:
$json = Get-Content '$(System.DefaultWorkingDirectory)\DirectLineCreate.json' | Out-String | ConvertFrom-Json

$key = $json.properties.properties.sites.key
$botName = "$env:BotName"

echo "##vso[task.setvariable variable=DIRECTLINE;]$key"
echo "##vso[task.setvariable variable=BOTID;]$botName"

5- After the Tests run, add a new Azure CLI Task to delete the resource group we've created in the third step.

The script is:
call az group delete -n "$(BotGroup)" --yes

We strongly recommend setting this task to run Even if a previous task has failed, even if the build was canceled. With this setting, we will assure that the resources will be deleted from Azure even if the build fails at any step.
After running these steps, the Bot is deployed, tested and deleted.

:shipit:

Replaced by #1807


iMessage - Voice Messages

iMessage crashes when an attempt is made to send audio message.

Dupe of #803


Just use an option instead of the first argument to skate().

It seems simpler to just have the component ID as part of the definition since we need it on there anyway.
skate({
  id: 'my-element'
});

Naming is hard:

id
name
is (Polymer)

name sounds more self-descriptive and clear to me. id and is could be a bit confusing as there are already attributes defined with those names.

FWIW, I prefer having the component name as the first arg to skate, since it makes it clear that it is a required and core part of the component, as opposed to it being in an object with configuration entries that may or may not be present (this is what I associate function-argument objects with, anyway).
When Polymer made this change (0.8 release or something) I had the same feeling, however for Skate this may be beneficial if only to make migration from Polymer easier

I tend to agree. If you're migrating from Polymer, there's going to be more than that you'd have to worry about anyways. It would also make extending a bit weird because you'd have to put the name on your class and ensure to override it. Small quibbles, but after playing with it I think we'll stick with it as the first parameter.


⚠ Bofrost.be has degraded performance

In 74c6c7f, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1566 ms

Resolved: Bofrost.be performance has improved in 9d5f28e.


r2.post not working vs fetch

At my wits end trying to get this POST to return properly.
What else does r2 do to the request vs fetch? Something is wrong. I've tried using different initialization methods and alternate response management to try to get a working result.
Universal variables for all four examples:
const url = 'https://example.com/path';
const params = { example: 'request data', other_data: 'test' };

const body = Object.keys(params).map(key => encodeURIComponent(key) + '=' + encodeURIComponent(params[key])).join('&');

const headers = {
  'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
};

Broken, returns 400 error html:
const r2_example = await r2.post(url, {
  headers,
  body
});
console.log('r2', await r2_example.text);

Broken; alternate configuration, alternate response handling, still returning 400 error in html:
const r2_example_2 = await r2({
  url,
  body,
  method: 'POST',
  headers,
});
console.log('r2_2', await (await r2_example_2.response).text()); // 400 Bad Request

Working example with raw Header/Body
const fetch_example = await fetch(url, {
  method: 'POST',
  headers,
  body
});
console.log('fetch', await fetch_example.text());   // Proper result

Working example with node-fetch.Headers (globals created by r2) and Buffer.from(string).
const fetch_example_with_headers = await fetch(url, {
  method: 'POST',
  headers: new global.Headers(headers),
  body: Buffer.from(body)
});
console.log('fetch_example_with_headers', await fetch_example_with_headers.text()); // Proper result

I just confirmed replacing node-fetch 2.0.0-alpha.8 with 2.2.0 resolves this issue.


Update ERC-2333: Move to Final

Bring ERC 2333, 2334, 2335 up to date.
These ERCs are already the standard for BLS keys as used within the Beacon chain. This PR brings the ERCs up to date and sets them to Last Call so they can become Final ASAP to line up with their actual status.

Taking this straight to final, there doesn't seem much sense in going via Last Call as this is already the standard across BLS key derivation


Module 1.1 Assignment

Given what you have learned about Git, why would you want to use Git in your classroom?

So the students would learn Git


Bump reno requirement to latest version and set default_branch

Summary
When we switched to using main as the default branch for qiskit-terra
we encountered a bug in reno that required a branch named master to exist
for traversing the git history. To workaround that issue we temporarily
added back a branch named master while we waited for the fix [1] to be
included in a release. A new reno release has been published [2] and we
can rely on that version and delete the idle master branch now. This
commit makes this change and switches to using reno>=3.4.0 and uses
the new default_branch configuration option added in 3.4.0 to fix the
issue.
Details and comments
[1] https://opendev.org/openstack/reno/commit/ed6bbae82e01edf781534d9e9cce21e0c55a49a6
[2] https://pypi.org/project/reno/3.4.0/

Ah, this explains things - I had tried to set this locally, but just got the error unknown configuration variable: default_branch.  I didn't realise it was so recent!


ERROR: Command errored out with exit status 128: git clone -q https://github.com/Xilinx/BNN-PYNQ.git /tmp/pip-req-build-juh8zblg Check the logs for full command output.

I enter to termina this line "sudo pip3 install git+https://github.com/Xilinx/BNN-PYNQ.git" I get "ERROR: Command errored out with exit status 128: git clone -q https://github.com/Xilinx/BNN-PYNQ.git /tmp/pip-req-build-juh8zblg Check the logs for full command output."
Please help me

please immediately help

I suggest to check the logs for the full command output. What error is raised in that log?

I suggest to check the logs for the full command output. What error is raised in that log?

I didn't understand what you mean. Can you write more clearly what is a log?

I mean log file (https://en.wikipedia.org/wiki/Log_file), as stated in the error you should check the generated log file.
p.s. are you sure the board is connected to internet?

So how can I check from where? Please help me

I performed earlier such as project in link https://www.youtube.com/watch?v=LoLCtSzj9BU&t=724s . It run

You might want to read the PYNQ getting started https://pynq.readthedocs.io/en/v2.6.1/getting_started/pynq_z1_setup.html.
If the board is plugged to your laptop, most likely it's not connected to the internet. You should connect it to a router in order to have the board connected to the internet.


fix: convert cents to dollars in MarketStats

The type of this PR is: Bugfix

This PR resolves CX-1004
Description
Corrects the display value of Average Value Sold by converting cents to dollars before formatting.
PR Checklist (tick all before merging)

[x] I have included screenshots or videos to illustrate my changes, or I have not changed anything that impacts the UI.
[x] I have added tests for my changes, or my changes don't require testing, or I have included a link to a separate Jira ticket covering the tests.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added an app state migration, or my changes do not require one. (What are migrations?)
[x] I have added a CHANGELOG.yml entry or my changes do not require one.

Nice catch!


refactor[gitlab-yaml]: Add openshift platform to mdap ci

Signed-off-by: nsathyaseelan<EMAIL_ADDRESS>
Update the e2e pipelines to run on openshift.
Maintain a separate baseline for different branches to use branch
specific tagged images.

superseded by https://github.com/openebs/jiva/pull/239


Found another minor improvement for the README.md

Otherwise the following statement would deliver undefined: Deno.env.get("TOKEN")

The current example doesn't use .env config for env variable, standard Deno.env should be enough here.
Try the command below, just pass your-bot-token
Mac, lunix:
TOKEN=your-bot-token deno run --allow-net --allow-env https://deno.land/x/telegram_bot_api/examples/sending-files/01-simple-media.ts

Please, let me know if there is something confused you.

@dfvgbh ok I thought this is needed as I read https://reactgo.com/deno-environment-variables/#auto-loading
personally I had not used Deno.env.get(...) earlier as I typically use .env.json files... - so I'm not too experienced in this :) - closing the PR
Thank you for this module. It works really reliably for all my use cases so far.


New develop

Codecov Report

Merging #202 into develop will decrease coverage by <.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #202      +/-   ##
===========================================
- Coverage    52.95%   52.94%   -0.01%
===========================================
  Files          217      217
  Lines        21656    21656
===========================================
- Hits         11467    11466       -1
+ Misses        8892     8891       -1
- Partials      1297     1299       +2

Impacted Files
Coverage Δ

bfe_modules/mod_static/mod_static.go
57.97% <ø> (ø)
:arrow_up:

bfe_balance/bal_gslb/sub_cluster.go
48.14% <0%> (-7.41%)
:arrow_down:

...e_modules/mod_trust_clientip/mod_trust_clientip.go
42.85% <0%> (-3.3%)
:arrow_down:

bfe_websocket/server_conn.go
74.24% <0%> (-1.52%)
:arrow_down:

bfe_spdy/server_conn.go
78.98% <0%> (+0.88%)
:arrow_up:

bfe_util/fixed_pool.go
100% <0%> (+16.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b5d1da...ed5e790. Read the comment docs.


feat: Added file id mapping in datasource config

Description
This PR adds:

File Id mapping of the google sheets selected by user, in datasource configuration, so that when creating queries on top of such gsheet datasource, only the selected spreadsheets can be seen in the spreadsheet dropdown.

Changes done on client side:

As soon as user selects file in file picker popup, the callback will get the file ids and update the datasource to contain file ids as a part of datasource configuration properties.
If user cancels the file selection, file ids is sent as empty array and datasource is updated.

Changes done on server side:

In GoogleSheetPlugin.java where we have defined execute and trigger methods for gsheet query, here I have added a new variable allowedFileIds, which gets the list of authorised file ids from datasource configuration object and the same list is passed to functions like transformTriggerResponse and transformExecutionResponse, which returns  file list data based on the allowedFileIds. In FileListMethod class, these methods contain the logic to send only authorised file data.
Since these two methods are a part of triggerMethod and executionMethod interfaces, all gsheet query operation classes that extend this method, their function definition needed to be updated with this third allowedFileIds parameter.
Similarly all gsheet query operations test classes were using these two methods, and hence this third parameter needed to be added there as well.

How to test:

With this improvement, when we select file1 and file2 for one datasource and file3 and file4 for another datasource, In the query dropdown for first ds, we should only see file1 and file2, for second datasource, we should only see file3 and file4.

Add a TL;DR when description is extra long (helps content team)

Fixes #21074
Media

A video or a GIF is preferred. when using Loom, don’t embed because it looks like it’s a GIF. instead, just link to the video

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Manual
JUnit

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] PR is being merged under a feature flag

QA activity:

[ ] Test plan has been approved by relevant developers
[ ] Test plan has been peer reviewed by QA
[ ] Cypress test cases have been added and approved by either SDET or manual QA
[ ] Organized project review call with relevant stakeholders after Round 1/2 of QA
[ ] Added Test Plan Approved label after reveiwing all Cypress test

/build-deploy-preview

/ok-to-test

/ok-to-test

/ok-to-test

/build-deploy-preview

/ok-to-test

/build-deploy-preview

/ok-to-test

Deploy-Preview-URL: https://ce-21699.dp.appsmith.com

@prapullac


[build] Fixes for experiments, poll strategy

General fixes to experiments and builds:

remove the command line parsing for poll strategy, experiments from test_config.cc, and instead read from environment variables directly (we can, and it means that our test strategy matches what we ask customers to do - and it means we can have tests that don't accidentally pull in all of grpc, which in the long term will save build time)
Fix configs for enabling/disabling experiments: we inadvertently weren't running one side of an on/off experiment
Consequently add support for broken experiments: experiments off by default and which should not be tested (because their tests are not expected to pass)

... I will get to the broken experiments in due course.

Marking this P1 as we're currently not testing experiments as we expected.


Allow regex pattern in id_replacement

Ref: https://github.com/naokazuterada/MarkdownTOC/issues/112#issuecomment-355807670

Support in 3.0.0.


RLS: 0.4.0

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender


Fix header nav styles

Not sure if I updated app version correctly. Should it be minor version update?

Would it be better if I addressed your comments in a new commit or rebasing like this (and having "outdated diff") is ok?

Since the GitHub UI has a squash button now, it's better to add additional commits.

Thanks :)


[Question] More information about the training process: hyperparameters and token number

Hi,
I am new to mechanistic interpretability and am working on a toy project with GPT-2 using some examples. However, it seems I might be missing something, particularly regarding the number of tokens and the hyperparameters. Let me explain with a graph:

The hyperparameters that i am using is are the following:
cfg = LanguageModelSAERunnerConfig(

    model_name=args.model_name,  # our model (more options here: https://neelnanda-io.github.io/TransformerLens/generated/model_properties_table.html)
    hook_name=f'blocks.{args.hook_layer}.hook_resid_mid',  # A valid hook point (see more details here: https://neelnanda-io.github.io/TransformerLens/generated/demos/Main_Demo.html#Hook-Points)
    hook_layer=args.hook_layer,  # Only one layer in the model.
    d_in=768,  # the width of the mlp output.
    dataset_path=args.dataset_path,  # this is a tokenized language dataset on Huggingface for the Tiny Stories corpus.
    is_dataset_tokenized=False,
    streaming=True,  # we could pre-download the token dataset if it was small.
    # SAE Parameters
    mse_loss_normalization=None,  # We won't normalize the mse loss,
    expansion_factor=64,  # the width of the SAE. Larger will result in better stats but slower training.
    b_dec_init_method="geometric_median",  # The geometric median can be used to initialize the decoder weights.
    apply_b_dec_to_input=False,  # We won't apply the decoder weights to the input.
    normalize_sae_decoder=False,
    scale_sparsity_penalty_by_decoder_norm=True,
    decoder_heuristic_init=True,
    init_encoder_as_decoder_transpose=True,
    normalize_activations="expected_average_only_in",
    # Training Parameters
    lr=5e-5,  # lower the better, we'll go fairly high to speed up the tutorial.
    adam_beta1=0.9,  # adam params (default, but once upon a time we experimented with these.)
    adam_beta2=0.999,
    lr_scheduler_name="constant",  # constant learning rate with warmup. Could be better schedules out there.
    lr_warm_up_steps=total_training_steps // 20,  # this can help avoid too many dead features initially.
    lr_decay_steps=total_training_steps // 5,  # this will help us avoid overfitting.
    l1_coefficient=1.7e-3,  # will control how sparse the feature activations are
    l1_warm_up_steps=total_training_steps // 20,  # this can help avoid too many dead features initially.
    lp_norm=1,  # the L1 penalty (and not a Lp for p < 1)
    lr_end=5e-6,  # we'll lower the learning rate at the end of training.
    train_batch_size_tokens=4096,
    context_size=1024,  # will control the lenght of the prompts we feed to the model. Larger is better but slower. so for the tutorial we'll use a short one.
    # Activation Store Parameters
    n_batches_in_buffer=128,  # controls how many activations we store / shuffle.
    training_tokens=total_training_tokens,  # 100 million tokens is quite a few, but we want to see good stats. Get a coffee, come back.
    store_batch_size_prompts=16,
    # Resampling protocol
    use_ghost_grads=False,  # we don't use ghost grads anymore.
    feature_sampling_window=1000,  # this controls our reporting of feature sparsity stats
    dead_feature_window=1000,  # would effect resampling or ghost grads if we were using it.
    dead_feature_threshold=1e-4,  # would effect resampling or ghost grads if we were using it.
    # WANDB
    log_to_wandb=True,  # always use wandb unless you are just testing code.
    wandb_project=args.wandb_project,
    run_name=args.experiment_name,
    wandb_log_frequency=args.wandb_log_frequency,
    eval_every_n_wandb_logs=args.eval_every_n_wandb_logs,
    # Misc
    device=device,
    seed=args.seed,
    n_checkpoints=args.n_checkpoints,
    checkpoint_path=args.checkpoint_path,
    dtype="float32",
)

It seems I might be misunderstanding some aspects. Is the "max context" the maximum possible length of a single sentence, or something else? Should the "max tokens" refer to the total number of tokens in the entire dataset? How exactly does this affect the training process?
I am observing the same behavior across all my training experiments, regardless of the layer, which has left me quite confused.
Could someone guide me on this? I’ve been reading blogs and papers, but they often assume prior knowledge of these concepts without explaining them in detail.
Best regards,

Hi,
What's the issue in the graph you'd like to point out? If you want to plot against tokens on the x-axis, you should be able to change that in wandb.
The context_size is the length of sequences passed to the GPT2. Batches of activations look like B x C x D where B is the batch dim, C is the context length and D is the hidden dimension of the model (assuming you're training on the residual stream. Generally, using the same context_size that the model was trained with is a good idea, but that's not always possible for some of the newer models with absolutely massive context sizes. Using a longer context size will mean you use more memory for a given batch size.
For instance, I believe GPT2 was trained with a context size of 1024 tokens. This means each sequence it was trained on had length 1024. If you use a shorter context size for your SAE, then you should make sure you don't use your SAE on a sequence longer than the sequence the SAE was trained on since you'll likely get bad results.
For instance, if you trained your SAE on context length 2 (just for demonstration purposes), then you could run the SAE on the text "Hi", but if you tried to run the SAE on the final token of "Hi there my name is", you'd likely get bad results since "is" is token position 4, but the SAE was trained with context size 2.
The "training_tokens" parameter is the total number of tokens the SAE will train on. Usually, the datasets used for training SAEs are the same as what you would train a normal LLM on, and thus have absolutely crazy numbers of tokens. Realistically, you probably won't need to train on more than 1 - 10 Billion tokens for a real SAE, and you can probably get away with 500M or less.
Not sure if this is what you're asking, but happy to answer if you have further questions!

Thank you for your insights. I will try to explain myself and try to use these opportunity to learn more. Here goes my questions:

My problems seems that my SAE gets to a plateau only after seen approximately 600k tokens, what is really strange for me. It is true that the dataset is not that diverse as i am trying to do some study in AI Safety related corpus.
In the case of the max context size do you recommend on having sentences just of the max size or have diverse sizes?
Another question related to the dataset size, if the size of 169M tokens is too low should i add some other random text like the fineweb to add more corpus to it?

Again thank you so much i am a noobie in this.

You really know your plays. It is true that my values are quite high, the explained variance is nearly 1 and the L0 is 13k taking into account i have 50k latents. Why is it high for SAELens? I have been reading articles, on bigger models, and it seems that they use very small L1 coefficient.

I'm not sure why the L1 coefficient in SAELens tends to be much larger than from other sources. I suspect SAELens is normalizing the L1 loss differently (either SAELens is taking the mean across the batch, or across the batch and context, while other sources are not or the inverse or something).
Regardless, if your L0 is higher than the number of dimensions of the SAE input (768 for GPT2), then the SAE can just learn the identiy matrix and get perfect reconstruction without doing anything interesting. You should probably be aiming for a L0 between 5-150 or so I would think. There's a trade-off there, the lower the L0 the more sparse the SAE, but the reconstruction will be worse.

Thank you so much, i will try to follow your advice and I will try to keep you updated


Snapshot and Xcode 10 issues - Unable to generate screenshots for "hi" Hindi app store using xcode 10.

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description

When I run snapshot with Xcode 9.4.1 to generate Hindi (the new apple released app store) screenshots, it generates them fine but when I Use xcode 10.1 for the same, it fails.
Please help.
Command executed

fastlane snapshot
Complete output when running fastlane, including the stack trace and command used

 [REPLACE THIS WITH YOUR INFORMATION]

Environment

 [REPLACE THIS WITH YOUR INFORMATION]

Please take not of what @fastlane-bot posted, we have to know more about your environment to be able to answer.


No header with custom row format option

Hi,
I am using the Griddle react module for implementing data grid in my project. And in this I am using Custom row format setting like useCustomRowComponent={true} customRowComponent={OtherComponent}, after this I am able to see the custom row grid but there is no header because of that I am not able to do the sorting. I can also see in your documentation that there is no header when you are using custom row format, but is there any way to use table header with cutom row format.
Please help

18 days and no answer? :\

Don't use customRowComponent as I've experienced same issue. Way I got around it is to render custom components through columnMetadata as demonstrated under Custom Columns on http://griddlegriddle.github.io/Griddle/customization.html.

Right, but I need to use customRowComponent in this case because I don't want cells...
I don't know what is the problem here... I need a customHeaderComponent + customRowComponent, so the Header will be rendered ONCE before the rows.
At this moment I have my header OUT of griddle and a customRowComponent but I have a problem with this: when I have a lot of rows, a scroll will appear but won't on header so It seems like its bad aligned

I would like to see this feature as well...

+1

This is coming in 1.0 -- if someone has time to PR for 0.x stuff would really love that! :)

We've released Griddle 1.0 along with a new customization and plugin model.
The map and list examples could be used as a reference on how to customize things in Griddle 1.0. Please note, in the examples, we do not show the heading as requested here but this could be added.
I'm going to close this one for now but please feel to reach out in the Gitter channel if you're having any trouble implementing this or open a new issue if any problems are encountered. 👍


Added a heart disease prediction model

Hi @vrittles I have Added a heart disease prediction model. I request you to pls review and merge my PR.

Hi @vrittles pls merge my PR.


Custom error handlers for GraphQL requests

Is your feature request related to a problem? Please describe.
I'm looking to measure availability of certain screens in my app (by reporting network traffic and traces in your config options to initialize the SDK) and computing availability as a metric (success/total) derived from RUM traces. It looks like the SDK automatically reports errors in GraphQL responses based on error code. However, we have use cases where a part of the query errors out, but the query still returns a 200 (the actual error is nested within the response body under a separate field).
How can I report these errors? Do you have any recommendations on how to handle this?
Describe the solution you'd like
A mechanism to override the existing behavior to report errors
Describe alternatives you've considered
Should I report an error using your APIs and alter my math when I run the availability calculations (to also include these custom errors reported within 200 responses)?
Additional context
Add any other context or screenshots about the feature request here.

Hi @sushilr, thanks for reaching out!
Using our addError method with NETWORK or ErrorSource.NETWORK as source would be the appropriate way to log these errors with the current state of the SDK.
Also feel free to reach out to our support team that will be more able to help you on reporting the data where you want to see it :)

Hi @sushilr,
As there has been no activity on this issue for a while I'm going to close it.
Feel free to reopen it with more details if you still encounter a problem :)


Prefix artifact ID with quarkiverse-

Closes #26

If it's good for you I will merge.

If it's good for you I will merge.


undefined variable pyv upon :PlugUpdate after updating MacVim to 7.4.105

I updated to MacVim 7.4.105 and am now experiencing an error upon running :PlugUpdate. I've run :PlugUpgrade which succeeded. The exact error I get is below:
Error detected while processing function <SNR>3_update[1]..<SNR>3_update_impl:
line   61:
E121: Undefined variable: pyv

:version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Aug  6 2016 06:13:38)
MacOS X (unix) version
Included patches: 1-2161
Compiled by<EMAIL_ADDRESS>Huge version with MacVim GUI.  Features included (+) or not (-):
+acl             +cryptv          +float           +lispindent      +mouse_xterm     +quickfix        +terminfo        +windows
+arabic          +cscope          +folding         +listcmds        +multi_byte      +reltime         +termresponse    +writebackup
+autocmd         +cursorbind      -footer          +localmap        +multi_lang      +rightleft       +textobjects     -X11
+balloon_eval    +cursorshape     +fork()          +lua/dyn         -mzscheme        +ruby/dyn        +timers          -xfontset
+browse          +dialog_con_gui  +fullscreen      +menu            +netbeans_intg   +scrollbind      +title           +xim
++builtin_terms  +diff            -gettext         +mksession       +num64           +signs           +toolbar         -xsmp
+byte_offset     +digraphs        -hangul_input    +modify_fname    +odbeditor       +smartindent     +transparency    -xterm_clipboard
+channel         +dnd             +iconv           +mouse           +packages        +startuptime     +user_commands   -xterm_save
+cindent         -ebcdic          +insert_expand   +mouseshape      +path_extra      +statusline      +vertsplit       -xpm
+clientserver    +emacs_tags      +job             +mouse_dec       +perl/dyn        -sun_workshop    +virtualedit
+clipboard       +eval            +jumplist        -mouse_gpm       +persistent_undo +syntax          +visual
+cmdline_compl   +ex_extra        +keymap          -mouse_jsbterm   +postscript      +tag_binary      +visualextra
+cmdline_hist    +extra_search    +lambda          +mouse_netterm   +printer         +tag_old_static  +viminfo
+cmdline_info    +farsi           +langmap         +mouse_sgr       +profile         -tag_any_white   +vreplace
+comments        +file_in_path    +libcall         -mouse_sysmouse  +python/dyn      -tcl             +wildignore
+conceal         +find_in_path    +linebreak       +mouse_urxvt     +python3/dyn     +termguicolors   +wildmenu
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/Applications/MacVim.app/Contents/Resources/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall -Wno-unknown-pragmas -pipe  -DMACOS_X_UNIX  -g -O2 -U_FORTIFY
_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang   -L. -L/usr/local/lib -L. -L/usr/local/lib  -L/usr/local/lib -o Vim -framework Cocoa -framework Carbon       -lm  -lncu
rses -liconv -framework Cocoa   -fstack-protector  -L/System/Library/Perl/5.16/darwin-thread-multi-2level/CORE

Type:

[X] Bug
[ ] Enhancement
[ ] Feature Request
[ ] Question

OS:

[ ] All/Other
[ ] Linux
[X] OS X
[ ] Windows

Vim:

[ ] Terminal Vim
[ ] GVim
[ ] Neovim
[X] MacVim

Hmm, I'm running into a different issue when using Python installer on Macvim, but the Ruby installer (which is chosen over Python) works flawlessly. I wonder why Ruby installer is not activated on your system. What do you see when you execute :ruby require 'thread'; puts RUBY_VERSION on Macvim?

I get
E370: Could not load library libruby.2.0.0.dylib
E266: Sorry, this command is disabled, the Ruby library could not be loaded.

Just checked for an update on MacVim, new version (106) released that fixes scripting issues

I spoke too soon; I'm still getting the same error when I run :PlugUpdate.
Anyone else experiencing this? I'm on MacVim 7.4.107.
Thanks.

sigh something was awry with my vim installation, reinstalled macvim and the problem is gone


About the calculation of FID

Hi, that's a wonderful work and thanks for your open source code. I'm confused about the calculation of FID on SMALL scale datasets, such as Set5 and BSD100, as I know the samples of dataset need at least 2048 for calculating the fid. Referece

能加你的qq问问行不

哪方面问题？1411217340这是我的QQ

搜不到你的qq

现在应该可以了

解决没 想请教下fid计算代码


First look at HFP and mSBC support

Hi,
Here is my first cut at HFP and mSBC support. Please take a look and let me know if this approach looks good to you.
It is not yet ready for merging, one big issue being that I haven't been able to make the mSBC output actually work. My headsets just ignore the frames I send them. The input through microphone does work as does standard CVSD audio.

Hi,
As a first cut it looks great. From your previous comment, I though, that the mSBC is handled internally by the bluez, but right now I see, that there is a lot of work :)
Just few minor issues regarding coding style (but it is not really important - I will correct this while merging).  Regarding more important things (e.g. io-msbc.c), I will have to read some specs, because right now I don't know much.


Provide default fallbacks for static error pages

Currently, the error pages for things like 401/403/404 response codes are provided by the pass-ui-public repository. However, the dynamic Ember app now provides default config.json without any reference to those pages because they are currently JHU branded.
Those error pages should likely be re-worked to be default fallbacks in case an institution does not provide pages in their own config.json.

This is actually already handled here: https://github.com/eclipse-pass/pass-ui/blob/main/app/templates/not-found-error.hbs


Transparent background in turn restriction signs

The EU left and right turn restriction signs (with wing arrows) have a transparent background instead of opaque white. See here:

See also Mapillary issue: mapillary/mapillary_issues/issues/2545

@GITNE we will fix that.


improve NR Examples

ARM (Control Plane) API Specification Update Pull Request
Purpose of this PR
What's the purpose of this PR? Check all that apply. This is mandatory!

[ ] New API version. (Such PR should have been generated with OpenAPI Hub, per this wiki doc.)
[ ] Update existing version for a new feature. (This is applicable only when you are revising a private preview API version.)
[ ] Update existing version to fix swagger quality issues in S360.
[X] Other, please clarify:

Improving some of the Example files

Due diligence checklist
To merge this PR, you must go through the following checklist and confirm you understood
and followed the instructions by checking all the boxes:

[X] I have reviewed the general guidance on the spec PR review process: https://aka.ms/specprreview.
[X] I confirm this PR is modifying Azure Resource Manager (ARM) related specifications, and not data-plane related specifications.
[X] I commit to follow the Breaking Change Policy.
[X] I have reviewed following Resource Provider guidelines, including
ARM resource provider contract and
REST guidelines (estimated time: 4 hours).
I understand this is required before I can request review from an ARM API Review board.

ARM API changes review

If you want for the ARM team to review this PR, you must add the ARMReview label.
The automation may automatically add the ARMReview label, if appropriate.
If this happens, proceed according to guidance given in GitHub comments also added by the automation.

Breaking change review
If you have any breaking changes as defined in the Breaking Change Policy,
follow the process outlined in the High-level Breaking Change Process doc.
Getting help

For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
For additional help, see https://aka.ms/azsdk/support/spectools.

Hey @rkmanda .
We are just improving the examples, please approve the PR.
Thanks.

"subscriptionId": "ddqonpqwjr",

If it is azure subscription id then please update it to GUID format example. "00000000-0000-0000-0000-000000000000" can be used here.

Refers to: specification/newrelic/NewRelic.Management/examples/Monitors_ListMonitoredResources_MinimumSet_Gen.json:6 in f30b134. [](commit_id = f30b134577dc7c266684442b10fa78584c93d616, deletion_comment = False)

"subscriptionId": "nqmcgifgaqlf",

"00000000-0000-0000-0000-000000000000" can be used here.
Please update this every places where we are specifying azure subscriptionIds.

Refers to: specification/newrelic/NewRelic.Management/examples/Monitors_GetMetricStatus_MinimumSet_Gen.json:6 in f30b134. [](commit_id = f30b134577dc7c266684442b10fa78584c93d616, deletion_comment = False)

"subscriptionId": "nqmcgifgaqlf",

"00000000-0000-0000-0000-000000000000" can be used here. Please update this every places where we are specifying azure subscriptionIds.
Refers to: specification/newrelic/NewRelic.Management/examples/Monitors_GetMetricStatus_MinimumSet_Gen.json:6 in f30b134. [](commit_id = f30b134, deletion_comment = False)

Done. Now please approve the PR. Thanks.

Hey @samsaha-ms, when will it be get merged?

Hey @samsaha-ms, when will it be get merged?

PR assignee will do the needful for the next steps, expect some delay due to high PR backlog as mentioned here https://teams.microsoft.com/l/message/19:0351f5f9404446e4b4fd4eaf2c27448d@thread.skype/1688755962143?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=3e17dcb0-4257-4a30-b843-77f47f1d4121&parentMessageId=1688755962143&teamName=Azure SDK&channelName=API Spec Review&createdTime=1688755962143


feat: simplify config format

before:
{
  server: {port: 5000},
  agent: {
    keepAliveMsecs: 5000,
    keepAlive: true,
    maxSockets: 10_000,
    timeout: 10_000
  },
  firewall: [{
    base: '/foo',
    registry: 'https://registry.npmjs.org',
    rules: [],
    cache: {
      ttl: 5,
      evictionTimeout: 1,
      limit: 1_000_000
    }
  }, {
    base: '/bar',
    registry: 'https://registry.yarnpkg.com/',
  }]
}

after:
{
  server: {port: 5000},
  agent: {
    keepAliveMsecs: 5000,
  },
  cache: {
    ttl: 5,
    evictionTimeout: 1,
    limit: 1_000_000
  },
  firewall: {
    '/foo': {
      registry: 'https://registry.npmjs.org',
      rules: [],
    },
    '/bar': {
      registry: 'https://registry.yarnpkg.com/',
    }
  }
}

:tada: This issue has been resolved in version 1.35.0 :tada:
The release is available on:

v1.35.0
npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


GEODE-4803: remove flaky category

Co-authored-by: Finn Southerland<EMAIL_ADDRESS>the most recent failure is a legitimate failure which was later corrected. Removing the flaky category for this test.

@jinmeiliao Did you run the test a bunch to find out if it's indeed not flaky anymore?  These things don't seem mutually exclusive to me.

the failure was corrected by later commits. the test hasn't been failing for a long time. and all the other tickets related to this test being flaky is marked as resolved and closed.


Rebrand the website

Description
As we're unifying the FlowForge platform and our website in terms of branding, the website will need an update.

cc @joepavitt

Merged into main through: https://github.com/flowforge/website/commit/9ee2b4102a7dfa1e19a68393c64cb5dcff6581b8


dpinger does not respect virtual IPs in different subnets

Hello,
we have 2 different WAN routers in front of our Firewall with a /30 transfer subnet for BGP each. Both routers and the firewall share the same Layer 2 connection on a switch, the firewall is configured on a LAG of two ports on the firewall itself connected to the same switch. All ports are in the same VLAN.
One transfer subnet is <IP_ADDRESS>/30 where the router's IP is <IP_ADDRESS> and our IP is <IP_ADDRESS>. For the other router the transfer subnet is <IP_ADDRESS>/30 where the other side is <IP_ADDRESS> and we are <IP_ADDRESS>.
For security reasons, the real IPs have been swapped out with RFC1918 ones.
To get this setup working, we set a static IPv4 <IP_ADDRESS>/30 on the LAG and set a virtual IP <IP_ADDRESS>/30 on the LAG too.
frr is able to establish a BGP session with both routers, which basically means that the L2 and L3 configuration is working. Now the following problem:
As we're only exporting prefixes to those two BGP neighbors without importing any, we wanted to use the gateway featrues of OPNsense itself.
We created a new gateway entry for each router (<IP_ADDRESS> and <IP_ADDRESS>) which resulted in <IP_ADDRESS> being marked as UP and <IP_ADDRESS> being marked as DOWN.
Pinging the gateways itself from CLI shows that both respond to ICMP echo requests. Investigating on CLI with tcpdump reveals the following:
10:35:05.493041 IP <IP_ADDRESS> > <IP_ADDRESS>: ICMP echo request, id 11556, seq 7641, length 8

where you can see, that the firewall tries to ping the gateway in the subnet of the virtual IP with the "real" IP set on the LAG which obviously does not work which is why the gateway is marked as DOWN.
Further investigating revealed, that the dpinger process itself is being started with the following parameters:
root           11556   0.0  0.0    8956   2400  -  Is   15:01        0:11.07 /usr/local/bin/dpinger -f -S -r 0 -i <name1> -B <IP_ADDRESS> -p [...]
root           86313   0.0  0.0    6908   2356  -  Is   15:01        0:10.79 /usr/local/bin/dpinger -f -S -r 0 -i <name2> -B <IP_ADDRESS> -p [...]

Looking into dpinger help: -B bind (source) address is set to a wrong address here on one of the processes which is why it tries to ping the gateway from the wrong source address / subnet.
The workaround at the moment is forcing the marked down gateway as always up in the gateway settings and creating a gateway group with both gateways and setting the forced-online gateway to a lower tier than the gateway where the ICMP monitoring works. This basically creates a active-standby configuration where it only uses the gateway with non-working monitoring only if the first one with working ICMP monitoring is marked as down.
The question is: Is this intended behavior, a bug or is this a case that has never been considered before? Would it be possible to add a dropdown to the gateway creation mask to set that bind IP?
Our version is OPNsense 19.1.4 (amd64/OpenSSL) on Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz with two Intel x520 NICs.
Thanks.

It looks like a feature request. Traditionally, you wouldn't select "WAN" or "OPTX" but it would also list the VIPs there. That's a slippery slope so simply adding the options will not achieve much and I expect this to go into improving gateway monitoring code to deal with VIPs, which will also likely increase complexity of the code to avoid a whole new class of gateway monitoring mismatch bugs.
I'll tag this as "help wanted" so that someone willing to work on it can pick it up.

@WRMSRwasTaken you create 2 /30 networks on the same L2 domain (VLAN)? You should really fix this, or just use one network for all three. Also you should consider default-originate on both upstream peers, would make the network design more consistent. BGP can handle this easily ...

@mimugmail Unfortunately, a subnet for all three devices is not possible as the upstream ISP only assings separate transfer subnets for each router. I've already asked for VRRP or HSRP (w.e.) on their side but they don't do this for this type of connection / contract we're using.
I am currently thinking of deleting the LAG itself and assign a /30 to each of those two interfaces.
There is a detail I did not mention earlier: We're actually using two switches, with the firewall's LAG being connected to both switches and the two ISP routers being connected to one switch each.
When one Switch fails, the dpinger will register one router as down and the ISP router on that switch is going to have a timed out BGP session which results in the prefixes not being routed to this router anymore (we're not using BFD so this takes a while).
We're not currently importing the default route via BGP (although the ISP has default-originate set) because we're using a Gateway on a different line currently. This is going to change in a later time.

@mimugmail Unfortunately, a subnet for all three devices is not possible as the upstream ISP only assings separate transfer subnets for each router. I've already asked for VRRP or HSRP (w.e.) on their side but they don't do this for this type of connection / contract we're using.

So you have too few interfaces on your Firewall, correct?

I am currently thinking of deleting the LAG itself and assign a /30 to each of those two interfaces in different VLANs.

There is a detail I did not mention earlier: We're actually using two switches, with the firewall's LAG being connected to both switches and the two ISP routers being connected to one switch each.
When one Switch fails, the dpinger will register one router as down and the ISP router on that switch is going to have a timed out BGP session which results in the prefixes not being routed to this router anymore (we're not using BFD so this takes a while).
So I think that might work.

Trust me, BGP failover is not the fastest, but the most stable. If your lines are such flaky that you insist on dpinger there's really something wrong with your ISP.

We're not currently importing the default route via BGP (although the ISP has default-originate set) because we're using a Gateway on a different line currently. This is going to change in a later time.

So you have too few interfaces on your Firewall, correct?
Yes, unfortunately. But the ISP also gave us only one Port per Router on their side. We plan on adding a 2nd node for HA later which we're going to need a shared L2 anyway for CARP to work on our side.

It's a single node for now, but as I already mentioned, we plan to add a 2nd node later on. But thinking about it, this seems to work to create a CARP /30 IP for each router subnet and run two BGP sessions from the active node to both routers instead of just running 1 BGP session per firewall because both firewall nodes are connected to both switches.

I don't get it, you create on a switch e.g. Vlan30, plug provider1 in and a second port with Vlan30 and plug OPNsense in it. Same for second ISP line with Vlan31. Isn't this what you want? It would also fit in HA examples since you just create for both Vlans new ports.

I don't get it, you create on a switch e.g. Vlan30, plug provider1 in and a second port with Vlan30 and plug OPNsense in it. Same for second ISP line with Vlan31. Isn't this what you want? It would also fit in HA examples since you just create for both Vlans new ports.

You got it. This is exactly going to be my approach right now.

So because I don't need multiple IPs per Interface anymore this issue is not going to be relevant for me anymore.


Creating tests to verify deployment

Create a test set for example with Terraform Kitchen sink to verify a deployment

WIP : create test with terratest


Skipfile works only with absolute paths and not with relative paths

Also,
-*

does not exclude all files.
Also,
-.*

does not exclude all files.
A path must always start with / otherwise it has no effect.

If you use a relative path is should start with a *.

@martong I could not reproduce, -* excludes everything for me. Do you have other entries in you skip file?

@gyorb
Yes. The use case is to analyse only one file.
+/home/egbomrt/WORK/CodeCheckerProjects/protobuf/src/google/protobuf/util/field_comparator.cc
-/*

It works for me and it is documented at here: https://github.com/Ericsson/codechecker/blob/master/docs/analyzer/user_guide.md#skip. There is also a warning: To skip everything use the -* mark. Watch out for the order!.
We have also introduced a new file option for the analyze command which can be used to analyze only the given files from your compilation database. For more information see #2552.


V2.0.1 Small Add for Discovery ... $nodes

In my pursuit to make a Homie Addon for Home Assistant, ive discovered that not having $nodes like $properties make a huge difference in how you discover nodes... So i propose a small change to add this to v2 as v2.0.1
Its also such an easy thing for implementations to implement so i see no reason not to. I know v3.0.0 has but thats a whole redesign and breaking changes to implement and will take time...
@marvinroger  @ThomDietrich

I agree. It is an addition, so it does not break compatibility.

Oh, btw, the titte is v<IP_ADDRESS>, but I think it should be v2.0.1 (as mentioned in your comment).

Typo...

I agree too, if it makes things easier.

I'll make a PR tomorrow then :)

@timpur which commit will this PR be based on? The one tagged as v2.0.0?

Yes

Sounds okay to me. Are you okay to wait till we sorted out #46 and #69 ? Seems like we will come to an understanding today.

@ThomDietrich, still confused on what im meant to do with this, can you just give me instructions. More confused about what to do with the branch after release and tagged ...... i thought we wanted to delete branches and only have master.

I've done you one better, here you go: https://github.com/marvinroger/homie/tree/release-v2.0.x
Feel free to commit changes into this branch as you see fit.
You are mixing topics. Of course we are still using branches. Just no constant develop branch as it is common for big software projects. Every iteration (i.e. improvement or addition) will be nurtured in a Pull Request and merged as soon as it is ready to be part of the convention in the master branch. Think of every PR as one of many small develop branches.
Now the topic of improving an older version of the convention is unrelated to the advancement of the master branch. You are thinking about taking an older state of the convention and adding a new detail. If you want to do that you need a new "workspace" for that, outside the advancements towards and beyond v3.0.0. That's what a branch created from the "v.2.0.0" tag is for. You are now able to add unrelated changes in this workspace unhinged from the master branch, in the end we will mark this branch with the "v2.0.1" version tag and voilà, your addition will be available to users alongside other version tags.
I hope that helped ;)

You didn't finish with what to do with with the branch after tagging. Do you just delete the branch and it should magically remain because of the tag ? You guys fail to explain this last part which I've asked acouple of times now ....

Nothing. At least not necessarily. The fact that a tag will point to the head of the branch allows us to delete the branch (and potentially recreate it later). This is something we can do either way @euphi wdyt?
I'm sorry that my excessive explanations did fail your expectations. You could always... you know... do some research on the matter. https://stackoverflow.com/questions/1307114/how-can-i-archive-git-branches

Haha sorry, long day. Didn't mean to come across as rude, but damm ripped. I'll just rip now.

@ThomDietrich, anything else i need to do ?
https://github.com/homieiot/convention/releases/tag/v2.0.1

Yes looks good. 👌
Now that you've also pushed the nodejs implementation, I'd say you should delete he tag and move it one commit forward.

True that. Not very on the ball aim i.


Big set of changes

Hi elm,
Thank you for bevy-input-sequence. It's been very helpful and instructive, especially the test suite you put together.
I apologize that this PR is so big. I started working with it, and I made a macro for describing key sequences like this: keyseq! { ctrl-A alt-B shift-C } originally as bevy-input-sequence-macros, but I decided to spin it off into its own crate keyseq once I realized I could support winit too, but it was specifically made to work in conjunction with bevy-input-sequence.
The biggest internal change is I switched to using a trie implementation for detecting sequences. That changes its algorithmic complexity noted in the CHANGELOG below. I had to submit a patch trie-rs for functional and performance reasons. I have not done any performance testing; however, using a trie avoids having to spawn and despawn entities during input sequence matching.
I've tried to document my changes most thoroughly in the change log, which I'll put an excerpt here:
### Features

- Add support for bevy v0.12
- Add key chord support
- Add `key!` and `keyseq!` macro from `keyseq` crate.
- Make controllers' sequences independent of one another.
- Add `add_input_sequence_run_if()` variant.
- Add "keymod" example that uses key chords.
- Add "run_if" example

### Refactor

- Make timeout setting more ergonomic.
- Use a trie for sequence detection.

  Changes the algorithmic complexity each tick from `O(number of key_sequences)`
  to `O(length of longest key sequence)`.

- Make `new()` generic: `new<T:Into<Act>>(inputs: [T])`

### Documentation

- Describe examples in README
- Add installation, usage, and license sections to README

### Test

- The `multiple_inputs` test is disabled.

  This test includes keyboard and controller inputs. It's not clear how to
  manage these with controller sequences being independent from one another.

I have put a lot of effort into the README, so I hope despite the internal changes that they all still align with your vision for this crate.
Thank you again for the excellent crate and liberal license.
-Shane

Thank you for the great PR!
Please wait a moment so I'll check changes.
After confirmation, i will merge then release the latest version.

Oh, if this does get merged, I'll be happy submit another PR to support bevy v0.13. I'd suggest having bevy-input-sequence v0.2.0 for bevy v0.12 and v0.3.0 for bevy v0.13 or something to that effect.

I don't usually write long sentences in English, and I use machine translation, so I'm sorry if it's difficult to read.
I have checked the code and its operation.
Thank you for revising it so thoroughly that it can even remove my name from the list of authors.lol
I think the modifications themselves are excellent. In particular, "trie_rs" was implemented in such a way that it can be neatly linked with "bevy_input_sequence," which was a good learning experience.
Just one point, Act::Any is not functioning, so it needs to be resolved. (I confirmed that it does not work in examples/multiple_input.rs)
Act::Any allows input from multiple keys or buttons.
For example, in the following sequence, two patterns will be allowed: [W,D,S,D] and [D,D,S,D]
[
	Act::from(KeyCode::W) | Act::from(KeyCode::D),
	Act::from(KeyCode::D),
	Act::from(KeyCode::S),
	Act::from(KeyCode::A),
]

"Ord::cmp" is used to search the query "Trie::bin_search_by_children_labels", but it seems that the cause is that "Ord" is not properly implemented on the Act side.
When Impl "Ord", the key must be "Ordering::Equal" if it is in Any, otherwise it must be "Ordering::Less" or "Ordering::Greater" as shown below.
    #[test]
    fn ord_eq_if_contains_key_in_lhs(){
        let key = Act::KeyChord(Modifiers::CONTROL, KeyCode::A);
        let lhs = Act::Any(vec![key.clone()]);
        let rhs = key;
        assert_eq!(lhs.cmp(&rhs), Ordering::Equal);
    }

   #[test]
    fn ord_not_eq_if_non_contains_key_in_lhs(){
        let lhs = Act::Any(vec![Act::KeyChord(Modifiers::CONTROL, KeyCode::A)]);
        let rhs = Act::KeyChord(Modifiers::CONTROL, KeyCode::B);
        // If not Equal, `lhs` is Greater or Less? It needs to be implemented with consistency.
        assert_eq!(lhs.cmp(&rhs), Ordering::Greater);
    }

Alternatively, I think it would be okay to delete Act::Any itself.
I'll leave the method of correction up to you, but could you please just correct this part?

Thanks! You got it.

Released version 0.2.0
https://github.com/not-elm/bevy-input-sequence/releases/tag/release-v0.2.0
Thank you for your assistance!


Windows service

Hello! Thanks for project!
I am trying to add support to logfan to work as windows service, but I don't understand how to doit with cobra library.
If cobra is used only for short flags, so can it be replaced by pflag library of the same author?

fixed


fix build error under ubuntu 18.04.1 default compiler and standard li…

…bs: include  for std::mem_fn

@atablash Thanks for testing. I will merge this.