Will "MvcReportViewer.aspx" work with "jquery-1.9.1.js" or not?

Dear Ilich (and all) --
Please help.
Will "MvcReportViewer.aspx" work with "jquery-1.9.1.js" or not?
Currently, inside "MvcReportViewer.aspx" the code is...
src="//code.jquery.com/jquery-1.11.0.min.js"
src="//code.jquery.com/jquery-migrate-1.2.1.min.js"
But, for those of us who (sadly) have to support IE8, the highest jQuery we can use is jquery-1.9.1 here.
And, furthermore, some of us (me at least) cannot link-out to "jquery.com" at all.
So, I suppose I could hack the "MvcReportViewer.aspx" page, and point to a local version of "jquery-1.11.0.min.js", but having 2 different versions of jQuery in the same project is not a good idea, IMHO.
Plus, with  MVC5 and RegisterBundles and the generic "get highest version of jQuery" syntax ("~/Scripts/jquery-{version}.js"), if I were to hack "MvcReportViewer.aspx" and package "jquery-1.11.0.min.js" in my app, then I suppose I would have to rename "jquery-1.11.0.min.js" to something else like "MySneakyExtraVersionOfJQuery.js" to help ensure the "bundles.Add" call does not include the wrong jQuery.
So, while I know IE8 support is my problem, (and it not necessarily your problem), and while I also believe it is not necessary for a modern app to support IE8, but still I must satisfy my customer, and so  I should ask you about this issue for your advice and suggestions, because you are the MvcReportViewer expert, etc.
What do you think?
Please advise.
Thanks.
-- Mark Kamoski

All --
This is a follow-up to my post above, about MvcReportViewer and jQuery, etc.
FYI, as a careful workaround to my plight describe above, I now simply downloaded the required MvcReportViewer JS and renamed the files and put the files  in my MVC5 project (a project that must support IE8 and only use "jQuery 1.9.1" here), and then I simple did the following in the "MvcReportViewer.aspx" file...
Changed from this...

``` <script type="text/javascript" src="//code.jquery.com/jquery-migrate-1.2.1.min.js"></script>

...to this...

``` <script type="text/javascript" src="<%=this.ResolveUrl("~/Scripts/MvcReportViewerEx1.js") %>"></script>

``` <script type="text/javascript" src="<%=this.ResolveUrl("~/Scripts/MvcReportViewerEx2.js") %>"></script>

...so that is perhaps a bit hacky  but it is clear and simple and easy and works fine for now, so I am going with it, until something better comes along.

Please let me know if there is a better way to do it, etc.

Thanks.

-- Mark Kamoski

//Changed from this...

<script type="text/javascript" src="//code.jquery.com/jquery-1.11.0.min.js"></script>

<script type="text/javascript" src="//code.jquery.com/jquery-migrate-1.2.1.min.js"></script>

//...to this...

<script type="text/javascript" src="<%=this.ResolveUrl("~/Scripts/MvcReportViewerEx1.js") %>"></script>

<script type="text/javascript" src="<%=this.ResolveUrl("~/Scripts/MvcReportViewerEx2.js") %>"></script>

(I hope I got the code block syntax right this time.)
(Wish I could edit a comment.)

I have reviewed out JavaScript code. You will safe downgrading the jQuery version to 1.9.1. I think you can also remove the reference to jQuery migrate if you will go this way. Please let me know if there will be some issues. I will apply a fix then.

FYI, "jQuery 1.9.1" is working fine with the MvcReportViewer for me, no problems at all.

I'm not sure I follow what the issue with 1.11.x and up would be. The 2.x version dropped support for non-compliant browsers (IE8 being the villain generally everyone hates), but not 1.11.x. Where you getting specific errors with 1.11.x aren't present with 1.9.x?

tiesont -- The problem that I was facing is that I personally need to support IE8. So per the jQuery site, if one wants to support IE8 and still use jQuery then the highest version of jQuery that a person can use is "jQuery 1.9.1". So, I had to make sure my project linked to "jQuery 1.9.1". But the MvcReportViewer, which my project also has, links to "jQuery 1.11 something". So I had to have 2 versions of jQuery in my project, which is not a good idea in general, especially with the MVC version-generic Bundle syntax, which grabs the highest version. So instead of faking out the standard bundling process, I instead hacked the file names of the "jQuery 1.11 something" to be non-colliding with "jQuery 1.9.1" during bundling, then I hacked the MvcReportViewer ASPX to use local-path (which I had to do anyway because I cannot link-out to non-local sites in my project). Etc. The code is above. I hope this is clear enough. Let me know if you have any other questions. Thanks. -- Mark Kamoski

@mkamoski Like I mentioned: 1.11.x supports IE8. jQuery Migrate is needed from 1.9.x onward to continue supporting some removed features ($.browser being the one causing the most issues). Remember, if you're basing your "I need 1.9.x to support IE8" off of this article, that post was part of the 1.9 release notes published in 2013. Every subsequent release still supports IE8.
Obviously, if what you have works, run with it; I'm just suggesting you might not need to the extra work, hence why I asked if you're seeing bugs with 1.11 that don't appear with 1.9.

@tiesont -- jQuery 1.11 did not run at all for me with IE11 EnterpriseMode=On or when in actual IE8. So maybe I needed  migrate shim? Or something? I do not know. Just looks like IE8 support in jQuery is flaky after 1.9.1 IMHO. So migrate shim or multiple jQuery versions or whatever, seems a jumble kludge to me. Anyway I am U and R now so....

@mkamoski It doesn't hurt to load the jQuery Migrate plugin/shim. It simply backfills the features that were removed from the 1.9 release and up, and logs some messages to the console to let you know what deprecated features are being used (logging can be disabled for production). It would need be loaded after jQuery but before any other plugins.
It's also worth noting that the emulator in IE11 doesn't faithfully reproduce the behavior of an actual IE8 instance, so you might want to try running one of the virtual machines from modern.ie if you haven't already.

BTW, I just went to....  http://jquery.com/download/  ....for another project and another reason... and I came across this...  "jQuery 2.x has the same API as jQuery 1.x, but does not support Internet Explorer 6, 7, or 8... Since IE 8 is still relatively common, we recommend using the 1.x version unless you are certain no IE 6/7/8 users are visiting the site"  ....so FWIW that's what the jQuery folks are currently saying.

Yes. There are (currently) two branches of jQuery. The 2.x branch drops browser sniffing in favor of feature detection. The 1.x branch still attempts to work around older IE/Trident bugs, with the Migrate plugin being needed mostly to backfill removed functions like $.browser.
So, any version that starts with a 1 still supports IE8. You can use 1.9.x if you want, or you can use 1.10, 1.11, 1.12, etc. if that's what NuGet is installing in the MVC template.
As I keep asking:
-- Where you getting specific errors with 1.11.x that aren't present with 1.9.x?
The point I've tried to make is that you shouldn't have to use an older version of jQuery than what this project references. I also note:
-- if what you have works, run with it.
So, if you have a workable solution, awesome.


Bug: Error Launching Electron executable created with electron-packager

I'm submitting a bug report

Aurelia Skeleton Version
skeleton-typescript
Framework Version:
major.minor.patch-pre

Please tell us about your environment:

Operating System:
Windows [10]
Node Version:
6.6.0
NPM Version:
3.4.0
JSPM OR Webpack AND Version
JSPM 0.16.46
Browser:
n/a
Language:
TypeScript (ES not tested)

Current behavior:
Packaging the skeleton-typescript application using the electron-packager npm package successfully creates an executable, but attempting to launch the executable results in the error:
"Uncaught Exception:
ReferenceError: define is not defined"

Expected/desired behavior:
Packaged app should run as an Electron app.
Workaround is to change the package.json file's 'main' property to point to index.js (the file that is used to load the app as an Electron app. The electron-packager uses this property to determine how to load the packaged app, and fails if the property is set to 'dist/main.js' (the default in the starter kit).

What is the motivation / use case for changing the behavior?
Running the app using electron-prebuilt is interesting, but ultimately not useful, since that isn't a practical way to distribute the app. If Electron support is desired, the app should be able to be packaged using electron-packager, or one of the other packaging/distribution tools.

It may be that this is just a matter of updating the README to include instructions for packaging, and highlighting the need to change the 'main' property before packaging the app.

Thanks @devhammer Any interest in submitting a PR to the readme?

Absolutely...will do.

PR #693 created:
https://github.com/aurelia/skeleton-navigation/pull/693
I haven't tried the skeleton-esnext project, but it appears to have the same instructions as skeleton-typescript, so I'm guessing the same issue would arise with packaging if you didn't change "main" to "index.js". Should I do a PR for that README as well, or can someone just merge the change?

@EisenbergEffect Just wanted to check...anything else you all need from me, aside from the PR?

Nope :)

PR merged,
Should be closed.


Fallback to 1500 MTU if it is not possible to set it to >1500 on vxlan-vport

This check started to look a bit as an overkill to me, mainly because we need to specify exact port for vxlan-vport.
Fixes #1853

The original PR: https://github.com/weaveworks/weave/pull/2373

PTAL. Once we are happy with changes, I'll squash / re-oder some commits.

Needs rebasing on 1.7 after the 1.7.0 release.

PTAL.

This change seems so complex compared to the level of pain users are experiencing (only one person reported it, and they were only trying to run a benchmark), I am going to close it.


New Validator Design

Run validator.validate()
Run a method that might check any preconditions by calling validationWillRun() - #34
Validate fields

Run Sanitizers - mentioned in #64
Validate each field

Call validator.validationSuccessful() or validator.validationFailed() but make them optional - #44
Run a method that checks any postconditions by calling validationDidRun()

Sounds like a good idea. I will upload a pull request with suggestive implementation of this by tomorrow.


The training script produce WER of 2.57% on librispeech test-clean

Averaging epoch 15 - 34, (decoding with flag --epoch 34 --avg 20)
Decoding with HLG + 4gram lm rescore + attention decoder rescore.
ngram_lm_scale_1.3_attention_scale_1.2 2.57  best for test-clean
ngram_lm_scale_1.1_attention_scale_0.9 2.58
ngram_lm_scale_1.1_attention_scale_1.0 2.58
ngram_lm_scale_1.1_attention_scale_1.1 2.58
ngram_lm_scale_1.2_attention_scale_1.1 2.58
ngram_lm_scale_1.2_attention_scale_1.2 2.58

ngram_lm_scale_1.2_attention_scale_1.1 5.94  best for test-other
ngram_lm_scale_1.5_attention_scale_2.0 5.95
ngram_lm_scale_1.0_attention_scale_0.7 5.96
ngram_lm_scale_1.2_attention_scale_0.9 5.96
ngram_lm_scale_1.3_attention_scale_1.1 5.96

Thanks!
LGTM, but will wait for Fangjun to check and merge.

Could you also add a file conformer_ctc/RESULTS.md and put the WERs for test-clean and test-other in a table
into that file, e.g.,

test-clean
test-other

WER
2.57
5.94

Please also put a link to the uploaded model after model averaging in the added file so that others can use it. See https://github.com/k2-fsa/icefall/pull/10#issuecomment-901735938

Did you observe any WER improvements due to recent changes in bucketing on either of the test sets?

Did you observe any WER improvements due to recent changes in bucketing on either of the test sets?

The changes in bucketing are included in the model, but there were also some other changes. I am not sure how much it benefits from the changes in sampler.

👍 👍

Alright -- anyway, good job :)

@pkufool when you have time, would you mind telling me an earlier WER, possibly one from --epoch 23?
I want to compare it with a version with a different optimizer and learning rate scheduler.  My best WER for test-clean is 2.8 and for test-other is 6.6, with --lattice-score-scale=0.5 --method=attention-decoder --epoch=23 --avg=10 --max-duration=30 (it's still decoding test-other).
I'm wondering whether this is worse than what I'd expect for that iteration.

conformer_ctc/exp/23_10.txt:ngram_lm_scale_0.9_attention_scale_0.9      2.76    best for test-clean
conformer_ctc/exp/23_10.txt:ngram_lm_scale_1.3_attention_scale_2.0      6.07    best for test-other

I only decoded epochs greater than 30 before, will search for more epochs and update results here.

Best WERs for some epochs, the *.txt filename means epoch_avg.txt, for example, 49_10.txt is the result run with --epoch 49 --avg 10.
conformer_ctc/exp/49_10.txt:ngram_lm_scale_1.5_attention_scale_1.9      2.62    best for test-clean
conformer_ctc/exp/49_10.txt:ngram_lm_scale_1.0_attention_scale_0.9      5.91    best for test-other

conformer_ctc/exp/49_12.txt:ngram_lm_scale_1.3_attention_scale_1.5      2.64    best for test-clean
conformer_ctc/exp/49_12.txt:ngram_lm_scale_1.3_attention_scale_2.0      5.94    best for test-other

conformer_ctc/exp/49_15.txt:ngram_lm_scale_1.0_attention_scale_1.3      2.64    best for test-clean
conformer_ctc/exp/49_15.txt:ngram_lm_scale_1.3_attention_scale_1.1      5.95    best for test-other

conformer_ctc/exp/49_18.txt:ngram_lm_scale_1.2_attention_scale_1.2      2.62    best for test-clean
conformer_ctc/exp/49_18.txt:ngram_lm_scale_1.0_attention_scale_0.9      5.92    best for test-other

conformer_ctc/exp/49_20.txt:ngram_lm_scale_1.7_attention_scale_2.0      2.62    best for test-clean
conformer_ctc/exp/49_20.txt:ngram_lm_scale_1.5_attention_scale_1.7      5.89    best for test-other

conformer_ctc/exp/49_8.txt:ngram_lm_scale_1.1_attention_scale_1.0       2.64    best for test-clean
conformer_ctc/exp/49_8.txt:ngram_lm_scale_1.3_attention_scale_1.3       5.96    best for test-other

conformer_ctc/exp/44_10.txt:ngram_lm_scale_1.5_attention_scale_1.7      2.66    best for test-clean
conformer_ctc/exp/44_10.txt:ngram_lm_scale_1.7_attention_scale_2.0      5.97    best for test-other

conformer_ctc/exp/44_12.txt:ngram_lm_scale_1.3_attention_scale_1.5      2.63    best for test-clean
conformer_ctc/exp/44_12.txt:ngram_lm_scale_1.7_attention_scale_2.0      6.01    best for test-other

conformer_ctc/exp/44_15.txt:ngram_lm_scale_1.2_attention_scale_1.1      2.62    best for test-clean
conformer_ctc/exp/44_15.txt:ngram_lm_scale_1.5_attention_scale_1.7      5.91    best for test-other

conformer_ctc/exp/44_18.txt:ngram_lm_scale_1.3_attention_scale_1.0      2.62    best for test-clean
conformer_ctc/exp/44_18.txt:ngram_lm_scale_1.9_attention_scale_1.9      5.93    best for test-other

conformer_ctc/exp/44_20.txt:ngram_lm_scale_0.9_attention_scale_0.6      2.63    best for test-clean
conformer_ctc/exp/44_20.txt:ngram_lm_scale_1.5_attention_scale_1.5      5.9     best for test-other

conformer_ctc/exp/44_8.txt:ngram_lm_scale_1.5_attention_scale_1.9       2.65    best for test-clean
conformer_ctc/exp/44_8.txt:ngram_lm_scale_1.1_attention_scale_1.0       5.99    best for test-other

conformer_ctc/exp/39_10.txt:ngram_lm_scale_1.1_attention_scale_1.0      2.63    best for test-clean
conformer_ctc/exp/39_10.txt:ngram_lm_scale_1.5_attention_scale_1.7      6.03    best for test-other

conformer_ctc/exp/39_12.txt:ngram_lm_scale_0.9_attention_scale_0.5      2.63    best for test-clean
conformer_ctc/exp/39_12.txt:ngram_lm_scale_1.7_attention_scale_1.9      6.0     best for test-other

conformer_ctc/exp/39_15.txt:ngram_lm_scale_1.0_attention_scale_0.7      2.64    best for test-clean
conformer_ctc/exp/39_15.txt:ngram_lm_scale_1.3_attention_scale_1.2      6.0     best for test-other

conformer_ctc/exp/39_18.txt:ngram_lm_scale_1.3_attention_scale_1.3      2.62    best for test-clean
conformer_ctc/exp/39_18.txt:ngram_lm_scale_1.7_attention_scale_1.7      5.95    best for test-other

conformer_ctc/exp/39_20.txt:ngram_lm_scale_1.5_attention_scale_1.9      2.59    best for test-clean
conformer_ctc/exp/39_20.txt:ngram_lm_scale_1.3_attention_scale_1.5      5.98    best for test-other

conformer_ctc/exp/39_8.txt:ngram_lm_scale_1.5_attention_scale_1.7       2.66    best for test-clean

conformer_ctc/exp/34_10.txt:ngram_lm_scale_1.3_attention_scale_1.0      2.67    best for test-clean
conformer_ctc/exp/34_10.txt:ngram_lm_scale_1.5_attention_scale_1.7      6.01    best for test-other

conformer_ctc/exp/34_12.txt:ngram_lm_scale_1.7_attention_scale_1.9      2.65    best for test-clean
conformer_ctc/exp/34_12.txt:ngram_lm_scale_1.1_attention_scale_0.9      6.0     best for test-other

conformer_ctc/exp/34_15.txt:ngram_lm_scale_1.5_attention_scale_1.7      2.62    best for test-clean
conformer_ctc/exp/34_15.txt:ngram_lm_scale_1.2_attention_scale_1.1      6.01    best for test-other

conformer_ctc/exp/34_18.txt:ngram_lm_scale_1.3_attention_scale_1.3      2.64    best for test-clean
conformer_ctc/exp/34_18.txt:ngram_lm_scale_1.0_attention_scale_0.9      6.07    best for test-other

conformer_ctc/exp/34_20.txt:ngram_lm_scale_1.3_attention_scale_1.3      2.59    best for test-clean
conformer_ctc/exp/34_20.txt:ngram_lm_scale_1.2_attention_scale_1.1      5.96    best for test-other

conformer_ctc/exp/34_8.txt:ngram_lm_scale_1.1_attention_scale_0.7       2.66    best for test-clean
conformer_ctc/exp/34_8.txt:ngram_lm_scale_1.2_attention_scale_1.3       6.04    best for test-other

conformer_ctc/exp/29_10.txt:ngram_lm_scale_1.2_attention_scale_1.1      2.62    best for test-clean
conformer_ctc/exp/29_10.txt:ngram_lm_scale_1.2_attention_scale_1.0      6.03    best for test-other

conformer_ctc/exp/29_12.txt:ngram_lm_scale_1.2_attention_scale_1.2      2.64    best for test-clean
conformer_ctc/exp/29_12.txt:ngram_lm_scale_1.3_attention_scale_1.5      5.96    best for test-other

conformer_ctc/exp/29_15.txt:ngram_lm_scale_1.1_attention_scale_1.1      2.62    best for test-clean
conformer_ctc/exp/29_15.txt:ngram_lm_scale_1.5_attention_scale_1.7      6.04    best for test-other

conformer_ctc/exp/29_18.txt:ngram_lm_scale_1.2_attention_scale_1.3      2.62    best for test-clean
conformer_ctc/exp/29_18.txt:ngram_lm_scale_1.3_attention_scale_2.0      6.03    best for test-other

conformer_ctc/exp/29_20.txt:ngram_lm_scale_1.2_attention_scale_1.1      2.66    best for test-clean
conformer_ctc/exp/29_20.txt:ngram_lm_scale_1.5_attention_scale_1.9      6.02    best for test-other

conformer_ctc/exp/29_8.txt:ngram_lm_scale_1.5_attention_scale_2.0       2.71    best for test-clean
conformer_ctc/exp/29_8.txt:ngram_lm_scale_1.5_attention_scale_1.7       6.09    best for test-other

conformer_ctc/exp/23_10.txt:ngram_lm_scale_0.9_attention_scale_0.9      2.76    best for test-clean
conformer_ctc/exp/23_10.txt:ngram_lm_scale_1.3_attention_scale_2.0      6.07    best for test-other

conformer_ctc/exp/23_12.txt:ngram_lm_scale_1.0_attention_scale_1.1      2.73    best for test-clean
conformer_ctc/exp/23_12.txt:ngram_lm_scale_1.3_attention_scale_1.7      6.04    best for test-other

conformer_ctc/exp/23_15.txt:ngram_lm_scale_1.0_attention_scale_0.7      2.77    best for test-clean
conformer_ctc/exp/23_15.txt:ngram_lm_scale_1.2_attention_scale_1.7      6.04    best for test-other

conformer_ctc/exp/23_18.txt:ngram_lm_scale_1.5_attention_scale_1.9      2.79    best for test-clean
conformer_ctc/exp/23_18.txt:ngram_lm_scale_1.1_attention_scale_1.0      6.1     best for test-other

conformer_ctc/exp/23_20.txt:ngram_lm_scale_1.2_attention_scale_1.3      2.89    best for test-clean
conformer_ctc/exp/23_20.txt:ngram_lm_scale_1.7_attention_scale_1.9      6.58    best for test-other

conformer_ctc/exp/23_8.txt:ngram_lm_scale_1.3_attention_scale_1.7       2.74    best for test-clean
conformer_ctc/exp/23_8.txt:ngram_lm_scale_1.3_attention_scale_1.5       6.05    best for test-other


Incorrect placement of the validation feedback when using input groups

A few days ago, I opened this ticket on the vue-bootstrap repository. However, they said, that this is a bootstrap related issue, and that they cannot help me. So I want to ask here, if you can help me with that.
Thank you in advance for any feedback.

Bug reports must include a live demo of the problem. Per our contributing guidelines, please create a reduced test case via CodePen/JS Bin or Stackblitz and report back with your link, Bootstrap version, and specific browser and OS details.
This is an automated reply

@ysds Sorry, but I have no idea how to create a bootstrap-vue based live demo. Keep in mind, that this is not just regular bootstrap. I askey the guys over there (see the linked ticket above) and they said, that it's a bug on your side, and that they cannot fix it. I've provided some source code to reproduce the error in the linked ticket.
Normally, the display looks like this:
[ Description | Input field | + ]
When the validation feedback appears, it looks like this:
[ Description | Input field |
Validation feedback
| + ]

The correct behavior would be:
[ Description | Input field | + ]
Validation feedback

Can you create a reduced test case with pure HTML?

The main problem is, that vue creates the HTML source on the fly with JavaScript. I can't even see any proper HTML source, only in DOM explorer.
The main problem is, that in order to connect the feedback text with the input field, they must be siblings in the dom. This, though, doesn't work in a scenario, where you want to use an input group. In that case, the feedback text can only be a sibling of the input group, not the input itself. Otherwise, it will break the layout as described above.
As far as I understand it, vue's problem is, that they technically aren't able to connect the feedback text control with the input, as long as they are not direct siblings in the DOM. But I don't have much insight in the inner technical implementation of either your validation system, nor their translation from the vue.js XML template into real HTML and JS.
In my specific case, I could work around the problem, because I can set the validation state manually through my code. So I just linked both state properties to the same dataEntryNumberInvalid flag in my data model.
<b-input-group prepend="Serial or part №">
  <b-form-input type="text" class="form-control" v-model="dataEntryNumber" v-on:keydown.enter.prevent="getOrder" placeholder="Please enter a serial or part number here" :state="dataEntryNumberInvalid ? false : null"></b-form-input>

  <b-input-group-append>
    <b-button variant="success" @click="getOrder()" :disabled="!dataEntryNumber || gettingOrder || complaint.articles.length >= 10">
      <font-awesome-icon v-if="gettingOrder" icon="spinner" pulse />
      <font-awesome-icon v-else icon="plus" />
    </b-button>
  </b-input-group-append>
</b-input-group>

<b-form-invalid-feedback :state="!dataEntryNumberInvalid">
  We could not find a product for your input. Please enter a valid reference.
</b-form-invalid-feedback>

However, as @tmorehouse explained in his answer to my original bootstrap-vue ticket, this would not work in any case, where I would only rely on browser validation. So please consider this ticket more as a "please fix this general problem together", instead of a "I have a specific problem with my code".

@AnReZa See https://github.com/twbs/bootstrap/issues/23454 (and search for similar open/closed issues).

Closing per last comment.


SPI slave support

Basic SPI slave support, starting to implement #469.
DMA only, as in slave mode the clock is controlled externally, so there's no way to force a send or receive. So the API shouldn't really be "send this set of bytes", but rather "here's a set of bytes to send whenever the master wants to get around to asking for them". DMA fits the latter quite well.
Thank you!
Thank you for your contribution.
Please make sure that your submission includes the following:
Must

[x] The code compiles without errors or warnings.
[ ] All examples work.
[x] cargo fmt was run.
[x] Your changes were added to the CHANGELOG.md in the proper section.
[x] You updated existing examples or added examples (if applicable).
[x] Added examples are checked in CI

Nice to have

[x] You add a description of your work to this PR.
[x] You added proper docs for your newly added features and code.

Notes
I don't know for sure if the example works. It should, but it requires hardware mods to the board holding the -c3 chip in order to separate the SPI slave and master. If it's ok to try a remote debugging sequence like I did in PR #475, that works for me too, but if nobody wants to try that, i can mark this PR private and wait until I have access to the -c3 hardware again. That being said, I can't test on any other variant of the hardware at all, even after I wait.
I also don't think this is fully ready yet - its feature flags support non -c3 chips, but I haven't read through their TRMs to see if the code is doing the right thing (or even the documented thing). Inverting the SPI mode bits on the original ESP32 is the only thing I've seen so far, and that was a happy accident. I need to read the SPI chapters of several other reference manuals before it's truly ready - but the main structure is here, and that's the big thing I'd like to get reviewed at this point.
Thanks!

Looking at the xtensa failures ... Ah, at least part of this is the duplex mode generic arg, and some if the rest is the SPI3 peripheral. Sorting those out.

I gave this a quick try and with the bit-banged master I didn't got it to work.
Then I removed the bit-bangging master code and connected an ESP32 running the spi_loopback_dma example.
I got this which looks quite good (despite the 0 reading in the first iteration)
slave got [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] .. [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
slave got [1, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 1]
slave got [2, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 2]
slave got [3, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 3]
slave got [4, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 4]
slave got [5, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 5]
slave got [6, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 6]
slave got [7, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 7]

On the ESP32 side I got
[b9, ba, bb, bc, bd, be, bf, c0, c1, c2] .. [75, 75, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 1, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 2, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 3, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 4, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 5, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 6, 75, 75, 75, 75, 75, 75, 75, 75]
[8, 9, a, b, c, d, e, f, 10, 11] .. [7b, 7, 75, 75, 75, 75, 75, 75, 75, 75]

That also looks almost good beside the read on first iteration - the slave should have set the iteration count as the first and last element.
It looks like the master is picking up the data starting at the 8th / 9th byte (I changed the slave example to fill the send buffer with bytes counting up)
I will be travelling for business next week so I probably won't be able to do testing next week

Turns out I haven't been able to get much time for this either lately. :-/
Maybe rewriting what's going on will help sort things out. On the first transfer, the first b9 = 185 bytes are missing -- or actually, it's 185+256*N for some integer N. So 185, 441, 697, 953, 1209, 1465, 1721, 1977, 2233, ... None of those look like particularly memorable numbers to me except possibly 1465, but even that is only because of MTU values on an old PPP encapsulated network link, so it has no relevance here.
Can you change the master side code to hunt down where the incrementing bytes stop, and print out how many extra 0x75 bytes are at the end? Something like an rposition call on the slice's iterator, maybe. Not sure if this number will make anything obvious, but maybe.
Further transfers always lose 8 bytes, as you said.
I wonder where the 0x75 is coming from. Is that transmitted on the SPI bus, or are those bytes untouched by the DMA on the master side? What happens if you fill the RX buffer with some other fixed pattern before the transfer starts, on the master side?
 I wonder if the FIFO resets are happening at the right time. That wouldn't necessarily explain the first transfer (nor the zeros printed in the slave side), but of course there might be more than one issue. (Or actually, if it's timing related, maybe the first transfer takes longer to set up so it loses more into the FIFO?) I'll have to re check all of that with the TRM at some point soon.
Or, I wonder if the slave hardware is looking for an 8 byte segmented transfer setup packet - the place that the TRM says to disable segmented transfers seems a bit late (it's after DMA is already started). That feels unlikely, but might explain something. I'll have to try to find some time to carefully reread the DMA chapter as well as the SPI one, with that in mind.
But one other question I do have when you get time again - does the logic analyzer show the earlier bytes getting sent out at all, possibly not synchronized to a clock pulse? I don't think there could be spurious pulses (and I think a FIFO reset is more likely to be the cause either way), but I wonder what shows up on the bus.
Thank you!

... Yeah, section <IP_ADDRESS> of the -c3 TRM has a diagram. The various FIFOs involved in the send and receive paths are all 8*8 bits long. That's super suspicious.
I think section <IP_ADDRESS> looks odd. Nowhere does it specifically say to turn on the DMA start bit, but the code does it at the "start the GDMA tx/rx engine" step, before it resets the three different afifos, and even before it configures the SPI peripheral for single transfer mode. I am guessing DMA needs to be started in between step 7 and 8, or maybe in the middle of step 8.
Let's see if I can get that code together any time soon.

Ok, let's see if that makes any difference. The last commit before rustfmt (74e14ef) makes some changes to the DMA code to let the SPI code start both tx and rx after the afifos get reset, but it also keeps compatibility for other peripherals, like i2s and the SPI master. That compatibility isn't strictly needed, but it's less to write to get this testable. I'll sort it out later, if it works.
(I partly wonder if SPI master code needs to do something similar ... It looks like the send path is working, at least on the esp32, though. So probably not.)
If this doesn't fix the first missing chunk, I would still be interested to find out how many 0x75 bytes are at the end of that one.

 We'll need the logic analyzer (that you were using in the LEDC fading PR) for that anyway, I think.
But for what's there now, both sides did eventually start producing the right numbers, on transmission number 0xb (so either 11 or 12; I can't remember for sure if that number starts at 0 or 1). So I think this is actually better in one way and worse in another: it's better in that it stops losing the first eight bytes, but it's worse in that it takes a lot more transmissions to get to that point. The earlier code got to that point after only one transmission. So I think the DMA change is needed, to only set the "start" bit after resetting the afifos.
It's also odd that the master side shows a continuous sequence of numbers even across transmissions - the first row ends with 5, 4, 3, and the second row starts with 2, 1, ff. It looks like something is out of sync between the two - and here, again, I think the logic analyzer would be most useful.
Not sure how many bits it can capture in a row, but if you can get it to sample all four lines (cs, miso, mosi, and sck) until the two sides get in sync, that would be best. If it can do that, can you post the capture file?
The other question is, how long are the slices that the code is printing chunks of? They're supposed to be 32000 bytes, but I'd like to actually check. Can you change the code to print out the length on both sides before the first and last bytes, as well?

(There, let's get rid of ten of those commits, and squash them into wherever is appropriate. The one remaining change to fix the build for non -c3 devices can be squashed as well, but it's different enough for now that I don't think I will yet.)

Yeah, maybe sorting it out on my own would be faster in terms of turnaround time.  There will definitely be some downtime before I can start that, though, as while I can mess with software here, I can't mess with hardware (even if I did have two of the dev boards).
But, weirdly, what I see from the logic analyzer capture looks perfect.  The MISO line is registering bytes that go down starting at 0xfe, while the MOSI line is registering bytes that go up starting at 1.
That tells me that the problem is somewhere on the DMA side, or the SPI peripheral isn't interpreting some of the registers the way I expected.  But while I could try to describe the changes I'd like to use to narrow things down more, I don't think the changes I'd be describing will be the only ones.  So I can see why you might not want to do a long remote debugging session -- so I'll definitely wait.  I might look through the code a couple more times to make sure I'm not missing something obvious.

So part of the rebase here was making changes to the Drop impl on the various transfer structs, so they checked both the DMA channel status as well as the SPI peripheral's idea of done. On transmit, especially, these are different - DMA is done when the 8-byte afifo gets filled, while we can't actually do anything with the peripheral until trans_done is set when the data is shifted all the way out onto the bus.
I'm not sure if that will help though. The transmit side isn't the issue anyway, or at least not in the LA sample from before - the bus is doing the right thing, just the data reception isn't working right.
... Hmm, there's an idea. Maybe clearing the afifos needs to be split as well, around the start bit being set. Maybe the one direction afifo needs to be cleared before starting, as today, but the other needs to be cleared after, as before the "split" commits I added. Maybe DMA is reading old data out of those afifos when it first starts up.
I'll see what I can do here - not sure if you were planning to test anything at all here, but don't do it yet if so.
Thank you very much again :)

Ok, so in the "received by master" in the most recent run, I see a pattern that I missed before. Maybe the issue isn't in the afifo resets, although I still wonder.
Because the code in the slave is filling the tx buffer with 255..1 sequences (not 255..0), I should not expect the end of the 32000 byte buffer to be on a byte rollover value (either 255 or 0). And indeed, in the very first transmission, I see the buffer ends with 87, 86, 85, 1, fe (where the 1 looks like the transmission index from the second iteration). So if the last two bytes had been transmitted (or received?), that buffer would end in 85, 84, 83.
And ff to 83, inclusive, is 125 - matching, exactly, the value of 32000 mod 255. So that first transmission is fine, other than missing the last two bytes. I'm not entirely sure why the code logged the first two bytes of the next transmission, but the DMA engine on the master side will save unused chunks of rx buffer and fill them the next time around, so maybe that's happening.
Well, except no, because the master side logs before it starts to fill the rx buffer again. So maybe that's from the slave side; maybe it moved on to the next descriptor in the DMA setup somehow, or maybe it managed to reset the DMA engine and re register the buffer in time for the master to get those two bytes when it sent the last 16 clocks.
The other thing is, it's not always two bytes. It's two the first time around, but it's those two plus five more, then plus five more, etc., each subsequent time around. Hmm.
On the slave side, the first transmission is all wonky, but the second starts with the three expected closing bytes of the first transmission - 7b, 7c, 7d (but 7d got overwritten with the transmission index 0). That's the same 125 bytes as on the send side, but we're off by three this time instead of two. Future transmissions have the same off by five pattern going on.
I don't see anything obvious in the code that would generate an off by five error, but maybe it's a consistent timing issue or something. Maybe there's something wrong in how the code decides the transfers are done, still.

Hmm, at least some of that last comment is nonsense - there's a 250ms delay between when the transfer says it's done, and when the code sets up another transfer. So it's not likely the case that the slave side is setting up another transfer before the master side sends the last 16 clocks.
So then where are those bytes coming from...
I might just have to dump out the DMA descriptors to see what's happening... That will of course require access to the hardware, so not until at least next week. Or maybe I could have it flip gpio lines when certain events happen, and see about that timing.
It's still odd that the SPI bus has all the right bits being sent in the right order too, though. Hopefully this all makes sense soon.

I still haven't been able to make much progress here. I did manage to get two dev boards connected together, and the right data is definitely being sent on both the miso and mosi lines. So I think something is off on the receive side only, but I still haven't been able to put a lot of effort into figuring out what yet.

There we go, that seems helpful.
On the first transmission, everything is fine except that the receive side logs a bunch of zeros before the data starts. After adding a ton more logging, and trying several .fill() calls on the rx buffer before starting DMA, I finally realized today that there are unchanged bytes in there - and the total number of them is exactly 4092*3, i.e. the first three descriptors are not being filled with SPI data. But the first byte of each of these chunks was being set to 0 -- even if I change the master side to never send a 0 byte.
I also tried changing the CHUNK_SIZE const to 124, and adding a bunch more descriptor u32s to match. It's always the first two or three descriptors that have a single zero byte added, regardless of the byte count.
I wonder if this is related to the glitches I see on the chip select and clock lines when setting up the master ... perhaps the zero byte is some kind of timeout or error result, after only one bit gets shifted into the SPI peripheral's buffer. Or after no bits get shifted in, if the issue is the high->low->high glitch on the chip select line.
And I'm surprised the recent change to report a Result<> from the wait method didn't show anything either ... if this is some kind of error, maybe there are more bits to check or something. That will need another read through the TRM.
Not sure on the best way to fix it either, but I think I see what's going on now. Two or three descriptors just aren't getting used the first time around.

Hmm. According to the esp32c3 datasheet, pages 20-21, in table 6 the default pins for SPI2 clock and chip select (which the SPI loopback example also configures) are gpio6 (mtck) and 10.  And on page 22, in table 7, both of those lines are documented to have a 5 microsecond low level glitch at startup.
When the master device resets and this glitch hits the listening slave device, I bet it's causing at least some of the issues.
Fortunately SPI2 can run through the gpio matrix to any data lines. Next week when I'm back in front of the hardware I'll try to find time to rewire the bus so that so least cs is on a line that doesn't glitch at startup, and let's see if that makes any difference.
This probably doesn't affect the single device example because it doesn't initialize the receive channel before reset is done so the glitches don't affect anything. Or actually, because its receive channel is not the same as an SPI slave (it's just doing a full duplex transfer), so nothing looks at chip select at all.
But it would really be nice if there was a way to ignore zero byte transfers in the SPI DMA handling, to let it ignore these on its own. Because really, it should be robust to something like this. Sadly I don't see any way to do that, so if this is the problem, then changing the wiring might be the only fix

I can't find any gpio that doesn't have those glitches at startup, even the ones that Espressif documents as not having the "G" note. All of them glitch low when the bootloader is running, before the SPI master code even starts.
So I went back to the bit bang master, because within a single MCU it's possible to avoid starting the slave until after the glitches are done. (And in fact it's impossible to start it before the glitches, because they're at bootloader time.) This fixes all the problems with missing bytes and with the slave receive side losing bytes.
But I had to fix the bit bang master, obviously. Turns out it had two bugs, one where it was iterating over a list of arrays instead of a list of integers (so it only spit out one bit per byte), and one where I got the default bit order of SPI backwards (so it went in lsb first mode instead of msb first).
Fixing those bugs makes it work perfectly, with only one device. As the checks are all passing as well, I think this is ready for review again.
If you think it looks reasonable, I'll see if all the intermediate commits are still needed, and drop the ones that aren't, then squash everything and rebase one more time.
Thank you for waiting!

Ok, I'm pretty sure the intermediate commits are needed (though most of them can be squashed). And I think this is ready for another review; I just rebased to the new head.

Thank you! I might not get to review this today since because of the changes to DMA I should test all DMA related examples on all chips
I understand you only have access to ESP32-C3 but it seems this is also intended for other targets - would be nice to have the example adapted for them - but you can wait doing that after the first tests on ESP32-C3 certainly

BTW if anyone else wants to take a look at this ... go ahead - I haven't started reviewing this

I was able to successfully test this on ESP32-C3 🚀
There are some minor nitpicks but overall looks really good to me!
Also briefly testing other DMA related examples was successful

Addressed your comments.
Time to copy the example to all the other chips' directories?

Addressed your comments.
Time to copy the example to all the other chips' directories?

Nice! Yes - let's see if this will work for all targets. Would be great

Ok, that passes the automatic build for everything - it doesn't include an example for the -c6-lp variant, though, since that looks very new in the codebase, and only has one blinky example so far. Hopefully it works on all of them. :)
There might be register differences I'll need to fix up, but let's see.
Thanks again!

C6-LP can't even use SPI or DMA so it's fine
here are my test results

C3: still works
ESP32: had to choose other pins, but didn't work

pins used: 23-22, 21-19, 18-5, 17-15

S2: didn't work
S3: WORKS!
C2: crashed, reduced buffers to 3200, didn't work
C6: WORKS!
H2: had to choose other pins, WORKED

pins used: 3-4, 2-1, 25-5, 10-9

For the esp32, were those pin numbers the pin number on the dev board (in which case, which dev board was this?), or the pin numbers on the esp32 package, or the gpio.pinX numbers in the rust source? Also, which of the two numbers were used for the slave_* local variables, and which for the master_*?
I'm glad that the s3 worked - that means that the xtensa variant is at least possible.
For the esp32, the s2, and the c2 with smaller buffers, what happened? Maybe focusing on one at a time is a good idea, unless they're all failing in the same way? It's the output the wrong data, or is there some error reported?
If it's the wrong data, does the bus look like it's working from the perspective of the logic analyzer?
Thanks!

For the esp32, were those pin numbers the pin number on the dev board (in which case, which dev board was this?), or the pin numbers on the esp32 package, or the gpio.pinX numbers in the rust source? Also, which of the two numbers were used for the slave_* local variables, and which for the master_*?

it's GPIO pin numbers on the dev-board (== gpio.pinX numbers). It's a generic dev-board with a revision 0 chip. In general, first number is master, second number is slave
I only had to use smaller buffers on C2 - with the original sized buffers it just crashes with "'Store/AMO access fault'" - apparently the stack overflows
"Doesn't work" means nothing is happening at all in my above post.

Maybe focusing on one at a time is a good idea

I think it's a good idea either way. I can check one of them with the LA - just let me know which one

Starting with the c2 is probably best since both the other chips are xtensa - from your description it sounds like the DMA transfer never finished, and so then maybe it's an issue recognizing the chip select line, or the DMA transfer is set up wrong... We'll see.
Thank you!

  I suspect they're present on the bus, but the 4 Msamples/sec rate was too slow to catch the low level of the clock line.  But if that's true, then the low state might have happened too fast for the SPI peripheral to notice, as well (the SPI peripheral on the -c3 chip can go up to 60MHz, but maybe the peripheral is slower on the -c2, since the overall clock speed is slower, and maybe it's just below the needed sample rate).
I think it'll help to add a short delay while the clock line is low.  That will slow down the transmission, but that seems fine; it's still way way faster than the 100kHz transmission in the spi_loopback_dma.rs example.
It looks like the delay driver doesn't support pausing for less than a microsecond, and the samples in here have a high time on the clock of 500ns or 750ns (so in reality it's somewhere between those) ... eh, oh well, let's just do one microsecond, since that will definitely be long enough.  It means the low clock pulses will be longer than the high ones, but that's fine.
I also shrank all the buffers.
Does this work any better?

  If they don't, let's get a capture of the S2 device's bus (with the delay) and see if I can figure out what's going on.
Also, I changed the pin mapping on the ESP32 example -- let me know if that's the mapping you meant, and I'll fix up the H2 example in the same way.  If I misunderstood, let me know how, and I'll try to get it to match what you were running.
Thank you!

I attached the capture of ESP32 and ESP32-S2. Not working yet but the pins for ESP32 are fine
capture.zip

Hmm, this is very weird.  The same thing is happening on both ESP32 and ESP32S2: MOSI is fine, clock is fine, chip-select is fine.  But MISO is constantly at 1 ... except for a few single-sample low glitches, that are not happening when the clock line transition sees them.  So the received data is always 0xff on the master side.
I wonder if the MISO line's pins are misconfigured on the master.  What happens on the LA when you disconnect the slave from the master; does the LA software correctly decode the SPI data if it's the only thing plugged into the slave's MISO line?
I don't see anything in the TRM that looks like it would cause the peripheral to always send 1 bits, unless that's the value that it reads from the DMA buffer.  (And it's not the value in that buffer, at least.)  Or the 0xff could come from the FIFO ... but that FIFO drains, so it's getting fed with 0xff bytes if the problem is in the DMA or SPI configuration.  (That is, if the problem was the FIFO then it would only affect the first few bytes on the line, not the whole transfer.)
...Wait just a minute.  The slave_receive buffer is filled with 0xff bytes.  Did the ordering of receive and transmit get messed up somewhere in the different chips' dma_transfer() functions, I wonder?  I don't see anything in the code... but that's a second thing to try: what does the LA capture (with the line connected to the master again) if you try the esp32s2 with this change, to use cc instead of ff?

Now MISO is constantly 0 and even when I change the init value back to 0xff it's just 0. So, it seems the peripheral just clocks out random garbage
I remember ESP32 and ESP32-S2 were a pain also for master mode with DMA. But don't remember the details t.b.h.
Weird stuff

So, it seems the peripheral just clocks out random garbage

Is it possible that the peripheral doesn't clock out anything instead?

So, it seems the peripheral just clocks out random garbage

Is it possible that the peripheral doesn't clock out anything instead?

possible - but at least it changes the logic level during the transfer and gets back to high in the end

Yes it's clear that it reacts to CS changes, so at least that's working :) hard to tell anything else without hardware to test things, but it feels to me like the peripheral just doesn't react to the clock input

adding this to the bit-banging loop:
            unsafe {
                let spi2 = &*esp32s2_hal::peripherals::SPI2::PTR;
                println!("{:032b}", spi2.dma_int_raw.read().bits() );
            }

prints this
00000000000000000000000000000000
00000000000000000000010000000000
00000000000000000000010000000000
.....

An empty outfifo.  Hmm.  That would correspond to not being able to read the descriptors properly, although that would surprise me.  Or maybe ... hmm, @bugadani, you mention enable_dma not doing anything.  It doesn't, which makes me re-read the TRM ... there's a blurb in there about setting the spi_dma_rx_ena and spi_dma_tx_ena bits in the in_link and out_link registers.  It doesn't say when to set those, other than "during the transmission".  I can assume that means the outlink_start and inlink_start bits need to be set first, and {rx,tx}_ena right after ... let's try that maybe.
That won't affect the ESP32, as it doesn't have those bits.  (And actually I'm surprised this didn't affect the SPI master mode, if it's the problem.)  But let's see what happens; I can try tracking down ESP32 later.

Unfortunately, no difference

Still trying to figure out what else it might be... It seems likely to be related to the pdma (instead of gdma) peripheral in the esp32 and s2 ... but it's not clear where the difference is.
Here's a question: Does the SPI loopback DMA example still work on the s2 chip with this change? I wonder if I broke pdma somehow. If it doesn't work here, I assume it does work at head, right?
If the master does work, I'll start digging through the differences between this code and the master side. Maybe I'll start that anyway, actually...

spi_loopback_dma still works on S2 but the changed variable (i to n) causes it to print different numbers
e.g. here
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 0]
[b, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, b]
[16, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 16]
[21, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 21]

vs main
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 0]
[1, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 1]
[2, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 2]
[3, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 3]
[4, 1, 2, 3, 4, 5, 6, 7, 8, 9] .. [73, 74, 75, 76, 77, 78, 79, 7a, 7b, 4]

Sigh, that's a pretty obvious thinko -- I'm incrementing i in the inner loop instead of n.  It's also a pretty easy fix.
I haven't had any time to dig into this recently; going to try a few more things here shortly though.  I think the rebase above ought to fix the spi_loopback_dma example again though.

What's the status on this? I'd really like to get it merged if possible, this PR has been open for ~4 months now. If you're not able to complete the work here I'm sure we can find somebody else to bring it across the finish line.

I've been super distracted with a lot of other stuff lately.  :(  I just managed to rebase it tonight, but yeah, it's been several weeks since I really had time to dig into the -s2 device's issues.  (And it seems the rebase is off somehow ... oh, right, I dropped the frequency arguments; at least that's relatively easy to fix.)
What do you think about disabling this module on esp32 and esp32s2, dropping the conditionals for those chips in spi_slave.rs, and merging it as-is?  Then, yeah, others can sort out the support for those two chips.  That would work for me.

What do you think about disabling this module on esp32 and esp32s2

I would be fine with that - we did similar things before

OK, I'm making slow progress on that -- still need to go through spi_slave.rs and drop cfg checks.
...Or actually, maybe not -- leaving it as-is means whoever follows up has less to do.  :)  I'll still take another look through before another review, though.  (I think maybe the DMA prepare_transfer split could be used by other peripherals too, and drop the compat fn that's still there now.)  Then I'll still have to squash down to 3 commits before merging.

 I could go back and rename the fn again, but my concern is that leaving it as-is wouldn't let others know the contract had changed.
I'll need to test it again, I think, and will only get a chance in a couple days.  But let's see what people think before too much more time passes.
Thank you!

I was able to test on all supported targets. It worked fine
However, on H2 I needed to change pins (since some are not available)
4 instead of 6, 12 instead of 7, 11 instead of 8
Generally, looks good to me.
One thing I'm not sure about: usage of the .and_then(|_| ....)? pattern.
We don't have it in the code base and in optlevel 0 it produces slightly more code than calling the two functions like function1()?; function2()?;
@jessebraham what do you think about that?

Rebased onto head, and fixed the pins.
I think the and_then construction reads better than the double question mark ("do this and then do that" makes sense to me) - but if opt-level zero is important, let me know and I'll switch it back. I'm sure the difference is just that the callback isn't being inlined.
Thanks!

I guess opt-level 0 is not really important - you can see the difference in https://godbolt.org/ ... the difference is also not huge.
It's 99.9% a matter of taste and consistency - I'm fine with it as it but would really like to hear a third opinion before merging
Thanks

Totally fair. I'll change it back if desired; let me know.

@jessebraham @MabezDev any opinions on the usage of the ?.and_then(|_| ...)?; pattern - if not I guess this is fine to merge


[BUG]: error: type "user_role[]" does not exist when run push for array enum

What version of drizzle-orm are you using?
0.29.1
What version of drizzle-kit are you using?
0.20.6
Describe the Bug
Hello Im trying to create table has column role with enum array, but I get error
error: type "user_role[]" does not exist
I am not sure why I have this error. Thank you for any help
export const userRoleEnum = pgEnum("USER_ROLE", ["ADMIN", "USER"]);

export const users = pgTable("user", {
    id: text("id")
        .notNull()
        .primaryKey()
        .$defaultFn(() => createId()),
    ....
    image: text("image"),
    role: userRoleEnum("role").array().notNull().default(["USER"]), // this make error
    ....
    state: userStateEnum("state").notNull().default("ACTIVE"), // this fine
});

Expected behavior
table has column enum array
Environment & setup
No response

Closed as duplicate of #1564. But thanks for reporting


Different time vectors for different exp. conditions

The time in the Plos Comp Bio measurement data differs slightly between conditions. It would be nice if DisFit would support that. More generally speaking, it may sometimes occur that each experiment and each observable has it's own time vector. In an attempt to support that sort of flexibility in time specification, I wrote the following code:
20200822_JuMP indexing issue.txt
Running the code results in the following error in line 153:
ERROR: MethodError: no method matching size(::JuMP.Containers.SparseAxisArray{Float64,2,Tuple{Int64,Int64}})
Closest candidates are:
  size(::AbstractArray{T,N}, ::Any) where {T, N} at abstractarray.jl:38
  size(::BitArray{1}) at bitarray.jl:70
  size(::BitArray{1}, ::Any) at bitarray.jl:74

To break things down a bit, the root case of the problem is that JuMP uses JuMP.Containers.SparseAxisArray instead of Array once the second index depends of the first like so:
# Define observables
println("Defining observables...")
@variable(m, 0.2 <= obs_a[j in 1:1, k in 1:length(t_exp[j])] <= 0.2, start=1.)
@NLconstraint(m, [j in 1:1, k in 1:length(t_exp)], obs_a[j, k] == A[j, t_sim_to_exp[k]])
@variable(m, 0.8 <= obs_b[j in 1:1, k in 1:length(t_exp)] <= 0.8, start=1.)
@NLconstraint(m, [j in 1:1, k in 1:length(t_exp)], obs_b[j, k] == B[j, t_sim_to_exp[k]])

julia> typeof(obs_a)
JuMP.Containers.SparseAxisArray{VariableRef,2,Tuple{Int64,Int64}}

julia> typeof(obs_b)
Array{VariableRef,2}

Anyway, in the end I would like to have a Dict of Dicts, where I can put in the simulation value of each observable in each condition like so: results_observable['condition_x']['observable_y'] = [1,2,3,4,5]
@sshin23 : Do you have a suggestion of how this can be achieved? Also, if you look at the code above, do you think this sort of flexible indexing will slow down JuMP substantially? Do you have an idea how to make the Julia code more elegant?
Thanks a lot!

Hi Paul,
Replacing Array(JuMP.value.(o))  to Dict(key=>val for (key,val) in o.data) should fix the issue; it will create a nested dictionary where the inner one has type Dict{Tuple{Int,Int},Float64}.
Once a JuMP model is created and passed to Ipopt, Ipopt.jl transcribes the data into a format that can be efficiently handled by Ipopt. So SparseAxisArray format won't significantly slow down the optimization process. The current code looks fine to me.

Thanks. Seems to work. Will try to implement it and push a new version when it is done.

I just looked into that a bit deeper, but this is not exactly doing what I want. I need observable_values to be a nested dict of the same structure as m_exp (except that it contains simulated rather than measured observable values). I am copying in an updated code (now for petab_test_suite case 0002, to better showcase the problem) below:
using CSV
using DataFrames
using Ipopt
using JuMP

n_steps = 101 # Setting number of ODE discretisation steps

# Data
println("Reading measurement data...")
data_path = "/media/sf_DPhil_Project/Project07_Parameter Fitting/df_software/petab_test_suite/cases/0002/_measurements.tsv"
df = CSV.read(data_path; use_mmap=false)
insert!(df, 1, (1:length(df[:,1])), :id)

df_by_o = groupby(df, :observableId)
df_by_c = groupby(df, :simulationConditionId)

# Setting variables
t_sim = range(0, stop=maximum(df[!, :time]), length=n_steps)
t_exp = Dict()
m_exp = Dict()
t_sim_to_exp = Dict()
deduplicated_time_idx = Dict()

cond2idx = Dict()
for i_cond in 1:length(keys(df_by_c))
    cond_name = values(keys(df_by_c)[i_cond])[1]
    cond2idx[cond_name] = i_cond
end

for i_obs in 1:length(keys(df_by_o))
    obs_name = values(keys(df_by_o)[i_obs])[1]
    t_exp[obs_name] = Dict()
    m_exp[obs_name] = Dict()
    t_sim_to_exp[obs_name] = Dict()
    deduplicated_time_idx[obs_name] = Dict()
    df_by_o_c = groupby(DataFrame(df_by_o[i_obs]), :simulationConditionId)
    for i_cond in 1:length(keys(df_by_o_c))
        cond_name = values(keys(df_by_o_c)[i_cond])[1]
        cond_idx = cond2idx[cond_name]
        t_exp[obs_name][cond_idx] = df_by_o_c[i_cond][!, :time]
        m_exp[obs_name][cond_idx] = df_by_o_c[i_cond][!, :measurement]

        t = df_by_o_c[i_cond][!, :time]
        tmp = []
        for i in 1:length(t)
            idx = argmin(abs.(t[i] .- t_sim))
            append!(tmp, idx)
        end
        t_sim_to_exp[obs_name][cond_idx] = tmp

        deduplicated_time_idx[obs_name][cond_idx] = []
        idx = 0
        prev = "a"
        for t in df_by_o_c[i_cond][!, :time]
            if t != prev
                idx = idx + 1
            end
            push!(deduplicated_time_idx[obs_name][cond_idx], idx)
            prev = t
        end
    end
end

obs2conds = Dict()
for obs_dict in t_exp
    obs_name = obs_dict[1]
    obs2conds[obs_name] = []
    for cond_dict in obs_dict[2]
        cond_idx = cond_dict[1]
        push!(obs2conds[obs_name], cond_idx)
    end
end

results = Dict()
results["objective_value"] = Dict()
results["parameters"] = Dict()
results["species"] = Dict()
results["observables"] = Dict()

j_to_cond_par = [1, 2]
cond_without_preequ = [1, 2]
cond_with_preequ = []
preequ = []

i_start = 1
    m = Model(with_optimizer(Ipopt.Optimizer))

    # Define global parameters
    println("Defining global parameters...")
    @variable(m, 0 <= b0 <= 10, start=0+(10-0)*rand(Float64))
    @variable(m, 0 <= k1 <= 10, start=0+(10-0)*rand(Float64))
    @variable(m, 0 <= k2 <= 10, start=0+(10-0)*rand(Float64))

    # Define condition-specific parameters
    println("Defining condition-specific parameters...")
    @variable(m, a0[1:2])
    @constraint(m, a0[1] == 0.8)
    @constraint(m, a0[2] == 0.9)

    # Define overrides
    # Define observable overrides
    println("Defining observableParameter overrides...")

    # Define noise overrides
    println("Defining noiseParameter overrides...")

    # Model compartments
    println("Defining compartments...")
    @variable(m, compartment == 1.0, start=1.0)

    # Model species
    println("Defining species...")
    @variable(m, 0.0 <= A[j in 1:2, k in 1:(length(t_sim)+1)] <= 8.8)
    @variable(m, 0.0 <= B[j in 1:2, k in 1:(length(t_sim)+1)] <= 8.8)

    # Model initial assignments
    println("Defining initial assignments...")
    @constraint(m, [j in 1:2], A[cond_without_preequ[j],1] == a0[cond_without_preequ[j]])
    @constraint(m, [j in 1:2], B[cond_without_preequ[j],1] == b0)

    # Model ODEs
    println("Defining ODEs...")
    println("A")
    @NLconstraint(m, [j in 1:2, k in 1:length(t_sim)-1],
        A[j, k+1] == A[j, k] + ( -1.0*( compartment * k1 * A[j, k+1] ) +1.0*( compartment * k2 * B[j, k+1] )     ) * ( t_sim[k+1] - t_sim[k] ) )
    println("B")
    @NLconstraint(m, [j in 1:2, k in 1:length(t_sim)-1],
        B[j, k+1] == B[j, k] + ( +1.0*( compartment * k1 * A[j, k+1] ) -1.0*( compartment * k2 * B[j, k+1] )     ) * ( t_sim[k+1] - t_sim[k] ) )

    # Pre-equilibration constraints
    @constraint(m, [j in 1:2], A[j, length(t_sim)+1] == 0) # Dummy preequilibration
    @constraint(m, [j in 1:2], B[j, length(t_sim)+1] == 0) # Dummy preequilibration

    # Define observables
    println("Defining observables...")
    @variable(m, -0.6000000000000001 <= obs_a[j in obs2conds["obs_a"], k in 1:length(t_exp["obs_a"][j])] <= 1.5, start=1.)
    @NLconstraint(m, [j in obs2conds["obs_a"], k in 1:length(t_exp["obs_a"][j])], obs_a[j, k] == A[j, t_sim_to_exp["obs_a"][j][k]])

    # Define objective
    println("Defining objective...")
    @NLobjective(m, Min, sum(0.5 * log(2*pi*(( 1 ))^2) + 0.5*((obs_a[j, deduplicated_time_idx["obs_a"][j][k]]-m_exp["obs_a"][j][k])/(( 1 )))^2 for j in obs2conds["obs_a"] for k in 1:length(t_exp["obs_a"][j]))
        )

    println("Optimizing:")
    optimize!(m)

    println("Transfering results to Python...")
    parameter_names = [b0, k1, k2]
    parameter_values = Dict()
    for p in parameter_names
        if occursin("[", string(p))
            parameter_values[split(string(p[1]), "[")[1]] = JuMP.value.(p)
        else
            parameter_values[string(p)] = JuMP.value.(p)
        end
    end

    species_names = [A, B, ]
    species_values = Dict()
    for s in species_names
        species_values[split(string(s[1]), "[")[1]] = JuMP.value.(s)
    end

    observable_values_v1 = Dict()
    for o in observable_names
        observable_values_v1[split(string(o[1,1]), "[")[1]] = Dict(key=>val for (key,val) in o.data)
    end

    observable_names = [obs_a, ]
    observable_values_v2 = Dict()
    for o in observable_names
        tmp = []
        results = JuMP.value.(o)
        for val in results
            push!(tmp, val)
        end
        observable_values_v2[split(string(o[1,1]), "[")[1]] = tmp
    end

    objective_val = objective_value(m)

    results["objective_value"][string(i_start)] = objective_val
    results["parameters"][string(i_start)] = parameter_values
    results["species"][string(i_start)] = species_values
    results["observables"][string(i_start)] = observable_values

results

Unfortunately, the structure of the observable_values_v1/2 dictionaries does not match the structure of m_exp.
julia> m_exp
Dict{Any,Any} with 1 entry:
  "obs_a" => Dict{Any,Any}(2=>[0.8, 0.2],1=>[0.7, 0.1])

julia> observable_values_v1
Dict{Any,Any} with 1 entry:
  "obs_a" => Dict((1, 2)=>obs_a[1,2],(2, 2)=>obs_a[2,2],(1, 1)=>obs_a[1,1],(2, 1)=>obs_a[2,1])

julia> observable_values_v2
Dict{Any,Any} with 1 entry:
  "obs_a" => Any[0.143449, 0.16138, 0.8, 0.9]

The code below perhaps creates the dict of the wanted format.
observable_values = Dict(name=>Dict(t1=>[value(eval(Symbol(name))[t1,t2]) for t2=1:length(arr)]
                                    for (t1,arr) in dict) for (name,dict) in m_exp)

Thanks. This does the trick for the most part, except when I set n_starts to > 1. Then the indented block after i_start is enclosed in a for loop. This causes trouble, as eval only accesses the global scope. Can you think of a way to do this without eval?
One workaround is to promote variables to the global scope like so:
global obs_a = @variable(m, -0.6000000000000001 <= obs_a[j in obs2conds["obs_a"], k in 1:length(t_exp["obs_a"][j])] <= 1.5, start=1.)

Please let me know if this should be the solution of our choice or if you can think of a better way. Thanks!

One another work around is to create a dictionary in the beginning, say
var_dict=Dict()

and let
var["obs_a"] = @variable(m, -0.6000000000000001 <= obs_a[j in obs2conds["obs_a"], k in 1:length(t_exp["obs_a"][j])] <= 1.5, start=1.)

and then replace eval(Symbol(name)) with var[name].

Perfect. That did the job. I just pushed a new commit that includes your suggestion. Thank you very much!


CDAT Migration Phase 2: Refactor arm_diags set (try 2)

Description

Closes #667  to replace #834

Checklist

[ ] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings
[x] Any dependent changes have been merged and published in downstream modules

If applicable:

[ ] New and existing unit tests pass with my changes (locally and CI/CD build)
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

Updates:
All sub sets code refactoring including (annual_cycle, diurnal_cycle(_zt), convection_onset, aerosol_activation) are completed.
Todo:
Fix mypy issues
Potential performance bottleneck from deriving variables using xarray dataset as input

Potential performance bottleneck from deriving variables using xarray dataset as input

Thank you for identifying the two formulas code as potential bottlenecks. I confirmed that these computations with Xarray are indeed much slower than CDAT.
Solution
I ran a performance benchmark and found the solution to the slowness: we need to call .load(scheduler="sync") in _get_dataset_with_source_vars() to speed up the computation.
Benchmark Results
The first runtime is the current code and the second runtime is with .load(). I also ran e3sm_diags with commit ef44fc6 (#842)  and confirmed a significant runtime improvement, similar to the benchmarks below.
"""
Results
----------
1. Elapsed time (Xarray non-chunked): 6.540755605790764 seconds
2. Elapsed time (Xarray non-chunked with .load()): 0.17097265785560012 seconds
3. Elapsed time (Xarray chunked): 0.1452920027077198 seconds
4. Elapsed time (numpy .values): 6.418793010059744 seconds
5. Elapsed time (numpy .data): 7.334999438840896 seconds
"""

Benchmark code: https://github.com/E3SM-Project/e3sm_diags/pull/842/files#diff-3ae80750c261ea56336100d0606654f4480ddef2e4549044366c93b27196a051

The first runtime is the current code and the second runtime is with .load(). I also ran e3sm_diags with commit ef44fc6 (#842)

@tomvothecoder thank you for performing the timing test. Interesting that when I test your commit. Running through a configuration with each subset, it took 24 mins. Without, .load(), the total time is about 3 mins, which I consider at least on par with cdat code. In this case, maybe we should drop the .load() change?

The first runtime is the current code and the second runtime is with .load(). I also ran e3sm_diags with commit ef44fc6 (#842)

@tomvothecoder thank you for performing the timing test. Interesting that when I test your commit. Running through a configuration with each subset, it took 24 mins. Without, .load(), the total time is about 3 mins, which I consider at least on par with cdat code. In this case, maybe we should drop the .load() change?

I would think that loading the derived variables dataset into memory shouldn't slow down performance unless the datasets were extremely large (which we should use Dask chunking for).
I only benchmarked performance for the formula computations. I will benchmark a complete run to verify your findings and determine if we should revert the commit or not.
Side-note:
It could be that the logic I implemented already stores the dataset in-memory, since it merges multiple xr.Dataset objects opened via open_dataset() (uses numpy arrays) instead of using open_mfdataset() (uses Dask arrays). If this the case, I don't see how .load() would improve the speed of the formula computations though.
https://github.com/E3SM-Project/e3sm_diags/blob/ef44fc6ffd538a5e257b097b99f7a1a79b79bc3b/e3sm_diags/driver/utils/dataset_xr.py#L1056-L1064

RE: https://github.com/E3SM-Project/e3sm_diags/pull/842#issuecomment-2344166562
I found adding .load() in _get_dataset_with_source_vars() adds 2-4 minutes of runtime to a complete arm_diags run.
I reverted this change and will now address the pre-commit issues.
# Commit: 58361c49-4b1b-11ec-9b3b-9c5c8e2f5e4e (no .load())
# run_set function took 281.78 seconds to complete.
# run_set function took 332.79 seconds to complete.
# Commit: ef44fc6ffd538a5e257b097b99f7a1a79b79bc3b (with .load())
# run_set function took 472.81 seconds to complete.

Hey @chengzhuzhang, I fixed the pre-commit issues in 2c248fb (#842). I also performed initial code cleanup since I would probably be doing that later anyways. Refer to the commit message and my review comments for more information.

Thank you @tomvothecoder I will do another pass to see if there are other sets need to be refactored. Thanks a lot for fixing and clean this branch.

@tomvothecoder I applied png regression tests. And all figures are produced and results are as expected as noted in the notebook.
Also performance-wise, the refactored codes are similar to the original codes. wall time is ~ 10mins with 4 workers for the full arm_diags set.

Hey @chengzhuzhang, I noticed you have several other PRs you're working on right now. I'm happy to help finish up refactoring the last diag function in this PR. Let me know.

Hey @chengzhuzhang, I noticed you have several other PRs you're working on right now. I'm happy to help finish up refactoring the last diag function in this PR. Let me know.

@tomvothecoder it would be great if you could help me on finish this last diag, so that I'm not holding back the progress to merge!

Hey @chengzhuzhang, I noticed you have several other PRs you're working on right now. I'm happy to help finish up refactoring the last diag function in this PR. Let me know.

@tomvothecoder it would be great if you could help me on finish this last diag, so that I'm not holding back the progress to merge!

I found that no sets actually run "annual_cycle_aerosol" (arm_diags_model_vs_obs.cfg, arm_diags_model_vs_model.cfg). Unless we expect to run this diagnostic in the future, I think we can delete _run_diag_annual_cycle_aerosol instead of refactoring it.

@tomvothecoder good for catching this. I think for now we can just create an issue to log this problem, and we can add back the code at a later time.

Regression test results
.png regression test shows all plots are identical except the following, which have missing data (white spaces) compared to main:

[ ] TODO: Investigate the following plots because they have white spaces on dev branch for some reason.

Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/arm_diags/armdiags-CLOUD-ANNUALCYCLE-nsac1-test.png
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags/armdiags-CLOUD-ANNUALCYCLE-nsac1-test.png
     * Difference path /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags_diff/armdiags-CLOUD-ANNUALCYCLE-nsac1-test.png
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/arm_diags/armdiags-CLOUD-ANNUALCYCLE-sgpc1-test.png
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags/armdiags-CLOUD-ANNUALCYCLE-sgpc1-test.png
     * Difference path /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags_diff/armdiags-CLOUD-ANNUALCYCLE-sgpc1-test.png
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/arm_diags/armdiags-CLOUD-ANNUALCYCLE-twpc1-test.png
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags/armdiags-CLOUD-ANNUALCYCLE-twpc1-test.png
     * Difference path /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags_diff/armdiags-CLOUD-ANNUALCYCLE-twpc1-test.png
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/arm_diags/armdiags-CLOUD-ANNUALCYCLE-twpc2-test.png
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags/armdiags-CLOUD-ANNUALCYCLE-twpc2-test.png
     * Difference path /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags_diff/armdiags-CLOUD-ANNUALCYCLE-twpc2-test.png
Comparing:
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/arm_diags/armdiags-CLOUD-ANNUALCYCLE-twpc3-test.png
    * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags/armdiags-CLOUD-ANNUALCYCLE-twpc3-test.png
     * Difference path /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/667-arm_diags-final/arm_diags_diff/armdiags-CLOUD-ANNUALCYCLE-twpc3-test.png

This PR should be good to go. The integration test image checker fails because there is now one additional plot for the sgpc1 region that was generated with a fix in this PR.
I will merge this PR even though the integration tests fail. I will update the complete test results on LCRC using the cdat-migration-fy24 branch. The complete test results should also include the QBO wavelet plot from #850.


[MINOR] Use SL4J string interpolation instead of string concatenation

@hachikuji adding on to your pile of reviews ;-)
This one is only two lines of change so should hopefully be quick.

@ijuma is the goal to look through the entire code base for such usage?

No, I just meant in the PR where this was introduced.

@ijuma I missed replying to this earlier. Those are the only two places with such usage in the chunked conversion PR.


Train/Inference between unevenly matched replicas

Hello,
I'm Ilya from The Stone Cross Foundation of Ukraine, a data-driven humanitarian volunteer cooperative tasked with solving some of the issues charities in Ukraine are having during these tumultuous times. We've discovered this project via HN a couple weeks back and have been evaluating it ever since. We're using a high-availability variant of TimescaleDB which is another Postgres extension, providing time-series and some advanced capabilities, so called hypertables and hyperfunctions. This is where we keep all our data, including but not limited to geospatial, analytics and everything related to day-to-day operations of our charity, the pleas and the volunteers we're working with.
For us, the primary area of interest is the following: How does postgresml store the trained models, and can these models be streaming-replicated to the other replicas in order to perform inference afterwards? Our database instances are not particularly beefy on the CPU side, and it wouldn't be cost-effective to have them so, but we would really love to perform some regressions on our data in the following way:

Provision a training replica on a highly-performant VM from the latest backup.
(Inquiry: Does pgml's backend implementation in Python support GPU devices?)
Give it time to get up to speed with the latest changes, WAL, etc.
Construct a materialised corpus of our data to be used in training.
pgml.train()
Have other replicas pick it up and perform inference on it.
The training replica can be terminated until the next time we would want to re-train the model.

I'm guessing whether if this approach is even possible, if not too far-out?
This could potentially significantly simplify our data analysis pipeline. Instead of having to worry about ingest, provisioning of the dedicated machine learning services, their inference APIs exposure, and many other things required to do basic machine learning, we could just play around with our Postgres-derived replicas and perform these scheduled training sessions without having to do any of that!
In our case this could translate to relief of actual human suffering.
Best regards,
Ilya

Your approach is one we've planned for, and should work, although it has not been tested in a production environment that I know of. I'd be happy to support your cause and help with any issues you come across.
Another option rather than replicating the entire DB is to use logical replication on the subset of tables required, or you could also create foreign data wrappers on your training server back to your host server to pull the data on demand without having to replicate at all. pgml.train creates a pgml.snapshot of the table when it is called with a relation name, so you could even test multiple algorithms or hyperparams with a single pull over FDW from the master dataset.
The algorithm will be a significant factor in the performance benefits of offloading to a second server from the master. For algorithms like linear regression my experience is that scanning the data from disk creates the majority of load on the server, the actual computation for those algorithms is small relative to the data movement from storage to RAM. For a large XGBoost model, then I agree, its worth at least testing the performance gains on a second worker. We need to add more metrics and visibility for where time is spent during training, so I'll prioritize that on our roadmap if it will help you.
The serialized models are stored as BYTEA pickled objects in pgml.models table, which should replicate the same as other tables. You should also be able to pg_dump the tables from one server to another if you want to avoid replication, or create backups. You'll also want to replicate pgml.projects and pgml.deployements. You may wish to drop the pgml.snapshots from replication back to the inference servers, since the snapshots are only used during training and later for analysis. We're working on our documentation to clarify this further, but the basics are outlined: https://postgresml.org/references/models/
Does pgml's backend implementation in Python support GPU devices?
AFAIK None of the scikit algorithms support GPU devices. XGBoost does: https://xgboost.readthedocs.io/en/stable/gpu/index.html
You'll need to ensure that the xgboost libraries installed on the PostgreSQL server are setup correctly, and pass the tree_method: gpu_hist argument.

Thank you for a quick response, I'm very glad to know that we can rely on your support!
PostgresML is perhaps the most interesting piece of postgres-related software I've come across in months. We would like to add it to our database container build, that is based on timescaledb-ha perhaps having the the built-in dashboard would prove beneficial. Wouldn't you happen to know how we should approach this? I took a look at pgml-extension's Dockerfile and perhaps we could make it part of our multi-stage build, and just copy the necessary files? I can imagine these are the pip3-provided packages and the extension build itself. We use Postgres 14, would pgml support that as well? The existing container seems to rely on 13, so perhaps our multi-stage build would have to be more involved. I'm not a Docker guru, and in this case it's not immediately clear what the final composition should look like.

You should also be able to pg_dump the tables from one server to another if you want to avoid replication, or create backups

This is perfect! At any rate, it's unlikely that we're going to be running these "large" xgboost models, the reason why I asked about GPUs was basically to learn whether if there are any inherent limitations in pgml itself that would not allow the underlying backend (xgboost) to use the available hardware. However, for our case it's almost irrelevant, as we're likely to only rely on the lightweight regressions, and the training sessions can be scheduled with pg_cron on our regular master nodes.
Basically, there are three types of regressions we would want to perform:
(1) Predict the time-to-fund (for a plea request to reach target funding) based on the time it has already spent since it was published, the number of views and the amount of donations it accumulated during this time.
(2) Predict what plea requests would need extra exposure, ie. currently we do this using the following aggregate formula:
    SELECT
        (1+funded/100) * log(t) / coalesce(nullif(n, 0), 1) *
        log(extract(epoch FROM now()-published_at)) as score,
        id
    FROM plea_view pv
    LEFT JOIN LATERAL (
        SELECT
            count(*) as n,
            coalesce(sum(extract(epoch FROM dt)), 1) as t
        FROM panorama pr
        WHERE pr.plea_id = pv.id
          AND dt is not null
        ) pr ON true

This formula is meant to promote the requests that have accrued relatively more donations in spite of relatively small number of views. (Thus, proving that they are potentially worthwhile.) Then we pick the requests for the frontpage "carousel" in designated proportions out of the score distribution (half of them come from p95, another quarter from p75, and another from p50) this way the selection can have somewhat of a "random" feel and won't discriminate for the super-performers.
We're looking to perhaps add sophistication to how this score is calculated, using a regression model.
(3) The requests are distributed among the volunteers that have them in their "working area", i.e. the volunteer will see a request in their pool if the applicant is in their geographical vicinity where they can operate. But with the number of requests growing, we're afraid that this will potentially overwhelm the volunteers, so perhaps we could do better than that— by promoting certain pleas in the pool for certain volunteers, if we deem them a good fit for it. So potentially we would want to extract some features out of the successfully completed requests to perhaps correlate these request features with the volunteers who've completed them. This way we could potentially "recommend" more requests like ones that a volunteer has completed in the past, by category, more precise geographic location, or some tf-idf keywords.
Do you believe pgml would be a good fit for this?

Installation
To install Postgres in Linux the standard instructions should work:
https://postgresml.org/guides/installation/#install-postgresql-with-plpython
You'll need to add those lines to your container setup, e.g.  for debian/ubuntu
# Install Postgres with  PL/Python
$ sudo apt-get install -y postgresql-plpython3-12 python3 python3-pip postgresql-12

# Install PostgresML in the database (after it is running)
$ sudo pip3 install --install-option="--database-url=postgres://user_name:password@localhost:5432/database_name" pgml-extension

Modelling
These sound like great ideas. I generally start with a linear model to see where the predictive baseline is, since building that model is usually in the same order of magnitude as a SELECT count(*) FROM ... on normal PostgreSQL. I'm not sure how much performance characteristics will change on TimescaleDB, but I'd love to see your performance with \timing on.
Roughly how many rows would your training sets have? For sub 100k training samples, I think a linear model should train on the order of seconds, but it'll depend on your disk IO speed or if everything is already resident in memory. I also find it helpful to work with a subsampling during algorithm selection if there are more than 1-10M rows in the full set and survey many different algorithms/hyperparameters. At the scale of 10M+ there may be enough data for deep learning to start to overtake more efficient algorithms in performance. Another thing on our backlog is https://neuralprophet.com/html/index.html for time series predictions, which I thought might help since you're using timescaledb, but it's not immediately clear to me that would be a good algorithm for any of these use cases.
I'm not sure if you've seen the regression example code.

We've just added some docs on how to use production data with PostgresML. Please let us know how it works out for you.

I think we've added answers to your questions in the documentation. Please don't hesitate to re-open or create a new issue if you have additional questions.

Thank you! I've just wanted to come by and say that PostgresML is the best thing that occurred to us.
Now that we've had limited success with PostgresML in our charity, I'm itching to apply it everywhere I set my foot!
What would you say is currently your biggest weakness, or perhaps, most underdeveloped area that you think is preventing wider adoption in the machine learning community? You have here a beautiful project, and a gateway drug, I think, for the kind of shops that have a BI team already, but for the time being— don't yet have a dedicated machine learning team— due to lack of initiative, or whatnot. (Like my company!) The barrier to entry is just so low, it seems anybody can simply pick it up, and suddenly produce insights out of thin air. I'm not a ML guy at all, but I can appreciate a good regression, and to me tools like it are positively liberating.
BTW, would you welcome a PR introducing one or more survival analysis models such as IPCRidge or Cox's PH— these can be used to predict for a particular event in the data, if and when it would happen, or not, at a given moment in time. I'm by no means a ML specialist or anything, but this is something one of my peers have played with in the past, and I couldn't help but wonder if we could apply this to our problem set in the charity, to guess whether if a particular supply link of our vendor networks is less or likely to fail given the data that have been accumulated by our partners and us.
Please let me know if I'm missing something, or if postgresml is already capable of this.

I'm glad you're finding success on your mission! I think the biggest weakness is simply awareness of the capability vs "the way things are done". PostgresML is still a pretty young project. It'd go a long way to have case studies we can discuss publicly. We'd love to do a writeup of how you're using it, and link to your charity to increase visibility there as well. Do you want to connect w/ me over email to coordinate? I'm<EMAIL_ADDRESS>I think the two algorithms you've linked look like good candidates for the regression API, and we'd welcome PRs to introduce new libraries/aglos/capabilities.


Type error: 'SimpleBar' cannot be used as a JSX component.

I am getting an error during the NextJS build process like in below:
Type error: 'SimpleBar' cannot be used as a JSX component.
I am using ^2.3.6 as simplebar-react version.
How can I fix this?

I have the same issue any workaround ?

As a hack fix I renamed the .ts file (node_modules/simplebar-react/dist/simplebar-react.d.ts) to .tsx.
I've added this to our build as an intrim fix:
[ -f "node_modules/simplebar-react/dist/simplebar-react.d.ts" ] && mv "node_modules/simplebar-react/dist/simplebar-react.d.ts" "node_modules/simplebar-react/dist/simplebar-react.d.tsx"
For me it was compiling locally with yarn but not npm (which is what our Dockerfile was using).
I guess ideally there's a way to specify the .ts file to be assumed to be JSX somehow?

I don't understand this issue. This used to work just fine AFAIK and we didn't change anything. I saw some people complaining about something like this with the latest version of @types/react could it be related?

Yes we didn’t change anything at our end either… I suspect it was a point release change in (about a caret ^ version number in package.json) but haven’t had chance to look further.

On 24 May 2022, at 01:14, Adrien Denat @.***> wrote:
I don't understand this issue. This used to work just fine AFAIK and we didn't change anything. I saw some people complaining about something like this with the latest version of @types/react could it be related?
—
Reply to this email directly, view it on GitHub https://github.com/Grsmto/simplebar/issues/604#issuecomment-1135262734, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNWTVD3XEJSW7PT44FNXSLVLQNQHANCNFSM5S2IVRGQ.
You are receiving this because you commented.

I have the same issue, full error text (maybe it will help):

TS2786: 'SimpleBar' cannot be used as a JSX component.
  Its instance type 'SimpleBar' is not a valid JSX element.
    The types returned by 'render()' are incompatible between these types.
      Type 'React.ReactNode' is not assignable to type 'import("./node_modules/@types/react-transition-group/node_modules/@types/react/index").ReactNode'.
        Type '{}' is not assignable to type 'ReactNode'.
          Type '{}' is missing the following properties from type 'ReactPortal': key, children, type, props

Here's a workaround for people looking for a solution:
I installed react-beautiful-dnd and also got the same error:
'SimpleBar' cannot be used as a JSX component.
  Its instance type 'SimpleBar' is not a valid JSX element.
    The types returned by 'render()' are incompatible between these types.
      Type 'React.ReactNode' is not assignable to type 'import("./node_modules/@types/react-beautiful-dnd/node_modules/@types/react/index").ReactNode'.
        Type '{}' is not assignable to type 'ReactNode'.

I managed to track the error and this dependency introduced types for react in v18. For context, I'm using Typescript with React 17 since I have some dependencies that are not fully compatible with v18. This is the command output:
npm ls @types/react
<EMAIL_ADDRESS><PROJECT_PATH>
├─┬<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>extraneous
│ └──<EMAIL_ADDRESS>extraneous
├─┬<EMAIL_ADDRESS>
The same package I installed also installed the types for React 18, so the only thing to be done is to override this dependency when installing. In npm is done by forcing the dependency version in the overrides section in the package.json file. I'm using yarn v1 so the directive is called resolutions.
After modifying the package.json file, I restarted VS Code and the error is now gone.

Here's a workaround for people looking for a solution: I installed react-beautiful-dnd and also got the same error:
'SimpleBar' cannot be used as a JSX component.
  Its instance type 'SimpleBar' is not a valid JSX element.
    The types returned by 'render()' are incompatible between these types.
      Type 'React.ReactNode' is not assignable to type 'import("./node_modules/@types/react-beautiful-dnd/node_modules/@types/react/index").ReactNode'.
        Type '{}' is not assignable to type 'ReactNode'.

I managed to track the error and this dependency introduced types for react in v18. For context, I'm using Typescript with React 17 since I have some dependencies that are not fully compatible with v18. This is the command output:
npm ls @types/react
<EMAIL_ADDRESS><PROJECT_PATH>
├─┬<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>extraneous
│ └──<EMAIL_ADDRESS>extraneous
├─┬<EMAIL_ADDRESS>
Now the only thing to be done is to override this dependency when installing. In npm is done by forcing the dependency version in the overrides section in the package.json file. I'm using yarn v1 so the directive is called resolutions:
 "resolutions": {
    "@types/react": "17.0.39"
  },

After modifying the package.json file and running yarn install, I restarted VS Code and the error is now gone.

It worked

Ran into similar issue here. Since simplebar-react is just a small (single file) wrapper around simple bar-core, I added the code directly to my project, removed simplebar-react dep (and added simplebar-core), and works well.


Build script modification

Describe the changes you have made in this PR -

Modified build script for vite.config.ts and netlify.toml
Added version.json to store version metadata and details.

cc @Prerna-0202  @pulkit-30 @gr455


Error checking metric_aggregation rule

What have I done wrong?
name: "excp_aggregation"
type: metric_aggregation
index: log1c-*

buffer_time:
  hours: 1

metric_agg_key: "t:computerName"
metric_agg_type: value_count
doc_type: _doc
max_threshold: 5

filter:
- query:
    - query_string:
        query: "Descr: \"...................\""
- term:
    type: "log1c-excp"

alert:
  - "command"
....

Error running your filter: TypeError("search() got multiple values for argument 'body'",) 1 rules loaded INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts Traceback (most recent call last):   File "/usr/local/bin/elastalert-test-rule", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.6/site-packages/elastalert/test_rule.py", line 445, in main     test_instance.run_rule_test()   File "/usr/local/lib/python3.6/site-packages/elastalert/test_rule.py", line 437, in run_rule_test     self.run_elastalert(rule_yaml, conf, args)   File "/usr/local/lib/python3.6/site-packages/elastalert/test_rule.py", line 318, in run_elastalert     client.run_rule(rule, endtime, starttime)   File "/usr/local/lib/python3.6/site-packages/elastalert/elastalert.py", line 884, in run_rule     if not self.run_query(rule, rule['starttime'], tmp_endtime):   File "/usr/local/lib/python3.6/site-packages/elastalert/elastalert.py", line 631, in run_query     data = self.get_hits_aggregation(rule, start, end, index, rule.get('query_key', None))   File "/usr/local/lib/python3.6/site-packages/elastalert/elastalert.py", line 565, in get_hits_aggregation     body=query, size=0, ignore_unavailable=True)   File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped     return func(*args, params=params, headers=headers, **kwargs) TypeError: deprecated_search() got an unexpected keyword argument 'headers'

We also recommend asking questions on the Gitter channel
https://gitter.im/Yelp/elastalert

@LazarenkoA Have you got the solution/workaround?

yes, use other software


Django 1.9 compatibility

Thanks @olarin!
{% load url from future %} is not needed since Django 1.5… We can probably drop them completely, what do you think?

I think, we can. Support of Django 1.4 was ended 5 moths ago.

Great. Do you want to update this pr?

Done.

:tada: Thanks @olarin!

0.3.0 is up on PyPI.


RabbitMQ Engine

I thought about this a little and I'd say for now let's do the following:

deleting a config is a no-op on Vault, it of course deletes the object in Kube
in the validation webhook, let's ensure that someone cannot define two config for the same reabbit mq engine endpoint endpoint

Resolved comments. Updated controller to utilize single Vault object and added creation validation for RabbitMQ configs.


Add VCR.py to mockup test API calls, and add test runner for setup.py

Fixes #9 Fixes #8

Thanks for this man!


Errors during installation

Followed the below instructions:
git clone --depth 1 https://github.com/NativeScript/nativescript-sdk-examples-ng.git
cd nativescript-sdk-examples-ng
tns run android
Errors:
user@LAPTOP-1MS538L2 MINGW64 /e/allsamples/nativescript-sdk-examples-ng (master)
$ tns platform add android
Copying template files...
Installing  tns-android
/ Installing tns-android+<EMAIL_ADDRESS>added 22 packages, removed 23 packages and updated 397 packages in 48.323s
cp: no such file or directory: E:\allsamples\nativescript-sdk-examples-ng\node_modules\tns-android\framework\libs
platform add
┌────────────────────────
Please help.
Regards,
LH

@LHLK can you pen the package.json and see if there is a tns-android listed as a dependency.
if so - remove the entry and rebuild the project.

Hi NickIliev, great support as usual and it's working now.  BTW, in light of ease, could you point me to some resources regarding the css gradient (reference to : https://github.com/NativeScript/NativeScript/issues/539).  My client would like to have the Android button native gradient look like this: https://www.dibbus.com/2011/02/gradient-buttons-for-android/
Great support again & Many Thanks!
LH


Adding tox env for pubsub sys tests w / emulator.

Modeled after #1393.
Also fixes a small docs bug in CONTRIBUTING.

@tseaver PTAL. I re-factored the datastore_emulator script so it could be used for pubsub as well.

LGTM


Saved remote connection are not actually saved

Create remote connection
Shut down VS
[I uninstalled R here but it prob doesnt matter]
Open VS

No remote connections in the list

Happens because settings get restored after ConnectionManager is instantiated

This will be fixed by my upcoming PR for #2589

https://github.com/Microsoft/RTVS/commit/0e32d5ffd6bfb6ac69a669b8238846cb564f77e3

Actually still broken

https://github.com/Microsoft/RTVS/commit/1ee7574d76997eff0abad7bde52f4b2f3d1c6605