Plots no longer being moved to storage drive

Hoping someone could shed some light on this. I have had little issue with the plotter but now all of a sudden the files are staying on my ssd and not being put on storage drive. Before it always moved them between phase 1 table 3 and table 4. I haven't changed anything so not sure why this is occurring. Any help would be greatly appreciated.

can you post the output at beginning of plot

I get the same issue, my SSDs were 3.5tb filled out of 4tb combined, and plots were not moved after compleition.
I got an error copy to ........  failed with fopen() failed

i think i have the same issue

This is the options im using on a rizen 2200g
#!/bin/bash
sudo ./chia-plotter/build/chia_plot -n 1 -r 4 -t /home/the-farm/Desktop/plots_temp/ -d /media/the-farm/genesis/ -c  -f

Multi-threaded pipelined Chia k32 plotter - 7c6d195
Final Directory: /media/the-farm/genesis/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 14923
Number of Threads: 4
Number of Buckets P1:    2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Puzzle Hash:  8eddfb24851519245cedc4e6770fb20b79fd80a863a9fd1137d839427ed86eba
Farmer Public Key: 81944d9facaa8b3c0d21fcb9a3f2deb6c271b5ce6534c4207be6d27eee0b3be19f31a5635ea7d311fa4bc775a91fcd84
Working Directory:   /home/the-farm/Desktop/plots_temp/
Working Directory 2: /home/the-farm/Desktop/plots_temp/
Plot Name: plot-k32-2021-07-11-18-31-64415009d6e1366af7c8d128936b0e72c047fa4fec99148e44e9aa8b061422cb
[P1] Table 1 took 50.7742 sec
[P1] Table 2 took 429.812 sec, found<PHONE_NUMBER> matches
[P1] Table 3 took 494.751 sec, found<PHONE_NUMBER> matches
[P1] Table 4 took 572.672 sec, found<PHONE_NUMBER> matches
[P1] Table 5 took 565.645 sec, found<PHONE_NUMBER> matches
[P1] Table 6 took 535.018 sec, found<PHONE_NUMBER> matches
[P1] Table 7 took 431.151 sec, found<PHONE_NUMBER> matches
Phase 1 took 3079.87 sec
[P2] max_table_size =<PHONE_NUMBER>
[P2] Table 7 scan took 30.3431 sec
[P2] Table 7 rewrite took 79.6773 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 140.234 sec
[P2] Table 6 rewrite took 251.155 sec, dropped 581375843 entries (13.5357 %)
[P2] Table 5 scan took 134.632 sec
[P2] Table 5 rewrite took 244.911 sec, dropped 762153318 entries (17.7444 %)
[P2] Table 4 scan took 132.278 sec
[P2] Table 4 rewrite took 237.688 sec, dropped 829046095 entries (19.3017 %)
[P2] Table 3 scan took 131.763 sec
[P2] Table 3 rewrite took 235.627 sec, dropped 855214396 entries (19.9113 %)
[P2] Table 2 scan took 131.434 sec
[P2] Table 2 rewrite took 236.052 sec, dropped 865762966 entries (20.1568 %)
Phase 2 took 1990.96 sec
Wrote plot header with 252 bytes
[P3-1] Table 2 took 107.927 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 2 took 101.396 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 3 took 141.612 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 3 took 98.3201 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 4 took 144.687 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 4 took 99.5501 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 5 took 145.796 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 5 took 101.687 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 6 took 152.436 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 6 took 107.542 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 7 took 127.23 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 7 took 122.568 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
Phase 3 took 1452.24 sec, wrote<PHONE_NUMBER>0 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 133.596 sec, final plot size is<PHONE_NUMBER>90 bytes
Total plot creation time was 6656.71 sec (110.945 min)
Started copy to /media/the-farm/genesis/plot-k32-2021-07-11-18-31-64415009d6e1366af7c8d128936b0e72c047fa4fec99148e44e9aa8b061422cb.plot

Same here..
Problem occurs one plot before last one...
Im using stotiks plotter for windows

Multi-threaded pipelined Chia k32 plotter - 82f92a9
Build 0.1.1 for Windows. Check for latest updates: https://stotiks.github.io/chia-plotter/
Final Directory: G:
Number of Plots: 4
Crafting plot 1 out of 4
Process ID: 8780
Number of Threads: 6
Number of Buckets P1:    2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Puzzle Hash:  714edbece1589406da826c9df5ee34062b3e9871e7e62ca0d2db7a0ace990c07
Farmer Public Key: ac87f3b326905065413d8fb535f0ae8512d576bdd27f6f125e54ff170db7fcbfeff7823f83bcc4fb808c382871272c41
Working Directory:   D:
Working Directory 2: D:
Plot Name: plot-k32-2021-07-13-07-36-b9b74d34bbf065caaa19fe3f90fa55124281b00bee53c4d21cb759e87aac3784
[P1] Table 1 took 19.8534 sec
[P1] Table 2 took 229.581 sec, found<PHONE_NUMBER> matches
[P1] Table 3 took 270.283 sec, found<PHONE_NUMBER> matches
[P1] Table 4 took 354.237 sec, found<PHONE_NUMBER> matches
[P1] Table 5 took 340.864 sec, found<PHONE_NUMBER> matches
[P1] Table 6 took 321.835 sec, found<PHONE_NUMBER> matches
[P1] Table 7 took 241.669 sec, found<PHONE_NUMBER> matches
Phase 1 took 1778.56 sec
[P2] max_table_size =<PHONE_NUMBER>
[P2] Table 7 scan took 15.7163 sec
[P2] Table 7 rewrite took 28.6632 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 49.9901 sec
[P2] Table 6 rewrite took 81.042 sec, dropped 581338020 entries (13.5376 %)
[P2] Table 5 scan took 47.0918 sec
[P2] Table 5 rewrite took 77.4978 sec, dropped 762024595 entries (17.7441 %)
[P2] Table 4 scan took 46.5234 sec
[P2] Table 4 rewrite took 76.5011 sec, dropped 828873015 entries (19.2999 %)
[P2] Table 3 scan took 45.9568 sec
[P2] Table 3 rewrite took 75.2904 sec, dropped 855090812 entries (19.9099 %)
[P2] Table 2 scan took 46.1183 sec
[P2] Table 2 rewrite took 75.9336 sec, dropped 865639694 entries (20.155 %)
Phase 2 took 669.201 sec
Wrote plot header with 252 bytes
[P3-1] Table 2 took 64.0573 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 2 took 53.9649 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 3 took 94.9797 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 3 took 55.2826 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 4 took 94.0258 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 4 took 55.5479 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 5 took 96.885 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 5 took 58.263 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 6 took 100.172 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 6 took 59.5409 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 7 took 68.7765 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 7 took 69.792 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
Phase 3 took 872.266 sec, wrote<PHONE_NUMBER>6 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 111.476 sec, final plot size is<PHONE_NUMBER>23 bytes
Total plot creation time was 3431.57 sec (57.1928 min)
Started copy to G:\plot-k32-2021-07-13-07-36-b9b74d34bbf065caaa19fe3f90fa55124281b00bee53c4d21cb759e87aac3784.plot
Crafting plot 2 out of 4
Process ID: 8780
Number of Threads: 6
Number of Buckets P1:    2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Puzzle Hash:  714edbece1589406da826c9df5ee34062b3e9871e7e62ca0d2db7a0ace990c07
Farmer Public Key: ac87f3b326905065413d8fb535f0ae8512d576bdd27f6f125e54ff170db7fcbfeff7823f83bcc4fb808c382871272c41
Working Directory:   D:
Working Directory 2: D:
Plot Name: plot-k32-2021-07-13-08-33-bc1899407c52ede1f73ffb74170a34e420c905f21fbd5358e40bf5a14e6b2755
[P1] Table 1 took 22.0643 sec
[P1] Table 2 took 241.894 sec, found<PHONE_NUMBER> matches
[P1] Table 3 took 295.591 sec, found<PHONE_NUMBER> matches
[P1] Table 4 took 370.742 sec, found<PHONE_NUMBER> matches
Copy to G:\plot-k32-2021-07-13-07-36-b9b74d34bbf065caaa19fe3f90fa55124281b00bee53c4d21cb759e87aac3784.plot finished, took 1140.99 sec, 90.951 MB/s avg.
[P1] Table 5 took 357.858 sec, found<PHONE_NUMBER> matches
[P1] Table 6 took 321.91 sec, found<PHONE_NUMBER> matches
[P1] Table 7 took 242.167 sec, found<PHONE_NUMBER> matches
Phase 1 took 1852.36 sec
[P2] max_table_size =<PHONE_NUMBER>
[P2] Table 7 scan took 16.0124 sec
[P2] Table 7 rewrite took 31.0222 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 49.956 sec
[P2] Table 6 rewrite took 81.2241 sec, dropped 581391793 entries (13.5383 %)
[P2] Table 5 scan took 47.0961 sec
[P2] Table 5 rewrite took 77.9601 sec, dropped 762017952 entries (17.7434 %)
[P2] Table 4 scan took 46.3043 sec
[P2] Table 4 rewrite took 76.4187 sec, dropped 828916621 entries (19.3003 %)
[P2] Table 3 scan took 46.4177 sec
[P2] Table 3 rewrite took 75.5315 sec, dropped 855127144 entries (19.9101 %)
[P2] Table 2 scan took 46.0472 sec
[P2] Table 2 rewrite took 78.9283 sec, dropped 865581692 entries (20.1537 %)
Phase 2 took 675.862 sec
Wrote plot header with 252 bytes
[P3-1] Table 2 took 65.3463 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 2 took 55.0009 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 3 took 94.8452 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 3 took 54.81 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 4 took 96.3398 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 4 took 54.9329 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 5 took 96.4551 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 5 took 56.54 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 6 took 100.401 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 6 took 59.5958 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
[P3-1] Table 7 took 69.4107 sec, wrote<PHONE_NUMBER> right entries
[P3-2] Table 7 took 69.1649 sec, wrote<PHONE_NUMBER> left entries,<PHONE_NUMBER> final
Phase 3 took 873.946 sec, wrote<PHONE_NUMBER>5 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 117.051 sec, final plot size is<PHONE_NUMBER>63 bytes
Total plot creation time was 3519.29 sec (58.6548 min)
Started copy to G:\plot-k32-2021-07-13-08-33-bc1899407c52ede1f73ffb74170a34e420c905f21fbd5358e40bf5a14e6b2755.plot
Crafting plot 3 out of 4
Process ID: 8780
Number of Threads: 6
Number of Buckets P1:    2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Puzzle Hash:  714edbece1589406da826c9df5ee34062b3e9871e7e62ca0d2db7a0ace990c07
Farmer Public Key: ac87f3b326905065413d8fb535f0ae8512d576bdd27f6f125e54ff170db7fcbfeff7823f83bcc4fb808c382871272c41
Working Directory:   D:
Working Directory 2: D:
Plot Name: plot-k32-2021-07-13-09-32-533b93287196ed06e2c0416699909009e3d8401a47f903f098b8d6a24c44406b
[P1] Table 1 took 22.7933 sec
[P1] Table 2 took 247.795 sec, found<PHONE_NUMBER> matches
[P1] Table 3 took 295.816 sec, found<PHONE_NUMBER> matches
Copy to G:\plot-k32-2021-07-13-08-33-bc1899407c52ede1f73ffb74170a34e420c905f21fbd5358e40bf5a14e6b2755.plot finished, took 705.514 sec, 0 MB/s avg.

after this in cmd i have to press enter and then program close, with plot on SSD, and ~200gb tmp files of new plot that failed to proceed after failing to copy finished plot.

Same issue here - the problem started in the last week. The machine experiencing this problem is running Ubuntu 21.04 with 4 NVMEs on 4 PCIEx16s transferring plots to 4x14TB drives. Transfers to the storage drives will randomly fail and the drives are no longer usable until a restart. However, this problem also occurs when transferring the plots manually from the NVMEs to the HDDs, so the issue may not be with Madmax and instead with Ubuntu/Linux.

The best solutions so far is to use a SSD as buffer. You copy to -d SSD and then use a bash script for moving from ssd to hdd.. Copy to HDD can never be linear.. With this method the used -d ssd has enough time to copy to hdd. And even if a new plot is done, a ssd can take both jobs without problem.

I think I found a solution.
When you are plotting, put it in root directory, but in CHIA, dont add this
directory to the plots.
Make a new directory like, Ready, and then put plotted plots inside it and
then add this directory to chia.
I think chia process when running on Root of the drive, creates fwrite
issues for the plotter
I did this for 3 days, and the problem is gone.

The best solutions so far is to use a SSD as buffer. You copy to -d SSD and then use a bash script for moving from ssd to hdd.. Copy to HDD can never be linear.. With this method the used -d ssd has enough time to copy to hdd. And even if a new plot is done, a ssd can take both jobs without problem.

This makes the most sense to me. I will attempt this and see if it solves the issue.

this above solution is also same to what I said, you have to avoid putting
plots inside a directory thats actively used by CHia. that will stop all
the issues
On Fri, Jul 16, 2021 at 6:24 PM vegaix9 @.***> wrote:

The best solutions so far is to use a SSD as buffer. You copy to -d SSD
and then use a bash script for moving from ssd to hdd.. Copy to HDD can
never be linear.. With this method the used -d ssd has enough time to copy
to hdd. And even if a new plot is done, a ssd can take both jobs without
problem.
This makes the most sense to me. I will attempt this and see if it solves
the issue.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/madMAx43v3r/chia-plotter/issues/777#issuecomment-881425821,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AR2WFXZYIRYWCD6OMILBBLLTYATYFANCNFSM5ACF5BLQ
.


Introduce cfgeditor, a GUI editor for Istio config files.

This is very basic right now, but gets the structure in place
for future work in this area.

In order to build, this has a dependency on the go-bindata binary. I tried a variety of things but couldn't get the WORKSPACE file setup to pick up that dependency properly. Some help would be appreciated with getting the right voodoo in place.

Code looks OK. This would be a good placeholder to start working on the viewer/visualizer of the config/state as well.
I can take a look at the data packaging for the go binary.


Upload artifacts to Maven Central

Hi,
It would be nice if Voldemort were available at Maven Central. I heard a few months back that some dependencies were been resolved but looks like this tasks was abandoned.
I also tried to import it using https://jitpack.io/ but it didn't success either because a pom file is expected:
https://jitpack.io/com/github/voldemort/voldemort/release-1.9.18-cutoff/build.log
Any news or thoughts?
Thanks in advance

Hi @miguel0afd,
This is something I'd like to do as well.
PR #247 is the only thing that separates us from being able to publish correct Voldemort artifacts.
If you want to pull it, build it, give it a spin and help us test it, that might speed up the process of merging it in.
Thanks.
-F

bump. Any progress on getting voldemort client jars published to Maven Central (or any publicly accessible maven-compatible repository)?

@busbey  we have plans to clean this up in the next 3 months as part of our internal effort. When we are done with this, How can we integrate it back with YCSB in the future ?

Opening a PR against the project should be it.
Undoing the commenting out done in YCSB-307 will put it back into the build system. Then you'll have to update the pom for the voldemort-specific module to use the artifact you've published in maven central.
After that, compilation errors should tell you how much the existing use of hte client varies from your current client and how much , if any, the voldemort datastore binding implementation has drifted from changes in the YCSB core while its been disabled.
If y'all have a way to test voldemort client use, adding something like that to the project would also be greatly appreciated. :)

Thanks @busbey  appreciate that, will get to it and update the comment.

@FelixGV @arunthirupathi I see that #247 was recently closed. Any chance we can have artifacts published to Maven Central? (I'd like to pull the Voldemort client into a project I'm working on...)

Elad it is maven ready but someone needs to enable the maven  publishing. I
don't have this experience nor time to work on this in next few weeks.
On Mon, Jul 11, 2016, 6:48 AM Elad Efrat<EMAIL_ADDRESS>wrote:

@FelixGV https://github.com/FelixGV @arunthirupathi
https://github.com/arunthirupathi I see that #247
https://github.com/voldemort/voldemort/pull/247 was recently closed.
Any chance we can have artifacts published to Maven Central? (I'd like to
pull the Voldemort client into a project I'm working on...)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/voldemort/voldemort/issues/287#issuecomment-231739130,
or mute the thread
https://github.com/notifications/unsubscribe/AGN-M1tSE7N1gMAskDrQTFHxkpvUr_XXks5qUkmjgaJpZM4FeoZL
.

I have done it for a few other projects and I could do it again, but I'm kind of in a crunch right now. Off the top of my head, I think the last missing pieces before publishing would be to release proper artifacts for the source code and JavaDoc. I know @gnb was working on that a little while back and someone else from the community also mentioned it in passing, but as far as I know, it has not been completed yet. If anyone wants to take a stab at producing these artifacts, feel free (:

@FelixGV @arunthirupathi I'll look into it. There are multiple Voldemort packages on Maven already, but all for different versions and not the latest one. At the very least, I'll publish the latest release. :)

Okay, so I found https://gist.github.com/lukasz-antoniak/4546021 and used it as a base.
Deploying to Maven should probably be something like this:
mvn deploy:deploy-file \
    -Durl=<MAVEN_CENTRAL_URL> \
    -DrepositoryId=<REPOSITORY_ID> \
    -Dfile=target/voldemort-1.10.18.jar \
    -DpomFile=pom.xml \
    -Dsources=target/voldemort-1.10.18-sources.jar \
    -Djavadoc=target/voldemort-1.10.18-javadoc.jar

If this looks reasonable to you, I can submit a pull request for the pom.xml file.

I'm not a big fan of maintaining both a Maven and Gradle build, even if they'd be for different purposes, when a single build system can do it all...
If you're in a hurry to get these artifacts published, feel free to run the command and publish them yourself, no one's going to prevent you. But if we're going to check it into the repo, I think it'd be cleaner to do it via Gradle.
And we should also probably publish the -contrib, -bnp and -protobuf jars that are generated by the build.
As for BDB, I'm not sure what's considered cleaner between referring to an external non-Maven-Central repo, or publishing our own version of the jar (I've seen some projects do that).

Right now we're probably going to use an internal repository to hold Voldemort jars. I'm much less familiar with Gradle, so I'll leave that to someone else.
(I just agree with the idea that, ideally, anyone who wants to develop software that interacts with Voldemort could do so by simply including a few lines in a configuration file - be it Maven or Gradle. :)

Just a brief reminder that the YCSB community needs the client jars published to a publicly accessible repo before we can reenable Voldemort testing (which we'd love to do).

Yeah definitely, it's a big gap for Voldemort jars not to be included in Maven Central.
No matter what build system publishes the jar to Maven Central, any build system will be able to pick it up with a single line or three.

FYI,
I've started the process to publish Voldemort artifacts into Maven Central.
https://issues.sonatype.org/browse/OSSRH-24377
-F

@FelixGV great news! Following closely. :)

Whoa - is there any update on this? I'd planned on using Voldemort for a project but if it doesn't live in any standard dependency repository I'll probably end up passing.

Unfortunately, pushing to Maven Central was never formally prioritized and it isn't completed yet. I don't recall what issue I had hit with Sonatype but I vaguely remember there was something which made it less straightforward than other projects I had published before. One could pick it back up and bring it to completion, I suppose, but I likely won't have time in the short term.
Sorry guys.

heya folks. checking in as I work my way through a round of clean up in YCSB.
is the voldemort project still active? almost 2 years since the last update on client libraries here. looks like nothing merged to master 2.5 years?
https://github.com/voldemort/voldemort/graphs/contributors?from=2000-12-31&to=2020-02-03&type=c


What would be the simplest way to implement a ltr/rtl text direction toggle for different paragraphs in a single editor?

I have as case where I need to support different languages within a single editor (specifically a rtl with a ltr language).
From what I've seen this isn't supported out of the box as the editor selects the text dir from the language file.
Would it be possible to implement a text direction toggle which would allow different paragraphs to have different text directions within a single editor?
If yes, could I please get advise on the best approach, I'm aware of Custom Buttons & Plugins.
Thanks

You just need to destroy and create a new instance with the new option :)

I believe I mentioned both in the title & description of this issue that I'm trying to have different paragraphs within a single editor to have different text directions.
I'm aware that I can create instances of the editor with a different language.

Ah sorry, so Trumbowyg can't handle that at all since the direction is set on the contenteditable element and not on subelements.
Sorry :/


Seismic example re-structuring

The current seismic examples (Acoustic and TTI) both use explicit wrappers for Forward, Adjoint, Gradient and Born wrapper methods that create new operators for every kernel invocation. This prohibits re-use in tight loops, such as checkpoint-based RTM, because every application of the wrapper bypasses the operator-level caching and triggers a full DSE/DLE/codegen/compilation cascade. We should re-structure the examples to, at the very least, utilise pre-setup operators for the different invocations, or even allow a seamless argument injection into the pre-defined operator objects.
The seismic utility wrappers also previously instantiated all symbolic data object at invocation time, which causes costly memory-copy operations. Some of this has moved to a slightly higher level, but a more concrete model of data flow for the various types of grid data and the various desired use cases (adjoint test, gradient test, RTM, FWI, etc.) is required, in particular with focus on which data lives at which temporal granularity and who instantiates it (user-level, operator-wrapper).

As an example let's consider the checkpointing case with all data created by the user (either explicitly or through seismic-specific abstractions/utility classes:
# Data allocation
model = Model(velocity, ...)
src = RickerSource(src_coordinates, time_series)
rec = Receivers(rec_coordinates)
u = TimeData(u, ...)
v = TimeData(v, ...)

# Operator instantiation, wrappers return type Operator/StencilKernel
op_fwd = ForwardOperator(model=model, u=u, source=src, receivers=rec, ...)
op_grad = GradientOperator(model=model, v=v, ...)

# Incremental forward loop
ckp = Checkpointer()  # Something clever that handles checkpointing/scheduling
nsteps = 0
while nsteps > 0:
   nsteps = ckp.how_many_timesteps_should_I_run()
   op_fwd(u=u, m=model.m, time_size=nsteps, ...)
   ckp.do_your_thing(u)

for all checkpoints:
    u = ckp.retrieve_forward_wavefield()
    op_grad(v=v, u=u, m=model.m, ...)

The key questions arising from this are

Can we make op_fwd/op_grad pure Operator/StencilKernel objects?
When/where do we instantiate the wavefields? Should these have an additional abstraction to capture ABCs, or can we hide the PML on the Model??
When/where are sources/receivers instantiated and populated?

Note: The above is a rough draft and likely to change.

1-Can we make op_fwd/op_grad pure Operator/StencilKernel objects?
Not sure about this one, a lot of setup is still required.
2-When/where do we instantiate the wavefields? Should these have an additional abstraction to capture ABCs, or can we hide the PML on the Model??
I think ABC should be abstracted out. The current version, as pointed out in #140 , as some problems. Also, for PML and ABC we have symbolic expression for it and should be special loops over the small portion of the domain concerned. I had a try at it, but Iteration seems to be broken with 3.0 so I need to rethink it.
For the wavefield, this is the tricky question, specially with the Julia interface. I owever think we can discuss it in person with philipp once we get in London, but this shouldn't be required to be solved for the GMD paper.
3- When/where are sources/receivers instantiated and populated?
This is usually done at the beginning, as this will never change during the inversion. However, with thousand of sources, it is likely that different sources will have different receiver number and/or positions.

Can we make op_fwd/op_grad pure Operator/StencilKernel objects?

Not sure about this one, a lot of setup is still required.

Can you please be more specific? What setup/which fields are required on a per-invocation basis, and which live beyond a single operator run?

I had a try at it, but Iteration seems to be broken with 3.0 so I need to rethink it.

Iteration objects are now used internally only for 3.0 while loop iterations are simply implied by the dimensions used. Can you provide a concrete example of what you tried, and how it failed (possibly in the appropriate issue for ABCs #140)?

For the wavefield, this is the tricky question, specially with the Julia interface. I owever think we can discuss it in person with philipp once we get in London, but this shouldn't be required to be solved for the GMD paper.

I agree this question does not necessarily need to be solved for GMD. Nevertheless I would like to have some idea of what the requirements are (and what rationale we have for them), since this problem still influences a lot of other design decisions. In a similar vain, being clear about the kind of domain-specific  utilities and the type of examples code we want to provide is important for further development, and will allow us to make an informed decision on what level of clean-up we can reasonably do for GMD, given a much changed operator API.

fixed with #243 ?

Yes, everything not covered in #243 is veing tracked in other issues.


feat(ci): update latest tag on release

Overview
Push flant/shell-operator:latest on release.
What this PR does / why we need it
There is a separate workflow to update a latest tag, but build supports pushing multiple tags.
Does this PR introduce a user-facing change?


MP2-PP2 hack shared layer to non-sharded layer to Step Alignment

PR types

Others
PR changes

Others
Description

MP2-PP2与MP2无法做到逐位对齐，原因是MP使用了共享Embedding层，共享层需要使用梯度累加，MP的梯度累加是使用加法来实现的，而PP由于跨卡，因此共享层的梯度同步是使用通信来完成的，具体的操作是Allreduce。这个PR将共享层hack成非共享层，来实验MP2-PP2与MP2的逐位对齐，目的是为了验证Allreduce和加法实现的梯度累加本身就存在diff，因此无法做到完全逐位对齐。一定要注意要注释掉self.tie_weights()这一行代码，这一行手动绑定了两个层作为共享层

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 3 committers have signed the CLA.:x: xysheng-baidu:x: iosmers:x: AnnaTrainingGYou have signed the CLA already but the status is still pending? Let us recheck it.


skip setting should take globs

I have a feature request: the "skip" setting should take glob expressions, to exclude multiple files at once. For instance **/*_pb2.py.

I'm unsure the proper way to write tests, but I will write them with some guidance on what to do with the base64 blob.
The patch adds an additional settings flag -sg / --skip-glob which works similar to -s / --skip . It uses fnmatch to perform path globbing.
Currently there are issues with backslash escapes and {} wildcards. A workaround for the former is to use [], e.g to match a literal *, use [*] instead of \* . The {} limitation is due to the settings splitting on commas.


Update imported file path while change folder path

Issue Type: Feature Request
suggestion:

while move one sub folder and make it to parent at that time imported all the component's path would be automatically update.

Regards,
Bhavin Shiroya
VS Code version: Code 1.30.1 (dea8705087adb1b5e5ae1d9123278e178656186a, 2018-12-18T18:12:19.495Z)
OS version: Windows_NT ia32 6.1.7601

Sounds like a duplicate of #58987
Let me know if that is not the case


compatibility issues

could please re add support for the mod "Color Code Gui" by NukeDuck in early beta versions it worked flawlessly with the mod and allowed me to have really smart Unicode symbols and allowed easy access to color codes in chat box i really miss being able to inject the nuke/plane unicode in chat when im chatting to my friends thanks for reading this i look forward to your reply :)

I'm going to have to say no for now. With the way that mod works right now, it would be difficult to add compatibility. It would actually be easier to create a new mod that works because it's so simple..


own train-test data split

i am following this notebook https://github.com/microsoft/recommenders/blob/main/examples/02_model_collaborative_filtering/ncf_deep_dive.ipynb
but i dont have any timestamp related column in my dataset . i have created train-test split externally and want to feed it to NCFDataset class but i m getting "Empty file error".
any way to solve this

it seems that you are either not linking the right file or your file doesn't have data.
The best way to debug your code is to start from the current notebook, make sure it works and then replace the original dataset with your one trying to change as less as possible. For example, what I would do is to change everything except the timestamp, or use a constant timestamp. Then keep iterating until you have something working.


Getting frequent D/OkHttp: <-- HTTP FAILED: java.net.SocketTimeoutException: failed to connect to / (port 8443) from / (port 50810) after 60000ms: isConnected failed: ETIMEDOUT (Connection timed out)

I am getting frequent(one in 10 times average) D/OkHttp: <-- HTTP FAILED: java.net.SocketTimeoutException from my servers and I have set all timeouts(read, write, connect) to 60seconds but I get this exception in 8-10 seconds and there are no hits on my server for these requests.
Some of the previously closed issues(https://github.com/square/okhttp/issues/6486) discuss IPv4 vs IPv6 issues at the client and/or server-side but my server supports only IPv4 and I made the client(android device) use IPv4 only still I am receiving this.
Can someone please help here,
Depos used -
implementation 'com.squareup.retrofit2:retrofit:2.7.1'
implementation "com.squareup.retrofit2:converter-moshi:2.6.2"
implementation "com.jakewharton.retrofit:retrofit2-kotlin-coroutines-adapter:0.9.2"
implementation 'com.squareup.okhttp3:logging-interceptor:4.4.0'
implementation 'com.squareup.retrofit2:converter-gson:2.7.1'
implementation 'com.squareup.retrofit2:adapter-rxjava2:2.7.1'
implementation 'com.squareup.retrofit2:converter-scalars:2.4.0'

Thanks for your question. This issue tracker is only for bug reports with test cases and feature requests. Please ask usage questions on Stack Overflow.
https://stackoverflow.com/questions/tagged/okhttp


Add sinon's sandboxing functionality to the test framework #1347

This change is

PR is added to the queue for testing as 1. task. (fa2680b)

@opetkevich,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

Test 112761991 (fa2680b) build started.

Test (Success) 112761991 build is completed (fa2680b)
See https://github.com/ThaliTester/TestResults/tree/112761991fa2680b_Add_sinon_s_sandboxing_functionality_to_the_test_framework__1347_opetkevich/ for the logs

Test 112761991fa2680b(fa2680b) has failed
See https://github.com/ThaliTester/TestResults/tree/112761991fa2680b_Add_sinon_s_sandboxing_functionality_to_the_test_framework__1347_opetkevich/ for the fail logs

-@chapko +@opetkevich

Reviewed 21 of 21 files at r1.
Review status: all files reviewed at latest revision, 11 unresolved discussions.

test/www/jxcore/bv_tests/testTests.js, line 45 at r1 (raw file):

test('test sinon sansbox spy', tape.sinonTest(function (t) {
  this.spy(testObject, 'testSpyMethod');
  t.equal(testObject.testSpyMethod, testSanboxObject.testSpyMethod,

This is a bit strange check.
More appropriate check would be just calling method and checking spy.callCount or something similar.

test/www/jxcore/bv_tests/testTests.js, line 70 at r1 (raw file):

  mock.verify();
  t.end();
}));

Also there should be a test to checks that sinonTest indeed restores all the spies, stubs and mocks.

test/www/jxcore/bv_tests/testThaliManager.js, line 43 at r1 (raw file):

});

function Mocks(t, sandboxObject) {

NB: why don't just call it sandbox? :)

test/www/jxcore/bv_tests/testThaliMobile.js, line 288 at r1 (raw file):

    process.nextTick(function () {
      t.equals(spy.callCount, 1, 'should not have been called more than once');
      ThaliMobile.emitter.emit.restore();

This line can be removed.

test/www/jxcore/bv_tests/testThaliMobile.js, line 1895 at r1 (raw file):

      t.equal(peerIdentifier, somePeerIdentifier, 'Peer still matches');
      t.ok(connectionErrorReceived, 'We got the connection error');
      ThaliMobileNativeWrapper.disconnect.restore();

This line can be removed too.

test/www/jxcore/bv_tests/testThaliNotificationClient.js, line 149 at r1 (raw file):

});

function stubGetPeerHostInfo() {

NB: I would pass sandbox explicitly as parameter to make it more readable and easier to use.

test/www/jxcore/bv_tests/testThaliNotificationLocal.js, line 82 at r1 (raw file):

};

function stubGetPeerHostInfo() {

NB: same as above

test/www/jxcore/bv_tests/testThaliNotificationServer.js, line 139 at r1 (raw file):

    globalVariables.kill().then(function () {
      globalVariables.spyStartUpdateAdvertisingAndListening.restore();
      globalVariables.spyStopAdvertisingAndListening.restore();

Why don't move it to the kill method itself? I think it belongs there.

test/www/jxcore/bv_tests/testThaliWifiInfrastructure.js, line 293 at r1 (raw file):

  }

  captureMessages.call(this, function (messages) {

NB: captureMessages is a local function in this test, so it probably would be simpler to use self = this.

test/www/jxcore/lib/SinonTest.js, line 4 at r1 (raw file):

var util     = require('util');
var inherits = util.inherits;

please, lint this file, eslint reports 11 errors.

test/www/jxcore/lib/SinonTest.js, line 10 at r1 (raw file):

function SinonTest (callback) {

This is not a constructor so don't capitalize it.

Comments from Reviewable

PR is added to the queue for testing as 1. task. (de7a3d0)

Test 112761991 (de7a3d0) build started.

Test (Success) 112761991 build is completed (de7a3d0)
See https://github.com/ThaliTester/TestResults/tree/112761991de7a3d0_Add_sinon_s_sandboxing_functionality_to_the_test_framework__1347_opetkevich/ for the logs

PR is added to the queue for testing as 1. task. (d9ed552)

Test 112761991 (d9ed552) build started.

Reviewed 11 of 11 files at r2, 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

test/www/jxcore/lib/sinonTest.js, line 32 at r2 (raw file):

      result = callback.apply(this, args.concat(sandbox.args));
    } catch (e) {
      t.fail(e);

I expect that a function wrapped in sinonTest should behave the same way as original one. So if original function throws, wrapped should throw too.
Fixed.

Comments from Reviewable

Test (Success) 112761991 build is completed (d9ed552)
See https://github.com/ThaliTester/TestResults/tree/112761991d9ed552_Add_sinon_s_sandboxing_functionality_to_the_test_framework__1347_opetkevich/ for the logs

Test 112761991d9ed552(d9ed552) has successfully finished without an error
See https://github.com/ThaliTester/TestResults/tree/112761991d9ed552_Add_sinon_s_sandboxing_functionality_to_the_test_framework__1347_opetkevich/ for the logs

I can't imagine this was any more fun to write than it was to review but good work, this should hopefully cause many current and future hysenbugs to go away.

Reviewed 12 of 22 files at r1, 10 of 11 files at r2, 1 of 1 files at r3.
Review status: :shipit: all files reviewed at latest revision, all discussions resolved, all commit checks successful.

Comments from Reviewable


issue #71

This request contains a solution to problem https://github.com/koenbeuk/EntityFrameworkCore.Projectables/issues/71 and a test case of use. The request was prepared with the support of P9avel.

Please to merge this PR

@koenbeuk  Please merge this PR


UTC time in templates

Maybe should add "expects_localtime=True" to filter silk_date_time?
# templatetags/filters.py

@register.filter(expects_localtime=True)
def silk_date_time(dt):
    return _silk_date_time(dt)

Not familiar with this - what does it do? Just translates into local time?

Yes, it is:
https://docs.djangoproject.com/en/1.7/howto/custom-template-tags/#filters-and-time-zones

Great, will add this in the next update also.

Done


Introduce presentation model blue prints

In Open Dolphin each presentation model is unique and thus contains all meta-information.
The Dolphin Platform maps objects onto presentation models. As we often have several instances of the same class, we are currently sending the same kind of meta-information over and over again. Instead we should introduce something like a presentation model blue print for a DolphinBean which is sent only once. When we create another instance of that DolphinBean, we can simply refer to the blue print and use its meta-data.

Sounds like a good idea :)


koordlet: adjust node comparison method to improve program performance

Ⅰ. Describe what this PR does
adjust node comparison method to improve program performance for nodeInformer event handler.
Ⅱ. Does this pull request fix one issue?
NONE

Ⅲ. Describe how to verify it
Ⅳ. Special notes for reviews
V. Checklist

[x] I have written necessary docs and comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in make test

how about also checking the content after version, since other field maybe updated.

how about also checking the content after version, since other field maybe updated.

@zwzhang0107 If the resource object is persisted to etcd, the resourceVersion of the resource object is increased. Do you mean that the Node resource object is also determined by replacing the Generation with the resourceVersion?

how about also checking the content after version, since other field maybe updated.

@zwzhang0107 If the resource object is persisted to etcd, the resourceVersion of the resource object is increased. Do you mean that the Node resource object is also determined by replacing the Generation with the resourceVersion?
Generation only when the Spec content of the resource object changes, will be increases.

I mean how about this:
if newNodeMetric.Generation == oldNodeMetric.Generation || deepEqualCheck()
in case got spec with the same value

how about also checking the content after version, since other field maybe updated.

@zwzhang0107 If the resource object is persisted to etcd, the resourceVersion of the resource object is increased. Do you mean that the Node resource object is also determined by replacing the Generation with the resourceVersion?
Generation only when the Spec content of the resource object changes, will be increases.

I mean how about this: if newNodeMetric.Generation == oldNodeMetric.Generation || deepEqualCheck() in case got spec with the same value

@zwzhang0107 I don't think it's necessary to add a deepEqualCheck(). If newNodeMetrics and oldNodeMetrics correspond to the same Spec, then their metadata.Generaton is the same.
https://stackoverflow.com/questions/47100389/what-is-the-difference-between-a-resourceversion-and-a-generation

@xigang According to what-is-the-difference-between-a-resourceversion-and-a-generation

in some objects, generation is incremented by the server as part of persisting writes affecting the spec of an object.

in some objects: This should not include the Node. After looking at the source code implementation, the generation is modified by the Deployment(used for Deployment version rollback)  or other controller, but  the controller of the Node ignores this field.
Back to the problem, when code enter from Informer.UpdateFunc, the resource has been modified. Therefore, deepEqualCheck() is always false. (refer to: deploy/ds/job‘s controller)
https://github.com/kubernetes/kubernetes/blob/8bba479f03fd27f1a54fcd1e02347c1a5b98b30c/pkg/controller/daemon/daemon_controller.go#L231-L250

@xigang According to what-is-the-difference-between-a-resourceversion-and-a-generation

in some objects, generation is incremented by the server as part of persisting writes affecting the spec of an object.

in some objects: This should not include the Node. After looking at the source code implementation, the generation is modified by the Deployment(used for Deployment version rollback) or other controller, but the controller of the Node ignores this field.
Back to the problem, when code enter from Informer.UpdateFunc, the resource has been modified. Therefore, deepEqualCheck() is always false. (refer to: deploy/ds/job‘s controller)
https://github.com/kubernetes/kubernetes/blob/8bba479f03fd27f1a54fcd1e02347c1a5b98b30c/pkg/controller/daemon/daemon_controller.go#L231-L250

@novahe Yes, you're right. Some object updates ignore metadata.Generation field (for example:Node, ConfigMaps). But the NodeMetric object can determine whether it changes by generation. So we don't need deepEqualCheck(). /cc @zwzhang0107 Please review the code and see if there is any problem?
Thanks.

spec data maybe updated twice with same value( A-> B -> A), is it possible that the second update(B) maybe lost because of network problem?

@xigang According to what-is-the-difference-between-a-resourceversion-and-a-generation

in some objects, generation is incremented by the server as part of persisting writes affecting the spec of an object.

in some objects: This should not include the Node. After looking at the source code implementation, the generation is modified by the Deployment(used for Deployment version rollback) or other controller, but the controller of the Node ignores this field.
Back to the problem, when code enter from Informer.UpdateFunc, the resource has been modified. Therefore, deepEqualCheck() is always false. (refer to: deploy/ds/job‘s controller)
https://github.com/kubernetes/kubernetes/blob/8bba479f03fd27f1a54fcd1e02347c1a5b98b30c/pkg/controller/daemon/daemon_controller.go#L231-L250

@novahe Yes, you're right. Some object updates ignore metadata.Generation field (for example:Node, ConfigMaps). But the NodeMetric object can determine whether it changes by generation. So we don't need deepEqualCheck(). /cc @zwzhang0107 Please review the code and see if there is any problem?
Thanks.

spec data maybe updated twice with same value( A-> B -> A), is it possible that the second update(B) maybe lost because of network problem?

@zwzhang0107 There is no problem with network jitter. When the NodeMetric spec value changes from A->B->A, the NodeMetric metadata.Generation value increases continuously. For example:

A->B: metadata.Generation value from 1 to 2.
B->A: metadata.Generation value from 2 to 3.

( A-> B -> A)

yep, so when koordlet receives metadata.Generation bumps from 1 to 3, there is no need to do the update since the spec info not changed.

@zwzhang0107  Yes, you are right, there is such a case. Let me remove part of the checking logic.

@zwzhang0107 Please take a look this code needs to merge?

/lgtm
/approve
Thanks! Acturally, we can still keep the generation comparing as an OR condition, which can save the cost of deep-equal comparing. @xigang
if newNodeMetric.Generation == oldNodeMetric.Generation || deepEqualCheck()

/lgtm /approve
Thanks! Acturally, we can still keep the generation comparing as an OR condition, which can save the cost of deep-equal comparing. @xigang if newNodeMetric.Generation == oldNodeMetric.Generation || deepEqualCheck()

@zwzhang0107 I will submit a pr for this issue later.

/lgtm /approve
Thanks! Acturally, we can still keep the generation comparing as an OR condition, which can save the cost of deep-equal comparing. @xigang if newNodeMetric.Generation == oldNodeMetric.Generation || deepEqualCheck()

@zwzhang0107 I will submit a pr for this issue later.

@zwzhang0107 pr has been submitted.


[FEAT] Reorder columns

Column re-ordering would be a nice feature, you already have resizing.

Agreed, will put in at some point. Thinking a popup with two lists for now,
with 'add' / 'remove' columns as well re-ordering. Then dragging columns
would be a nice 'show off' feature.
On 8 March 2015 at 02:23, Austin<EMAIL_ADDRESS>wrote:

Column re-ordering would be a nice feature, you already have resizing.
—
Reply to this email directly or view it on GitHub
https://github.com/ceolter/angular-grid/issues/6.

--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Statements and opinions expressed in this e-mail may not
represent those of the sender. Any review, retransmission, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient is prohibited. If you
received this in error, please contact the sender immediately and delete
the material from any computer.

Just wanted to second the request for the column re-ordering / dragging re-order.
Haven't implemented yet but looks awesome, thanks for making it!

Thanks cnd5041 for your input, it's deffo on my 'to do' list . . . . just finishing 'checkbox selection' at the mo, next on my list is 'cell selection and keyboard navigation'.

+1 for this... was thinking the same

I want to use the column reordering feauture using drag drop in my app.
Could you please let us know the request's status / priority.
Thanks in advance.

status: not started. priority: medium. estimate: two or three months from now.

Any updated time frame on this?

+1 on column reordering!

you can reorder using the toolpanel

Excellent, just tried the new Test Drive! :)

+1 on draggable Reorder of columns (not having to use the toolpanel).

+1 . . . on my list

+1

+1

+1

+1

+1

+1

Guys, you have custom header renderer, you can implement drag&drop there, and call gridOptions.columnApi.moveColumn(fromIndex, toIndex) on ondragover.
I'm working on something like this just now. But it's with angularCompileHeaders: true also.

Here you go, a simple example: (w is instead of $scope here):
I needed angular compile headers because I'm accessing angular data. (gridOptions). You may want to remove that if you're not using angular.
    var columnDefs = [
        {headerName: "Make", field: "make"},
        {headerName: "Model", field: "model"},
        {headerName: "Price", field: "price"}
    ];

    var rowData = [
        {make: "Toyota", model: "Celica", price: 35000},
        {make: "Ford", model: "Mondeo", price: 32000},
        {make: "Porsche", model: "Boxter", price: 72000}
    ];

    w.gridOptions = {
        columnDefs: columnDefs,
        rowData: rowData,
        angularCompileHeaders: true,
        headerCellRenderer: headerCellRendererFunc
    };

    function headerCellRendererFunc(params) {

        var eHeader = document.createElement('div');
        eHeader.setAttribute('draggable', true);

        eHeader.addEventListener('dragstart', function (event) {
            var sourceIndex = w.gridOptions.columnDefs.indexOf(params.colDef);
            event.dataTransfer.setData("fromIndex", sourceIndex);
        });

        eHeader.addEventListener('dragover',function (event) {
            event.preventDefault();
            event.target.style.color = 'blue';
        });

        eHeader.addEventListener('dragleave',function (event) {
            event.preventDefault();
            event.target.style.color = '';
        });

        eHeader.addEventListener('drop', function (event) {
            var toIndex = w.gridOptions.columnDefs.indexOf(params.colDef);
            event.preventDefault();
            var fromIndex = event.dataTransfer.getData("fromIndex");
            w.gridOptions.columnApi.moveColumn(fromIndex, toIndex);
            //console.log('fromIndex: '+fromIndex);
            //console.log('toIndex: '+toIndex);
        });

        var eTitle = document.createTextNode(' ' + params.colDef.headerName + ' ');
        eHeader.appendChild(eTitle);

        return eHeader;
    }

+1
Thanks @lucaalex87, really nice example

+1 is there any estimation about that?

dragging column headers will be in in the next week or two. it's the next 'big ticket' item i'll be doing.

Works great with the lastest version, thanks !
Can you add an option/api to disable/enable this manually ?

@Wykks
Is possible on:
Grid level: suppressMovableColumns (https://www.ag-grid.com/javascript-grid-properties/index.php)
Column level: suppressMovable: true

Whoa I missed that. Thanks ! :)

closing this issue, complete :)

I found an issue.
Using ag-grid.min.noStyle.js: dragging column moves only the header and not the whole column.
Using ag-grid.min.js: dragging column moves whole column (expected)

that's really strange, the only difference is 'noStyle' doesn't contain the css.
are you including the css yourself directly then?

@nsteenbeek i've just tested, it works fine for me

@ceolter My fault! Forgot to update the ag-grid.css. That's why the version including styles worked. Thumbs up for ag-grid!

Thanks cnd5041 for your input, it's deffo on my 'to do' list . . . . just finishing 'checkbox selection' at the mo, next on my list is 'cell selection and keyboard navigation'.

For keyboard accessibility.
Column re-ordering would be a nice feature with keyboard.

Thank you for your comment @chenokn.
You'll be pleased to know that we are looking to add this functionality with our next v31 release. I have added the backlog item reference to this Github issues for your reference.


Add a link to slide and its thumbnail

I added a thumbnail of slide to the index page.

LGTM :+1:  We will need more slides on Chainer :)


upgrade from 2.7.9 to 2.7.12 and the consumer got NoSuchMethodError for curator "CreateBuilder.creatingParentsIfNeeded".

Environment

Dubbo version: 2.7.12
Operating System version: windows 10
Java version: 1.8.0
zookeeper version: 3.6.2
curator version: 2.12.0

Steps to reproduce this issue
upgrading version 2.7.9 to 2.7.12 and got this exception:
Injection of @DubboReference dependencies is failed; nested exception is java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModeStatPathAndBytesable;

It works ok in version 2.7.9 but got this exception in 2.7.12. Provider code works ok but Consumer is not. the extra infomation:
***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery.registerServiceWatcher(ZookeeperServiceDiscovery.java:196)

The following method did not exist:

    org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModeStatPathAndBytesable;

The method's class, org.apache.curator.framework.api.CreateBuilder, is available from the following locations:

    jar:file:/D:/m2/repository/org/apache/curator/curator-framework/2.12.0/curator-framework-2.12.0.jar!/org/apache/curator/framework/api/CreateBuilder.class

The class hierarchy was loaded from the following locations:

    org.apache.curator.framework.api.CreateBuilder: file:/D:/m2/repository/org/apache/curator/curator-framework/2.12.0/curator-framework-2.12.0.jar

Action:

Correct the classpath of your application so that it contains a single, compatible version of org.apache.curator.framework.api.CreateBuilder

This high probability is a problem of zk version conflicts.
Do you use other zk version in your project?

This high probability is a problem of zk version conflicts.
Do you use other zk version in your project?

Yes. zk 3.5.9 and 3.4.14 also have this problem.

can you just use the zk version in dubbo and exclude other zk ？

我们遇到了同样的问题，可不可以不升级curator的version(我们目前也使用2.12.0)，修复这个bug因为要升级的话，我们有几个百应用要动，但又想使用dubbo2.7.12的新版本。

我们遇到了同样的问题，可不可以不升级curator的version(我们目前也使用2.12.0)，而修复这个bug
因为要升级的话，我们有几百个应用要动，这个成本太大，能否在2.7的版本下，加个小版本修复？
We have the same problem. Can you fix this bug without upgrading the version of curator？ (we also use 2.12.0 now.)
Because we have hundreds of applications to upgrade. The cost is too high.
Thank you very much！

这部分逻辑是需要使用应用级服务发现才会有的，当前 2.7 版本的应用级服务发现是处于原型状态，后续不再继续维护的。关于 zookeeper 版本的短期是可以做一个兼容。但是长期来看肯定是要往高版本升级的，目前 zookeeper 官方也对低版本不维护了。

@AlbumenJ 谢谢。
我目前的解决方案是：dubbo升级到2.7.12，curator升级到4.2.0，排除curator本身的zk版本，引用了自己的3.4.8的zk版本
       <dependency>             <groupId>org.apache.zookeeper</groupId>             <artifactId>zookeeper</artifactId>             <version>3.4.8</version>         </dependency>         <dependency>             <groupId>org.apache.curator</groupId>             <artifactId>curator-recipes</artifactId>             <version>4.2.0</version>             <exclusions>                 <exclusion>                     <groupId>org.apache.zookeeper</groupId>                     <artifactId>zookeeper</artifactId>                 </exclusion>             </exclusions>         </dependency>
另外，还发现一点：用过2.7.9的，在zookeeper目录下，找到/dubbo/config，下面有个mapping目录，这个在用2.7.12前要删除，否则会报错。

in fact it still recommend to use zookeeper: 3.4.9 and curator: 2.12.0 in the doc for dubbo 2.7. https://dubbo.apache.org/zh/docs/v2.7/user/dependencies/
I would appreciated if it could be updated for dubbo 2.7.12 and other subversion for production use. I'm not very clear that which zookeeper cluster version, zookeeper client version and curator version whould be perfect for dubbo 2.7.12.


fix(sqllab): invalid sorting order on db selector

SUMMARY
Currently, the order of the db selector dropdown in SQL Lab is always sorted by record ID, regardless of the intended sort order.
This issue arises due to two reasons:

an incorrect order_column name is being passed
AsyncSelect component's default sortComparator is always sorting by ID.

This commit fixes these issues, ensuring that the db selector is sorted by name as originally intended.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
Added unit tests.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

justinpark if you look at the default DEFAULT_SORT_COMPARATOR implementation, you can see that it will sort items alphabetically if label is of type string. In this case, it's not sorting alphabetically because label is being set to a React component here instead of using customLabel for that purpose. If you set label to database_name and customLabel to the React component, you'll get the desired behavior.

You are right, but I think that if the API that includes the actual sorting items is implemented, we would want the results to be sorted according to the API. Also, relying on client-side sorting can lead to inconsistencies, especially when handling paginated results, so I believe it is better to follow the API's results.
Additionally, I have optimized the code to use the suggested order field.

@justinpark Could you also update the PR description to say that the objective is to sort by the API order?


modis colormap added to init.py

Description
The colormaps were not registered. Adding it to the init.py folder.
Type of change
How Has This Been Tested?
Checklist:

Thanks!


Upgrade KSM

What this PR does / why we need it
Towards https://github.com/giantswarm/giantswarm/issues/23226
Checklist

[X] Update changelog in CHANGELOG.md.

Trigger e2e tests
/test create

/test create


SPSA has an assert that is too loose

Grep for "TODO: this assert looks totally wrong."

Is it the same assertion error that I meet?
from cleverhans.attacks import SPSA
SPSA_params = {
    'epsilon': 0.6,
    'num_steps': 40
}
################
SPSA_adv = SPSA(adv, sess=sess)
SPSA_adv_gen = SPSA_adv.generate(x, **SPSA_params)
adv_SPSA = adv.get_logits(SPSA_adv_gen)
adv_SPSA_acc = model_eval(sess, x, y, adv_SPSA, X_test, Y_test, args=eval_params)
print('Adv training model accuracy on SPSA test examples: %0.4f' % adv_SPSA_acc)

/usr/local/lib/python3.6/dist-packages/cleverhans/attacks_tf.py in _compute_gradients(self, loss_fn, x, unused_optim_state)
1650         """Compute gradient estimates using SPSA."""
1651         # Assumes x is a list, containing a [1, H, W, C] image
-> 1652         assert len(x) == 1 and x[0].get_shape().as_list()[0] == 1
1653         x = x[0]
1654         x_shape = x.get_shape().as_list()
AssertionError:

That's a different assert

Closing as we update the library to v4 and drop support for version < 4.

Closing as we update the library to v4 and drop support for version < 4.


Adjust useDatabricksOAuthInAzure behavior

useDatabricksOAuthInAzure = true means that user wants to use Databricks OAuth method on given Azure instance. And if that Azure host is not supported - we shouldn't silently fall back to Azure OAuth but throw an error instead. Azure OAuth should be used only when  useDatabricksOAuthInAzure = false

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (f3c53a5) 90.59% compared to head (d1ce0c7) 90.59%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #225   +/-   ##
=======================================
  Coverage   90.59%   90.59%
=======================================
  Files          62       62
  Lines        1435     1435
  Branches      245      245
=======================================
  Hits         1300     1300
  Misses         84       84
  Partials       51       51

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[test] tcp-ping-pong Does Not Wait for Full Receive Buffer

Description
This PR fixes https://github.com/demikernel/demikernel/issues/390.
Summary of Changes

Fixed tcp-ping-pong to buffer received response on client-side
Changed tcp-ping-pong to send buffers with different payload at each round.

This change is okay (assuming this test isn't about performance).
But it brings into question what the main purpose of this test is. If it is for correctness, then it's a reasonable test of the basics, but it doesn't check for data integrity inside each "round". If it is for performance, then this change is going to hurt, as it introduces an extra data copy - if we care about performance here, we should hang on to the received scatter-gather arrays until we get all the data for one "round" and then sanity check the array data correctly (or perform partial checks on each receive as they arrive). Ideally, on the server side we'd then reuse that same scatter-gather array to return that data to the client, rather than recreate it on the server side.
Even more ideally, our API would be flexible enough to allow us to modify multiple received scatter-gather arrays that compose a "round" (potentially including overlap with adjacent "rounds") to make the returning push operation take a single scatter-gather array that just describes the current "round" (without requiring any copying of the data).

@BrianZill this is about fixing the correctness of the test only. Concerning the sanity check of the data in each round, we are doing that: at each round, we sanity check if the received buffer matches the fill_char. Are you suggesting us to have something different than that?


Config doesn't work on mod menu

I've removed release 2.0 for now as it has a few issues, including it not being bundled with the config library required for the config screen.

Fixed in 2.0.1, thanks for reporting!


as: Add Datacake webhook template

Summary
Adding Datacake webhook template
Checklist

[x] Scope: The referenced issue is addressed, there are no unrelated changes.
[x] Compatibility: The changes are backwards compatible, they don't break existing deployments.
[x] Testing: The changes are tested.
[ ] Documentation: Relevant documentation is added or updated.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please edit the commit message to exclude the as: prefix.


There is any event to search the specified location in search box I cant find any event?

No desceiption, no help.


OperatorHub Integration - WIP

Description
this PR prepares the operator to be made available in operator hub when a release is done.
the readme contains the steps to be followed when releasing. For now the steps are manual.
Please follow the steps until you see what this operator would look like in operator hub and give some feedback there.
Once satisfied I'll test the installation from operator hub actually works and then at the following release we can also push this to operator hub.
because of how the operator-sdk and operator-courier commands work I had to create some manifest files that are actually duplicates of the manifest that are in the helm charts. May in the future we can think of more sophisticated solutions.

There seem to be many files very similar (identical?) to ones already in the repo (e.g. the CRD); is this PR intending to add duplicated files? can't we just prepare a temporary dir when making a release to OperatorHub? is it a requirement of OperatorHub that the repo has specific files at specific locations?

@akavel deploy/olm-catalog/eunomia directory needs to be submitted to community-operators repo and I believe we don't need it here, but in this PR it's for code review purposes.

@rhalat please go ahead and submit the pull request to https://github.com/operator-framework/community-operators/tree/master/community-operators. Please @ mention me on the pull request.
I believe we will want to still merge the file .github/RELEASE.md file in this pull request.

@seanmalloy I tried to follow @raffaelespazzoli instructions and I managed to add eunomia v0.0.2 to my private quay application registry https://quay.io/application/zezima/eunomia?tab=releases
I also created temporary PR from eunomia-0.0.2 to master branch in my fork here https://github.com/rhalat/community-operators/pull/1 but as for now I didn't test deployment, because I don't know exactly how. In this case I'm not sure if I should submit PR to https://github.com/operator-framework/community-operators/tree/master/community-operators now or first try to test it somehow.

@rhalat did you see this documentation ... https://operatorhub.io/contribute. This document has a link to https://github.com/operator-framework/community-operators/blob/master/docs/testing-operators.md#manual-testing-on-kubernetes. It looks like the second link provides instructions for testing with minikube and OLM.
Also, why did you need to create a personal quay.io? Can't you just use https://quay.io/repository/kohlstechnology/eunomia-operator?

One more question has occurred to me: in future, do we expect to need to redo this process? — e.g. when the CRD changes, like with #134 — or not?
If yes, wouldn't it make sense to try and put as much of this process as possible in scripts, to make our life in future easier? I would imagine the scripts could e.g. build a temporary directory, with structure as expected for copying into community-operators; do you think this makes any sense, or not really?

@seanmalloy Thank you for provided link. I did all the steps to test it in minikube, but operator installation failed with following events:
Events:
  Type     Reason               Age                From                        Message
  ----     ------               ----               ----                        -------
  Normal   RequirementsUnknown  54m (x2 over 54m)  operator-lifecycle-manager  requirements not yet checked
  Normal   RequirementsNotMet   54m (x2 over 54m)  operator-lifecycle-manager  one or more requirements couldn't be found
  Warning  InstallCheckFailed   49m (x2 over 49m)  operator-lifecycle-manager  install timeout
  Normal   NeedsReinstall       49m (x2 over 49m)  operator-lifecycle-manager  installing: Waiting: waiting for deployment eunomia-operator to become ready: Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
  Normal   AllRequirementsMet   49m (x4 over 54m)  operator-lifecycle-manager  all requirements found, attempting install
  Normal   InstallSucceeded     49m (x3 over 54m)  operator-lifecycle-manager  waiting for install components to report healthy
  Normal   InstallWaiting       49m (x4 over 54m)  operator-lifecycle-manager  installing: Waiting: waiting for deployment eunomia-operator to become ready: Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
  Warning  InstallCheckFailed   44m (x2 over 44m)  operator-lifecycle-manager  install failed: Timeout: deployment eunomia-operator not ready before timeout: deployment "eunomia-operator" exceeded its progress deadline

It was required for testing to have eunomia in application registry. In kohlstechnology quay.io account we have it only in container registry, that's why I created my private one to push it there.

Looks like it's working after I added all the resources from deploy directory manually.
$ kubectl get clusterserviceversion -n marketplace
NAME             DISPLAY   VERSION   REPLACES         PHASE
eunomia.v0.0.2   Eunomia   0.0.2     eunomia.v0.0.1   Succeeded

I also downgraded to kubernetes 1.15.0 from 1.16.2, but I'm not sure if this was required. Will do more testing.

@akavel yes I believe we will need to do this with every time we release a new version. And yes in the future we will want to have as much automation as possible around this process, but for now we just want to see if we can get the first version listed on operatorhub.io.

@seanmalloy I created a PR: https://github.com/operator-framework/community-operators/pull/902
I mentioned you and @raffaelespazzoli in a comment there.
To see how exactly eunomia-operator will look like in operatorhub, please paste the contents of eunomia.v0.0.3.clusterserviceversion.yaml to operatorhub preview.
I tested eunomia-operator locally as described in manual tests.
I tested watching:

OwnNamespace (in this case both eunomia-operator and hello-world have been deployed to the marketplace namespace),
SingleNamespace (here eunomia-operator has been deployed to the marketplace namespace, and hello-world to the eunomia-hello-world-yaml-demo namespace),
AllNamespaces (eunomia-operator as above, hello-world to hello-world-yaml-demo and hello-world-yaml-demo2 namespaces).

Here is a screenshot from the last case:

CVP/operator-olm-deployment check is failing with the message:
install failed: Timeout: deployment eunomia-operator not ready before timeout: deployment "eunomia-operator" exceeded its progress deadline
That's because a manual step mentioned in my previous comment is required. Automatic check cannot perform it.

@seanmalloy #162 needs to be completed before eunomia is made available on operatorhub either way, but I now see the PR I created and @rhalat's temporary PR are wrong - where-to-contribute. We should be putting eunomia in upstream-community-operators directory, not community-operators.

@sbar95 I did not fully understand all of the steps to get an operator added to operatorhub.io when we started down this path. Thanks for your hard work on this!
Looking at the  where-to-contribute doc that you linked to I think we will want eunomia added to both upstream-community-operators and community-operators. We can start with just getting it added to community-operators for now.
I'll get #162 added to the v0.0.5 milestone.

Closing this PR. Created a new one
https://github.com/KohlsTechnology/eunomia/pull/235


Remove package-lock from source control

Since there are no dependencies, there is no ned for package-lock.json .

Done


feat: add support for observables to withDataService

This PR would close issue 85: suggestion/question: withDataService support for Observables
What was done

The interface for the implementing services that use withDataService was expanded to take Observable methods

type PromiseOrObservable<Entity> = Promise<Entity> | Observable<Entity>;
ex) loadById(id: EntityId): PromiseOrObservable<Entity>;

Existing functionality supports something like this

loadById(id: EntityId): Promise<Flight> {...}

These changes would now also support something like this

loadById(id: EntityId): Observable<Flight> {...}

The withDataService methods were adapted to take an observable (returns a subscription) as well as already taking promises

type PromiseOrSubscription<Entity> = Promise<Entity> | Subscription;
ex) [K in Collection as `load${Capitalize<K>}Entities`]: () => PromiseOrSubscription<void>;

The method checks if something is an instance of an observable. If so, the method manually pipes and subscribes. If not, the method resolves the call as a promise.

edit:
What was chosen not to be done, as per talk in the issue

For the moment, rxMethod is not used. The solution currently just manually subscribes. Supporting rxMethod would widen the feature to need to take non-static values such as signals or observables.

"I'd say for the first version, it should be fine of you check if the response is of type Promise or Observable. I guess you would do an automatic subscribe. If there's demand, we can later maybe add the rxMethod and map a Promise to an Observable." - Rainer

What is left to be done (before turning from Draft PR to regular PR)

Tests for both the promises and observables

There are not already existing tests for withDataService as is with promises, so I should probably make those first and get those in as a PR before moving onto this one. Thoughts?
Observable tests should follow
I am admittedly not very experienced/good at testing, so testing is probably the main hurdle now that implementation is hopefully all good apart from iteration on feedback.

Explicitly handle unsubscribing

HTTP observables tend to be considered self handling observables if I understand correctly, but it would be good to unsubscribe anyways
Pass in an optional DestroyRef? Otherwise basic take(1) or first or something?

I forgot to actually push up the usage of the RXJS equivalent service and point that service at the right endpoint... fixed https://github.com/angular-architects/ngrx-toolkit/pull/88/commits/47fc853d373905dd81c267ecc8ac972e071350f8


Add resume and interrupt events on the MediaSession object

Ideally, we should allow MediaSession instances decide how to behave when they are resumed or interrupted. Exposing events that could be cancelled would help. This is particularly important for Web Audio for which the best a UA can do is to break the link between the producing source and the output device. A MediaSession instance might stop the producing source and resume it when needed.
In addition, the interrupt event could have information about the ducking behaviour.
Both events could have information regarding how and why the session was resumed/interrupted. For example, whether it happened from a user interaction.

Yep, I agree that we should fire events for these state transitions. However, it does make it necessary to deal with the situation where a session is interrupted and in response the script does something weird like playing something else or calls preventDefault() without actually stopping playback itself. There's some relevant platform differences here. Android will happily let this kind of situation happen, while it looks like iOS will just mute the app but let it think that it's getting its way. Does that sounds right, @doomdavve?

Yes, the difference seems to be that on iOS apps can be muted by the system as dictated by the currently now playing app.

How detectable is that for the web page? Can we leave a door for the UA to ignore .preventDefault()? If my website is correctly written and I stop my audio processing and streams when the interrupt event is received and call preventDefault(), would the behaviour be any different between iOS and Android?

There is no issue for code that we would deem correct, all we have to contend with is accidental failure to pause primarily, deliberate abuse of such a difference seems less worrying, to me at least.
The way I see this, we ought to have a default behavior that pauses all participating media elements and mutes/suspends all participating audio contexts. So it comes down to whether or not the event is cancelable. I lean towards saying that at least the interrupt even should be non-cancelable, unless we can see some plausible use case that would justify the resulting incompatibility.

Closing this issue since we are moving audio focus out to a separate API.

This issue was moved to WICG/audio-focus#22


Please add support for TWA WebView Fallback

When a user starts a Trusted Web Activity (TWA) based application, a browser that supports TWAs may not be available on the device. In those situations, the default implementation will fallback to opening the URL in a Custom Tab or the default browser, if a browser that supports Custom Tabs is not found. This approach may be undesirable as it breaks the full-screen experience for users.
I would therefore suggest you to please add an option to the CLI that allow user to enable WebView Fallback with a single switch.
This could be based on
https://github.com/GoogleChrome/android-browser-helper/tree/5a0accde2341108fcc7eb7d0b0e0d7ce45a29962/demos/twa-webview-fallback

Thanks @naveedahmed1
The goal is to add the WebView fallback directly to android-browser-helper. The issue we're tracking this is here: https://github.com/GoogleChrome/android-browser-helper/issues/15, and the demo is the first step in that direction. We will start working on that soon.
Once that is done, llama-pack will automatically get it.

That's great, thanks for the update @andreban :)

#105 adds this

Update twa-manifest.json and include / update the fallbackStrategy field to webview.
Run npx @bubblewrap/cli update to apply the changes to the template
Run npx @bubblewrap/cli build to generate the APK.

We still default to customtab, as it provides better compatibility for Web APIs. We strongly recommend  to comprehensively test the PWA inside the WebView when using it.

bubblewrap is the new name for @llama-pack cli right?

Can you please also add the basic instruction about this library to project read me file https://github.com/GoogleChromeLabs/bubblewrap/blob/master/README.md covering at least these two topics:

Creating new project
Updating existing project
Different available settings

Work on more comprehensive docs will start soon.

Should a new project generated through @bubblewrap/cli already have the option fallbackStrategy?
I just installed @bubblewrap/cli and generated a new app, but twa-manifest doesn't contain fallbackStrategy.

We havent' published a version with the feature yet. Will publish it this week.


Maintenance: Lint/validate examples Iac

Summary
We should validate the example IaC:

validate the template (lint), ex: sam validate --lint
validate that it "works", ex: sam build (we already have the compilation with maven/gradle)

Not only for SAM but for others too.
Why is this needed?
Make sure the provided templates (IaC) are working.
Which area does this relate to?
Governance, Tests
Solution
Use sam validate --lint (see https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/validate-cfn-lint.html and https://github.com/aws-cloudformation/cfn-lint#config-file) + sam build for SAM.
See also https://awslabs.github.io/serverless-rules/cfn-lint/
For terraform, use terraform validate and tflint.
Acknowledgment

[X] This request meets Powertools for AWS Lambda (Java) Tenets
[ ] Should this be considered in other Powertools for AWS Lambda (Java) languages? i.e. Python, TypeScript

Please assign this to me.

note that it is done for terraform already: https://github.com/aws-powertools/powertools-lambda-java/blob/main/.github/workflows/pr_build.yml#L82-L109

Also, if you could extract this plus what you will do for SAM in a pr_iac_lint.yml, so that we don't have all these if java 11 everywhere.


feat: refactor ingest

Description
This refactors the current ingest CLI process to support better granularity in how the steps are ran

Both multiprocessing and async now supported. Given that a lot of the steps are IO-bound, such as downloading and uploading content, we can achieve better parallelization by using async here
Destination step broken up into a stager step and an upload step. This will allow for steps that require manipulation of the data between formats, such as converting the elements json into a csv format to upload for tabular destinations, to be pulled out of the step that does the actual upload.
The process of writing the content to a local destination was now pulled out as it's own dedicated destination connector, meaning you no longer need to persist the content locally once the process is done if the content was uploaded elsewhere.
Quick update to the chunker/partition step to use the python client.
Move the uncompress suppport as a pipeline step since this can arbitrarily apply to any concrete files that have been downloaded, regardless of where they came from.
Leverage last modified date to mark files to be reprocessed, even if the file already exists locally.

Callouts
Retry configs haven't been moved over yet. This is an open question because the intent was for it to wrap potential connection errors but now any of the other steps that leverage an API might run into network connection issues. Should those be isolated in each of the steps and wrapped with the same retry configs? Or do we need to expose a unique retry config for each step? This would bloat the input params even more.
Testing

If you want to run the new code as an SDK, there's an example file that was added to highlight how to do that: example.py
If you want to run the new code as an isolated CLI:

PYTHONPATH=. python unstructured/ingest/v2/main.py --help

If you want to see which commands have been migrated to the new version, there's now a v2 short help text next to those commands when running the current cli:

PYTHONPATH=. python unstructured/ingest/main.py --help
Usage: main.py [OPTIONS] COMMAND [ARGS]...main.py --help

Options:
  --help  Show this message and exit.

Commands:
  airtable
  azure
  biomed
  box
  confluence
  delta-table
  discord
  dropbox
  elasticsearch
  fsspec
  gcs
  github
  gitlab
  google-drive
  hubspot
  jira
  local          v2
  mongodb
  notion
  onedrive
  opensearch
  outlook
  reddit
  s3             v2
  salesforce
  sftp
  sharepoint
  slack
  wikipedia

You can run any of the local or s3 specific ingest tests and these should now work.

This is really cool! And its also a lot to understand. Could you add notes on how to test it out in the description?

are there doc changes that need to happen for users of the connectors via Python?

Overall I think there are a lot of good changes in here! This does come at the cost of quite a bit of complexity (though to be fair some of this is coming back in a bit cold). My biggest concern is that it's going to be tough for an average person to walk in and debug or be effective here, and currently you have to do a lot of tracing to grok what's going on. My main ask: can we equivalently (in size and scope) push the documentation? Including a lot more comments and docstrings as well as a comprehensive breakdown of everything happening with maybe some diagrams as well?


Where are the asset .obj files?

Hi,  I really appreciate you cleaning up the mess that was the original HelloAR.
For anyone interested, the Asset*.obj files are missing. You can get them from here:
https://github.com/xamarin/XamarinComponents/tree/master/Android/ARCore/samples/HelloAR/Assets

Thanks for tracking them down @ramiss! I'm sorry it took so long to get back to you. I'll put a pointer to them in the readme, too. Cheers 👍


if filter width =3, how to do fast inference?

In the new paper, Google use filter width =3 to increase the receptive field.
Then how could we do inference with filter width 3?
My idea is use to Queue, because the dilation is still 2 times increased, the first Queue is used to store the first half of middle value, and the second Queue is used to store the second half middle value.
Output of first Queue then be enqueued into the second Queue.
such as:
        current_state = q.dequeue()
        push = q.enqueue([current_layer])
        init_ops.append(init)
        push_ops.append(push)

        pre_state = None
        if self.filter_width == 3:
            q2 = tf.FIFOQueue(
                 1,
                 dtypes=tf.float32,
                 shapes=(self.batch_size, self.quantization_channels))

            init2 = q2.enqueue_many(tf.zeros((1, self.batch_size, self.quantization_channels)))

            pre_state = q2.dequeue()
            push2 = q2.enqueue([current_state])

            init_ops2.append(init2)
            push_ops2.append(push2)

        if self.filter_width == 2:
            current_layer = self._generator_causal_layer(
                            current_layer, current_state)
        if self.filter_width == 3:
            current_layer = self._generator_causal_layer(
                            current_layer, current_state, pre_state)

...
        with tf.name_scope('dilated_stack'):
            for layer_index, dilation in enumerate(self.dilations):
                with tf.name_scope('layer{}'.format(layer_index)):

                    q = tf.FIFOQueue(
                        dilation,
                        dtypes=tf.float32,
                        shapes=(self.batch_size, self.residual_channels))
                    init = q.enqueue_many(
                        tf.zeros((dilation, self.batch_size,
                                  self.residual_channels)))

                    current_state = q.dequeue()
                    push = q.enqueue([current_layer])
                    init_ops.append(init)
                    push_ops.append(push)

                    pre_state = None
                    if self.filter_width == 3:
                        q2 = tf.FIFOQueue(
                             dilation,
                             dtypes=tf.float32,
                             shapes=(self.batch_size, self.residual_channels))

                        init2 = q2.enqueue_many(tf.zeros((dilation, self.batch_size, self.residual_channels)))

                        pre_state = q2.dequeue()
                        push2 = q2.enqueue([current_state])

                        init_ops2.append(init2)
                        push_ops2.append(push2)

                    output, current_layer = self._generator_dilation_layer(
                        current_layer, current_state, layer_index, dilation,
                        global_condition_batch, local_condition, pre_state)
                    outputs.append(output)

is that make sense?

@twidddj , thanks very much.


HTTPS issues with macosx CURL

loading resources from github website (same as oryol does) requires HTTPS for new repositories - thus not seen as an issue with the oryol gh_pages repo.
error message from CURL:
curlURLLoader: curl_easy_peform failed .'Protocol "https" not supported or disabled in libcurl', httpStatus='301'
Reproduction : creation or usage of a new repository for data. Only happens in macosx (did not test android yet) works in windows.
-neshume

Yes I know, https in curl requires OpenSSL (not sure what other ssl are supported), and this complicates the build process, it will be the same problem on Android. I haven't looked at the curl build process for a while, I think they support cmake now and may be it became easier.
For local testing I usually run a local web server (e.g. node.js 'http-server', or 'python -m SimpleHTTPServer'), and temporarily hack 'http://<IP_ADDRESS>:xxxx/' into the code as source url.
On Windows I'm using native Win32 functions which automatically support https so the problem doesn't happen there.
There's also a similar code path for iOS/OSX, but this is currently only used on iOS, not on OSX (don't remember why I switched to curl for OSX, the native solution must have had some problems on OSX).

thx man .. I'll see what I can do about it cause reading resources from the bundle is a pain in the &&^*&^R&%&^^ ;o)
-neshume


add class to slick grid cells

I'd like to add a class to specific slick grid cells, for example, if the value of a cell anywhere on grid is less than 5 it should be green. I've looked at the example of conditional formatting which shows how to format columns and rows but not specific cells.
is there a method available that will tell me the value of a cell then apply a css class to it?

See https://github.com/volkanceylan/Serene/tree/master/Serene/Serene.Web/Modules/BasicSamples/Grids/ConditionalFormatting

Thanks @estrusco but as said I've seen that example which does column and row level. In either case I've found another example where it's possible to create custom formatter and as as attribute to row.cs which is much preferred.

for anyone who also needs this I accomplished with this code
    @Serenity.Decorators.registerFormatter()
    export class WimWinTelRotaOnCallFormatter implements Slick.Formatter {

        format(ctx: Slick.FormatterContext) {

            var onCallPattern = /\*/ig;
            var user: string = ctx.value;

            if (user.match(onCallPattern)) {

                return "
" + user + "
";
            }

            return user;
        }
    }

all the thanks goes to edson chen

I was looking for this since monday, thanks !
Here's the link for Edson's post:
https://github.com/volkanceylan/Serenity/wiki/SlickGrid-Formatters


Updated the userdata template to work properly with custom AMI

Few changes to allow proper work of osd-network-verifier with our AMI :

could not find a clean way to redirect all outputs to the console (found one requiring bash but could not get the generated usrcmd to not use sh) so staging the outputs to a file and sending it to /dev/console as last step
removing the start of docker as it is failing on our custom AMI (and faillure is detected as 'not internet connection'). From quick investigations, seems docker is aliased to podman in our AMI (except for systemd entries)

Codecov Report

Merging #108 (39f2edb) into main (a4b2576) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #108   +/-   ##
=======================================
  Coverage   26.39%   26.39%
=======================================
  Files          17       17
  Lines         682      682
=======================================
  Hits          180      180
  Misses        477      477
  Partials       25       25


Switch language lag

When changing the language between English and Español, there is sometimes a lag in the translations switching.

Duplicate of #263


Chorus depth doesn't smoothly oscillate

Try a chorus and set it's depth to something above 1 and you can hear that one of the LFOs aren't smoothly oscillating. It sounds a bit like it's set to a pulse wave.
Previously you used to be able to set the depth and delaytime high values and it would add interesting harmonies which sounded great.

I wasn't having this issue on Chrome, but when i tested it with Safari, I heard what you're referring to. The LFO was in fact being forced into a square wave shape. I traced this problem to discrepancies between the WaveShaperNode on the two platforms, specifically in Tone.AudioToGain which uses a WaveShaper to map incoming values between [-1, 1] to a range of [0, 1]. I think i've fixed the problem and i'll be committing those changes shortly.

What you're describing sounds right except it was happening for us in chrome. I copied in the commit but that didn't change anything. I haven't tested on your dev branch yet so maybe I'm missing some other LFO or WaveShaper changes.

When the depth is above 1, the square wave is inevitable with the current logic. The depth controls the deviation from the delayTime in each of the LFOs, so when the depth is above 1, they are trying to modulate to negative delayTime values and instead it just stops at 0.
I can add some logic to keep the min delayTime above 0 which will give you a smooth oscillation, but the depth and delayTime values will not really be accurate anymore if i do this since the effect will be equivalent to increasing the delayTime and decreasing the depth. Though, maybe this is the best path to go since modulating to a negative delayTime is undefined behavior.


Grimlite Rev Amogus

I'm getting this error, and when I'm in game my character doesn't load up

same also


Add Storybook field to set Cesium.Ion.defaultAccessToken

After some time, the default access token expires and Cesium Ion stops functioning.
It would be nice if the user could override this, so when accessing the examples at https://resium.darwineducation.com/examples/ we could just set our own user token somewhere and it be used so the examples work properly and consistently.

Hi, I've been fixed it, and now storybook is available. Thanks!


Xyz

Hi, thank you for your PR.
These changes are not relevant to the course content, since they are user-specific, and would break the learning material. Consequently, the changes would break the learning process for future students.
Please keep the changes in your fork, and continue the course with your fork. I am going to close this PR.
If you believe this to be an error, feel free to reach out, or open a new PR with further explanation.

Hi, thank you for your PR.
These changes are not relevant to the course content, since they are user-specific, and would break the learning material. Consequently, the changes would break the learning process for future students.
Please keep the changes in your fork, and continue the course with your fork. I am going to close this PR.
If you believe this to be an error, feel free to reach out, or open a new PR with further explanation.


Release v1.5.8 is broken? No Arm images and make fails

Hi,
Links provided for this release are broken - getting 404 on them:
https://releases.rancher.com/os/v1.5.8/arm64/initrd
https://releases.rancher.com/os/v1.5.8/arm64/vmlinuz
https://releases.rancher.com/os/v1.5.8/arm64/rootfs_arm64.tar.gz
https://releases.rancher.com/os/v1.5.8/arm64/rancheros-raspberry-pi64.zip
Also running 'make' fails with:
+ echo 'build-image: Building rancher/os-rootfs:fcdc9ca4'
build-image: Building rancher/os-rootfs:fcdc9ca4
+ '[' -x images/00-rootfs/prebuild.sh ']'
+ images/00-rootfs/prebuild.sh
+ dapper -d --build -f images/00-rootfs/Dockerfile -- -t rancher/os-rootfs images/00-rootfs
DEBU[0000] Building os:master using images/00-rootfs/Dockerfile
FATA[0000] pattern contains path separator
+ '[' 1 '!=' 42 ']'
+ exit 1
FATA[0025] exit status 1
make: *** [Makefile:11: default] Error 1

Looks like https://releases.rancher.com/os/v1.5.5/arm64/rancheros-raspberry-pi64.zip is the last working version

Caused by #3052 ?

Note that RancherOS has reached EOL #3000
I highly encourage you to join our journey with BurmillaOS (fork of RancherOS):

https://burmillaos.org/
https://github.com/burmilla/os/

On that side we already have also support for Raspberry Pi 4B: https://github.com/burmilla/os/releases/tag/v2.0.0-beta4

The issue is that the EOL is a very well kept secret. The websites still refer to Rancher/OS as a “alive and well” project and whatever is on github does not mention it either.
Ondrej
Get Outlook for iOShttps://aka.ms/o0ukef

From: Olli Janatuinen @.>
Sent: Thursday, April 29, 2021 9:13:01 AM
To: rancher/os @.>
Cc: Krajíček Ondřej @.>; Comment @.>
Subject: Re: [rancher/os] Release v1.5.8 is broken? No Arm images and make fails (#3050)
Note that RancherOS has reached EOL #3000https://github.com/rancher/os/issues/3000
I highly encourage you to join our journey with BurmillaOS (fork of RancherOS):

https://burmillaos.org/
https://github.com/burmilla/os/

On that side we already have also support for Raspberry Pi 4B: https://github.com/burmilla/os/releases/tag/v2.0.0-beta4
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/rancher/os/issues/3050#issuecomment-828995085, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AATIAPQVY3JCAKFISAC5OK3TLEBH3ANCNFSM4YGOV63A.

CONFIDENTIALITY NOTICE
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information.
If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify us by email to @.*** with a copy of this message. You must not, directly or indirectly, use, disclose, distribute, print or copy any part of this message if you are not the intended recipient. Y Soft and any of its subsidiaries each reserves the right to monitor all e-mail communications through its networks.
Y Soft is neither liable for the proper, complete transmission of the information contained in this communication nor any delay in its receipt. This email was scanned for the presence of computer viruses. In the unfortunate event of infection Y Soft does not accept liability.
Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to state them.

Looks like https://releases.rancher.com/os/v1.5.5/arm64/rancheros-raspberry-pi64.zip is the last working version

Thank you! I've been looking for the supported version and wasn't able to find it.


[Feature]: Implement encryption for personal ID data

For the highest-tier consent for re-contacting for research purposes, the data collected and stored should be encrypted. Find a library that implements this well using some kind of key stored in ENV

Related to #26


Monitoring non-output values

My network has two different outputs, let's call them A and B. While training, I have 10 monitoring values: loss, loss_A, loss_B, binary_accuracy_A, binary_accuracy_B, val_loss, val_loss_A, val_loss_B, val_binary_accuracy_A, val_binary_accuravy_B.
My problem is that I am not interested in all of these outputs. I would like to determine which are the ones I see and I want to create my own monitor value.
My own monitor value should be something like the percantage A > B.
Is this possible?

I believe you could do this by writing a custom callback: https://keras.io/callbacks/


Allow newlines and trailing commas in inline tables

Hi!
TOML allows newlines and trailing commas for inline tables:

https://github.com/toml-lang/toml/issues/516

Was wondering if we could support this as well
If you are open to it, then I can work on it

The new spec is not out yet and we are tracking it in #397