question about minimum volume for new segments - performance implications

When the quality engine is looking at node outflows, the following logic is used in determining if new segments are produced.
In short, the volume of a new segment must be greater than 0.0 and its quality value has to be outside of Ctol - otherwise no new segment is produced.
https://github.com/OpenWaterAnalytics/EPANET/blob/fbd005d6044cbd5dd0efddbc933be61c75a1a76c/src/qualroute.c#L341-L357
This approach has some pretty severe performance impacts on large networks, run for long times, that have dead-end pipes. After a while, even with large tolerance values, the number of segments skyrockets because the volume of these new segments, while greater than 0.0, is still vanishingly small.
I'm wondering about the introduction of a "volume tolerance" term here to skip new segment creation when the volume of the new segment would be negligible in the eyes of the user.

The segment explosion occurs mainly for Water Age simulations, yes? Instead of a volume tolerance, which may be difficult to have an intuitive feel for, what about a flow tolerance. Because EPANET's hydraulic solver can't allow a flow to be exactly 0, pipes downstream of a closed pipe or on the path to a dead end with no demand will always see some small amount of flow which I believe contributes to segment growth. The program already has an optional Flow Tolerance (FlowChangeLimit) used to determine hydraulic convergence. Maybe we can also use it to limit WQ segment growth as shown below:
    // ... case where link has a last (most upstream) segment
    seg = qual->LastSeg[k];
    if (seg)
    {
        // ... if node quality close to segment quality or inflow is negligible
        //    then mix the nodal outflow volume with the segment's volume
        if (fabs(seg->c - c) < qual->Ctol ||
            (hyd->FlowChangeLimit > 0.0 && fabs(LINKFLOW(k)) < hyd->FlowChangeLimit))
        {
            seg->c = (seg->c*seg->v + c*v) / (seg->v + v);
            seg->v += v;
        }

It would be easy to try out since only one line of code needs to be added.

that's a really good point, to lean on a convention that's already in use. I'll see if I can benchmark this kind of change in my worst case scenario - thanks for the feedback.

Lew and Sam, I think that a flow tolerance would work just as well as a volume tolerance, and as Lew suggested maybe that’s more intuitive to people (for example, it’s obvious that you’d want to choose a value somewhat larger than the current epanet minimum flow value).
But I think we should not link the resolution of this issue to a hydraulic convergence tolerance. That just seems confusing two different issues, and the one we’re dealing with here is super specific. I don’t think we should be concerned about a proliferation of tolerances – so long as they are very specific and have good default values. A “MinimumFlowForSegmentCreation” or “MinimumVolumeForSegmentCreation” tolerance would both be pretty easy to explain, it would be clear that they have a very restrictive scope, and they would both have good defaults of zero, which would not change current behavior.
-Jim
From: Lew Rossman @.>
Date: Tuesday, August 6, 2024 at 7:12 PM
To: OpenWaterAnalytics/EPANET @.>
Cc: Subscribed @.***>
Subject: Re: [OpenWaterAnalytics/EPANET] question about minimum volume for new segments - performance implications (Issue #813)
The segment explosion occurs mainly for Water Age simulations, yes? Instead of a volume tolerance, which may be difficult to have an intuitive feel for, what about a flow tolerance. Because EPANET's hydraulic solver can't allow a flow to be exactly 0, pipes downstream of a closed pipe or on the path to a dead end with no demand will always see some small amount of flow which I believe contributes to segment growth. The program already has an optional Flow Tolerance (FlowChangeLimit) used to determine hydraulic convergence. Maybe we can also use it to limit WQ segment growth as shown below:
// ... case where link has a last (most upstream) segment

seg = qual->LastSeg[k];

if (seg)

{

    // ... if node quality close to segment quality or inflow is negligible

    //    then mix the nodal outflow volume with the segment's volume

    if (fabs(seg->c - c) < qual->Ctol ||

        (hyd->FlowChangeLimit > 0.0 && fabs(LINKFLOW(k)) < hyd->FlowChangeLimit))

    {

        seg->c = (seg->c*seg->v + c*v) / (seg->v + v);

        seg->v += v;

    }

It would be easy to try out since only one line of code needs to be added.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/OpenWaterAnalytics/EPANET/issues/813*issuecomment-2272312072__;Iw!!OKzgfr8!YVCRQabWfyJynKgCklAOGwk3Gzv7fRhqllLlyXeB5ry-Lux0tRc8fFt6LV9OR7hy-o6sjzSWNbGV-4uzj57v0DbnUi8$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AASMACNFYN3KPZULEV47XG3ZQFJ3XAVCNFSM6AAAAABMC7UFQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZSGMYTEMBXGI__;!!OKzgfr8!YVCRQabWfyJynKgCklAOGwk3Gzv7fRhqllLlyXeB5ry-Lux0tRc8fFt6LV9OR7hy-o6sjzSWNbGV-4uzj57vfl4yLd0$.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc..

I agree with @jamesuber 's point. Using the existing hydraulic flow tolerance for water quality as well is just an easy way to test if it is effective in limiting WQ segment growth without sacrificing accuracy. To add a separate new WQ flow tolerance option requires making changes to 8 different files (which doesn't speak well for the overall design of the code base -- but that's a separate issue). We can do so if testing shows that a WQ flow tolerance is worthwhile.
P.S. Another option is to add a user-specified maximum segment limit for pipes. This was present in EPANET 1.x which used an Eulerian scheme for WQ routing, where each pipe was divided into a fixed number of equal size segments with each new flow solution.

I just found something interesting when simulating water age. If you set the Quality Tolerance to the reporting time step you can significantly reduce the number of segments used without affecting the reported results. I tested this on a 5,000 pipe network with a 72 hour duration. With a 1-hr reporting step there was a 10-fold decrease in segments as compared to the default tolerance of 0.01 hrs. With a 15-minute reporting step there was a 4-fold decrease. In both cases I couldn't see any difference in the time series of water age at selected nodes.

That is interesting Lew! I guess, that is basically guaranteeing that you will only get new segment creation every report time step (at most)?
For what it’s worth, the default tolerance of 0.01 maybe should be revisited, given that it’s a really small value for water age in hours, and water age seems to be the most common quality analysis by far. We’ve used a value of 1 hour routinely without ill effects, given that we aren’t concerned about changes in water age on the order of +/- an hour.
I’m curious if this cheap way out could also fix the problem of steadily increasing number of segments in dead end pipes with no demand.
Thanks, -Jim
From: Lew Rossman @.>
Date: Friday, August 9, 2024 at 12:13 PM
To: OpenWaterAnalytics/EPANET @.>
Cc: Uber, James - Xylem @.>, Mention @.>
Subject: Re: [OpenWaterAnalytics/EPANET] question about minimum volume for new segments - performance implications (Issue #813)
I just found something interesting when simulating water age. If you set the Quality Tolerance to the reporting time step you can significantly reduce the number of segments used without affecting the reported results. I tested this on a 5,000 pipe network with a 72 hour duration. With a 1-hr reporting step there was a 10-fold decrease in segments as compared to the default tolerance of 0.01 hrs. With a 15-minute reporting step there was a 4-fold decrease. In both cases I couldn't see any difference in the time series of water age at selected nodes.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/OpenWaterAnalytics/EPANET/issues/813*issuecomment-2278295379__;Iw!!OKzgfr8!dMVVhyoQWbgrbVysC7mR_WzuBh42H4fSVmCEFffBWmZU53blkTJIgkjJ96ur4a-EbjD0nuMhITzMgC18xuX7xPIYr-w$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AASMACIO3LAZBJXOO66THHLZQTTC7AVCNFSM6AAAAABMC7UFQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZYGI4TKMZXHE__;!!OKzgfr8!dMVVhyoQWbgrbVysC7mR_WzuBh42H4fSVmCEFffBWmZU53blkTJIgkjJ96ur4a-EbjD0nuMhITzMgC18xuX73FM0sqY$.
You are receiving this because you were mentioned.Message ID: @.***>
CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc..


Migration guide from 0.1 to 0.3

I've been trying to update some code written with futures 0.1 to use futures 0.3, but I've found that the API has changed so much that I just don't know how to proceed in some cases. I did find this helpful blog post: https://www.ncameron.org/blog/migrating-a-crate-from-futures-0-1-to-0-3, but that's about it. I think a migration guide would be really nice, and could help with more quickly moving the ecosystem from 0.1 to 0.3.
A migration document could contain the following sections:

A list of simple renames, e.g. try_ready -> ready.
How to update implementations of traits that have evolved, e.g. Future and Sink. (As that blog post notes, Sink in particular is not so easy to migrate.)
What to do for APIs that no longer exist, e.g. the Notify trait.

https://github.com/rust-lang/async-book seems like a good place for this to go. I suspect that unless someone takes the lead on authoring this document it won't happen though; @nicholasbishop would this be something you'd like to do?


Incorrect .env path in Go server

The Go server is using ../../.env as the .env path. According to your docs, the user should move the file to the chosen server directory. Accordingly the path should be .env.
If a user follows the instructions and opens the page in the browser, they'll get 404.
The node server on the other hand correctly checks .env

Thank you. Fixed with this: https://github.com/stripe-samples/set-up-subscriptions/commit/c160fa3f6c292a02f67d2f78ea7bbe123dd63010


The tendermint light client can not work in substrate host chain

Hi,guys:
I am using ibc-rs and substrate-ibc for cross-chain interaction testing, and my test environment is as follows：

Chain a:
substrate        V0.9.13
pallet-ibc        V3.0.0-pre.0
ibc-rs              V0.14.1

Chain b:
cosmos V0.44.0
ibc-go   V1.2.0

After testing, tendermint light  client can not work properly in substrate host chain, which I think may be related to the verification logic of tendermint light client!
I found that the tendermint light client needs to rely on the native data provided by the host chain to complete the verification, which may be the reason for the test failure. The context.rs files under ics02_client,ics03_connection and ics04_channel define some methods prefixed with host that need to return data on the host chain.
     /// Returns the current height of the local chain.
    fn host_height(&self) -> Height;

    /// Returns the current timestamp of the local chain.
    fn host_timestamp(&self) -> Timestamp {
        let pending_consensus_state = self
            .pending_host_consensus_state()
            .expect("host must have pending consensus state");
        pending_consensus_state.timestamp()
    }

    /// Returns the current height of the local chain.
    fn host_current_height(&self) -> Height;

    /// Returns the oldest height available on the local chain.
    fn host_oldest_height(&self) -> Height;

    /// Returns the ConsensusState of the host (local) chain at a specific height.
    fn host_consensus_state(&self, height: Height) -> Result<AnyConsensusState, Error>;

    /// Returns the pending `ConsensusState` of the host (local) chain.
    fn pending_host_consensus_state(&self) -> Result<AnyConsensusState, Error>;

If the tendermint light client works in the substrate host chain, then these methods prefixed with host return data from the substrate host chain, not native data from the tendermint host chain, so the validation does not pass.
I think that tendermint light client should not rely on host chain native data for verification if it wants to work properly in a heterogeneous environment, such as a substrate environment, I am not sure if this is the correct understanding?
Will the tendermint light client provide support for other heterogeneous chains in the future?

Hello @boern. Is this still an ongoing issue for you? Note that as of 0.20.0, the light-client API has changed to allow for any type that implements ConsensusState (or similarly for other types), instead of just Tendermint, as was the case before.

The process of consensus generation is different for different host chains, so is there any guidance on how to implement these two interfaces?

We have not addressed this problem yet though; that different light clients/hosts might have different requirements on each other (e.g. a GRANDPA light client might need different information from the host compared to a tendermint light client; similarly, a grandpa-based host might or might not be able to implement every method in the trait that a tendermint light client requires).
We would love your input on this if this is still relevant to you.

Closing as stale. Please re-open if this is still an issue for you.


chore(master): release 0.4.4

:robot: I have created a release beep boop
0.4.4 (2023-09-03)
Bug Fixes

cursor: optimized cursor synchronization debouncing (#1406) (7326aae)
mode: synchronize mode after startup (#1411) (3804505)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/vscode-neovim/vscode-neovim/releases/tag/v0.4.4 :sunflower:


python312Packages.pony: 0.7.17 -> 0.7.18

Description of changes
Diff: https://github.com/ponyorm/pony/compare/refs/tags/v0.7.17...v0.7.18
Changelog: https://github.com/ponyorm/pony/releases/tag/v0.7.18

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.


Develop

Current coverage is 100% (diff: 100%)

Merging #50 into master will not change coverage

@@           master   #50   diff @@
===================================
  Files          36    36
  Lines         593   593
  Methods       177   177
  Messages        0     0
  Branches        0     0
===================================
  Hits          593   593
  Misses          0     0
  Partials        0     0

Powered by Codecov. Last update cb45628...95a783a


Ideas for guiding devs to build performant apps and avoid common mistakes.

Was mulling this over the past few days; the biggest struggle we're having developing on top of Rust and Dioxus itself (not a fault of either of course) is bringing traditional web folks into the world of Rust; I was curious, would it be possible to convert things like componentDidMount or whatever the equivalent is now days into a scoped block that executes when the component is rendered into view, and "supporting" other features to get basic functionality. I suspect this will largely create confusion and anti-patterns carried over from JS but wanted to get the feedback of people more intelligent than myself before asking some poor soul on our team to go down trying to modify ASTs and whatnot.
I think the hardest thing to grasp has been (for me at least, a total noob to building anything real in rust) is the automatic execution of the code before the rsx macro on every re-render, once it's understood a lot of performance mistakes make themselves apparent and you start to understand why you're running at 200% cpu 😅.
TL;DR is there way we could help to onboard new devs (to the world of rust and dioxus) to help guide them to build performant applications using Dioxus and avoiding common mistakes that can cause larger issues and needless re-rendering of components? If so, is there any way we can help contribute to this effort?
Thank you in advanced for reading!

Node Refs
For Node refs and onmounted I opened a new issue to discuss ideas: https://github.com/DioxusLabs/dioxus/issues/631
We need some sort of solution here but it is probably best to find an api that works well both for rust and many different platforms before we implement anything. Different platforms handle nodes very differently so a small subset will probably be added first and then expanded. There are also serialization boundaries on some platforms (webview and liveview) that might make synchronous query's preform poorly.
Performance
The mental model of react/dioxus is very different from traditional programming. Docs is probably the easiest thing we can improve here, but there are also some areas we can improve tooling.
Docs
We could add more examples to the docs in the best practices section. It might be useful to see how react handles onboarding new devs with the new beta docs.
It's also worth noting that the master version of dioxus performs very differently from the 0.2.4 version and react. The previous version diffed every node every time; the master version only diffs dynamic nodes. This makes performance a lot better for most apps as we need to do less diffing, but for apps with similar branches it will preform worse. We can provide some diffing in the new version, but the more diffing we do the less of a benefit template will be.
For example in this code if x changes from true to false the div and text node will change in the dom in the master version. In the 0.2.4 version only the text would change.
if x {
  render!{
    div{ "0" }
  }
}else {
  render!{
    div{ "1" }
  }
}

What users should do instead is:
let y = if x {0} else{1};
render!{
  div{
    "{x}"
  }
}

The Docs should probably emphasize this for the next version.
Here are the relevant sections the guide:
Performance of updating state during rendering: https://dioxuslabs.com/nightly/guide/best_practices/antipatterns.html#avoid-updating-state-during-render
Hooks causing re-renders: https://dioxuslabs.com/nightly/guide/interactivity/hooks.html#hooks-and-component-state
Tooling
There are some tooling areas we can improve to make it more clear what is causing re-renders:

Adding a method that gives a reason for re-rendering https://github.com/DioxusLabs/dioxus/issues/475
A browser extension that visually shows how state flows between components. This would be more fine-grained and could provide a timeline of events (this event caused this specific state to update which re-rendered this component and caused this other state to update)


Add ContextEnumeratorT::unique and a default parameter

Often, ContextEnumerator is used to obtain a single item.  Provide a helper routine which checks the single item assumption and also simplify the constructor syntax to make ContextEnumeratorT easier to use.

Note: ContextEnumerator::iterator doesn't satisfy the requirements of the ForwardIterator concept (see http://en.cppreference.com/w/cpp/concept/ForwardIterator ), so it can't be used in std::next, for example.


Add test against node8 and node6

What does this PR do ?
This PR use the new core-dev container to run the test suite against Node8 and Node6.
The proxy image also use Node8. The image is built with a multi-stage build.
This image is build with Travis and no longer with Dockerhub automated build.
How should this be manually tested?

Step 1 : Run tests `sh test/travis-bin/run-tests.sh

Step 1 : Run docker-compose docker-compose -f docker-compose/dev.yml up

Other changes

Add PR/issue template

Codecov Report

Merging #102 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #102   +/-   ##
=======================================
  Coverage   97.78%   97.78%
=======================================
  Files          17       17
  Lines         723      723
=======================================
  Hits          707      707
  Misses         16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7636bb4...2f5ea7d. Read the comment docs.


[BUG]: Error: Unable to find integration meta info file

dmno version
0.0.21
Steps to reproduce
bunx dmno init
What is expected?
Probably not an error
What is actually happening?
✅ dmno already installed
📦 dmno @ "^0.0.21"
✅ DMNO + astro - integration already installed
📦 @dmno/astro-integration @ "^0.0.12"
📚 see docs @ https://dmno.dev/docs/integrations/astro
Error: Unable to find integration meta info file
at initDmnoForService (file:///D:/_dev/astro-playground/astro-nomy/node_modules/dmno/src/cli/lib/init-helpers.ts:90:14)
at DmnoCommand. (file:///D:/_dev/astro-playground/astro-nomy/node_modules/dmno/src/cli/lib/init-helpers.ts:446:3)
at DmnoCommand.parseAsync (D:_dev\astro-playground\astro-nomy\node_modules\dmno\node_modules\commander\lib\command.js:1092:5)
at file:///D:/_dev/astro-playground/astro-nomy/node_modules/dmno/src/cli/commands/init.command.ts:171:9
System Info
win32, bun 1.1.24, astro

Any additional comments?
No response

Closing for now - I think we tracked these issues down to windows support? For now we'll just recommend using WSL2.
Please reopen if you encounter similar with WSL2.


WIP add e-values support

as discussed with @tbng @bthirion @pneuvial, this PR aims at adding Knockoff aggregation using e-values as described in Ren and Barber 2022 (https://arxiv.org/pdf/2205.15461.pdf)

Thanks a lot! I think it is better that you also create a small test case to check the FDP and Power of the method on simulated data (ideally FDP < threshold and Power is greater than 0); you can check examples in the test folder.
A related thing is that maybe we need a general procedure for creating e-value, since as far as I'm aware there is no such implementation on Python.
Note that there are many unorganized things with the knockoff module that I might plan to do a refactoring soon, but I have not found the time yet. That being said, I will do it after the merge of this PR.

Indeed,  this PR obviously needs test + example.

Also note that Zhimei Ren has an R implementation here: https://github.com/zhimeir/derandomized_knockoffs_fdr

Resurrecting this PR as I'm working to finish it right now.


Map Review Process

add an isReviewed bool to maps
add admins table which holds a userId
a page which let's admins see all unreviewed maps
a button to let admin's accept the map
some how prevent the map from being played if it has no been reviewed.  You may need to make the level be optional on maps

working!


Automation for auto-sync of prod and stage databases

Recently I went through the process of downloading a recent prod database dump and restoring it into the stage ibutsu-server DB.
The steps looked something like:

Upload recent backup dump file to stage env PVC. This was done using the image quay.io/appuio/oc:v4.9 in the stage env. The steps looked like

oc new-app quay.io/appuio/oc:v4.9
oc env deployment/oc HOME=/tmp
Attach the stage postgresql PVC to the oc deployment (if ReadWriteOnce you have to scale the postgresql DC down to 1 pod)
oc debug deployment/oc
oc login to prod db
oc cp/oc rsync dump file to postgresql PVC
Scale the postgresql DC up again, drop the current ibutsu DB and restore pg_restore -v -C -d postgres < backup-ibutsu-<data>.dump
Scale up the backend, worker, and scheduler DC

It'd be nice if we did this regularly as a cronjob and had some automation around the process. Keeping the stage DB in sync with the prod DB will allow us to catch issues in the master branch of ibutsu that can't be caught in a local dev environment.

We should probably add a step in to anonymise the user data once that starts rolling in.


Fix exception keys so that values do not reappear on deletion

Checklist

[x] Appropriate branch selected (all PRs must first be merged to dev before they can be merged to master)
[x] Any modified or new methods or classes have helpful JSDoc and code is thoroughly commented
[x] The description lists all applicable issues this PR seeks to resolve
[x] The description lists any configuration setting(s) that differ from the default settings
[ ] All tests and CI builds passing

Description
Previously, issues with conflicting component keys were causing values to be transferred between exceptions mysteriously on deletion. This PR resolves that issue.

I'm a little confused why the e2e tests are failing on the CI, they pass locally for me:

well they're passing now.. the x files theme plays in my head whenever I run these things


Can I convert tf saved_models to tf/HLO dialect using iree

Hi,
Can I convert tf saved_models to tf/HLO dialect using iree? If yes what are the commands I should use?
Thanks

Iree does that for its own purposes but doesn't have a general way (ie. We convert to a mix of hlo and other dialects based on our requirements). The upstream tensorflow project has some standalone tools that people have used for this, though.

You are, of course, free to look at the iree-import-tf source for inspiration: https://github.com/google/iree/blob/main/integrations/tensorflow/iree_tf_compiler/iree-import-tf-main.cpp


Grant @vlatombe permissions to release kubernetes plugin

I would like to co-maintain the plugin. CC @carlossg who is a maintainer now
When adding new uploaders (this includes newly created permissions files)

[X] Make sure to @mention an existing maintainer to confirm the permissions request, if applicable
[X] Use the Jenkins community (LDAP) account name in the YAML file, not the GitHub account name
[X] All newly added users have logged in to Artifactory at least once

Explicit authorization is needed

@oleg-nenashev done


[SPARK-14642][SQL] import org.apache.spark.sql.expressions._ breaks udf under functions

What changes were proposed in this pull request?
PR fixes the import issue which breaks udf functions.
The following code snippet throws an error
scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> import org.apache.spark.sql.expressions._
import org.apache.spark.sql.expressions._

scala> udf((v: String) => v.stripSuffix("-abc"))
<console>:30: error: No TypeTag available for String
       udf((v: String) => v.stripSuffix("-abc"))

This PR resolves the issue.
How was this patch tested?
patch tested with unit tests.
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Can you update the pull request to have the full title and alos fix the description to elaborate the problem? Thanks.

Hi @rxin , I updated the title & description. Please have a look.

Can you try
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions._
import org.apache.spark.sql.types._

udf((v: String) => v.stripSuffix("-abc"))

and see if it throws something like the following exception?
error: object lang is not a member of package org.apache.spark.sql.expressions.java
      val normalizeVersion = udf((v: java.lang.String) => v.stripSuffix("-abc"))

Hello @yhuai ,
I tested the scenario you mentioned & it works fine for me without any errors.
However, as you rightly mentioned, the following statement need not be removed.
import implicits._

Hence, I reverted that particular change. Please comment.

Can one of the admins verify this patch?

@sbcd90 could you resolve the conflicts? thanks!

Hello @zsxwing ,
I have resolved the conflicts. Please have a look.

ok to test

Test build #58093 has started for PR 12458 at commit 0d2310f.

Test build #58093 has finished for PR 12458 at commit 0d2310f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58093/
Test FAILed.

Merged build finished. Test FAILed.

jenkins test this please

Test build #58105 has started for PR 12458 at commit 80af641.

Test build #58105 has finished for PR 12458 at commit 80af641.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58105/
Test PASSed.

Merged build finished. Test PASSed.

Hello @zsxwing ,
Can this be merged now? I fixed the scala style tests.

LGTM. @rxin what do you think about the new package name?

maybe change it to scalalang and javalang.
otherwise scala and java have different hierarchy.

Hello @rxin , @zsxwing ,
I made the suggested changes.

Test build #58173 has started for PR 12458 at commit 139e1b9.

Test build #58173 has finished for PR 12458 at commit 139e1b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58173/
Test PASSed.

LGTM. Merging to master / 2.0

Thanks! @sbcd90


Colour as Input

It would be a nice feature to set the colour via Input message.
My use case is, that I want to show two different colours for one Button.
Example:  Switch Button "Auto":   State Auto on -> Auto Button Green,  State Auto off -> Auto Button Red

Not sure about this. By supporting this I see the perspective which leads to having rebuild whole node on demand and that is against of concept of dashboard and not really supported by angular. Yes it can be done but introduces a lot of edge cases to cover and that all makes widget unreliable.

@hotNipi : I will have to trust on your CSS knowledge for this request...
Is it really needed to rebuild the whole node when the color needs to be updated?

For color I'd think not, but that is already part of options list properties which by nature is dangerous thing to change. Widget behavior does not rely on button text and color, but definitely on count of options and state  id-s. And again - the option change will not be permanent so the result is always half solution. Be prepared to answer the questions why it does not remember the change and how should the one set up the flow to always show the widget like the user wants. Replay-messages, init state, reload, connection loses, using the ui_control node ... all that stuff.


fix: clean dom when tooltip unmounts

This removes leftover dom nodes when the component unmounts
we are displaying paginated search results where hovering a search result shows some additional details. We are finding that users who paginate a lot will fill up the dom with a lot of empty

 as they move to the next page
if you want I could also flag this behind a prop cleanDOMOnUnmount with a default of false

if you want I could also flag this behind a prop cleanDOMOnUnmount with a default of false

Definitely not. Don't want an apropcalypse!


Testcafe never exits after successful tests using browserstack in docker

What is your Test Scenario?
Asserting true on google.com using browserstack (any browser) in container using node:12.16.1-buster image
What is the Current behavior?
After tests complete the process never exits
What is the Expected behavior?
Testcafe finishes after successful tests
What is your web application and your TestCafe test code?

[Your website URL (or attach your complete example):](www.google.com)

Your complete test code (or attach your test files):

import { t } from 'testcafe';
fixture('google').page('www.google.com');

test.only('example', async () => {
  await t.expect(true).ok();
});

Your complete configuration file (if any):

BROWSERSTACK_NO_LOCAL=1
BROWSERSTACK_USERNAME=<username>
BROWSERSTACK_ACCESS_KEY=<password>

Your complete test report:

 Running tests in:
 - Safari 13.0.3 / iOS 13.2
 (https://automate.browserstack.com/builds/...)

 google
 ✓ example

 1 passed (16s)

--- process never exits after this point ---

Steps to Reproduce:

Execute docker run -v $PWD:/tests -w /tests -it node:12.16.1-buster npm run testcafe -- "browserstack:iPhone 11 Pro Max" "src/tests/smokeTest.js" "-q"
Skipping the only test also results in the process never exiting
If I add process.exit(0) inside the test, it exits correctly

Your Environment details:

testcafe version: 1.8.3
node.js version: 12.16.1
command-line arguments:  "browserstack:iPhone 11 Pro Max" "src/tests/smokeTest.js" "-q"
browser name and version: Safari 13.0.3 / iOS 13.2
platform and version: macOS Mojave v10.14.6 (also seeing the issue on our bamboo agent)
other

Issue only appeared after upgrading testcafe from version "0.23.2" to "1.8.3"

If I add process.exit(0) inside the test, it exits correctly

I was able to reproduce the issue only when the BROWSERSTACK_NO_LOCAL variable was set to 1. Could you please run the test without this setting?
Please also specify the version of testcafe-browser-provider-browserstack you use with<EMAIL_ADDRESS>

I am still getting the issue without BROWSERSTACK_NO_LOCAL set on my end
testcafe-browser-provider-browserstack was previously at version 1.8.0

Like @AlexKamaev, I was able to reproduce the issue only when the BROWSERSTACK_NO_LOCAL variable was set to 1.
Could you please run the following commands in your terminal?
echo %BROWSERSTACK_NO_LOCAL%
testcafe "browserstack:iPhone 11 Pro Max" test.js
set BROWSERSTACK_NO_LOCAL=1
echo %BROWSERSTACK_NO_LOCAL%
testcafe "browserstack:iPhone 11 Pro Max" test.js

and share the screenshot of the terminal after the commands are executed.

Hi @miherlosev
Adapting the above for bash:
echo "$BROWSERSTACK_NO_LOCAL"

testcafe "browserstack:iPhone 11 Pro Max" src/tests/sample.js
Using locally installed version of TestCafe.
 Running tests in:
 - Safari 13.0.3 / iOS 13.2.3
 (https://automate.browserstack.com/builds/<buildid>)

 google
 ✓ example

 1 passed (13s)

BROWSERSTACK_NO_LOCAL=1

echo "$BROWSERSTACK_NO_LOCAL"
1

testcafe "browserstack:iPhone 11 Pro Max" src/tests/sample.js
Using locally installed version of TestCafe.
 Running tests in:
 - Safari 13.0.3 / iOS 13.2.3
 (https://automate.browserstack.com/builds/<buildid>)

 google
 ✓ example

 1 passed (11s)

However as per my question if I do the above via a Docker container I still see the process fail to exit

Hello @ljowen,
Could you please send us your project (at least, the package.json file) so that we can research it on our side? Also, please make sure that you have updated the testcafe-browser-provider-browserstack version to the latest build.

Here is my package.json:
{   "name": "e2e-tests",   "version": "1.4.0",   "private": true,   "description": "",   "main": "index.js",   "engines": {     "node": ">=12.14.0",     "npm": ">=6.13.4"   },   "author": "",   "license": "ISC",   "dependencies": {     "glob": "7.1.3",     "pretty": "2.0.0",     "request-promise": "4.2.2",     "testcafe": "1.8.3",     "testcafe-browser-provider-browserstack": "1.12.0",     "testcafe-browser-tools": "2.0.10"   },   "devDependencies": {     "axe-testcafe": "1.1.0",     "babel-cli": "6.26.0",     "babel-eslint": "10.0.1",     "babel-preset-flow": "6.23.0",     "browserstack-local": "1.3.6",     "dotenv": "6.0.0"   } }
The sample project I've created just consists of the test file included above and this package.json.
Do you mean the v1.13.0-alpha.1 or is 1.12.0 fine for the browserstack provider?

Thank you for your package.json file. However, I was unable to reproduce the issue. Please give us additional time to research it.

Update on my end, the tests all pass fine if I use the automate api. BROWSERSTACK_USE_AUTOMATE=1 I switched to the JS api as we were experiencing a lot of sessions hanging and using the JS api seemed to improve that. I've contacted browserstack to see if they've made any changes / deprecated the JS api.

@ljowen
I ran tests on "browserstack:iPhone 11 Pro" (and other iPhones) - they are not executed with BROWSERSTACK_USE_AUTOMATE=0. I can reproduce the issue even without a docker.
It looks like Browserstack introduced some changes that caused hanging. So, I will close this issue since it doesn't occur on our side.
If you have any further questions, feel free to contact us. We will reopen the issue if some modifications are required on our side.


[SPARK-36352][SQL] Spark should check result plan's output schema name

What changes were proposed in this pull request?
Spark should check result plan's output schema name
Why are the changes needed?
In current code, some optimizer rule may change plan's output schema, since in the code we always use semantic equal to check output, but it may change the plan's output schema.
For example, for SchemaPruning, if we have a plan
Project[a, B]
|--Scan[A, b, c]

the origin output schema is a, B, after SchemaPruning. it become
Project[A, b]
|--Scan[A, b]

It change the plan's schema. when we use CTAS, the schema is same as query plan's output.
Then since we change the schema, it not consistent with origin SQL. So we need to check final result plan's schema with origin plan's schema
Does this PR introduce any user-facing change?
No
How was this patch tested?
existed UT

Test build #142318 has finished for PR 33703 at commit e9d7595.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142318/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46826/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46826/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47061/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47061/

Test build #142559 has started for PR 33703 at commit e6a80c3.

FYI @cloud-fan

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47062/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47062/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47062/

Test build #142559 has finished for PR 33703 at commit e6a80c3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142559/

retest this please

Test build #142738 has started for PR 33703 at commit e6a80c3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47238/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47238/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47238/

Test build #142738 has finished for PR 33703 at commit e6a80c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142738/

ping @cloud-fan & should not merge this to spark-3.0

thanks, merging to 3.1!


[azservicebus] If the links instance has an old connection ID, recover the links (not the connection)

(Found this while running tests (and some simulations) for azeventhubs in PR#19281
AMQPLinks relies on Namespace to determine if it should recover links. I had code in there that would return true if the connection was recreated, but missed the case where the connection was NOT recreated but the user's connection ID was outdated.
The odds of this happening increase the more Receivers or Senders you have on a single Client instance.

/azp run go - azservicebus


fix windows update

The tmpBinPath did not have the .exe extension on Windows which would cause the update to fail because Windows only considers files that correlate with the PATHEXT variable and that does not include no extension files.
Closes #55

Merged! Thanks :)

Tagged v1.1.6

wow thanks for merging it so quick @jpillora. Awesome.


[SPARK-33423][ML] DoubleParam/FloatParam parse JSON support type conversion

What changes were proposed in this pull request?
Make DoubleParam/FloatParam parse JSON support simple type conversion
Why are the changes needed?
as https://issues.apache.org/jira/browse/SPARK-33423 show,  DoubleParam can't parse a JSON number without decimal places,  we can support it.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added UT

Test build #131532 has started for PR 30462 at commit 2a9ac2a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36135/

Test build #131532 has finished for PR 30462 at commit 2a9ac2a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131532/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36135/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36135/
Test PASSed.

Test build #131535 has started for PR 30462 at commit 1625da5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36139/

Test build #131535 has finished for PR 30462 at commit 1625da5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131535/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36139/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36139/
Test FAILed.

Test build #131588 has started for PR 30462 at commit 1625da5.

Test build #131588 has finished for PR 30462 at commit 1625da5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131588/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36189/


Please update and Notify

Spring Creators Update contains lots of new glyphs. Please document those glyphs and notify when doing it. Thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 633955d7-89b3-04a5-ad62-4fa860785150
Version Independent ID: ce94b443-0257-8414-0c12-0490c0bad8b6
Content: Segoe MDL2 icon guidelines - UWP app developer
Content Source: windows-apps-src/design/style/segoe-ui-symbol-font.md
Service: windows
GitHub Login: @mijacobs
Microsoft Alias: mijacobs

Thanks for the feedback--the update is in progress.

THIS. This page is a disaster to navigate and use and it's missing so much

Checking on the updates.

@ahainen can you elaborate on what's missing? Do you have a more specific suggestion about what could be improved?

@ahainen can you elaborate on what's missing? Do you have a more specific suggestion about what could be improved?

https://youtu.be/RzohF-Wbdy8

It's in our queue to update this. Thanks!

It's in our queue to update this. Thanks!

It's in our queue to update this. Thanks!

https://github.com/MicrosoftDocs/windows-uwp/issues/411

We published the updated on June 21st. Please re-open if there are items missing.
https://github.com/MicrosoftDocs/windows-uwp/commit/d5667ea32e90acef44f903806aad356813ed5e62#diff-e461a5aa680b93a61bdac9007555bbf6


Missing file from Verilog file list

Could this missing Verilog module file:
+${ADAMSBRIDGE_ROOT}/src/ntt_top/rtl/ntt_shuffle_buffer.sv
be added to this file list:
src/mldsa_top/config/mldsa_top.vf
to fix Verilog compilation error?
========
Additionally, the following two lines cause compilation errors:
${ADAMSBRIDGE_ROOT}/src/ntt_top/tb/ntt_ram_tdp_file.sv
${ADAMSBRIDGE_ROOT}/src/ntt_top/tb/ntt_wrapper.sv
However, excluding the above two lines from the file list allows Verilog compilation to finish error-free, so perhaps these two lines should be deleted.
Thank you.

Thanks for raising this issue. It should be fixed now with the latest commit. Please check and let me know if you see any other issues.

The compilation completes error-free now.  Thank you!


Go 1 13 compatibility.

The new code following @yahavi's PR was committed to the dev branch via https://github.com/jfrog/jfrog-cli/commit/a28356964f04658dfa414018f8116894049c0d8f


Make a meaningful change for upgrade test.

Previously, we were making a change to README.md. This is no longer triggering a new image to be built. This actually makes sense that it should only trigger on a meaningful change to the source code.
Root cause for the change: https://github.com/paketo-buildpacks/go-build/pull/335
Co-authored-by: Rasheed Abdul-Aziz<EMAIL_ADDRESS>

Needs a uname check for Darwin and switch sed implementations, or some cross platform edit instead.


luci-proto-vxlan does nothing

Installed luci-proto-vxlan but still got unsupported info.

You need to restart netifd/network for this to appear.


[Feature] RayGui?

Is your feature request related to a problem? Please describe.
I need a gui toolkit and imgui tends to crash alot due to the unconstrained function nature of it
Describe the solution you'd like
Can we get a wrapper for RayGui?
Describe alternatives you've considered
ImGui.NET
Additional context
Maybe a layout and docking system could be included in your version of the wrapper.

Unsure of what kind of DearImGui code you're writing for it to crash a lot, but I do agree with the need for raygui bindings. However a layout and docking system wouldn't really fit in with bindings in my opinion.

Is your feature request related to a problem? Please describe. I need a gui toolkit and imgui tends to crash alot due to the unconstrained function nature of it
Describe the solution you'd like Can we get a wrapper for RayGui?
Describe alternatives you've considered ImGui.NET
Additional context Maybe a layout and docking system could be included in your version of the wrapper.

Hello, i wanna do a own project for RayGui called Raylib-CSharp.RayGui. (WIP)

Hi! Very helpful library! I was also looking for RayGui bindings the same way. Do you have an ETA when your RayGui bindings will be available? :)


Fix multi-GPU training

Codecov Report

Merging #131 into master will decrease coverage by 0.08%.
The diff coverage is 4.87%.

@@            Coverage Diff             @@
##           master     #131      +/-   ##
==========================================
- Coverage   39.46%   39.37%   -0.09%
==========================================
  Files          83       83
  Lines        9991     9996       +5
  Branches     2106     2106
==========================================
- Hits         3943     3936       -7
- Misses       5850     5861      +11
- Partials      198      199       +1

Impacted Files
Coverage Δ

neural_sp/bin/asr/train.py
0.00% <0.00%> (ø)

neural_sp/bin/lm/train.py
0.00% <0.00%> (ø)

neural_sp/models/data_parallel.py
0.00% <0.00%> (ø)

neural_sp/models/base.py
43.10% <100.00%> (-2.21%)
:arrow_down:

...eural_sp/models/seq2seq/decoders/rnn_transducer.py
74.92% <0.00%> (-0.95%)
:arrow_down:

neural_sp/models/seq2seq/decoders/transformer.py
62.24% <0.00%> (-0.21%)
:arrow_down:

neural_sp/models/modules/cif.py
88.60% <0.00%> (+1.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 634765f...5a8a89d. Read the comment docs.


Switch to rye for building

@armaan-abraham I've set up a rye build here. I've noticed that you added several ruff rules in your other repositories. Are there any you think that should be included here? Others will likely copy these settings to other repos.

@armaan-abraham I've set up a rye build here. I've noticed that you added several ruff rules in your other repositories. Are there any you think that should be included here? Others will likely copy these settings to other repos.

Here are the descriptions for each of the rules I specified. Only "E" (pycodestyle) and "F" (pyflakes) are included by default.
"E" (pycodestyle):
These are style checks from pycodestyle, which enforces style conventions from PEP 8, the official Python style guide.
Examples include checks for indentation, line length, and whitespace usage.
"F" (Pyflakes):
Pyflakes performs various static checks to find errors in Python code without actually running it.
It can detect issues like undefined names, unused imports, and duplicate arguments in function definitions.
"UP" (pyupgrade):
This rule set suggests ways to upgrade your Python code to use newer language features.
It might suggest using f-strings instead of .format() or % formatting, for example.
"B" (flake8-bugbear):
Bugbear is a plugin that detects various bugs and design problems in your code.
It can find issues like likely bugs, risky code patterns, or complexity issues.
"SIM" (flake8-simplify):
This plugin suggests simplifications for your code.
It might recommend using return x > y instead of if x > y: return True else: return False, for example.
"I" (isort):
isort is a utility that sorts and formats import statements.
It ensures your imports are organized consistently and grouped properly.
"ARG" (Unused arguments):
This rule checks for unused function arguments.
It helps identify parameters that are defined but not used within a function.
There is a tradeoff between allowing lab members who are less proficient in python code faster and maintaining code quality off course. I would say that they're all worth it actually. All of the above except isort would seem to be worth the extra manual work, and ruff automatically sorts the imports according to isort so that's also covered.

@armaan-abraham I agree that these are all useful. I think these added guardrails are especially helpful for novice coders because they are often least familiar with what readable code looks like. The experts have a better feel for when to break the rules.
Could you add these to the pyproject.toml?


Document stability attributes?

I was trying to hack in libcore and when compiling it I got:
error: This node does not have a stability attribute

and couldn't find anything detailing stability attributes.  Should that info live here?

There's some detail in https://github.com/rust-lang/rust/blob/1.30.1/src/libsyntax/attr/builtin.rs#L104-L123.

@dwijnand We now have https://rust-lang.github.io/rustc-guide/stability.html
I will close this issue, but feel free to reopen or create a new issue.


Implement DMA based on embedded-dma traits

DMA implementation based on the embedded-dma traits and the existing implementation in the stm32f4xx-hal. ~Currently some parts are very similar - the Stream trait and its implementation are completely identical similar, and much of dma/mod.rs is similar. However it remains to be seen how much will stay the same when implementing the BDMA and/or MDMA.~ However some changes are required since we'd like to support the BDMA and MDMA also.
Implementation / examples for:

[x] Memory-to-Memory
[x] SPI TX
[x] I2C RX

TODO:

[x] BDMA implementation
[ ] Confirm that owning the target peripheral is the right API design
[x] ~Do the compiler fences also need matching synchronisation~ Memory barriers for the Cortex-M7
[x] Add an example using RTIC - Thanks ryan-summers!

Contributions to this branch are very welcome, especially implementations and examples for other peripherals.
Ref #80

I've just pushed a proof-of-concept example using the SAI peripheral with DMA:
https://github.com/antoinevg/stm32h7xx-hal/blob/sai-dma/examples/sai-dma-passthru.rs
I coded this up before you pushed this branch so I'll probably need to go back and update it for your changes!

Please excuse me if I'm a little off-track here but I'm still very new to embedded-hal implementation!
re: Soundness of taking a mutable reference to the target peripheral, rather than consuming it completely
Are we assuming that a given peripheral will only ever have a single DMA stream associated with it?
The SAI peripheral, in particular, has two channels: A & B
Each of these channels has its own DMA stream associated with it.
Usually configured as one stream each for transmitting and receiving audio data.
So there are two related issues here I think:

If we are passing a mutable ref or moving the entire peripheral into the Transfer constructor for SAI Channel A then I can no longer create a second Transfer for SAI Channel B.
Because the data register for each SAI channel is a sub-register we cannot pass it into the peripheral_target_address! macro.

Would it make sense to modify Transfer to just accept the address of the data register rather than the whole peripheral?
i.e. something like:
let mut dma1_str0: Transfer<_, _, MemoryToPeripheral, _> = Transfer::init(
    dma1_stream.0,
    sai1.target_address(),
    tx_buffer,
    None,
    dma1_config,
);

let mut dma1_str1: Transfer<_, _, PeripheralToMemory, _> = Transfer::init(
    dma1_stream.1,
    sai1.target_address(),
    rx_buffer,
    None,
    dma1_config,
);

And also modify the peripheral_target_address! macro to allow us to specify a sub-register using something like:
peripheral_target_address!(
    (pac::SAI1, cha.dr, u32, M2P, DMAReq::SAI1A_DMA),
    (pac::SAI1, chb.dr, u32, P2M, DMAReq::SAI1B_DMA),
);

@antoinevg
My expectation is that it would work like Serial.split() but returning some Channel<> struct that's just used for DMA.
I'm not familiar with the SAI peripheral but for example if you need to ensure the peripheral settings aren't changed during the DMA transfer, do it like Serial.split() and consume the serial port while the DMA is happening, then you can have a function that consumes the Channels and gives you access to the peripheral again.
If you don't need to ensure that peripheral settings aren't changed during the DMA transfer you can just have the peripheral store an Option<Channel> that you take() so you can't get multiple copies of the channel.
If you need to ensure that only some peripheral settings aren't changed during the DMA transfer you can do something like Serial.split() but also return a version of the peripheral with some typestate so that certain functions aren't implemented in that state (see spi::Enabled and spi::Disabled).

I've had some DMA code using the serial peripheral: https://github.com/mattico/stm32h7xx-hal/tree/add-dma-example
I'm going to rebase it on this and try to modify the HAL serial to support DMA.

@richardeoin Re: Do the compiler fences also need matching synchronisation barriers for the Cortex-M7?
https://community.st.com/s/article/FAQ-DMA-is-not-working-on-STM32H7-devices has good information about this if you haven't seen it.
From what I remember, I couldn't get DMA to work with the serial port without disabling the dcache on the DMA buffers. No combination of dsb() compiler_fence() or even not enabling the global dcache (!) worked. (so I was probably doing something wrong...)

With DMA in a cached region you'd have to use the cache control operations to specifically invalidate (on DMA writes) or clean (on DMA reads) the memory in use for DMA.

@mattico

My expectation is that it would work like Serial.split() but returning some Channel<> struct that's just used for DMA.
<schnip/>

Thank you for taking the time to answer.
I think I understood about 1% of what you were saying there, I really meant it when I said I'm new to embedded-hal development! 🤣
Could you show what you mean in a few lines of hypothetical code? Alternatively, is there a tutorial or documentation I could read that can help bridge the gaps between the working SAI/DMA implementation code I posted earlier in this thread, my two questions and your reply?
I could probably piece things together on my own from the sources but time is expensive and it would be cool if I could put the little I have towards contributing functionality rather than reverse engineering!

Hi @antoinevg @mattico ! Those examples are great, thanks for sharing them. Also nice that you're trying to update them to work with this PR.
Indeed DMA with the SAI isn't well supported without several changes, so right now you're running a bit ahead of where this PR is @antoinevg

Are we assuming that a given peripheral will only ever have a single DMA stream associated with it?

Yes, the current instances of peripheral_target_address!( assume that a given peripheral only has one associated DMA stream (at once). That's a result of us representing ownership to the compiler in blocks of 'peripheral', which is the approach most current PACs take (but not the only possible approach).
Indeed the approach taken by the serial HAL would be a good option. Here's the function where the serial HAL (which owns the serial PAC) is split into two ZSTs. Here we're using ZSTs to represent ownership, whilst not consuming resources at runtime.
You'd then implement the DMA like so:
peripheral_target_address!(
    (Rx<pac::USART1>, rdr, u8, P2M, DMAReq::USART1_RX_DMA),
    (Tx<pac::USART1>, tdr, u8, M2P, DMAReq::USART1_TX_DMA),
);

Potentially this approach would work for the SAI channels too.

@mattico The link about memory layout and dcache is useful, although it's a separate issue (I think). This PR doesn't consider the dcache at all yet, it just assumes it's off.

@antoinevg I added an extra option to the peripheral_target_address! macro, so the proposal from your comment above should compile now if you add it!
peripheral_target_address!(
    (pac::SAI1, cha.dr, u32, M2P, DMAReq::SAI1A_DMA),
    (pac::SAI1, chb.dr, u32, P2M, DMAReq::SAI1B_DMA),
);

If you have a neater way of modifying the macro that's also great, I didn't spend too much time thinking about it.
Of course it doesn't solve the problem with actually initialising both streams at the same time, since they both want a mutable reference to pac::SAI1

@antoinevg I added an extra option to the peripheral_target_address! macro, so the proposal from your comment above should compile now if you add it!

Awesome, working nicely thank you!

Of course it doesn't solve the problem with actually initialising both streams at the same time, since they both want a mutable reference to pac::SAI1

We also need access to pac::SAI1 after DMA has started up.
The SAI peripheral is a bit weird in that it needs DMA to be up & running before setting you can set its dmaen and en bits.
I see that transfer.start(|sai1| {}) supplies a reference to the peripheral's register block so I could just twiddle the bits directly.
That said, wouldn't it be better to rather have the hal peripheral wrapper passed so we can call e.g. enable_dma() and fifo_is_empty() methods?

I see that transfer.start(|sai1| {}) supplies a reference to the peripheral's register block

Yup, this closure allows access to whatever memory the Transfer has exclusive mutable access to. That that would be the place to set the dmaen and dma bits. I experimented with implementing TargetAddress for both the PAC and HAL types for I2C. It seems to work well, so I think that can be done for other peripherals too.

It looks like the SAI peripheral needs the DMA to be fully configured before it can be successfully configured

Ah, you mean that the DMA must be configured before the call to SAI1.i2s_ch_a(...)? That does make it very difficult to give the transfer exclusive ownership to anything.
If you're willing to use unsafe, then working around this is quite easy. The most obvious escape hatch that already exists is the steal() method.
let mut sai1_a_for_dma = unsafe { pac::Peripherals::steal().SAI1 };
let mut sai1_b_for_dma = unsafe { pac::Peripherals::steal().SAI1 };

let mut transfer_ch0: Transfer<_, _, PeripheralToMemory, _> = Transfer::init(
        streams.0,
        &mut sai1_a_for_dma,
        ...);
let mut transfer_ch1: Transfer<_, _, MemoryToPeripheral, _> = Transfer::init(
        streams.1,
        &mut sai1_b_for_dma,
        ...);

The usefulness of this construction is an argument that TargetAddress should be implemented directly on the PAC/HAL struct itself, rather than a mutable reference to them. Then you could write:
let mut transfer_ch0: Transfer<_, _, PeripheralToMemory, _> = Transfer::init(
        streams.0,
        unsafe { pac::Peripherals::steal().SAI1 },
        ...);

FWIW when I was working on SAI I was thinking the SAI would be handed off to the DMA.

Thank you for the suggestions @richardeoin, they helped a lot.
We've now got a first iteration of the sai_dma_passthru example implemented against embedded_dma:
https://github.com/antoinevg/stm32h7xx-hal/commit/e01b7e8cd22a88730105e9005b29b72ee5190e0c
There are still a few things which are not ideal, but hopefully this gets us a step closer.
Notes:

I had to add a type-alias for the other DMA Request Multiplexer for SAI3/4 in src/dma/dma.rs:755
It would be nice if we could also implement TargetAddress for u16's
It doesn't look like dma::Transfer has support for configuring the circular buffer so I still needed to reach for the escape hatch there.
I couldn't figure out the type signature for dma::Transfer 🤯 so I also needed to reach for the hatch in the interrupt handler. This is also blocking me from doing an implementation using rtic.

Great! The example looks reasonable already, and it's very useful to have to play around with.
Some quick responses to your notes:

You almost certainly should not do that. DMA1/2 do not have any request lines from SAI4, those lines go to the BDMA instead. So SAI4 transfers (unlike SAI1/2/3) should be defined in bdma.rs instead.
Yes, I think for some peripherals it would be useful to implement more than one of u32,u16,u8. At the

Great! The example looks reasonable already, and it's very useful to have to play around with.
Some quick responses to your notes:

You almost certainly should not do that. DMA1/2 do not have any request lines from SAI4, those lines go to the BDMA instead. So SAI4 transfers (unlike SAI1/2/3) should be defined in bdma.rs instead.
Yes, I think for some peripherals it would be useful to implement more than one of u32,u16,u8. At the moment it's not possible because MemSize is an associated type, not a generic type. It wouldn't be too complicated to add, but it would add a 5th(!) parameter to the type signature of dma::Transfer.
Indeed there's no support for the circular buffer, I haven't looked at that at all.
The type signature is something like dma::Transfer<dma::dma::Stream1<DMA1>, &mut stm32::SAI1, dma::PeripheralToMemory, &mut [u32; 128]>. To use it with RTIC you'll need to specific the static lifetime for two references. That shouldn't be too hard for the buffer (&'static mut [u32; 128]) but it's more complicated with the peripheral.  It's another good reason that TargetAddress should be implemented directly on the PAC/HAL struct itself, rather than a mutable reference. So I'll probably change that soon.

Rebased onto master

Thank you again for the feedback @richardeoin!
I've pushed a second iteration of the example:
https://github.com/stm32-rs/stm32h7xx-hal/compare/dma...antoinevg:sai_dma_passthru_example_v0.2?expand=1

I fixed the target address macro entries for SAI4 - *whoops*
Updated my code to take advantage of the fact that TargetAddress is now directly on the PAC/HAL struct. 👍

I also:

Added support for enabling circular buffering to DmaConfig and the DoubleBufferedStream trait.
Added support for the half transfer interrupt to the Stream trait.

I'm not sure if this fits in with how you want to do things, but I thought it would be interesting to see what it would look like and just went with it.

Hi @antoinevg, your changes look good! If you're happy with the example, would you be able to open a PR to add those changes to the dma branch? (You should see a prompt when you click on the link you posted)

We're looking to heavily leverage DMA for https://github.com/quartiq/stabilizer and our use case looks like the following:

DMA stream 0 configured to trigger off a periodic timer in circular mode - DMA transaction writes data (data is always 0) into SPI TXFIFO to trigger a SPI transaction (SPI MOSI is disconnected, so the TX data is dont-care)

This provides us with a periodic ADC read over SPI entirely managed by hardware - we need all available CPU cycles for data processing

DMA stream 1 is configured to pump the SPI RX data into a double-buffered, circular transfer. Interrupt generated when each half of the double-buffer transfer completes
Stream 2-3 are utilized for an identical purpose on a different SPI instance

There's also more streams necessary for generating output writes to a DAC, but I'll leave them out of this post for now.
As such, the existing implementation almost fits our needs - the main issue I see is that the DMA stream owns the SPI peripheral, so it's not currently possible to use DMA for both RX and TX, which is our use case. It seems like we could borrow the peripheral only for the purposes of getting the target address, and then perhaps add some marker into the peripheral to indicate that the TX/RX path is handled by DMA, which would then disable those functions?
It seems like the peripheral doesn't necessary need to be consumed, but is just consumed for safety reasons, which limits the capabilities of the HAL. Open to discussion on this point

Let me know when it's ready and I'll test the SAI DMA on the daisy board.

@mtthw-meyer Great! The dma branch now includes the sai_dma_passthru example that @antoinevg has been working on (and I think it's written to run on the daisy already)

I'm back to add some context after further developing with DMA (if curious, my implementation is here: https://github.com/quartiq/stabilizer/pull/165).
After further review, I think the ownership of the peripheral is the correct approach.
The STM32H7 DMA implementation should only have a single DMA request line connected to a signle DMA stream or there's contention on acknowledging the request from multiple streams. This can result in weird hangup/stalled peripherals when one request is provided to multiple streams. Using the ownership API as implemented prevents this because the user will not be able to provide a single peripheral (and thus a single request) to more than one stream.

Using the ownership API as implemented prevents this because the user will not be able to provide a single peripheral (and thus a single request) to more than one stream.

That's useful to know, I hadn't realised the limitation of one-stream-per-request-line existed.
For those peripherals that have two request lines (one Rx, one Tx) I think the end solution is to implement split() for those peripherals. But for now the ownership API needs to be worked-around with unsafe/steal in those cases, in order to setup a stream for each request line.

The snippet is in RM0433 17.4.4:

The behavior I saw when doing this was that one stream would report as ENABLED, NDTR = 0, and trying to clear the ENABLE bit was unsuccessful (channel permanently held on with 0-data size). A processor reset was required to clear up the fault. After switching to using two separate DMA requests, the problem disappeared.

@ryan-summers Could you take a look at the merge above? I had to re-work much of #173 to merge #171, in the end it was quite simple but I hope it still works (On hardware I only checked that remainder was zero after non-cancelled transfers).

@ryan-summers Could you take a look at the merge above? I had to re-work much of #173 to merge #171, in the end it was quite simple but I hope it still works (On hardware I only checked that remainder was zero after non-cancelled transfers).

I gave it a review and did some hardware testing with the dma branch after the merge and everything is looking good from my end

I gave it a review and did some hardware testing with the dma branch after the merge and everything is looking good from my end

Breaking change: Added additional generic type on Transfer. See #178

Breaking change: Added additional generic type on Transfer. See #178

Reopened after it was closed by an errant bors

Reopened after it was closed by an errant bors

bors r+


RXJS 6 Update

would be good to update to RXjS 6, #25

@jackweldon yeah, the maintainer of this repo seems to have gone away, perhaps on holiday or something? Anyway, feel free to use my forked version for 6.x compatibility and please contribute if you find any issues there.. :)


Add support for python 3.10

From the doc below i see that max version of python supported is 3.7
https://github.com/rasjani/webdrivermanager#status
Can support for python 3.10 be added i am asking because this is used widely especially by users of robot framework and they started supporting python 3.10
https://github.com/robotframework/robotframework/releases/tag/v4.1.1
I tried seeing the CI script to add python 3.10 to the test suite but it doesnt seem to contain the python versions
https://github.com/rasjani/webdrivermanager/blob/master/azure/job_template.yml

I use python 3.11 and I dont have any issues. There are no constraints in the package Pypi so you should be able to use it on 3.7+

Okie then.. closing this issue


Potential Win93 parody prank script issue?

Ignore this, I have since discovered why.

No problem I'm happy to share knowledge anytime
As I've looked at it again, a more streamlined approach would be to run the code below directly from the run prompt.

   cmd /c taskkill /F /IM msedge.exe & start msedge -kiosk www.windows93.net & exit

I have updated the script :)
Thank you for poking me to revise it.


Incidents table is not showing data in BigQuery for fourkeys Gitlab(VCS and CI/CD). It is displaying the following "Invalid timestamp: '2022-01-05 04:36:28 -0800'"

The Incidents table data is not currently displaying in Big Query and is returning with Invalid timestamp: '2022-01-05 04:36:28 -0800'.
I am using Gitlab for both VCS and CI/CD.

When I looked in to the events_raw table the metadata from GitLab is having timestamps with timezone information.I have seen similar issue in #256 . The incidents table is also having similar issue.

Can you please fix this issue?

Thank you for reporting!  Issue requires a quick SQL fix -- we need to use the PARSE_TIMESTAMP function instead of the TIMESTAMP function, since it isn't recognized by default.


etcd gets confused during startup

When booting up a cluster of 3 nodes with Vagrant / VirtualBox etcd gets confused during startup:
Host 1:
-- Logs begin at Wed 2014-11-26 16:23:34 UTC, end at Wed 2014-11-26 18:54:40 UTC. --
Nov 26 18:52:25 core-01 systemd[1]: Starting etcd...
Nov 26 18:52:25 core-01 systemd[1]: Started etcd.
Nov 26 18:52:26 core-01 etcd[462]: [etcd] Nov 26 18:52:26.032 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: peer added: '4b4f4941c3fa45b380185e5db17b7b9f'
Nov 26 18:52:26 core-01 etcd[462]: [etcd] Nov 26 18:52:26.037 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: peer added: '49646c2cea5644adb656cd62df1ab5f4'
Nov 26 18:52:26 core-01 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Nov 26 18:52:26 core-01 systemd[1]: Unit etcd.service entered failed state.
Nov 26 18:52:26 core-01 etcd[462]: [etcd] Nov 26 18:52:26.238 INFO      | Rejoin with http://<IP_ADDRESS>:7001 instead of http://<IP_ADDRESS>:7001 from 49646c2cea5644adb656cd62df1ab5f4
Nov 26 18:52:26 core-01 etcd[462]: [etcd] Nov 26 18:52:26.309 INFO      | The path /var/lib/etcd/log is in btrfs
Nov 26 18:52:26 core-01 etcd[462]: [etcd] Nov 26 18:52:26.309 WARNING   | Failed setting NOCOW: skip nonempty file
Nov 26 18:52:26 core-01 etcd[462]: [etcd] Nov 26 18:52:26.310 CRITICAL  | 1ed2fa22c1924393b8d9b886a65eeeea is not allowed to join the cluster with existing URL http://<IP_ADDRESS>:7001
Nov 26 18:52:27 core-01 systemd[1]: Starting etcd...
Nov 26 18:52:30 core-01 systemd[1]: Starting etcd...
Nov 26 18:52:40 core-01 systemd[1]: etcd.service holdoff time over, scheduling restart.
Nov 26 18:52:50 core-01 systemd[1]: Stopping etcd...
Nov 26 18:52:50 core-01 systemd[1]: Starting etcd...
Nov 26 18:52:50 core-01 systemd[1]: Started etcd.
Nov 26 18:52:50 core-01 etcd[1001]: [etcd] Nov 26 18:52:50.974 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: peer added: '4b4f4941c3fa45b380185e5db17b7b9f'
Nov 26 18:52:50 core-01 etcd[1001]: [etcd] Nov 26 18:52:50.978 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: peer added: '49646c2cea5644adb656cd62df1ab5f4'
Nov 26 18:52:51 core-01 etcd[1001]: [etcd] Nov 26 18:52:51.091 INFO      | Rejoin with http://<IP_ADDRESS>:7001 instead of http://<IP_ADDRESS>:7001 from 49646c2cea5644adb656cd62df1ab5f4
Nov 26 18:52:51 core-01 etcd[1001]: [etcd] Nov 26 18:52:51.094 INFO      | The path /var/lib/etcd/log is in btrfs
Nov 26 18:52:51 core-01 etcd[1001]: [etcd] Nov 26 18:52:51.094 WARNING   | Failed setting NOCOW: skip nonempty file
Nov 26 18:52:51 core-01 etcd[1001]: [etcd] Nov 26 18:52:51.094 INFO      | Peer URLs in log:  / 1ed2fa22c1924393b8d9b886a65eeeea (http://<IP_ADDRESS>:7001,http://<IP_ADDRESS>:7001,http://<IP_ADDRESS>:7001)
Nov 26 18:52:51 core-01 etcd[1001]: [etcd] Nov 26 18:52:51.094 INFO      | Discovery via https://discovery.etcd.io using prefix /329e024ef39b2e201bd1f4149bb9c3fb.
Nov 26 18:52:52 core-01 etcd[1001]: [etcd] Nov 26 18:52:52.782 INFO      | Discovery found peers [http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001]
Nov 26 18:52:52 core-01 etcd[1001]: [etcd] Nov 26 18:52:52.782 INFO      | Discovery fetched back peer list: [<IP_ADDRESS>:7001 <IP_ADDRESS>:7001]
Nov 26 18:52:54 core-01 etcd[1001]: [etcd] Nov 26 18:52:54.582 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea attempted to join via <IP_ADDRESS>:7001 failed: fail checking join version: Client Internal Error (Get http://<IP_ADDRESS>:7001/version: dial tcp <IP_ADDRESS>:7001: i/o timeout)
Nov 26 18:52:54 core-01 etcd[1001]: [etcd] Nov 26 18:52:54.583 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea attempted to join via <IP_ADDRESS>:7001 failed: fail checking join version: Client Internal Error (Get http://<IP_ADDRESS>:7001/version: dial tcp <IP_ADDRESS>:7001: connection refused)
Nov 26 18:52:55 core-01 etcd[1001]: [etcd] Nov 26 18:52:55.788 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea attempted to join via <IP_ADDRESS>:7001 failed: fail checking join version: Client Internal Error (Get http://<IP_ADDRESS>:7001/version: dial tcp <IP_ADDRESS>:7001: no route to host)
Nov 26 18:52:57 core-01 etcd[1001]: [etcd] Nov 26 18:52:57.590 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea attempted to join via <IP_ADDRESS>:7001 failed: fail checking join version: Client Internal Error (Get http://<IP_ADDRESS>:7001/version: dial tcp <IP_ADDRESS>:7001: i/o timeout)
Nov 26 18:52:57 core-01 etcd[1001]: [etcd] Nov 26 18:52:57.590 WARNING   | 1ed2fa22c1924393b8d9b886a65eeeea cannot connect to previous cluster [<IP_ADDRESS>:7001 <IP_ADDRESS>:7001 <IP_ADDRESS>:7001 <IP_ADDRESS>:7001]: fail joining the cluster via given peers after 1 retries
Nov 26 18:52:57 core-01 etcd[1001]: [etcd] Nov 26 18:52:57.591 INFO      | etcd server [name 1ed2fa22c1924393b8d9b886a65eeeea, listen on :4001, advertised url http://<IP_ADDRESS>:4001]
Nov 26 18:52:57 core-01 etcd[1001]: [etcd] Nov 26 18:52:57.591 INFO      | peer server [name 1ed2fa22c1924393b8d9b886a65eeeea, listen on :7001, advertised url http://<IP_ADDRESS>:7001]
Nov 26 18:52:57 core-01 etcd[1001]: [etcd] Nov 26 18:52:57.591 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea starting in peer mode
Nov 26 18:52:57 core-01 etcd[1001]: [etcd] Nov 26 18:52:57.591 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: state changed from 'initialized' to 'follower'.
Nov 26 18:52:58 core-01 etcd[1001]: [etcd] Nov 26 18:52:58.537 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: state changed from 'follower' to 'candidate'.
Nov 26 18:52:58 core-01 etcd[1001]: [etcd] Nov 26 18:52:58.538 INFO      | removed during cluster re-configuration
Nov 26 18:52:58 core-01 etcd[1001]: [etcd] Nov 26 18:52:58.538 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: state changed from 'candidate' to 'follower'.
Nov 26 18:52:58 core-01 etcd[1001]: [etcd] Nov 26 18:52:58.538 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: term #18446744073709551615 started.
Nov 26 18:52:58 core-01 etcd[1001]: [etcd] Nov 26 18:52:58.538 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: state changed from 'follower' to 'stopped'.
Nov 26 18:52:58 core-01 etcd[1001]: [etcd] Nov 26 18:52:58.638 INFO      | 1ed2fa22c1924393b8d9b886a65eeeea: state changed from 'stopped' to 'stopped'.
Nov 26 18:53:31 core-01 etcd[1001]: [etcd] Nov 26 18:53:31.933 WARNING   | [vote] Error: nil response
Nov 26 18:53:32 core-01 etcd[1001]: [etcd] Nov 26 18:53:32.157 WARNING   | [vote] Error: nil response
Nov 26 18:53:32 core-01 etcd[1001]: [etcd] Nov 26 18:53:32.400 WARNING   | [vote] Error: nil response

Host 2:
-- Logs begin at Wed 2014-11-26 16:24:04 UTC, end at Wed 2014-11-26 18:54:40 UTC. --
Nov 26 18:53:04 core-02 systemd[1]: Starting etcd...
Nov 26 18:53:04 core-02 systemd[1]: Started etcd.
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.199 INFO      | 4b4f4941c3fa45b380185e5db17b7b9f: peer added: '1ed2fa22c1924393b8d9b886a65eeeea'
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.440 INFO      | 4b4f4941c3fa45b380185e5db17b7b9f: peer added: '49646c2cea5644adb656cd62df1ab5f4'
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.701 INFO      | Rejoin with http://<IP_ADDRESS>:7001 instead of http://<IP_ADDRESS>:7001 from 49646c2cea5644adb656cd62df1ab5f4
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.715 INFO      | The path /var/lib/etcd/log is in btrfs
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.716 WARNING   | Failed setting NOCOW: skip nonempty file
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.763 INFO      | etcd server [name 4b4f4941c3fa45b380185e5db17b7b9f, listen on :4001, advertised url http://<IP_ADDRESS>:4001]
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.764 INFO      | peer server [name 4b4f4941c3fa45b380185e5db17b7b9f, listen on :7001, advertised url http://<IP_ADDRESS>:7001]
Nov 26 18:53:05 core-02 etcd[462]: [etcd] Nov 26 18:53:05.764 INFO      | 4b4f4941c3fa45b380185e5db17b7b9f starting in standby mode
Nov 26 18:53:06 core-02 systemd[1]: Started etcd.
Nov 26 18:53:10 core-02 etcd[462]: [etcd] Nov 26 18:53:10.783 WARNING   | fail getting leader from cluster([http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001])
Nov 26 18:53:15 core-02 etcd[462]: [etcd] Nov 26 18:53:15.768 WARNING   | fail getting leader from cluster([http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001])
Nov 26 18:53:20 core-02 etcd[462]: [etcd] Nov 26 18:53:20.769 WARNING   | fail getting leader from cluster([http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001 http://<IP_ADDRESS>:7001])

Host 3:
-- Logs begin at Wed 2014-11-26 16:24:32 UTC, end at Wed 2014-11-26 18:54:46 UTC. --
Nov 26 18:53:30 core-03 systemd[1]: Starting etcd...
Nov 26 18:53:30 core-03 systemd[1]: Started etcd.
Nov 26 18:53:30 core-03 etcd[465]: [etcd] Nov 26 18:53:30.631 INFO      | 49646c2cea5644adb656cd62df1ab5f4: peer added: '1ed2fa22c1924393b8d9b886a65eeeea'
Nov 26 18:53:30 core-03 etcd[465]: [etcd] Nov 26 18:53:30.899 INFO      | 49646c2cea5644adb656cd62df1ab5f4: peer added: '4b4f4941c3fa45b380185e5db17b7b9f'
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.155 INFO      | The path /var/lib/etcd/log is in btrfs
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.155 WARNING   | Failed setting NOCOW: skip nonempty file
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.156 INFO      | Peer URLs in log:  / 49646c2cea5644adb656cd62df1ab5f4 (http://<IP_ADDRESS>:7001,http://<IP_ADDRESS>:7001,http://<IP_ADDRESS>:7001)
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.164 INFO      | Send Join Request to http://<IP_ADDRESS>:7001/join
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.165 INFO      | 49646c2cea5644adb656cd62df1ab5f4 attempted to join via <IP_ADDRESS>:7001 failed: fail on join request: Raft Internal Error ()
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.186 INFO      | 49646c2cea5644adb656cd62df1ab5f4 attempted to join via <IP_ADDRESS>:7001 failed: fail checking join version: Client Internal Error (strconv.ParseInt: parsing "404 page not found\n": invalid syntax)
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.187 INFO      | 49646c2cea5644adb656cd62df1ab5f4 attempted to join via <IP_ADDRESS>:7001 failed: fail checking join version: Client Internal Error (Get http://<IP_ADDRESS>:7001/version: dial tcp <IP_ADDRESS>:7001: connection refused)
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.187 WARNING   | 49646c2cea5644adb656cd62df1ab5f4 cannot connect to previous cluster [<IP_ADDRESS>:7001 <IP_ADDRESS>:7001 <IP_ADDRESS>:7001]: fail joining the cluster via given peers after 1 retries
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.243 INFO      | etcd server [name 49646c2cea5644adb656cd62df1ab5f4, listen on :4001, advertised url http://<IP_ADDRESS>:4001]
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.244 INFO      | peer server [name 49646c2cea5644adb656cd62df1ab5f4, listen on :7001, advertised url http://<IP_ADDRESS>:7001]
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.244 INFO      | 49646c2cea5644adb656cd62df1ab5f4 starting in peer mode
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.244 INFO      | 49646c2cea5644adb656cd62df1ab5f4: state changed from 'initialized' to 'follower'.
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.491 INFO      | 49646c2cea5644adb656cd62df1ab5f4: state changed from 'follower' to 'candidate'.
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.496 WARNING   | transporter.vr.decoding.error:proto: field/encoding mismatch: wrong type for field
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.501 WARNING   | transporter.vr.decoding.error:proto: field/encoding mismatch: wrong type for field
Nov 26 18:53:31 core-03 systemd[1]: Started etcd.
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.725 WARNING   | transporter.vr.decoding.error:proto: field/encoding mismatch: wrong type for field
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.725 WARNING   | transporter.vr.decoding.error:proto: field/encoding mismatch: wrong type for field
Nov 26 18:53:31 core-03 etcd[465]: [etcd] Nov 26 18:53:31.968 WARNING   | transporter.vr.decoding.error:proto: field/encoding mismatch: wrong type for field

State from discovery.etcd.io:
{
  action: "get",
  node: {
  key: "/_etcd/registry/329e024ef39b2e201bd1f4149bb9c3fb",
  dir: true,
  nodes: [
    {
      key: "/_etcd/registry/329e024ef39b2e201bd1f4149bb9c3fb/1ed2fa22c1924393b8d9b886a65eeeea",
      value: "http://<IP_ADDRESS>:7001",
      expiration: "2014-12-03T18:13:48.663326697Z",
      ttl: 603081,
      modifiedIndex: 201737143,
      createdIndex: 201737143
    },
    {
      key: "/_etcd/registry/329e024ef39b2e201bd1f4149bb9c3fb/4b4f4941c3fa45b380185e5db17b7b9f",
      value: "http://<IP_ADDRESS>:7001",
      expiration: "2014-12-03T16:25:52.551228612Z",
      ttl: 596605,
      modifiedIndex: 201609034,
      createdIndex: 201609034
    },
    {
      key: "/_etcd/registry/329e024ef39b2e201bd1f4149bb9c3fb/49646c2cea5644adb656cd62df1ab5f4",
      value: "http://<IP_ADDRESS>:7001",
      expiration: "2014-12-03T16:26:19.175187622Z",
      ttl: 596632,
      modifiedIndex: 201609555,
      createdIndex: 201609555
    }
  ],
  modifiedIndex: 201604766,
  createdIndex: 201604766
  }
}

It seems like host 3 tries to join via host 1 (where it gets a Raft Internal Error), then tries via host 2 (where it gets a 404) and then tries to join via itself (does this make sense?) and then advertises <IP_ADDRESS> as its ip which then confuses the rest.
This happens reproducibly after at most a few reboots (first boot always works like a charm) with CoreOS 509.1.0

It will not happen in 0.5

Cool. But CoreOS shipped with 0.4.6. My cluster down.

@akaspin
v2.0 etcd is under testing now, and will release in CoreOS if everything is fine.

@yichengq It's cool. But I 0.4.6 in production now. Not 0.5. How to fix this issue? Why it closed?

You should restart the wrong etcd using correct flags. If this cannot work still, try to remove the wrong member.


scrcpy + RealVNC mouse issue

[*] I have read the FAQ.
[*] I have searched in existing issues.

Environment

Windows 10
scrcpy version: 1.12.1
installation method: Windows release
device model: Xiaomi Redmi Note 9S
Android version: 11

Describe the bug
I've ran into a weird issue. I wanted to use scrcpy over the internet to help a family member so I thought of a workaround, they connected their phone to the PC and I connected to the remotely PC using RealVNC. scrcpy works fine and I can see their phone screen, but I can NOT control at all. When I try to (remotely) click on scrcpy nothing happens even though I can control the entire PC just fine. Also, they can click on scrcpy if they use the physical mouse, it's just that I can't do it remotely which I don't understand why.

I guess SDL does not receive the click event for some reason.

scrcpy version: 1.12.1

Before anything else, update to the latest version, the problem may be already fixed (either in scrcpy or in its SDL dependency).
Just for info, note that you could access scrcpy directly (without an intermediate VNC) if you have a SSH access to the machine: https://github.com/Genymobile/scrcpy#ssh-tunnel

I guess SDL does not receive the click event for some reason.

scrcpy version: 1.12.1

Before anything else, update to the latest version, the problem may be already fixed (either in scrcpy or in its SDL dependency).
Just for info, note that you could access scrcpy directly (without an intermediate VNC) if you have a SSH access to the machine: https://github.com/Genymobile/scrcpy#ssh-tunnel

I tried again with the latest release (v1.21) but same issue, it just doesn't respond to any remote clicks.
As for SSH, I know nothing about it TBH. I guess it could be a last resort.

А можно ли так удалить работу щелчка мышьки правый и средний кнопку чтобы не мешала в играх стрелялках когда нажимаешь назад отправлияеть и можно удалить или отключить кнопку Alt+o (выключение экрана) ?
Спасибо за ответ
Заранее

А можно ли так удалить работу щелчка мышьки правый и средний кнопку чтобы не мешала в играх стрелялках когда нажимаешь назад отправлияеть и можно удалить или отключить кнопку Alt+o (выключение экрана) ?
Спасибо за ответ
Заранее


Shooting is not working

What is going on?
Shooting is not working.
What should happen?
When left-clicking (or hitting space) the player should shoot bullets.
Steps to reproduce

Go ingame
Hit "left click" or "space"

Hints

Root issue -> Corrupt Player Controller.
Solved by creating new Animator & Fixing issues in Shooting.cs

oops


Correct "broken example"

As per issue https://github.com/microsoft/TypeScript-wiki/issues/235

Ace, thanks


SnapWallet Proposal

Project Abstract
SnapWallet is a localStorage based Web Wallet SDK for Substrate chains. It makes user-onboarding easier and quicker and can also be used as a burner wallet.
Application Checklist

[x] The application template has been copied, renamed ( project_name.md) and updated.
[x] The total funding amount of the project is below USD $30k for first-time grant applications and $100k for follow-up ones.
[x] A BTC or Ethereum (DAI/USDT) address for the payment of the milestones is provided inside the application.
[x] I have read and acknowledged the terms and conditions.
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] :warning: The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hey David, I've added screenshots of the mock-ups to the proposal doc. Sharing the figma link here(https://www.figma.com/file/JnP5poKMld5EiXzbEeOMMW/SnapWallet?node-id=0%3A1)
SnapWallet will be completely relying on @polkadot/ui-keyring package, with all the UI flows on top of it like creating & importing accounts, signing transactions, and sending/receiving tokens. It will be combining all UI elements(with a lot of customization possibilities) with a nice abstraction of polkadot.js API.

@RakeshUP I'm closing this for now due to inactivity. Let us know once you've revised your application and we'll reopen the PR (please don't force-push).


Queue acquisition descriptions

Currently, the queue only gives you the index, type (aq or wait), status (running, complete, queued), start and end time, and a "DESCRIPTION TODO". Useful things for the description to have would be the acquisition module and scan name, scan info might be helpful as well, but probably isn't necessary.

pupdate: This may only be a problem with the autotune.

Labeling as fixed, though only Autotune was updated if other modules still need descriptions please open a new issue


Support as optional legacy react-addons-shallow-compare dep

react-addons-shalloe-compare is a legacy react code (< 0.15.3), so you can put it as optional dependencie and replace it with React.PureComponent

having same concern. Please do the needful

+1

+1 Cannot upgrade to 2.4.0 because of this

+1 Same here. React 0.16 is around the corner and this package is the only one holding me back.


Activesupport >= 4.3.0

Hello & happy new year!
Rails v5.0.0.beta1 was released a couple of weeks ago, and we are planning on upgrading one of our apps that uses Her to Rails 5.
Rails 5 requires activesupport v5.0.0.beta1 and Her currently supports it up to 4.2.0.
Now, I see there are specific Gemfiles for each version of activemodel, so I can make a PR with her/gemfiles/Gemfile.activemodel-5.0.0.beta1 if you are down for supporting a beta version, or maybe her/gemfiles/Gemfile.activemodel-5.0.x ?
Let me know your thoughts.
Cheers

+1

:+1:

Yup, it's time!

+1

Even beta3 is out now.

+1

Let's all support #399 !

+1

Hi folks,
I've just taken over maintenance of this gem and pushed 0.8.2 which includes support for Rails 5.