Hash new types as specified in #284

Progress report:

List of the new hash types was discussed and new types were defined: https://github.com/rust-bitcoin/rust-bitcoin/issues/284#issuecomment-524618641
bitcoin_hashes new_hashtype macro is fixed to work correctly and minimize amount of code changes required for adopting the new hash types: https://github.com/rust-bitcoin/bitcoin_hashes/pull/63 (it depends on the other PR here: https://github.com/rust-bitcoin/bitcoin_hashes/pull/64)
All new types are defined: https://github.com/rust-bitcoin/rust-bitcoin/pull/349/commits/e9c4d7d9070f317bb0de51a21bf3ce9680ae6051#diff-107236903cdf97a062c0be08467a2769
Txid is adopted, the library is compiling and tests are running correctly: https://github.com/rust-bitcoin/rust-bitcoin/pull/349/commits/e9c4d7d9070f317bb0de51a21bf3ce9680ae6051
All proposed hashes are implemented

@apoelstra The PR is ready for a review; the build with rust 1.22 fails since rust compiler fails to recognise that FromHex use is actually used by the macro, and allow_unused does not work for this version (but works fine for the rest). I can't do anything with this false error...

Maybe make #![deny(unused_imports)] conditional on a feature , and turn that feature on travis builds of non-1.22.0 builds?

I think we should just put allow(unused_imports) on hash_types.rs
Alternately, we could remove the deny line entirely; these are just a random collection of rust-0.6-era style lints which are handled today by better error messages and better coding conventions. But I don't like the warnings, even if they're only on 1.22.

Everything works for now with allow(unused_imports) moved on hash_types mod in lib.rs. Also merged recent changes from the master.

Codecov Report

Merging #349 into master will increase coverage by 0.02%.
The diff coverage is 86.41%.

@@            Coverage Diff             @@
##           master     #349      +/-   ##
==========================================
+ Coverage   82.35%   82.38%   +0.02%
==========================================
  Files          39       40       +1
  Lines        7901     9025    +1124
==========================================
+ Hits         6507     7435     +928
- Misses       1394     1590     +196

Impacted Files
Coverage Δ

src/lib.rs
100% <ø> (ø)
:arrow_up:

src/consensus/encode.rs
79.9% <ø> (-2.2%)
:arrow_down:

src/internal_macros.rs
59.47% <ø> (ø)
:arrow_up:

src/network/message_filter.rs
0% <0%> (ø)
:arrow_up:

src/network/message_blockdata.rs
36.17% <0%> (ø)
:arrow_up:

src/util/contracthash.rs
79.78% <100%> (+1.28%)
:arrow_up:

src/hash_types.rs
100% <100%> (ø)

src/util/address.rs
85.58% <100%> (+0.09%)
:arrow_up:

src/util/bip158.rs
92.76% <100%> (-0.39%)
:arrow_down:

src/util/bip143.rs
100% <100%> (ø)
:arrow_up:

... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d20936...a48bc65. Read the comment docs.

Could you please clean up the history? Squash down fixup commits, remove merge commits, etc?

I have tried to do my best in structuring my changes into logical structure of sequential commits plus the last two for rebasing master and resolving conflicts with the master

Can you rebase instead of merging?

I can, but when I did it it said about conflicts (even after rebasing), so I had to merge. Now I see that there are new conflicts...

Right, if there are conflicts you need to fix the conflicts

If you prefer I don't mind helping with rebasing/fixing conflicts.
(all you'll need to do is tick the allow edits from maintainers)

Sorry for replying late, being out for our first client-side validation event we were doing in Milan. @elichai, thank you for your proposal, I will address @stevenroose review today with a new commit and will allow edits for maintainers - I am completelly missed with all this rebase vs merge + conflicts...

Travis is completely broken: it fails to build in 50-75% of times at the stage of apt install etc, i.e. before the actual compilation. I have tried more than 10 times, and still can't get through...

I made some optimizations to the merkle root code. If you want I can add the commit to this branch.
It makes the merkle root calculation generic. It also removed MerkleRooted. And it fixes the bitcoin_hashes minimum version that you forgot. The code could be a little bit more readable with as_hash() on the newtypes (https://github.com/rust-bitcoin/bitcoin_hashes/pull/70), but it works like this.
This is the diff (you can also just git apply the diff):
diff --git a/Cargo.toml b/Cargo.toml
index 3b9ae51..9530c4b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -22,7 +22,7 @@ use-serde = ["hex", "serde", "bitcoin_hashes/serde", "secp256k1/serde"]

 [dependencies]
 bech32 = "0.7.1"
-bitcoin_hashes = "0.7"
+bitcoin_hashes = "0.7.2"
 bitcoinconsensus = { version = "0.17", optional = true }
 serde = { version = "1", optional = true }
 hex = { version = "=0.3.2", optional = true }
diff --git a/src/blockdata/block.rs b/src/blockdata/block.rs
index 044494c..b637e99 100644
--- a/src/blockdata/block.rs
+++ b/src/blockdata/block.rs
@@ -22,9 +22,9 @@

 use util;
 use util::Error::{BlockBadTarget, BlockBadProofOfWork};
-use util::hash::{BitcoinHash, MerkleRooted, bitcoin_merkle_root};
+use util::hash::{BitcoinHash, bitcoin_merkle_root};
 use hashes::{Hash, sha256d, HashEngine};
-use hash_types::{Txid, Wtxid, BlockHash, TxMerkleRoot, WitnessMerkleRoot, WitnessCommitment};
+use hash_types::{Wtxid, BlockHash, TxMerkleRoot, WitnessMerkleRoot, WitnessCommitment};
 use util::uint::Uint256;
 use consensus::encode::Encodable;
 use network::constants::Network;
@@ -93,6 +93,12 @@ impl Block {
         false
     }

+    /// Calculate the transaction merkle root.
+    pub fn merkle_root(&self) -> TxMerkleRoot {
+        let hashes: Vec<sha256d::Hash> = self.txdata.iter().map(|obj| obj.txid().into()).collect();
+        bitcoin_merkle_root(hashes).into()
+    }
+
     /// compute witness commitment for the transaction list
     pub fn compute_witness_commitment (witness_root: &WitnessMerkleRoot, witness_reserved_value: &[u8]) -> WitnessCommitment {
         let mut encoder = WitnessCommitment::engine();
@@ -103,16 +109,10 @@ impl Block {

     /// Merkle root of transactions hashed for witness
     pub fn witness_root(&self) -> WitnessMerkleRoot {
-        let mut txhashes = vec!(Wtxid::default());
-        txhashes.extend(self.txdata.iter().skip(1).map(|t|t.wtxid()));
-        let hash_value: sha256d::Hash = bitcoin_merkle_root(txhashes).into();
-        hash_value.into()
-    }
-}
-
-impl MerkleRooted for Block {
-    fn merkle_root(&self) -> TxMerkleRoot {
-        bitcoin_merkle_root::<Txid>(self.txdata.iter().map(|obj| obj.txid().into()).collect())
+        let mut txhashes: Vec<sha256d::Hash> = Vec::with_capacity(self.txdata.len());
+        txhashes.push(Wtxid::default().into());
+        txhashes.extend(self.txdata.iter().skip(1).map(|t| sha256d::Hash::from(t.wtxid())));
+        bitcoin_merkle_root(txhashes).into()
     }
 }

@@ -212,7 +212,6 @@ mod tests {

     use blockdata::block::{Block, BlockHeader};
     use consensus::encode::{deserialize, serialize};
-    use util::hash::MerkleRooted;

     #[test]
     fn block_test() {
diff --git a/src/hash_types.rs b/src/hash_types.rs
index 6b0a635..56305a7 100644
--- a/src/hash_types.rs
+++ b/src/hash_types.rs
@@ -67,9 +67,3 @@ impl_hashencode!(TxMerkleRoot);
 impl_hashencode!(TxMerkleBranch);
 impl_hashencode!(WitnessMerkleRoot);
 impl_hashencode!(FilterHash);
-
-/// Generic trait for functions which may either take a Txid type of Wtxid type as an imput.
-/// For instance, used for Merklization procedure
-pub trait TxidType: Hash<Inner = [u8; 32]> + Encodable + Decodable { }
-impl TxidType for Txid { }
-impl TxidType for Wtxid { }
diff --git a/src/util/hash.rs b/src/util/hash.rs
index 1aac8fb..86759fa 100644
--- a/src/util/hash.rs
+++ b/src/util/hash.rs
@@ -16,37 +16,32 @@
 //! Utility functions related to hashing data, including merkleization

 use std::cmp::min;
-use std::default::Default;
+use std::io;

 use hashes::Hash;
-use hash_types::{Txid, TxidType, TxMerkleRoot};
-
-
-/// Any collection of objects for which a merkle root makes sense to calculate
-pub trait MerkleRooted {
-    /// Construct a merkle tree from a collection, with elements ordered as
-    /// they were in the original collection, and return the merkle root.
-    fn merkle_root(&self) -> TxMerkleRoot;
-}
+use consensus::encode::Encodable;

 /// Calculates the merkle root of a list of txids hashes directly
-pub fn bitcoin_merkle_root<T: TxidType>(data: Vec<T>) -> TxMerkleRoot {
+pub fn bitcoin_merkle_root<T>(data: Vec<T>) -> T
+    where T: Hash + Encodable,
+          <T as Hash>::Engine: io::Write,
+{
     // Base case
     if data.len() < 1 {
         return Default::default();
     }
     if data.len() < 2 {
-        return TxMerkleRoot::from_inner(data[0].into_inner());
+        return T::from_inner(data[0].into_inner());
     }
     // Recursion
     let mut next = vec![];
     for idx in 0..((data.len() + 1) / 2) {
         let idx1 = 2 * idx;
         let idx2 = min(idx1 + 1, data.len() - 1);
-        let mut encoder = TxMerkleRoot::engine();
+        let mut encoder = T::engine();
         data[idx1].consensus_encode(&mut encoder).unwrap();
         data[idx2].consensus_encode(&mut encoder).unwrap();
-        next.push(Txid::from_engine(encoder));
+        next.push(T::from_engine(encoder));
     }
     bitcoin_merkle_root(next)
 }
diff --git a/src/util/merkleblock.rs b/src/util/merkleblock.rs
index 974a962..1532402 100644
--- a/src/util/merkleblock.rs
+++ b/src/util/merkleblock.rs
@@ -496,7 +496,7 @@ impl Decodable for MerkleBlock {
 mod tests {
     use std::cmp::min;

-    use hashes::Hash;
+    use hashes::{Hash, sha256d};
     use hashes::hex::{FromHex, ToHex};
     use hash_types::{Txid, TxMerkleRoot, TxMerkleBranch};
     use secp256k1::rand::prelude::*;
@@ -518,7 +518,8 @@ mod tests {
                 .collect::<Vec<_>>();

             // Calculate the merkle root and height
-            let merkle_root_1 = bitcoin_merkle_root(txids.clone());
+            let hashes = txids.iter().map(|t| sha256d::Hash::from(*t)).collect();
+            let merkle_root_1: TxMerkleRoot = bitcoin_merkle_root(hashes).into();
             let mut height = 1;
             let mut ntx = num_tx;
             while ntx > 1 {

I made some quite good optimizations that reduces the number of allocations to a single allocation of half the size of the current first allocation:
diff --git a/src/blockdata/block.rs b/src/blockdata/block.rs
index b637e99..7a71515 100644
--- a/src/blockdata/block.rs
+++ b/src/blockdata/block.rs
@@ -95,7 +95,7 @@ impl Block {

     /// Calculate the transaction merkle root.
     pub fn merkle_root(&self) -> TxMerkleRoot {
-        let hashes: Vec<sha256d::Hash> = self.txdata.iter().map(|obj| obj.txid().into()).collect();
+        let hashes = self.txdata.iter().map(|obj| sha256d::Hash::from(obj.txid()));
         bitcoin_merkle_root(hashes).into()
     }

@@ -109,10 +109,15 @@ impl Block {

     /// Merkle root of transactions hashed for witness
     pub fn witness_root(&self) -> WitnessMerkleRoot {
-        let mut txhashes: Vec<sha256d::Hash> = Vec::with_capacity(self.txdata.len());
-        txhashes.push(Wtxid::default().into());
-        txhashes.extend(self.txdata.iter().skip(1).map(|t| sha256d::Hash::from(t.wtxid())));
-        bitcoin_merkle_root(txhashes).into()
+        let hashes = self.txdata.iter().enumerate().map(|(i, t)| sha256d::Hash::from(
+            if i == 0 {
+                // Replace the first hash with zeroes.
+                Wtxid::default()
+            } else {
+                t.wtxid()
+            }
+        ));
+        bitcoin_merkle_root(hashes).into()
     }
 }

diff --git a/src/util/hash.rs b/src/util/hash.rs
index 86759fa..0de4731 100644
--- a/src/util/hash.rs
+++ b/src/util/hash.rs
@@ -21,8 +21,11 @@ use std::io;
 use hashes::Hash;
 use consensus::encode::Encodable;

-/// Calculates the merkle root of a list of txids hashes directly
-pub fn bitcoin_merkle_root<T>(data: Vec<T>) -> T
+/// Calculates the merkle root of a list of hashes inline
+/// into the allocated slice.
+///
+/// In most cases, you'll want to use [bitcoin_merkle_root] instead.
+pub fn bitcoin_merkle_root_inline<T>(data: &mut [T]) -> T
     where T: Hash + Encodable,
           <T as Hash>::Engine: io::Write,
 {
@@ -34,16 +37,43 @@ pub fn bitcoin_merkle_root<T>(data: Vec<T>) -> T
         return T::from_inner(data[0].into_inner());
     }
     // Recursion
-    let mut next = vec![];
     for idx in 0..((data.len() + 1) / 2) {
         let idx1 = 2 * idx;
         let idx2 = min(idx1 + 1, data.len() - 1);
         let mut encoder = T::engine();
         data[idx1].consensus_encode(&mut encoder).unwrap();
         data[idx2].consensus_encode(&mut encoder).unwrap();
-        next.push(T::from_engine(encoder));
+        data[idx] = T::from_engine(encoder);
     }
-    bitcoin_merkle_root(next)
+    let half_len = data.len() / 2 + data.len() % 2;
+    bitcoin_merkle_root_inline(&mut data[0..half_len])
+}
+
+/// Calculates the merkle root of an iterator of hashes.
+pub fn bitcoin_merkle_root<T, I>(mut iter: I) -> T
+    where T: Hash + Encodable,
+          <T as Hash>::Engine: io::Write,
+          I: ExactSizeIterator<Item = T>,
+{
+    // Base case
+    if iter.len() == 0 {
+        return Default::default();
+    }
+    if iter.len() == 1 {
+        return T::from_inner(iter.nth(0).unwrap().into_inner());
+    }
+    // Recursion
+    let half_len = iter.len() / 2 + iter.len() % 2;
+    let mut alloc = Vec::with_capacity(half_len);
+    while let Some(hash1) = iter.next() {
+        // If the size is odd, use the last element twice.
+        let hash2 = iter.next().unwrap_or(hash1);
+        let mut encoder = T::engine();
+        hash1.consensus_encode(&mut encoder).unwrap();
+        hash2.consensus_encode(&mut encoder).unwrap();
+        alloc.push(T::from_engine(encoder));
+    }
+    bitcoin_merkle_root_inline(&mut alloc)
 }

 /// Objects which are referred to by hash
diff --git a/src/util/merkleblock.rs b/src/util/merkleblock.rs
index 1532402..7a3308c 100644
--- a/src/util/merkleblock.rs
+++ b/src/util/merkleblock.rs
@@ -518,7 +518,7 @@ mod tests {
                 .collect::<Vec<_>>();

             // Calculate the merkle root and height
-            let hashes = txids.iter().map(|t| sha256d::Hash::from(*t)).collect();
+            let hashes = txids.iter().map(|t| sha256d::Hash::from(*t));
             let merkle_root_1: TxMerkleRoot = bitcoin_merkle_root(hashes).into();
             let mut height = 1;
             let mut ntx = num_tx;

Could you apply one more diff?
Just git apply and then paste the diff and ctrl + D and then git commit --amend.
diff --git a/src/blockdata/block.rs b/src/blockdata/block.rs
index 7a71515..83ef624 100644
--- a/src/blockdata/block.rs
+++ b/src/blockdata/block.rs
@@ -23,7 +23,7 @@
 use util;
 use util::Error::{BlockBadTarget, BlockBadProofOfWork};
 use util::hash::{BitcoinHash, bitcoin_merkle_root};
-use hashes::{Hash, sha256d, HashEngine};
+use hashes::{Hash, HashEngine};
 use hash_types::{Wtxid, BlockHash, TxMerkleRoot, WitnessMerkleRoot, WitnessCommitment};
 use util::uint::Uint256;
 use consensus::encode::Encodable;
@@ -95,7 +95,7 @@ impl Block {

     /// Calculate the transaction merkle root.
     pub fn merkle_root(&self) -> TxMerkleRoot {
-        let hashes = self.txdata.iter().map(|obj| sha256d::Hash::from(obj.txid()));
+        let hashes = self.txdata.iter().map(|obj| obj.txid().as_hash());
         bitcoin_merkle_root(hashes).into()
     }

@@ -109,14 +109,14 @@ impl Block {

     /// Merkle root of transactions hashed for witness
     pub fn witness_root(&self) -> WitnessMerkleRoot {
-        let hashes = self.txdata.iter().enumerate().map(|(i, t)| sha256d::Hash::from(
+        let hashes = self.txdata.iter().enumerate().map(|(i, t)|
             if i == 0 {
                 // Replace the first hash with zeroes.
-                Wtxid::default()
+                Wtxid::default().as_hash()
             } else {
-                t.wtxid()
+                t.wtxid().as_hash()
             }
-        ));
+        );
         bitcoin_merkle_root(hashes).into()
     }
 }
diff --git a/src/util/merkleblock.rs b/src/util/merkleblock.rs
index 7a3308c..cb81203 100644
--- a/src/util/merkleblock.rs
+++ b/src/util/merkleblock.rs
@@ -496,7 +496,7 @@ impl Decodable for MerkleBlock {
 mod tests {
     use std::cmp::min;

-    use hashes::{Hash, sha256d};
+    use hashes::Hash;
     use hashes::hex::{FromHex, ToHex};
     use hash_types::{Txid, TxMerkleRoot, TxMerkleBranch};
     use secp256k1::rand::prelude::*;
@@ -518,7 +518,7 @@ mod tests {
                 .collect::<Vec<_>>();

             // Calculate the merkle root and height
-            let hashes = txids.iter().map(|t| sha256d::Hash::from(*t));
+            let hashes = txids.iter().map(|t| t.as_hash());
             let merkle_root_1: TxMerkleRoot = bitcoin_merkle_root(hashes).into();
             let mut height = 1;
             let mut ntx = num_tx;

I think the TxMerkleBranch type doesn't really make sense as a wrapper around a single hash. If this is supposed to represent a proof, it should be multiple hashes ... if it's supposed to represent an internal node of a Merkle tree, I think it makes more sense to reuse TxMerkleRoot for this purpose.

I was discussing typing and Merkle Trees with @petertodd a while ago, he had some strong arguments for having a type for each branch. The point is that "Type" is something that means the same in-memory serialization AND the same semantics. You should not be able to perform some operation on two branches placed at the different levels, and this must be prohibited at the level of the code itself. So may be it even worth doing MerkleBranch<LEVEL: const u16> and MerkleRoot<NO_ELEMENTS: const u64> types

I was discussing typing and Merkle Trees with @petertodd a while ago, he had some strong arguments for having a type for each branch. The point is that "Type" is something that means the same in-memory serialization AND the same semantics. You should not be able to perform some operation on two branches placed at the different levels, and this must be prohibited at the level of the code itself. So may be it even worth doing MerkleBranch<LEVEL: const u16> and MerkleRoot<NO_ELEMENTS: const u64> types

Though remember that you're using const generics in your example, which are still unstable. Conversely trying to do this with type-system trickery on stable Rust leads to at best bad code generation, and at worse can even crash the compiler (I've tried!).
What I've experimented with in Proofmarshal is using unsized types, with a dummy variable at the end of a struct containing a [()] to basically reuse the slice length as a tree depth/size parameter. But I haven't yet actually done a full implementation of that idea, so it's not clear how feasible it is to implement. It does require a decent amount of unsafe boilerplate to coerce pointers (this will be improved when unsized locals lands, specifically allocating dynamically sized arrays).

Though remember that you're using const generics in your example, which are still unstable. Conversely trying to do this with type-system trickery on stable Rust leads to at best bad code generation, and at worse can even crash the compiler (I've tried!).

You were able to crash the compiler on stable rust?

You were able to crash the compiler on stable rust?

Yes. It's not hard at all: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=ab27fc5442da5bb8f8d76970bcbdcab7
The Rust compiler just isn't designed for extremely deeply nested types.

I don't understand how the semantics of an internal node differ from that of the root, or why one should be called a "branch".

Yes. It's not hard at all: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=ab27fc5442da5bb8f8d76970bcbdcab7

Oh ok you meant panic not crash(ICE) doesn't matter.

I don't understand how the semantics of an internal node differ from that of the root, or why one should be called a "branch".

I agree, leafs and nodes should be separate but node==root for all I(and everything else involved?) care

I also don't understand how leaves are different from any other node.

I also don't understand how leaves are different from any other node.

For hashing purposes to differentiate what's a hash of data and what's a hash of other nodes.

If you have a single-node tree which type would be used?

On December 19, 2019 11:27:38 AM EST, Elichai Turkel<EMAIL_ADDRESS>wrote:

Yes. It's not hard at all:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=ab27fc5442da5bb8f8d76970bcbdcab7

Oh ok you meant panic not crash(ICE) doesn't matter.

Well, I've found a number of ICE's too. :) They're not that uncommon, even on stable.
I've never found a case where Rust miscompiled something. That's the scary case, which I have run into before with other compilers back when I was doing microprocessor stuff.

If you have a single-node tree which type would be used?

You probably will need to hash it twice? (once to get the node hash and another to get the root hash)
Isn't not differentiating between nodes and leafs what leads to things like: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016697.html

On December 19, 2019 12:40:18 PM EST, Elichai Turkel<EMAIL_ADDRESS>wrote:

If you have a single-node tree which type would be used?

You probably will need to hash it twice? (once to get the node hash and
another to get the root hash)
Isn't not differentiating between nodes and leafs what leads to things
like:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016697.html

There's other ways to achieve that. For example, commit to the total size/depth of the tree at the very top, and treat that commitment as type metadata. IMO that's a better way to do it, as it does less hashing and doesn't duplicate the information, so less stuff to verify.

If you have a single-node tree which type would be used?

You probably will need to hash it twice? (once to get the node hash and another to get the root hash)
Isn't not differentiating between nodes and leafs what leads to things like: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016697.html

That's about differentiating between nodes and data. Of course transactions should have a different type than Merkle tree nodes, and of course they should be hashed in a way that they cannot collide (except that in this case, we have to match Bitcoin's consensus, so we do have to allow for collisions...thanks Satoshi).
My point is that all nodes in a Merkle tree are treated identically, regardless of whether they are roots or leaves or internal.

Heh. Trying to express the depth-level of each hash with const generics reminds me of the well-known solving N-Queens problem in Haskell's type system story.
One can express all sorts of properties with the type systems, and argue about the semantics, but ultimately the question is ... is the cost (complexity) worth the benefit (preventing mistakes)? I'm guessing in C++ people just use one hash type for everything, and Bitcoin network is still running, so I would side with not introducing an additional type here.

@apoelstra and others, here is my take on the Merkle tree-related types:
We need to differentiate tree root and branch because the branch represent incomplete/partial knowledge of/commitment to some state (like set of transactions) in terms of some specific and defined protocol, while the root represent a complete type. This is the semantic difference in these types I was talking about.
Of course the leaves, if they represent a hashed data, a 0-level branches.
While we can't rely on the unstable rust features like const generics, we can just have two Merkle tree-related types: for branches and root, and we need one set of them per the type of the tree (Txids and Wtxids, which is  done with generics), if we'd like to follow the same design logic by which we have introduced specific hash types for semantically-different (but not serialization-different) hashes.
Also, since Merkle tree represent a common algorithm like hashing, I'd like to propose to do a genericl Merkle tree implementation and move it somewhere to bitcoin_hashes, and not have it inside rust-bitcoin — or even a complete independent crate; here we may join our efforts with @petertodd.

@dr-orlovsky Could you merge these changes in, please? https://github.com/pandoracore/rust-bitcoin/pull/2/
Please use --ff-only to avoid creating a merge commit. The branch should be rebased correctly.
It fixes all my nits made above and makes some optimizations I discussed with @apoelstra .

In what sense is an internal node "incomplete knowledge"? It is the root of a complete tree, just not the particular tree that is directly anchored in a particular block header.

In particular it seems that if I need external knowledge to differentiate two types whose meaning and computation and use are otherwise identical, they should not be distinct types.

@apoelstra Let me explain. Bitcoin protocol requires that ALL txid in a block have to be Merkelzed, not some subset of them. Thus, in terms of Bitcoin protocol, MerkleRoot is the "complete knowledge", i.e. successful end-result of the protocol execution, while any MerkleBranch is an intermediate product, covering only a subset of all Txid in the block.

My point is that all nodes in a Merkle tree are treated identically, regardless of whether they are roots or leaves or internal.

Oh than we agree  :) I just call "leafs" the data object. anyway you're also right about matching whatever Satoshi did. so let's stop this bikeshed, do whatever bitcoin core does and copy the usual warning saying this is bat shit but we're stuck with it :)

This isn't bikeshedding - I have never seen this distinction outside of this PR, and when I asked for an explanation (assuming, charitably, that this was another insane satoshi-ism we had to replicate) I was told "you need to have a complete block to distinguish between these types". That's not an explanation, that's just stating one of many ergonomic issues that this causes.
What if I'm making a block and I need to extend the tree? Do I need to convert my old merkle root into a merkle branch because the root is now internal? Why do I need to do that? What goal is accomplished by maintaining this rust-bitcoin-specific artificial distinction?

My take was that rust-bitcoin is used for block validation, not creation, and from this point the distinction of an intermediate type and the end form of some computation process may be a good API practice. However if you think that it's not, I will just change and make the type to be the same.

The word "branch" does not seem correct for a single hash. Either "root" or "node" would be fine by me.

Replaced with the node

I'm not sure I like that latest commit. I think TxMerkleRoot is a type that makes sense with a clear semantic. Personally, I'd opt for just using the internal hash type for the intermediate nodes by lack of agreement on a better representation. In the merkle calculation, we're using the internal type already.

Sure, I wouldn't mind using the underlying hash for internal nodes.

Ok, I started doing that, but came to the point at https://github.com/rust-bitcoin/rust-bitcoin/pull/349/files#diff-8a3492fb2ab337ced7eb96f9120594a3R117
So what about structures like PartialMerkleTree? We can use normal sha256d::Hash there as before, but it seems to be strange not to have a type for this hash, which is clearly semantically different from any other SHA256d hash. Nevertheless, we decided to move from MerkleNode type.
Sorry, but with such cases it seems to me that having two distinct types for partial and complete data in Merklization process is rational. Or, if you are still strongly disagree with that, pls let's stick with the latest version where we have MerkleNode type for all levels.

Sure, MerkleNode is also fine with me. I'll review the PR as is.

Well, a PartialMerkleTree is a partial merkle tree. It doesn't imply anything about the data hashed under the leaves of the tree. So since a partial merkle tree consists of a collection of internal nodes, it's perfectly fine to just hold hashes. It could be generic over H: Hash, but that's overkill as Bitcoin always uses SHA-256d for merkleization for now.

Sure, MerkleNode is also fine with me. I'll review the PR as is.

MerkleNode should do as well.

Can you please rebase so that the unit tests compile and pass for all the commits?

And while you're at it, can you squash in the fixup commits?

@apoelstra everything is ready

Can you run
git rebase -x 'cargo test && cargo +1.22.0 test && cargo test --features "use-serde rand"' master and update each commit so that this command runs clean?

@apoelstra the command succeeds with all of the commits; so there is no change

:( on the first commit, 4b1fb9970f6ce32977945440db8668e60f0ecff7, I get

error: unused imports: `FromHex`, `ToHex`
  --> src/hash_types.rs:22:52
   |
22 | use hashes::{sha256, sha256d, hash160, Hash, hex::{ToHex, FromHex}};
   |                                                    ^^^^^  ^^^^^^^
   |
note: lint level defined here
  --> src/lib.rs:44:9
   |
44 | #![deny(unused_imports)]
   |         ^^^^^^^^^^^^^^

Then on 985bfb70c58d9b82662155b3db0711ce31fbf402 I get many errors like
error[E0599]: no function or associated item named `from_hex` found for type `hash_types::XpubIdentifier` in the current scope
  --> src/hash_types.rs:48:1
   |
48 | hash_newtype!(XpubIdentifier, hash160::Hash, 20, doc="XpubIdentifier as defined in BIP-32.");
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   | |
   | function or associated item `from_hex` not found for this
   | function or associated item not found in `hash_types::XpubIdentifier`
   |
   = help: items from traits can only be used if the trait is in scope
   = note: the following trait is implemented but not in scope, perhaps add a `use` for it:
           `use hashes::hex::FromHex;`
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

Very strange, what I get is:
rust-bitcoin % git rebase -x '~/.cargo/bin/cargo test && ~/.cargo/bin/cargo +1.22.0 test && ~/.cargo/bin/cargo test --features "use-serde rand"'
Executing: ~/.cargo/bin/cargo test && ~/.cargo/bin/cargo +1.22.0 test && ~/.cargo/bin/cargo test --features "use-serde rand"
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s
     Running target/debug/deps/bitcoin-bfc5fbd5ae8e345a

and after quite a lot of time I get
Successfully rebased and updated refs/heads/hashtypes.

rust-bitcoin % git status
On branch hashtypes
Your branch is up to date with 'origin/hashtypes'.

nothing to commit, working tree clean

What I am doing wrong?

@apoelstra done

Great, thanks! I'm unlikely to get it done today on account of Christmas :)

Sure, I am too celebrating the Winter Solstice and Solar Eclipse :)

My rebase command still fails on the first three commits (at least) :(
One persistent error is
error: expected one of `,` or `as`, found `::`
  --> src/hash_types.rs:22:49
   |
22 | use hashes::{sha256, sha256d, hash160, Hash, hex::{ToHex, FromHex}};
   |                                                 ^^ expected one of `,` or `as` here

on 1.22.

@apoelstra very strange, it could be that I had pushed a wrong version. Have re-done rebasing and double-checked, it seems to be fine now.
PS. Travis still fails due to its internal problem, needs to reset cache.


[HELP WANTED] The example of Sanic and Dask does not work. Really need a new one.

Nowadays RapdisAI platform uses the Sanic as backend for cuXfilter. I would like to work with sanic, dask and dask-cuda. The example in Sanic Wiki is the only example on this subject. It is a pity that it does not work anymore.
from sanic import Sanic
from sanic import response

from tornado.platform.asyncio import BaseAsyncIOLoop, to_asyncio_future
from distributed import LocalCluster, Client

app = Sanic(__name__)

def square(x):
    return x**2

@app.listener('after_server_start')
async def setup(app, loop):
    # configure tornado use asyncio's loop
    ioloop = BaseAsyncIOLoop(loop)

    # init distributed client
    app.client = Client('tcp://localhost:8786', loop=ioloop, start=False)
    await to_asyncio_future(app.client._start())

@app.listener('before_server_stop')
async def stop(app, loop):
    await to_asyncio_future(app.client._shutdown())

@app.route('/<value:int>')
async def test(request, value):
    future = app.client.submit(square, value)
    result = await to_asyncio_future(future._result())
    return response.text(f'The square of {value} is {result}')

if __name__ == '__main__':
    # Distributed cluster should run somewhere else
    with LocalCluster(scheduler_port=8786, nanny=False, n_workers=2,
                      threads_per_worker=1) as cluster:
        app.run(host="<IP_ADDRESS>", port=8000)

I have tried a lot to fix but does not succeed. I need something like the following.
from sanic import Sanic
from sanic import response

from tornado.platform.asyncio import BaseAsyncIOLoop, to_asyncio_future
from distributed import Client
from dask_cuda import  LocalCUDACluster

app = Sanic(__name__)

def square(x):
    return x**2

@app.listener('after_server_start')
async def setup(app, loop):
    # configure tornado use asyncio's loop
    ioloop = BaseAsyncIOLoop(loop)

    # init distributed client
    app.client = Client('tcp://localhost:8786', loop=ioloop, start=False)
    await to_asyncio_future(app.client._start())

@app.listener('before_server_stop')
async def stop(app, loop):
    await to_asyncio_future(app.client._shutdown())

@app.route('/<value:int>')
async def test(request, value):
    future = app.client.submit(square, value)
    result = await to_asyncio_future(future._result())
    return response.text(f'The square of {value} is {result}')

if __name__ == '__main__':
    # Distributed cluster should run somewhere else
    with LocalCUDACluster(scheduler_port=8786, nanny=False, n_workers=2,
                      threads_per_worker=1) as cluster:
        app.run(host="<IP_ADDRESS>", port=8000)

Really hope an expert can fix it. That would help others to code real machine learning apps with Sanic and RapidsAI. Thanks

Can you let us know what is broken?


Feature: History items ttl

Overview
Fixes #674
Notes
Shows only the history items that fit the threshold, delete the others

@jobala, +1 on switching the terminology to "expiry". I agree that it will be easier to reason about with that change since the concept is well-understood in the world of cookies etc.

I am sold on switching the terminology to 'expiry'

@jobala I have added an issue to tackle the unit tests in another PR


eslint --fix on botbuilder-dialog* libraries

Description
Auto-fix of eslint on all dialogs.  Sorry for the large number of files updated, but probably better than more PRs 😊
Specific Change-

eslint --fix for botbuilder-dialogs
eslint --fix for botbuilder-dialogs-adaptive
eslint --fix for botbuilder-dialogs-adaptive-testing
eslint --fix for botbuilder-dialogs-declarative

Testing
N/A

Pull Request Test Coverage Report for Build 171840

653 of 798 (81.83%)  changed or added relevant lines in 134 files are covered.
22 unchanged lines in 14 files lost coverage.
Overall coverage decreased (-0.1%) to 84.91%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

libraries/botbuilder-dialogs-adaptive-testing/src/testActions/assertReplyOneOf.ts
2
3
66.67%

libraries/botbuilder-dialogs-adaptive/src/actionContext.ts
1
2
50.0%

libraries/botbuilder-dialogs-adaptive/src/actions/actionScope.ts
3
4
75.0%

libraries/botbuilder-dialogs-adaptive/src/actions/baseInvokeDialog.ts
1
2
50.0%

libraries/botbuilder-dialogs-adaptive/src/actions/codeAction.ts
0
1
0.0%

libraries/botbuilder-dialogs-adaptive/src/actions/deleteActivity.ts
2
3
66.67%

libraries/botbuilder-dialogs-adaptive/src/actions/deleteProperty.ts
2
3
66.67%

libraries/botbuilder-dialogs-adaptive/src/actions/endDialog.ts
2
3
66.67%

libraries/botbuilder-dialogs-adaptive/src/actions/forEachPage.ts
3
4
75.0%

libraries/botbuilder-dialogs-adaptive/src/actions/getConversationMembers.ts
2
3
66.67%

Files with Coverage Reduction
New Missed Lines
%

libraries/botbuilder-dialogs-adaptive/src/actions/switchCondition.ts
1
81.4%

libraries/botbuilder-dialogs-adaptive/src/conditions/onChooseEntity.ts
1
26.32%

libraries/botbuilder-dialogs-adaptive/src/conditions/onChooseProperty.ts
1
29.41%

libraries/botbuilder-dialogs-adaptive/src/input/inputDialog.ts
1
79.92%

libraries/botbuilder-dialogs-adaptive/src/languageResourceLoader.ts
1
77.21%

libraries/botbuilder-dialogs-adaptive/src/recognizers/recognizer.ts
1
86.54%

libraries/botbuilder-dialogs-declarative/src/factory/customTypeBuilder.ts
1
33.33%

libraries/botbuilder-dialogs-declarative/src/factory/typeFactory.ts
1
61.9%

libraries/botbuilder-dialogs/src/prompts/prompt.ts
1
88.32%

libraries/botbuilder-dialogs/src/skillDialog.ts
1
87.04%

Totals

Change from base Build 171833:
-0.1%

Covered Lines:
16035

Relevant Lines:
18038

💛  - Coveralls


Fix typo in README.md

generarte -> generate

Thank you for all your hard work!

You are welcome @andrewmelis.
:smile:


Differential summary potential bug: Reuse child of differential summary

Found a potential bug with differential summaries which might cause ops to be lost.
Scenario
Suppose tree contains some path: /A/B/...

At ref seq 10, suppose /A and /A/B summarize successfully
An op with seq 15 is sent to /A/B
At ref seq 20, suppose /A generates a differential summary
At ref seq 30, a summary is being generated, and suppose /A/B latest op is still seq 15

The state of /A latest summary is:
{
    referenceSequenceNumber: 20,
    basePath: "/",
    localPath: "A",
    additionalPath: "_baseSummary",
}

The state of /A/B latest summary is:
{
    referenceSequenceNumber: 20,
    basePath: "/A/_baseSummary",
    localPath: "B",
    additionalPath: undefined,
}

So now when generating a summary at seq 30, /A/B will see 15<20 and think that it can reuse previous summary handle, which will yield path: /A/_baseSummary/B (which is the correct path).
The problem is that the /A/_outstandingOps blob will be lost in the latest summary, so op 15 will be missing. The summary at seq 30 will be something like this:
/ -> [tree] {
    A -> [tree]: {
        B -> [handle]: "<summaryAtSeq20>/A/_baseSummary/B",
        X -> [tree]: { ... },
    },
    ...
}

So when a client loads from that summary it will load /A/B at the state of seq 10, but start processing ops from seq 20, thus missing seq 15.
Detailed Explanation
This is the result of the code in SummarizerNode.completeSummaryCore which detects if the parent node sent a differential summary or handle via wipSkipRecursion. In the case of handle, this all works fine, since there should not be an op with seq 15 addressed to /A/B, or else it would not send a handle at all.
In the case of differential summary, the problem above is possible. It does a second check: if the node /A/B in this example has never seen a previous acked summary (i.e. latestSummary is undefined), then it will be fine because it will skip recursion.  This node and child nodes will essentially have ref seq 0, and always think it has changed since last summary.
In the case that there is a previous acked summary, it will correctly update the paths to the children, but will continue to send ref seq 20 down to children. If we change it to always skip recursion, the problem should not resolve, because the paths are incorrect. For example if op seq 15 did not exist, then it would try to send a handle but with path /A/B which does not exist in the summary at seq 20, it should be /A/_baseSummary/B. There would be a similar issue with subsequent differential summaries from /A/B's perspective.
Proposed Fix
If we propagate the correct paths down, but keep the reference sequence number unchanged, that should fix it. In the case where op seq 15 does not exist, then the handle sent would have the correct path and there would be no ops between 10 and 20. In the case where op seq 15 does exist, 15>20 would prevent a handle from being sent. A differential summary would reference the correct path as a baseline, but would include ops between 10 and 20.
We would need to refine the assertion in SummarizerNode.refreshLatestSummaryFromPending that the passed in ref seq may be different from the one in the pending node, and the one from the pending node should be used for clearing outstanding ops. It might be useful to store both the real reference sequence number of the summary and the effective one for clearing outstanding ops, because the real one would be used for clearing the pendingSummaries.

Currently differential summaries are completely disabled, so this should only arise when turned back on.
We need automated testing around SummarizerNode and differential summaries before we can turn them on again.

#4296


Incomplete dogapi histograms

I'm not sure if the problem come from dogapi or the way we use it.
Some histograms report incomplete percentiles.

Some metrics only have the 95th percentile, while they all run the same version of dogapi.
And we should report more: https://github.com/DataDog/dogapi/blob/master/src/dogapi/stats/metrics.py#L73
Thanks @miketheman for noticing it.

Closing and reopening on  datadogpy repo.


Added missing translations for wipeing city and additional DE translations

Thanks for taking care of the translations!

Thanks for fixing these :)


Automatic deployment of generated event website to the user's Github Account

I have been working on this feature for quite some time. This will give the users the ability to publish the event website on their Github account in the form of gh-pages.
Workflow

Users will generate the event website with the help of the generator.
If they want to deploy the generated website to their GitHub account, they will press on the Deploy button in the form.
After clicking on the Deploy button, they will be taken to the GitHub authentication page where they will authorize the web app to make changes to their account.
After they approve the requested permissions and enter their credentials, the process of uploading of files and making commits will start.
After the whole website is completed, a live link to the gh-pages of the event website will appear on the page.

Some Notes

I am still working on this feature. It would take some more time to get all the details right(interface, looks, error handling and display). But the core functionality has been somewhat implemented.
Currently, the speed of uploading and committing of the file has been kept a bit low. On higher speeds, there were some conflicts since the Github database was not responding fast enough. I have emailed Github Support for an optimal no of requests per second.
Authentication is managed through the library passport js.
A repo named eventSiteWebApp is created in the user's account which contains the files of the event. User can delete that repository later if he wants.
The app will ask for the user's permission to create and delete a repo from his/her account.
Since the process is a bit slow, please use the EHealth Africa sample for quick testing of the app.

Here is the link to the deployed app:
https://shrouded-ocean-89460.herokuapp.com
@mariobehling @aayusharora Please test and review. Thanks.

@Princu7 Awesome!
I could not try out the whole thing right now as the apps demands access to all repositories and I am not seeing a way to limit it to one repository. I would be ok to share with my personal repositories but sharing of organizations is a bit tricky. Is there a way to allow access to a specific repository or something along the lines?

Currently the "Deploy" button is already shown before the site is generated. Expected: It should show up together with the other buttons after site generation.

@mariobehling I have removed the permissions which asked for access to the private repositories. Also, I have set the organizations permissions to read only. It should be sufficient, I guess.
I am sorry that I didn't elaborate on the permissions required for the app to work. To give power to the app, we have to associate certain scopes to our OAuth token. Scopes let us specify exactly what type of access we need. Scopes limit access for OAuth tokens. They do not grant any additional permission beyond that which the user already has.
Currently, the app requests for OAuth token with three scopes-
public_repo( Used For Creating New Repository containing the contents of event Site)
Grants read/write access to code, commit statuses, collaborators, and deployment statuses for public repositories and organizations. Also required for starring public repositories.
delete(Used In case the repository containing the event site is already there, then we need to remove it and make a create one)
Grants access to delete adminable repositories.
read:org(Used for read-only access to organizations repositories)
Read-only access to organization, teams, and membership.
I will fix the deploy button issue soon. I also made a video demonstrating the whole process. You can view it here
https://www.opentest.co/share/4378d8d0d37011e68531cf81a6426bba
Please review the deployed app. Thanks.

@Princu7  Is there anything left in this issue to work on or is it closed.

@SparshBansal It's finished. There is already a PR open for it :)


Feedback from a bbdiff user

BBEdit "diff" mode is very efficient, here are the tremendously useful features that I miss when I use brackets-compare:

bbdiff, a CLI which takes two filenames as arguments and opens the editor in doff mode between the two files
compare mode opens 3 windows:

TEXT1 | TEXT2
    diff

in the diff window, you get a list of chunks with line numbers, and you can go quickly through the list with the keyboard or mouse, updating the TEXT1 and TEXT2 windows ; then with a click you can apply changes from TEXT1 to TEXT2 or the other way.
3. you can select two filenames in the list/project pane and select a contextual "compare selected documents" menu
4. (There's a directory diff mode also, but I personally prefer to directly use diff -r)

hey i've released v0.2.0. Please try it out and let me know.

I don't know why I was not able to install it this time. Install from URL > https://github.com/bomsy/brackets-compare gives me an (unspecified) error ("Erreur d'extension" in French)

Have you tried installing from the extension registry?

Try installing again! thanks

Installing from the registry works (it loads 0.2.4 from amazon). But installing from https://github.com/bomsy/brackets-compare/ stil doesn't work.

yeah only the install from registry works


Turn of symbols

Hi,
how can I remove the symbols?
Klaus

Just remove the slot code like this.
<img class="like-pointer" slot="like" src="./assets/like-txt.png">

OK, thanks! Nice work!


Automated cherry pick of #4067: fix(region): Guest template & Service catalog

Cherry pick of #4067 on release/2.13.
#4067: fix(region): Guest template & Service catalog

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.13@0620840). Click here to learn what that means.
The diff coverage is 0%.

@@              Coverage Diff               @@
##             release/2.13   #4078   +/-   ##
==============================================
  Coverage                ?    6.9%
==============================================
  Files                   ?     508
  Lines                   ?   85840
  Branches                ?       0
==============================================
  Hits                    ?    5927
  Misses                  ?   79412
  Partials                ?     501

Flag
Coverage Δ

#aFlag
6.9% <0%> (?)

Impacted Files
Coverage Δ

pkg/compute/models/guest_template.go
4.09% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0620840...33ee2dd. Read the comment docs.


Fix incorrect comparison of MEID lastSeenTimestamp

Ticket: https://dt-rnd.atlassian.net/browse/DAQ-2355
Description
Currently instead of using the newest and more up-to-date MEID we use the most outdated one as we compare the timestamps (the bigger the more up-to-date it is) incorrectly
How can this be tested?
Have two MEs for the same clusterID -> check if the one with the bigger timestamp is used for metadata enrichment

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 64.28%. Comparing base (fa4ce9e) to head (1588cd3).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff              @@
##           release-1.3    #4052   +/-   ##
============================================
  Coverage        64.28%   64.28%
============================================
  Files              353      353
  Lines            23747    23747
============================================
  Hits             15265    15265
  Misses            7229     7229
  Partials          1253     1253

Flag
Coverage Δ

unittests
64.28% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update homepage

Made the following changes in the home page:

Removed the "Recruiting Now" button.
Spacing above "So, what do we do?" heading.
Added 6 more domains.
Added an extra slide in the carousel.

Write this in PR description as well @mahiprasad


fix: includes/mobile-services-enable-google-cloud-messaging.md

Escape swalled element with code block

@nschonni : Thanks for your contribution! The author, @spelluru, has been notified to review your proposed change.

@nschonni - Thanks for updating the article.
#sign-off


Bug 2516 fix master

Same as https://github.com/Microsoft/pxt/pull/2590 but for master branch.

@caitlin-hennessy,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Added AMP Gtag UI

refs https://github.com/TryGhost/Ghost/issues/11980
This adds a new UI to set the amp_gtag_id setting.
When AMP is disabled:

When AMP is enabled:

What about custom trackers? For example Yandex.Metrica?


Azure files planning guide should mention limitations of the naming

This documentation or some other Azure file documentation should mention storage account naming limitation:
Join-AzStorageAccount : Parameter -StorageAccountName 'storxxxxxxxxxxxshareprod' has more than 15 characters, which is
not supported to be used as the SamAccountName to create an Active Directory object for the storage account. Azure
Files will be supporting AES256 encryption for Kerberos tickets, which requires that the SamAccountName match the
storage account name. Please consider using a storage account with a shorter name.

if share points have similar limitations they should be also listed.
I tried to find mentions about the limitation but could not find any.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 836fd4cd-0afb-4c93-c2cc-57a6d50e7315
Version Independent ID: 3a1420f8-4b20-100a-4c3e-223f00b079d0
Content: Planning for an Azure Files deployment
Content Source: articles/storage/files/storage-files-planning.md
Service: storage
Sub-service: files
GitHub Login: @khdownie
Microsoft Alias: kendownie

@jikuja
Thanks for your feedback! We will investigate and update as appropriate.

@jikuja We recommend explicitly specifying $SamAccountName in addition to $StorageAccountName. If you don’t specify $SamAccountName, the script will fail if the $StorageAccountName is > 15 characters in length. Storage account names can be up to 24 chars if $SamAccountName is specified, but $SamAccountName is recommend to be 15 characters or less due to AD limit.
The planning doc you referenced is intended to be high level and isn't the right place for this kind of detail. Can you tell me where in the documentation you ran into Join-AzStorageAccount? Was it here:
https://learn.microsoft.com/en-us/azure/storage/files/storage-files-identity-ad-ds-enable

Checked the tooling.
AzFilesHybrid.psm1 @ v0.2.4 automatically converts too long storage account names to shorter. No documentation changes needed because of that.

#signoff


mockService unexpected behaviour

I am trying to use mockService in my sinon-chai tests. It looked really helpful when I read the README file. However, I ran into several problems trying to use it, and even now that I solved the problem, I don't fully understand the function's behaviour nor effect, yet. Using my example:
I have a service that injects UI-router's $state service. Since I am not really interested in having to deal with the effects of $state or any inconveniences, I thought in giving mockService a go. This was my beforeEach block on the first try:
beforeEach(function () {
    bard.appModule('mmApp.core');
    bard.inject(this, '$q',
                'userProgressService',
                'surveyDataService',
                'userInputService',
                '$state');
    bard.mockService($state);
});

I left the mockService with just one argument because I thought it would make every function return void and everything would be alright. When I ran the tests, though, I got the following error:
TypeError: 'undefined' is not an object (evaluating 'config._default')
This was in the exact line of mockService. In order to solve this I used an empty object as the second argument, like so
bard.mockServices($state, {});
based on the assumption that the function isn't prepared for not receiving the second argument. With this code, when I ran the tests, I got rid of the first error and got this one instead:
TypeError: 'undefined' is not a function (evaluating '$state.go('/phase/step', {phaseId:phaseNumber})')
I thought it was related to the $state.go function not being defined, not even as a stub or spy... I tried again, now with
bard.mockService($state, {
    go : sinon.stub()
});

in order to explicitly tell the function that I wanted an empty stub for $state.go (which, BTW, is the only function of $state I am currently using). This got rid of the errors!
Now, my question actually is... Why did those 2 errors happened in the first place?
Also, I recommend some more explanation for this function in the README, since it is one of the most useful of the provided. I checked the docs in the source and they are much more complete, you should try to make something more like it.
Anyway, great project, I don't even remember how to inject dependencies without this, anymore.

@fplgusmao Saved my day this workaround!


Rewrite user groups settings UI to work more like "Manage streams"

As part of the overall project in #19525, we need to substantially upgrade the"User groups" management UI.  Rather than something tactical like #16453, the new plan is to make a new /#groups overlay similar to "Manage streams", with the group names and descriptions on the left, designed in a way to support #19525, and then at some point just delete the legacy "user groups" management UI.  Here's a technical design for how to do this transition effectively:

We rename static/js/settings_user_groups.js to static/js/settings_user_groups_legacy.js, just to make it clear that code will be deleted and developers should avoid spending much time on it.
We create a new static/js/user_groups_ui.js to be the main module for the new project. We register it as #groups in the hashchange.js system, but don't document or link to it anywhere (since at this point it'll be basically an empty overlay); this will let us iteratively do pull requests to implement it.
We implement the left panel with a list_widget similar to the stream settings one; in particular we'll want a "Filter by name" widget, the group name and description, and maybe some indicator for whether you're a member.
We implement the right panel with a group selected also similar to "stream settings", with a tab switcher. Right now the tab switcher might seem unnecessary, but with #19525, we'll very much want it, and life will be a lot easier if we just design this (still hidden) UI how we want it once #19525 is completed; we can cut over to it once it's ready.
One tab will be "Direct members", showing the users who are members of the group in the same format that we show subscribers to a stream (and with the same filter/add/remove widgets).
One tab will be "Group settings", which initially will be just name and description. We'll likely want to copy the UI from "Stream settings" designed in https://github.com/zulip/zulip/issues/19519.
One tab will be "Group members", showing the groups that are members of this group in some format, once #19525 is far enough along to support this.
We'll probably also want a "Permissions" tab where you can see all the permissions this group has in some format.  We haven't designed this at all, beyond it being a natural thing we'll want after #19525; I mention it mainly because it'll affect how much
space we need.
We make a "Create user group" widget. We should avoid repeating the mistakes of "stream settings", and have this form share its template with the "Edit user group" UI to the extent that we let you choose things when creating it (vs. just have you specify a name and then let you edit from there).  It's worth mentioning that we probably want the "Create user group" widget to be a dialog_widget modal that can be rendered on top of the "Organization settings" overlay as well, so that we can offer "Create user group" as an option when selecting the user group to have a given permission without navigating all over the place.
Once the above are done, we can delete the settings_user_groups.js UI and cutover the UI, perhaps after a bit more work

This work is a high priority for the project, since it's a mandatory component of the overall vision in #19525, which a very large and complicated project (and thus one where we want to push on its dependencies as much as possible to manage complexity and risk).
@m-e-l-u-h-a-n do you think you'd be able to take this up?  I think your experience working on the "Manage streams" UI would give you a great deal of comparative advantage for this work.

Closing in favor of  #28012, as I think that's the main remaining piece of this issue.
@timabbott @sahil839 Please file any follow-up issue that you think will help track the remaining points. I think they'll mostly come up organically, though -- e.g., we'll create a permissions management panel when we start adding group management permissions.


can not run setup

Hello,
when i run "npm run setup" under C:\dev\js\frame that is cloned from github on 14-04-2015, i get the following error. Please advise and many thanks in advance.
0 info it worked if it ends with ok
1 verbose cli [ 'c:\\Program Files\\nodejs\\node.exe',
1 verbose cli   'c:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'setup' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'presetup', 'setup', 'postsetup' ]
5 info presetup<EMAIL_ADDRESS>6 info setup<EMAIL_ADDRESS>7 verbose unsafe-perm in lifecycle true
8 info<EMAIL_ADDRESS>Failed to exec setup script
9 error<EMAIL_ADDRESS>setup: `./setup.js`
9 error Exit status 1
10 error Failed at the<EMAIL_ADDRESS>setup script.
10 error This is most likely a problem with the frame package,
10 error not with npm itself.
10 error Tell the author that this fails on your system:
10 error     ./setup.js
10 error You can get their info via:
10 error     npm owner ls frame
10 error There is likely additional logging output above.
11 error System Windows_NT 6.2.9200
12 error command "c:\\Program Files\\nodejs\\node.exe" "c:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "setup"
13 error cwd c:\dev\js\frame
14 error node -v v0.10.35
15 error npm -v 1.4.28
16 error code ELIFECYCLE
17 verbose exit [ 1, true ]

Thanks for opening an issue. I need to download a VM to test on Windows and I'm not sure when I'll have a chance to do that.
Did you start MongoDB before running $ npm run setup?
What version of Windows are you running?
Do you plan on hosting your app on a Windows server? If not, have you considered developing on a virtual machine running *nix?

Thanks for replying...I am using windows 8.1 and I did have mongodb running before I ran "npm run setup".
I ll try it on linux today.
Thanks

Closing for now. Feel free to post again.


Nav lists

Hi @jonathantneal, thank you for this work. It's now time to move on and leave back old utilities like normalize (that support very old browsers) and resets.
I really like your clean and surgical approach but I have a thought about the nav ol, nav ul code. The nav element specification says (EXAMPLE 11) that it doesn't have to contain a list element and provide an example with paragraphs inside.
So i think that users can choose to use (or not) a list inside a nav, then providing a reset code for this situation can be useles for those who don't use any list inside a nav (like me), resulting in unused code.
What do you think about removing this assumption from the code?

Yes, it may not always be used, but it would be expected in many scenarios, and it is officially recommended:

In cases where the content of a nav element represents a list of items, use list markup to aid understanding and navigation.
— https://www.w3.org/TR/html/sections.html#the-nav-element

Does this seem reasonable?

Yes, but right now the css does not provide a css fix for browser consistency, it provide a custom and personal look and feel bu just removing a standard rapresentation only inside the nav element. I don’t think this is related to browesers inconsistency at all..

Please forgive my confusion or any hint of aggression, but I never said anything about inconsistencies, so you might be reading more into what I’m saying than I mean to express.
normalize.css is the library that deals with exclusively normalizing things. This library also includes common developer preferences, like box-sizing: border-box, etc.
Your opinion on whether or not to use lists is totally allowed and shared by others. In fact, many years ago, we all kind of hashed out our thoughts https://css-tricks.com/wrapup-of-navigation-in-lists/
In the end, I think requests like this are a good reason to make a PostCSS version of this library, where one can remove certain rules by options at build time. 😄

sanitize.css is a CSS library that provides consistent, cross-browser default styling of HTML elements alongside useful defaults.


Experiment test cases have reference to user path

https://github.com/persephone-tools/persephone/blob/aa695758e675f47e91ef79beb0f470e97f754e6e/persephone/tests/experiments/test_na.py#L260-L273
This test depends on being able to locate a Tensorflow checkpoint file in a local directory. This will fail for other users.
Same with:
https://github.com/persephone-tools/persephone/blob/aa695758e675f47e91ef79beb0f470e97f754e6e/persephone/tests/experiments/test_na.py#L275-L282
And
https://github.com/persephone-tools/persephone/blob/aa695758e675f47e91ef79beb0f470e97f754e6e/persephone/tests/experiments/test_na.py#L284-L292

Creating a minimal test corpus as per #166 will make this far less of a big deal.


Does it support fourier transform?

Hello
I'm looking for a faster alternative to OpenCV's dft function.
Does ComputeLibrary include a Neon-based Fourier transform?

Hello @zanazakaryaie,
Unfortunately, we don't have a NEON- based DFT in ComputeLibrary.
Is this something you are interested in? If yes we can mark it as feature request which we can try and schedule when we can or you can contribute one yourself as ACL is open for contributions here .

Hello GeorgeARM
Thanks for your reply
Actually, I was looking for a way to accelerate object trackers (e.g. KCF, MOSSE, etc.) on ARM CPUs and I found that DFT is the main time-consuming part of such algorithms. I tried to use FFTW library which has mentioned to benefit from Neon coprocessor but it was 3-4 times slower than OpenCV's DFT!
Since I've seen similar requests for a faster DFT (link1, link2, link3, link4), I think it would be very nice if you add a Neon-based DFT in upcoming versions of ACL.

HI @zanazakaryaie
Are you interested in complex to complex or real to complex transform?

Hi @zanazakaryaie
In 19.05 we introduced DFT
https://github.com/ARM-software/ComputeLibrary/blob/master/arm_compute/runtime/NEON/functions/NEFFT1D.h
https://github.com/ARM-software/ComputeLibrary/blob/master/arm_compute/runtime/NEON/functions/NEFFT2D.h
This is used in the convolution layer for specific shapes.
Please reopen the ticket if you need additional support.


Refactor tests in pytest.run.test.ts

Currently the tests are just not manageable, nor readable.
src/test/unittests/pytest/pytest.run.test.ts

Does this "needs PR"?

Closing as stale.


Why we need ts-node for setup

Question

I apologise if this has been asked many times, but I asked this question because I couldn't find it when I looked for it.
Why does the Setup section ask me to install ts-node?
I read the code in the typia command tools, but it didn't seem to be running ts-node.

The library requires ts-node is not typia but nestia.
This is just my mistake.

I should upgrade the setup wizard not to install the ts-node.


chore: switch to debian buster for arm

With Electron 22, chromium and v8 builds with C++17 and this in turn has implications for our native modules to be built with the same version. GCC 6.3 toolchain we ship with out stretch image does not support all of C++17 features and leads to the following failure, refs https://monacotools.visualstudio.com/Monaco/_build/results?buildId=199711&view=logs&s=c041ccde-92b7-5163-1d93-b1c06077f4b4&j=15ca7ee2-e153-5761-fc3c-51cb7d92e001
error: 'is_lvalue_reference_v' is not a member of 'std'

We will switch to buster that comes with GCC 8 to support C++17 feature usages.

@deepak1556 use bionic 18.04 as mentioned to avoid bumping glibc requirements
see -> https://github.com/microsoft/vscode/issues/181720#issuecomment-1542643649


Make output file names unique when rendering videos in documentation

A quick follow-up to #402.
List of Changes

Allow duplicate scene names for examples rendered by the manim directive.

Motivation
Scenes with the same name are currently saved with the same name. Avoid this so that people can use Example(Scene) however often they wish. :-)
Explanation for Changes
This adds a dict classnamedict keeping track of the already rendered scenes. With it, a number is added to the output file name to make it unique.
Testing Status
Works locally as intended; RTD should also be compatible.
Acknowledgement

[x] I have read the Contributing Guidelines

Looks good to me, and a very convenient feature!
I did not test it, but I think if it works on your machine, it will also work on others.
One question that comes to my mind:
Will this also work in case of class names:
Example, Example, Example1 , Example2
Won't the second Example then collide with Example1?

One question that comes to my mind:
Will this also work in case of class names:
Example, Example, Example1 , Example2
Won't the second Example then collide with Example1?

No, this cannot happen as the name of every scene is modified: in your example, the output names would be Example-1, Example-2, Example1-1 and Example2-1.


Add timeout for handling of Command on agent's implementation

ISSUE TYPE

Improvement Request

COMPONENT NAME

Agent, ServerResource

CLOUDSTACK VERSION

All versions

CONFIGURATION

N/A
OS / ENVIRONMENT

N/A
SUMMARY

The value for wait set in Command is not honored by agents (SSVM, host agent, etc.) while processing the request. These requests can also pile up if the same request is sent multiple times because of retries by user or the management server itself. This can potentially cause issues on SSVM and agents while executing long tasks leading to high wait times for commands to get executed on host agents, ssvm, etc.
STEPS TO REPRODUCE

Add a sleep in a ssvm command for 60 seconds and a log line before and after the sleep to check number of executions for that command.
Send 100 requests in parallel which will execute the above ssvm command with cmd.setWait(5).
Check logs on ssvm to check the actual number of requests received.

Note: The above example is for a Command for which executeInSequence is false.

EXPECTED RESULTS

If the request on management server times out in 5 seconds, the processing of command on SSVM should stop after 5 seconds as well.

ACTUAL RESULTS

The number of requests received on the SSVM would be executing these requests in batches of 10 requests or so. The requests would still get processed even if the server itself has timed out and no longer waiting for the answer. You will the log for both before sleep and after sleep.

cc: @DaanHoogland

@vishesh92 I think you are right in your expectations. There are some more cases to define/consider;

will responses be sent of timed out but completed commands?
will the time-out be from receiving of the command? I think you want the commands to not be executed if these have timeout before being on top of the queue, right?
are time-out responses being sent?
?


About page not responsive on Tablet device.

Describe the bug
The About page is not responsive on tablet device. It is only responsive on desktop and mobile but when you are on a tablet, the text begins to overlap on each other.
To Reproduce
Steps to reproduce the behavior:

Go to 'https://www.digitomize.com/about
2.Minimize you screen to a tablet view
Notice the heading in the about page, scroll down and notice how the Contest and portfolio heading is on top of the paragraph. Scroll down again on how to contribute and notice that the texts are compacted together without any margin or spacing.

Expected behavior
The heading in the page need to be visible on all devices.
The contests and portfolio direction need to be changed to column on this device for it to be responsive and ensure good user experience.
How to contribute section, the content need more spacing so it doesn't overlap on each other.

If this is a change you will love to implement, I would like to work on it.

hello, Can you assign this to me?

hello, Can you assign this to me?

It has been fixed.


Azure Function Linux Nodejs and @azure/storage-blob gives error: Cannot read property 'startsWith' of undefined

Package Name: @azure/storage-blob
Package Version:  "@azure/storage-blob": "^12.15.0",
Operating system: MacOS Big Sur 11.2.3
[ ] nodejs

version:   v14.17.0

[ ] typescript

version:  "typescript": "^4.0.0"

Describe the bug
I am testing my Azure Nodejs Linux function locally and this code:
import { BlobServiceClient } from "@azure/storage-blob";

const connectionString = process.env[
  "AZURE_STORAGE_CONNECTION_STRING"
] as string;

export const blobServiceClient =
  BlobServiceClient.fromConnectionString(connectionString);

is generating this error message:
john@MacBook-Pro:~/myApp/code/seal/azure_function$ rm -rf dist/ && npm run-script build &&  func start

><EMAIL_ADDRESS>build /Users/john/myApp/code/seal/azure_function
> tsc && tsc-alias

Azure Functions Core Tools
Core Tools Version:       4.0.5198 Commit hash: N/A  (64-bit)
Function Runtime Version: <IP_ADDRESS>67

Functions:

        sealworker: [GET,POST] http://localhost:7071/api/sealworker

For detailed output, run func with --verbose flag.
[2023-08-15T09:06:03.440Z] Worker was unable to load entry point "dist/sealworker/index.js": Cannot read property 'startsWith' of undefined
[2023-08-15T09:06:03.551Z] Worker process started and initialized.
[2023-08-15T09:06:03.564Z] Worker was unable to load function sealworker: 'Cannot read property 'startsWith' of undefined'
[2023-08-15T09:06:03.568Z] Worker failed to load function: 'sealworker' with functionId: 'c146c9d1-05ef-4aa2-bb18-d436186b7f65'.
[2023-08-15T09:06:03.568Z] Result: Failure
[2023-08-15T09:06:03.568Z] Exception: Worker was unable to load function sealworker: 'Cannot read property 'startsWith' of undefined'
[2023-08-15T09:06:03.568Z] Stack: TypeError: Worker was unable to load function sealworker: 'Cannot read property 'startsWith' of undefined'
[2023-08-15T09:06:03.568Z]     at extractConnectionStringParts (/Users/john/myApp/code/seal/azure_function/node_modules/@azure/storage-blob/dist/index.js:13654:26)
[2023-08-15T09:06:03.568Z]     at Function.fromConnectionString (/Users/john/myApp/code/seal/azure_function/node_modules/@azure/storage-blob/dist/index.js:24328:32)
[2023-08-15T09:06:03.569Z]     at Object.<anonymous> (/Users/john/myApp/code/seal/azure_function/dist/sealworker/utils/Azure/blobServiceClient.js:6:62)
[2023-08-15T09:06:03.569Z]     at Module._compile (internal/modules/cjs/loader.js:1068:30)
[2023-08-15T09:06:03.569Z]     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
[2023-08-15T09:06:03.569Z]     at Module.load (internal/modules/cjs/loader.js:933:32)
[2023-08-15T09:06:03.569Z]     at Function.Module._load (internal/modules/cjs/loader.js:774:14)
[2023-08-15T09:06:03.569Z]     at Module.require (internal/modules/cjs/loader.js:957:19)
[2023-08-15T09:06:03.569Z]     at Object.apply (/usr/local/Cellar/azure-functions-core-tools@4/4.0.5198/workers/node/dist/src/worker-bundle.js:2:44135)
[2023-08-15T09:06:03.569Z]     at require (internal/modules/cjs/helpers.js:88:18)
[2023-08-15T09:06:03.569Z]     at Object.<anonymous> (/Users/john/myApp/code/seal/azure_function/dist/sealworker/services/attachment.js:58:29)
[2023-08-15T09:06:03.571Z]     at Module._compile (internal/modules/cjs/loader.js:1068:30)
[2023-08-15T09:06:03.571Z]     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
[2023-08-15T09:06:03.571Z]     at Module.load (internal/modules/cjs/loader.js:933:32)
[2023-08-15T09:06:03.571Z]     at Function.Module._load (internal/modules/cjs/loader.js:774:14)
[2023-08-15T09:06:03.571Z]     at Module.require (internal/modules/cjs/loader.js:957:19).
[2023-08-15T09:06:03.785Z] Host lock lease acquired by instance ID '000000000000000000000000DB4F1B81'.
^C[2023-08-15T09:34:22.578Z] Language Worker Process exited. Pid=53985.
[2023-08-15T09:34:22.578Z] node exited with code 130 (0x82). .

Why is simply importing the package giving an error?
Not sure what I should do now...
Any hints?
To Reproduce
Steps to reproduce the behavior:
1.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

@DaCao from the error stack it looks like it's having trouble parsing the connection string. Can you try logging it and check that it isn't malformed?

@xirzec  Thank you! I have found the cause and indeed the string was malformed.
This issue can be closed


[NavigationMenu] Fix menu reopening on mouse move when closed with Esc

Fixes #1529
Makes the menu stay closed after pressing the Esc key

I made the changes to make a shallower approach, though the root cause of the focus issue seems to be related to the document focus variable in UseEscapeKeydown and the fact that Storybook has two documents (One inside an iframe). Quoting from the jQuery Docs:

[the keydown event] can be attached to any element, but the event is only sent to the element that has the focus. Focusable elements can vary between browsers

Therefore, since Storybook uses an iframe, there are two documents and only one of them can be focused, and if the external one (Sidebar + Header) is focused then the UseEscapeKeydown handler won't trigger and the dropdown won't close, which is likely what you saw happening.
This is issue is likely to be exclusive to Storybook though, since regular web pages usually don't have more than one document, and it won't be a problem.
An easy way to test this behavior is adding a console.log(ownerDocument.hasFocus()) inside the useEffect on the useEscapeKeydown hook, you'll see that when you focus the sidebar and proceed to hover on a menu it'll say false and Escape won't close the menu, though if you focus the inner document (iframe) then it'll say true and the menu will work fine. Tell me if you find any issues 😄


PR: server scope podrestart-java-integ-test

Adding tests for the changes of server/cluster scope properties that cause server pods to be restarted
The properties being tested are: containerSecurityContext, podSecurityContext, resources
Adding usecases to the README.md

The latest Jenkins clean run: http://wls-jenkins.us.oracle.com/view/weblogic-operator/job/weblogic-kubernetes-operator-javatest/1447/consoleFull

Because of the last "@maggiehe00
jenkins-ignore javadoc update" Jenkins weblogic-kubernetes-operator is aborted

I'm going to sync to the latest develop branch and resolve integration-tests/README.md conflict.

@markxnelson @anpanigr Please kindly review the code. Thanks!

Approved


[ionic2] EXCEPTION: No provider for Http! (MyApp -> TranslateService -> Http)

Hi,
I'm having similar issue like someone else had with its Project. Mine was Ionic2 (2.0.0-alpha.49), ng2-translate (1.3.0) and angular2 (2.0.0-beta.0- yes, beta0 since ionic2 relies on it.
EXCEPTION: No provider for Http!

Help me out (if possible) :+1:

Hmm could you make me a minimal plunkr with the problem please?

Hi @ocombe,
sadly, plunker doesn't have the support for alpha/beta version of ionic2. That's why I've created a simple project for you here. To install:

git clone
npm install -g ionic@beta gulp
npm install
ionic serve

Thanks a lot for your repository, this bug has been fixed in the latest version (1.3.1), you shouldn't have a problem like this anymore.
The main issue with ionic 2 is that you cannot use bootstrap to instantiate HTTP_PROVIDERS, but I used the injector and now it's all right !

Hmm as a matter of fact I will revert this change. Using the injector to get HTTP_PROVIDERS is not the right solution.
The problems comes from the fact that ng2 translate uses beta 1 and ionic2 is still stuck at beta 0.
If you change the package.json version for ng2 translate to 1.2.4 (last version on beta0) then it will work.
By using a different version of angular 2, webpack will load 2 different instances of Angular 2, and the one that ng2 translate gets is not the same, which means that the app is not bootstrapped for ng2 translate and the injector is different.
The solution for you is to use ng-translate 1.2.4 until ionic 2 moves to angular 2 beta 1.

Firstly, thank you that you took your time and look up in this issue. As you said 1.3.1 was working. However, I thought it has something to do with angular2 versions. Thank you for replying this; hopefully, we'll get new version of Ionic2 out soon with the beta1 dependency of Angular2.
Best regards! :+1:

You can use 1.3.1 but I think that you might run into trouble (and loading 2 versions of angular will double your file size!).
If you don't need the new features, you can stay on 1.2.4 until then.
By the way, there is a Translate service bundled into ionic 2.

Translate service will be taken out of Ionic2 (discussed with Mike Hartington from Ionic team yesterday). And, as far as I can tell... your ng2-translate is on its way to get the success of Pascal Precht's Translate for Angular1. Will move to 1.2.4 as per your advice :8ball:

Hmm I know what I could do, I could change the dependency for ng2-translate from "angular2": "~2.0.0-beta.1" to "angular2": "~2.0.0-beta.0", this way it would use the beta.0 for you (as defined in your package.json), and beta.1 for others. There was no breaking change in beta.1 for me.

Ok you can try version 1.3.2, it should work for you !

@ocombe thank you. it works. :+1:
You're doing awesome job, keep up the good work! :)

I changed it to peerDependencies in 1.4.0, it should still work for you :)

@lordgreg this is not directly related to this problem, but I'm not sure if I should log a bug for this and maybe you already have the solution: I also use ng2-translate in my ionic2 app, and since then the file translate.parser.ts gets picked up by ionic and appears under build/js in Chrome debbuger, and basically it makes debugging impossible because somehow every breakpoint that I try to set on app.bundle.js gives me a breakpoint at the end of the translate.parser.ts file, and I can't figure out why

This looks like a problem of sourcemaps to me.

@mebibou why setting breakpoints in webpack created bundle? Enable sourcemaps and make breakpoints in original files. See the ticket here

@lordgreg oh thanks never saw this one :)

I have same problem just like you @mebibou and that ticket didn't help me @lordgreg . Do you guys have any solution? Here's an example. Thanks


Matched Training isn't working

When I try to match applicant(s) to a particular training the "Matched Training" option doesn't allow me to type in the box, or it doesn't list any of the worskhops for me to select.

For some reason, Select2 (drop-down) widget is not rendering for MatchTrainingRequestForm and BulkMatchTrainingRequestForm. Thanks @sheraaron for reporting this issue!


Unable to parse output of www.google.com

This test:
simpleRequestTest : String -> IO ()
simpleRequestTest testName = do
  Right response <- simpleHttp "www.google.com" 80 "/" | Left err => failed testName err
  if Strings.isInfixOf "google" $ responseBody response then ok testName
  else failed testName "`google` not found within response"

fails with:
At 16:11738:
	crlf new-line
At 16:11738:
'	character '
At 16:11738:
	a different token

I believe this is fixed after b1a2a6e4ab741096e53001d36df3780e067c7cd0 and a few commits prior. But I've made a number of changes, and things are generally quite fragile right now, so no promises.
Feel free to re-open if it's still breaking.


Individual package artifacts are ignored

For small projects a global artifact is used:
package:
  artifact: path/to/artifact.zip

but in later projects we used to separate the functions to reduce upload size and isolate dependencies:
package:
  individually: true

functions:
  tagGenerator:
    handler: hakuna.matata.Handler
    package:
      artifact: path/to/artifact-for-function.zip

This is valid using serverless framework, but the plugin ignores the artifacts per function. so the are missing in the generated sam template.

:+1:


Docstring support

I have a feature request for this project that I would like to propose. I think, the addition of this feature would greatly enhance the project's functionality and usability.
Description
The proposed feature is to incorporate a standardized function, such as doc, that enables retrieving the docstrings for classes and methods and seamlessly inserting them into the UML diagram. This addition would significantly contribute to clarifying the purpose and functionality of classes within the UML representation. It would provide developers with a convenient and consistent way to access the documentation, resulting in improved understanding and maintainability.
Previous Status
Furthermore, I am unsure whether this feature has been requested previously. If it has, I kindly request information regarding its current status or any ongoing efforts to implement it.

Duplicate of https://github.com/lucsorel/py2puml/issues/34.
Thank you for posting feature requests, but please check beforehand whether a similar issue already exists. There are only 10 open issues, it is easy to check :smiley:
You can contribute on #34 to suggest how the docstring of a module and a data structure (class, enum, dataclass, etc.) should appear in the generated PlantUML documentation (see https://github.com/lucsorel/py2puml/issues/34#issuecomment-1352265035), that would help implementing something satisfying. Thanks :pray:


Fix a few more SSR problems

Tested SSR render with bootstrap-vue-3. Found these.

Thank you @xvaara  :heart:

Closing it as not answered. The focusTrap changes transferred to #34989

I did answer... ?


combining Actions (sending initial Key)

I really like the app and how it can use the extra ke on the remote. Since i have a openhab instance i would like to use this app to also create a "clone remote" in openhab. For this i would need a new funktionality:

beeing able to combine e.g.: "execute" and "replace" (or just sending the key AND executing stuff)

i went through the code but i dont think its as easy as i imagined.
but basically a flag/checkbox to still send the initial key would be great, i will try it but if you/anyone with more experience wants to implement it, that would be great :-D.
cheers Lukas

I would also love to have this functionality. Was looking through the code, I guess the buttons that launch an app can be set to "Execute" and then call the proper luna-send command together with other commands one wants to execute.
But when it comes to "normal" key presses e.g. play/pause I've not understood how/whether it's possible to send the keystroke as a command.
Thanks for all the work to set this app up! Would be a great addition to have.

I have kinda worked around it now, by just executing a curl command that invokes a webhook trigger on home assistant. Home assistant will then send the original button press via the webOS TV integration + any addiitonal actions I want it to execute. That works fairly well so far.

Hey, nice to see i am not the only one :-D.  i actually got a modified app to work, it will send all key presses normally, except the "netflix" buttons. If i get a chance i can upload the fork properly. For now: i hope ai uploaded the right ipk. https://easyupload.io/9ozjhi
Greetings on the go,
Lukas


[WIP] #204 Transformer Abstraction

The primary goal with this PR is to help the discussion in https://github.com/RasaHQ/whatlies/issues/204. If folks agree we might expand the idea here to a code-change.

@mkaze you're right. Object oriented would be better.
I'll go ahead and iterate using an object variant for Pca. Once we both agree that it's what we want I'll continue implementing the other ones.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

There's an issue that seems related to tqdm which is a progress bar used by ivis. It also seems like this project just got an update according to pypi.

@mkaze no rush, but I think the branch is looking green on my end.


patch: file name changes and exports

:tada: This PR is included in version 4.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Freeze on window close

Hyprland Version
Hyprland, built from branch main at commit 5c50fac907df3463492a019f01f84d547ab93650 dirty (windowrules: add workspace param). Tag: v0.27.2-65-g5c50fac9
Bug or Regression?
Bug
Description
Hyprland freezes sometimes when closing a window. This is recoverable by SSHing in and doing killall Hyprland -9. Switching VTs does not work.
How to reproduce

Label: loop

Open foot
Close foot

Goto loop
Eventually, Hyprland will freeze one or two frames into the animation of closing the window

Crash reports, logs, images, videos
GDB backtrace (OCR'd from screenshot):
[New LWP 2321]
[New LWP 2322]
[New LWP 2323]
[New LWP 2324]
[New LWP 2478]
[New LWP 2479]
[New LWP 2726]
[New LWP 2845]
[New LWP 2846]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x000077d9f09a0fb in __lll_lock_wait_private ()
from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007f7d9f09a0fb in __lll_lock_wait_private ()
at /usr/lib/libc.so.6
#1  0x00007f7d9f0ad960 in malloc () at /usr/lib/libc.so.6
#2  0x00007f7d9f087c11 in _IO_file_doallocate ()
at /usr/lib/libc.so.6
#3  0x00007f7d9f096d20 in _IO_doallocbuf ()
at /usr/lib/libc.so.6
#4  0x00007f7d9f095a9d in _IO_file_underflow ()
at /usr/lib/libc.so.6
#5 0x00007f7d9f096dd6 in _IO_default_uflow ()
at /usr/lib/libc.so.6
#6 0x00007f7d9f0897db in _IO_getline_info ()
at /usr/lib/libc.so.6
#7 0x00007f7d9f088570 in fgets () at /usr/lib/libc.so.6
#8 0x00005584214b1e38 in execAndGet[abi:cxx11](char const*) ()
#9 0x0000558421481e78 in CrashReporter::createAndSaveCrash(int) ()
#10 0x000055842141affc in handleUnrecoverableSignal(int) ()
#11 0x00007f7d9f04fab0 in <signal handler called> ()
at /usr/lib/libc.so.6
#12 0x00007f7d9f09f26c in () at /usr/lib/libc.so.6
#13 0x00007f7d9f04fa08 in raise () at /usr/lib/libc.so.6
#14 0x00007f7d9f038538 in abort () at /usr/lib/libc.so.6
#15 0x00007f7d9f0392db in () at /usr/lib/libc.so.6
#17 0x00007f7d9f0ab3d8 in () at /usr/lib/libc.so.6
#18 0x00007f7d9f0adcb3 in free () at /usr/lib/libc.so.6
#19 0x00007f7d9fc7e9d9 in () at /usr/lib/libwayland-server.so.0
#20 0x00007f7d9fc807a9 in wl_resource_destroy ()
at /usr/lib/libwayland-server.so.
#21 0x000055842152fa30 in CWaylandResource::~CWaylandResource() ()
#22 0x0000558421530025 in destroyManagerResource(wl_client*, wl_resource*) ()
#23 0x00007f7d9ebdd4f6 in () at /usr/lib/libffi.so.8
#24 0x00007f7d9ebd9f5e in () at /usr/lib/libffi.so.8
#25 0x00007f7d9ebdcb73 in ffi_call () at /usr/lib/libffi.so.8
#26 0x00007f7d9fc7bada in () at /usr/lib/libwayland-server.so.0
#27 0x00007f7d9fc80180 in () at /usr/lib/libwayland-server.so.0
#28 0x00007f7d9fc7eae2 in wl_event_loop_dispatch ()
at /usr/lib/libwayland-server.so.0
#29 0x00007f7d9fc7f2d7 in wl_display_run ()
at /usr/lib/libwayland-server.so.0
#30 0x000055842140b061 in main ()

this is a crash, but an odd one.
patch.txt
?

I can't seem to repro with the patch applied, nice!

wtf

ef0d97153ac121a75be1478808123ab8e7cdf4de


[3.0] Update manifests to match haproxy changes to work as an http proxy

(cherry picked from commit 4f26406f018dc46f4e4d7087ff7966fa0c0fd2e9)

 Bug 1071994 - kubectl run --attach exits early

Please, do not merge: figuring out why the two PR's together aren't passing in Jenkins: https://github.com/kubic-project/jenkins-library/pull/170

@flavio @kiall: We need to force merge this one. Passing on https://github.com/kubic-project/jenkins-library/pull/174


Enable configuring service port for loadblancer services

Istio is using loadbalancer service for tracing and istio-ingressgateway by default, and the default service port for those two services are 80.
The problem is that if there is already a service using port 80, the request to the tracing and istio-ingressgateway loadbalancer may be intercepted by services which is already using port 80.
Take my env as example, I have an ingress service running and the default-backend already using 80, and when I access tracing via loadbalancer service, it always report default backend - 404 which is actually returned by default-backend service.
[root@gyliu-icp-1 ~]# kubectl get svc -n istio-system | grep tracing
tracing                        LoadBalancer   <IP_ADDRESS>   <IP_ADDRESS>    80:31134/TCP                                                                                2d

[root@gyliu-icp-1 ~]# kubectl get svc -n kube-system | grep default
default-backend                               ClusterIP      <IP_ADDRESS>    <none>        80/TCP               6d

[root@gyliu-icp-1 ~]# curl <IP_ADDRESS>
default backend - 404

After I change the service port of tracing to a new port, the tracing loadbalancer works.

/cc @sdake @linsun @costinm @morvencao

I did not play with other loadbalancers, but only keepalived as loadbalancer in my cluster.
I think we need at least let customer can customize the port in order the resolve this issue if it happens.


chore: do not use executable tx to calculate hash

This change is

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#517 : 2  dependent PRs (#518 , #519 )
#657  👈
#636
#611
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ArniStarkware and the rest of your teammates on  Graphite

crates/gateway/src/gateway_test.rs line 96 at r1 (raw file):
Previously, MohammadNassar1 (mohammad-starkware) wrote…
please do not use clone

The original code also uses clones (it was just hidden inside rpc_tx_to_account_tx). This is only for testing.

This function will be deleted.


Appstudio update test-component-pac-eayd

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-eayd-on-pull-request has successfully validated your commit.


Disable inspection for starting a newline in a tabular after '&'

The inspection for starting a new line after each period should not trigger inside a tabular if an & occurs directly after the period.
Type of JetBrains IDE (IntelliJ, PyCharm, etc.) and version
TeXiFy IDEA version
Beta 0.5

Try updating the plugin to 0.6.
I cannot reproduce, do you have an MWE?
This works for me:
\documentclass{article}
\begin{document}

    \begin{tabular}{cc}
        stuff.& asdf \\
    \end{tabular}

\end{document}


How to use @GraphQLNonNull annotation in DTO model from library jar file

I am using graphql-spqr-spring-boot-starter and annotating @GraphQLApi and @GraphQLMutation @GraphQLQuery in the service layer. However, the DTO data models come from another library project. How can I define some filed as mandatory in this situation? Thanks.

You could implement a custom TypeMapper that detects the desired DTOs and wraps them with GraphQLNonNull.
E.g. for a DTO class such as:
public class BookDTO { .. }
You could a mapper like this:
public class BookMapper implements TypeMapper {

        @Override
        public GraphQLNonNull toGraphQLType(AnnotatedType javaType, Set<Class<? extends TypeMapper>> mappersToSkip, TypeMappingEnvironment env) {
            GraphQLOutputType wrappedType = env.operationMapper.toGraphQLType(javaType, singleton(getClass()), env);
            return GraphQLNonNull.nonNull(wrappedType);
        }

        @Override
        public GraphQLNonNull toGraphQLInputType(AnnotatedType javaType, Set<Class<? extends TypeMapper>> mappersToSkip, TypeMappingEnvironment env) {
            GraphQLInputType wrappedType = env.operationMapper.toGraphQLInputType(javaType, singleton(getClass()), env);
            return GraphQLNonNull.nonNull(wrappedType);
        }

        @Override
        public boolean supports(AnnotatedElement element, AnnotatedType type) {
            return type.getType().equals(BookDTO.class);
            // or GenericTypeReflector.isSuperType(BookDTO.class, type.getType()) to support subtypes as well
            // You can also use the `element` to detect a specific method or class where this type is being used,
           // in case BookDTO should only sometimes be non-null
        }
    }

Since you say you're using Spring, you can register the custom TypeMapper like this:
@Bean
public ExtensionProvider<GeneratorConfiguration, TypeMapper> customMappers() {
       // Order of mappers matter, but a generally good place to insert customizations is after `IdAdapter`
        return (config, mappers) -> mappers.insertAfter(IdAdapter.class, new BookMapper());
}

Sorry for the long response time, the project was on a bit of a hiatus.


Not Found After Successfully Generated

The requested URL /php/CRUDAG/admingenerator/web/barang was not found on this server.
barang is one of my tables in my database. I have successfully generated admin page, but I cannot see the data list page and add data page. How to solve it? Should I insert some data first to my tables?

This is a nice project but I think the developer has stopped working on it. Please try this new project: https://github.com/Muhammadinaam/speed-admin


Questions about query blocks

How to query block information through API

The explorer webapi is documented at: https://cardanodocs.com/technical/explorer/api/
However, I would advise against using the explorer in this repo (due to known bugs) and instead use the explorer in the https://github.com/input-output-hk/cardano-explorer/ respository which was coded to match the explorer documentation above as closely as possible.

@erikd How to create addresses offline, sign some examples of broadcast transactions offline, preferably in Java, thanks.

In the cardano-node repository there is a CLI that allows the creation of a transaction (yes it works offline).
In the cardano-explorer there is an exectuable that allows a pre-created transaction to be published to the network.


Update content scope scripts to version 4.22.4

Task/Issue URL: https://app.asana.com/0/488551667048375/1204964860447173/f

Automated content scope scripts dependency update

This PR updates the content scope scripts dependency to the latest available version and copies the necessary files.
If tests have failed, see https://app.asana.com/0/1202561462274611/1203986899650836/f for further information on what to do next.

[ ] All tests must pass

test


Crash while doing specific things

I will get straight to the point. I was deleting 31 videos of a total size of around 15gb and right after deleting I got a notification that the app crashed due to an unknown issue or something of that sort. Another instance was keeping the app idle, after around 20 mins of the app being idle it stops responding and crashes soon after
App version - <IP_ADDRESS>

@GiornoJJ thanks for the report. Please upload the debug files.

@GiornoJJ thanks for the report. Please upload the debug files.

debug.txt
debug_fulltrust.txt
Are these the files?

@GiornoJJ thanks! Is this still an issue in v2.0.34?