Waku Sync

Waku Sync
In the Waku family of protocols Relay is responsible for the dissemination of messages but cannot guarantee delivery. The push based propagation (PubSub) of Relay can be augmented with a pull based method, in our case Sync. It's purpose is to help with network wide consistency of the set of messages stored. The Store protocol is also related as it is used to query peers for messages.
Protocol
The Sync protocol will be used to merge the set of messages two Store nodes posses. As not all nodes are interested in all messages it is important for Sync to take into consideration different attributes; content topics, timestamps, shards and message hashes. Messages transmission can be left to Store, synchronizing hashes will suffice. Sync is also not concerned with how messages are stored or their validity, syncing with trusted peers is assumed.
Solutions
The basic requirement is to find the differences between 2 sets of message hashes. In addition to this, limiting this operation to messages with certain attributes would fit common use cases where only a sub-set of message hashes need to be synced. The bandwidth, number of requests and responses and the efficiency of the computation are all factors to consider but we can expect network latency to be the biggest performance factor
Prolly Trees
At the highest level, a Prolly tree is an immutable, ordered collection of either keys and values or keys only (a set). The tree is constructed in a similar fashion to Merkle trees but does not care about order of operation. The same elements will always result in the same tree. There is currently no known general purpose Prolly tree implementation we could use.
Tree Structure
Starting at the leaves, all the elements are first ordered by theirs keys and then feed into a chunking algorithm. This algorithm decide the size of each leaf nodes by selecting an element as the boundary. All the boundary elements are then ordered again and feed to the algorithm. This process is repeated for each level of the tree until only one node remains, the root. To traverse the tree, non-leaves node elements store the hash of the node at the lower level they are a boundary of.
The number of elements a node store is probabilistic and varies based on the choice of chunking algorithm and it's parameters.
E.G.
lvl 2 AQ -> each element store the hash of a node below
lvl 1 AEHM, QUX -> store hashes of the nodes below
lvl 0 ABCD, EFG, HIJKL, MNOP, QRST, UVW, XYZ -> each element store values
Set Operations
To efficiently find differences between 2 trees, the keys order and nodes hash equivalence are used to determine which path to follow or not. Starting at the root if 2 elements have the same hash then they can be ignored. If the hash differ, elements of nodes at the lower level have to be checked recursively. The elements are checked in order and are added to the differences if they are not shared between the 2 trees.
References

https://www.dolthub.com/blog/2020-04-01-how-dolt-stores-table-data/#prolly-trees
https://www.dolthub.com/blog/2020-06-16-efficient-diff-on-prolly-trees/
https://github.com/ipld/ipld/blob/prolly-trees/specs/advanced-data-layouts/prollytree/spec.md

Future Work
Prolly trees can be used to search vast amount of data efficiently. With this technology we could build a database that is proveable, replicatable and decentralized. See this experiment.
Range Based Set Reconciliation
A good way to think about RBSR is to imagine an iterative binary search between 2 peers. Keys and values are first ordered and then ranges of elements are hashed and exchanged between the peers. Identical ranges are ignored and different ones are sub-divided into smaller ranges recursively. Current implementations includes C++, JS and Rust. JS and Rust only support in-memory vectors as storage with C++ adding B trees and LMDB.
Divide and Conquer
Actual implementation can split ranges into more than 2 sub-ranges. It allow smaller payload to be sent but increases the number of round trips between 2 peers or more ranges can be sent, resulting in a bigger payload with fewer round trips. Ranges don't have to be identical either, if payloads are limited to a certain size, ranges can stay under the limit imposed.
Hashing
Incremental hashing is used, instead of recomputing the hash of a range each time, the hash of each elements are XOR together. This method makes inserting or removing elements more efficient when using a tree-based storage. Hash the element to insert then XOR with the other hashes. Deletions are done by hashing the elements, reversing the bits then XOR with other hashes.
References

https://logperiodic.com/rbsr.html
https://github.com/hoytech/negentropy
https://github.com/hoytech/negentropy/blob/master/cpp/README.md
https://www.symas.com/lmdb

Future work
Multiple ranges can be combined to sync specific sub-sets of elements. See this experiment. Set reconciliation could be done with more than 3 "dimensions"
Prollies VS RBSR
State
To sync 2 prolly trees a snapshot of both side needs to be maintained while the sync is ongoing. This is made easier by the fact that prolly trees are immutable but it also means that more state needs to be stored. Some kind of garbage collector would need to clean up old version of the local tree that are no longer in use. Updating the local tree would also require a message buffer so that updates are done on batches of new messages instead of every new message for efficiency.
This delay between new message being added and synced is not a problem for RBSR. While the set reconciliation is taking place new messages can be added to the underlying storage. If any new messages fall in a range that is yet to be sent then they will be synced the same way as if they were there before the reconciliation started.
Also, Prolly trees are only probabilistic balanced but for RBSR the tree storage can be rebalanced and packed because it is not part of the protocol.
Multi-party Sync
Both method could sync with multiple peers at the same time.
For RBSR, a client would sends the initial payloads to multiple "servers" and then respond to each server payload independently. Care must be taken if one wants to differentiate which "server" has which hashes, if not a single list of missing hashes is populated.
For Prolly trees, every "server" would walk their own tree and send the differences to the "client". In turn the client would have to remember each "server" differences and walk each path separately. Because the trees are immutable each sync would be independent from each other.
Overall, RBSR multi-party sync is simpler because the state required to track each sync "session" is less.
Ease of development
RBSR are implemented in the form of the Negentropy protocol. Some work would be needed to adapt it to our needs but much less than Prolly trees as no implementation is available.

As not all nodes are interested in all messages it is important for Sync to take into consideration different attributes; content topics, timestamps, shards and message hashes.

I would focus for now on a simplifying assumption that Sync is between two nodes that already "know" they want to reconcile all cached message hashes - in other words, sync everything. Not only is this how Sync will first function (with the exception of perhaps a sync list per shard), but it focuses our initial efforts into a building block that can be adapted for all these use cases.

Sync is also not concerned with how data is stored

Well, it does assume a key-value store with unique addressable keys, message hashes in our case.

instead of recomputing all hashes each time, the hash of each elements are XOR together.

But when a diff is found the fingerprint for the "dynamic" subranges would have to be recomputed each time, or am I missing something? This remains my main concern for RBSR scalability if frequently syncing with multiple peers for an ever-growing data set. Not a showstopper, but scalability may be something to model or explicitly verify in experiments.

Overall, RBSR multi-party sync is simpler

In our case, assuming regular syncing, most reconciliation would happen on the "right side" for the most recent messages. Presumably this has approximately equal impact on RBSR and Prolly Trees (?), but worth keeping this in mind when considering options.

RBSR are implemented in the form of the Negentropy protocol

If this is a simple(ish) integration, this would already be good enough reason to build POC using RBSR above Prolly Trees IMO.

I would focus for now on a simplifying assumption that Sync is between two nodes that already "know" they want to reconcile all cached message hashes - in other words, sync everything. Not only is this how Sync will first function (with the exception of perhaps a sync list per shard), but it focuses our initial efforts into a building block that can be adapted for all these use cases.

:100:

Sync is also not concerned with how data is stored

Well, it does assume a key-value store with unique addressable keys, message hashes in our case.

RBSR can function well without a DB.  I'll rephrase, I meant in the sense that Sync only deals in hashes not messages themselves.

But when a diff is found the fingerprint for the "dynamic" subranges would have to be recomputed each time, or am I missing something? This remains my main concern for RBSR scalability if frequently syncing with multiple peers for an ever-growing data set. Not a showstopper, but scalability may be something to model or explicitly verify in experiments.

Messages are already hashed. At no point hashing occurs. The "hash" of a range is the XOR (plus hardening) of all the elements.

In our case, assuming regular syncing, most reconciliation would happen on the "right side" for the most recent messages. Presumably this has approximately equal impact on RBSR and Prolly Trees (?), but worth keeping this in mind when considering options.

Yes, good point! RBSR can limit the sync to the most recent messages as well as Prolly trees.

RBSR are implemented in the form of the Negentropy protocol

If this is a simple(ish) integration, this would already be good enough reason to build POC using RBSR above Prolly Trees IMO.

I estimate that ~4-6 months of work would be required to make a Prolly tree Sync. If we use Negentropy 3-4 weeks might be enough.

This method makes inserting or removing elements more efficient when using a tree-based storage. Hash the element to insert then XOR with the other hashes. Deletions are done by hashing the elements, reversing the bits then XOR with other hashes.

In case of Waku sync, deletion would not be a use-case right as only stored messages are being synced across and a node will only ever be missing or having additional messages. In a case is missing it needs to sync those from peer, and if it is having extra the peer needs those messages.
Or do you see some scenario where something already existing has to be deleted during sync?

deletion would not be a use-case right

Correct. No deletion (only removing oldest entries based on dimensioning) and no modification of entries.

I've been thinking of adding a new protocol. Let's call it StoreSync. it would be useful in the case when Sync returns have hashes. ATM, we can't send or tell the other node about those. Also, we can't negotiate sync range or frame size between nodes.
The new protocol would be used for this. The idea would be to send our desired max frame size and sync range then wait for a response. We then compare the response with ours and pick the smallest of the values and initiate sync. Once Sync is done we send the have hashes to the other node if any, then we request our need hashes via Store from the other node.

it would be useful in the case when Sync returns have hashes. ATM, we can't send or tell the other node about those.

Not sure I follow - isn't this exactly what negentropy sync does? return a diff list of message hashes?

we can't negotiate sync range or frame size between nodes.

Perhaps sync range can be a network specification, but indeed - we probably need two nodes to agree on the exact time range to sync for this to work. However, this sounds to me like an extension of the Sync protocol itself rather than a new protocol? (I.e. a wrapper and extension on negentropy)? What do you mean by "frame size"?

Not sure I follow - isn't this exactly what negentropy sync does? return a diff list of message hashes?

Yes but there's 2 lists; hashes I want, hashes the other node needs.

Perhaps sync range can be a network specification, but indeed - we probably need two nodes to agree on the exact time range to sync for this to work. However, this sounds to me like an extension of the Sync protocol itself rather than a new protocol? (I.e. a wrapper and extension on negentropy)? What do you mean by "frame size"?

Maybe just an extention would work and maxFramSize is the param that can limit the size of payloads.

Perhaps sync range can be a network specification, but indeed - we probably need two nodes to agree on the exact time range to sync for this to work. However, this sounds to me like an extension of the Sync protocol itself rather than a new protocol? (I.e. a wrapper and extension on negentropy)? What do you mean by "frame size"?

I too think this should be the approach where-in we just create a wrapper protocol called Sync inside which the negentropy payload is a field. Other fields can be  time range of sync, max-frame-size for now.
In future we can include parameters like shards and/or content-topics as well.
If this is the case i am wondering if the protocol sequence would be something like this:

Client sends a request to sync with params like maxFrameSize, timerange etc.
Server acks the params that it can support sync for these.
Client would then start the negentropy sync based on agreed params and then client-server would continue as they do now.

@SionoiS In this case, we may want the server side also to create a new negentropy instance for each sync request it handles. Because the frameSize is set during creation of negentropy instance (similar to how we are doing for each sync method call on client side). The storage would remain same across all instances.
But this may not be required to be implemented immediately, as the first usage wouldbe for TWN where we would expect this param to be hard-coded for the network.

Yes but there's 2 lists; hashes I want, hashes the other node needs.

But this doesn't need to be communicated separately right. During the negentropy sync procedure itself both client and server can know what hashes they need and the other have. This can be achieved by simple using the reconcile method that takes in have and needHashes on server-side as well. We don't need to communicate this separately from client to server.

I've been thinking of adding a new protocol. Let's call it StoreSync. it would be useful in the case when Sync returns have hashes. ATM, we can't send or tell the other node about those. Also, we can't negotiate sync range or frame size between nodes.

I think the StoreSync protocol should define something like below:

Sync protocol would be used for syncing between store nodes periodically and sync-time should not be more than x hours
Any node that falls behind more than x hours should not use the sync protocol, rather use historyQuery to fetch the messages and then in-turn start using sync protocol for periodic syncing. Note that once incentivization of Store protocol is implemented, then historyQuery would involve payment to the node from which history is being queried.
It can define parameters like how many store nodes to sync with periodically
rate-limiting sync requests to handle???

Yes but there's 2 lists; hashes I want, hashes the other node needs.

But this doesn't need to be communicated separately right. During the negentropy sync procedure itself both client and server can know what hashes they need and the other have. This can be achieved by simple using the reconcile method that takes in have and needHashes on server-side as well. We don't need to communicate this separately from client to server.

Are you sure? From what I can see of the code, only the initiator can use the reconcile with list as output. This restriction seams arbitrary but there's must be a good reason for it...

What I see is that if we want to verify the messages we receive we can't just use Sync and Store it fits either scope of responsibility. That is why I was suggesting a new protocol.
Maybe RLN-Relay could expose a proc that can be used to verify any messages? We can't verify them without RLN anyway.

What I see is that if we want to verify the messages we receive we can't just use Sync and Store, it fits either scope of responsibility. That is why I was suggesting a new protocol.
Maybe RLN-Relay could expose a proc that can be used to verify any messages? We can't verify them without RLN anyway.

I too agree we need a StoreSync protocol a layer above sync protocol. I tried to list down its functions here.
But I am wondering if Message verification can be part of Store protocol as any client requesting messages from a StoreNode can verify if needed. But then again, if a client(other than storeNode) is requesting, it can validate using application logic as well. So, maybe we can keep it restricted to StoreSync to start with.

Yes but there's 2 lists; hashes I want, hashes the other node needs.

But this doesn't need to be communicated separately right. During the negentropy sync procedure itself both client and server can know what hashes they need and the other have. This can be achieved by simple using the reconcile method that takes in have and needHashes on server-side as well. We don't need to communicate this separately from client to server.

Are you sure? From what I can see of the code, only the initiator can use the reconcile with list as output. This restriction seams arbitrary but there's must be a good reason for it...
Maybe we could try making some modification to the original code?

Will take a look at this today and confirm the behaviour.

Sync protocol would be used for syncing between store nodes periodically and sync-time should not be more than x hours

Yes, but the x in this case is also a factor of the chosen archive policy. If archive keep the last 30m then Sync can't be 1h.
We can either assume Store keep more than x hours of messages OR negotiate the time range before syncing.

Any node that falls behind more than x hours should not use the sync protocol, rather use historyQuery to fetch the messages and then in-turn start using sync protocol for periodic syncing. Note that once incentivization of Store protocol is implemented, then historyQuery would involve payment to the node from which history is being queried.

Yes when a Store node start it should check how long it's been offline. Maybe Store should be responsible for that?

It can define parameters like how many store nodes to sync with periodically. It can keep track of various sync scopes and periodicity for each of them in future (like which shards to be synced with which nodes etc).

Yes but this seams lower priority.

It would also manage negentropy storage i.e should be maintained on x time period basis and delete it.

Sync should talk to Store and keep the set of messages the same. On the technical side we should use 1 DB if possible. Having an in-memory storage for Sync seams not necessary. Let's store the negentropy tree in the DB instead.

It would also define when sync for a certain time-range should be initiated so that we don't duplicate work done at gossip layer where nodes ask each other for messages.

If gossips were reliable we would not have to create Sync. Work duplication seams inevitable?

It should also maintain some sort of a state as to what was the last point that was sycned(with mutliple nodes) i.e no need to sync a particular range etc.

Yes, but for now lets just sync with a random peer at certain interval.

rate-limiting sync requests to handle (Maybe this would be part of sync protocol itself)

To rate-limit Sync requests, preventing the same peer from touching the same time range would be a good thing. I feel, this and the one above should be the responsibility of Sync.
@chaitanyaprem WDYT?

For time range nego. I feel the simplest would be to always pick the smallest range.
As for the closing Sync payload, it should include the server "need" hashes.

@chair28980 This should fall under Sync epic, not the Store v3 message hashes.

Sync protocol would be used for syncing between store nodes periodically and sync-time should not be more than x hours

Yes, but the x in this case is also a factor of the chosen archive policy. If archive keep the last 30m then Sync can't be 1h. We can either assume Store keep more than x hours of messages OR negotiate the time range before syncing.

Agreed, it should definitely consider the archive policy and should not overshoot it.

Any node that falls behind more than x hours should not use the sync protocol, rather use historyQuery to fetch the messages and then in-turn start using sync protocol for periodic syncing. Note that once incentivization of Store protocol is implemented, then historyQuery would involve payment to the node from which history is being queried.

Yes when a Store node start it should check how long it's been offline. Maybe Store should be responsible for that?

I think this should be the responsibility of StoreSync. I am looking at StoreSync as a protocol that helps store nodes to be in sync with each other. So, it would indicate how Store nodes can be in sync considering various scenarios such as this one.

It can define parameters like how many store nodes to sync with periodically. It can keep track of various sync scopes and periodicity for each of them in future (like which shards to be synced with which nodes etc).

Yes but this seams lower priority.

Agreed, this doesn't need to be part of first iteration of StoreSync. But i see that this would be requried eventually.

It would also manage negentropy storage i.e should be maintained on x time period basis and delete it.

Sync should talk to Store and keep the set of messages the same. On the technical side we should use 1 DB if possible. Having an in-memory storage for Sync seams not necessary. Let's store the negentropy tree in the DB instead.

It would also define when sync for a certain time-range should be initiated so that we don't duplicate work done at gossip layer where nodes ask each other for messages.

If gossips were reliable we would not have to create Sync. Work duplication seams inevitable?

To start with maybe we can just let Sync run in parallel to gossip and see. But i see that this would just add redundant traffic in the network, which is why i think we should try to optimize/improve this so that Sync doesn't kick-in for messages that are in-flight due to gossipsub's own IWANT/IHAVE interaction. But then again, this is more of an optimization and can be taken-up in second iteration.

It should also maintain some sort of a state as to what was the last point that was sycned(with mutliple nodes) i.e no need to sync a particular range etc.

Yes, but for now lets just sync with a random peer at certain interval.

To start with this is fine. Maybe as part of next iteration we can include such capabilities.

rate-limiting sync requests to handle (Maybe this would be part of sync protocol itself)

To rate-limit Sync requests, preventing the same peer from touching the same time range would be a good thing. I feel, this and the one above should be the responsibility of Sync.
@chaitanyaprem WDYT?

I agree that rate-limiting can be part of sync protocol implementation itself.
But wrt maintaining state of sync with a particular peer, i think it is better to be part of a higher level protocol like StoreSync. Will think little more on this though.

It would also manage negentropy storage i.e should be maintained on x time period basis and delete it.

Sync should talk to Store and keep the set of messages the same. On the technical side we should use 1 DB if possible. Having an in-memory storage for Sync seams not necessary. Let's store the negentropy tree in the DB instead.

Did not get this. Do you mean when Store purges/deletes the messages they should be deleted from sync storage as well?
I don't know currently how that is done, but i think @Ivansete-status is working on changing this using partitions where-in instead of deleting each message once time limit crosses, a partition is removed. I was thinking we should follow something similar for Sync storage as well. i.e maintain storage separately for different time-ranges and keep deleting older storage once it crosses the time we want to sync.
Wrt storing sync data in-memory, i agree that we should switch to using LMDB since it is already supported.
Since Store/Archive uses postgreSQL, I don't think it is a good idea for the Sync to use postgres because Sync uses a BTree for which a key-value kind of database is more suited. So, we can start by using lmdb for storing sync data and at a later stage think of which db to use.  RLN also uses another DB to store its data (which is also a tree), maybe we can use a single DB to store both syncData and RLN (not sure but this needs some study).

For time range nego. I feel the simplest would be to always pick the smallest range.
As for the closing Sync payload, it should include the server "need" hashes.
1. client payload

time range start
time range end
max frame size

2. server payload (same as above)

3. client use the smallest range and initiate sub-range storage.

4. client create first payload with smallest frame size.

5. round-trips of negentropy payloads

6. client sync is complete and now send last payload containing server "needs" hashes if any then done.

7. server receive hashes it needs from the client if any and then done.

Agree with this flow, i too kind of envisoned something like this but with some differences.
We need to evaluate whether to use sub-range for syncing or maintain different storage(s) for different time ranges.
Will analyze the pros and cons of the approaches of using a subrange vs separate storage per time period for syncData.

I have given some thought about using SubRange vs using separate storages itself and i think it is best we follow the approach of using separate storages.
Below is the simplest approach using separate storages and its pros.
Approach:

Each node shall always create a new storage based on duration (e.g: maintain separate storage for every 1 hour). i.e a node shall keep creating new storage for let's say 12PM and 1PM etc.
When a node starts, it can sync with other nodes in the network to have the data for the current hour (e.g if a node is started at 12:10, it would sync with a peer to get messages from 12PM).
We can follow an approach similar to partition where-in the future storages are created in advance. Messages are stored in appropriate storage based on timestamp.

Pros of this approach:

A storage can be deleted once its appropriate archive partition is removed or if it crosses the threshold for sync. (There can be cases where a node wants to archive 30 days of data but only support syncing of recent 2 hours of data)
We can avoid usage of sub-range completely by this approach as the server side of negentropy already supports inherently subrange (based on initiator's selected range)
If we have to take a sub-range approach then, a single storage would be maintained and purging of older elements can only be done 1 by 1 which will be time-consuming and less performant.

One change we would have to make to the flow of the Sync protocol with this approach, where-in the range will always be based on 1 storage duration. This is under assumption that 2 nodes will not have to sync using a sub-range at all. This should be good to start with. Later we can enhance with below if required:

Allow a node to sync more than 1 range (i.e more than 1 storages or 1 hour of data)
Allow a node to sync less than 1 hour/storage of data (for this subrange would be required)

I had thought briefly of the alternate approach of using subrange and i see following challenges:

If we are maintaining single storage and use sub-range everytime for syncing, then deleting items from storage would be costly as each item has to be deleted separately. This would be a challenge as we need to know the hash and timestamp of each item to remove it. Currently the archive itself deletes items from DB with a single timerange query and not item by item.
It would lead to more complex flows to handle which may not be required to being with for storeSync.
In future, when we want to store based on pubsubTopic and/or contentTopic, this approach would not be useful as anyways separate storages would be required at that time. This is because sync would be done for specific pubsubTopic or specific list of contentTopics.

Hence let us proceed with simplified approach of having a mechanism to keep creating new storages and deleting old ones (once they go out of sync duration). We can restrict sync also to single storage size per session.

We can follow an approach similar to partition where-in the future storages are created in advance. Messages are stored in appropriate storage based on timestamp.

On this note, maybe we should also change archive paritioning logic to be in sync with this where-in partitions are created based on absolute hours of time such as 1PM , 2PM etc rather than relative to the node start time. Then we can get events from archive when a parition is being deleted, its associated sync storage can also be deleted.

I agree with the suggested approach. One more to add here, to collect some evidence based data on the distribution of messages during a day. If we can have such data we can fine tune the logic. My assumption is that it is rare that an application has a well distributed amount of messages (count and size) in a day. So there will be hours of partitions that will store much more messages than others. It means dropping such partitions will shrink the time based retention of messages more. Maybe we can have a better balance between the size vs. time retention. I think such a data collected we can have better suggestions for the operator for setting up the retention policy for their needs.

We are assuming that node have clock and are in sync. In what way things would break if that's not the case? I can't think of any but just to be sure. @chaitanyaprem

We are assuming that nodes have clock and are in sync. In what way things would break if that's not the case? I can't think of anything major but just to be sure. @chaitanyaprem

That is a valid point. Looking at how partitions were implemented i got tempted to move to an approach of creating storage as well in the same fashion.
To avoid this, we can follow the below alternate approach.
We don't have to rely on node's clock if we just use the messageTimestamp to determine and create a new storage based on date and hour of the day. That would shield from this scenario of node clock's not being in sync. But the only challenge i saw with this approach is that we are creating storages/partitions on the fly which should be synchronized from parallel execution. The nodes would also use start and end time of the storage to initiate sync as well.

Timestamp are not trustless. If we trigger any costly logic based on them we open ourselves to DOS attacks.

Timestamp are not trustless. If we trigger any costly logic based on them we open ourselves to DOS attacks.

Hmm, But since these message are already RLN validated, I am assuming we can trust them since they are meant to be validly published during that epoch for which a user has permission to send.
Maybe I am missing something, Do you see any challenge even after RLN verification?

I am assuming we can trust them since they are meant to be validly published during that epoch for which a user has permission to send.

Good point! I'll read more on RLN I don't know what it verify exactly wrt timing.

I've been thinking and talked with @jm-clius about the sub-range vs separate Negentropy storage and I'm not sure it's the best course of action anymore.
If we keep many separate storage we can't merge them and so will need to initiate multiple sync (one per storage).
We will always have a "roll-over" period where at least 2 sync must be done.
The cost of local computation for removing elements from the storage is small compared to the bandwidth and network latency of an extra sync.
I think we should use the sub-range functionality of Negentropy and remove old message hashes. A store query could be use to get the hashes and timestamps of messages older than 1H.

If we keep many separate storage we can't merge them and so will need to initiate multiple sync (one per storage).

Do we see a need to merge these storages? This is only used for syncing and if sync duration is going to be 1 hour, then why would these need merging?

The cost of local computation for removing elements from the storage is small compared to the bandwidth and network latency of an extra sync.

Considering the below explanation, would cost of removing elements be still small? As per negentropy spec, for 1 millino entries max-round trips is 3 to perform sync right. I am wondering if that would have soo much latency in comparison to removing all old entries.

The cost of local computation for removing elements from the storage is small compared to the bandwidth and network latency of an extra sync.

It is not just the cost of removing elements from storage, we need to identify which elements to be removed which means we need to know the list of hashes for all elements to be removed which would be an overhead. Hence, having separate storage would address this problem where-in the storage can just be removed.

Do we see a need to merge these storages? This is only used for syncing and if sync duration is going to be 1 hour, then why would these need merging?

No matter the time range of each storage, at some point the last 1h will have to cross into another storage no?
IMO it would be simpler to, at interval, do a store query for message hashes older than 1h and remove those from the storage.
It's all speculation at this point. I guess we could try both methods.

Am I right in my understanding that the main issue here is actually the difficulty of removing old items from a negentropy storage? My understanding is you basically need a list of timestamps and hashes to remove after which you can proceed to erase each item individually.

Yes or we could change the API Negentropy provide. The tree already store elements in order it would be trivial to impl. "erase element older than X" IMO.

creating new Negentropy storages dynamically

The cost is multiple sync and the logic to handle storage juggling.

creating SubRanges dynamically from existing Negentropy storage

The cost is either do a store query for list of timestamp and hashes to remove or impl a new prune function into Negentropy.

Unless we do some further secondary caching, I think we already know that we're likely to use SubRange as we never want to consider the latest messages (up to 20s in the past) during a sync. We expect some "real time" jitter for ~20s, while everything older than 20s can be considered settled.

Even if we don't do that syncing a couple of message extra is not a problem.

Is there a way to do a partial sync over a Negentropy storage other than SubRange?

No, sub-range is the intended solution.

If it's not feasible to create dynamic SubRanges

No way it would be that inefficient (famous last words).

I'd also be interested in what the cost is to create ad-hoc larger Negentropy storages for e.g. multi-hour sync requests by querying the archive DB.

It would take longer to query the DB than to build the tree IMO.

The tree already store elements in order it would be trivial to impl. "erase element older than X" IMO.

I too had this thought initially, but considering it is a B-Tree, I don't think it would be that trivial as removal of entries require rebalancing of the tree and I am not sure of pruning is going to be as simple.
From the first look at BTree implementation in negentropy, it looks like removal of elements will have to be 1 by 1 which then felt very similar to invoking erase from storage of all elements that we want to remove.
Also, another thing to note is once we move to DB backed storage such as lmdb, each node will have to removed separately which would be its own operation and i am not sure if there would be a simple way to prune the tree.
But, then again not sure if there is an algorithm to prune a B-Tree without removing elements 1 by 1. Could not find anything in a quick search. Will spend some more time again on this to see if there is any other alternative.

We could build our own storage impl. if the deleting of KVs is too costly. Some kind of growable ring buffer maybe?

We could build our own storage impl. if the deleting of KVs is too costly. Some kind of growable ring buffer maybe?

Each delete operation on B-tree is supposed to be O(logn) complexity. But i am worried if we delete 1 by 1 in sequence, it will just result in a lot of tree rebalances.
And considering our final solution will be to use a database (either lmdb or something else), having to delete entries would be more costly than maintaining multiple storages.
How about an alternative approach of mixed use of multiple storages and SubRanges. What if we maintain separate storage for a longer duration (Maybe few hours) and use Subranges for syncing using the latest storage? This would have following benefits:

We don't need to prune storages separately, rather just delete that specific storage which would be a single operation as it deletes entire tree.
The problem of 2 syncs will happen only at the boundary which would not be that bad.
It would also address the point @jm-clius was trying to make, i.e for longer term sync we can have nodes that maintain longer Sync state just because they have larger disk space.

Also, i was taking a relook at code of StorageManager (that rotates and manages storage) and it doesn't look too complex to maintain.

The problem is that the storage impl. is not adapted to our use case. The good news is that RBSR is agnostic to how elements are stored. Instead of mitigating the problem at a higher layer, let's actually fix the problem.
The first and easiest thing to impl. would be to delete elements 1 by 1 so that the POC is somewhat usable in a long running node (inefficiently it may be) soon. We then follow with storage rotation to mitigate the problem but we also work on a actual fix in the form of a new storage impl.
At that point, we could maybe benefit from swapping the C++ code for Rust or Nim since, we have to impl. a new storage and it would make interop easier too.

It does indeed seem like the storage implementation is too limited. However, if we can get the simplest POC solution implemented (even if it does a fairly expensive erase operation) we can evaluate the actual impact of the three avenues available:

fix/maintain/rewrite the storage implementation for our use
use some mitigating strategy, such as storage rotation (if this is significantly easier than (1) I think this is a reasonable approach)
discard negentropy and implement some other sync structure

RBSR is agnostic to how elements are stored.

Not sure if that is completely true. I mean from a protocol design it is.IIRC, it also considers a form of hashing that is tree-friendly so that by adding a single element in the tree doesn't result in recomputing hashes all again for whole tree.
We will have to design or come up with another data structure that fits this bill and not a tree-like if we have to support pruning (AFAIK all tree based data-structures would have same problem ).
The underlying persistence of the tree is flexible in the sense, we can easily store the tree in a postgres DB. I had noticed that this is going to be very easy job after going through the tree implementation. But changing the data-structure itself for storage is not going to be an easy task i think.

The first and easiest thing to impl. would be to delete elements 1 by 1 so that the POC is somewhat usable in a long running node (inefficiently it may be) soon. We then follow with storage rotation to mitigate the problem but we also work on a actual fix in the form of a new storage impl.

Agreed, and this would be easier done from top level so that any synchronization can be done at application (i.e in this case store sync level).

In the context of the recent focus on reliability, I've been thinking of more use cases for RBSR.
We could replace light push and reliability checks by syncing. Since we need to verify that the messages have been sent correctly, a sync session could be used to do the "push" and the verification at the same time. Node A would put a new message in it's own local cache then initiate a sync with node B. This would verify that the previous messages were sent correctly and also send the new message at the same time. Node B would have to keep a sub-set for each of it's light push clients.
The same logic and flow could also be use to replace the Filter protocol. A sync could be initiated from node B when it has received one or more messages that are interesting for one of it's light clients (IE part of their sub-set).
If we assume that nodes, light and "full" continuously sync with each other at opportune moments and each keep a set of the messages they are interested in, the same code can be used everywhere.
With the current impl. of Waku Sync it would be possible but awkward as it is not possible to efficiently create sub-sets, each cache would have to be managed on it's own and duplicate data. A better impl. would allow the creation of light weight sub-sets or to sync with set sorted by more than just time (probably content topic).
This would preserve the status-quo in term of privacy and increase bandwidth slightly.
@fryorcraken


rebar on a linux with non-smp kernel

I've recently had an issue on a cheap VPS which had kernel built apparently without smp support. rebar (rebar2) just hangs wor a while on such VPS, then fails with the 'cannot create aux thread' error or something like that. erl, built for smp kernels, fails with the same error, but it is still possible to start it with the -smp disabled command line option. Is there somethink like this for rebar? The only way to run it that i found is to use ERL_AFLAGS env.

You can (1) configure the environment to always have ERL_FLAGS="-smp disable" (you wrote disabled, but it must be disable) and (2) use a patched rebar binary, where you override the default escript_emu_args by adding {escript_emu_args, "%%! -pa rebar/rebar/ebin -smp disable"} in rebar/rebar.config before building. Perhaps you also need +A0.

The suggestions will work with Erlang older than 20, but going forward the non-SMP emulator has been removed. Therefore, I think this can be closed.

Hi, this issue was closed in an attempt to do quick basic filtering. All tickets prior to March 2017 were closed and will be reopened on a per-request basis, please give rebar3 a try and feel free to reopen if it's a pressing issue that is still unresolved.


Only output db_debug in tests when env variable is set

Normally the sqlite debug output is not very useful but included in every failing test. This sets the output for the tests depending on an environmental variable. If you want to include it you can set the env variable before running the tests.

For the CI, shall this variable exist and set to True as default? for eample, for azure, shall it be configured in our "pipeline" so that it can be also manually changed when manually queueing a build?

I'd say "yes" and "yagni".

I would argue that it should be set to False by default also in CI. I cannot think of a single time this has been useful but I can think of lots of times where it actively made it harder to figure out what is going on

I yield and agree make it "no" everywhere by default.


[Event Request] Table 7001 "Price List Line" - OnBeforeCopyFilteredLinesToTemporaryBuffer

Describe the request
Hi, would it be possible to get the OnBeforeCopyFilteredLinesToTemporaryBuffer event in table 7001 (Price List Line), like this:
    procedure CopyFilteredLinesToTemporaryBuffer(var TempPriceListLine: Record "Price List Line" temporary) Copied: Boolean;
    var
        IsHandled: Boolean;
    begin
        IsHandled := false;
        OnBeforeCopyFilteredLinesToTemporaryBuffer(Rec, TempPriceListLine, Copied, IsHandled);
        if IsHandled then
            exit;

        if FindSet() then
            repeat
                TempPriceListLine := Rec;
                OnCopyFilteredLinesToTemporaryBufferOnBeforeInsert(TempPriceListLine);
                if TempPriceListLine.Insert() then
                    Copied := true;
            until Next() = 0;

        OnAfterCopyFilteredLinesToTemporaryBuffer(TempPriceListLine);
    end;

    [IntegrationEvent(true, false)]
    local procedure OnBeforeCopyFilteredLinesToTemporaryBuffer(var TempPriceListLine: Record "Price List Line" temporary; var Copied: Boolean; var IsHandled: Boolean)
    begin
    end;

Additional context
Customization requires to implement calculation depending of Sales Setup
Internal work item: AB#523892

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 18389.


Three.js r119

Updating from r117 -> r119. Mainly differences in ACESFilmic tone mapping, but it seems to be generally bringing us slightly closer to the other renderers, so good enough. There is also a bug in the new WebGL2Renderer, probably with the RoughnessMipmapper that means I had to switch us to WebGL1Renderer for now. I need to upstream a fix into three.js that we can allow WebGL2.

FYI @mrdoob


Fix flaky test of TestRestartStoppedContainer (#21211).

This fix addressed the issue of test TestRestartStoppedContainer in #21211. Inside the test, a docker restart command is followed by a docker logs command. However, docker restart returns immediately so there is no guarantee that docker logs will wait until the restarted container completes the command echo foobar.
This fix use the check of {{.State.Running}} = false to make sure that the restarted container has already finished, before invoking the docker logs command.
This fixes #21211.
Signed-off-by: Yong Tang<EMAIL_ADDRESS>

the error in windowsTP4 is legit:
05:02:21 ----------------------------------------------------------------------
05:02:21 FAIL: docker_cli_restart_test.go:13: DockerSuite.TestRestartStoppedContainer
05:02:21
05:02:21 docker_cli_restart_test.go:25:
05:02:21     c.Assert(err, checker.IsNil)
05:02:21 ... value *errors.errorString = &errors.errorString{s:"condition \"\"true\" == \"false\"\" not true in time"} ("condition \"\"true\" == \"false\"\" not true in time")
05:02:21
05:02:21
05:02:21 ----------------------------------------------------------------------

@calavera Guess the issue is because of the long timeout in Windows. I increased timeout to see if the issue persists.

It seem like the test passed with added timeout (the test takes 23s on Windows to finish).

LGTM

yay, it's green now
LGTM


pressing the 'r' key resets the fox back to spawn in case of emergency

players can get stuck, 'r' key should reset them back to spawn

won't do


docs: make sidebar nav sticky w/ accordion sections

"Getting Started" is considered the default section and
auto-expanded on landing page
Otherwise, current page's section is pre-expanded, others
are collapsed by default
Sticky and vertically scrolls if still overflows
CSS-only, no Javascript

It MIGHT make sense to throw in some progressive enhancement JS to scrollTo() when expanding a section 🤔

WIP but opened PR so @SurbhiSGupta can see Netlify preview and @lizzthabet can help me make sure it's accessible

Superseded by #906 (collapsible) + #907 (sticky)


First time IntelliSense trigger occasionally takes much longer time than expected

Closed with the new runspace in place.


New version: Zoom.ZoomOutlookPlugin version <IP_ADDRESS>2

Checklist for Pull Requests

[ ] Have you signed the Contributor License Agreement?
[ ] Is there a linked Issue?

Manifests

[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Hi @PckgrBot,

This PR's version number <IP_ADDRESS>2 has 4 parameters (sets of numbers between dots - major, minor, etc), which is greater than the current manifest's version 6.0.0, which has 3 parameters.

Does this PR's version number exactly match the version reported in the Apps & features Settings page? (Feel free to attach a screenshot.)
(Automated response - build 876)


Test rework, part 4

This is the next part of the test rewriting project.
Background and motivation:
See:

#3418
#3455
#3464

and the initial commit message from https://github.com/containerd/nerdctl/pull/3418/commits/2c95d090237c534b9ddec59b6488386445c41da6
along with the linked issues from above PRs.
The gist of it is really simple: our testing situation is not good (see prior discussions for full list of issues - but then, the sheer fact we have to retry our CI multiple times to get a green tree should say enough).
This here is the part 4 of the effort to rewrite our tests using a new, custom developed testing toolkit that emphasizes test expressiveness, debugability and better isolation.
Changes
Individual commits messages have extensive details about the changes where appropriate.
Status
a. the following has been ran locally 100x+ with no failure:

. (< 1 sec)
completion (< 1 sec)
issues (< 10 secs)
system (~ 5 secs)
volume (~ 5 secs)
network  (~ 5 secs)

although, #3500 did show up and might still be lurking (and #3501, albeit this one probably got fixed by #3508)

b. the following have ran locally 20x+ with no failure:

ipfs (~ 100 secs)

c. the following are failing after about 20 iterations:

image (~40 secs)

likely due to one of the variants in #3513 (the one with images --filter or the one using a platform reduced push)

d. the following are NOT part of this PR:

builder
container
compose
login

Also left out of this PR: disabling retries for non flaky tests.
We do now have the infrastructure for that (eg: -test.only-flaky) but this can wait.

Review
Close to 10k lines is indeed a chunky PR, but then, as with the previous parts, there is absolutely no "code" or "logic" changes in nerdctl itself - this is "just" tests: if they run, and if the CI is green...
PR has been split in as many commits as feasible to facilitate review, along with a number of inline comments in the github UI where appropriate.
Let me know if I can make this easier in any way.
The north star for this whole effort remains:

we must get to a consistently green CI without retries - with no or very few flaky test.
go test -p 1 ./cmd/nerdctl/... must work locally out of the box on developers laptop without randomly breaking for missing requirements

@AkihiroSuda this is not completely green yet.
There is likely one more adjustment needed to get it there.
Also, I would prefer to get #3499 in first - and I am likely to land a few small fixes as well for issues spotted while stress-testing (specifically around the digest not found cropping back up).
Nevertheless, given this here is a bit chunky, I think it is fair to get a head start to review asap.
At your convenience as usual.
Thanks.

@AkihiroSuda green except:

the windows reg
arm64 <- can you poke it?

Needs rebase

Needs rebase

Rebased.
I am giving up (temporarily) on the missing content bugs - so, no upcoming PR wrt to that problem for now.
Let's just mark flaky whatever test is going to show this particular problem for the time being.

Mildly concerned about the repeat failure on arm - annoying part being it is a retry/timeout, hiding the actual issue :s

There is clearly a deadlock happening in image tests, but only for arm64.
Debugging right now - marking draft until I find the culprit.

Fixed.

Thanks, I'm merging this, but this amount of PR was almost unreviewable and even causing Safari to almost hang up 😓

@AkihiroSuda I really appreciate your help and support on this.
I could not have done it otherwise.

In the next series of the PR please consider splitting the PR to smaller parts.

Upcoming parts are roughly:

github actions clean-up with non-flaky/flaky(with retry) split up - this will give us a lot of bang for the buck and likely a good speed-up of the total testrun time
migration of container tests
migration of builder tests
migration of login tests
migration of compose tests

I think the test framework is mature enough now that we can have a separate PR for each of these ^ - making it much less painful for everybody (me included 😓).

Also consider reducing the amount of the helper functions and constants (see the comments)

Eventually all of it will fuse into one - rn I have to make sure existing tests still work unaltered, and the new ones are not constrained... It is tough. I almost gave up on the effort multiple times 😰. Debugging some of these issues was mind altering.
Thanks again!
I am absolutely convinced we will be in a much better situation soon, and the quality of 2.0 will rock!


Doesn't work with Chartjs v2.7.0

I've tried it with v2.7.1 and it also doesn't work. The dependencies for the plugin is "chart.js": "2.6.0"... so it's not surprising.
Looking at the changelog from v2.6.0 to v.2.7.0 and there are quite a lot of changes: https://github.com/chartjs/Chart.js/releases
Hoping an update is in the works.

v1.2.0 supported Chart.js 2.6 only. Today we released v1.3.0 which supports Chart.js 2.7 and you will see that the issue is fixed.


Game gets hung up on walking

[20:06:31] Now at Pokestop: Dinosaur Topiary
[20:06:31] Spinning ...
[20:06:32] Loot:
[20:06:32] 50 xp
[20:06:32] - 2x Pokeball (Total: 3)
[20:06:32] - 1x Ultraball (Total: 87)
[20:06:32] PokeStop on cooldown. Time left: 5 minutes, 16 seconds
[20:06:38] -- Recycled 3x Pokeball (keeps only 0 maximum)
[20:06:38] -- Recycled 1x Potion (keeps only 0 maximum)
[20:06:44] [x] Next egg incubates in 1.00 km
[20:06:51] Something rustles nearby!
[20:06:52] A Wild Weedle appeared! [CP 122] [Potential 0.2]
[20:06:52] IV [Stamina/Attack/Defense] = [4/3/2]
[20:06:56] Using Greatball (chance: 55.28%)... (142 left!)
[20:06:56] Captured Weedle! [CP 122] [Potential 0.2] [4/3/2] [+110 exp]
[20:07:01] Moving towards fort f83cfce12d4a4654afa61cf99a35fba4.16, 0.05km left
[20:07:11] Moving towards fort f83cfce12d4a4654afa61cf99a35fba4.16, 0.04km left
[20:07:39] Now at Pokestop: Whooping Crane Mozaic
[20:07:39] Spinning ...
[20:07:39] Loot:
[20:07:39] 50 xp
[20:07:40] - 1x Pokeball (Total: 1)
[20:07:40] - 1x Potion (Total: 1)
[20:07:40] - 1x Super Potion (Total: 4)
[20:07:40] PokeStop on cooldown. Time left: 5 minutes, 16 seconds
[20:07:42] Moving towards fort 792270782f08406ca6571a84200cd8dc.16, 0.05km left
[20:07:46] -- Recycled 1x Pokeball (keeps only 0 maximum)
[20:07:46] -- Recycled 1x Potion (keeps only 0 maximum)
[20:07:56] Now at Pokestop: Wetland Overlook
[20:07:56] Spinning ...
[20:07:56] Loot:
[20:07:56] 50 xp
[20:07:57] - 3x Pokeball (Total: 3)
[20:07:57] PokeStop on cooldown. Time left: 5 minutes, 16 seconds
[20:07:59] Moving towards fort 5d6245e1af014456ad869c625ea842ec.16, 0.07km left
[20:08:02] -- Recycled 3x Pokeball (keeps only 0 maximum)
[20:08:20] Moving towards fort 5d6245e1af014456ad869c625ea842ec.16, 0.08km left
[20:08:49] Moving towards fort 5d6245e1af014456ad869c625ea842ec.16, 0.10km left

is there anyway to start the app fresh like clearing the cache to start it stable again?

PokeStop on cooldown. Time left: 5 minutes, 15 seconds
[20:15:53] Moving towards fort a3edd5483a05455aaadf537b46404b22.16, 0.06km left
[20:15:57] -- Recycled 2x Pokeball (keeps only 0 maximum)
[20:16:11] Moving towards fort a3edd5483a05455aaadf537b46404b22.16, 0.05km left
[20:16:31] Now at Pokestop: Gathering Place
[20:16:31] Spinning ...
[20:16:32] Loot:
[20:16:32] 50 xp
[20:16:32] - 2x Pokeball (Total: 2)
[20:16:32] - 1x Greatball (Total: 140)
[20:16:32] PokeStop on cooldown. Time left: 5 minutes, 16 seconds
[20:16:34] Moving towards fort 31f1402009b64e30910c522603a0e66d.16, 0.11km left
[20:16:38] -- Recycled 2x Pokeball (keeps only 0 maximum)
[20:16:51] Moving towards fort 31f1402009b64e30910c522603a0e66d.16, 0.12km left
[20:17:28] Moving towards fort 31f1402009b64e30910c522603a0e66d.16, 0.15km left

Looks like a duplicate of #1885.


Cannot export data

Hello! I have run into a problem with DailyTxT. When trying to export data it just simply doesn't
I have tried:

Restarting docker contrainer (assuming it's done in docker restart dailytxt way)
Logging out and logging in
Different browser
Do it on mobile (in fact I left it "trying" to export for a good 5-7 minutes no reaction.)

I am not really sure what to do other than contact here. I feel really unsafe without a backup. Don't wanna lose all my entries you know. Thanks in advance!

Do you still see all your postings of each day? (So are you correctly logged in?)

The Export-time strongly depends on the amount of uploaded files! If you only have text, then the export is instant. If you have just uploaded a few files (like about 10 Photos), then it will already take several seconds. Having a lot of uploaded Photos/Files might actually take several seconds.
Sadly there is no possibility to get any progress-bar of the export at the moment...

As long as you see your postings everything is fine and your data is not corrupted! In this case it is probably 'just' an error in the export. But in this case you can simply "backup" your data by making a backup of the data-volume/-folder and make sure to save the user/password-combination

It really is just 10 (a bit more) photos and videos there. Seems like some error. I'll follow your 3 recommendation to backup the folder. :/

Would be really nice to somehow report the bug if your app had some way to view an error dump

Sadly there isn't a really good possibility to view an error dump.
The only two ways to get a little bit more information are probably the following:
a) Take a look at the browser "console" and the "network analysis" tabs while exporting. Perhaps there are more details visible in the POST-response when trying to export.
b) You could try to view the docker logs. Either by docker logs dailytxt or by connecting into the running container (to do this first run docker exec -it dailytxt /bin/sh and then view the nginx-logs e.g. by running cat /var/log/nginx/error.log).

And secondly, for test-reasons you could perhaps try the following:

Make a backup of your data-folder!
Download and delete (!) the few files that you uploaded. Especially the video-uploads. (I'm wondering a bit that you uploaded video-files. How big are those files? The maximum-size of uploads should be limited to 128 MB.)
Try to make an export again (now without the files). If it's working, then you know that the files produced the errors.

In case you did those steps: please post the results here, since I'm interested in the reason of your error as well...

Errors and traceback
Unfortunately I didn't get really valued errors here
Console did tell me a bit here's what happend:
xhr.js:184          POST http://(my website IP)/api/exportData 500 (INTERNAL SERVER ERROR)
Error: Request failed with status code 500
    at t.exports (createError.js:16:15)
    at t.exports (settle.js:17:12)
    at f.onreadystatechange (xhr.js:69:7)

(couldn't copy it)
sudo docker logs dailytxt returns nothing.
NGINX log has a lot of (probably not helpful) warnings. I assume it happens when I try to download many files at once?
[warn] 20#20: *6245 an upstream response is buffered to a temporary file /var/lib/nginx/tmp/proxy/5/03/0000000035 while reading upstream, client: (MY IP), server: , request: "POST /api/downloadFile HTTP/1.1", upstream: "http://<IP_ADDRESS>:5000/api/downloadFile", host: "(SERVER IP)", referrer: "http://(SERVER IP)/"

Another "unique" thing is this:
[emerg] 21#21: io_setup() failed (38: Function not implemented)
Testing
I have tried exporting data with no files and nothing happened
Testing returned same errors in console (browser console)
There were no errors in docker logs nor nginx error logs.
My idea of the error

There's a possibility it is a problem in my processor on my server. (aarch64)
Since it refused to run because of my processor I used a "hacky?" way to run it using  --platform=linux/amd64 in my pleaserun.sh script file
According to dailytxt docker page https://hub.docker.com/r/phitux/dailytxt/tags it only supports amd64 while I have aarch64 (lscpu)
Not sure if it's source of the problem
My writing in dailytxt is NOT english and it somehow errors?

That's pretty much it what I could find
I hope this helps a bit. I'll stick to copying my data folder in case something terrible happens

Thanks very much for your detailed answer!
Actually directly the first information says 'enough', since it's an error 500, which means, that it is an internal error on the server and so it finished the export with an error. Sadly at this point it's still not possible to get a more detailed idea, what exactly is the reason for this error.
Normally I would now presume, that It's a bug in dailytxt. But thanks for your information about your processor architecture! Since exactly this export needs more interaction with the filesystem and the OS it is absolutely possible that this is the source of your problems!
Dailytxt is definitely just made to run on amd64 and not on ARM... I have no support for other platforms!
The only way that I see: you could try to create you own docker-image for your platform... Clone the repo and create the image by running docker build .
The language you write in is not important! You probably just mean the syntax-checking? You could try to disable it (some browsers allow this)

The only way that I see: you could try to create you own docker-image for your platform... Clone the repo and create the image by running docker build .

Oh no not again. I guess I'll just resort to copying the folder for now. That thing is torture!! I built the image, tagged it and docker can't find it anyway :/ I'll pass
Little question: When running the container with docker run there's a variable SECRET_KEY with a bunch of random characters. Does it play a role in accessing my data? (So I would need to back it up aswell)

Well, aarch64 is probably not a lot of fun...
No, the SECRET_KEY is only necessary to be able to stay logged in to your current session when restarting the docker container! You can always access your data as long as you know your username+password (and/or backup-keys)!


When will The figma tokens support the Gogs Git synchronization

I'll be closing this one. Please provide a description, not just the title and reopen.


inferno-compat need "import Inferno from 'inferno'

I tried the inferno-compat package and it's better, but I have to add this a line at the beginning of each file.
import Inferno from 'inferno'
import React from 'react';
...
...
Without the import Inferno I have this error : Uncaught ReferenceError: Inferno is not defined
I installed 'inferno-compat' and changed the webpack.config.js to add the resolves section. Do I have to do something else ?
Thanks

@vinej I believe this is because you're using the babel-plugin-inferno, I've updated inferno-compat in the dev branch and it should now also include the dependency functions. You will need to alias inferno -> inferno-compat too in your webpack/browserify config.

I have updated to 0.7.17, but I still have an error with 'inferno-compat' with the 'inferno-router'
ERROR in ./~/inferno-compat/inferno-compat.js
Module not found: Error: Cannot resolve module 'inferno-compat/dist/inferno-compat' in D:\git\inferno-remux\node_modules\inferno-compat
 @ ./~/inferno-compat/inferno-compat.js 3:17-55

ERROR in ./~/inferno-router/inferno-router.js
Module not found: Error: Cannot resolve module 'inferno-compat/dist/inferno-router' in D:\git\inferno-remux\node_modules\inferno-router
 @ ./~/inferno-router/inferno-router.js 3:17-55

I tried to use 'inferno-router' as is (no alias), but I also have an error
I tried to use an alias for the router but none is working
'react-router' -> 'inferno-compat'
'react-router' -> 'inferno-router'
'inferno-router' -> 'inferno-compat'
Is 'inferno-router' compatible with 'inferno-compat' ?

I don't have experience with those packages, but I think it would help if you can provide "empty" github repo with the setup you are using.

You can use the repo https://github.com/vinej/inferno-remux
This repo is working. To test for the bug:

remove the lines : import Inferno from 'inferno';  at the beginning of each component
add the alias 'inferno' : 'inferno-compat' as stated in the inferno documentation

alias: {
'react': 'inferno-compat',
'react-dom': 'inferno-compat',
'inferno': 'inferno-compat'  // for usage of babel-plugin-inferno
}
The repo is not working anymore : todos.js:4 Uncaught ReferenceError: Inferno is not defined
Thanks

Please check https://github.com/trueadm/inferno/blob/master/packages/inferno-compat/README.md
I've updated the README to include a better config. Remove this line:
'inferno': 'inferno-compat' // for usage of babel-plugin-inferno
Also you can update inferno-compat to the latest version.

It's working now,
Thanks

Great!


The python code of ising-gpu has problem.

I execute the command "python ising_basic.py -x 2048 -y 2048 -n 100" five times, but I get the same configuration of the Ising model every time (i.e. the same "final_rank0.txt"). However, this is Monte Carlo simulation, so it should be wrong. I did not inspect the details of this code. Can someone fix it?
My OS: ubuntu 16.04
GPU: Quadro P4000
Thank you very much!

The code uses a fixed default seed (see here: https://github.com/NVIDIA/ising-gpu/blob/master/basic_python/ising_basic.py#L48). Unless you provide a different seed (via the -s command line argument), running the code repeatedly will produce the same result. Hope that helps!

The code uses a fixed default seed (see here: https://github.com/NVIDIA/ising-gpu/blob/master/basic_python/ising_basic.py#L48). Unless you provide a different seed (via the -s command line argument), running the code repeatedly will produce the same result. Hope that helps!

Thank you very much! I use the RNG supported by cupy (i.e. cupy.random), so I do not know the details of the RNG. I'm trying to use this code to do some research, and it helps me a lot, thank you again!


Arnovel (No Result Found)

Source information
Arnovel
Steps to reproduce
1.open extinction
2. It doesn't open
3. It's say no result found
Expected behavior
It should open
Actual behavior
This happens instead ..

LNReader version
1.1.8
Android version
Android 10
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 1.1.6.
[X] I have updated all installed extensions.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[X] I will fill out all of the requested information in this form.

Works fine for me. If the issue is still there reopen this.


BUG: there is no learning rate parameter for  Conjugate Gradient

Describe your issue.
there is no learning rate parameter for  Conjugate Gradient
https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.cg.html
per paper
https://arxiv.org/abs/1009.5839
learning rate should be specified
Reproducing Code Example
no code example
no need

Error message
no error message
no need

SciPy/NumPy/Python version information
latest

The "conjugate gradient" algorithm referred to in the paper (optimizing a kernel regression model) is not the conjugate gradient used in scipy.sparse.linalg.cg() (solving a linear system). Both are nominally related to a very general "conjugate gradient" technique for solving general continuous optimization problems, their specific formulations for their specific optimization problems don't necessarily have the same input requirements. Solving a linear system is a very constrained problem that permits the step size to be automatically determined and does not require a learning rate parameter to scale the step size. An Introduction to the Conjugate Gradient Method Without the Agonizing Pain is a classic reference.

great thanks for soon answer
"Solving a linear system is a very constrained problem that permits the step size to be automatically determined"
thanks for explaining, sorry for false alarm
it would be very kind of you to help to find plain python code from scratch to better understand
how conjugate gradient works without learning rate tuning
especially in ridge regression case for
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html
my guess scikit-learn Ridge regression  uses  your conjugate gradient code implementation

great code was found
https://towardsdatascience.com/complete-step-by-step-conjugate-gradient-algorithm-from-scratch-202c07fb52a8
but still may you help to understand how your implementation deals with non-symmetric matrices
as mentioned in
https://en.wikipedia.org/wiki/Conjugate_gradient_method
https://en.wikipedia.org/wiki/Biconjugate_gradient_method
https://en.wikipedia.org/wiki/Biconjugate_gradient_stabilized_method

like
https://stats.hohoweiya.xyz/2017/08/13/Conjugate-Gradient-for-Regression/

I'm afraid this isn't going to be a forum where you will get this kind of general help. But to clarify a couple of relevant points:

scipy.sparse.linalg.cg() does only handle symmetric positive-definite matrices.
RidgeRegression constructs a symmetric positive-definite operator to pass to cg() (see create_mv() for how it is constructed).

thanks for soon valuable answer


Replace file format providers

Rationale for this change
As discussed in https://github.com/apache/arrow-datafusion/pull/1010, this PR replaces the old file format datasources with the new one (both table providers and execution plan)
What changes are included in this PR?
These are mostly mechanical changes that come from the replacement of the file format datasources. A few changes are noteworthy:

the way the default values are provided to the table provider is somewhat chaotic. Some parameters come from the context, some from  the struct XxxReadOptions<'a> objects. This PR mimics the current behavior as much as possible, but a separate work should be conducted to make the configuration system more consistent and intuitive
the ExecutionContext.read_xxx and ExecutionContext.register_xxx are now async because they might require to infer the schema
the get_file_metadata service in Ballista does not return the list of files anymore

Are there any user-facing changes?

ExecutionContext.read_xxx and ExecutionContext.register_xxx now async

What changes are left for later?
When possible, the behavior was left to be the same. For this reason, some todos were left:

in Ballista, the schema should be resolved on the scheduler (calling a service like get_file_metadata) instead of locally
the way the configuration is passed around between the context and the logical plan builder lacks consitency (in particular in terms of default management). This should be improved.
the useage of PartitionedFile, ParquetPartition, FilePartition is not very clear. The structures should need to be simplified a bit.
the LocalFileSystem ObjectStore is hardcoded in Ballista

the useage of PartitionedFile, ParquetPartition, FilePartition is not very clear. The structures should need to be simplified a bit.

I introduced PartitionedFile and FilePartition here: https://github.com/apache/arrow-datafusion/pull/932
PartitionedFile -> Single file (for the moment) or part of a file (later, part of the row groups or rows), and we may even extend this to include partition value and partition schema (see below) to support partitioned tables:
/path/to/table/root/p_date=20210813/p_hour=1200/xxxxx.parquet
FilePartition -> The basic unit for parallel processing, each task is responsible for processing one FilePartition which is composed of several PartitionFiles.
I think ParquetPartition is a legacy abstraction and we could use FilePartition instead directly in various scan physical plans. Since FilePartition and PartitionedFile are meant to be format agnostic.


Incorrect unity value of package.json

"unity": "2013.3",

It seems 2013.3 is not a valid Unity release.

@favoyang Fixed in v0.0.12


feat(ArtQuiz): main screen

The type of this PR is: Feature

This PR solves GRO-1290
Description
This PR creates a dumb interface for the artwork quiz. Following up in Web Practice about the use of CSSGrid, and will refactor according to the consensus there (or in a follow-up RFC, if requested). A good deal of work is still to be done here based on the ongoing epic.
Completed here:

[x] Create responsive desktop/mobile layouts per figma

For follow-up PRs:

[ ] Finalize context logic
[ ] Add animations
[ ] Add tooltips
[ ] Add mutation to dislike button
[ ] Replace fixture data w/ query
[ ] Write integration tests

Desktop

Mobile Web Large

Mobile Web Small

@dzucconi now that you're back wanna give this a review?

damon faked us out! he's not back yet. 😂 definitely going to get his input on this!

(I'm technically back tomorrow; but I've got a computer in front of me 👋 )

@TMcMeans - note that you can always create branches off of a branch and then rebase after the merge; things don't need to be merged to main in order to continue working. Stuff that contains any complexity benefits from having the whole team look at it, especially coming right out of a draft on a Monday morning.

@damassi I'm gonna push back on that thinking. As the assignee for this PR, I looked at the comments on this PR and chose to approve because they didn't seem blocking/could be dealt with in a follow up PR. Why does the entire team need to do a thorough review in order for the assignee to approve the work?

Mostly because you were out all of last week and Laura and I had been discussing a lot of the particulars in this PR at great length. Further, if you look up at the previous convo, there were issues that were yet to be resolved before you merged. So Its good to read back over that stuff and at least acknowledge that there were prior convos ongoing.

@damassi I may not have had context for any slack convos or otherwise but from my perspective the issues that were brought up have been resolved. I've done my due diligence here, I'm not exactly sure why this is a problem.

It's no problem! Just saying that there were in flight convos here, you were out, and having more eyes on it would make for a better final product. I had intended to do a final review this week when it was taken out of draft, and to find that it was already merged when I got in seems to be a process hiccup given Laura and my convo last week. And of course we can always follow up as we're doing now!

sure, when I first looked at this it seemed like you've already done review and it was ready to go so that could have been based on lack of context so I understand the frustration. I also feel like there isn't anything here that can't be done via a follow up PR, a refactor PR or a fix.

@damassi this is on me. Since I brought @pam- up to speed, and we discussed going ahead with the merge until I took my thoughts on CSSGrid to Web Practice, I told Tanjie that the code here & it's associated logic were the only things of concern.


Master Zoning Law

ReverRace:
all construction will be low density.
50% of any give area must be greenspace
Yeul:
only small trees though, no shadows allowed.

only small trees though, no shadows allowed.


Chore: Improvement on config templates

Just small touches that I think can look better

thanks for the cleanup!


dropbox folder empty

Hello im curious if you have a revised link for the dropbox directory, the current is empty.
👍

Hello @thePegasusai!
Nice point, unfortunately there is not enough space in this dropbox to upload every file. However, I'll provide some example in the following days and a definitive solution asap.
Thank you for the notification.

great!


Allow running queue.php from any working dir

This prevents scripts executing the queue from having to cd into shell/ before executing the script.  Just a little peeve of mine, having to add cd && everywhere.
Note that cli populates SCRIPT_FILENAME with argv[0].  This works fine on Windows, Mac, and Linux.  It specifically supports a symlinked setup where shell/queue.php is a symlink, but works also when copied.
If for some reason someone does e.g. cd shell/ && php ../.modman/magento-lilmuckers_queue/shell/queue.php, it will fall back to the existing behavior and use abstract.php in the current working dir.

@lilmuckers bump? I think this is a needed feature!


Allow parsing JavaScript proto definitions from JSON

This PR adds the new spine-proto-js-plugin which generates the fromJson(json) method for the JS proto messages.
The fromJson method allows to parse the message from the JSON string.
Usage
To use the plugin, add
classpath "io.spine.tools:spine-proto-js-plugin:$spineBaseVersion"

to the buildscript section of the Gradle configuration, and then add
apply plugin: 'io.spine.tools.proto-js-plugin'

to the module compiling proto files to JavaScript.
Needed Project Config
The plugin currently relies on the set of hard-coded settings in the Gradle project configuration.
The full list of needed settings can be found in the ProtoJsPlugin class doc but in general they mimic how the Spine Web compiles protos to JS.
Thus, the Spine Web is almost ready to use the plugin, except it needs to enable the descriptor set file generation via
task.generateDescriptorSet = true
task.descriptorSetOptions.path = "${projectDir}/build/descriptors/${task.sourceSet.name}/known_types.desc"

specified in protobuf section of js.gradle.
In future the plugin should be able to setup the necessary project config by itself.
Windows build
Also, the PR contains minor build script changes that should've repaired the Windows build which was previously broken.

@armiol PTAL.

@armiol PTAL.

@armiol PTAL.


Azure Container Apps is not mentionned as supporting Managed Identity

Please provide us with the following information:

This issue is a: (mark with an x)

[ ] bug report -> please search issues before submitting
[X] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description
Azure Container Apps is not mentionned as supporting Managed Identity
Azure Container Apps is not mentionned as supporting Managed Identity
Please update the docs at https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/managed-identities-status#azure-services-that-support-managed-identities-for-azure-resources
Steps to reproduct
N/A
Expected behavior [What you expected to happen.]
N/A
Actual behavior [What actually happened.]
N/A
Screenshots
N/A
Additional context
see also https://github.com/MicrosoftDocs/azure-docs/issues/99664

The doc has been updated throught the other issue mentionned, closing this one


Enhancement: ExtNetInfo for lane grouping

NetInfo objects will be paired with new ExtNetInfo objects to provide extended information. Initially this will be lane grouping data based on medians and directionality. The ExtNetInfo objects will be generated on first request, and stored in a static instance of a backported ConcurrentDictionary keyed on the NetInfo reference.
Each lane will be assigned a set of flags:
public enum LaneGroupFlags {
    Service = 1 << 0,
    Express = 1 << 1,
    Displaced = 1 << 2,
    Forward = 1 << 3,
    Backward = 1 << 4,
    Median = 1 << 5,
}

Forward and Backward do not directly map to the vanilla lane direction. They identify a lane group's prevailing vehicle direction, and may not match a specific lane's direction in some cases.
The lanes are organized in these steps:

Apply Forward/Backward flags to vehicle lanes.
Analyze vechicle lane directionality, identify lanes whose direction+position are nonstandard (as defined below), and apply the Displaced flag.
Identify medians based on rules described below, and apply the Median flag.
For all lanes that are not Displaced:

If a lane is inside a median in relation to other lanes in the same direction, apply the Express flag.
If a lane is outside a median in relation to other lanes in the same direction, apply the Service flag.

For each non-vehicle lane that is not a Median, apply the flags of the nearest vehicle lane.

Once this is done, a collection of lane group objects are created and stored in the ExtNetInfo object.
ExtNetInfo itself will have the following flags:
public enum ExtFlags {
    ForwardDisplacedLanes = 1 << 0,
    BackwardDisplacedLanes = 1 << 1,
    OuterDisplacedLanes = 1 << 2,
    ForwardExpressLanes = 1 << 3,
    BackwardExpressLanes = 1 << 4,

    AnyDisplacedLanes = ForwardDisplacedLanes | BackwardDisplacedLanes,
    AnyExpressLanes = ForwardExpressLanes | BackwardExpressLanes,
}

Displaced lanes are considered "outer" if the leftmost lanes are forward lanes, or if the rightmost lanes are backward lanes. Otherwise they are considered to be "inner."
The presence of express lanes implies the presence of service lanes.
Nonstandard lane direction+position
"Left" and "right" here assuming you are looking down a segment from its starting end, so that negative positions are on the left and positive positions are on the right.
A forward lane is in a nonstandard direction+position if both of the following conditions are met:

The forward lanes are not all contiguous, but have one or more backward lanes inbetween.
The lane being evaluated has any backward lanes on its right.

For backward lanes, the above criteria are reversed.
Medians
A non-vehicle lane is considered a Median if either of these criteria are met:

The lane is at a higher vertical offset than that of the vehicle lanes around it, but the difference does not exceed 3 meters. (The 3-meter limit will prevent false positives on certain types of mass transit.)
The gap between the nearest vehicle lanes on either side is at least as wide as the widest of those lanes.

In addition, a Median lane cannot overlap any vehicle lane.

The lane is at a higher vertical offset than that of the vehicle lanes around it, but the difference does not exceed 3 meters. (The 3-meter limit will prevent false positives on certain types of mass transit.)

what if they are both car lanes?

LaneGroupFlags can be added to ExtLane from ExtLaneManager. unless if you are worried about manager's dependencies and order of operations.

what if they are both car lanes?

That's the point. Two car lanes with a non-vehicle lane between at a higher vertical offset.

LaneGroupFlags can be added to ExtLane from ExtLaneManager.

That would produce an insane amount of redundant data. This information is all at the NetInfo level.

Having said that, it might make more sense for this to be classified as a more general extension to NetInfo, rather than being specifically about lane grouping.

I have decided that managers are not an appropriate pattern for this, and so it will follow the basic design of the displaced lane detection code I have already written.

what if there are two car lanes at different heights with no lanes in between?
or two car lanes at same height with another car lane of different height in between?

what if there are two car lanes at different heights with no lanes in between?

That sounds like a use case this wasn't designed to cover. Do you have any examples?

or this?

maybe in future there will be something like this?

I mean its not technically median but cars should sill not cross lanes.

The example where the inner lanes are above grade at the interchange seems like it would be way outside the scope of this project.
The example with the lanes running alongside but below grade looks to me like it should be separate roads.

NYC actually has express lanes that go above or below grade at some intersections. I chose not to include that in this project because there's a lot more involved than just routing and traffic light control.

WIP can be found here: https://github.com/Elesbaan70/TMPE/tree/feature/medians/ext-net-info


apns:close_connection don't closed h2_connect

https://github.com/inaka/apns4erl/blob/7be3493011ecc43439526338ba5e62ae54fc4737/src/apns_connection.erl#L219
i modify apns_connect.erl and fixed this issues.
handle_cast(stop,  #{http2_connection := HTTP2Conn} = State) ->
   h2_client:stop(HTTP2Conn),
   {stop, normal, State};

by the way , i use chatterbox 0.7.0 ,and it replace gen_fsm to gen_statem

Hi @boboxiaodd , thanks for the fix. Could you please raise a PR with your fix?

sorry @ferigis i don't know how to do it.
i use apns:close_connection every 10min ，and use apns:connect a new h2_connect
to solve chatterbox's memory leak


Specifix SQL Statements in Method leads to error

I have a method with the following SQL Statement in it:
    SELECT ( SUM( netwr ) + SUM( mwsbp ) ) AS order_amount
       FROM vbap
       WHERE vbeln = '1234567890'
       INTO @DATA(l_order_amount).

When I run ABAP Cleaner on it, i get a Parse error about "ENDMETHOD found but no corresponding METHOD". If I comment the SQL Statement, it works...

I wonder if SELECT SINGLE makes any difference / sense?

I wonder if SELECT SINGLE makes any difference / sense?

SELECT SINGLE works, but then I cant SUM ;)

I wonder if SELECT SINGLE makes any difference / sense?

SELECT SINGLE works, but then I cant SUM ;)

Have you tried? I don't think it makes a difference.

mmh, ok. You are right, it works. To be honest, I didn't expect this 😄
Ok, great. Problem half solved. At least for me.
This also works with ABAP Cleaner:
    SELECT
      FROM vbap
      FIELDS ( SUM( netwr ) + SUM( mwsbp ) ) AS order_amount
      WHERE vbeln = '2004678933'
      INTO @DATA(l_order_amount).
``´

Hi bastianStr and Fabian,
thanks a lot for reporting and discussing this! This was a bug, because ABAP cleaner wrongly assumed that this SELECT requires an ENDSELECT (and therefore expected an ENDSELECT before the ENDMETHOD).
To determine whether or not SELECT opens a loop is not trivial, and in this case, ABAP cleaner got the arithmetic expression in parentheses wrong. Will be fixed in the next release!
It's nice that there is a workaround, but of course ABAP cleaner should be able to deal with anything that is correct ABAP syntax (well, apart from chain colons inside parentheses as in any_method( iv_value = : 1 ), 2 )., those are just too horrible ;-).
Kind regards,
Jörg-Michael

Hi bastianStr,
thanks again for opening this issue! This should now be fixed with version 1.7.0, which was just released!
Kind regards,
Jörg-Michael


appcoins

https://github.com/Catappult/appcoins-wallet-android/releases/download/v<IP_ADDRESS>/AppCoins_Wallet_v<IP_ADDRESS>_213_release.apk

what?

Spam

SPAM. Should be closed


Find max of three numbers

Description
Write a C++ program to take input 3 numbers and find the max in them
Domain
CPP
Type of Contribution
Addition
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of LearnCPP.

ASSIGN

/assign

/assign

/assign


Changing monitors makes window appear out of the screen

Changing monitors makes window appear out of the screen. Even if quit it, and restart, it still although less, has the top bar with the zoom, etc, buttons out of reach.

Which interface are you using? (You can find out under Preferences → Appearance)

Hi,
Bisel (included), with Superfluous visual effects ON

That one should center itself every time it becomes visible. Can you describe what you mean by “changing monitors”? I have an external display at work, but moving the focus to that one and calling up Bezel shows it right in the center.

Well, that's exactly my situation. At work I have I fairly wide (ultra-wide, in fact) monitor.  But if it happens I open Quicksilver on that big monitor, then, like right now, opening it on the MacBookPro retina13, meshes totally its position.  It is impossible to handle and even if you close/open QS it, stills is wrongly placed.
Sending a partial screen capture.

Well, just now, after quitting/restarting it has centered it, not in previous experiences… But the problem is of course, as I have sent you in the screenshot, still there.

Oh, you’re talking about the Preferences window. I was only looking at the actual Quicksilver interface.
It seems to remember the screen and position every time I open it, so it’s never in the “wrong” place. Does this only happen if you open it and make it large on the external display, then disconnect that display and try to open it again?

Not exactly, in fact, I didn't change its size in the external display, as far as I can remember.  It happens always after opening on the external (laptop screen close) and then on the laptop own screen (external disconnected).  The problem here is it takes negative Y coords for the window, so is impossible to access it

I think you might be able to workaround the issue by Mission Controlling the preference window on another Space and back. Hopefully the window server will notice the weird coordinates and fix them by itself.

I tried that, but it seems OSX knows it is not a normal window and doesn't allow you to assign it any Space but the main

Well, you might be glad to hear I just reproduced it here ;-). I'll try to see if it can be fixed easily, sadly my second monitor is at work, and it seems quitting restores it correctly.

@juanfal If you happen to have your external monitor handy (as I won't come near mine until Monday), can you try it again while running that build. You'll get signing errors because I can't sign so you might need to ⌃clic > Open to bypass GateKeeper but otherwise it's straight out of #2183.
I think the reproduction steps for your issue are :

open preference window
disconnect monitor.

Maybe 1.5) arrange the pref window to be flushed against the top of the external, just to make sure the coordinates will be off when windows will be rearranged at 2).

Oh, sorry, same situation right now, tomorrow I'll try, I hope!

Ok! you have made it.  Tried it and it works.  Just after opening it in my external monitor, I've opened it in the laptop and I saw how it repositioned itself moving the upper part of the preference window inside the screen frame.  Thanks!!!

I know this issue is closed, but I have noticed a repeat of this behavior since I got new monitors at work.  I have 2 2560x1440 monitors and I also use my MBP's display (total 3 displays).  I've noticed that it is possible to move the QS interface to a place that will make it pop up totally off-screen if I switch to a set of displays with lower resolution.  I can fix it by switching interfaces from Primer to Bezel and back, but it would be nice if that wasn't necessary.

@cro This issue was about the preference window, not the main interface. I'd appreciate it if you opened a new issue for that, and provided a little more information, as I haven't been able to reproduce using one external Apple Thunderbolt Display, set as my main display.
Things I've tried, with Primer :

drag the interface to the bottom right corner of the main display, check that it opens here each time, close iface, disconnect monitor, invoke QS. Interface is flush against the left side, but appears in its entirety.
drag the interface to the center of my laptop's screen, close iface, disconnect, invoke. Interface shows up as above.

What do you mean by "switch to a set of displays with lower resolution" ? Resolution change vs. monitor disconnection ?


aten.any.default and aten.all.default and fx converters

Relies on #3206
Converts to bool and then uses reduce_any or reduce_all in MIGraphX
Don't merge this till #161 is merged into master

We might have to modify the jenkinsfile for CI to be able to pass these test cases

This PR is pointing into pytest_skip_version instead of master btw

Passes on my local machine with MIGraphX on develop branch


[LEDGER] Move UpdatesBytesBuilder to txmgr pkg

Type of change

Improvement (improvement to code)

Description
Currently, we create the bytes representation of UpdateBatch to compute the hash of UpdateBatch which would be used to compute the block commit hash along with the validation results. The utility for converting an UpdateBatch to deterministic bytes is present in the privacyenabledstate pkg, i.e., DeterministicBytesForPubAndHashUpdates(). We have made the following changes to this utility.

This utility is used only by the txmgr and hence, we move this utility function to txmgr pkg from privacyenabledstate pkg to keep the utility closer to the consumer.
We have removed the UpdateBytesBuilder struct and unexport functions as they are not needed any longer.
We have defined proto messages such as KVWriteProto and KVWriteBatchProto to construct bytes from the UpdateBatch. As we shouldn't append the keyword proto to messages, we renamed KVWriteProto to KVWrite and KVWriteBatchProto to Updates.
The function name such as buildForKeys(), buildForColls() are not very explicit in what they do. When the buildForColls() calls buildForKeys(), it adds even more confusion as the first-level function, i.e., deterministicBytesForPubAndHashUpdates() is also calling buildForKeys(). Hence, we have used the following function names instead:

genKVsFromNsUpdates()
genKVsFromCollsUpdates()
genKVs()

Related issues
FAB-17830

I agree @manish-sethi. I also felt that get reads like reading it from the DB as we have used this prefix majoritily for DB access. I thought of construct but the name became long. I think the name suggested by you makes sense. I have changed the function names to genKVsXXX.

LGTM from code change point of view. Though, the code changes are mainly related to refactoring, still, it touches a critical code for commit-hash computation. Unfortunately, we lack a test that verifies that the behavior is still the same as before. May be this may be a good time to add a test in kvledger/tests - wherein, the stored ledger in the testdata folder can be used as a reference and one of the reset/rollback tests can be enhanced to verify the newly computed commit-hash - wdyt?

We have UT to ensure that the behavior of deterministicBytesForPubAndHashUpdates() is unchanged. Further, TestAddCommitHash() in kv_ledger_test.go also adds more robustness.
I understood the importance of ledger-level integration tests using testdata to ensure we are not changing the way the hash is computed. I would prefer to push a separate PR. If it must be done in this PR itself, I am fine with that too.

Yes, there is a safety check at the UT level. I am fine for the higher level test in a separate PR. TestAddCommitHash() does not serve that purpose though.

Thanks for merging this @manish-sethi Next, I will work on the commit hash test and push a PR -- https://jira.hyperledger.org/browse/FAB-17832


OpenStack "openstack_volume_size" not included in json results output

OpenStack provider specific flags such as "openstack_volume_size" are not making their way into the result json which could be misleading, since defaults make their way in.
Flag Values:
2016-09-27 13:46:35,551 browbeat MainThread INFO     Flag values:
--openstack_volume_size=5
--machine_type=m1.small
--os_type=rhel
--benchmarks=copy_throughput
--openstack_network=browbeat_private
--image=centos7
--cloud=OpenStack
--timing_measurements=runtimes
--copy_benchmark_mode=cp
--run_uri=browbeat
--openstack_floating_ip_pool=browbeat_public
--openstack_image_username=centos

And in the newline delimited results json:
{"labels": "|data_disk_0_type:standard|,|zone:nova|,|scratch_disk_type:standard|,|perfkitbenchmarker_version:v1.7.0|,|num_striped_disks:1|,|machine_type:m1.small|,|scratch_disk_size:500|,|data_disk_0_size:500|,|data_disk_0_num_stripes:1|,|image:centos7|,|cloud:OpenStack|,|vm_count:1|", "timestamp":<PHONE_NUMBER>.938512, "metric": "cp throughput", "official": false, "value":<PHONE_NUMBER>137134, "test": "copy_throughput", "run_uri": "browbeat-eed78df5-5389-4b37-b717-ed062cea8d99", "owner": "stack", "product_name": "PerfKitBenchmarker", "unit": "MB/sec", "sample_uri": "3f3a66a2-7a1c-4be7-9378-1a9533518d92"}

Looking through the results, you might think there is a 500G disk vs just a 5G cinder volume.  Shouldn't "openstack_volume_size" be included under "labels"?

Yes,  good catch.
So, we could remove deprecate openstack_volume_size and use data_disk_size, and if openstack_volume_size is set then use it as the data_disk_size.
What do you think?

Yes, good catch.
So, we could remove deprecate openstack_volume_size and use data_disk_size, and if openstack_volume_size is set then use it as the data_disk_size.
What do you think?

Thanks, For consistency with the other providers it sounds like data_disk_size would be the way to go.
On another note, should --data_disk_type=local allow the use of my flavor's ephemeral disk?

Sounds good, I'll work on patch soon.
Regarding data disk types, we could define our own. At the moment, it always uses Cinder Volumes regardless of the data_disk_type. I had a PR that added support for ephemeral disks, as well as using  a path within the boot partition, but I closed since it needs rework after I made the changes to move the OpenStack CLI.
Here's the PR #759
I'm open to suggestions for other disk types as well.

Sounds good, I'll work on a patch soon.

Cool I can help contribute to the patch as well.

Regarding data disk types, we could define our own. At the moment, it always uses Cinder Volumes regardless of the data_disk_type. I had a PR that added support for ephemeral disks, as well as using a path within the boot partition, but I closed since it needs rework after I made the changes to move the OpenStack CLI.
Here's the PR #759

Got it.

I'm open to suggestions for other disk types as well.

I'll let you know if we think there should be other disk types for OpenStack we should support.

Hi there, @akrzos.  I provided a fix to this issue #1220.  Please confirm and close whenever you have time.