Provide the ability to replicate specific data on an external redis-protocol KV storage

There still has to be a lot defined in this issue, but we can already outline the goal:

Be able to filter on one or multiple addresses, and store the data (mapped to those addresses) in a KV storage. The storage is externally and accessed via the Redis protocol.

By default it would do this for the entire blockchain (for which the addresses match), but I think it shouldn't be too difficult to support an optional min/max blockheight as well, should this be desired.
The original idea was to simply add it as an optional feature to the explorer module, given that it is already doing partially the work that has to be done. However @robvanmieghem came up with the idea, that we could just turn this into a separate binary, which uses the consensus and gateway modules, and which would run as a separate daemon service just for this purpose. I quite like that idea as well.
Also to be clear, for now there is no requirement to filter on data, I think the data should be considered arbitrary data, and filtering only ever happens based on addresses (sender and/or receiver(?)).
Prior to this issue #117 has to be resolved in master.

But does it allow to do this for multiple addresses at once? As it would be a waste of resources if we need to scan the entire blockchain for each address, separately.

If you are referring to getting the current api call, the explorer (and consensus set) should have the unlockhashes stored in a bucket. So its just some database calls

The goal is that services can subscribe to a (local(?)) explorer node, such that all data send to specific
addresses is also forwarded to those subscribed services. When a service starts subscribing for a given address, it will receive already all data that has been send since the specified min heigh/time or even from the genesis block, should no beginning have been specified.
High level requirements and principles

be able to subscribe to the blockchain for a given address:

immediately this would give you all data send to that address up til now;
it would give you the data send to that address from then on, in real time;

data (also called records) will be replicated per address, where each address can be seen as a namespace;
each record has a unique ID:

this ID has to be ordered, only be 32 or 64 bit, unsigned (?);
this ID can be generated by the data replicator code which receives the data in order anyhow;
Open questions:

should we somehow keep a link between this unique ID and the actual hash?
how is this new ID useful, other than that it takes less space? As in, what meaning does it really give just by seeing it? Or is that last question a wrong question?

when subscribing an address it should also be possible to define a specific data from which to start;

thus a subscriber has to provide the address and optionally can provide a min block to start from (if not given it will start searching from the genesis block);
Open Questions:

this was suggested by KDS, but I wonder if it would be useful to (also) allow a from_block_height, in case both are given we could simply use the value that results in the oldest block to start from;

if some record with the same ID has already been replicated prior, it will simply be overwritten if newer data with the same ID is somehow replicated:

Open Question: how can this even happen?

for sending data there would be a specific minimum transaction fee, which for example in ThreeFold might be 0.1 TFT (to start which, as this could be modified later);
this would mean the owner of the data (the one that owns the address (=namespace)) would get paid for putting data in a namespace.

Open Questions:

How to prevent malicious actors from spamming random data to those addresses. Sure it would cost them money, but still, you would have crap in your namespace (forever);

or is this not a concern, and if not, why not?

format of data which is binary stored in the blockchain (as suggested by kds)
[system_byte][id=4byte][acl=optional][payload]

payload is opaque to blockchain;
system byte defines whether or not ACL is to be used;
if acl is used (system byte = 0) we would have a structure in the next bytes, prior to the payload;
id is incremental id for each record;

Implications of this suggestion:

it would mean that the generation of the ID is the responsibility of the user sending the data;
we would require a fixed format of the data (except for the payload part which is still opaque);
we would have to support some ACL struct (but what would it really define/contain???);

This is different from how I saw it, as I had planned to just keep the entire arbitrary data opaque (except for the fact that we know we have to base64-decode it first and remove the prefix).
implementation possibilities
Here we have several options. We could for example (a) expose this API over web sockets, allowing services to subscribe and receive data like that. However that would be an indirect approach. For example, for our use cases we wish to replicate to a (local) redis(-ish) server, meaning we would need to write another service, run that as well, having the data needless having to go through an intermediatary.
We could instead provide the option (b) to give the address(es) of redis server(s), such that the explorer replicates directly to those addresses data when given, over a TCP connection. While that is a more straight-forward and performant solution, it also doesn't give any flexibility. The user would have no way to define how it would want that data stored, nor how it should be processed prior to storage. Further it would force people to use the redis-based replication, while others might prefer to use something else that doesn't support the redis protocol.
A last (?) option (c) could be to provide a seperate "data replicator" binary. Due to the modular design of Rivine it should be possible to use (only) the needed modules (e.g. explorer) in order to look up old(er) transaction(s) for a specific address as well as make sure to stay in sync with the blockchain at all times, such that specific incoming transactions can also be consumed. If this is indeed possible it should allow us to provide this seperate binary to allow people to setup a node that would only do data replication (and thus only ever read from the blockchain) and nothing else.
So many options...
In option (c) we could design the code such that subscribers are represented by an interface. This would allow anyone to provide their own specific data replicators, without having to define the entire read-and-spread logic themselves, given that they fork-modify-and-compile the binary.
As people would be forced to fork-modify-and-compile the binary anyhow, we could just as well make it just an optional feature into the regular daemon. Still I think this data-replication feature is something that will most likely be used stand-alone by a node/machine which is destined only for the data replication onto the KV storage server which is most likely running on that same node/machine, unless it's replicating to a cluster I suppose.
An entirely different approach (d) could be to have this as a seperate data-replication module, which would simply become another consensus-user, just as the explorer is one. This way we could make it another optional module that one could use, and would also not mangle the data-replication feature together with the exlorer one. At the same time it would allow both features to be used at the same time, togethe with any other already available feature. Just as with option (c) this would mean that people would have to fork-modify-and-compile the binary should they want their custom replication logic.
Somehow I like this last option, (d), the most. Should we go for option (c) or (d) we still have several possible approaches:

We implement a direct replication to a TCP server which implements the redis protocol, forwarding it the data directly, without touching it at all (except to decode the base64 encoding as well as removing the prefix);
We implement a direct replication to a TCP server which implements the redis protocol, but allow an optional interface which would allow you to define how the data needs to be consumed (after the base64 decoding and NonSia prefix removal) prior to replication, this would require a soft fork however;
We provide a more generic subscription model (based on channels?), where anyone can provide subscribers, given that they implement a common interface. The default implementation (and provided together with rivine) could still do (direct) replication to a redis-protocol-supported server, but at least it would allow people to redefine this replication-specific logic, given that they are willing to soft fork once again;

.please define "data send to an address" in respect to a transaction, I think this is a pretty important definition since a  lot of the logic depends on that.
the format of the data: [system_byte][id=4byte][acl=optional][payload], shouldn't we prefix this as well? It is implied  in the "So many options" first option but not earlier.

the format of the data:[system_byte][id=4byte][acl=optional][payload], shouldn't we prefix this as well? It is implied in the "So many options" first option but not earlier.

Yes, it has to be prefixed. I'll edit my comment. In the next few weeks it should be possible for blockchain implementations to define this prefix though.

please define "data send to an address" in respect to a transaction, I think this is a pretty important definition since a lot of the logic depends on that.

How you mean? I might misunderstand your question, but when we talk about data send to an address I'm simply talking about a transaction created on the blockchain with the destination being that address. I might misunderstand your question though...

@GlenDC  Well, you can send tokens to an address, being the unlockhash of an output but data is just a property on a transaction that can have multiple token outputs.

Well, you can send tokens to an address, being the unlockhash of an output but data is just a property on a transaction that can have multiple token outputs.

Well it's why I want to involve you all in defining this issue, as this is indeed very true. I suppose it would mean that data could be "send" to multiple addresses at once? I don't think there is a reason we should somehow add something on top, to make the address just be owned by one specific address?

While having the "replicator" and explorer be different modules could be nice, code-seperation wise, it would mean that a user wishing to both use the explorer and "replicator" module in a single daemon has the data twice, while probably this data is similar, so perhaps we should have it just be part of the normal explorer module after all, and be it opt-in.

It is important that the developer using that replication-enabled daemon, only ever has to interface using redis (rather than using a tfchainc CLI client for example). The daemon will still have to be running, and a wallet should be there (as otherwise transactions can't be made), but the developer's server will only ever communicate using the redis interface. Meaning that the "replicator" module has to somehow also watch the redis server (where the data will be replicated onto), to listen to the subscribe-related commands. This keeps it very simple for a user of that replicator to control it, as there is only 1 interface to learn: the redis one.
https://redis.io/topics/pubsub could help in this, but I'm not too sure. A queue implementation is possible as well.

The unique ids are based on the order that the blocks are stored on the blockchain, and are generated on-the-fly by replicators when replicating the data from the blockchain onto the redis server.

The originally proposed format [NonSia][system_byte][id=4byte][acl=optional][payload] no longer applies and instead you can just store it as expected as [NonSia][payload].

While having the "replicator" and explorer be different modules could be nice, code-seperation wise, it would mean that a user wishing to both use the explorer and "replicator" module in a single daemon has the data twice, while probably this data is similar, so perhaps we should have it just be part of the normal explorer module after all, and be it opt-in.

Not sure if duplicate data is that big of an issue, as we will mainly be storing only the data coupled with an unlock hash. Moving it into the explorer module also means all the bolt code has to be separated, moved under an interface, and then the interface implemented for redis. Which seems tedious given the timeframe, and vulnerable to introduce bugs.

It is important that the developer using that replication-enabled daemon, only ever has to interface using redis (rather than using a tfchainc CLI client for example). The daemon will still have to be running, and a wallet should be there (as otherwise transactions can't be made), but the developer's server will only ever communicate using the redis interface. Meaning that the "replicator" module has to somehow also watch the redis server (where the data will be replicated onto), to listen to the subscribe-related commands. This keeps it very simple for a user of that replicator to control it, as there is only 1 interface to learn: the redis one.
https://redis.io/topics/pubsub could help in this, but I'm not too sure. A queue implementation is possible as well.

So if I understand you correctly, there should also be a redis interface which can do the same as our client right now? Thats way outside the scope of this issue, and probably needs some discussion as well

The unique ids are based on the order that the blocks are stored on the blockchain, and are generated on-the-fly by replicators when replicating the data from the blockchain onto the redis server.

Do they still need to be ordered? Then we'd need some scheme like blockheight-txindex-part_of_hash. Given the constraint to the ID size, that probably boils down to (32bit blockheight -32bit txnindex)

Moving it into the explorer module also means all the bolt code has to be separated, moved under an interface, and then the interface implemented for redis.

I don't think that's true, as we would still want a local database to get already fetched blocks from, to have faster look ups for new subscribers. As remember, when they subscribe we need to give them also all data already send to an unlock hash in the past.

So if I understand you correctly, there should also be a redis interface which can do the same as our client right now? Thats way outside the scope of this issue, and probably needs some discussion as well

No, we don't want all our client functionality over a redis interface. What we (apparently) want is that the user would subscribe to an address using their redis client, this would probably be added to some kind of queue, and the replicator will pick up this request from the redis server, so it knows it has to start replicating the data for that address to the redis server. That subscribe request would define the address as well as the "from_date".

Do they still need to be ordered? Then we'd need some scheme like blockheight-txindex-part_of_hash. Given the constraint to the ID size, that probably boils down to (32bit blockheight -32bit txnindex)

Why is that needed? The data has a natural order order on the blockchain, no? When we collect data to be replicated, we would do so in order as well, no?

If we keep the local boltdb, then there is little to no gain in terms of storage space.

I was talking that if we would go for 2 separate modules, and thus which their own boltdb database, you would have twice as much data, while if we merge this logic directly as an opt-in feature into the explorer, we would only have that data once.

A misunderstanding on my part then. For now I'll focus on storing ALL the data, and once that is working, start on subscribing

Sounds good, just make sure to know that for KDS both are required for the launch, as he really wants the user of that replication feature (developer of the server that consumes that replicated data), only has to work with one interface (a redis client).

Or do we simply use the txnID?

I don't think KDS really minds what ID generation we use, as long as it has the following properties:

it is only 32 or 64 bit (thus small);
it is ordered (where the order HAS to be the same order as the data is stored into the blockchain);
each ID has to be unique to each record (data);
when 100 replicators replicate the data of the same namespace (address) from the same blockchain, it should all generate the same IDs for the same record;

I was talking that if we would go for 2 separate modules, and thus which their own boltdb database, you would have twice as much data, while if we merge this logic directly as an opt-in feature into the explorer, we would only have that data once.

I see what you mean, but I wasn't planning on keeping a local boltdb. Sure it would be nice since this way the data is indexed already in case we connect to a second redis instance, but for now that would also mean adding some complexity in the syncing, which we can also do later when we have more time for this (if we even want to continue with this setup in the future)

just make sure to know that for KDS both are required for the launch

yeah, problem is my experience with redis is very little so I'll need to do some research first
as for the ID generation, I think I'll go with 32bit blockheight + 32bit index in the block of the transaction for now. That way all the required properties are statified

yeah, problem is my experience with redis is very little so I'll need to do some research first

If so, please try to already replicate it for now, which is as simple as a HSET call to a given server (where the key is the address and the field the ID), that should be simple enough with limited experience with Redis.
I'll try to think tonight and allocate some time to come up with a plan to also handle the subscription using redis, so that you don't lose too much time in researching that.

as for the ID generation, I think I'll go with 32bit blockheight + 32bit index in the block of the transaction for now. That way all the required properties are statified

To be honest, for me that sounded like an awesome plan. But it seems that the requirements really are that it has to be 32 bit. So I don't think it will be allowed.
Not sure if you know how to make a similar scheme work (safely) for 32 bit? Otherwise we'll really somehow to just keep a counter, which is deterministic as long as you start counting from the genesis blockheight and only increase the counter for each record per address.

By the way, not sure if I communicated this in the meanwhile properly, but the address is indeed the  address which receives the fee via the output (and thus the receiver address). If there are multiple addresses so be it, as long as they all receive that data transaction fee, it is OK I guess.

If so, please try to already replicate it for now, which is as simple as a HSET call to a given server (where the key is the address and the field the ID), that should be simple enough with limited experience with Redis.

Yeah, HSet is one of the few things I do know, so already doing that.
As for the subscription, we could store an HSET somewhere which has the addresses we follow as field and a consensuschangeid as value. That way we can register every address as its own subscriber to the consensusdb. This would solve the issue of managing all the different addresses which may not be in the same state. Though I'm not sure how manageable this will get. Ideas are always welcome ofcourses
As for the ID to become 32 bit, I'm not sure. Obviously I could shrink it a little bit, but I'd need to check how much txns can fit in a single block to see how small I can make the index part, and we still need a reasonable number for the blockheight part so it does not overflow in this scheme. Counting from genesis could work, but then we always need to start at genesis, not at an arbitrary position

obviously I could shrink it a little bit, but I'd need to check how much txns can fit in a single block to see how small I can make the index part, and we still need a reasonable number for the blockheight part so it does not overflow in this scheme.

Yeah no, I don't think that's a good idea. As I said I would be totally happy to go with your scheme, but alas, it seems for some reason 8 bytes is not good enough. 4 bytes it is.

Counting from genesis could work, but then we always need to start at genesis, not at an arbitrary position

Yes, that would indeed mean that. Obviously for each subscription that only needs to happen once, after that you can just keep counting based on the last ID in redis, I would assume. You can also store values used for replication in redis, I don't think that's an issue either.
And given that walking over the blockchain are local DB calls, having to start from the genesis now and then, doesn't seem like it's that bad of a thing either, or is it?

4 bytes it is.

I'm curious, did you get any reason why we go with 32 bit fields in a 64 bit era?
In the scheme I have in mind, where each "tracked" address runs as its own subscriber, we'll indeed need to count from genesis every time, loop over every block and every txn in those blocks. But we need to do that anyway since we want to get all the data out of it, so incrementing a counter while we store data over tcp to a possibly remote redis server won't impact performance if we look at the bigger picture

I'm curious, did you get any reason why we go with 32 bit fields in a 64 bit era?

There doesn't seem to be a reason other than the fact that 32 bit seems the smallest integral value possible, while still allowing sufficient amount of unique IDs for a single namespace at once.

In the scheme I have in mind, where each "tracked" address runs as its own subscriber, we'll indeed need to count from genesis every time, loop over every block and every txn in those blocks. But we need to do that anyway since we want to get all the data out of it, so incrementing a counter while we store data over tcp to a possibly remote redis server won't impact performance if we look at the bigger picture

It should indeed not need to hurt performance. However we have to make sure to not lose the deterministic property. Should you have a lot of independent subscribers for a single namespace, even if some subscribers start from different dates/block-heights, still should those subscribers all use the same UniqueID for the same transaction.

Sadly last night I didn't find time to work on a design around this feature any longer. However this morning I came up with a design which I think covers the entire "replicator" interface, using the Redis protocol as a (means to establish a) communication and storage medium.
The "replicator", when enabled, knows what to do thanks to the commands send to the "replication channel". When the "replicator" is enabled and activated it subscribes to that channel using SUBSCRIBE replication. The user would use this channel to indicate to the replication which address is to be (un)subscribed:

subscribe: PUBLISH subscribe:$address[:$from_date]
unsubscribe: PUBLISH unsubscribe:$address

NOTE: as far as I know this doesn't allow to return an error back to the user,
which might be desired in case the user publishes a misformatted/unknown menssage,
or in case the action couldn't be applied for whatever reason.
Not sure what the best solution would be for this.

One solution could be to have a second channel,
on which the replicator publishes negative/positive replies,
this seems the most sane solution, as you can have many subscribers as you want,
and they all receive the same published content.

(!) the beautiful thing about channels is also that if you have no subscribers,
you can still safely publish on the other side without having to be worried that you fill up space,
as publish content is simply discarded if no subscriber is available,
the PUBLISH command returns the amount of times it published that content,
which is equal to the amount of active subscribers, and thus can be 0.

Replicated data is stored in a hashmap, and stored by the data "replicator":

each record is stored using the command: HSET $address $uid <data>
min/max indices are stored for a namespace as well:

start/min: HSET $address s $uid
end/max: HSET $address e $uid
this metadata relies on the assumption that all indices between start and end are used/stored;
having this metadata we can handle following situations:

subscribe starting from an index X lower than min:
we simply have to store from X until min
subscribe starting from an index X in between min and max:
we have to do nothing, storage wise
subscribe starting from an index X, after max:
store starting from max+1, until we either reached X,
or until we have no more records available

All this means that the user of the replicator has to
only support 4 redis commands (HSET, HGET, SUBSCRIBE, PUBLISH),
and it has enough to be able to communicate with "replicator",
as well as to consume the replicated data.
Obviously some other commands (e.g. HMGET, HSCAN, HGETALL, ...),
could be useful as well for the user, and would make more sense in certain situation,
but they still remain optional to be supported by the user,
and the interface could work fine just without. Even so,
this is all on the user side, and has no effect on the replicator side at all.

this metadata relies on the assumption that all indices between start and end are used/stored;

So this means that the uid is linked to the address (multiple addresses can have the same uid set), and that the uid is only incremented in case of a txn which contains the linked address in one of the outputs, and that the txn has arb data.
Is the range a hard requirement, or something that will need to be supported at some time? since it obviously adds (a lot of) complexity to the design.
Finally, how to store the data? Right now, arb. data on a txn is a [][]byte, whereas I store it as []byte, using the sia encoder to flatten it. this produces the followig output:
field: "\xe2\x8d\x93" -> the uint32 uid as bytes
data:
"\x01\x00\x00\x00\x00\x00\x00\x00\x18\x00\x00\x00\x00\x00\x00\x00NonSia\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00testdata" -> siaencoder of 2 dimensional array, first 8 bytes indicate the size of the first array dimension is 1, next 8 bytes indicate the size of the first (and only) 2nd dimenstion byte array is x18, followed by the 16 prefix bytes (converting those that can be converted to valid chars), followed by the actual "testdata" which was inserted.
Btw, the sia encoder encodes in little endian, while the string(uint32) builtin appears to convert to big endian.
I was wondering if we can just flatten the arb data by appending each row to the previous, then we  don't have the overhead of the encoding, but we won't be able to reconstruct the 2 dimensional array (if this is ever required)

Btw, the sia encoder encodes in little endian, while the string(uint32) builtin appears to convert to big endian.

You could convert numbers using the binary package as well.

So this means that the uid is linked to the address (multiple addresses can have the same uid set), and that the uid is only incremented in case of a txn which contains the linked address in one of the outputs, and that the txn has arb data

Yes, I thought this was already clear, I guess not. Either way, I'm happy that requirement is clear and well understood now.

Is the range a hard requirement, or something that will need to be supported at some time? since it obviously adds (a lot of) complexity to the design.

I think it is a useful feature to have, not sure if it is super important, but knowing that kds already freaks out about 4bytes vs 8bytes for an ID, I would say replicating all possible data, rather than a limited scope is very much out of the question.
What kind of complexity would it add? We already have the dates, so all it would mean is that we not only filter based on the address but also the date, right? Where the from_date is optional. If you have a better idea on how to provide the option to the user to provide a range, I'm all ears to, if that would make it easier.

data on a txn is a [][]byte

That's weird, why is this doen? I would have made this a []byte instead o.o
And i'm fairly sure KDS is expecting a flat binary payload as well.
Can you elaborate on why it is not a flat slice?

Also, I suggest to change the uid in:

start/min: HSET $address s $uid
end/max: HSET $address e $uid

to use the consensuschangeid. Since the consensusset reports the block history as it happend, with rollbacks due to possible forks, which would mean that a uid can be generated twice

As for subscription ranges, it means that the subscriber for an address needs to be aware of this. It needs to be able to unregister itself.

The subscriber might be aware from it, but this is not required. I do see where complexities can arise. At all times we can simply use the lowest chosen from_date, of any active subscriber. This min_from_date can be stored as metadata as well I suppose.
If there is a startdate, and this data is later adjusted to a lower value, we need to restart from genesis anyway since the uid needs to be calculated
Yes that is

Since the consensusset reports the block history as it happend, with rollbacks due to possible forks, which would mean that a uid can be generated twice

Not sure that is a real concern, as that would simply mean we overwrite an existing ID with the actual transaction, which the design allows for. But if we can use concensuschangeid without having to give something up in return, than why not.

As discussed on telegram, we should only replicate data which is for sure is not going to be forked. This way the replicated data can be seen as WRITE-ONLY, with no data previously replicated suddenly deleted due to a fork.

Updated the WIP branch with latest code, still need to test with actual data, will do that once I fixed my test setup. Management through redis ((un)subscribe) works.

When closing the daemon, it hangs until redis throws a SIGSEGV, because the PubSub channel somehow gets garbage collected. It seems however that the library exposes a somewhat lower lvl API which allows us to get a read only buffered channel, so I'll change the current subscription strategy to make use of that
Also right now data will be written immediatly (and still deleted), so there will need to be some delay(prevent deletes) and maybe caching (performance so we can use pipes, don't this is all that useful though once we've caught up).
Check if there needs to be code cleanup since some of it might be out of place (or unused) after the latest requirements change

Also right now data will be written immediatly (and still deleted), so there will need to be some delay(prevent deletes) and maybe caching (performance so we can use pipes, don't this is all that useful though once we've caught up).

Let's only replicate data, which transactions live in blocks that have at least  blocks after them.

Further requirements:

when using the data gets created using the REST API, the user should receive a long ID identifying that data (the transaction hash?), if this is already the case, excellent;

Open "Questions":

for now anyone would be able to create data for a given namespace, we need a way to restrict such access (an implementation based on the research done in #141 could help);

data gets created using the REST API

I suppose you mean a transaction is posted which has arbitrary data
As for the ID, we could return the txn ID to the user after a successful post. But that is a separate issue I feel as it does not seem to have anything to do with the replication feature itself (unless the ID's need to be linked somehow)

I suppose you mean a transaction is posted which has arbitrary data

yes

As for the ID, we could return the txn ID to the user after a successful post.

Yes that's what we should do. It has to do with the replication feature as we need to be able to return an ID immediately to the user of that REST API (and thus also the python light client), as the short ID (based on the blockchain order etc) will only be known once the transaction has both been pooled into a block, has been validated and accepted, and has survived long enough to guarantee its survival. At which point the data is replicated using the short ID. At that point however the user only has this long ID, so we need to be able to go from that long ID to that short ID. And thus those 2 IDs can be seen as a two different references to the same stored data.

Is there an idea about this link already? if the data replication module needs to give some feedback or be able to resolve long IDs to short ones, it will need some kind of persistent mapping, since the short ID's have no relation to the actual location of the data

For now I have no precise idea about what to do, certainly not one I'm happy with. Just as you were thinking, with the current requirements in mind I do indeed think that such a link is only possible using some kind of persistent mapping. For example, a naive link could be easily established by using just the first x bytes of the long id as the short ID. However this would break the ordered requirements of the short ID.
That said, I think it can be persistent and at the same time volatile. The short id could be communicated over a channel from server to user, using the same subscriber-kind-of system. At the same time we could also make sure that the first time the user requests data using the long ID, the new (short) ID is returned together with the data, and to from than on only accept the short ID (as we could at that point delete the link between the long and short).
However, I'm not sure you really have to worry about this for now. I think all parties involved would accept the non-existence of that short ID for now, and simply have the long ID for now.
@robvanmieghem might have some ideas/input/feedback for this in specific as well.

Good news. It has been agreed with KDS that we can use 8 bytes for the short ID. See #149 for more information. This means that this feature gets much simpler as well. Once issue #149 is implemented (any suggestions for the new endpoint URL mentioned at the end of that issue description by the way), we can rebase and integrate it in this feature as well.
It will make the mapping also much more obvious. Even though it will not resolve the mapping at all, and you will still need to find a way to map the long to the short IDs.


Question on NATS Java client tooling

There is a tool in NATS which provides testing the specifics to the NATS Java client called Jetstream Multi tool - https://github.com/nats-io/java-nats-examples/tree/main/js-multi-tool.
Can this be used for testing vert.x?

Scott mentioned that it could be used. I will circle back with Scott and check.

Need more info


Label calculation

❓ Questions and Help
In the following documentation it says the following

In the Alpha158, Qlib uses the label Ref($close, -2)/Ref($close, -1) - 1 that means the change from T+1 to T+2, rather than Ref($close, -1)/$close - 1, of which the reason is that when getting the T day close price of a china stock, the stock can be bought on T+1 day and sold on T+2 day.

However, in Aplha360 should we use the same equation - Ref($close, -2)/Ref($close, -1) - 1 - for label? Or should we use Ref($close, -1)/$close - 1 ?
In this other page from the documentation it says,

For ic

The Pearson correlation coefficient series between label and prediction score. In the above example, the label is formulated as Ref($close, -1)/$close - 1. Please refer to Data Feature for more details.

Are those two formulas related? Shouldn't them be the same?

@you-n-g if it is of any help I can fix this documentation.
I just need to undersand what exactly is wrong.
Is  it the IC definition? Or the equation that should be used in Aplha360 which should be Ref($close, -1)/$close - 1 ?

Sorry for taking so long to reply to you, I have made changes to this issue in this PR.


Alpha.4 has many TS7006 error "implicitly has an any type"

Hi.
In alpha.4 & TS 1.8.9 "moduleResolution": "node", "noImplicitAny": true
node_modules/angular-ui-router/commonjs/common/common.d.ts(13,37): error TS7006: Parameter 'X' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/common.d.ts(163,63): error TS7006: Parameter 'T' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/common.d.ts(163,66): error TS7006: Parameter 'key' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/common.d.ts(165,71): error TS7006: Parameter 'T' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/common.d.ts(165,74): error TS7006: Parameter 'key' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/hof.d.ts(113,27): error TS7006: Parameter 'ctor' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/hof.d.ts(113,37): error TS7006: Parameter 'x' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/hof.d.ts(115,27): error TS7006: Parameter 'comp' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/common/hof.d.ts(115,37): error TS7006: Parameter 'x' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/ng1/viewDirective.d.ts(10,9): error TS7010: 'resolve', which lacks return-type annotation, implicitly has an 'any' return type.
node_modules/angular-ui-router/commonjs/state/stateBuilder.d.ts(3,54): error TS7006: Parameter 'parent' implicitly has an 'any' type.
node_modules/angular-ui-router/commonjs/transition/transitionService.d.ts(75,35): error TS7006: Parameter 'error' implicitly has an 'any' type.

I encountered many TS7006 errors. please describes any correctly. thanks.

+1

@armorik83 @majo44
Which package are you both referring to... the ng1 package?
We now publish angular-ui-router, ui-router-ng2 and ui-router-core to npm.  You should use npm install --save angular-ui-router@next.  The package's main: points to the release/angular-ui-router.js bundle, and typings: points to the typescript definitions.
Then use import {Thing} from "angular-ui-router";  Your Typescript compatible IDE should find the typings automatically.

Closing as wontfix because typescript 2.0 has an option to skip lib checks
https://github.com/Microsoft/TypeScript/pull/8735

Unfortunately that means that users of the latest router will have to be on TypeScript 2.0 and have that option set given that using noImplicitAny is pretty commonplace.  That's a real shame.  I realise (having taken a look at it) that adding the necessary type annotations to resolve this issue isn't the work of 5 minutes but it's a shame to add extra requirements onto the users.  Would you be open to PRs to resolve this?  I may be able to slowly resolve this issue (though it might take a while as I don't know the codebase yet).

So I got bold @christopherthielen - I've made a PR that makes ui-router noImplicitAny compliant.  I hope this works for you?

Awesome!

Thanks @johnnyreilly.
@christopherthielen  Btw, this not only produced error in the declaration files from ui-router but also in your own code if you do for example:
import {StateParams} from 'angular-ui-router';  class MyController($stateParams:StateParams) {   let x = $stateParams['param']; }
This throws on $stateParams being an implicity any even with --skipLibCheck, with this fix the problem is gone. Is there a nightly build one can use? or could we have a release with this fix in. Thanks!

FYI I think noImplicitAny compatibility is broken again in 1.0.0-rc.1:
ERROR in E:\repos\resgroup\smart-ui\node_modules\ui-router-core\lib\resolve\resolveContext.d.ts
(7,22): error TS7005: Variable 'NATIVE_INJECTOR_TOKEN' implicitly has an 'any' type.

ERROR in E:\repos\resgroup\smart-ui\node_modules\ui-router-core\lib\transition\transitionHook.d.ts
(13,37): error TS7006: Parameter 'error' implicitly has an 'any' type.

ERROR in E:\repos\resgroup\smart-ui\node_modules\ui-router-core\lib\common\common.d.ts
(361,31): error TS7006: Parameter 'a' implicitly has an 'any' type.

ERROR in E:\repos\resgroup\smart-ui\node_modules\ui-router-core\lib\common\common.d.ts
(361,34): error TS7006: Parameter 'b' implicitly has an 'any' type.

using --noLibChecks as a workaround works as expected.


assign two move_replica tasks within very short interval cause 2nd task block forever

It can be reproduce by this test:
#[test]
fn dup_move_replica() {
    block_on_current(async {
        let mut ctx = TestContext::new("group-test--move-replica");
        ctx.disable_all_balance();
        let nodes = ctx.bootstrap_servers(4).await;
        let c = ClusterClient::new(nodes.clone()).await;
        let group_id = 10;
        let mut node_id_list = nodes.keys().cloned().collect::<Vec<_>>();
        let node_id = node_id_list.pop().unwrap();
        create_group(&c, group_id, node_id_list).await;

        info!("issue moving replicas request");
        c.assert_group_leader(group_id).await;

        let mut group = c.group(group_id);

        let follower = c.must_group_any_follower(group_id).await;
        group
            .move_replicas(
                vec![ReplicaDesc {
                    id: 123123,
                    node_id,
                    role: ReplicaRole::Voter as i32,
                }],
                vec![follower.to_owned()],
            )
            .await
            .unwrap();

        // tokio::time::sleep(std::time::Duration::from_secs(10)).await;

        group
            .move_replicas(
                vec![ReplicaDesc {
                    id: 123124,
                    node_id,
                    role: ReplicaRole::Voter as i32,
                }],
                vec![follower],
            )
            .await
            .unwrap(); // this should panic with EpochNotMatch
    });
}

The 2nd move_replicas decision can be made when root using out-of-date metadata, the 2nd move_replicas should return error but block forever.
2022-09-15T10:04:19.111017Z  INFO engula_server::node::replica::eval::cmd_move_replicas: group 10 replica 102 receive moving replicas requests, incoming [123123], outgoing [103]
2022-09-15T10:04:19.112153Z  INFO engula_server::node::replica::eval::cmd_move_replicas: group 10 replica 102 receive moving replicas requests, incoming [123124], outgoing [103]

Node also receives 2 requests, but the scheduler seems to have some problem in this situation.
(On the other side, if we uncomment the sleep statement, test case should works as desired)

5. 2nd move_replicas request acquire acl_guard in group10's leader and set up the new move_replica task and waiting for node_scheduler to handle the task, but now node_scheduler is handled 1st request's action_task,  so 2nd request waiting for 1st request's job in logical here

This step should be the crux of the problem, the second operation should not be blocked, instead it should return an error like busy directly. I'll see how to fix this.


Make legacy scopmath solvers header only, drop kinderiv.py and dimplic.cpp

Description
Move scopmath solvers into header files, drop kinderiv.py/dimplic.cpp workarounds.
TODO:

[ ] Update submodule once https://github.com/BlueBrain/mod2c/pull/81 is merged.
[ ] Change NMODL code generation

Use certain branches in CI pipelines.

CI_BRANCHES:NEURON_BRANCH=master,NMODL_BRANCH=olupton/scopmath-inline,SPACK_BRANCH=develop

It seems like the simulation stack is failing with unrelated errors and could be ignored.


SymbolPropertiesDataBase and MarketHourDataBase get outdated

In the data folder the lean-CLI has a copy of the SPDB and MHDB get outdated and are not refreshed with the latest version from Lean

Hey team! I have custom market hours in the MHDB. Can I prevent lean from overwriting these?


Make osbs ironman command bypassage

Description:
Currently when a user makes a new ironman alt, they have to perform many steps:
0. Ask a mod to give them bronze role so they can talk in server.

Ask a mod to bypassage command them
Run +m ironman, and confirm
Ask a mod to be bypassed again, because +m ironman resets everyting
Run +m ironman --permanent. (--permanent flag only works if you're already iron)
Ask mod to link accounts.

Changes

Adds BypassAge account bitfield when an OSB (+) user runs +m ironman

Other checks:

Copy/pasted working and tested code.

Looks like magna is solving this a different way


Rarity configuration update

rarity.config.json generation and check rarity should be in one command.
Rarity check should run on DNA database, not on generated metadata json files.

Issues fixed. in version 0.2.4


Unable to install kubewarden-defaults

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Following installation steps described in https://www.kubewarden.io/ produces error.
~ e2e-tests:main* ε helm install --create-namespace -n kubewarden kubewarden-crds kubewarden/kubewarden-crds
NAME: kubewarden-crds
LAST DEPLOYED: Mon Nov  7 10:58:28 2022
NAMESPACE: kubewarden
STATUS: deployed
REVISION: 1
TEST SUITE: None

~ e2e-tests:main* ε helm install --wait -n kubewarden kubewarden-controller kubewarden/kubewarden-controller
NAME: kubewarden-controller
LAST DEPLOYED: Mon Nov  7 10:58:35 2022
NAMESPACE: kubewarden
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
kubewarden-controller installed.

You can start defining cluster admission policies by using the
`clusteradmissionpolicies.policies.kubewarden.io` resource.

For more information checkout https://kubewarden.io/

> **Warning:** the `ClusterAdmissionPolicy` resources of version
> `v1alpha1` are now removed. Refer to the `v0.2.0` release notes to
> find out how to migrate them to `v1alpha2` if you still need to do
> so.

~ e2e-tests:main* ε helm install --wait -n kubewarden kubewarden-defaults kubewarden/kubewarden-defaults                                                                             [13s 846ms]
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "default" namespace: "" from "": no matches for kind "PolicyServer" in version "policies.kubewarden.io/v1"
ensure CRDs are installed first

Expected Behavior
No response
Steps To Reproduce
No response
Environment
- OS:
- Architecture:

Anything else?
No response

Fixed by https://github.com/kubewarden/helm-charts/pull/147, thanks.


addons/packages/pinniped/0.12.1: fix update-strategy-overlay.yaml matching

addons/packages/pinniped/0.12.1: fix update-strategy-overlay.yaml matching
Signed-off-by: Benjamin A. Petersen<EMAIL_ADDRESS>What this PR does / why we need it
The recently added update-strategy-overlay.yaml assumed a single match for deployments, but pinniped package may have several. This resulted in a package error:
      Document on line overlay/update-strategy-overlay.yaml:14:
        Expected number of matched nodes to be 1, but was 3 (lines: overlay/dex-deployment.yaml:7, upstream/_ytt_lib/concierge/deployment.yaml:102,

Which issue(s) this PR fixes

Fixes: #
Describe testing done for PR

Special notes for your reviewer

Just noticed https://github.com/vmware-tanzu/community-edition/pull/4230 and assessing


Bugfix: Ternary Conditionals Inside OSMD Component setupOsmd Method

Update ternary conditionals to function correctly. Previously they would return true even if the prop was defined as 'false' when constructing the component.

Good catch, thanks! Merging.


Unable to access the Google Cloud Client Library [PHP] • API Documentation

Infinite loop encountered while trying to access the API Documentation for the PHP Google Cloud Client Library.
URL: https://googleapis.github.io/google-cloud-php/#/docs/google-cloud/latest/servicebuilder

Hi @friendlypoints,
Thanks for bringing this to our attention, we're looking into it! In the meantime, the second most-recent version documentation is available here.
I'm sorry for the inconvenience!

Thanks for the report :), there was a hiccup in the gh-pages build that has been resolved. Please make sure to refresh your browser cache and this should be all set!


[BUG] Non-lethal Error parsing Modern Door variants on a server

Thank you for reporting!
For issues:

If the issue seems to affect only one mod, report to them instead of here.
Remove Dramatic Doors to see if they are causing the issue before reporting.
If you crashed, please paste the crash log to gist and link it here.

Important Information
Minecraft Version: 1.19.2
Mod Loader System: Fabric
DramaticDoors Version: 2.0.1
Describe the bug
Error parsing Modern Door variants on a server
To Reproduce
Steps to reproduce the behavior:

Launch a server
Receive error

Expected behavior
The server is still able to launch, and the doors seem to be functional, so I'm not exactly sure what this error is for -- just thought you may be interested.
Screenshots

I am getting similar Parsing errors
https://mclo.gs/ObIJ2F9


FHIRCodePrimitive validation

Hey @dcarbone ,
I noticed you removed _isValid() method from FHIRCodePrimitive sub-classes (those List classes). I also noticed you added some prep work in order to implement validation of data for these classes. Any idea when this would be implemented? Thanks.

Not just yet, that functionality is still a bit in-flux at the moment.  More immediately, I am working on a way to generate better tests to validate the json unmarshalling -> marshalling workflow as part of the phpunit test cases.
I would like to have something in place within the next few weeks, however.

@shehi: i have finished the initial validation implementation with v2.0.4.
Let me know if this fits your purposes.

Works well, thanks a lot !


Threat 'other symbols' as letters

The unicode class 'other symbols' contains emojis, math symbols, etc. Treat these as letter characters to support searching for them.

Yes, they should always be separate tokens, and so must be treated as a separate class, which required some additional changes.


Pushing to GCR errors with "docker-credential-gcloud": executable file not found in $PATH

$ gcloud version
Google Cloud SDK 304.0.0
bq 2.0.58
core 2020.07.31
gsutil 4.52

wasm-to-oci version 0.1.1
Running wasm-to-oci push target/wasm32-wasi/debug/demo.wasm gcr.io/<registry_path> returns
`Error: error getting credentials - err: exec: "docker-credential-gcloud": executable file not found in $PATH, out: ```
docker pushing to gcr.io works fine.

Hi, @glennslaven!
I am not familiar with how docker-credential-gcloud is working - wasm-to-oci reads the ~/.docker/config.json file and uses the credentials there to push to the registry - in short, if you logged in to the registry, and the registry supports OCI Artifacts, it should work.
I suspect that for GCR repositories, the credentials are retrieved just-in-time by the docker-credential-gcloud utility - I think that if you login using the access token, as described in the GCR authentication docs, this should work.

I can confirm it works fine logging in with Docker using the access token

Thanks a lot for the confirmation, @jiayihu!
Closing for now, but feel free to reopen if this is still an issue.


envoy: Update to release 1.21.0

Update Envoy to release 1.21.0. Envoy Go API is updated to contain
the generated validation code.
cilium-envoy image is updated to support the newEgressMarkSourceEndpointId
option for the Cilium listener filter. NPDS field 'Policy' is renamed as
'EndpointID'. 'Policy' field was not used for anything, so might as
well recycle it while this API is not yet public.
Envoy retries may fail on "address already in use" when the original
source address and port are used on upstream connections. Cilium
typically does this in the egress proxy listeners. Fix this by using a
Cilium Envoy build that always sets SO_REUSEADDR when original source
address and port is used.
Signed-off-by: Jarno Rajahalme<EMAIL_ADDRESS>Cilium host proxy is updated to Envoy release 1.21.0

/test

/test-only  --focus="K8sPolicyTest.*using namespace label and L7"

/test-only --focus="K8sPolicyTest.*using.namespace.label.and.L7"

Focused test run "failed" due to artifact collection failing when no tests were run:
09:59:19  Ran 0 of 401 Specs in 9.062 seconds
09:59:19  SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 401 Skipped
09:59:19  PASS

Fixed typo.

/test

runtime test failed with unrelated error:
14:44:28      runtime: /tmp/provision/compile.sh: line 28: cd: /home/vagrant/go/src/github.com/cilium/cilium: Stale file handle

This likely indicates a problem in the VM provisioning.

/test-runtime

Multicluster workflow was broken on master for a while, fixed in https://github.com/cilium/cilium/pull/18808 (merged yesterday). It's likely only failing because the PR has not been rebased. I think we're good to optimistically just merge this to unblock next steps.


Implemented client version metric

This PR addresses issue #1483. This PR adds a new middleware to collect and parse the user-agent from client requests and increment a counter metric with the label being the client version.

I'll leave this to @jhiemstrawisc for the final round of review as he has more context to the PR than me at this point, but I hope the changes should be minor


Pubsub stops receiving messages

I have storage as my publisher and a subscriber on the compute engine as the receiver.
Below is the subscriber code that I use.
def receive_messages_with_flow_control(project, subscription_name):
    """Receives messages from a pull subscription with flow control."""
    subscriber = pubsub_v1.SubscriberClient()
    subscription_path = subscriber.subscription_path(
        project, subscription_name)
    idocStorageAbsolutePath = config.idocsStorageAbsolutePath
    makeDirIfNotExist(idocStorageAbsolutePath)
    def callback(message):
        storage_data = message.data
        process_data(storage_data,idocStorageAbsolutePath)
        message.ack()

    # Limit the subscriber to only have ten outstanding messages at a time.
    flow_control = pubsub_v1.types.FlowControl(max_messages=50)
    subscriber.subscribe(
        subscription_path, callback=callback, flow_control=flow_control)

    # The subscriber is non-blocking, so we must keep the main thread from
    # exiting to allow it to process messages in the background.
    logger.info('Listening for messages on {}'.format(subscription_path))
    while True:
        time.sleep(5)

After a day or two of the subscriber running, it stops consuming the messages. I have to manually restart it to make it work again.
Please suggest what I am doing wrong.
PS: The data flowing speed into the topic is high during the day and reduces at night.

@anmol1vw13 , which version of google-cloud-pubsub are you using? Could you please share the output of pip freeze?

I have the same issue, this is my pip freeze:
backports-abc==0.5
cachetools==2.0.1
certifi==2017.11.5
chardet==3.0.4
cognitive-luis==0.1
colorama==0.3.2
dill==<IP_ADDRESS>
ecdsa==0.11
enum34==1.1.6
euca2ools==3.1.0
funcsigs==1.0.2
future==0.16.0
futures==3.2.0
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
gcimagebundle==1.3.1
google-api-core==0.1.4
google-auth==1.4.1
google-cloud==0.32.0
google-cloud-bigquery==0.28.0
google-cloud-bigquery-datatransfer==0.1.1
google-cloud-bigtable==0.28.1
google-cloud-container==0.1.1
google-cloud-core==0.28.1
google-cloud-datastore==1.4.0
google-cloud-dns==0.28.0
google-cloud-error-reporting==0.28.0
google-cloud-firestore==0.28.0
google-cloud-language==1.0.1
google-cloud-logging==1.4.0
google-cloud-monitoring==0.28.1
google-cloud-pubsub==0.33.1
google-cloud-resource-manager==0.28.1
google-cloud-runtimeconfig==0.28.1
google-cloud-spanner==0.29.0
google-cloud-speech==0.30.0
google-cloud-storage==1.6.0
google-cloud-trace==0.17.0
google-cloud-translate==1.3.1
google-cloud-videointelligence==1.0.1
google-cloud-vision==0.29.0
google-gax==0.15.16
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
googlemaps==2.5.1
grpc-google-iam-v1==0.11.4
grpcio==1.11.0
html5lib==0.999
httplib2==0.11.3
idna==2.6
intervaltree==2.1.0
Jinja2==2.10
lxml==3.4.0
MarkupSafe==1.0
oauth2client==3.0.0
paramiko==1.15.1
ply==3.8
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-error-reporting-v1beta1==0.15.3
proto-google-cloud-logging-v2==0.91.3
protobuf==3.5.2.post1
psutil==5.4.5
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycrypto==2.6.1
python-apt==<IP_ADDRESS>
pytz==2018.4
PyYAML==3.11
requestbuilder==0.2.3
requests==2.18.4
rsa==3.4.2
singledispatch==<IP_ADDRESS>
six==1.10.0
sortedcontainers==1.5.7
tornado==4.5.3
urllib3==1.22
virtualenv==1.11.6

I upgraded to google-cloud-pubsub==0.33.1, but this issue isn't solved yet. My callback is the same (process the data, ack the message).

Hi there, please try using the subscribe_experimental function present in pubsub 0.34.0.  This will soon become the default behavior in 0.35.0 and it resolves several issues with streaming pull. If you're still experiencing issues after that, let us know and we'll re-open.

Sorry for the delayed response.
pip freeze:
acme==0.14.2
Babel==1.3
beautifulsoup4==4.4.1
boto==2.38.0
cachetools==2.0.1
certbot==0.14.2
certbot-apache==0.14.2
certifi==2018.1.18
chardet==3.0.4
ConfigArgParse==0.11.0
configobj==5.0.6
crcmod==1.7
cryptography==1.7.1
dnspython==1.15.0
docutils==0.12
enum34==1.1.2
feedparser==5.1.3
funcsigs==0.4
futures==3.2.0
gdata==2.0.18
google-api-core==0.1.4
google-api-python-client==1.6.5
google-auth==1.4.1
google-cloud-core==0.28.1
google-cloud-datastore==1.5.0
google-cloud-pubsub==0.32.0
google-cloud-storage==1.7.0
google-compute-engine==2.3.6
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4
grpcio==1.10.0
gyp==0.1
html5lib==0.999
httplib2==0.11.1
idna==2.6
ipaddress==1.0.17
Jinja2==2.8
linecache2==1.0.0
lxml==3.5.0
Mako==1.0.3
MarkupSafe==0.23
mock==1.3.0
ndg-httpsclient==0.4.2
oauth2client==4.1.2
parsedatetime==2.1
pbr==1.8.0
Pillow==3.1.2
protobuf==3.5.2.post1
psutil==5.4.3
psycopg2==2.6.1
pyasn1==0.4.2
pyasn1-modules==0.2.1
PyChart==1.39
pycrypto==2.6.1
pycurl==7.43.0
pydot==1.0.29
Pygments==2.1
PyICU==1.9.2
pyinotify==0.9.6
PyMySQL==0.8.0
PyOpenGL==3.0.2
pyOpenSSL==17.0.0
pyparsing==2.0.3
Pyrex==<IP_ADDRESS>
pyRFC3339==1.0
python-apt==1.1.0b1
python-augeas==0.5.0
python-dateutil==2.4.2
python-http-client==2.4.0
python-ldap==2.4.22
python-openid==2.2.5
python-stdnum==1.2
python2-pythondialog==3.3.0
pytz==2014.10
PyWebDAV==0.9.8
PyYAML==3.11
reportlab==3.3.0
requests==2.18.4
roman==2.0.0
rsa==3.4.2
sendgrid==4.2.0
simplejson==3.8.1
six==1.10.0
traceback2==1.4.0
unittest2==1.1.0
untangle==1.1.1
uritemplate==3.0.0
urllib3==1.22
uTidylib==0.2
vatnumber==1.2
vobject==0.8.1rc0
Werkzeug==0.10.4
xlwt==0.7.5
zope.component==4.2.2
zope.event==4.2.0
zope.hookable==4.0.4
zope.interface==4.1.3
ZSI==2.1a1

Hi, getting an exception with 0.34.0:
[2018-05-13 12:30:14,744 - ERROR - Thread-17 - _channel] Exception iterating requests!
Traceback (most recent call last):
  File "/Users/alonshomrat/Library/Python/2.7/lib/python/site-packages/grpc/_channel.py", line 187, in consume_request_iterator
    request = next(request_iterator)
  File "/Users/alonshomrat/Library/Python/2.7/lib/python/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 100, in __iter__
    yield self._initial_request()
  File "/Users/alonshomrat/Library/Python/2.7/lib/python/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 313, in _get_initial_request
    lease_ids = self._leaser.ack_ids
AttributeError: 'NoneType' object has no attribute 'ack_ids'

I think it's the CleanupThread.
Created a script to reproduce:

Create topic
Insert message to it.
Create subscription.
Start subscribe.
Get exception (CleanupThread in _channel -> grpc package).

# Create a topic
# Insert data to the topic
# Create subscription and connect it to the topic
# Start subscriber
# Get exception
from google.cloud.pubsub_v1 import PublisherClient, SubscriberClient

publisher_obj = PublisherClient()
subscriber_obj = SubscriberClient()
topics_path = [u'projects/your-project-name/topics/topic_general']
subscribers_path = [u'projects/your-project-name/subscriptions/instance_general']

def create_topics():
    for topic in topics_path:
        try:
            publisher_obj.create_topic(topic)
        except Exception as e:
            print("Topic {}: {}. Notice: don't forget to run pubsub emulator before".format(topic, e.message))

def publish_messages(data, message_type):
    response = publisher_obj.publish(topics_path[0], data=data, type=message_type)
    return response

def create_subscriptions():
    subscriptions = []
    for subscribe, topic in zip(subscribers_path, topics_path):
        subscriptions.append(subscriber_obj.create_subscription(subscribe, topic))
    return subscriptions

def start_subscribe():
    subscription_path = SubscriberClient().subscription_path("your-project-name", "instance_general")
    _ = SubscriberClient().subscribe(subscription_path, callback=lambda x: x)

def pubsub_error():

    create_topics()
    a = publish_messages(data='{"key3": "value3", "key2": "value2", "key1": "value1"}', message_type="/webhook/webhook_type")
    create_subscriptions()
    start_subscribe()

if __name__ == "__main__":
    pubsub_error()

@theacodes What I've done is replace subscriber.subscribe(...) with subscriber.subscribe_experimental(...).
Is that correct? I didn't find any exception while running it.
Btw, could you tell me what's different between subscribe and subscribe_experimental? I mean what in subscriber() was causing it to stop!

@theacodes @crwilcox Please reopen the issue.  Below is the part of the code which has subscriber.subscribe changed to subscriber.subscribe_experimental.
The script this time, stopped after just a few hours. I had to restart it to get it to start working.
Need your help to figure this out.
`def receive_messages_with_flow_control(project, subscription_name):
"""Receives messages from a pull subscription with flow control."""
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
idocStorageAbsolutePath = config.idocsStorageAbsolutePath
makeDirIfNotExist(idocStorageAbsolutePath)
def callback(message):
storage_data = message.data
process_data(storage_data,idocStorageAbsolutePath)
message.ack()
# Limit the subscriber to only have ten outstanding messages at a time.
flow_control = pubsub_v1.types.FlowControl(max_messages=50)
subscriber.subscribe_experimental(
    subscription_path, callback=callback, flow_control=flow_control)

# The subscriber is non-blocking, so we must keep the main thread from
# exiting to allow it to process messages in the background.
logger.info('Listening for messages on {}'.format(subscription_path))
while True:
    time.sleep(5)`

@anmol1vw13 sorry for the delay. Both @theacodes and myself are at PyCon US so responses may be a bit more asynchronous than normal.
Thanks for trying out the experimental method. As @theacodes said, that is the future implementation so figuring out how that implementation works would be very beneficial to us and hopefully other users.
Could you describe what you mean by 'it stopped'? Is the process terminating, are you monitoring unprocessed messages?

@crwilcox Thanks for the response. Badly need it.
The process is still alive but there are unprocessed messages left.
If you could help me in getting the process killed when the messages stop being consumed, that would do too. I say this because, I have a cron which restarts the process if the process isn't alive.

I haven't put any logger.debug statements in my code. All my logs are of info level. Would debug level logs do any good?

Yes, the subscriber does a significant amount of logging specifically to
help in this case.
On Sun, May 13, 2018, 2:25 PM anmol1vw13<EMAIL_ADDRESS>wrote:

I haven't put any logger.debug statements in my code. All my logs are of
info level. Would debug level logs do any good?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5314#issuecomment-388646189,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUcwBenc8kIpf-qICzQT_WnoHQNRTPks5tyHqhgaJpZM4T3wEP
.

PFB, the logger code I had written. I changed the logging level to debug but it still isn't giving the subscriber logs in.
This project was the first time I put logging into a python code. Prior to that I mostly used print statements.
Please let me know where I am wrong.
import logging
import logging.config
import time
import os
# def logger():
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

try:
    os.mkdir("../logs")
except:
    None
hdlr = logging.FileHandler('../logs/topicSubscriber.log')
# os.environ["TZ"] = "US/Pacific"
# logging.Formatter.

formatter = logging.Formatter('[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s','%Y-%m-%d %H:%M:%S')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.info('Completed configuring logger()!')

You may need to config the root logger. Please try logging.basicConfig.
On Sun, May 13, 2018, 2:34 PM anmol1vw13<EMAIL_ADDRESS>wrote:

PFB, the logger code I had written. I changed the logging level to debug
but it still isn't giving the subscriber logs in.
This project was the first time I put logging into a python code. Prior to
that I mostly used print statements.
Please let me know where I am wrong.
import logging
import logging.config
import time
import os
def logger():
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
try:
os.mkdir("../logs")
except:
None
hdlr = logging.FileHandler('../logs/topicSubscriber.log')
os.environ["TZ"] = "US/Pacific"
logging.Formatter.
formatter = logging.Formatter('[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s','%Y-%m-%d %H:%M:%S')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)logger.info('Completed configuring logger()!')
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5314#issuecomment-388646760,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc1_6qJwLko4tIlKBULiQ_pR2qa1fks5tyHy8gaJpZM4T3wEP
.

So, I have got the debug logging working and yes, the subscriber logs are coming in. But, it stops arbitrarily in a few hours and I don't know how to notify myself when that happens. If I could know that, I might as well could kill the process :)
Do you have an idea of how that could be done?
If not, I'll check up on it after a few hours.

It stopped.
The logs: https://gist.github.com/anmol1vw13/eea5d5a8a15798e2d5221a67adb444b0

Thanks, @anmol1vw13. Your logs confirm this is a duplicate of #5306. I'm closing this - you can follow the progress of the fix over there.

@theacodes I'm not using subscriber_experimental coz it's not very stable right now but would continue to use subscriber present in 0.32.0.
All I want to know is, when subscriber stops it gives out the following in the logs. How should I handle it, so I'm able to restart the subscriber?
[2018-05-16 21:24:03] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/thread.py:291} DEBUG - Handling 1 batched requests
[2018-05-16 21:24:03] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_consumer.py:246} DEBUG - Sending request on stream
[2018-05-16 21:24:06] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:362} DEBUG - The current p99 value is 10 seconds.
[2018-05-16 21:24:06] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:368} DEBUG - Renewing lease for 0 ack IDs.
[2018-05-16 21:24:06] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:388} DEBUG - Snoozing lease management for 8.475584 seconds.
[2018-05-16 21:24:14] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:362} DEBUG - The current p99 value is 10 seconds.
[2018-05-16 21:24:14] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:368} DEBUG - Renewing lease for 0 ack IDs.
[2018-05-16 21:24:14] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:388} DEBUG - Snoozing lease management for 5.516442 seconds.
[2018-05-16 21:24:20] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:362} DEBUG - The current p99 value is 10 seconds.
[2018-05-16 21:24:20] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:368} DEBUG - Renewing lease for 0 ack IDs.
[2018-05-16 21:24:20] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:388} DEBUG - Snoozing lease management for 4.373885 seconds.
[2018-05-16 21:24:22] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_consumer.py:399} DEBUG - Resuming consumer
[2018-05-16 21:24:22] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_consumer.py:440} DEBUG - Stopping helper thread Thread-ConsumeBidirectionalStream
[2018-05-16 21:24:22] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/_consumer.py:243} DEBUG - Request generator signaled to stop.
[2018-05-16 21:24:24] {/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/subscriber/policy/base.py:355} DEBUG - Consumer inactive, ending lease maintenance.

@theacodes @crwilcox Anything on this?

@crwilcox Where should this be checked? In the while loop that I have.(I have given the sample code above)
Could you provide me with an example of how it must be used.
Also, is there any other way? subscriber_experimental isn't reliable.

@crwilcox @theacodes Please help me out people?


Update provision-workspace.md

Add note about lack of support for multiple Workspaces
Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the CODEOWNERS.
What Type of PR is This?

[ ] Issue
[ ] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[ ] Tested Locally
[ ] Optional Screen Shoot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://658223101a8b8e7c04402791--harness-developer.netlify.app


Allow for separate inboard and outboard first wall builds

Description of issue / requirement to address
PROCESS currently only assumes an identical inboard and outboard first wall in terms of pipe size, pitch armour etc.
It would be good to enable a separate set of variables for the inboard and outboard builds. This will allow for more accurate radial builds and pumping requirements.

Will need to check blanket changes.
Possibly add different fw_wall variations for plasma and non-plasma facing sides

Proposed solution
Create a separate set of variables for the inboard and outboard FW parameters

@chris-ashe @geograham I see a branch with changes here, is this issue still going on?

Chris has said conversion of that could be affected by this can go ahead


Fixed some paths related to managing-a-custom-domain-for-your-github-pages-site

Why: Because I was trying to access to this page and I can't do it!

Closes [issue link]
What's being changed: The spanish version of Troubleshooting custom domains and GitHub Pages

Check off the following:

[x] I have reviewed my changes in staging. (look for the deploy-to-heroku link in your pull request, then click View deployment)
[x] For content changes, I have reviewed the localization checklist
[x] For content changes, I have reviewed the Content style guide for GitHub Docs.

@ChristianECG Thank you so much for opening a PR to try and make this fix! I'm not sure why that link isn't redirecting correctly 🤔
Do you want to try opening a PR on the English page to update the link to the current relative link, which should be /pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site.
From there the change will go to Crowdin, our translation service, and hopefully be updated in the Spanish version soon 💛


[plugin] firebase_messaging crashes the app when it is closed with back button (android)

Firebase messaging is crashing on android devices when app is put in background by pressing physical back button.
Steps to Reproduce

Get firebase_messaging example project
Make it work by adding google_services and all that stuff
Once app runs, press back, it will close the app (put it in background), open up  the app again.

Logs
Run your application with flutter run and attach all the log output.
Launching lib/main.dart on SM G935F in debug mode...
Initializing gradle...                                1.0s
Resolving dependencies...                             1.4s
Running 'gradlew assembleDebug'...                    2.5s
Built build/app/outputs/apk/app-debug.apk (24.1MB).
I/FlutterActivityDelegate(14995): onResume setting current activity to this
Syncing files to device SM G935F...
I/flutter (14995): Push Messaging token: eEetlfVtUtU:APA91bEC-QqyMCDNkAerg-FvcAq7X-_EW0CaRaVPRXc-8pGLxb5k8lznYNgxvsxs1azMbzUw3CQOgaoBa5yMdapTNoFD2tDX0YvaiOZ5_JO85YTdcvbuvA-bvFnGTtwlh9zp4akbx1mf
D/libGLESv2(14995): STS_GLApi : DTS is not allowed for Package : com.yourcompany.notificationtest

🔥  To hot reload your app on the fly, press "r". To restart the app entirely, press "R".
An Observatory debugger and profiler on SM G935F is available at: http://<IP_ADDRESS>:8105/
For a more detailed help message, press "h". To quit, press "q".
D/ViewRootImpl@17fb4af[MainActivity](14995): ViewPostImeInputStage processKey 0
D/ViewRootImpl@17fb4af[MainActivity](14995): ViewPostImeInputStage processKey 1
V/FA      (14995): Recording user engagement, ms: 4044
V/FA      (14995): Activity paused, time: 668671972
D/FA      (14995): Logging event (FE): user_engagement(_e), Bundle[{firebase_event_origin(_o)=auto, engagement_time_msec(_et)=4044, firebase_screen_class(_sc)=MainActivity, firebase_screen_id(_si)=4669346971372851144}]
D/ViewRootImpl@17fb4af[MainActivity](14995): MSG_WINDOW_FOCUS_CHANGED 0
D/InputTransport(14995): Input channel destroyed: fd=90
D/ViewRootImpl@17fb4af[MainActivity](14995): dispatchDetachedFromWindow
D/InputTransport(14995): Input channel destroyed: fd=74
V/FA      (14995): onActivityCreated
I/FlutterActivityDelegate(14995): onResume setting current activity to this
V/FA      (14995): Activity resumed, time: 668674299
D/InputTransport(14995): Input channel constructed: fd=81
D/ViewRootImpl@e1e0184[MainActivity](14995): setView = DecorView@7550d6d[MainActivity] touchMode=true
D/FA      (14995): Logging event (FE): screen_view(_vs), Bundle[{firebase_event_origin(_o)=auto, firebase_previous_class(_pc)=MainActivity, firebase_previous_id(_pi)=4669346971372851144, firebase_screen_class(_sc)=MainActivity, firebase_screen_id(_si)=4669346971372851145}]
D/mali_winsys(14995): EGLint new_window_surface(egl_winsys_display*, void*, EGLSurface, EGLConfig, egl_winsys_surface**, egl_color_buffer_format*, EGLBoolean) returns 0x3000,  [720x1280]-format:1
D/mali_winsys(14995): EGLint new_window_surface(egl_winsys_display*, void*, EGLSurface, EGLConfig, egl_winsys_surface**, egl_color_buffer_format*, EGLBoolean) returns 0x3000,  [720x1280]-format:1
D/ViewRootImpl@e1e0184[MainActivity](14995): MSG_RESIZED_REPORT: ci=Rect(0, 42 - 0, 0) vi=Rect(0, 42 - 0, 0) or=1
D/ViewRootImpl@e1e0184[MainActivity](14995): MSG_WINDOW_FOCUS_CHANGED 1
V/InputMethodManager(14995): Starting input: tba=android.view.inputmethod.EditorInfo@94271ee nm : com.yourcompany.notificationtest ic=null
I/InputMethodManager(14995): [IMM] startInputInner - mService.startInputOrWindowGainedFocus
D/InputTransport(14995): Input channel constructed: fd=92
D/AndroidRuntime(14995): Shutting down VM
E/AndroidRuntime(14995): FATAL EXCEPTION: main
E/AndroidRuntime(14995): Process: com.yourcompany.notificationtest, PID: 14995
E/AndroidRuntime(14995): java.lang.NullPointerException: Attempt to invoke virtual method 'void io.flutter.view.FlutterNativeView.send(java.lang.String, java.nio.ByteBuffer, io.flutter.plugin.common.BinaryMessenger$BinaryReply)' on a null object reference
E/AndroidRuntime(14995):        at io.flutter.view.FlutterView.send(FlutterView.java:841)
E/AndroidRuntime(14995):        at io.flutter.plugin.common.MethodChannel.invokeMethod(MethodChannel.java:82)
E/AndroidRuntime(14995):        at io.flutter.plugin.common.MethodChannel.invokeMethod(MethodChannel.java:69)
E/AndroidRuntime(14995):        at io.flutter.plugins.firebasemessaging.FirebaseMessagingPlugin.onReceive(FirebaseMessagingPlugin.java:60)
E/AndroidRuntime(14995):        at android.support.v4.content.LocalBroadcastManager.executePendingBroadcasts(LocalBroadcastManager.java:297)
E/AndroidRuntime(14995):        at android.support.v4.content.LocalBroadcastManager.access$000(LocalBroadcastManager.java:46)
E/AndroidRuntime(14995):        at android.support.v4.content.LocalBroadcastManager$1.handleMessage(LocalBroadcastManager.java:116)
E/AndroidRuntime(14995):        at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime(14995):        at android.os.Looper.loop(Looper.java:154)
E/AndroidRuntime(14995):        at android.app.ActivityThread.main(ActivityThread.java:6692)
E/AndroidRuntime(14995):        at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime(14995):        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1468)
E/AndroidRuntime(14995):        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1358)
Lost connection to device.

Run flutter analyze and attach any output of that command also.
Analyzing /Users/snoopy/Documents/dreamImpl/notification_test...
No issues found!
Ran in 5.4s

Flutter Doctor
[✓] Flutter (on Mac OS X 10.13.1 17B48, locale en-HR, channel master)
    • Flutter at /Users/snoopy/flutter
    • Framework revision d79c12e5b8 (27 hours ago), 2017-11-22 17:35:35 +0100
    • Engine revision 93b2179597
    • Tools Dart version 1.25.0-dev.11.0

[✓] Android toolchain - develop for Android devices (Android SDK 26.0.2)
    • Android SDK at /Users/snoopy/Library/Android/sdk
    • Android NDK at /Users/snoopy/Library/Android/sdk/ndk-bundle
    • Unable to locate compiler in Android NDK.

    • Platform android-26, build-tools 26.0.2
    • ANDROID_HOME = /Users/snoopy/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_112-release-b06)

[✓] iOS toolchain - develop for iOS devices (Xcode 9.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 9.1, Build version 9B55
    • ios-deploy 1.9.2
    • CocoaPods version 1.3.1

[✓] Android Studio
    • Android Studio at /Applications/Android Studio 3.1 Preview.app/Contents
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1012-b01)

[✓] Android Studio (version 2.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Java version OpenJDK Runtime Environment (build 1.8.0_112-release-b06)

[✓] IntelliJ IDEA Community Edition (version 2017.2.6)
    • Flutter plugin version 19.1
    • Dart plugin version 172.4343.25

[✓] Connected devices
    • SM G935F • 9885b651454a4a3034 • android-arm • Android 7.0 (API 24)

I was just about to raise this same issue.
To me it looks like the issue might be something to do with a new flutter view being initialized when the phone app is re-opened (I get the white 'loading' sort of screen for while flutter is initializing) and maybe the plugin keeps a reference to the old FlutterView which has been detached/closed at that point.

Yeah, I think it's something with not putting it in background correctly.
Workaround I've found:
@override
  didPopRoute() {
    Navigator.of(context).pop();
    exit(0);
    return new Future(() => true);
  }

While it does end abruptly it works, opening app again won't crash it.
I've used exit() because SystemNavigator.pop() didn't work.

cc @collinjackson

@knezzz I could not reproduce the issue with the latest plugins. Is it still happening to you?

@cyanglaz Nope, I can't reproduce it with latest plugins, seems to be fixed :)

@knezzz Thank you for the quick response.
Seems the issue is fixed, closing the issue ticket.
Please feel empowered to reopen with details if the same issue is found again.

@knezzz
This issue has been moved to https://github.com/FirebaseExtended/flutterfire/issues/933. Any further collaboration will be done there.


Backend Support of Planner infotips

As a part of the story https://openshift.io/openshiftio/openshiftio/plan/detail/1744 we need backend support for infotips.
Problem
The docs team releases infotips via https://docs.prod-preview.openshift.io/json/infotips.json endpoint. We need to the fetch JSON from the docs endpoint and return them to the front-end.
NOTE - We're not implementing localization/Internationalization support for now. (This part over complicates the problem for now)
Solution
Option 1 - We fetch the JSON data from the docs endpoint at build time and load the database. A table with the following schema should be enough

ID
Infotip-UUID
Term
Description
WorkItemType-UUID

27899b37...
e8d54bf3...
Bug
Defect that ...
26787039...

We return the infotips data in the workItemType response.
Pros - Frontend doesn't need to process the data. The infotip is part of the workitemtype response.
Cons - This solution will work only for workItemTypes. We have infotip for the term Iteration also. This cannot be stored in this table since iteration is not a workitemtype. Right now it has been hardcoded in the frontend
Option 2 - We store the infotips in database as a generic infotip and return all the infotips to the front-end. The front-end maps the id of the infotip to the correct HTML element.
We return infotips through a /api/docs endpoint.

Infotip-UUID
Term
Description

e8d54bf3...
Bug
Defect that ...

Pros - We can store infotips for all terms/elements in the database.
Cons -

Frontend needs to store the mapping between HTML element and infotip UUID.
Frontend needs to process the response from fabric8-wit. This response might grow in size when we have a  large number of infotips.

Issue with both the options
Updating the infotip data. How do we ensure the data between docs endpoint and wit database is consistent? Do we allow manual update with /api/updateDocs that will fetch the latest docs and update the database? Do we periodically check the docs endpoint for update?
@kwk @aslakknutsen @baijum @DhritiShikhar @michaelkleinhenz @sanbornsen I'd like to hear your thoughts on this.

I think second solution with a and update API would be good.

This feature is a GA critical. So our primary goal should be

Keep it as simple as possible for now for us
Keep it as simple as possible for the doc team to add or update info tip.
Updating info tips should be a very seamless process without touching the build and whatnot every time.

So, here is an idea that includes, minor backend work and minor frontend work.

The backend needs to generate the UUID for info tip.
Front-end fetches and looks up to the JSON file for any info tip UUID coming from the backend.
If found, then frontend shows the string from the JSON.
Else front-end shows the raw UUID in the info tip.
Now the Doc team can easily find out the UUID needs to be added in the JSON, by going through the front-end. Once they do it, it'll show up in the front end without any effort.

Since the IDs of group types and work item types are hardcoded in the YAML file, they will never change, so we can directly use that ID for info tips too.
This solution is also scalable for future, so it's not something will be thrown away but will be extended in an efficient way.

I wouldn't make the info tips uuids, but some logical string+number sequence. And probably default to the text that is there now not to break preview/prod in the process. Then use feature flag group internal to show non found mappings, but default for everyone else.

I wouldn't make the info tips uuids, but some logical string+number sequence

@aslakknutsen using UUID allows us to map workitem types directly to it's info tips. Using a local string+number sequence would require us to store the mapping between infotip and workitem types somewhere (maybe backend).

I wouldn't make the info tips uuids, but some logical string+number sequence
That involves some amount of backend work. That exactly what I plan for future. We want to keep it as simple as possible at this moment. The architecture will be same. Now can use the attribute IDs directly.
Then use feature flag group internal to show non found mappings, but default for everyone else.
Again, we can use some default hard coded string for now. We can go with feature falg post summit.

Just a note from the docs perspective: the JSON endpoint was suggested some time ago by the Planner team, but it's not something the Docs team would be hung up on. If you need some other way to consume the content (or some other objects in the JSON), let us know. The JSON works for us because its generation from our source can be reasonably automated, but we're open to other suggestions.

@jarifibrahim I agree with all the points you have mentioned and I also think this is the right way to go ahead but according to me the implementation details and its pros and cons should have been discussed before implementation. Although I am willing to work on it again, it seems to me like an overhead and duplication of work now as the frontend has been implemented and merged.
CC : @michaelkleinhenz @nimishamukherjee

it seems to me like an overhead and duplication of work now as the frontend has been implemented and merged.

@divyanshiGupta can you explain how is it an overhead? This plan will relatively shorten the overall effort of the team. Often we implement something and later realize the flaw in the architecture and redo things. It's good that @jarifibrahim gave it a direction. How would you compare half an hour of work on planner vs a number of changes in the backend?

@sanbornsen I agree, but the only point I want to make here is that we should exhaustively discuss the implementation details before starting the implementation so that we don't have to put twice the effort in a thing that could have been fixed the first time. Shouldn't we avoid that?  Because previously a lot of discussions happened on the need for a backend and then it was decided to go with a backend. I agree, looking at the points mentioned by @jarifibrahim that certainly this is a better way of implementing it but my only concern is that time should not be wasted on deciding upon something then implementing it and then again having the same discussion and changing the previous decision and implementing the whole thing again.

@sanbornsen @jarifibrahim prod preview and production will have the same UUIDs for work item types, work item type groups, etc?

@nimishamukherjee I understand your point. Thank you. Agree that this would be a way forward and a new task in itself, so as I said earlier I would like to work on this task as well since I've already worked on the previous one. Will start the work once all the implementation details are finalised.

Will start the work once all the implementation details are finalised.
@sanbornsen @jarifibrahim can you confirm if the option outlined by @sanbornsen will be taken forward?

@sanbornsen @jarifibrahim can you confirm if the option outlined by @sanbornsen will be taken forward?

I also talked to @Preeticp and it looks fine every way. I'll just check with @rkratky once. Is there any problem with updating the UUID in the JSON if we provide the UUIDs?

@sanbornsen If you need to change the UUIDs (just curious... why?), the best way would be to submit a PR against this file: https://github.com/fabric8io/fabric8-online-docs/blob/master/docs/topics/modules/glossary.adoc -- in the source, you'll see the UUIDs with the respective term definitions.

@sanbornsen @jarifibrahim can you confirm if the option outlined by @sanbornsen will be taken forward?

Yes @nimishamukherjee. We'll go forward with the option https://github.com/fabric8-services/fabric8-wit/issues/2032#issuecomment-383470520 suggested by @sanbornsen

Thanks @jarifibrahim @sanbornsen for closing this quickly.
@divyanshiGupta the implementation details have been finalised. Start work in this on a priority basis.

@nimishamukherjee sure.

Is my opinion required on this any longer. I understand that the backend is not doing anything for this in the beginning, right?

Yes Konrad, we don't need any backend work for now.

I'm closing this issue since we're implementing Infotips in the frontend and we don't need any backend work. Thanks everyone!
Infotips PR in planner - https://github.com/fabric8-ui/fabric8-planner/pull/2595


quiet parameter for cli

add parameter to quiet all outputs for command run.
kirby myJob --quiet
set this somewhere... took this from janitor which also uses CLImate. somewhere like your --global command but it has to affect all commands.
$command = $climate->arguments->get('quiet');
if ($command) {
    $climate->output->add('quiet', new \Kirby\CLI\QuietWriter());
    $climate->output->defaultTo('quiet');
}

<?php

namespace Kirby\CLI;

use League\CLImate\Util\Writer\WriterInterface;

class QuietWriter implements WriterInterface
{
    public function write($content)
    {
        // be quiet here
    }
}

This is now implemented on develop


Unify UX on how different plugins schedule workers/tasks

Description
In each backend plugin we host in Janus-IDP, we offer a different mean to schedule workers. Not only when it comes how and where this is configured, but also how they are scheduled.
See and compare configuration for following plugins:

3Scale https://github.com/janus-idp/backstage-plugins/tree/main/plugins/3scale-backend#configure
Keycloak https://github.com/janus-idp/backstage-plugins/tree/main/plugins/keycloak-backend#installation
OCM https://github.com/janus-idp/backstage-plugins/tree/main/plugins/ocm#setup-janus-idpbackstage-plugin-ocm-backend

Expected behavior
I think 3Scale plugins provides the easiest method to configure the scheduler, please migrate the other plugins to this format:
# app-config.yaml

catalog:
  providers:
    <PLUGIN_NAME>:
      <ENV>:
        schedule: # optional; same options as in TaskScheduleDefinition
          # supports cron, ISO duration, "human duration" as used in code
          frequency: { minutes: 1 }
          # supports ISO duration, "human duration" as used in code
          timeout: { minutes: 1 }

Acceptance Criteria

[ ] Keycloak supports scheduler config through app-config.yaml
[ ] OCM supports scheduler config through app-config.yaml

I can take a look at this.


The inconsistent results of your modified 3D U-Net and mine

Hello,
I try to reproduce the result of the modified 3D U-Net model as described in your paper, and I use the default setting of your code except the loss function (use Dice loss), but I got pretty worse Dice score than yours (summary_report.txt.txt). Could you share your training details or tricks about training this 3D model? Thanks a lot!

There was indeed a bug in the 3D evaluation function. I am surprised you were able to get any Dice scores because it was predicting everything as zero. Anyway, it is now fixed.


[WIP] Add an authenticated HTTP client

Resolves #238. Any requests to the Cloudflare API will now use auth_client. Also added some operating system information to the user agent. Will promote from draft after testing each endpoint.

Also added some operating system information to the user agent

I don't think we need that, I would prefer removing it.
Also could we replace the client() in favor of the auth_client() completely?

Will do on the agent. For the client, there are some requests, like the playground, where we don't need auth and probably shouldn't send the headers.


Unofficial Azure CLI rpm packages are distributed by openSUSE

Similar to https://github.com/Azure/azure-cli/issues/19640.
openSUSE packagers release unofficial Azure CLI rpm packages from their own repository (https://software.opensuse.org/package/azure-cli):

openSUSE Leap 15.2 delivers 2.16.0
openSUSE Tumbleweed delivers 2.31.0

These packages are not maintained by Microsoft, outdated, and contain various bugs:

https://github.com/Azure/azure-cli/issues/20839
https://github.com/Azure/azure-cli/issues/13209

Unofficial Azure CLI rpm packages

Since openSUSE packagers decides to distribute their own Azure CLI packages (https://github.com/Azure/azure-cli/issues/20839#issuecomment-1001517488), currently Microsoft doesn't support Azure CLI on openSUSE or its related systems.


Auth fails: Missing client_token. Please have admin generate a token for you.

Trying to use the Workspaces API now, and it's failing because I don't have a UWS token... stepping back from that, there's an earlier failure when I try to get my token from the Workspaces endpoint. For context, here's the client code, with a couple added print statements:
    token_object = tokens.by_resource_server['groups.api.globus.org']
    groups_token = token_object['access_token']

    auth_token_object = tokens.by_resource_server['auth.globus.org']
    auth_token = auth_token_object['access_token']

    workspaces_post_url = current_app.config['WORKSPACES_ENDPOINT'] + '/tokens/'
    workspaces_post_data = dumps({'auth_token': groups_token})
    print(f'>>> POST {workspaces_post_url}\n{workspaces_post_data}')
    workspaces_post_resp = requests.post(
        workspaces_post_url,
        data=workspaces_post_data)
    print(f'<<< {workspaces_post_resp}')

In the log I see:
>>> POST https://workspaces.api.hubmapconsortium.org/tokens/
{"auth_token": "Ag38y1y--REDACTED--Too8q"}
<<< <Response [400]>
[2022-06-28 17:30:38,385] ERROR in routes_auth: Workspaces auth failed: 400 {"detail":"Missing client_token. Please have admin generate a token for you."}

The string client_token does not appear in out code, so I'm not sure what this is about.

@mccalluc this was a result of prod instance being out of date. Updated and seems to be resolved.


docs for running maven build with services from docker-compose

Added answer to question that was asked in #476

Checklist

[ ] Test your changes as thoroughly as possible before you commit them. Preferably, automate your test by unit/integration tests.
[ ] Make sure mvn install runs for the whole project and, if you touched any code in the respective service, submission and distribution service can be run with spring-boot:run

Description

@nils4cosee or @RomanNess could you please rebase again?


[ota-provider-app] Support nrfConnect

Problem
ota-provider-app does not have an implementation for nrfConnect
Proposed Solution
Support nrfConnect

Spoke with @Damian-Nordic offline. Closing this issue since there are no plans to add OTA Provider on nrfConnect platform as it only supports Thread for now.


Empty commit for rerender

Checklist

[X] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[X] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

It looks like we hit a random rerender glitch in #43 which deleted all the builders, so merging that was a noop. (I didn't notice it because the CI was green, and when the CI goes green it collapses the list of runs.)

Wait, this is still weird... now we only get osx_arm64 builds!?

@conda-forge-admin, please rerender

Ok, the rerender is definitely glitching since it's not idempotent. It looks like this time we have at least most of the builds.


Update to newer Barcode Reader

This screen is out-of-date.  The "Barcode Scanner" has been deprecated and replaced with the "Barcode Reader" and it seems to work differently.  I tried to follow the lab but it didn't seem to work for me with the newer control.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c0d04b70-974d-d5ad-1728-552a45cd0c4f
Version Independent ID: 3989f667-a7d3-111d-9cf2-322a5782638e
Content: Create a canvas app with the barcode reader control (contains video) - Power Apps
Content Source: powerapps-docs/maker/canvas-apps/how-to/create-app-barcode-scanner.md
Service: powerapps
Sub-service: canvas-maker
GitHub Login: @joel-lindstrom
Microsoft Alias: anuitz

Thanks for your comment, @tylerfarmer1. This issue has been assigned to the doc author for review.