Issue 77 - Persistence Query for Cassandra design discussion.

Do not merge.
I have done a few iterations on the Persistence Query api implementation for Cassandra and before spending too much time on it I wanted to discuss possible implementations, because I still don't clearly understand all the implications.
Please have a look at the below and provide feedback, ideas and let me know where I am wrong. When we are happy with the design I will work on the actual implementation and publish the results along the way.
Assumptions
I am not aware of any exposed api for streams subscription in Cassandra so we will have to pull (query the database in configured intervals) or push from write side (see below).
The Cassandra read journal is different from the reference LevelDB read journal, because we it can be distributed and therefore we can not trivially assume shared knowledge.
Implementation
I can think of two options how to implement the read query api given the above assumptions.
1) Pull only approach
CassandraReadJournal would be very similar to LevelDBReadJournal. It would examine the query, parameters and create actor for each query that represents the source. There may be a few potential optimizations (e.g. for not updating queries creating source from iterator provided by Cassandra). Generally however we need to create an actor, because it needs to manage the current state of the stream. By state of the stream I mean the management of already seen elements and only emitting new ones. This may be a Set of already seen elements, an offset etc. Depends on query and data model.
For AllPersistenceIds query a AllPersistenceIdsPublisher actor would be spawned and it would have to maintain the state of the stream. On initialization it would query the database for all currently existing persistenceIds in the data store. Then it would use configured timer to query the database again, but  only publish the new persistenceIds to the stream. It should be possible to further reduce the size of the in memory state and optimize by sharing one set of already seen persistenceIds for all AllPersistenceIds queries, because the resulting events in the stream are the same for all queries (e.g. single Set[String] maintaining known persistenceIds instead of one Set[String] per query). Each query may start at different time and have a different refreshInterval, but that can be handled (i.e. merging the refreshIntervals so for each query the new persistenceIds would be emitted on specified refreshInterval or earlier, but never later). This optimization would only be possible for AllPersistenceIds query. In other cases we would have to manage the state with higher granularity. But such optimization does not change the concept.
Therefore in this implementation there is no push from database to subscribers and pulls are happening after configured intervals. The database here would be represented by CassandraJournal. I am not happy with that - I would either prefer each publisher to talk to database directly or through a read journal. Some of the queries could however use some of the existing methods in CassandraJournal (CassandraRecovery). It would require some refactoring to decouple that functionality from CassandraJournal which I don’t want to immediately do without a review, but could be a way forward.
The advantage is that the implementation is very simple, does not have to manage subscriptions, push from database/write side etc. On the other hand it may be less efficient, because database may be queried even if there were no new events.
2) Push - pull hybrid approach
If my assumption is correct and multiple CassandraJournal actors may live in a clustered system it is not possible to trivially push to read journal the way it is done in the LevelDB read journal implementation. It would only push changes in the local system which is not sufficient (for AllPersistenceIds because other nodes may accept writes and for EventsByPersistenceId even if all updates went through local journal because actors may potenatially be rebalanced).
Instead of a pure pull approach described above we could still consider a hybrid push pull approach similar to the LevelDB implementation. CassandraJournal would then have to manage subscriptions. Then changes going through the write journal could be pushed to the subscribers (e.g. new persistenceId, new event for persistenceId x). We would have to utilize replication, e.g. using Akka Distributed Data to propagate the knowledge about updates to other nodes.
This option would definitely be more complicated to implement. On the other hand it could help to avoid unnecessary database queries. For AllPersistenceIds for example in optimized case there would only be one query to database on CassandraJournal start and then only updates would be received using a replicated Set between the write sides.

@chbatey, @ktoso, @krasserm please review and provide feedback if you are interested.

An example of 1) could be this MongoDB implementation https://github.com/scullxbones/akka-persistence-mongo/tree/wip-akka24/common/src/main/scala/akka/contrib/persistence/mongodb (although it currently does not seem to support refreshing streams). An example of 2) could be the LevelDB implementation https://github.com/akka/akka/tree/master/akka-persistence-query/src/main/scala/akka/persistence/query/journal/leveldb.

This is outdated. Let's unify the conversation in the Issue comments.


Can I customize the /tags and /tags/term pages?

I'd like to replace the /tags page with a tag cloud, and the /tags/term to have the section title added to the link (not all my pages have unique titles). How should I go about this? I can't find where /tags and /tags/term get generated.

Yes, you can. This is not something currently offered by the theme so you would have to implement this yourself. I suggest you start by reading https://gohugo.io/templates/.
Any improvements or features you produce are welcome in pull requests.


Developer tools

Recent changes removed the refresh-views and field-counts commands and their --remove option. Also, individual files don't drop tables before creating them. As such, it's more effort to re-run a single routine/file.
Here are some options for developer tools:

[ ] Add an image to the docs (perhaps created using dot) to show the dependency graph, so that developers know which files to run before others.
[ ] Add a dev command to run the SQL files on which a specific SQL file depends (easy with existing framework – just prune non-dependencies from the dependency graph).
[ ] Add a dev command to "undo" a SQL file (easy to do if no SQL files that depend on it were run (just drop that file's "exports"; otherwise, can use existing framework to identify files that use the file's "exports" and drop their exports).
[ ] Add a dev command to re-run the field-counts routine (just requires DROP field_counts before calling the method)

All the dev commands should be sub-sub-commands e.g. ocdskingfisher-views-cli dev field-counts, to clearly mark them as for development only and not part of the public API, and therefore having no guarantee to exist in future.
As part of this, can move docs_table_ref to that dev scope.

Noting that in my experience, running add-view with —no-field-counts completes in 3s using the test data as documented, so there’s very little motivation to have learn more elaborate commands.

Closing until there is demand.

I added an option to skip a SQL file, so that it can be tested and run manually (e.g. to EXPLAIN different versions of the query): dd03a025f769cf4235b53be96c34082e04ab402f


proposal:  runtime: add support for the Arm’s ArmV8.5-A Memory Tagging Extension (MTE) in Go

What is Memory Tagging Extension (MTE)?
It was first introduced as part of the armv8.5 instruction set in August 2019 and built into the first Armv9 compliant CPUs that were announced in May 2021. It is a security architecture feature to detect and prevent memory safety vulnerabilities before and after deployment.
Why is MTE needed for Go?
Go has structures and its memory layout is visible to the developers. We also provide pointers for accessing some memory directly. This provides a lot of flexibility to the developers and makes the interaction with C code easier. But it also opens the door for the developers to make mistakes on using the memory. So, it is necessary to enable the memory detector in Go.
We have deployed ASan, a software memory error detector. It consists of a compiler instrumentation module and a runtime library. The disadvantage is that ASan comes with a heavy performance cost (it is roughly 2x slower), which makes it unsuitable for widespread deployment. MTE can reduce this performance overhead (Its performance overhead estimate is 5% in asynchronous mode and 25% in synchronous mode) whilst offering some level of protection provides the mechanism to detect out-of-bounds and use-after-free bugs in production code with no instrumentation.
How does MTE work?
At a high level, MTE tags each memory allocation/deallocation with additional metadata. It allocates a tag to a memory location, which then can be associated with pointers that reference that memory location. At runtime the CPU checks that the pointer and the metadata tags match on each load and store.
There are two types of tagging, one is Address Tagging, it adds four bits to the lowest 4-bit of the top-byte of every pointer. Another is Memory Tagging, it also consists of four bits, linked with every aligned 16-byte region in the application’s memory space. To implement the address tagging bits without requiring larger pointers, MTE only works with 64-bit applications since it uses the Top Byte Ignore (TBI) feature, which is an feature of Arm 64bit Architecture.
MTE offers a tuneable level of performance, protection, and precision at runtime, it has 4 operating modes: OFF, Asynchronous mode, Synchronous mode, and Asymmetric mode (this is added in MTE3, for more information, see chapter D9 in Arm Architecture Reference Manual for A-profile architecture).
The details are explained in Armv8.5-A Memory Tagging Extension.
Prototype heap tagging in Go.
As we know, Glibc2.33 has support for MTE on arm64 and it just has the userspace heap tagging, see https://elixir.bootlin.com/glibc/glibc-2.33/source/sysdeps/aarch64. Glibc enables MTE using a memory tunable glibc.mem.tagging, which takes a value between 0 and 255 and acts as a bitmask that enables various capabilities, please see Memeory Related Tunables for details.
Referring to the implementation of glibc, we plan to enable MTE in go using an environment variable as well, named GOMTE, which has the same values and capabilities as glibc’s memory tunable. The GOMTE behaves as follows:

The default value is 0, which disables all memory tagging.
The go runtime will enable memory tagging support if GOMTE has any non-zero value.
It is only used to control the way the Go code uses MTE feature.
It is only supported on AArch64 systems with the MTE feature; it is ignored for all other systems.
It cannot be used with other sanitizers (-asan, -msan or -race).
For cgo, user needs to set glibc.mem.tagging for C code and GOMTE for Go code, respectively.
If GOMTE is inconsistent with glibc.mem.tagging, go runtime will print a warning message and MTE is still enabled.

Note: Given the cgo situation, both GLIBC and Go runtime may configure the MTE protection level for the threads. If the configurations are inconsistent, some threads may be protected at a different level as developer expected. To avoid confusion, we prefer making GOMTE definition always aligned with glibc.mem.tagging definition. Considering that GLIBC tunables are not guaranteed to be stable, GOMTE definitions may also be changed in the future to match the behavior of GLIBC.
The implementation:

Add CPU feature detection to check whether the hardware supports MTE.
Add an environment variable GOMTE.
Enable memory tagging and set tag checks fault mode based on the value of GOMTE.
Modify the allocator when tagging is needed.

Allocate 16 bytes aligned pointer.

Tag the pointer and the memory.

Modify the deallocator when tagging is needed.

Re-tag all freed memory areas with zero tagging.

Any feedback on the current design is welcome. Thank you.

I'm having a hard time seeing how this can work with the garbage collector.  The garbage collector must obviously be able to examine any memory location.  Is the intent that the garbage collector record the tag for each page, and use that tag to construct a pointer when examining that page?
Alternatively, since Go is already a memory-safe language, should we simply reserve a memory tag for all Go memory allocations?  That would let C code check its use of Go pointers, and let Go code check its use of C pointers.

@ianlancetaylor  Yes, you are right. When MTE is enabled, we should do special handling of pointers (remove and add the Address Tagging)  in the process of the garbage collector. For example, a pointer should be removed its address tag when the GC checks whether it is a heap-allocated object. And when loading an underlying pointer from it, the pointer should be added its address tag.
As for the implementation, use p & 0xf0ffffffffffffff to remove the address tag of the pointer p.  To add the address tag for a pointer p , we have the following three options:

Load the memory tag from the memory location referencedby p, and add it to p.
The address tag added for allocations is not randomly generated, but calculated by a hash of the start address of a span instead, so we can get the start address of the span of p, and then calculate the address tag for p.
Just reserve a memory/address tag for all Go memeory allocations.

These three options offer difference level of performance, protection and precision. Like, the option 1 would be incorrect if this pointer p is invalid. The option 2 has higher precision but with weaker performance. The option 3 has strong performance but with weaker protection. We are considering the best option. Thank you.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

The discussion above is mostly about mechanics. That's about how we would add MTE.
The unanswered question is why we would add MTE. It seems like there are three parts to that question:

What must Go do to play nicely with glibc as far as cgo uses are concerned?
What opportunities do we have to use MTE to better isolate Go memory from C memory?
What opportunities do we have to use MTE in the Go runtime to our own benefit? (We could do what is described above for allocation, but what benefits would that give the Go runtime? In C the benefit is catching dangling pointers. Go programs have a GC, so they almost never have dangling pointers.)

Does anyone want to take a stab at answering any of these?

@rsc Sorry for the late reply.

What must Go do to play nicely with glibc as far as cgo uses are concerned?
In terms of implementation, we only need to modify the allocator and deallocator of go. In the case of cgo, no special handling is required. See the implementation of supporting MTE for heap mentioned in the above proposal.
From the MTE user interface,  both GLIBC and Go runtime may configure the MTE protection level for the threads. If the configurations are inconsistent, some threads may be protected at a different level as developer expected. To avoid confusion, we prefer making GOMTE definition always aligned with glibc.mem.tagging definition. Considering that GLIBC tunables are not guaranteed to be stable, GOMTE definitions may also be changed in the future to match the behavior of GLIBC.

What opportunities do we have to use MTE to better isolate Go memory from C memory?
With MTE, the memory will naturally be isolated, but the tag bits are random, and the memory will not be isolated with Go and C as the boundary. But if MTE is enabled. When the C code out of bounds accesses the Go memory, there is a high probability that the tag bits will be different, which will cause the MTE to report an error.

What opportunities do we have to use MTE in the Go runtime to our own benefit? (We could do what is described above
for allocation, but what benefits would that give the Go runtime? In C the benefit is catching dangling pointers. Go rograms
have a GC, so they almost never have dangling pointers.)

There should be less abuse of Go pointers. But we found 3 abuse in top 20 GitHub Go project by just running stock tests with -asan option. The item Why is MTE needed for Go? in the proposal above mentions some benefits.
As for whether the go runtime can use MTE to do some optimization, I think of a point, that is, during the GC process, address tagging can be used to determine whether the pointer is allocated by the heap.
Thank you.

In terms of implementation, we only need to modify the allocator and deallocator of go.

Can you be more specific as to exactly how the allocator and deallocator should be modified?  And are you referring to the allocator that allocates a single Go object, or are you referring to the allocator that allocates a set of memory pages, or what?  Thanks.

@zhangfannie, can you provide details about the kinds of problems you found with -asan?
There are various pieces of the runtime that clearly need to be aware of MTE, especially if we are coordinating with C.
What's not clear to me is whether we need GOMTE as a separate variable. If MTE is good, shouldn't we just turn it on any time it is available? Why would we give users control over this or initiate the use of MTE? (If we expect bugs we could always add a GODEBUG=mte=off to let users override it.)
It sounds like from the runtime side we don't really know how big the changes are, and we'd like to hold the decision for understanding how invasive this all is. Do you have a prototype CL or any sense of what the changes involve and how invasive they are?
Perhaps it would make sense to put this on hold until we have a prototype CL?

@rsc Sorry for the late reply.

@zhangfannie, can you provide details about the kinds of problems you found with -asan?

We found problems from three different projects, but we were not familiar with these projects and did not spend much time looking at too many details. We submitted these issues we found, please see issues below for details.
https://github.com/pingcap/tidb/issues/32952
https://github.com/grafana/grafana/issues/46395
https://github.com/go-delve/delve/issues/2919

What's not clear to me is whether we need GOMTE as a separate variable. If MTE is good, shouldn't we just turn it on any time it is available? Why would we give users control over this or initiate the use of MTE? (If we expect bugs we could always add a GODEBUG=mte=off to let users override it.)

I do not know if it is a good behavior to turn on MTE any time. Because MTE has overhead, it has about 3% memory consumption overhead for tags, as for performance overhead, a document introducing MTE mentions that a speculative expectation is an overhead lower than 5% on average for asynchronous mode, but this data needs to be tested after MTE is enabled.
We design an environment variable to control MTE, this is the implementation of reference glibc, Because considering that Go will interact with C, it is best to be consistent.
As for the use interface of MTE in Go, this can be discussed.

It sounds like from the runtime side we don't really know how big the changes are, and we'd like to hold the decision for understanding how invasive this all is. Do you have a prototype CL or any sense of what the changes involve and how invasive they are?

We found that there are four main places for the invasion of runtime.

 When MTE is enabled, we should do some special handling of pointers (remove or re-add Address Tagging) in the process of garbage collector. For example:

A heap pointer's address tag should be removed when the GC checks whether it is a heap-allocated object by calling spanOf() function. The line 682 of mheap.go file.

func spanOf(p uintptr) *mspan {
         p = p & 0xf0ff ffff ffff ffff                 //  we should add code like this to remove address tagging from this pointer.
 }

The pointer should have its address tag added when loading an underlying pointer from it. See the sanobject() function. The line 1321 of mgcmark.go file.

func scanobject(b uintptr, gcw *gcWork) {
     ...
     addr = reAddTag(addr)                      //  we should add code like this to re-add address tagging for this pointer.
     obj := *(*uintptr)(unsafe.Pointer(addr))
     ...
}

There are many places in the GC process that need to add such handling.

One is the pack and unpack lfstack.
The current pack process takes the 16 bits from the top of a 64-bit address, which conflicts with the 4-bit mte tag.  We should remove the address tagging during the pack process, and then re-add the address tagging during the unpack process when the node is heap allocated.

One is some assembly funtions have intentional out-of-bounds access operations. However, the MTE will treat these out-of-bounds accesses as illegal accesses and report a segmentation fault error.
For example, the indexbyte() function implemented in assembly code, it intentionally out-of-bounds to load data in 32-byte alignment when the data is not 32-byte aligned. The related codes are as follows.

BIC    $0x1f, R0, R3                              // R0: data
...
VLD1.P (R3),   [V1.B16, V2.B16]           //  this is out-of-bounds access.

To be compatible with MTE, we need to modify the implementation of the indexbyte function.  We can refer this implementation
strrchr-mte.S

The last one is some assembly functions that subtract the two addresses to get an offset. When MTE is turned on, the tags of the two addresses are different, and an incorrect offset will be obtained.
For example, the memmove() funtion implemented in assembly code, the line line 148 of memmove_arm64.s file, the related codes are as follows. We should add some extra code to handle this case.

backward_check:
	// Use backward copy if there is an overlap.
	SUB	R1, R0, R14                                // R0: dst, R1: src, the offset is incorrect when the tags of src and dst are different.
	CBZ	R14, copy0
	CMP	R2, R14
	BCC	copy_long_backward

The above are some changes we found during the implementation process. Among them, the processing in GC is most difficult, because GC has many operations on addresses and memory, we cannot guarantee that every place has been considered. And the implementation of GC has been changing, and the changes need to take these special treatments into account.

Perhaps it would make sense to put this on hold until we have a prototype CL?

It is ok to put this on hold until we have a prototype CL.
Thank you.

Placed on hold.
— rsc for the proposal review group

https://github.com/golang/go/issues/27610#issuecomment-2381919857 notes that some versions of Android have started enabling MTE and encounter compatibility issues with Go.
This gets back to Russ's first question in https://github.com/golang/go/issues/59090#issuecomment-1551784405:

What must Go do to play nicely with glibc as far as cgo uses are concerned?

#27610 shows us one case: some of our optimized routines (like IndexByte via runtime.gostring -> runtime.findnull) access memory within a page but potentially beyond the MTE 16-byte boundary. If the memory passed to these routines was allocated by C and tagged, then we must use MTE-compliant read patterns.
cc @randall77


Workflow files become invalid when we re-containerize a cell

Each time we re-containerize a cell it gets a new ID which causes the cell ID saved on the workflow file to be invalid as it does not exist anymore

We can try and maintain the ID.
https://github.com/QCDIS/NaaVRE/blob/main/jupyterlab_vre/component_containerizer/handlers.py#L371
We delete and re-create. We should update

Cells are now updated instead of deleted and recreated. That probably solves the issue.
→ Check whether it does solve the issue.

It does not, because we generate a new cell ID each we extract a cell (ExtractorHandler.post). As a result, (Catalog.get_cell_from_og_node_id(current_cell.node_id) in CellsHandler.post always returns None.
We should find another way to maintain the ID.


import @livechat/livechat-visitor-sdk/ does not work and generates error

error: bundling failed: Error: While trying to resolve module @livechat/mittfrom<EMAIL_ADDRESS>the package<EMAIL_ADDRESS>successfully found. However, this package itself specifies amain module field that could not be resolved<EMAIL_ADDRESS>Indeed, none of these files exist:

/root/pg7Mob/node_modules/@livechat/mitt/dist/livechat-mitt.native.js(.native||.android.js|.native.js|.js|.android.json|.native.json|.json|.android.ts|.native.ts|.ts|.android.tsx|.native.tsx|.tsx)
/root/pg7Mob/node_modules/@livechat/mitt/dist/livechat-mitt.native.js/index(.native||.android.js|.native.js|.js|.android.json|.native.json|.json|.android.ts|.native.ts|.ts|.android.tsx|.native.tsx|.tsx)
at ResolutionRequest.resolveDependency (/root/pg7Mob/node_modules/metro/src/node-haste/DependencyGraph/ResolutionRequest.js:65:15)
at DependencyGraph.resolveDependency (/root/pg7Mob/node_modules/metro/src/node-haste/DependencyGraph.js:273:16)
at Object.resolve (/root/pg7Mob/node_modules/metro/src/lib/transformHelpers.js:261:42)
at dependencies.map.result (/root/pg7Mob/node_modules/metro/src/DeltaBundler/traverseDependencies.js:391:31)
at Array.map ()
at resolveDependencies (/root/pg7Mob/node_modules/metro/src/DeltaBundler/traverseDependencies.js:388:18)
at /root/pg7Mob/node_modules/metro/src/DeltaBundler/traverseDependencies.js:261:33
at Generator.next ()
at asyncGeneratorStep (/root/pg7Mob/node_modules/metro/src/DeltaBundler/traverseDependencies.js:87:24)
at _next (/root/pg7Mob/node_modules/metro/src/DeltaBundler/traverseDependencies.js:107:9)

`
Hi I am trying to integrate simple chat into  the module but just a simple import breaks the whole project and gives the above error whats wrong with it.

Hi,
can you try adding this file to your project:
https://github.com/livechat/react-native-livechat/blob/master/LiveChat/livechat-visitor-sdk.min.js
instead of importing VisitorSdk from LiveChat. The LiveChat module itself is not mobile friendly :/

thanks for replying to be more clear so you mean instead of importing i add it inside  tags

Hi it should be more like this:
import {init} from './sdk/livechat-visitor-sdk.min';

thanks I will test it and let you know

I have tried using that approach it gives me another error says
TypeError: undefined is not an Object (evaluating 'Global.cookie.split')
please do remember I am using react native all I want to achieve is to set customer name and remove title from the window because i have my own back button or if I can manipulate your back button it would be really awesome that will save me 100 hours to develop something from ground zero.
Thank you

Hi, please set somewhere in your code:
Global.cookie = ''
and make it before importing visitorSDK

that worked like charm thanks Ventis. I hope in future this sdk more user friendly and add more features to chat widget like remove title bar, full screen windows as default feature, remove bubble. etc

Related issue: https://github.com/livechat/react-native-livechat/issues/25
Asking there if there will be a release with fixes on this issue. Thanks!


error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

I will try to do my best in orden to explain the problem.
The problem is the same for "Logging Service", "Document API" and "Keyring API"
If I follow the following documentation:
https://github.com/Fraunhofer-AISEC/ids-clearing-house-service/tree/master/clearing-house-app#logging-service
There is a point in which the following command has to exectute:
cargo build --release
Therefore, the binary files are created for the 3 services, the problem is that these binaries are compiled according ssl version in the machine that is compiled, in these case ubuntu 22.04 with openssl 3
but when images are created according to documentation with files document-api.Dockerfile, keyring-api.Dockerfile and logging-service.Dockerfile, these dockerfiles use FROM ubuntu:20.04 and these version of ubuntu uses openssl 1
Therefore libssl.so.3 is not found because ubuntu 20.04 has libssl1.1 by default.

I upgraded the docker files to use ubuntu 22.04 and hope that this fixes your issue.


Clear cache of azurefs containers which are not accessible anymore

If an azurefs container is not accessible for hubble host, it's cache folder is deleted so that the stale files are not served from that container anymore.
NOTE: In case all the containers are inaccessible, cache folder will deleted for all containers. But in this case, the code will not reach the part where 'hashes' directory is deleted. As a result, hubble will keep serving the file from the last azurefs state if all the containers are inaccessible.

Thank you for making this default to False. And I really like the fact that it continues working in case all of them are inaccessible, because otherwise this could be problematic during a network outage.

Thank you for making this default to False. I'm not sure that your evaluation of the "if all are inaccessible" situation is correct, however. The hashes directory won't be lost which means that files that have been previously served will continue to be served correctly. But new files will not. What happens in the case of a network outage? Is the error different from <AuthenticationErrorDetail>Signature did not match., saving us from accidental deletion in this case?

What happens in the case of a network outage?

I tried to hinder the internet connectivity ( by creating a fake dns server in /etc/resolv.conf ).
The exception message was different in this case, something like
Failed to establish a new connection...
And containers were not affected anyhow in this case.
If you can think of some other way(s) to test it, do let me know. I will check with them as well.

I'm not sure that your evaluation of the "if all are inaccessible" situation is correct, however....But new files will not.

If even a single container is accessible, 'hashes' folder will be deleted. So the updated files will be served.
'hashes' folder will not be deleted only in one case, i.e. when all the containers are inaccessible. And in this case, it would be best to serve stale files. Fortunately, this is already happening with current code.

Let me explain a little more about this change and why we need it.

We want to use two ( or more ) azurefs containers for fallback/backup/safety. Idea is that if we deactivate first container ( c1 ) , hubble should start picking files from second  container ( c2 ).
This does not happen as of now. Right now, if you rotate the storage key of c1, or delete c1 altogether from azure, hubble still keeps serving files from c1 ( because the local cached folder of c1 is not deleted).
So, hubble never switches to c2. Hubble keeps serving stale files from (deleted ) c1 container's cache.

In order to achieve the first task, we have decided to remove the cached folder of c1 ( or any other container that is not accessible ).

Next thing we want is that in case all the containers are inaccessible ( because of network glitch perhaps ), hubble should not stop working. Hubble should serve stale/cached profiles in that case.

For this second task, we are relying on the below mentioned logic:
Azurefs' 'hashes' directory is deleted whenever a container's contents are updated from internet. Updated files are served only if 'hashes' directory is deleted. If 'hashes' folder is  not deleted, hubble keeps picking up the files from '.../files/base' directory.
Now, in this second case, all the containers are unreachable. So, an exception is thrown for each container, and the for loop ( after catching exception ) would continue without executing the rest of code. Since the code for deletion of 'hashes' directory is never reached, 'hashes' folder is never deleted. So, hubble will keep picking old files from '.../files/base' directory although the cached folder for all the containers have been deleted.
I have verified this.

<AuthenticationErrorDetail>Signature did not match. messages appears in both the cases where we want to delete the cache:

When we rotate the key of storage container's account
When we delete the storage container for azure cloud.

This error message does not appear if there is network connectivity issue.
So, my code change is based on this 'string' logic.
I could not think of any other better solution.
If you have some other idea for this task, let me know. I can give it a try.

Thanks for the clarifications. I think this is fine.


Error when using Wav2vec-u to launch GAN training

❓ Questions and Help
Hi, everyone. I am using the wav2vec Unsupervised by following indicates on README. However, when I executed the below command, it raised the error AttributeError: 'NoneType' object has no attribute 'transpose'.
Commands :
PREFIX=w2v_unsup_gan_xp
TASK_DATA=/workspace/wav2vec_u/fairseq/examples/wav2vec/dataset/prepare_audio/precompute_pca512_cls128_mean_pooled
TEXT_DATA=/workspace/wav2vec_u/fairseq/examples/wav2vec/dataset/data/phones
KENLM_PATH=/workspace/wav2vec_u/fairseq/examples/wav2vec/dataset/data/phones/lm.phones.filtered.04.bin
fairseq-hydra-train \
    -m --config-dir unsupervised/config/gan \
    --config-name w2vu \
    task.data=${TASK_DATA} \
    task.text_data=${TEXT_DATA} \
    task.kenlm_path=${KENLM_PATH} \
    common.user_dir=examples/wav2vec/unsupervised \
    model.code_penalty=2,4 model.gradient_penalty=1.5,2.0 \
    model.smoothness_weight=0.5,0.75,1.0 'common.seed=range(0,5)'

Here is all the log and error information.
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - task: UnpairedAudioText
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - model: Wav2vec_U
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - criterion: ModelCriterion
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - num. shared model params: 1,166,337 (num. trained: 1,166,337)
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - num. expert model params: 0 (num. trained: 0)
[2021-07-08 13:50:57,276][unsupervised.data.extracted_features_dataset][INFO] - loaded 1680, skipped 0 samples
[2021-07-08 13:50:57,276][unsupervised.tasks.unpaired_audio_text][INFO] - split valid has unpaired text? False
[2021-07-08 13:50:58,923][fairseq.utils][INFO] - ***********************CUDA enviroments for all 1 workers***********************
[2021-07-08 13:50:58,924][fairseq.utils][INFO] - rank   0: capabilities =  6.1  ; total memory = 10.917 GB ; name = GeForce GTX 1080 Ti
[2021-07-08 13:50:58,924][fairseq.utils][INFO] - ***********************CUDA enviroments for all 1 workers***********************
[2021-07-08 13:50:58,924][fairseq_cli.train][INFO] - training on 1 devices (GPUs/TPUs)
[2021-07-08 13:50:58,924][fairseq_cli.train][INFO] - max tokens per device = 200000 and max sentences per device = 16
[2021-07-08 13:50:58,925][fairseq.trainer][INFO] - Preparing to load checkpoint ./checkpoint_last.pt
[2021-07-08 13:50:58,926][fairseq.trainer][INFO] - No existing checkpoint found ./checkpoint_last.pt
[2021-07-08 13:50:58,926][fairseq.trainer][INFO] - loading train data for epoch 1
[2021-07-08 13:50:58,942][unsupervised.data.extracted_features_dataset][INFO] - loaded 4620, skipped 0 samples
[2021-07-08 13:50:58,942][unsupervised.tasks.unpaired_audio_text][INFO] - split train has unpaired text? False
[2021-07-08 13:50:59,080][fairseq.optim.adam][INFO] - using FusedAdam
[2021-07-08 13:50:59,146][fairseq.optim.adam][INFO] - using FusedAdam
[2021-07-08 13:50:59,154][fairseq.logging.progress_bar][WARNING] - tensorboard not found, please install with: pip install tensorboard
[2021-07-08 13:50:59,157][fairseq.trainer][INFO] - begin training epoch 1
[2021-07-08 13:50:59,157][fairseq_cli.train][INFO] - Start iterating over samples
Traceback (most recent call last):
  File "/tmp/fairseq/fairseq_cli/hydra_train.py", line 45, in hydra_main
    distributed_utils.call_main(cfg, pre_main)
  File "/tmp/fairseq/fairseq/distributed/utils.py", line 369, in call_main
    main(cfg, **kwargs)
  File "/tmp/fairseq/fairseq_cli/train.py", line 180, in main
    valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/tmp/fairseq/fairseq_cli/train.py", line 291, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/tmp/fairseq/fairseq/trainer.py", line 710, in train_step
    ignore_grad=is_dummy_batch,
  File "/tmp/fairseq/fairseq/tasks/fairseq_task.py", line 502, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/tmp/fairseq/fairseq/criterions/model_criterion.py", line 50, in forward
    net_output = model(**sample["net_input"])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/tmp/fairseq/fairseq/../examples/wav2vec/unsupervised/models/wav2vec_u.py", line 567, in forward
    token_y = self.discriminator(token_x, token_padding_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/tmp/fairseq/fairseq/../examples/wav2vec/unsupervised/models/wav2vec_u.py", line 266, in forward
    x = x.transpose(1, 2)  # BTC -> BCT
AttributeError: 'NoneType' object has no attribute 'transpose'

Can someone help on this ? Thanks a lot in advance.

Same issue. random_label is None for some reason

The argument random_label is None. Carefully check your dataset.

@xiabingquan If you don't mind me asking, could you elaborate more on your explanation? I have the same issue, and I am not clear about what exactly I have to do. Should I change the argument random_label or prepare audio features again?


added workflow template and service, plus minor updates

Set image_pull_policy on a Task level
WorkflowTemplate Service and WorkflowTemplate implementation
Option to create a Workflow with a WorkflowTemplate

The image pull policy thing was added because I'm working on a demo for our company where I can showcase how we can use this library instead of the yml configs and I had some issues when I wanted to run my local docker image on minikube.
Workflow template and service was added because we usually create template in during our projects and we use the same docker image, we just pass different arguments in the different steps.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the really thorough review. I hope now everything is looking good. But please take a look and tell me if you still have anything that is not acceptable.


No active language error handling

Our NstackBuilder fails at finding the default language when no language is set to active on nstack.io. Ideally we print a nice message in that case instead of throwing.

@markusrubey @johsoe what about forcing nstack.io to force their users to set a language as active? This would prevent this unwanted behavior


Cannot make run if you set $APOLLO_KEY to a user API key

For my local rover development, I have APOLLO_KEY set to a user API key in my shell profile.
The default behavior for apollo-server is to enable usage reporting if APOLLO_KEY is present, but unfortunately it does not work if you set APOLLO_KEY to a user API key.
This is already tracked in this issue on the apollo-server repository, but I wanted to also document it here in case someone runs into this problem while trying out the demo.
$ node --version
14.16.0
$ rover config whoami
Checking identity of your API key against the registry.
Key Type: USER
User ID: gh.EverlastingBugstopper
Origin: $APOLLO_KEY
API Key: user************************************************
$ make run
node index.js local
Starting Apollo Gateway in local mode ...
Using local: ./supergraph.graphql
No graph variant provided. Defaulting to `current`.
Error: You've enabled usage reporting via ApolloServerPluginUsageReporting, but you also need to provide your Apollo API key, via the APOLLO_KEY environment variable or via `new ApolloServer({apollo: {key})
    at Object.serverWillStart (/home/avery/work/forks/supergraph-demo/node_modules/apollo-server-core/dist/plugin/usageReporting/plugin.js:62:23)
    at /home/avery/work/forks/supergraph-demo/node_modules/apollo-server-core/dist/ApolloServer.js:261:122
    at Array.map (<anonymous>)
    at ApolloServer.<anonymous> (/home/avery/work/forks/supergraph-demo/node_modules/apollo-server-core/dist/ApolloServer.js:261:73)
    at Generator.next (<anonymous>)
    at fulfilled (/home/avery/work/forks/supergraph-demo/node_modules/apollo-server-core/dist/ApolloServer.js:5:58)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
$ unset APOLLO_KEY
$ make run
node index.js local
Starting Apollo Gateway in local mode ...
Using local: ./supergraph.graphql
🚀 Server ready at http://localhost:4000/

@EverlastingBugstopper this should be fixed now via #143 which also fixes #136. Note the APOLLO_KEY and APOLLO_GRAPH_REF override behavior by ./graph-api.env in #143 for your CI usage. 🙂


[T3A2][T16-A2] Varun Gupta

Ready for review

@varung97 Some comments added. Please close the PR after reading comments.

Noted, thanks :)


mixer filter segfault with latest Envoy

referring to #1047 where mixer was updated new SHA from envoy.. here is the backtrace
W0817 19:03:26.095] }I0817 19:00:01.952818       1 watcher.go:233] Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 2 --parent-shutdown-time-s 3 --service-cluster istio-proxy --service-node sidecar~<IP_ADDRESS>~a-2969276591-3td37.istio-test-391zw~istio-test-391zw.svc.cluster.local]
W0817 19:03:26.095] [2017-08-17 19:00:01.959][19][warning][main] external/envoy/source/server/server.cc:151] initializing epoch 0 (hot restart version=8.2490552)
W0817 19:03:26.095] [2017-08-17 19:00:01.971][19][warning][main] external/envoy/source/server/server.cc:320] starting main dispatch loop
W0817 19:03:26.095] [2017-08-17 19:00:02.157][19][warning][main] external/envoy/source/server/server.cc:315] all clusters initialized. initializing init manager
W0817 19:03:26.095] [2017-08-17 19:00:02.251][19][warning][config] external/envoy/source/server/listener_manager_impl.cc:479] all dependencies initialized. starting workers
W0817 19:03:26.095] [2017-08-17 19:00:05.263][19][warning][main] external/envoy/source/server/drain_manager_impl.cc:57] shutting down parent after drain
W0817 19:03:26.096] [2017-08-17 19:00:18.868][19][warning][upstream] external/envoy/source/server/lds_api.cc:86] lds: fetch failure: route: unknown cluster 'in.70'
W0817 19:03:26.096] [2017-08-17T19:00:42.120Z] "GET /a HTTP/1.1" 200 - 0 454 70 1 "-" "Go-http-client/1.1" "7ea12db2-02dc-9c07-abfb-7e9c9ed2131e" "a" "<IP_ADDRESS>:8080"
W0817 19:03:26.096] [2017-08-17 19:00:43.452][19][critical][assert] src/envoy/mixer/thread_dispatcher.cc:39] assert failure: thread_dispatcher
W0817 19:03:26.096] [2017-08-17 19:00:43.452][19][critical][backtrace] bazel-out/local-dbg/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:101] Caught Aborted, suspect faulting address 0x53900000013
W0817 19:03:26.096] [2017-08-17 19:00:43.453][19][critical] Backtrace (most recent call first) from thread 0:
  #1 ?? ??:0
  #2 ?? ??:0
  #3
  #4 Envoy::Http::Mixer::GetThreadDispatcher() at thread_dispatcher.cc:39 (discriminator 1)
  #5 Envoy::Http::Mixer::GrpcTransport::onSuccess() at grpc_transport.cc:66
  #6 Envoy::Grpc::RpcChannelImpl::onSuccess(std::unique_ptr<Envoy::Http::Message, std::default_delete<Envoy::Http::Message> >&&) at rpc_channel_impl.cc:67
  #7 Envoy::Http::AsyncRequestImpl::onComplete() at async_client_impl.cc:176
  #8 Envoy::Http::AsyncRequestImpl::onTrailers(std::unique_ptr<Envoy::Http::HeaderMap, std::default_delete<Envoy::Http::HeaderMap> >&&) at async_client_impl.cc:199
  #9 Envoy::Http::AsyncStreamImpl::encodeTrailers(std::unique_ptr<Envoy::Http::HeaderMap, std::default_delete<Envoy::Http::HeaderMap> >&&) at async_client_impl.cc:105
  #10 Envoy::Router::Filter::onUpstreamTrailers(std::unique_ptr<Envoy::Http::HeaderMap, std::default_delete<Envoy::Http::HeaderMap> >&&) at router.cc:502
  #11 Envoy::Router::Filter::UpstreamRequest::decodeTrailers(std::unique_ptr<Envoy::Http::HeaderMap, std::default_delete<Envoy::Http::HeaderMap> >&&) at router.cc:617
  #12 Envoy::Http::StreamDecoderWrapper::decodeTrailers(std::unique_ptr<Envoy::Http::HeaderMap, std::default_delete<Envoy::Http::HeaderMap> >&&) at codec_wrappers.h:40
  #13 Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) at codec_impl.cc:412
  #14 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::{lambda(nghttp2_session*, nghttp2_frame const*, void*)#6}::operator()(nghttp2_session*, nghttp2_frame const*, void*) const at codec_impl.cc:696
  #15 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::{lambda(nghttp2_session*, nghttp2_frame const*, void*)#6}::_FUN(nghttp2_session*, nghttp2_frame const*, void*) at codec_impl.cc:697
  #16 session_call_on_frame_received at nghttp2_session.c:3262
  #17  (inlined by) session_after_header_block_received at nghttp2_session.c:3762
  #18  (inlined by) nghttp2_session_mem_recv at nghttp2_session.c:6108
  #19 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) at codec_impl.cc:300 (discriminator 2)
  #20 Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) at codec_client.cc:100
  #21 Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&) at codec_client.h:139
  #22 Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*) at filter_manager_impl.cc:61
  #23 Envoy::Network::FilterManagerImpl::onRead() at filter_manager_impl.cc:71
  #24 Envoy::Network::ConnectionImpl::onRead(unsigned long) at connection_impl.cc:209
  #25 Envoy::Network::ConnectionImpl::onReadReady() at connection_impl.cc:414
  #26 Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) at connection_impl.cc:362
  #27 Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::DispatcherImpl&, int, std::shared_ptr<Envoy::Network::Address::Instance const>, std::shared_ptr<Envoy::Network::Address::Instance const>, bool, bool)::{lambda(unsigned int)#3}::operator()(unsigned int) const at connection_impl.cc:80
  #28 std::_Function_handler<void (unsigned int), Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::DispatcherImpl&, int, std::shared_ptr<Envoy::Network::Address::Instance const>, std::shared_ptr<Envoy::Network::Address::Instance const>, bool, bool)::{lambda(unsigned int)#3}>::_M_invoke(std::_Any_data const&, unsigned int&&) at functional:1871
  #29 std::function<void (unsigned int)>::operator()(unsigned int) const at functional:2267
  #30 Envoy::Event::FileEventImpl::assignEvents(unsigned int)::{lambda(int, short, void*)#1}::operator()(int, short, void*) const at file_event_impl.cc:60
  #31 Envoy::Event::FileEventImpl::assignEvents(unsigned int)::{lambda(int, short, void*)#1}::_FUN(int, short, void*) at file_event_impl.cc:61
  #32 event_persist_closure at event.c:1580
  #33  (inlined by) event_process_active_single_queue at event.c:1639
  #34 event_process_active at event.c:1738
  #35  (inlined by) event_base_loop at event.c:1961
  #36 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) at dispatcher_impl.cc:164 (discriminator 4)
  #37 Envoy::Server::InstanceImpl::run() at server.cc:323
  #38 Envoy::main_common(Envoy::OptionsImpl&) at main_common.cc:82
  #39 main at main.cc:42
  #40
  #41 ?? ??:0
  #42
  #43 _start at ??:?
  #44

cc @kyessenov @qiwzhang @lizan

@rshriram can we get a log with debug or trace level?

It seems it run into this assertion
https://github.com/istio/proxy/blob/master/src/envoy/mixer/thread_dispatcher.cc#L39
Mixer grpc client code is using a dispatcher stored in thread local.  it is set at setDecoderFilterCallback for HTTP filer,  and initializeReadFilterCallbacks  for tcp filter.
Could it be destroyed?  or it was never set?
Is it possible that we could not cache dispatcher pointer?
What kind of traffic?  HTTP right?

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_pilot/1047/pilot-presubmit/400/ is the pilot test where the code is segfaulting. It's occurring for all cases.

Log confirmed that it is assertion in
https://github.com/istio/proxy/blob/master/src/envoy/mixer/thread_dispatcher.cc#L39

Addressed by https://github.com/istio/proxy/pull/433, https://github.com/istio/proxy/pull/434, and https://github.com/istio/proxy/pull/427


Enable MPIPartition to work with an input MPI communicator

This PR consists of a few changes to the Partition class in order for it to be able to take in an input MPI communicator if needed. I think this feature is useful when Partition() is called within an MPI application which already initialized the MPI communicator.
I successfully tested both approaches with this example code in the documentation. In the case where Partition() takes in the MPI communicator, one just needs to initialize MPI before importing mpipartition and pass comm as an input, as follows
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

from mpipartition import Partition
from mpipartition import distribute, overload, exchange
...
partition = Partition(comm=comm)
...

Note also that the example code does not run as is, it requires a few minor fixes, namely:

is_valid = np.ones(n_local_after, dtype=np.bool_) needs to be is_valid = np.ones(n_local_distributed, dtype=np.bool_)
for i, x in enumerate(['xyz']): needs to be for i, x in enumerate('xyz'):

Thanks! Let me know what else I can assist with to get this PR merged, MPIPartition is a very useful package!

Thanks for the PR, looks good!
Out of curiosity, are you running mpi4py without automatic init/finalize on import and program exit? (Didn't know one could do that, but just saw the env variables in the mpi4py docs).
I'll fix the examples, thanks for pointing it out!

@rickybalin , I undid some of the changes in your PR, see commit 54d9314 . I removed
mpi4py.rc.initialize = False

since I don't think the library should change the behavior of mpi4py. If you need this, you can always add this line into your scripts before importing MPIPartition.
Comm is now also a keyword argument for Partition instead of the first positional argument. This way, we don't have to update existing codes that use dimensions as the first positional argument.


Multi destination trips

About
Home page modifications to allow for multi destinations trip:

User can add a destination at the end of the trip
User can remove a destination, except for the 2 first ones (at least 2 required; origin & destination)

Closes #18
Screenshots

Not covered in this PR
What users can't do (possible improvements, but can be done as a separate PR according to me):

Re-order destinations
Delete first 2 destinations, even if there are more than 2 destinations present

@serge1peshcoff I had some time and motivation yesterday and today, so here is a proposition for multiple destinations on home page.
What do you think of it?

@NoryDev that is super lovely


maa-assistant-arknights: init at 4.23.8, maa-cli: init at 0.4.0

Description of changes

https://github.com/MaaAssistantArknights/MaaAssistantArknights
MaaAssistantArknights is a one-click tool for the daily tasks of Arknights, supporting all clients with about 10k stars on Github and 180k daily users. It only contains cross-platform library and windows client.
https://github.com/MaaAssistantArknights/maa-cli
maa-cli is a simple CLI for MAA.
Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg build maa-assistant-arknights maa-assistant-arknights.passthru.tests maa-cli maa-cli.passthru.tests


REST Usage workflow

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 150, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 161, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/lib/python3.9/site-packages/openmetadata/workflows/ingestion/common.py", line 50, in metadata_ingestion_workflow
    config = json.loads(workflow_config.json(encoder=show_secrets_encoder))
AttributeError: 'NoneType' object has no attribute 'json'

fixed by https://github.com/open-metadata/OpenMetadata/pull/4279


Add missing methods to InternalDeploy

Codecov Report

Merging #161 (01e435d) into main (03dbfef) will increase coverage by 1.77%.
The diff coverage is 29.62%.

@@            Coverage Diff             @@
##             main     #161      +/-   ##
==========================================
+ Coverage   71.84%   73.61%   +1.77%
==========================================
  Files          34       34
  Lines        1861     1884      +23
==========================================
+ Hits         1337     1387      +50
+ Misses        524      497      -27

Impacted Files
Coverage Δ

src/business_logic/state/cached_state.rs
71.95% <ø> (ø)

src/business_logic/transaction/internal_objects.rs
0.00% <0.00%> (ø)

...ic/transaction/objects/internal_invoke_function.rs
54.83% <ø> (ø)

src/definitions/constants.rs
100.00% <ø> (ø)

...rc/core/syscalls/business_logic_syscall_handler.rs
66.82% <56.00%> (+15.67%)
:arrow_up:

src/core/syscalls/syscall_request.rs
85.45% <100.00%> (+10.22%)
:arrow_up:

src/core/syscalls/syscall_response.rs
100.00% <100.00%> (ø)

src/core/syscalls/os_syscall_handler.rs
85.14% <0.00%> (-0.86%)
:arrow_down:

src/core/syscalls/syscall_handler.rs
80.12% <0.00%> (+1.23%)
:arrow_up:

... and 2 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Closing this one since everything was implemented in other PRs.


Remove unused coordinator dynamic configs

Description
After #13197 , several coordinator configs are now redundant as they are not being used anymore, neither with smartSegmentLoading nor otherwise.
Changes

Remove dynamic configs emitBalancingStats: balancer error stats are always emitted, debug stats can be logged by using debugDimensions
useBatchedSegmentSampler, percentOfSegmentsToConsiderPerMove: batched segment sampling is always used
Add test to verify deserialization with unknown properties

Release note
The following Coordinator dynamic configs have been removed:

emitBalancingStats: Stats for errors encountered while balancing will always be emitted. Other debugging stats will not be emitted but can be logged by setting the appropriate debugDimensions.
useBatchedSegmentSampler, percentOfSegmentsToConsiderPerMove: Batched segment sampling is always on.

This PR has:

[ ] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

Could you please make changes to coordinator-dynamic-config.tsx and the configuration docs as well?

Thanks for the reminder, @AmatyaAvadhanula !


Implement autogenerated Record Namespaces

Provide autogenerated Record Namespaces with more high-level work with basic operations upon records (CRUD + list records)
this is how we work with records now:
self.com.atproto.repo.delete_record(
            models.ComAtprotoRepoDeleteRecord.Data(
                collection=ids.AppBskyFeedPost,
                repo=uri.hostname,
                rkey=uri.rkey,
            )
        )

this is how it could be (.post is autogenerated PostRecordNamespace):
self.bsky.feed.post.delete(repo, rkey)

let's think about the benefits.

code is smaller
you don't need to care about collection arg
you definitly know which record type required for input
you definitly know which record type will be returned. good auto complition


Fix for name resolution responses containing <IP_ADDRESS>

Description
Note that the DUTs described below power up with an IPv4 endpoint set to <IP_ADDRESS> and after a short initial wait, that endpoint gets assigned a valid IPv4 address like for example <IP_ADDRESS>
Even though this may be a bit controversial, configuring +TCP to use a DHCP server would have the very similar effect of the IPv4 endpoint having its ipv4_settings.ulIPAddress set to <IP_ADDRESS> in the very early stages of the boot process.
I was working with 2 DUTs both running FreeRTOS+TCP. On power up, DUT_1 attempts to resolve DUT_2's IP address by name and then send UDP data to it. Since DUT_1 is pretty quick to begin sending mDNS and LLMNR request, DUT_2 was responding to them with <IP_ADDRESS> in the answer field.
This caused DUT_1 to call the user callback given to FreeRTOS_getaddrinfo_a() and also to store this <IP_ADDRESS> address in the DNS cache.
My resolution was 4-fold:

Modify my xApplicationDNSQueryHook_Multi() to return pdFALSE if my IPv4 endpoint had and IP address of <IP_ADDRESS> This of course is up to the user and as such is not part of this PR.
Modify my FreeRTOS_getaddrinfo_a() callback to ignore responses with <IP_ADDRESS>. As this is also part of the user code, it is not part of this PR.
Force +TCP to drop mDNS and LLMNR responses the mention <IP_ADDRESS> in their answer. This prevents storing this invalid address in the DNS cache.
Ensure that +TCP never sends responses when the IPv4 endpoint modified by  xApplicationDNSQueryHook_Multi() has its ipv4_settings.ulIPAddress equal to zero.

In my specific test scenario, any one of the fixes above solves the problem, but in the general case where one of the devices is not a FreeRTOS+TCP device, I think it's best to have all 3 safeguards in place.
Wireshark capture:
For the purposes of the capture, I allowed DUT_1 to power up first. Then, after it began sending it's LLMNR/mDNS request, I powered up DUT_2.
LLMNR_Responses_0_0_0_0.zip
In this capture, DUT begins responding around the 2 second mark, but it still does not have a valid IP.
Around the 4 second mark DUT responds with it's valid IP.
This was a fairly quick fix, so please feel free to suggest better/cleaner solutions.
I also have not examined or tried to reproduce similar issues with IPv6. Potentially there may be similar problems if we receive responses with "::" in the answer.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

It makes sense to not reply to DNS requests when there is no valid IP address for DUT. Thanks, @evpopov, for creating this PR.

@evpopov
While fixing the UTs, I have modified the PR at 2 places to make the coverage 100%. Please take a look to see if it is in conformance with your changes:

Change 1
Change 2

@tony-josi-aws, I agree. The checks that you modified were redundant and the changes look good.
Thanks.


New varint with unique values

In the old code, they have the same value (0)
In the new code, they have:
varint [0] = 0
varint [128 0] = 128
varint [128 128 0] = 16512

The new logic is handling varints incorrectly. I'm not sure what this is trying to achieve.
The binary.PutUvarint function uses the same varint encoding as protos:
	var arr [10]byte
	b := arr[:binary.PutUvarint(arr[:], 16512)]
	fmt.Println(b) // [128 129 1]

Its output disagrees with what your change does.

Maybe I chose the wrong place for my proposal. This is new more compact varint. All it values is unique. MSB now used in value calc.
Maybe need to add it as new functions DecodeCompactVarint and EncodeCompactVarint.

This is the wrong place for this. Please submit wire format encoding proposals to the issue tracker at github.com/google/protobuf.

Done: https://github.com/google/protobuf/issues/4376


B2CExtensionApplication name change in Azure

The extension application seems to have changed it's name in the portal to:
b2c-extensions-app. do not modify. used by aadb2c for storing user data.
The existing code in GetB2CExtensionApplication looked for an exact name match.  This has been changed to do a "startsWith" search on b2c-extensions-app.

Should have looked at the pull requests earlier; literally was about to create a pull request with that exact change. Good stuff!

Thanks for this contribution @rndthoughts !


Does this still work?

No updates in three years :/

Nope, just tried it - failing on NPM Install

same, getting error on npm Install

when i  fix ReferenceError: primordials is not defined error, it work for my.
how-to-fix-referenceerror-primordials-is-not-defined-in-node-js

running npx npm-force-resolutions and then run npm install solves my problem.

It doesn't work！ failing on NPM Install

I've no idea why GitHub search pops up this repo as the first search result for "cross browser extension boilerplate" and "Updated on May 2", but what you want is extension.js.

Also no idea why there are so many other Chrome-specific extension boilerplate projects, when extension.js supports Firefox, as well as both Svelte and React, and the author worked on Brave for four years.


feat: support uri redirect in rest client

See the test code for details

cc @flyrain Would you take a look?


Plane is a common name that causes problems when not in a namespace

Hi,
Plane is a very common name in Unity and can cause naming collision.
Consider putting the project in its own namespace or changing the Plane name to something different.

That's a good point, I'll put it all into it's own namespace to keep it separated.

I dropped everything into it's own namespace and changed the Plane class name as well.
This change might cause some merge conflicts but should be fairly easy to solve since no functionality changed.


Don't kill thread when event handling fails

Currently , if the EventsDispatcher fails deserialize an event payload, or if an exception occurs during the implementing handler, the thread polling the queue will die and not continue polling. This means you'll miss all events after the first failure. Instead, it should log an error but continue handling events.

@sean-heller Thank you for your contribution. This change is included in the 1.5.3 release I published yesterday. Please try it out when you have a chance.


Add Universiti Sains Islam Malaysia (USIM) to domains

This pull request adds Universiti Sains Islam Malaysia (USIM) to the domains. The file usim.txt has been created in the path lib/domains/my/edu/.

@ilmimunirah lib/domains/my/edu/usim: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.


Only parses Undelivered message for emails with bounced emails

Describe the bug
I have a raw email file in which one of the emails bounced. I would assume that I could parse both the 1) Content-Description: Undelivered Message email, as well as 2) the original email that had the bounced email, as there are valid addresses in that message.
To Reproduce
Steps to reproduce the behavior:

import mailparser

mail = mailparser.parse_from_bytes(f)

mailparsed_email.to only contains the return email back to sender

See error

Expected behavior
A clear and concise description of what you expected to happen.
Raw mail
Raw email gist
Environment:

OS: no
Docker: no
mail-parser version [e.g. 3.15.0]

Additional context
If there's another place that I need to be referencing to get information about both emails (or parse them into 2), please let me know!

To review in the new version.
I will open a new issue if needed.


Reconnect

Websocket reconnection
Updated websocket-client version to 1.6.4
Analog sensor integration
Switch integration
Fix for entities received after unsupported partition resource
typos code quality ...

Websocket reconnection
Updated websocket-client version to 1.6.4
Analog sensor integration
Switch integration
Fix for entities received after unsupported partition resource
typos code quality ...


calc: <IP_ADDRESS> -> <IP_ADDRESS>

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/lcn2/calc/releases.
meta.description for calc is: C-style arbitrary precision calculator
meta.homepage for calc is: http://www.isthe.com/chongo/tech/comp/calc/
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found <IP_ADDRESS> with grep in /nix/store/msyql1sqx8vg1adkyhpgxc581993gpq5-calc-<IP_ADDRESS>
found <IP_ADDRESS> in filename of file in /nix/store/msyql1sqx8vg1adkyhpgxc581993gpq5-calc-<IP_ADDRESS>

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
calc

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/msyql1sqx8vg1adkyhpgxc581993gpq5-calc-<IP_ADDRESS> \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A calc https://github.com/r-ryantm/nixpkgs/archive/70723fce93cb3db3961fa7ccba08325aeacb38d3.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/msyql1sqx8vg1adkyhpgxc581993gpq5-calc-<IP_ADDRESS>
ls -la /nix/store/msyql1sqx8vg1adkyhpgxc581993gpq5-calc-<IP_ADDRESS>/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

calc

Maintainer pings
cc @matthewbauer for testing.

Result of nixpkgs-review pr 221042 run on x86_64-linux 1

1 package built:

calc

[nix-shell:~/.cache/nixpkgs-review/pr-221042]$  results/calc/bin/calc
C-style arbitrary precision calculator (version <IP_ADDRESS>)
Calc is open software. For license details type:  help copyright
[Type "exit" to exit, or "help" for help.]

; 1+2
        3
;

Result of nixpkgs-review pr 221042 run on aarch64-darwin 1

1 package built:

calc


Add Pur Abo bypass to existing extraction pipelines

The Pur Abo bypass needs to be combined with the preexisting extraction functions. Might be a bit finicky due to selenium being in the mix, but manageable nonetheless. Important to remember: these functions will have to be the default if a ZEIT url is given as input, else users won't be able to access the articles.

See progress in https://github.com/Leibniz-HBI/newsfeedback/tree/pur-abo-bypass-in-pipelines_issue-13

Done!


Image rotation

I made a brief foray into using Images… and one thing I needed was image rotation.  Perhaps I'm missing something, but I couldn't find any existing code for this.  In fact, I'm not quite sure where it should live.  I ended up writing my own simple method that punts to Interpolations… and doesn't even have any dependence on Images.jl at all!  Which makes finding a home for this a little more confusing.  Although, as I write this, I see that you're going to be depending upon Interpolations soon here, which may simplify the choice a bit.
In any case, here's the fly-by code dump.  Feel free to use (MIT license), modify, or discard as you see fit.
"""
Rotate a matrix. Note that by default, this does not increase the
size of the returned image, so it crops portions that rotate outside the viewport

Optional keyword arguments:
* origin - a tuple of indices (x,y) about which the matrix should rotate
* region - a tuple of UnitRanges to specify the indices of the new image, with respect to the indices of the current one
* fill - the extrapolation fill spec

Note: Unless you compose a rotation as a combination of three
rounded skews, rotations need to be interpolated. So we flip the problem around and
instead rotate the indices into the array, allowing us to punt to Interpolations.
"""
function rotate{T}(im::AbstractMatrix{T}, θ;  # Maybe import Graphics: rotate?
        origin=(size(im, 1)/2+0.5, size(im, 2)/2+0.5),
        region::NTuple{2, UnitRange{Int}}=(1:size(im, 1), 1:size(im, 2)),
        fill=zero(T))
    etp = extrapolate(interpolate(im, BSpline(Linear()), OnGrid()), fill)

    out = similar(im, length(region[1]), length(region[2]))
    x₀, y₀ = origin
    sinθ, cosθ = sin(-θ), cos(-θ) # trickery: angle is negative since we're rotating the indices
    for (ix, x) in enumerate(region[1])
        for (iy, y) in enumerate(region[2])
            # Unwrap the rotation multiplication [cosθ sinθ; -sinθ cosθ]*[x̂, ŷ] to prevent allocating
            x̂, ŷ = x - x₀, y - y₀
            x′ =  cosθ*x̂ + sinθ*ŷ
            y′ = -sinθ*x̂ + cosθ*ŷ
            out[ix,iy] = etp[x′ + x₀, y′ + y₀]
        end
    end

    out
end

And here's a quick demo of 5, 10 degree rotations at various origins.

There's an unadvertised TransformedArrays capability in AffineTransforms.jl. While there is no mention in the README, there are internal docstrings. Try ?transform.

Oh wow that's fantastic, I didn't even know about AffineTransforms.jl and would never have found it! I think it's worth documenting TransformedArrays and advertising that package in Image.jl's readme. (I know, I know, time is limited.)

:+1:.  That's just what I had been looking for, and more general, too.  I should have figured that you had done this already.  :)


Add combined Wilderness layer to SGID from BLM and USFS

New combined BLM/USFS wilderness BOUNDARIES.UtahWildernessAreas to replace outdated BOUNDARIES.USFSWilderness
Summary

Proposed Dataset Name: UtahWildernessAreas
Proposed ArcGIS Online Name: Utah Wilderness Areas
Proposed ISO Category: BOUNDARIES
Data Description: Combined BLM/USFS wilderness areas
Source Agency/Entity: BLM and USFS
Contact: Zach Beck (for now)
Update Frequency: As needed

Triage

[ ] Data Team Triage (@gregbunce)
[x] Dev Team Triage (@steveoh)
[ ] Cadastre Team Triage (@rkelson)

We are introducing data
The data was or will be added on 2020/12/10 to the following areas

[x] Internal SGID
[x] ArcGIS Online

The data is high quality

[x] Sweeper checks have run and passed
[x] The minimum requirements for metadata are populated
[x] The data complies with our domain rules

The data should propagate automatically to

[x] ArcGIS Online
[x] Internal SGID (assigned to @ZachBeck)
[x] Open SGID
[x] Open Data
[x] SGID.META.ChangeDetection

References to the data need to be manually added to

[x] gis.utah.gov data pages
[x] Stewardship doc
[x] SGID.META.AGOLItems
[ ] AGOLItems_shelved metatable (AGOL version of the metatable for any static or shelved items)

We have verified the data is live

[x] ArcGIS Online (@ZachBeck on 12/21/2020)
[x] Internal SGID (@ZachBeck on 12/10/2020)
[x] Open SGID (@ZachBeck on 12/22/2020)
[x] Open data (@ZachBeck on 12/21/2020)
[x] gis.utah.gov data pages (@ZachBeck on 12/21/2020)
[x] sgid-index (@ZachBeck on 12/21/2020)
[x] SGID.META.AGOLItems (@ZachBeck on 12/21/2020)
[x] SGID.META.ChangeDetection (@steveoh  on 01/06/2021)
[ ] agol-validator items (@jacobdadams on ``)

Notification

[ ] Blog post (@assigned)
[x] Twitter (@steveoh)

Entries in META.AGOLItems and sgid-index were started, waiting for AGOL_ID

conductor results for tasks - 100

check
status

@gregbunce has completed 0 out of 1 tasks
:no_entry:

steveoh has completed 3 out of 3 tasks
:+1:

@rkelson has completed 0 out of 1 tasks
:no_entry:

ZachBeck has completed 8 out of 8 tasks
:+1:

@jacobdadams has completed 0 out of 1 tasks
:no_entry:

conductor results for tasks - 100

check
status

@gregbunce has completed 0 out of 1 tasks
:no_entry:

steveoh has completed 3 out of 3 tasks
:+1:

@rkelson has completed 0 out of 1 tasks
:no_entry:

ZachBeck has completed 8 out of 8 tasks
:+1:

@jacobdadams has completed 0 out of 1 tasks
:no_entry:

conductor results for BOUNDARIES.UtahWildernessAreas

check
status

internal sgid
:+1:

meta table

- item id
:+1:

- item name
:+1:

open sgid
:+1:

open data
:+1:

arcgis online
:+1:

stewardship

- Endpoint
:no_entry:

- Description
:+1:

- Data Source
:+1:

- Website URL
:+1:

- Data Type
:+1:

conductor results for BOUNDARIES.UtahWildernessAreas

check
status

internal sgid
:+1:

meta table

- item id
:+1:

- item name
:+1:

open sgid
:+1:

open data
:+1:

arcgis online
:+1:

stewardship

- Endpoint
:no_entry:

- Description
:+1:

- Data Source
:+1:

- Website URL
:+1:

- Data Type
:+1:

Endpoint has been updated

Endpoint has been updated

Nice! Are we done or waiting on the last @jacobdadams checkbox? The scheduled run should have happened this morning. Did that work properly?

Nice! Are we done or waiting on the last @jacobdadams checkbox? The scheduled run should have happened this morning. Did that work properly?

It completed but threw an error during clean up (working on that now).
Looking at the log, all the checks for this item were good, though.

It completed but threw an error during clean up (working on that now).
Looking at the log, all the checks for this item were good, though.


R Package: force silence?

Is there a way to for silence in the R package? I'm coding a stochastic optimizer which makes use of LightGBM, and LightGBM writes so much to the interactive console that I cannot use my optimizer without silencing it (it slows down heavily over time - I would need about 1M, or even more boosting iterations scattered across 10K+ models, R interactive console can't bear 1M lines in console).
The specific conditions I would need:

no warning/info messages (another verbosity level?)
keep logging the metric during training (can't use verbose=0 and log metrics at the same time, I want to keep metric but keep silent also)
not show randomly "free booster handle" and "free dataset handle" (this one is easy to fix)
not print line breaks (providing no verbose and no validation data makes line breaks print)

This would be useful for users who tend to runs batched code and don't want to see anything other than errors.
For instance, I want all these messages to never appear on the console (nor all the line breaks appearing when verbose is 0):
[1] "free booster handle"
[1] "free dataset handle"
[1] "free dataset handle"
[1] "free booster handle"
[1] "free dataset handle"
[1] "free dataset handle"
[LightGBM] [Warning] Ignoring Column_60 , only has one value
[LightGBM] [Warning] Ignoring Column_67 , only has one value
[LightGBM] [Warning] Ignoring Column_95 , only has one value
[LightGBM] [Info] Number of data: 1459, number of features: 133
[LightGBM] [Info] No further splits with positive gain, best gain: -inf, leaves: 22
[1]:	test's l2:43118
Will train until hasn't improved in 10 rounds.

[LightGBM] [Info] No further splits with positive gain, best gain: -inf, leaves: 40
[2]:	test's l2:42209.8
[LightGBM] [Info] No further splits with positive gain, best gain: -inf, leaves: 32
[3]:	test's l2:41768.4

or such wall of line breaks:
[LightGBM] [Warning] Ignoring Column_101 , only has one value
[LightGBM] [Warning] Ignoring Column_110 , only has one value
[LightGBM] [Warning] Ignoring Column_178 , only has one value
[LightGBM] [Warning] Ignoring Column_180 , only has one value
[LightGBM] [Warning] Ignoring Column_319 , only has one value
[LightGBM] [Warning] Ignoring Column_392 , only has one value
[LightGBM] [Warning] Ignoring Column_797 , only has one value
[LightGBM] [Warning] Ignoring Column_798 , only has one value
[LightGBM] [Warning] Ignoring Column_801 , only has one value
[LightGBM] [Warning] Ignoring Column_802 , only has one value
[LightGBM] [Warning] Ignoring Column_809 , only has one value
[LightGBM] [Warning] Ignoring Column_810 , only has one value
[LightGBM] [Warning] Ignoring Column_817 , only has one value
[LightGBM] [Warning] Ignoring Column_818 , only has one value
[LightGBM] [Warning] Ignoring Column_821 , only has one value
[LightGBM] [Warning] Ignoring Column_822 , only has one value
[LightGBM] [Warning] Ignoring Column_846 , only has one value
[LightGBM] [Warning] Ignoring Column_895 , only has one value
[LightGBM] [Warning] Ignoring Column_899 , only has one value
[LightGBM] [Warning] Ignoring Column_954 , only has one value

@Laurae2 I separate the verbose and record now.
verbose`` will control all message output and set record=TRUE``` will save eval result into booster.

@guolinke Thanks, works perfectly, it requires the three following parameters to force full silence and keep recording:
verbosity = -1 # Set higher if you want messages such as "[LightGBM] [Warning] Ignoring Column_85 , only has one value"
verbose = -1 # Hides nearly all output
record = TRUE # Keep recording messages

This will be very useful for any batch users who wants to see only the issues/errors.


Example Training data

Hi, this seems like an amazing breakthrough to me, and I'm not sure how it isn't getting more attention.
Anyway, I was looking at your docs and particular your scripts. Do you all have links to the training/testing jsonl files you used? I wanted to look at it to get an idea of what to pass in.

Hi, thanks a lot for your attention! I simply don't know how to earn this work more attention, very glad to hear any advice :-).
As for the dataset, I have released it here: https://huggingface.co/datasets/dwzhu/PoSE-Datasets/tree/main. Hope they may help!

Frankly, I don't either. I am currently looking into the ability to use Llama to upgrade old dotnet code to a newer version. The problem is that most repos are over 4k tokens. I'm hoping to use the PoSE technology to extend the context of a specialized dotnet llama2 model.


Position Sticky fails in Edge on  or

This is a follow up to #4166
We have had Microsoft confirm its a bug on Edge 16 and Edge 17.
I also added linked to the bug.

Thanks!


feat: prioritize banxa over onramper

Description
Makes sure Banxa is always prioritized over Onramper, should both
options be available
Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[x] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #4914
Risk

None
Testing

Go to fiat ramps -> buy ETH
Ensure Banxa is displayed before OnRamper

Engineering

^

Operations

^

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #5154   👈

This comment was auto-generated by Graphite.


MINOR: Reset timer when all the buffer is drained and empty

For scenarios where the incoming traffic of all input partitions are small, there's a pitfall that the enforced processing timer is not reset after we have enforce processed ALL records. The fix itself is pretty simple: we just reset the timer when there's no buffered records.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

To me the test numbers are kind of arbitrary. Could we define certain time point as constant, like 100L and 250L?

We set the max.task.idle to 100 and that's why we tested 99 v.s. 100 and 249 v.s. 250 etc for that config. I think it's kind of an overkill to define them as constant though.


Fixes for 2.0

Hi,
Here is my changes of library, that my code is working.
DateRangeExpression fix typo
TypeMapping missing SourceField
MultiSearchRequest operation index, type should override request index, type if exists

Thanks for the PR @lukapor and for making me look at multisearch indices and types! :+1:

Thanks for merge.
New logic for setting index and types for multi search is ok, I am setting only operation index, request index/type is null by me.

Awesome :smile:


 Can someone help me?

How did you install NM?

How did you install NM?

I dragged KoboRoot.tgz to the .kobo folder. it did show the updating and the dont power off screen

Which OS are you using?

Which OS are you using?

macOS Monterey 12.0.1 (21A559) and I have the standard M1 MacBook Air.

Im waiting for the new version

Sorry about the delay, I missed the reply.
How did you install NM? It should be compatible with 18838.

I will try again later today.

It worked! Found out that my MacBook automatically de-zips files, so it came out as KoboRoot.tar, but the .tgz file was there in the trash bin!

Oh, that. I forgot that macOS does that since I haven't used it in a while. I'm going to have to figure out a way to stop it from doing that.

Maybe..... Just package KoboRoot.tgz in a zip file?


Test isolation

What changed?

Why?

How did you test it?

Potential risks

Release notes

Documentation Changes

Pull Request Test Coverage Report for Build 018993b0-a85d-474f-8449-1dd5928ca34a

35 of 59 (59.32%)  changed or added relevant lines in 2 files are covered.
27 unchanged lines in 7 files lost coverage.
Overall coverage increased (+0.04%) to 57.084%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

service/matching/taskListManager.go
3
4
75.0%

service/matching/taskReader.go
32
55
58.18%

Files with Coverage Reduction
New Missed Lines
%

common/task/weightedRoundRobinTaskScheduler.go
1
89.64%

service/matching/matcher.go
1
89.21%

common/backoff/retry.go
2
92.11%

service/history/task/transfer_active_task_executor.go
2
72.24%

service/matching/taskReader.go
2
80.0%

service/matching/taskListManager.go
3
80.89%

common/persistence/sql/sqlExecutionStore.go
16
63.86%

Totals

Change from base Build 01898f3c-1d7c-470e-93d6-4073fc9d2ada:
0.04%

Covered Lines:
87207

Relevant Lines:
152769

💛  - Coveralls


KI Web V2 - competition form - Make list support multi level

Finding :

finding on new editor, list only support one level.
Expectation :

Make sure list support multi level, minimum 2 level, sample https://kompetisi.id/competition/TVRnek53PT0/regulations/ikutan-rapha-festive-500-bisa-dapat-saldo-linkaja-dari-maugowes

hold, editor not supported


Stronger 'this' warnings - suggestion

Would it be possible to add (or did I miss an already open issue regarding) support for a stronger this assertion?
Coming from strongly typed languages I have to say that worrying about the this context adds a lot of noise to coding.
I've read this in typescript wiki page and it helps identifying the issue a lot, but I really don't feel safe coding when this issue is only identified in runtime.
The wiki page suggests red flags for this issues, is there a chance of requesting this support by the compiler? Some strict mode flag that makes the compiler merely identify places that instance methods are passed by reference instead of invoked and place a warning/error?

See #513

It is worth mentioning that another option is to simply avoid this entirely. Unlike the languages to which you are likely alluding, this is only necessary when you are using classes. again, the languages you are referring to as comparison points, likely do require the use of  classes, but TypeScript does not.
Honestly, without experimental and prospective features like decorators, classes have relatively weak expressive power.

@aluanhaddad well, while this is desirable in a class, it is still quite easy to make mistakes.  For a lot of people classes are not really optional.  In some cases they are preferred and in some cases, if people are using a framework, they are required.  But it isn't just classes, it is any of the instance based object creation patterns you find in JavaScript, including constructor functions and factories.
I think the point is, almost all of the issues related to this usage are collated in #513 and have their respective issues.  Scoping and binding of this is one of the biggest challenges for JavaScript and I don't know anyone who hasn't screwed it up now and again, and TypeScript can only help so much.  The most common mistake I would see these days is something like this:
class Foo {
  value: string;
  constructor(target: HTMLElement) {
    this.value = 'bar';
    target.addEventListener('click', this.onclick);
  }
  onclick() {
    console.log(this.value);
  }
}

const button = document.createElement('button');
button.innerHTML = 'Click Me!';
document.body.appendChild(button);
const foo = new Foo(button);

When you click on the button, you won't get bar logged, you will get an empty string.  The challenge is, can TypeScript help us statically identify the error?  If so, there is likely an issue linked off of #513, otherwise it is up to us.  this scoping can and is dangerous, even with the help of TypeScript.

@kitsonk you are of correct.

Automatically closing this issue for housekeeping purposes. The issue labels indicate that it is unactionable at the moment or has already been addressed.


Support IMAP username separate from email address

As a security measure, some services use a random token as the IMAP username, rather than the associated email address.
Currently the IMAP username is also used as the user's own email address, as noted in #999.
I suggest that the form for entering IMAP details should have a field for email address, which is then used as the default for the IMAP username.
For backwards compatibility, the IMAP username should continue to be used where no email address is provided.

Starting from the email address, it should also be possible to suggest an IMAP server name, as<EMAIL_ADDRESS>Not sure we'd want to go the whole hog with autodiscovery (probing DNS for various domainnames, then probing connections on various ports)


jcr:createdBy value not displaying in asset detail page

Hi,
We need to display the created user info in the detail page. For that, we have mapped jcr:createdBy property to a metadata property in the asset detail page. In author, it's showing the user name. But when published, it's displaying 'sling-distribution-importer' instead of the actual user name/email. Could you please help on how to resolve this? Any permission missing?
Thanks,
Rahul

@RahulMohann hi - sorry for the delay - i was out of town.
This sounds like something to investigate with AEM rather than ASC, since ASC has nothing to do w/ how this value is set.
Assuming you receive feedback from Adobe Support that this is the expected value on AEM Publish, I expect youll need to create a custom AEM Author workflow that copies the jcr:createdBy value to a custom property on the asset ex (myApp:createdBy or whatever you want to call the property). Depending on ehwat you want to show, you may want to copy more human friendly user info over as well -- for example, if your user ID's are email address, you could use this WF to look up the first/last name on AEM Author for that user, and copy that into myApp:createdByFullName and also display that incase ppl havent memorized emails (and its very possible all your authors will have accounts on AEM Publish, so you wont be able to look them up "on render")
Here's some info on how to create a custom namespace in AEM if you end up wanting to go down that route:
https://experienceleague.adobe.com/docs/experience-manager-learn/cloud-service/developing/advanced/custom-namespaces.html?lang=en
Please re-open if you have more Qs on this!

Hi @davidjgonzalez ,
Thanks for the update. I've tried using the available workflow process step - 'Copy Properties' , which I think is part of acs commons. But it's not working as expected. This is the property mapping I've given in the workflow step

./jcr:createdBy => ./jcr:content/metadata/dam:uploadedby .

I'm getting workflow failure with message - com.adobe.granite.workflow.WorkflowException: Process resource is null.
Could you please let me know if there is anything wrong in this?
Thanks,
Rahul

What version of acs commons?
What line is that exception on?
And how are you tying that wf process step in the wf? (Which wf and how is it triggered)
This is saying the workflow engine is passing a null resource to the workflow model, which is strange.

ACS commons version is 5.6.0.
What I've done -

Created a new workflow model.
Added a process step - 'Copy Properties' with argument './jcr:createdBy -> ./jcr:content/metadata/dam:uploadedby'
Synced the workflow.
Started the workflow - pointed to an asset as payload that have these fields in its metadata.
Workflow failed.

Complete exception:
com.adobe.granite.workflow.WorkflowException: Process resource is null
at com.adobe.granite.workflow.core.job.HandlerBase.executeProcess(HandlerBase.java:134)
at com.adobe.granite.workflow.core.job.JobHandler.process(JobHandler.java:300)
at org.apache.sling.event.impl.jobs.JobConsumerManager$JobConsumerWrapper.process(JobConsumerManager.java:502)
at org.apache.sling.event.impl.jobs.queues.JobQueueImpl.startJob(JobQueueImpl.java:351)
at org.apache.sling.event.impl.jobs.queues.JobQueueImpl.access$100(JobQueueImpl.java:60)
at org.apache.sling.event.impl.jobs.queues.JobQueueImpl$1.run(JobQueueImpl.java:287)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

so, based on this stacktrace this is a product issue (you can see there are no ASC or ACS commons package sin the stacktrace).
This makes me think:

You started the WF Model on the payload incorrectly (how exactly did you do this?)
There's a bug in AEM, which seems unlikely for something like this.

Hi @davidjgonzalez ,
Just uploaded few screenshots if that helps. Or if you need, I can share the workflow model as well. Thanks for your support.
Thanks,
Rahul

Hi @davidjgonzalez ,
Any idea why this is happening.
Thanks,
Rahul

@RahulMohann nope - can you try starting that WF Model

without the custom WF Process step
With a WF process step, but pick one of the OOTB ones

If if has the same error with 1 or 2 then it sounds like there's an issue in AEM we cant fix.
If this only effects the ACS Commons WF process, then we could look to fix it there - tho im skeptical of this since i dont see any ACS packages in the error stack strace.

Hi @davidjgonzalez ,
I tried the below two steps.

without the custom WF Process step --> Worked fine.
With a WF process step, but pick one of the OOTB ones --> Worked fine.

I'm just sharing the argument I'm giving for the copy properties process step. Hope this is the correct format?
./jcr:createdBy -> ./jcr:content/metadata/dam:uploadedby
Thanks,
Rahul

Hi @davidjgonzalez , A gentle reminder. Any update to share? Or do I need to write a custom process to copy the value from createdby property to my custom field?
Thanks,
Rahul

@RahulMohann i recommend writing your own - im not sure why your AEM isnt processing a custom WF process -- the stacktrace doesnt seem to invoke any of the custom code so im at a loss. Sorry!


Access Hive on HBase metadata

Currently Astro has its own metadata stored in HBase. It create inconvenience if user want to reuse the same table metadata, if there are already tables created in Hive on HBase.
Request to implement a feature that astro can access and use the hive hbase storage handler metadata.

I think we should support at least following scenario:
scenario 1: create table in Hive,load/query/update hbase through hbase bulkload, query from Astro or Hive
scenario 2: create table in Astro,load data into astro with Astro fomat,update from Astro, query from astro
scenario 3: create table in Astro,load data into astro with HBase format,update from Astro/Hbase, query from astro/hbase/Hive

To support above scenarios .

Astro should read the metadata from Hive and  query Hbase tables by using that metadata.
Astro should support the same compatible datatypes of Hbase.
Astro should open KeyFactory interface to compatible with HbaseKeyFactory, so that any types of  complex composite keys of Hive Hbase can be accessed through Astro.

Cons : Critical Point ranges may not work properly if you open the KeyFactory interface.Need to verify this.
I will work on it.

PR https://github.com/HuaweiBigData/astro/pull/45 has been raised. Please review it.


Example universal app

Not really an issue per se, but I'm unsure of where to file this (short of sending a presumptuous PR). I've put together a no-frills, easy to understand example of a universal app using react-helmet with react and react-router. I'd like to suggest its inclusion in the docs.

react-helmet-example: https://github.com/mattdennewitz/react-helmet-example.

Thanks!

This is great, thanks!  I'd like to see some of the more advanced usage (like nesting and duplicate tags in the same component).  The app is very easy to follow.

@cwelch5 Great! I've added nested routes. Take a look!

Thanks for the great examples.  We got a link added to the README in 1.1.3


Clarification on Network Flow Limit

Does the network flow limit apply to internal VM traffic? That is, application A and B residing in the same VM and communicating between A and B via internal ports. Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c7c054b7-422c-c3b0-fc8e-91314231ee0c
Version Independent ID: 42825540-2137-92e3-08b0-0dbd84089eda
Content: Azure virtual machine network throughput
Content Source: articles/virtual-network/virtual-machine-network-throughput.md
Service: virtual-network
GitHub Login: @steveesp
Microsoft Alias: steveesp

@delphikit , Ideally you will not see the flows between 2 applications in the same host in the metrics. These network limit will not apply for the communication between 2 application in the same host.

@delphikit , Do you have any update for me?

@delphikit , Please tag me if you have any further questions.


both visual and textual prompts demo?

After training, our model becomes familiar with all prompt types and supports a variety of compositions, such as no prompts, one prompt type, or both visual and textual prompts using the same model and weights.
the demo code is only use one prompt type? How can i test both visual and textual prompts?

The demo is including all types of prompt.


Adds an import test to devicelab

The test ensures that certain less common import styles work with hot reload in Flutter.

Unfortunately, I have no idea why the --watch analyzer command returns errors that the regular analyzer command doesn't produce: https://travis-ci.org/flutter/flutter/jobs/225421624.
@pq Any idea what to do?

Nevermind, I found the problem. Will update this PR tomorrow.

I'd be curious what the root of the problem was...  Glad you got it sorted!


Fixes 2583,2425: add account_id to task table, trim error

Summary
Adds account_id to tasks table.  This makes querying by account_id easier and faster.
Note that after this change, any existing tasks will have a null account_id, but since we are still in stage, that is okay.
Change error length to 4000 chars, and auto trim error if its longer than that
I did these in a single PR as they would normally conflict
Testing steps

Deploy this pr and migrate the db
Snapshot some repository
Use the admin tasks interface
filter by account id

Verify nothing is broken, behavior should be the same
Unsure how to test the error length, i wasn't able to reproduce it but the unit test should prove that its solved

https://issues.redhat.com/browse/HMS-2583

https://issues.redhat.com/browse/HMS-2425

Hi, introspection seems to be misbehaving. Repos with example.com never finish introspecting. I see three  "delete-repository-snapshots" tasks which have been running for nearly 24 hours.

/retest

i wasn't able to reproduce, are you able to?

i wasn't able to reproduce, are you able to?

Hi, I deployed new image and now introspection completes as expected.
Thank you.

Hi
I deployed this PR, ran automated tests and made manual check in UI. Works as described, I can filter on account ID, org ID, all works as expected.
API test result pending.

Tested again and ran all API tests.
Thank you.

@rverdile Merge after your approval.