Regression in rule handling

I was messing around examples in order to test CLI extensibility for doc generation, and got some weird behavior when switching from distributed version to master.
Here is an example:
import textx

grammar = """
DSL:
    commands*=BaseCommand;
BaseCommand:
    (Require | Group) '.'?;

Require:
    'require' /[a-z]/;
Group:
    'group' hello=/[0-9]/;
"""

metamodel = textx.metamodel_from_str(grammar)

With current distributed version (the one installed by pip, 1.8.0), i got no problem. Now, if i use the master branch (by cloning this repository), i got the following stacktrace:
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    metamodel = textx.metamodel_from_str(grammar)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/metamodel.py", line 597, in metamodel_from_str
    language_from_str(lang_desc, metamodel)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/lang.py", line 931, in language_from_str
    TextXVisitor(parser, metamodel))
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/arpeggio/__init__.py", line 1244, in visit_parse_tree
    getattr(visitor, "second_%s" % sa_name)(asg_node)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/lang.py", line 218, in second_textx_model
    self._determine_rule_types(model_parser.metamodel)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/lang.py", line 369, in _determine_rule_types
    _determine_rule_type(cls)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/lang.py", line 365, in _determine_rule_type
    _add_reffered_classes(rule, cls._tx_inh_by, start=True)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/lang.py", line 353, in _add_reffered_classes
    _add_reffered_classes(r, inh_by)
  File "/home/lbourneu/Documents/projects/textx-dsldoc/venv/lib/python3.6/site-packages/textX-1.8.0-py3.6.egg/textx/lang.py", line 358, in _add_reffered_classes
    'is not allowed.'.format(cls.__name__))
textx.exceptions.TextXSemanticError: b'Referencing multiple rules from a single choice of abstract rule "BaseCommand" is not allowed.'

Removing the hello= do fixes the problem.
The other way to fix it is to create an intermediate rule, like this:
BaseCommand:
    BaseCommandRaw '.'?;
BaseCommandRaw:
    Require | Group;

Is seems to me that it is a regression. Or is it linked to #155 ?

Thanks for the report and sorry, I totally missed this issue. Stumbled upon it today as I was working on some object processor calling issues.
This do look like a regression related to #134 and #128 and indeed this PR introduced check that disallows referencing multiple rules from a single alternative of abstract rule. In this case it seems that it got it wrong.
I'm investigating the issue.
BTW, if nobody responds on the issue for a week or so please ping the tread by asking on progress. We should at least put labels whether it is a bug or some expected behavior.


Propose to add docker-compose file

Many developers will be glad to use loghouse for local development or development without kubernetes.

Thank you for your issue! Sadly it's not in our focus at the moment. However PRs are very welcome to implement it.


Support for post-installation hooks

We have an use case where we would like to perform additional configuration steps once the cluster has been installerd. These steps will most probably need the kubeconfig of the admin of the cluster to run. Will Hive support this kind of post-install hooks?
CC: @zgalor

Can you provide some examples?

In general I'm hoping yes, if you have to reach out to the end cluster, ideally Hive will do it. We just need to work together to beat the requirements down into something generic.

Some examples that we have already been asked to support:

Configure image streams.
Apply custom branding to the console.
Create custom cluster roles, for example a role that gives the customer a limited admin capability.
Add custom Prometheus rules and alerts.

I think that what all those have in common is that they are "programs" that need admin access to the cluster, using just the API. You could maybe consider having something similar to the list of init containers of a pod, so that the user can tell Hive "I want to run this container with this command line, and with access to the kubeconfig of the cluster". That way the post-installation steps can be quite isolated from Hive, and developed independently by the user of Hive. I can, for example, write one post-installation step in Python, another in Go, another in Ruby, etc, without interfering with the Hive code.

The idea of having an operator for this looks reasonable to me. @aweiteka, @mwoodson did you consider creating that kind of operator?
Federation looks interesting as well, but maybe for a more long term future.
@dgoodwin If we use the operator approach, then we will need a mechanism to have that operator deployed as part of the installation of the cluster. Will Hive do that? How will the API look like?

Hive will do that (unless we can find a better way to meet your need), it might even be backed by Federation. We don't have a design for it yet so I can't really comment on what the API would look like, I'd need to do some exploring around the data types that are involved with delpoying operators in a cluster today.
Is this a blocker for your side @jhernand ?

It isn't a real blocker today, but it will become a blocker very soon, in the next sprint. We need to provide @aweiteka and @mwoodson with a mechanism that they can use to customize the cluster after installation. If it isn't part of Hive then we will need to invent something, which I'd rather avoid. So, I think it is better if you do consider it a blocker.

CC @abutcher @csrwng @joelddiaz  we'll discuss how and when we can find someone to work on this during scrum.

Operator-based post-install API config management seems consistent with the 4.0 goals we've been tracking. I would prefer to leave federation out of the discussion since it's a significant architecture shift from ansible playbooks run external to the cluster. The requirements include each cluster to have some unique config so we cannot apply the same config across all clusters. I was interested in understanding what role OLM plays in this use case.

Ok @csrwng is going to try to start looking into how we can best support this ASAP.

Can we get examples of what kind of things you might do with this operator in a cluster that would vary depending on the cluster?
We're going to do a little investigation into Federation because if we build a mechanism to deploy arbitrary operators into clusters, it winds up looking like something dangerously close to federation, and they've specifically asked us not to re-implement federation without talking to the team first. From ops/SD standpoint I doubt much will change, it's still an operator you write being deployed to do what you need to do, it's just the means by which we get it deployed into the cluster.
Config will need to be considered carefully if there are substantial variations per cluster.
Sounds like your operator would be OLM managed, but we're not experts in this area, just getting used to it now.

I've done some investigation on Federation and it looks like the right fit for what's needed here.
Assuming that we can handle the joining of the child cluster to the Hive cluster for you automatically, all you would need to do for every artifact that you need created in child clusters is to create a template resource and a placement resource. The template resource describes the resource you want created in every child cluster. This can be a namespace, configmap, deployment, etc. You can see an example configmap template here. The placement resource tells federation in which clusters to create the artifacts. This can be done by explicitly enumerating cluster names or simply selecting them via labels as described here. This is a sample placement resource.
For now, I will focus on the work to automatically federate each child cluster when federation is present.

@csrwng When is federation expected to be part of OpenShift? I mean, will be able to use a solution based on federation by Feb 2019?

@aweiteka @mwoodson @jhernand I think you could proceed with development of an operator to run in-cluster and test against 4.0 clusters, and assume that we will deliver a way for this to be deployed automatically by the Hive cluster. Essentially don't treat federation as a blocker while we get the integration sorted out, just start with the operator whenever you can.

@jhernand federation is usable today and we will deploy it to the hive cluster along with the other hive components.

My only request is that we don't couple too tightly to federation. The architecture may be perfect but right now it introduces yet another change in how we interact with clusters. Its maturity level is concerning.
On the working assumption that operators are the primary way to extend a cluster, I am focused on ensuring OLM, and possibly the Marketplace, is being leveraged to manage operators. How we bootstrap CSVs, and possibly a catalog with subscriptions, is an open question in my mind.


additional props for keep-alive to specify the pages

Featured
additional prop for keep-alive to specify pages that needs to be kept alive. as right now all pages get kept alive, but it is not needed (and cause problems) on some pages! for example on my search page my route query differ but still gets the previous result when come to page again from a different search!
Solution
just like :keep-alive-props="{max: 10, exclude: component}" add exclude and include for pages too!
Describe alternatives you've considered
i tried to use v-if="myPageRoute" on <Nuxt /> to check if i'm on excluded page, no keep-alive, but didnt work.
i also moved all my codes to a component and excluded  that component, but as that had branched child components, didn't work either.

it also affects the savedposition on scrollbehavior

We are approaching the Nuxt 2 EOL date (June 30, 2024) - see this article for more information. I'm closing this issue as it's marked as a Nuxt 2 related enhancement and it's not a critical issue.
That doesn't mean it might not be relevant for Nuxt 3. If it is, please feel free to open a new issue (or just comment, and I can reopen it). 🙏
Thank you for your understanding and for your contribution to Nuxt! 🎉


feat(snownet): automatically discover host candidates

I had an idea that is quite a big deal. Instead of manually discovering host candidates by iterating interfaces and adding all of them, we automatically generate host candidates every time we receive traffic on from a certain interface. Initially, you might think that this is a catch-22: How do we generate traffic without having host candidates? The answer is: relays!
When we initiate a new connection using snownet, we add a list of STUN and TURN servers. We will immediately attempt to talk to both of them, sending STUN bindings to the former and ALLOCATE requests to the latter. As the replies come in, we know, which interface they have been received on. That particular interface is an excellent host candidate because we've already proven connectivity to a STUN or TURN server. In fact, for hole-punching, we will need to send traffic via that same interface to reach the gateway (otherwise the srflx candidate won't work anyway).
There is only one "edge"-case in which this doesn't work: When you want to make a connection between a client and a gateway on the same subnet yet without connectivity to a relay. At that point, I'd argue that your network topology is broken anyway. If you can't talk to a relay, you probably also cannot talk to the portal, meaning the signaling protocol also doesn't work.

There is only one "edge"-case in which this doesn't work: When you want to make a connection between a client and a gateway on the same subnet yet without connectivity to a relay. At that point, I'd argue that your network topology is broken anyway. If you can't talk to a relay, you probably also cannot talk to the portal, meaning the signaling protocol also doesn't work.

This is a very common edge case. It will happen any time a remote worker comes into the office for example. It also happens when folks are evaluating whether to use Firezone or not -- one typical test would be to install a gateway on another machine and then ping it. It will also happen for servers with headless clients on them talk to other servers (resources) in the same VPC.
Would that mean all of those connections need to go through a Relay? That would be... not ideal.
I know I'm probably not fully grasping why we can't gather host address directly from interfaces, but I'll give the discussion another read.
But unless we run many more Relays I don't think this will be a competitive enduser experience (or I'm not fully understanding)

Ah, I think I'm understanding a bit better now (coffee helps!).
This PR would essentially ensure that any data plane communication only happens on the interface used to reach the Relay, an external service.
This is probably OK for the "workers in office" use case and possible "evaluating in my homelab", but not great for server-server communication. Those are typically "multi-homed" to have an interface for default egress and then another attached to the VPC. I first thought maybe this isn't an issue because hairpinning could still work off their shared NAT gateway, but I'm not so sure AWS or other cloud providers and data centers will support that. That means these servers will "talk to each other" using their public IP addresses, which won't be ideal for latency or bandwidth costs.
So, back to the original problem then. I'll chat with @conectado today about it and see if something comes up.
I think being able to use host candidates that are not the default egress interface will soon be a requirement, if it's not already. We can kick the can down the road, but we have full control now right? ;-). We can make this an excellent (read: competitive) experience for customers.

Ah, I think I'm understanding a bit better now (coffee helps!).
This PR would essentially ensure that any data plane communication only happens on the interface used to reach the Relay, an external service.
This is probably OK for the "workers in office" use case and possible "evaluating in my homelab", but not great for server-server communication. Those are typically "multi-homed" to have an interface for default egress and then another attached to the VPC. I first thought maybe this isn't an issue because hairpinning could still work off their shared NAT gateway, but I'm not so sure AWS or other cloud providers and data centers will support that. That means these servers will "talk to each other" using their public IP addresses, which won't be ideal for latency or bandwidth costs.
So, back to the original problem then. I'll chat with @conectado today about it and see if something comes up.
I think being able to use host candidates that are not the default egress interface will soon be a requirement, if it's not already. We can kick the can down the road, but we have full control now right? ;-). We can make this an excellent (read: competitive) experience for customers.

Adding to this, there's another problem with this approach, we can't discover the address of any interface that isn't used. Meaning, it's not only interfaces that can't reach the relays that aren't used as local candidates, it's also interfaces that could reach the relays but are not prioritized by the routing table.
@thomaseizinger I think it's better to leave things as they are right now, we're losing a lot of conditions where we should be able to hole-punch for a bit of simplicity on the API

@thomaseizinger I think it's better to leave things as they are right now, we're losing a lot of conditions where we should be able to hole-punch for a bit of simplicity on the API

I'd like to challenge the "a lot of conditions" bit. Can you draw a network topology where we don't have connectivity with this apptoach?
I'd like to suggest that we continue down this path. The behaviour suggested here is still correct. Traffic coming in on a socket 100% gives us a viable host candidate. We can always add another API to manually add them later. But before we do that, I'd like to understand the network topology where this doesn't work and be able to write a test for it instead of blindly grabbing candidates from all interfaces.
If you want, we can also put the API back and just not use it for now but I'd like to merge this PR because it is a step forward.

but not great for server-server communication. Those are typically "multi-homed" to have an interface for default egress and then another attached to the VPC. I first thought maybe this isn't an issue because hairpinning could still work off their shared NAT gateway, but I'm not so sure AWS or other cloud providers and data centers will support that.

Sone questions / points:

Server-server in the same VPC assumed headless clients right? I didn't know that we want to support this already (and that it is a usecase at all running them in the same VPC).
Only party needs to add an extra host candidate for this to work. For example, it is probably better to add functionality to the fateway to iterate all its interfaces and add additional candidates but not do this on the client where we might have stuff like docker networks, other VPNs etc.

Do we agree that this PR is an improvement and that we can always add an API back that manually adds host candidates?

I'd like to challenge the "a lot of conditions" bit. Can you draw a network topology where we don't have connectivity with this apptoach?

We will always have connectivity due to relayed connections but we do lose direct connections, in these 2 cases:

2 hosts in a local network through a given iface, yet they connect to WAN through a separate interface that due to symmetric nat can't connect to each other
The LAN interfaces can even be connected to WAN but just no the default route to reach the relays

Do we agree that this PR is an improvement and that we can always add an API back that manually adds host candidates?

I'm not clear on what we gain, save for having to walk through local interfaces which we know that we can do

2 hosts in a local network through a given iface, yet they connect to WAN through a separate interface that due to symmetric nat can't connect to each other

They also must each go through separate WAN interfaces to the relay, otherwise the router would directly connect them.
You are right that in this case, we don't discover a more optimal connection. Some notes though:

That is not hole-punching.
Is that really a common enough topology to say we are missing "a lot of conditions"?
Like I said in the previous comment, adding all interfaces on the gateway is enough to fix it. We can avoid a lot of (likely rubbish) candidates on the client.

Do we agree that this PR is an improvement and that we can always add an API back that manually adds host candidates?

I'm not clear on what we gain, save for having to walk through local interfaces which we know that we can do

Error handling within snownet is improved. The error condition of "you gave me data on a socket I don't know about" simply no longer exists. (This reminds me I need to remove the error variant).
No combinatoric explosion of candidate pairs.

I am happy to add the API back. Then this PR is a strict improvement over what we had before. I'll also try to model the above network topology as a docker-compose test.

2 hosts in a local network through a given iface

This might be a silly question but what is the point of using firezone in this topology? You are already in the same network as your gateway so you should be able to reach all the resources.

They also must each go through separate WAN interfaces to the relay, otherwise the router would directly connect them.

Only if hair-pinning is supported

You are right that in this case, we don't discover a more optimal connection. Some notes though:
* That is not hole-punching.

* Is that really a common enough topology to say we are missing "a lot of conditions"?

* Like I said in the previous comment, adding all interfaces on the gateway is enough to fix it. We can avoid a lot of (likely rubbish) candidates on the client.

but we need to use local candidates from both to achieve this, or am I missing something?

* Error handling within `snownet` is improved. The error condition of "you gave me data on a socket I don't know about" simply no longer exists. (This reminds me I need to remove the error variant).

* No combinatoric explosion of candidate pairs.

👍 alright, yeah.

I am happy to add the API back. Then this PR is a strict improvement over what we had before. I'll also try to model the above network topology as a docker-compose test.

I'd be okay with that.

This might be a silly question but what is the point of using firezone in this topology? You are already in the same network as your gateway so you should be able to reach all the resources.

There are a few reasons why:

Rarely, a gateway might have another interface that can reach the resources
More commonly, the user always logs into firezone, since not all resources are available locally and can be using it on different networks to different gateways, some of which are within these conditions

This might be a silly question but what is the point of using Firezone in this topology?

I think this gets to the heart of the discussion.
Firezone is only applicable if there's a stateful firewall allowing egress in between the client and gateway that need to communicate. Otherwise, what's the point?
In many cases, there's only one stateful firewall, especially for user clients. In cloud topologies, data centers, and on-prem, however, networks are commonly more complex. It's not uncommon to have topologies like this:

In the topology above, the Firewall between App Servers and DB would need to be configured to explicitly allow traffic. Firezone wouldn't be useful.
But keep in mind that host firewalls are also stateful firewalls.
So with Firezone, instead of managing lots of firewalls in between each of your networks that need to communicate, instead you just enable the host firewall (ufw enable) for each host, and hole-punching happens between each host directly. That's the Zero Trust vision at least.
With the candidate gathering change introduced in this PR, communication can only happen between two peers' nearest WAN ancestor, and in some cases not allowed at all (and thus Relayed).
tl;dr
All of the above is to say, I think I'm agreement:

This is a step forward and we should move ahead with this API
Keep the old one in place, but unused (with an appropriately detailed comment)

Awesome! Thank you for the detailed explanation. That makes a lot of sense! :)
Today, we only have clients and gateways and only clients can connect to gateways. Unless I am misunderstanding, then the above diagram suggests that we might eventually have more of a peer-to-peer system where each node can establish connections and also allow other nodes to connect to them. Think multiple (micro) services talking to each other.

Like I said in the previous comment, adding all interfaces on the gateway is enough to fix it. We can avoid a lot of (likely rubbish) candidates on the client.

I have to correct myself here, the above is wrong. We will need host candidates on either end because the local host candidate is the "source" of the traffic and it also needs to target the right interface.
The above topology is very interesting and I think we should explicitly write tests that demonstrate that we can establish a direct connection in that case. Today, that would boil down to correctly using snownet by manually adding that extra candidate. I am curious if we can do better here and somehow discover those reliably.

the above diagram suggests that we might eventually have more of a peer-to-peer system where each node can establish connections and also allow other nodes to connect to them. Think multiple (micro) services talking to each other.

From a graph-theory perspective, yes. A peer-to-peer system would be a digraph where each node can connect to another and vice-versa, and imply that if node A can connect to node B, then B can connect to node A. All edges are bidirectional.
I want to be very clear of an important nuance though: in Firezone (and zero trust networks), we do not want to assume that the converse is also true. That is, by default, edges in our graph are not bidirectional. In the example above, it would be suspicious if the DB node initiated a connection to the App server.
To build such a digraph, we have a couple options. The first is that we can consider two different color nodes. One color node is the "from" node (client), and the other a "to" node (gateway). With these constructs we can build a digraph with all appropriate edges to allow desired communication. In reality, today, that means the possibility running a client and gateway on the same host (possible now if we rename interfaces and handle routing loops). We could optimize this down the road by combining the headless client and gateway into a "headless agent" that can function in client or gateway mode. But I digress, that's a lot of work, and I don't want to get distracted optimizing for that now.
The other way to build such a graph is to consider every node the same color, able to both initiate and receive communication. This bidirectional (by default) digraph is the model Tailscale uses. It then requires an ACL to explicitly state which nodes can talk to each other based on attributes the node has, and which direction the edge points.
Both approaches lead to an appropriate digraph that satisfies desired communication; our bet is an optimization that assumes most edges will only need to be one color or another, which simplifies a lot of things from a UX perspective.

our bet is an optimization that assumes most edges will only need to be one color or another, which simplifies a lot of things from a UX perspective.

Makes sense. I do like the simplification too, let's hope it pans out :crossed_fingers:


Regenerate draft/Edit prompt

Is there a way to regenerate a draft or edit a prompt? I can't find a way to do any of them.

It is unclear which prompt requires clarification. Please specify the desired action you are referring to.

Look.
When asking Google Bard something on https://bard.google.com/, you have a little pencil on the right of your message which lets you to edit your prompt. On Bard's response, you can regenerate draft or select one of three generated drafts (or regenerate them too).

Oh, and also rating responses. That would be cool even if this is waste of time.

Oh, and also rating responses. That would be cool even if this is waste of time.

Nah, its really a waste of time as of now!

Oh, and also rating responses. That would be cool even if this is waste of time.

Nah, its really a waste of time as of now!

Yeah, this is. But at least editing prompt and regenerating draft is useful at least in some way.

Yeah, this is. But at least editing prompt and regenerating draft is useful at least in some way.

That's more of UI implementation tbh. You can configure the UI to edit a provided prompt or re-gen it using the existing API implementation

If we use a reusable session and avoid establishing a new connection for each request, it will likely reduce the overall time.
Secondly, try utilizing the timeout variable.
In fact, the Python program itself does not have a significant impact on latency.
The 'Prompt edit' feature you mentioned essentially involves modifying the existing prompt to fulfill my request. In reality, using Python, it can be resolved more easily by directly modifying the text and pressing Enter.
Therefore, there are no further errors to be resolved regarding this issue, so I am closing it.


Suggestion: auto-determine chunk-size and if chunking is necessary

It would be nice if the filepond field would:

determine the max chunk size dynamically (example)
feed a treshold to JS from which uploads would have to be chunked (if FilePondJS supports this)
OR maybe just always chunk uploads instead (may simplify code maintenance as we it can return to just one handler, and may also ease up server load etc with multiple smaller requests instead of one big upload(?))

Background/rationale
Currently chunkin is a (php level) configurable option. This means a dev would have to know the server environment a project will be ran on, on beforehand to activate chunking. This is often not the case, eg when developing a re-usable module but also when moving existing projects to a different server/environment etc.
Activating chunkin later would currently also require editing the project-code or even the code of another module utilizing FilePond. Not sure if the default 5MB chunk size comes from FilePondJS or if this module sets it, but the default max upload size of an unconfigured PHP is often even just 2MB.

I made a couple of changes in https://github.com/lekoala/silverstripe-filepond/commit/939c3ec9faf4110c704998d7a4bb5ad0e77ebe25
you can set chunk as the default option. chunk size is computed automatically. still need to tweak a few things (like default validation size if chunking is applied, how to determine what is the "reasonable" max size)

@lekoala what do you mean with "how to determine what is the reasonable max size"?
Did you notice the example? That's how I used to determine the reasonable max [chunk] size (eg 90% of the max upload size).

@micschk no i mean let's say you have a server that is by default limited to 10 mb. You don't expect users to upload anything larger. if you enable chunking, they can suddenly upload very large files so maybe it's still a good thing to make this "opt in". Currently i simply removed the limit and you have to set a limit yourself, which is fine i guess since you are expected to know what you are doing if you enable chunking by default on all uploaders

Oh, yes wel basically it is always possible to upload large amounts of data to a server. Only the max amount of data per request is restricted, so if you have a 'max 5MB' regular uploadfield I can in theory still upload 1000 times 5MB.
I think there's two 'parts' to this issue;

the transfer mechanism (to chunk or not to chunk)
the validation (on finishing the upload)

The chunking could be considered a given, as that simply allows uploading large files. Then there's the validation part, that's where this issue should be fixed. I think usually the webserver & PHP layers take care of the low-level stuff like writing the uploaded data to a temp file and then a field like UploadField handles the validation of this written data.
This is comparable to what's happening now with chunked filepond. So if you set a max size on a filepond field instance, it should just return an error once the total size of uploaded chunks has become larger than this maximum.
One extra thing which is actually a benefit of the chunked uploading mechanism is that it might already be possible to return an error before finishing all chunks, namely from the 'handle chunk' function (not sure what's its actual name).

This is comparable to what's happening now with chunked filepond. So if you set a max size on a filepond field instance, it should just return an error once the total size of uploaded chunks has become larger than this maximum.

that's a good one, will look into it :)

closed with https://github.com/lekoala/silverstripe-filepond/commit/f2269503448e8ef8fb0f5d6d0a9384edebe7c4bb


[sival, rv_dm] convert rv_dm_ndm_reset_req test to SiVal

Based on #20355
Fix #19880

I think this can be merged but I'll just ping @matutem and @msfschaffner to double check the device side.

Your code looks good. I do have some doubts on the device code though: from what I can see, it says that it checks that devices in Group2 do not reset while Group3 resets. But all the registers that it checks are in Group3. I am missing something here?

This was updated in #16012, which I don't really understand.

The comments weren't updated for hardware changes to make NDM reset actually do its job, hehe. Reset were reorganized according to https://docs.google.com/presentation/d/1kDKSQTSYhP-Q9GoQLcmapb00ovSeLP6lZ4l0lBfxW18/edit#slide=id.p
In effect, almost everything was moved to the LC reset domain, so we expect to see the reset values on the device side. What is perhaps missing from this test is a check to distinguish between NDM reset or power-on reset, but that likely would necessitate some sort of scratch register in the Debug Module that can be guaranteed not to change. I'm not sure if openocd might cause the values of the dscratch CSRs to change underneath.

In effect, almost everything was moved to the LC reset domain (which then became the sole domain to be reset by NDM reset), so we expect to see the reset values on the device side. What is perhaps missing from this test is a check to distinguish between NDM reset or power-on reset, but that likely would necessitate some sort of scratch register in the Debug Module that can be guaranteed not to change. I'm not sure if openocd might cause the values of the dscratch CSRs to change underneath.

Ah that makes sense, thanks for the clarification.
If one uses the low-level jtag interface, I believe openocd should not touch the dscratch registers. When I looked at the code (some time ago), those were used exclusively for implementing operation that require to steal execution from the Ibex. Hence, if the host is careful and does not issue any such high-level operation, it should be feasible.


Failed to go get on Debian

Installing via
go get github.com/wagoodman/dive

results in the following error
$ go get github.com/wagoodman/dive

# github.com/wagoodman/dive/dive/image/docker
go/pkg/mod/github.com/wagoodman/dive@v0.9.1/dive/image/docker/engine_resolver.go:53:19: undefined: client.Opt
go/pkg/mod/github.com/wagoodman/dive@v0.9.1/dive/image/docker/engine_resolver.go:81:34: undefined: client.WithAPIVersionNegotiation

Environment:

ChromeOS Linux Env (Debian 9)
Go lang: go1.13.4 linux/amd64

Cannot reproduce in a docker container:
docker run --rm debian:stretch-slim bash -c "
apt update -qq
apt install -y git curl
curl -fsSL https://dl.google.com/go/go1.13.4.linux-amd64.tar.gz | tar xzf - -C /usr/local/
export PATH=\$PATH:/usr/local/go/bin
go get -v github.com/wagoodman/dive
"

Thanks for digging into this @eine. Looks like I had to unset GO111MODULES on my box to make this work. 🙈 Closing.

Same on Fedora 30, go1.13.5.  This worked (no it's GO111MODULE not GO111MODULES), not sure why as this project does have a go.mod & go.sum :shrug: :
env GO111MODULE=off go get -v github.com/wagoodman/dive

Aha, got it.  I happened to be running the go get github.com/wagoodman/dive inside another go project directory.
That was interpretted as "add a depedency to the containing project", messed with my go.mod & go.sum, and got conflicts.  Using GO111MODULE=off didn't touch my project, but that's a silly fix, the real fix is run go get outside any go projects (say from the home dir).


Switch tools management to bingo

All the files in .bingo are auto-generated

@everettraven the link you referenced in your comment points to the entire diff, were you referring to anything specific? I am assuming you're referring to removing goreleaser from the fetch script. If so, I fixed it in the latest commit.
Let me know if I'm missing anything else.


dockerTools: Support files directly under /nix/store

Motivation for this change
See: https://github.com/NixOS/nixpkgs/pull/91084#issuecomment-653496223
Previosly, stream_layered_iamge.py was assuming that all the paths under
/nix/store were directories, which is clearly wrong. This PR adds a test
for that and fixes the issue.
Also makes sure that the files inside a layer added in a sorted order
to make the results more deterministic.

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Looks alright. rglob appears to include dotfiles as well, so that's good.
For the test you could run nix-store --verify --check-contents (with buildLayeredImageWithNixDb). That should catch all tarball issues.

For the test you could run nix-store --verify --check-contents (with buildLayeredImageWithNixDb).

Thank you @roberth , this was a great idea. It already uncovered one more issue regarding the handling of the hardlinks, so this PR now also fixes that.

Great job fixing both issues. Thank you @utdemir!


Finalize plans for UP Comms Content Changes and Align with stakeholders

Problem Statement
We want to develop a plan to update our comms based on the new information we learned and to assist with Phase 2's focus on migrating senders.
What you need to know

Current drafting space

Acceptance criteria

[x] Remove support for CSV from website and UP comms
[ ] Connect with Glenn and Chris to review existing HL7 Sender responses (that haven't sent us data in the past 60 days)
[x] Finalize plans/roadmap for implementing changes
[ ] Review with stakeholders (Max, Chris, Glenn)

Reviewing and getting feedback today/tomorrow


Crashes with imaginary numbers

I tried converting my complex sympy expression to jax, and got the following error.
I wrote a minimum working example. The I is sympy's variable for a complex number. 1j is Python's version, and they are both treated the same.
from sympy import symbols, I
import sympy2jax

x = symbols("x")

expr = x*I # or x*1j

sympy2jax.SymbolicModule(expr)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:213, in _sympy_to_node(expr, memodict, func_lookup, make_array)
    212 try:
--> 213     return memodict[expr]
    214 except KeyError:

KeyError: I*x

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:213, in _sympy_to_node(expr, memodict, func_lookup, make_array)
    212 try:
--> 213     return memodict[expr]
    214 except KeyError:

KeyError: I

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:180, in _Func.__init__(self, expr, memodict, func_lookup, make_array)
    179 try:
--> 180     self._func = func_lookup[expr.func]
    181 except KeyError as e:

KeyError: <class 'sympy.core.numbers.ImaginaryUnit'>

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
/Users/thomas/Documents/vilde.ipynb Cell 6 in <cell line: 8>()
      [4](vscode-notebook-cell:/Users/thomas/Documents/vilde.ipynb#Y103sZmlsZQ%3D%3D?line=3) x = symbols("x")
      [6](vscode-notebook-cell:/Users/thomas/Documents/vilde.ipynb#Y103sZmlsZQ%3D%3D?line=5) expr = x*I # or x*1j
----> [8](vscode-notebook-cell:/Users/thomas/Documents/vilde.ipynb#Y103sZmlsZQ%3D%3D?line=7) sympy2jax.SymbolicModule(expr)

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/equinox/module.py:131, in _ModuleMeta.__call__(cls, *args, **kwargs)
    129 object.__setattr__(self, "__class__", initable_cls)
    130 try:
--> 131     cls.__init__(self, *args, **kwargs)
    132 finally:
    133     object.__setattr__(self, "__class__", cls)

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:257, in SymbolicModule.__init__(self, expressions, extra_funcs, make_array, **kwargs)
    250     self.has_extra_funcs = True
    251 _convert = ft.partial(
    252     _sympy_to_node,
    253     memodict=dict(),
    254     func_lookup=lookup,
    255     make_array=make_array,
    256 )
--> 257 self.nodes = jax.tree_map(_convert, expressions)

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/jax/_src/tree_util.py:205, in tree_map(f, tree, is_leaf, *rest)
    203 leaves, treedef = tree_flatten(tree, is_leaf)
    204 all_leaves = [leaves] + [treedef.flatten_up_to(r) for r in rest]
--> 205 return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/jax/_src/tree_util.py:205, in <genexpr>(.0)
    203 leaves, treedef = tree_flatten(tree, is_leaf)
    204 all_leaves = [leaves] + [treedef.flatten_up_to(r) for r in rest]
--> 205 return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:224, in _sympy_to_node(expr, memodict, func_lookup, make_array)
    222     out = _Rational(expr, make_array)
    223 else:
--> 224     out = _Func(expr, memodict, func_lookup, make_array)
    225 memodict[expr] = out
    226 return out

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/equinox/module.py:131, in _ModuleMeta.__call__(cls, *args, **kwargs)
    129 object.__setattr__(self, "__class__", initable_cls)
    130 try:
--> 131     cls.__init__(self, *args, **kwargs)
    132 finally:
    133     object.__setattr__(self, "__class__", cls)

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:183, in _Func.__init__(self, expr, memodict, func_lookup, make_array)
    181 except KeyError as e:
    182     raise KeyError(f"Unsupported Sympy type {type(expr)}") from e
--> 183 self._args = [
    184     _sympy_to_node(arg, memodict, func_lookup, make_array) for arg in expr.args
    185 ]

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:184, in <listcomp>(.0)
    181 except KeyError as e:
    182     raise KeyError(f"Unsupported Sympy type {type(expr)}") from e
    183 self._args = [
--> 184     _sympy_to_node(arg, memodict, func_lookup, make_array) for arg in expr.args
    185 ]

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:224, in _sympy_to_node(expr, memodict, func_lookup, make_array)
    222     out = _Rational(expr, make_array)
    223 else:
--> 224     out = _Func(expr, memodict, func_lookup, make_array)
    225 memodict[expr] = out
    226 return out

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/equinox/module.py:131, in _ModuleMeta.__call__(cls, *args, **kwargs)
    129 object.__setattr__(self, "__class__", initable_cls)
    130 try:
--> 131     cls.__init__(self, *args, **kwargs)
    132 finally:
    133     object.__setattr__(self, "__class__", cls)

File ~/mambaforge/envs/kaggle/lib/python3.10/site-packages/sympy2jax/sympy_module.py:182, in _Func.__init__(self, expr, memodict, func_lookup, make_array)
    180     self._func = func_lookup[expr.func]
    181 except KeyError as e:
--> 182     raise KeyError(f"Unsupported Sympy type {type(expr)}") from e
    183 self._args = [
    184     _sympy_to_node(arg, memodict, func_lookup, make_array) for arg in expr.args
    185 ]

KeyError: "Unsupported Sympy type <class 'sympy.core.numbers.ImaginaryUnit'>"

I should add - it's absolutely fine if this is not fixed. I tried sympy2jax just for fun now, since I had already written my (rather large) expression in sympy.

So it should be pretty straightforward to add support. I'd be happy to accept a pull request on this.


hardhat-zksync-upgradeable dependency issue with @openzeppelin/upgrades-core

Running yarn hardhat verify <myAddress> --network zcSyncTestnet results in the following error (it seems like the upgrades-core package doesn't have the ast-dereferencer module now):

The plugins seem to be imported in correct order in my hardhat.config.ts; hardhat-zksync-verify plugin is installed with version 0.1.8

Manually installing the previous version @openzeppelin/upgrades-core worked for me

Manually installing the previous version @openzeppelin/upgrades-core worked for me

Which previous version of @openzeppelin/upgrades-core did you install to make it work?

Manually installing the previous version @openzeppelin/upgrades-core worked for me

Which previous version of @openzeppelin/upgrades-core did you install to make it work?

1.27.3

Hi @statskiy-sovetnik @anhlh-edsolabs, this issue was caused by hardhat-zksync-upgradable plugin not yet supporting ethers v6 upgrade, while the latest openzeppelin packages have new major versions that include this change.
We are actively working on this ethers upgrade, but in the meantime we released a new version of our upgradable plugin - 0.1.3 - that fixes this dependency issue.
Thanks for pointing it out!

Ok thanks!


vBump azdata version

Pull Request Test Coverage Report for Build 496970584

0 of 0   changed or added relevant lines in 0 files are covered.
3 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.001%) to 43.089%

Files with Coverage Reduction
New Missed Lines
%

extensions/import/src/wizard/pages/fileConfigPage.ts
3
72.99%

Totals

Change from base Build 489251011:
-0.001%

Covered Lines:
24328

Relevant Lines:
51753

💛  - Coveralls

Pull Request Test Coverage Report for Build 496970584

0 of 0   changed or added relevant lines in 0 files are covered.
3 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.001%) to 43.089%

Files with Coverage Reduction
New Missed Lines
%

extensions/import/src/wizard/pages/fileConfigPage.ts
3
72.99%

Totals

Change from base Build 489251011:
-0.001%

Covered Lines:
24328

Relevant Lines:
51753

💛  - Coveralls


Fix context and window sizes sides.

Motivation:
As mentioned in RFC 7692 :

The "server_no_context_takeover" Extension Parameter should be used on server side for compression and on client side for decompression.

The "client_no_context_takeover" Extension Parameter should be used on client side for compression and on server side for decompression.

Right now, in PerMessageDeflateClientExtensionHandshaker, the decoder uses clientNoContext instead of serverNoContext and the encoder uses serverNoContext instead of clientNoContext.
The same inversion is present in PerMessageDeflateServerExtensionHandshaker: the decoder uses
serverNoContext instead of clientNoContext, while the encoder uses serverNoContext instead of clientNoContext. Besides the context inversion, the sliding window sizes seem to be inversed as well.
Modification:
Inverse clientNoContext with serverNoContext and clientWindowSize with serverWindowSize for both the Decoder and Encoder in PerMessageDeflateServerExtensionHandshaker and PerMessageDeflateClientExtensionHandshaker.
Result:
This fixes the decompression fail in the case that one of the contexts is set and the other one is not.

@netty-bot test this please

@langchristian96 can you please add a unit test ?

@normanmaurer, not really sure how should I test this (except maybe hardcode the values that didn't work for me while I was using it?). Basically, the contexts for both the server and the client were reversed (meaning if I use the encoder from PerMessageDeflateServerExtensionHandshaker, it will match the decoder from PerMessageDeflateClientExtensionHandshaker in the previous version as well). What didn't work was that when I was connecting to a websocket server (eg bitmex) with client_no_context_takeover set, the server was compressing the message with context, while i was decompressing with no context.

Feeding in data that did not work for you should do the trick

Am 17.10.2018 um 16:34 schrieb Christian Lang<EMAIL_ADDRESS>@normanmaurer, not really sure how should I test this (except maybe hardcode the values that didn't work for me while I was using it?). Basically, the contexts for both the server and the client were reversed (meaning if I use the encoder from PerMessageDeflateServerExtensionHandshaker, it will match the decoder from PerMessageDeflateClientExtensionHandshaker in the previous version as well). What didn't work was that when I was connecting to a websocket server (eg bitmex) with client_no_context_takeover set, the server was compressing the message with context, while i was decompressing with no context.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@normanmaurer Done. Sorry for the delay

@normanmaurer Did the modifications. Also signed the ICLA.

@langchristian96 thanks!

@langchristian96 thanks a lot!
The release is expected within a week.


C# Generic class support

code:
using MemoryPack;

[MemoryPackable]
[MemoryPackUnion(0, typeof(IntAttr))]
[MemoryPackUnion(1, typeof(FloatAttr))]
public partial interface IAttr : IAttrBase
{

}

public interface IAttrBase
{

}

public interface IObjectPool
{

}

[MemoryPackable]
[MemoryPackUnionFormatter(typeof(IAttr))]
public abstract partial class ZAttr<T> : IAttr,IObjectPool
{

}

[MemoryPackable]
public partial class IntAttr : ZAttr<int>
{

}

[MemoryPackable]
public partial class FloatAttr : ZAttr<float>
{

}

error:
MemoryPack.Generator.Roslyn3\MemoryPack.Generator.MemoryPackGenerator\ZAttr_T_.MemoryPackFormatter.g.cs(91,79): error CS0246: The type or namespace name 'T' could not be found (are you missing a using directive or an assembly reference?)
MemoryPack.Generator.Roslyn3\MemoryPack.Generator.MemoryPackGenerator\ZAttr_T_.MemoryPackFormatter.g.cs(91,69): error CS0144: Cannot create an instance of the abstract type or interface 'ZAttr'

mark

Hi. There seems to be a misunderstanding of usage.
MemoryPackUnionFormatter is an option when the base type cannot be given a MemoryPackUnion directly.
Your example works by removing the attributes from ZAttr<T>.
( If that is not what you want to do, I would appreciate it if you could tell me how you use the those types.


CacheBackend doesn't catch redis ConnectionError exception

When Redis service is down and a request to get the status of the services are done, the request returns an exception:
File "/root/.local/lib/python3.6/site-packages/health_check/backends.py", line 30, in run_check
  self.check_status()
File "/root/.local/lib/python3.6/site-packages/health_check/cache/backends.py", line 12, in check_status
  cache.set('djangohealtcheck_test', 'itworks', 1)
File "/root/.local/lib/python3.6/site-packages/django_redis/cache.py", line 39, in _decorator
  raise e.parent
File "/root/.local/lib/python3.6/site-packages/django_redis/client/default.py", line 143, in set
  return client.set(nkey, nvalue, nx=nx, px=timeout, xx=xx)
File "/root/.local/lib/python3.6/site-packages/redis/client.py", line 1519, in set
  return self.execute_command('SET', *pieces)
File "/root/.local/lib/python3.6/site-packages/redis/client.py", line 836, in execute_command
  conn = self.connection or pool.get_connection(command_name, **options)
File "/root/.local/lib/python3.6/site-packages/redis/connection.py", line 1053, in get_connection
  connection.connect()
File "/root/.local/lib/python3.6/site-packages/redis/connection.py", line 515, in connect
  raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error -2 connecting to redis:6379. Name does not resolve.

CacheBackend has an except to capture ConnectionError, but when I debugged the execution, the execution pass through without being catch. Then a BaseException is captured on BaseHealthCheckBackend and raised causing the non-desired exception data returned by the request.

That is expected. Since we don't know which backend you might use. Sadly redis uses their own ConnectionError not the Python 3 one, which we are catching.

If it's expected, why return the exception?
I will try to explain my point of view. CacheBackend can try to check if the backend is alive. Whatever the cache is used by the user, this cache can throw an exception that is not the Python 3 one (e.g. Redis). If a except BaseException is set, that would catch every exception from whatever cache is being used (e.g. RabbitBackend).
This would avoid raise an exception and make the app more useful because it would be able to show the information of all the backends even when the CacheBackend is down.

I don't see the issue with caching problematical or syntax error (but AFAIK it shouldn't happen). I say this because, if the error is catched and CacheBackend returns whatever error message without raising the exception, the health_check service won't be returning the exception response. The response will be the expected one with CacheBackend status as "Unknown error".
But, if you don't like this approach, a more specific catch can be set:
try:
    # some code here
except redis.exceptions.RedisError:
    # some error message

With this code, it will cached the majority of the Redis exceptions. That is because, these Redis exceptions inherit from RedisError that is the base Exception class from Redis.

The change you propose will mean we have to import redis. What if you use memcached and don't have redis installed? The cache backend is pretty generic. This would be something to consider for the redis backend.


Bug Fix issue #36

Fixes bugs with multiple zoomContainers being appended to body when window resized and also multiple zoomWrappers  being wrapped round zoom image when gallery is used with imageCrossfade set to true

Super, thanks!

No problem.


Machine Readable Output

What is the purpose of this change? What does it change?
Provide machine readable output (json format) so that other applications can easily consume output generated by odo. for example, vscode plugin
Was the change discussed in an issue?
fixes #1062
How to test changes?

odo app describe  -o json
odo app list -o json
odo describe  -o json
odo list -o json
odo url list -o json
odo storage list -o json

:white_check_mark: odo build 1323 completed and artifacts can be found here (for commit https://github.com/redhat-developer/odo/commit/b552b57d74 by @surajnarwade)

:white_check_mark: odo build 1324 completed and artifacts can be found here (for commit https://github.com/redhat-developer/odo/commit/32f181f043 by @surajnarwade)

:white_check_mark: odo build 1325 completed and artifacts can be found here (for commit https://github.com/redhat-developer/odo/commit/0b5dc956f0 by @surajnarwade)

[ ]  odo url list shows port, but there are no ports in odo url list -o json

this probably can be fixed in the followup PR

[ ]  odo describe  -o asdf is not failing

but this should be fixed before we merge this

@kadel , fixed as per your suggestion for odo describe and odo url list, can you please review again and merge it

:white_check_mark: odo build 1357 completed and artifacts can be found here (for commit https://github.com/redhat-developer/odo/commit/5bccabecdb by @surajnarwade)

:white_check_mark: odo build 1358 completed and artifacts can be found here (for commit https://github.com/redhat-developer/odo/commit/eaeab280be by @surajnarwade)

if its possible to resolve the code climate issues @surajnarwade

@girishramnani , I think they are resolved now

if its possible to resolve the code climate issues @surajnarwade

@girishramnani Good point

if its possible to resolve the code climate issues @surajnarwade

@girishramnani Good point

I would say, that in this case, it is ok.
almost all the code with code climate issues will have to be rewritten as part of https://github.com/redhat-developer/odo/issues/1263


REST compliant URL for each post

Reason (Why?)
The URLs of einander-helfen.de shall be REST compliant and ther user shall be able to share the link of a post.
Solution (What?)
The URL of the result page should be able to take all needed data as an parameter:

Query
Location
Radius

Also the user should be able to copy the link to a post are page of posts and share it in the internet. For this, the id of the post and page index must be present in the URL.
Not doing
This issue is focused on the REST compliance of the result page.
Relation to other Issues
#54: A resolved bug where the post index was used instead of the post id.
Acceptance criteria
The URLs of the result page are REST compliant.

#67 addresses this issue but there is one limitation we should keep in mind.
We currently need the original search parameters to find the post again.
If the user cuts these parameters from the URI we can´t directly adress the post:
e.g.
/posts/8bba27c43f202713c745f2d1704a1725?q=Jugendarbeit&location=&radius=&page=1
will find the post and open it
/posts/8bba27c43f202713c745f2d1704a1725
won´t find the post.
This is due to the fact that we have no page for one single post but display it in the page with the search results.

I think that should be covered by a separate issue.
The issue will be linked here in the comments once it has been created.


name change

looks like there's already an older (abandoned?) plugin with this name, hinders search results, only way to search this is to search it along with my name. a new name would be a good idea (wouldn't need to break anything, could automatically rename files etc). can't really think of a good one yet though

Honestly just add something to the name that evokes the fact that its newer. That way people cant get confused if they see both when searching AND you can find the new one easier if you know its full name.
See for example Modern LWC, EssentialsX, and McMMO returns.

Hmm... Not a bad idea. "ExplosionRegen9000" lol

I've got it: ExplosionReversal. It captures the fact that this plugin is particularly thorough and pretty much completely reverses the explosions xD

That sounds like a good name


refactor: hide format deserializer state type

tldr: go from (Serde_json.De.state, User.t) Serde.De.t to User.t Serde.De.t
This refactor should make the ergonomics of Serde much better for library authors that need to provide flexibility on the format deserializer type, by making the format deserializer state type go away or be hidden at least from the user facing code.
We've first observed this when working on DbCaml with @emilpriver and noticed that a common error was unifying the types of the state of the format deserializer, which wasn't known in some cases by design, to allow a user to switch up which database format deserializer module they wanted to use (Postgres, SQLite, etc).

Confirmed with the bug repro repo (https://github.com/leostera/serde-de-state-bug-repr/commit/174d2409f93e1186fc5dded34babe8bcde7d6c74) and with @emil_priver that this works for DbCaml too.
Merging 🚀


feat(clerk-js,types): Update OrganizationPreview appearance descriptors for OrganizationSwitcher

Description

Checklist

[ ] npm test runs as expected.
[ ] npm run build runs as expected.
[ ] (If applicable) JSDoc comments have been added or updated for any package exports
[ ] (If applicable) Documentation has been updated

Type of change

[ ] 🐛 Bug fix
[ ] 🌟 New feature
[ ] 🔨 Breaking change
[ ] 📖 Refactoring / dependency upgrade / documentation
[ ] other:

Packages affected

[ ] @clerk/backend
[ ] @clerk/chrome-extension
[X] @clerk/clerk-js
[ ] @clerk/clerk-expo
[ ] @clerk/fastify
[ ] gatsby-plugin-clerk
[ ] @clerk/localizations
[ ] @clerk/nextjs
[ ] @clerk/clerk-react
[ ] @clerk/remix
[ ] @clerk/clerk-sdk-node
[ ] @clerk/shared
[ ] @clerk/themes
[X] @clerk/types
[ ] build/tooling/chore

!preview

@BRKalow Added!


pkg/process: Only keep executable mappings

We only ever care about executable mappings, never about any other kinds of mappings. Executable mappings are a small fraction of all mappings typically so this should save a significant amount of memory.
This change was made based on a customer reported heap profile: https://pprof.me/edde8e7/

Thanks for doing this -- I am pretty sure we had this filtering before but it must have been accidentally deleted somewhere 😢


Could not start multi-instance queue manager with Azure NetApp Files NFS 4.1/OpenShift 4.8/Trident 21.10.1

Hi,
I tried to deploy the multi-instance queue manager using the helm chart and found that the queue manager could not start when use  Azure NetApp Files NFS 4.1/OpenShift 4.8/Trident 21.10.1. However the same helm chart works when I switch to Azure NetApp Files NFS 3.
Here are the steps I took.
(1) Set up Azure NetApp Files NFS 3/OpenShift 4.8/Trident 21.10.1
(2) Configure helm-values.yaml with following mq settings.
     **mq:
    version: <IP_ADDRESS>-r1
    license: "L-APIG-BMKG5H"
    use: "Development"
    metrics:
       enabled: false
    availabilityType: 'MultiInstance'
    storageType: 'persistent-claim'
    storageClassName: 'tpf-azure-netapp-files-gold'
    security:
      context:
        fsGroup:**

(3) Install SPM Helm Chart into OpenShift 4.8
(4) Confirm that both queue managers (mqserver-curam and mqserver-rest) are in running state
(5) Login to the mqserver-curam pod and list out the mount directory /mnt
_drwxrws---. 3 root<PHONE_NUMBER> 4096 Jan 25 01:16 mqm
drwxrwx---. 3 root root       4096 Jan 25 01:16 mqm-data
drwxrwx---. 3 root root       4096 Jan 25 01:16 mqm-log_

(6) Login to the mqserver-curam pod and list out the mount directory /mnt/mqm
_drwxrwsr-x. 13<PHONE_NUMBER><PHONE_NUMBER> 4096 Jan 25 01:16 data_
(7) Login to the mqserver-curam pod and list out the mount directory /mnt/mqm-data
_drwxr-xr-x. 3<PHONE_NUMBER> root 4096 Jan 25 01:16 qmgrs_
(8) Login to the mqserver-curam pod and list out the mount directory /mnt/mqm-log
_drwxr-xr-x. 3<PHONE_NUMBER> root 4096 Jan 25 01:16 log_
(9) Login to the mqserver-curam pod and list out the queue manager bin folder /usr/local/bin
_-rwxr-xr-x. 1 1001 root  2734017 Mar 12  2021 chkmqhealthy
-rwxr-xr-x. 1 1001 root  3531461 Mar 12  2021 chkmqready
-rwxr-xr-x. 1 1001 root  2721302 Mar 12  2021 chkmqstarted
-rwxr--r--. 1 root root     1264 Mar 12  2021 install-extra-packages.sh
-rwxr--r--. 1 1001 root     3039 Mar 12  2021 install-mq-server-prereqs.sh_

(10) The following are the log from the mqserver-curam pod
2022-01-25T01:16:45.860Z CPU architecture: amd64
2022-01-25T01:16:45.860Z Linux kernel version: 4.18.0-305.28.1.el8_4.x86_64
2022-01-25T01:16:45.860Z Container runtime: kube
2022-01-25T01:16:45.860Z Base image: Red Hat Enterprise Linux 8.3 (Ootpa)
2022-01-25T01:16:45.860Z Running as user ID<PHONE_NUMBER> with primary group 0, and supplementary groups<PHONE_NUMBER>
2022-01-25T01:16:45.860Z Capabilities: none
2022-01-25T01:16:45.860Z seccomp enforcing mode: disabled
2022-01-25T01:16:45.860Z Process security attributes: system_u:system_r:container_t:s0:c14,c29�
2022-01-25T01:16:45.861Z Detected 'nfs' volume mounted to /mnt/mqm-log
2022-01-25T01:16:45.861Z Detected 'nfs' volume mounted to /mnt/mqm
2022-01-25T01:16:45.861Z Detected 'nfs' volume mounted to /mnt/mqm-data
2022-01-25T01:16:45.861Z Multi-instance queue manager: enabled
2022-01-25T01:16:46.494Z Using queue manager name: QM1
2022-01-25T01:16:46.888Z Created directory structure under /var/mqm
2022-01-25T01:16:46.888Z Image created: 2021-03-12T09:57:54+00:00
2022-01-25T01:16:46.888Z Image tag: ibm-mqadvanced-server-dev:<IP_ADDRESS>-r1.20210312094918.14987c1-amd64
2022-01-25T01:16:46.930Z MQ version: <IP_ADDRESS>
2022-01-25T01:16:46.930Z MQ level: p922-L210310.DE
2022-01-25T01:16:46.930Z MQ license: Developer
2022-01-25T01:16:49.108Z Creating queue manager QM1
2022-01-25T01:16:57.048Z Created queue manager
2022-01-25T01:16:57.099Z Starting queue manager
2022-01-25T01:16:53.152Z AMQ6287I: IBM MQ V<IP_ADDRESS> (p922-L210310.DE). [CommentInsert1(Linux 4.18.0-305.28.1.el8_4.x86_64 (MQ Linux (x86-64 platform) 64-bit)), CommentInsert2(/opt/mqm (Installation1)), CommentInsert3(<IP_ADDRESS> (p922-L210310.DE))]
2022-01-25T01:16:53.112Z AMQ5051I: The queue manager task 'LOGGER-IO' has started. [ArithInsert2(1), CommentInsert1(LOGGER-IO)]
2022-01-25T01:16:53.234Z AMQ5051I: The queue manager task 'CHECKPOINT' has started. [ArithInsert2(1), CommentInsert1(CHECKPOINT)]
2022-01-25T01:16:53.270Z AMQ5041I: The queue manager task 'CHECKPOINT' has ended. [CommentInsert1(CHECKPOINT)]
2022-01-25T01:16:53.293Z AMQ5041I: The queue manager task 'LOGGER-IO' has ended. [CommentInsert1(LOGGER-IO)]
2022-01-25T01:16:53.946Z AMQ5051I: The queue manager task 'LOGGER-IO' has started. [ArithInsert2(1), CommentInsert1(LOGGER-IO)]

(11) The other queue manager (mqserver-rest pod) has the same behavior and both queue manager pods are in running state
/////////////////////////////////////////So far everything works properly////////////////////////////////////////////////////////////////
Now, I changed the NFS version from v3 to V4.1 in Azure NetApp Files and kept all the other settings the same, the queue manager pods could no longer run successfully. The log from the queue manager pod is:
2022-01-23T02:23:57.869Z CPU architecture: amd64
2022-01-23T02:23:57.869Z Linux kernel version: 4.18.0-305.28.1.el8_4.x86_64
2022-01-23T02:23:57.869Z Container runtime: kube
2022-01-23T02:23:57.869Z Base image: Red Hat Enterprise Linux 8.3 (Ootpa)
2022-01-23T02:23:57.869Z Running as user ID<PHONE_NUMBER> with primary group 0, and supplementary groups<PHONE_NUMBER>
2022-01-23T02:23:57.869Z Capabilities: none
2022-01-23T02:23:57.869Z seccomp enforcing mode: disabled
2022-01-23T02:23:57.869Z Process security attributes: system_u:system_r:container_t:s0:c14,c29�
2022-01-23T02:23:57.869Z Detected 'nfs4' volume mounted to /mnt/mqm
2022-01-23T02:23:57.869Z Detected 'nfs4' volume mounted to /mnt/mqm-data
2022-01-23T02:23:57.869Z Detected 'nfs4' volume mounted to /mnt/mqm-log
2022-01-23T02:23:57.869Z Multi-instance queue manager: enabled
2022-01-23T02:23:58.032Z Using queue manager name: QM1
2022-01-23T02:23:58.225Z Created directory structure under /var/mqm
2022-01-23T02:23:58.225Z Image created: 2021-03-12T09:57:54+00:00
2022-01-23T02:23:58.225Z Image tag: ibm-mqadvanced-server-dev:<IP_ADDRESS>-r1.20210312094918.14987c1-amd64
2022-01-23T02:23:58.272Z MQ version: <IP_ADDRESS>
2022-01-23T02:23:58.272Z MQ level: p922-L210310.DE
2022-01-23T02:23:58.272Z MQ license: Developer
2022-01-23T02:23:58.747Z Creating queue manager QM1
2022-01-23T02:23:58.793Z Detected existing queue manager QM1
2022-01-23T02:23:58.838Z Starting queue manager
2022-01-23T02:23:58.914Z Error 119 starting queue manager: AMQ7077E: You are not authorized to perform the requested operation.

2022-01-23T02:23:58.914Z /opt/mqm/bin/strmqm: exit status 119
2022-01-23T02:23:59.339Z stat /mnt/mqm-data/qmgrs/QM1/errors/AMQERR01.json: no such file or directory
2022-01-23T02:23:59.339Z stat /var/mqm/errors/mqhtpass.json: no such file or directory

Then I looked into the folder permission inside the queue manager pod and found the followings.
(a) /mnt
_drwxrws---. 3 99<PHONE_NUMBER> 4096 Jan 25 03:27 mqm
drwxrwx---. 3 99         99 4096 Jan 25 03:27 mqm-data
drwxrwx---. 3 99         99 4096 Jan 25 03:27 mqm-log_

(b) /mnt/data
_drwxrwsr-x. 13<PHONE_NUMBER><PHONE_NUMBER> 4096 Jan 25 03:27 data_
(c) /mnt/mqm-data
_drwxr-xr-x. 3<PHONE_NUMBER> 99 4096 Jan 25 03:27 qmgrs_
(d) /mnt/mqm-log
_drwxr-xr-x. 2<PHONE_NUMBER> 99 4096 Jan 25 03:27 log_

It appears that the user and group for the mount folders are changed. They are root in NFS3 and 99 in NFS4.1.
What would be the cause of this kind of permission error? Is there a way to debug the queue manager to see what went wrong? Are there additional parameters we need to set inside helm-values.yaml so the queue manager can run with NFS 4.1?
Thanks

@tomyangdxp
without having access to the helm-charts and the override file you are using im not sure what the issue could be.
What it could be is you are missing the supplementalGroups, or the value is blank from your override file.
please see:
(Storage considerations for IBM MQ Operator)[https://www.ibm.com/docs/en/ibm-mq/9.2?topic=containers-storage-considerations-mq-operator]
and the documented known issues (Storage initialized)[https://ibm.github.io/spm-kubernetes/known_issues/#storage-initialized-using-a-runbook-version-older-than-20.9.0]

Yes, you are right. It looks like the only way to make it work with NFS 4.1 is to specify the group id inside supplementalGroups. The issue is consider resolved. Please feel free to close it. Thanks.

Closing as  per previous comment.


Documentation

[x] Worker/Queue implentation boilerplate
[x] Comparison to celery (no funny business -- automatically importing your code, importing code by string, autodiscovery)
[x] mention not to use transaction.on_commit
[ ] systemd unit file for worker
[x] how to log to a file
[x] Monitoring queries/pgrowlocks
[x] sentry handler
[ ] How to run tasks in tests

[ ] Scheduler


Add ping message verification to the websocket connection.

This should further increase the durability of our websocket connections. See https://github.com/knative/serving/issues/3138
Ping messages are sent to actively probe the websocket connection. By default, the websocket servers we use send a pong message back. This can be used to break indefinite read calls.

/assign @vagababov
This was your idea, so let's continue our review journey 🙂.

This was a legitimate race found on Shutdown. Added a WaitGroup to ensure everything's shut down after Shutdown finishes.

/approve


[Question] How to change log format?

Information

Type
Version

Question
4.x

Description
I am trying to change the log format so it will be easier to parse. I can't find a way to do it. any tips? 😄
Thanks!

For now, I did the following:
@ServerSettings(conf)
export class Server extends ServerLoader {

	public $onInit() {
		$log.appenders.set('stdout', {
			type: 'stdout',
			layout: {
				type: 'pattern',
				pattern: '%[%d{yyyy/MM/dd-hh.mm.ss} %p%] %m%n'
			}
		});
	}
}

Shouldn't there be a way to do it in the server configurations?

Hi @TomerAmirV,
Currently the logger format cannot be configured from the configuration.
But you can manage the configuration and the routes logged in the console with a service, like this:
import {Env} from "@tsed/core";
import {RouteService, OverrideService, ServerSettingsService} from "@tsed/common";
import {$log} from "ts-log-debug";

@OverrideService(RouteService);
export class CustomRouteService extends RouteService {
  constructor(private controllerService: ControllerService, private serverSettingsService: ServerSettingsService) {
      super(controllerService)
  }

  public $onInit() {
        const logFormat = this.serverSettingsService.get('logFormat');
	$log.appenders.set('stdout', {
		type: 'stdout',
		levels: [ 'info', 'debug' ],
		layout: {
			type: 'pattern',
			pattern: logFormat
		}
	});

	$log.appenders.set('stderr', {
		levels: [ 'trace', 'fatal', 'error', 'warn' ],
		type: 'stderr',
		layout: {
			type: 'pattern',
			pattern: logFormat
		}
	});
   }

   public $afterRoutesInit() {
        const env = this.serverSettingsService.env;
        if (env !== Env.PROD) {
           $log.info("Routes mounted :");
           this.printRoutes($log);
        }
   }
}

See you ;)
Romain

Thanks!

Hi!
any news?

Hi @TomerAmirV ,
I completely forgot this point. Sorry :/
I fixed that ASAP.
Romain

No problem...
it's the world cup right now... and France is doing well 😉

Release :) v4.26.0


Add streaming documentation into msg handling page

update have been made to make the API cleaner and easy to use. Luos-io/Luos#14


Solver acceleration

Currently, per #21 the IP solver is about 5x slower than PATH on random sparse QPs. It would be great to get it at least on par.

I got a solid speedup from switching to a Krylov linear solver; see #24
I didn't do any larger-scale benchmark yet though since the benchmark/ stuff compiles forever on my machine.

SymbolicTracingUtils.jl now supports generating lazy Jacobians that behave like a matrix (one can multiply them with A * x or mul!(result, A, x) ) but that don't allocate. These are supported by the LinearSolve.jl and thus we should be able to make the hot loop entirely matrix-free (while keeping the readable look-and-feel of matrices).
See: https://github.com/JuliaGameTheoreticPlanning/SymbolicTracingUtils.jl/pull/2

Niiiiice let’s do it!

Btw if you want to try the benchmark stuff you can change the number of primals and constraints as keyword args in the benchmark() function. I’ll definitely also run it when I’m back at a computer next week.

Can you elaborate how you concluded from the benchmarks that PATH is 5x faster? When I run the benchmarks as
data = SolverBenchmarks.benchmark()
SolverBenchmarks.summary_statistics(data)

I get:
(ip = (success_rate = 0.995, μ = 0.08008527012261307, σ = 0.01090576568028957), path = (success_rate = 0.0, μ = NaN, σ = NaN))

Which suggests that PATH is never able to solve the problem. Am I calling the wrong function? Maybe you could add a little README to the benchmark/ directory to clarify how to use it.

That is the right way to run the benchmarks... I am surprised that PATH is never solving correctly though. The two solvers have always matched which problems they solved in past trials I've run. I think we need to dig into these failures.

Btw did you possibly try it on a machine where you haven't set the PATH license string? Not sure what else would have changed since it's just calling the solver through ParametricMCPs and I haven't seen breaking changes there.


Image description: Lasiurus semotus 2.jpg

Which image is this for?
Lasiurus semotus 2.jpg
What is your suggested image description?
A close-up photo of a Hawaiian hoary bat being held in somebody's hand.

Hi @necco-atsume, I think this suggestion could be improved by describing what a Hawaiian hoary bat looks like, rather than expecting a reader to know that information already. Would you be willing to amend your suggestion to include this information?
Thanks for contributing!

Closing due to inactivity.


Fix history

currently:

listing from earliest to latest.
ideally:
list from latest commands to earliest.

I think this has been implemented already.


Unable to extract video stream

Overview
I'm trying to access frames from video.
container = av.open(path_to_vid)
However, container.streams.video[0] is NoneType.
Expected behavior
container.streams.video[0]  is supposed to extract video stream, from which frames could be extracted.
Actual behavior
Unable to extract any frames.
Investigation
I'm able to extract frames directly from the video by using ffmpeg. The video is playable with a media player. So, nothing is wrong with the video.
Versions

OS: Ubuntu 18.04
PyAV runtime:

{{ Complete output of `python -m av --version`. If this command won't run, you are likely dealing with the build issue and should use the appropriate template. }}

PyAV v6.1.0
git origin: git@github.com:mikeboers/PyAV
git commit: unknown-commit
library configuration: --prefix=/mnt/pai/home/mayoore/anaconda3/envs/powerai16 --cc=/root/worker/containers/00002bgehr8/tmp/build/80754af9/ffmpeg_1576694050084/_build_env/bin/powerpc64le-conda_cos7-linux-gnu-cc --disable-doc --enable-gpl --enable-avresample --disable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-openssl --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --enable-nonfree
library license: nonfree and unredistributable
libavcodec     58. 54.100
libavdevice    58.  8.100
libavfilter     7. 57.100
libavformat    58. 29.100
libavutil      56. 31.100
libswresample   3.  5.100
libswscale      5.  5.100

PyAV build:

{{ Complete output of `python setup.py config --verbose`. }}

FFmpeg:

{{ Complete output of `ffmpeg -version` }}
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7.3.0 (crosstool-NG <IP_ADDRESS>9-a04d0)
configuration: --prefix=/mnt/pai/home/mayoore/anaconda3/envs/powerai16 --cc=/root/worker/containers/00002bgehr8/tmp/build/80754af9/ffmpeg_1576694050084/_build_env/bin/powerpc64le-conda_cos7-linux-gnu-cc --disable-doc --enable-gpl --enable-avresample --disable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-openssl --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --enable-nonfree
libavutil      56. 31.100 / 56. 31.100
libavcodec     58. 54.100 / 58. 54.100
libavformat    58. 29.100 / 58. 29.100
libavdevice    58.  8.100 / 58.  8.100
libavfilter     7. 57.100 /  7. 57.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  5.100 /  5.  5.100
libswresample   3.  5.100 /  3.  5.100
libpostproc    55.  5.100 / 55.  5.100```

## Research

I have done the following:

- [ x] Checked the [PyAV documentation](https://pyav.org/docs)
- [x ] Searched on [Google](https://www.google.com/search?q=pyav+how+do+I+foo)
- [x ] Searched on [Stack Overflow](https://stackoverflow.com/search?q=pyav)
- [ ] Looked through [old GitHub issues](https://github.com/PyAV-Org/PyAV/issues?&q=is%3Aissue)
- [ ] Asked on [PyAV Gitter](https://gitter.im/PyAV-Org)
- [ ] ... and waited 72 hours for a response.

## Additional context

{{ Add any other context about the problem here. }}

The problem seems to be specific to whatever file you're manipulating, the test suite would have picked up an error as broad as "PyAV is unable to read video streams". Can you please provide the specific file you're encountering an error with making sure you own the copyright to this file?


readable-stream

npm: https://www.npmjs.com/package/readable-stream
The description states that "This package is a mirror of the streams implementations in Node.js 18.19.0."

https://nodejs.org/dist/v12.19.0/docs/api/stream.html
Don't know whether it's safe to replace it.


fix(TimelineRow): add flex-grow

FX-3281
Description
Added flex-grow: 1 to the row content so it can fit the entire container width
How to test

Code looks good, you can't see the possibility of something being broken
Use the code snippet to test in deployed storybook and on current Picasso master. Deployed storybook should have the button all the way to the left, content taking up the entire width of the container.

Code example
import React from "react";
import {
Grid,
Timeline,
Typography,
Link,
Container,
Button
} from "@toptal/picasso";

const Example = () => (
<Timeline>
  <Timeline.Row date="May 21, 11:17 PM">
    <Grid justifyContent="space-between">
      <Grid.Item small={9}>
        <Container>
          <Typography as="div">
            <Link>Lavinia Maluf</Link> commented task{" "}
            <Typography weight="semibold">
              "Mark meetings' outcome (through the Unresolved Meetings page)"
            </Typography>{" "}
            on <Link>Obfuscated subject for meeting 124555</Link>
          </Typography>
        </Container>
      </Grid.Item>
      <Button>Action</Button>
    </Grid>
  </Timeline.Row>

  <Timeline.Row date="May 23, 9:19 AM" hasConnector={false}>
    <Typography as="div">
      <Link>Tomas Urban</Link> commented task{" "}
      <Typography weight="semibold">
        "Mark meetings' outcome (through the Unresolved Meetings page)"
      </Typography>{" "}
      on <Link>Obfuscated subject for meeting 124555</Link>
    </Typography>
  </Timeline.Row>
</Timeline>
);

export default Example;

Screenshots

Before.
After.

Not possible to push the button all the way to the right

Development checks

[x] Add changeset according to guidelines (if needed)
[x] Read CONTRIBUTING.md and Component API principles
[x] Make sure that additions and changes on design follow Toptal's BASE design, and it's been already discussed with designers at #-base-core
[x] Annotate all props in component with documentation
[x] Create examples for component
[x] Ensure that deployed demo has expected results and good examples
[x] Ensure the changed/created components have not caused accessibility issues. How to use accessibility plugin in storybook.
[x] Self reviewed
[x] Covered with tests

Breaking change

[n/a] codemod is created and showcased in the changeset
[n/a] test alpha package of Picasso in StaffPortal

All development checks should be done and set checked to pass the
GitHub Bot: TODOLess action

PR commands

List of available commands:

@toptal-bot run package:alpha-release - Release alpha version
@toptal-anvil ping reviewers - Ping FX team for review

PR Review Guidelines

When to approve? ✅
You are OK with merging this PR and

You have no extra requests.
You have optional requests.

Add nit: to your comment. (ex. nit: I'd rename this variable from makeCircle to getCircle)

When to request changes? ❌
You are not OK with merging this PR because

Something is broken after the changes.
Acceptance criteria is not reached.
Code is dirty.

When to comment (neither ✅ nor ❌)
You want your comments to be addressed before merging this PR in cases like:

There are leftovers like unnecessary logs, comments, etc.
You have an opinionated comment regarding the code that requires a discussion.
You have questions.

How to handle the comments?

An owner of a comment is the only one who can resolve it.
An owner of a comment must resolve it when it's addressed.
A PR owner must reply with ✅ when a comment is addressed.

@toptal-anvil ping reviewers


GitHub Actions - Connection References not connecting after solution import

Hello, I'm fairly new to GitHub Actions so I'm not sure if this is an actual problem, or if it's even possible.
We are currently experimenting with GitHub Actions and have a workflow set up which is to export a solution from DEV into TEST, which I copied from these example workflows: https://github.com/microsoft/powerplatform-actions-lab/tree/main/sample-workflows
In this solution we have:

Canvas App
Simple flow that sends an email notification to myself
Connection reference for the "Notification" in that flow

The solution export and import is working fine, but when we run the flow we get this error message:

Are we missing something in the workflow that will re-connect the connection references after import so that flows successfully run?
I know that when you export/import a solution from PowerApps itself, it will prompt you about connections, but because we're bypassing this stage is this the problem?  We've also attempted this with an Azure DevOps pipeline and got the same result.
Thank you.

Aaron, the underlying pac CLI the GH import-solution action uses doesn't yet support connection references. But that work is in progress and should appear as preview in here in 3-4 weeks (same work in progress dependency for the AzDevOps task you mentioned).
There


Removed CI test branch

:tada: This issue has been resolved in version 0.21.0-development.7 :tada:
The release is available on:

npm package (@development dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 0.22.0-development.1 :tada:
The release is available on:

npm package (@development dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 0.22.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix Gatling test used for https://www.baeldung.com/gatling-jmeter-grinder-comparison

The Gatling results in https://www.baeldung.com/gatling-jmeter-grinder-comparison are completely broken:

dynamic parameter injection is incorrect, always the same value is injected, causing the application to actually crash because of the duplicates
one of the response parameter capture (check) is incorrect and should be optional
the test includes 1 second pauses between consecutive requests. The tests with the other tools don't have them. The result is that the Gatling throughput is artificially limited.

On my local machine, I get 4100 requests executed in ~3 seconds (instead of the 31s in your article), which give an average throughput of 1025 requests/second.
Generally speaking the performance figures in the article are very suspicious:
  | Successful Requests | Errors | Total Test Time (s) | Average Response Time (ms) | Peak Throughput
-- | -- | -- | -- | -- | --
Gatling | 4100 Requests | 100 (soft)* | 31 | 64.5 | 132 req/s
JMeter | 4135 Requests | 0 | 35 | 81 | 1080 req/s
The Grinder | 5000 Requests | 0 | 5.57 | 65.72 | 1284 req/s

How come JMeter gets 4135 requests in 35s and get a throughput of 1080 req/s?!?!
How do you measure "Peak Throughput"?
Those tests are actually super short. With only 5,000 iterations, the JIT compiler hasn't kicked in yet. You should run longer tests to get meaningful results.

@eugenp PTAL

@lor6 Thanks for merging. What's your plan regarding the broken published results? Do you plan on updating the existing post, or publish an update? Any ETA?

@slandelle  I've updated the results here, using 100 instead of 10 users: https://www.baeldung.com/gatling-jmeter-grinder-comparison
One issue is that Gatling is running fewer requests than the other tools, which is because of the if here: https://github.com/eugenp/tutorials/blob/master/testing-modules/load-testing-comparison/src/main/resources/scripts/Gatling/GatlingScenario.scala#L33 It seems that after the first run this always evaluates to false, even if the request above doesn't return a result: https://github.com/eugenp/tutorials/blob/master/testing-modules/load-testing-comparison/src/main/resources/scripts/Gatling/GatlingScenario.scala#L31 Is there a way to set the variable back to undefined?
Let me know if you see any other issues.
Thanks.

@lor6 Sorry, I don't understand.

It seems that after the first run this always evaluates to false, even if the request above doesn't return a result: https://github.com/eugenp/tutorials/blob/master/testing-modules/load-testing-comparison/src/main/resources/scripts/Gatling/GatlingScenario.scala#L31

The test tries to create a new customer based on a random id (between 1 and 10,000 like the JMeter test), and then adds a new reward if there's not already one.
Every test run will generate 1,000 customers.
Now, the application under test uses an in-memory h2 database. This means that if you run tests without restarting the application in between, you'll increase the probability of hitting a customerId that already exists and hence having an already existing award.
In short. You should restart the application before running a test with any of the 3 tests tools so you start with the same state and don't compare apples and oranges.
Did I get it right?

@slandelle Yes, that's correct, but I've restarted the app before running the test with each tool so each starts with a clean db. The issue with the Gatling script is that after the first run, the rwdId variable never evaluates to undefined again even if the previous request that saves the id to the rwdId returns null.

@lor6 OK, sorry, now I get it. Please see https://github.com/eugenp/tutorials/pull/10146
Modern versions would make it easier but the post target old versions.
Could you please merge and let me know?

Thanks @slandelle It seems ok now.
Let me know if you see any other issues in the post: https://www.baeldung.com/gatling-jmeter-grinder-comparison

Thanks @lor6. However, yes, I see other issues:
Number of requests:
The 3 tests don't generate the same number of requests while the loop is configured with a fixed number of iterations. This means the 3 tests don't do the same thing.
This is most likely because the random generators are not configured the same way:

Grinder: Random#nextInt()
Gatling: Random#nextInt(1, 10000)
JMeter: <stringProp name="maximumValue">10000</stringProp>

As a result: the Grinder is very unlikely to suffer from custId collisions. That's why you get exactly 50000 requests with the Grinder and less with the other ones.
You should fix this and have the same behavior for the 3 tests.
Typo in 3.3
The 10 runs => The 100 runs
Test duration
The tests last ~1 minutes. This is not a realist usage for load testing where tests lasts way longer. All 3 tools run on the JVM but might benefit differently from the JIT that barely has time to kick in. I would recommend running tests with more iterations, eg 1,000 instead of 100.
Tools versions
Performance comparison results only hold for the versions of the 3 tools that were tested.
The article doesn't mention them explicitly, nor their release dates.
If I'm to trust:

the pom.xml for the Gatling version: the Gatling version used is 2.2.5, released on 2017-04-20
The JMeter test plan for the JMeter version: 5.1-SNAPSHOT.20181219, so a custom made version from 2018-12-19 (JMeter 5.0 was released on 2018-09-18 while JMeter 5.1 was released on 2019-02-19)

I can't find the Grinder version.
So if those versions are correct, there are 20 months between the JMeter and Gatling versions.
You can imagine the improvements that happened during this period of time. Gatling 2 has reached end of life in October, 2018.
As you're revisiting the article, I would recommend:

updating all the tool versions to the latest ones (I can provide a PR for upgrading Gatling)
mentioning explicitly the tested versions and their respective release dates

@slandelle I'm running the tests again with the latest versions, but if there's any improvement you suggest for the Gatling script, please add a PR.
I'll up the numbers and add the versions used in the text.

@lor6 PR incoming.
Regarding the nextInt, I would really recommend removing the bound in all the tools. There seems to be a race condition (spring-data, hibernate, h2?) in the application that can cause it to crash on duplicated entries:
javax.persistence.NonUniqueResultException: query did not return a unique result: 2
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:128) ~[hibernate-core-5.4.20.Final.jar:5.4.20.Final]
	at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1588) ~[hibernate-core-5.4.20.Final.jar:5.4.20.Final]
	at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getSingleResult(CriteriaQueryTypeQueryAdapter.java:111) ~[hibernate-core-5.4.20.Final.jar:5.4.20.Final]
	at org.springframework.data.jpa.repository.query.JpaQueryExecution$SingleEntityExecution.doExecute(JpaQueryExecution.java:196) ~[spring-data-jpa-2.3.3.RELEASE.jar:2.3.3.RELEASE]
	at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:88) ~[spring-data-jpa-2.3.3.RELEASE.jar:2.3.3.RELEASE]
	at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:154) ~[spring-data-jpa-2.3.3.RELEASE.jar:2.3.3.RELEASE]
	at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:142) ~[spring-data-jpa-2.3.3.RELEASE.jar:2.3.3.RELEASE]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor$QueryMethodInvoker.invoke(QueryExecutorMethodInterceptor.java:195) ~[spring-data-commons-2.3.3.RELEASE.jar:2.3.3.RELEASE]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152) ~[spring-data-commons-2.3.3.RELEASE.jar:2.3.3.RELEASE]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:130) ~[spring-data-commons-2.3.3.RELEASE.jar:2.3.3.RELEASE]

@lor6 PTAL #10157

Thanks @slandelle  The post is updated.

Thanks @lor6!
However, IMHO, you should really align the random behavior on the JMeter and Gatling tests (remove the 10,000 upper bound) to match the behavior of the Grinder test and not compare apples and oranges.
Also, the numbers in 4.2 don't add up, eg:

Gatling: 46348 / 59 = 786, not 772
JMeter: 46369 / 67 = 692, not 694
Grinder: 50000 : 74 = 676, not 684

Hi @slandelle Yep, the upper bound is now removed in all the tests.
I think you're seeing a cached version of the text, as I've now upped the number of runs to 1000 and there are close to 500k requests running. Please check again in a few hours.

@lor6 Awesome, thanks a lot!
FYI, if someday you want to spend time and have more realistic results, here are a few tips:

The load test tool and the applications should not sit on the same machine. Otherwise, first both compete for CPU, secondly loopback (localhost) is very different from actual network and doesn't produce the same results.
Use a very fast test application so you're sure this is not the bottleneck. The application we use internally can withstand 200,000 requests per second.
Run on Linux, not Windows. Java NIO's implementation on Windows is not great. Gatling works best on Linux.

@lor6 I was very surprised with the latest results and tried to reproduce on my side.
In turns out that, whatever the tool, the throughput is quickly dropping. The throughput is initially higher with Gatling, but then the drop is worse, things might be related (Gatling harming the application more).
In short: the current implementation of the test application is completely not suited for a load testing benchmark.
I suspect the culprit is h2 (we've had bad experiences with in in the past too): fine for functional tests, very bad for a demo application under load.
IMHO, if you want to fix this, you should use something like a ConcurrentHashMap, not a h2 database.

@slandelle Yes, I noticed the same behavior. The throughput really starts to drop after 400k requests. For reference, I did run these on a Linux machine, though the initial commands in the text were Windows-specific. I'll update those as well, as Linux is the more common choice anyway.
I could try with MySQL instead, since that could be used for a production db. I'll test with a different db, and with a ConcurrentHashMap, and let you know.
Thanks.

Hi @lor6
I got it sorted.
2 issues:

the JSON payloads defined in the Gatling script don't exactly match what the app expects causing an overhead, please see #10167
the h2 database doesn't have any index, so the findBy queries get slower and slower as the tables grow. I'm not familiar with SpringBoot and JPA to add those indexes myself. I went with in-memory maps and got the expected stable (and high) throughput, please see #10168

Regards,


Run ct_icp on Darpa tunnel circuit datasets

Hi, I have manage to make the slam work on Darpa urban datasets, now I am trying to test it in the tunnel ciruits.
As you links say, them al compressed, so I decompress all of them with rosbag decompress file.bag .
I have correctly remap the pointcloud topic, lidar model and frequency are the same and at the moment I have not change anything inside the param yaml file.
When I launch the simulation the simulation starts, but the odometry topic /ct_icp/pose/odom is not published.
As you can see from the following images, also the tf from odom to base_link is not generated.
This image are the TF from the urban datasets:

This one are the TF from the tunnel:

This one is a screenshot from RViz after some seconds: as you can see the position frame are not published even if the robot is already inside the tunnel (the robot stars outside the tunnel)

This one are the topic from rqt:

What I am doing wrong? Can someone please help me?

Hi,
I will look into it,
In the mean time, do you have some error message in the terminal ?

Hi, Thank you!
No, in the terminal I have some warning at the launch. But I have read in some post that they are normal.
I use this bag:
https://bitbucket.org/subtchallenge/subt_reference_datasets/src/master/
https://subt-data.s3.amazonaws.com/SubT_Tunnel_Ckt/sr_B_route2.bag (16.3 GB)


Add "Whistleblowing" to the software categories list

Whistleblowing software is a category of software that currently does not fit in any of the currently defined software categories list and that is defined by many existing and upcoming regulations (e.g. 190/2012, 179/2017, EU Regulation 2016/679) that determine specific software capabilities.
This ticket is to propose the addition of a specific software category "Whistleblowing".
Here are examples of two softwares that would fit in this category:

https://developers.italia.it/it/software/globaleaks-globaleaks-f22648.html
https://developers.italia.it/it/software/c_l736-comune_venezia-whistleblowing

Thanks a lot @evilaliv3 also for the PR!
This looks OK to me, whisteblowing should be a standalone category.
WDYT @ruphy @sebbalex @bfabio?

reposted this on the forum: https://publiccode.discourse.group/t/add-whistleblowing-to-the-software-categories-list-repost-issue-96/57

@evilaliv3 if you can register on the forum, I can change the ownership of the issue too :-)

I just received the notification by github that this issus has been opened again! this is c00l.
Is there anything that i could do to support thi this work to move forward @ruphy ?
Thank you!

Hey @evilaliv3, we're setting up a new governance and an international working group that should be able to vote on proposals pretty soon!

Rebased! thank you @bfabio !


pull request for e2e test at 01/05/2017 10:45:09

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Allow consecutive_failures setting to apply to warnings, too

Currently, consecutive_failures only applies to high alerts. This change makes them apply to warnings as well.
I made some significant refactors to metricsapp/api/base.py. The diff may not be extremely helpful; you may want to read the new code directly.

trigger build


Convert doc images to text

Fixes #203
This is direct conversion to text without any intended changes to the documentation. I have not checked if the documentation is up-to-date

🥳