Implement eBPF native redirect functionality

Feature Summary
Introduce an eBPF native redirect functionality that enables pod traffic, destined to IP/Port tuple or Kubernetes service, to be redirected locally to a backend pod within a node. Users will create a "local redirect policy" (aka LRP) to express the configurations needed for traffic redirection.
Implementation overview

API: Introduce a new CRD in order for users to express an LRP, and register the CRD using Kubernetes APIs.
Data path: Add a new service type that only has node-local backends, and enable the functionality.
Control plane: Add new constructs to keep track of updates to such LRPs, event handlers to get updates about resources like cilium endpoint, kubernetes service, and plumb data path.

Tasks

[x] API changes to define the new CRD structs and register with K8s
[x] Data path changes with unit tests
[x] Control plane changes with unit tests
[x] Add documentation

To limit the scope of the PR, following TODOs will be handled in a follow-up PR -
LRP updates and status
K8s service sync
Cluster wide LRP processing
End-to-end CI tests
CLI to dump entries (#13156)
Monitor health status of backend pods and update service entries (#13168)
Fixes:#11646
Signed-off-by: Aditi Ghag<EMAIL_ADDRESS>

Coverage decreased (-0.07%) to 37.062% when pulling e4498813760e6ee0f900b15abd69c41b13ce09a0 on aditighag:native-redirect into 00bbb9cfec1ef56df797706761e3cf39cc584973 on cilium:master.

I've marked the PR as draft since the title has WIP in it. Feel free to re-open it and add the label dont-merge-preview-only if you only want people to review it

test-me-please

test-me-please

All tests have passed -- marking the PR as ready for review to get an early feedback.

test-me-please

Thanks @aanm and @joestringer  for reviews. Addressed review comments in the latest commits.

test-me-please

Re-adding docs codeowners for review since docs are not yet done.

BPF Check fail is a false positive on commit message line longer than 75 characters, can be safely ignored as discussed in the Cilium community meeting.

test-me-please

Latest commits include :

Fix for the "any" source key from policy to pod labels matching logic
Reliable endpoint metadata retrieval logic
Lint fixes reported by CI

Error reported by cilium/coccicheck:1.0 doesn't seem to be related to my PR.

test-me-please

Error reported by cilium/coccicheck:1.0 doesn't seem to be related to my PR.

Fixed via https://github.com/cilium/cilium/pull/12962.

test-me-please

test-me-please

Hi @aditighag , any updates on the upgrade impact issue?

Hi @aditighag , any updates on the upgrade impact issue?

Hi @borkmann,
I'll address that in the follow-up patch.

To assist reviewers, I had added notes in one of the comments, but that comment got buried in subsequent updates.
Re-adding the note for the topmost commit :
Event handlers for LRPs and other resources like cilium endpoint, kubernetes services, and plumb datapath (core logic in pkg/redirectpolicy/manager.go. GH UI hides it by default, you may need to load the diff. ;) )

Hi @aditighag , any updates on the upgrade impact issue?

Hi @borkmann,
I'll address that in the follow-up patch.

I'd prefer if this could be addressed in this PR (upgrade impact but also documentation) given potential flag mismatch is a known issue. So either adding another flags field or better widening it and keeping bits in same position plus adding a small bpf test for it as assertion for our CI. (Mentioned doc as well here given this can help with review, thx.)

Latest commits:

Addressed review comments
Added a datapath test for testing local redirect policy override
Added a gsg
make render-docs runs successfully

test-me-please

Do we have a list of real world examples which we want to support with LRP? I'm trying to find a justification for ServiceMatcher. Currently, I'm aware only of KIAM and node-local DNS. I guess both fall into a category "ClusterIP service with a custom by user specified VIP". Maybe adding a custom service annotation would have been enough? Or did we choose the CRD way because some users might want to configure RBACs for redirection?

Do we have a list of real world examples which we want to support with LRP? I'm trying to find a justification for ServiceMatcher. Currently, I'm aware only of KIAM and node-local DNS. I guess both fall into a category "ClusterIP service with a custom by user specified VIP". Maybe adding a custom service annotation would have been enough? Or did we choose the CRD way because some users might want to configure RBACs for redirection? Also, what is a user case for cluster-wide LRP?

That's a valid point. The reasons we decided to go with CRDs -

CRDs are gated by Kubernetes RBACs so cloud admins can decide who can enable traffic redirections within their clusters.
We didn't want applications owners to be able to circumvent traffic redirections. With service/pod annotations, anyone can remove the annotations, and thus escape redirection.
For the KIAM use case, a user needs to create an LRP with address matcher (and not service matcher). The reason being, there is no service for metadata traffic. The metadata traffic IP address (<IP_ADDRESS>) for which traffic redirection needs to be enforced for, is outside of the service-cluster-ip-range [1].

[1] https://kubernetes.io/docs/concepts/services-networking/service/

@aditighag Thanks, that helps with the review. Do you know any real world use case for ServiceMatcher? From the short glimpse to https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/, it seems that node-local DNS is accessed by non-ClusterIP.

@aditighag Thanks, that helps with the review. Do you know any real world use case for ServiceMatcher? From the short glimpse to https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/, it seems that node-local DNS is accessed by non-ClusterIP.

node-local dns pod listens on kube-dns clusterIP, that's how the pod dns requests are redirected to the dns proxy.

test-me-please

I saw CI failures related to guest book and istio test cases in the last run. The delta between last CI run and the previous CI run that passed successfully should be minimal, and AFAIK, shouldn't break any CI runs. For now, I've re-triggered a new run.

Thanks very much for the detailed review, Joe. Appreciated!
I've fixed most of the comments, would like to get feedback from others around certain areas before pushing new commits.
Other fixes will be done in the follow-up PR as you acknowledged.

test-me-please

test-me-please

test-me-please

test-me-please

test-net-next

Fix an issue in the test flagged by the CI run.

test-me-please

Hitting net-next flake for the test - K8sServicesTest_Checks_service_across_nodes_Tests_NodePort_BPF_Tests_with_direct_routing_Tests_NodePort_with_externalTrafficPolicy_Local

test-me-please

test-me-please

test-focus K8sPolicyTest Multi-node policy test validates ingress CIDR-dependent L4 connectivity is restored after importing ingress policy

Both the test failures are flakes. I re-ran one of them, and it passed the 2nd time.
2nd failure is a known flake - https://github.com/cilium/cilium/issues/12756
I checked with André, and I'll address his comments in a follow-up PR.

I noticed some of the previous comments were marked "resolved" without a clear response or pointer that it's either addressed or being tracked somewhere. I'd usually interpret "thumbs up" + "mark resolved" as the comment has been addressed in the code

I've been giving "thumbs up" to most of the comments that I addressed but didn't intend to reply. In some cases, it's possible that I missed replying to the duplicate comments. Other times, they were lost in the big pile of updates. :(
Sorry :( The scope of the PR continued to expand as we reiterated design/spec and I tried to address the review comments.
@joestringer @aanm @christarazi Thanks for patiently reviewing the PR. Appreciated!

Hm, based on starting at the agent code again, I'm not fully sure what are the semantics & implications when there are already existing services present. I saw you've added the special case here:
		// Local Redirect Policies with service matcher would have same frontend
		// as the service clusterIP type. In such cases, if a Local redirect service
		// exists, we shouldn't override it with clusterIP type (e.g., k8s event/sync, etc).
		if svc.svcType == lb.SVCTypeLocalRedirect && p.Type == lb.SVCTypeClusterIP {
			err := fmt.Errorf("Local redirect service exists for "+
				"frontend %v, skip update for svc %s", p.Frontend, p.Name)
			return svc, !found, prevSessionAffinity, prevLoadBalancerSourceRanges, err

		}

Also based on discussion from yesterday, we should for https://docs.cilium.io/en/latest/gettingstarted/local-redirect-policy/:

Clearly describe the use case for when LPR is needed and what limitations it addresses compared to existing services, e.g. a comparison discussion would be good to have the GSG there.
Clearly describe the semantics / behavior in the GSG on how it interacts if there are existing services in the system.

My questions are:

What happens if the LPR entry is clashing with e.g. NodePort service (or in general any other type aside from ClusterIP)? Given a NodePort service has multiple front-end svc entries in the svc map, do we clean these up, or do we continue with a half-working state? E.g. that is, direct matches go to the LPR, but wild-card matches go to the NodePort service?
Why do we special case ClusterIP above? Do we restrict the CiliumLocalRedirectPolicy in a way where it can only ever overlap with ClusterIP range?

My other open questions are:

What happens if the LPR entry is clashing with e.g. NodePort service (or in general any other type aside from ClusterIP)? Given a NodePort service has multiple front-end svc entries in the svc map, do we clean these up, or do we continue with a half-working state? E.g. that is, direct matches go to the LPR, but wild-card matches go to the NodePort service?
Why do we special case ClusterIP above? Do we restrict the CiliumLocalRedirectPolicy in a way where it can only ever overlap with ClusterIP range?

serviceMatcher LRPs only apply to clusterIP services for now. This is documented in the CRD spec that users typically reference - https://github.com/cilium/cilium/blob/master/pkg/k8s/apis/cilium.io/v2/clrp_types.go#L122
I'll add it to the gsg as well, among other things including motivations, run-time behavior, etc
Also, this is enforced in the code - https://github.com/cilium/cilium/blob/master/pkg/k8s/watchers/service.go#L85

I'm not fully sure what are the semantics & implications when there are already existing services present.

I have been operating on the assumption that LRP always takes precedence. I would assume that adding and then deleting LRP that overlaps with another service type would end up putting the maps back into the initial state.

What happens if the LPR entry is clashing with e.g. NodePort service (or in general any other type aside from ClusterIP)? Given a NodePort service has multiple front-end svc entries in the svc map, do we clean these up, or do we continue with a half-working state? E.g. that is, direct matches go to the LPR, but wild-card matches go to the NodePort service?

This sounds like a great example for e2e test cases. Create and destroy variations of overlapping NodePorts, ClusterIPs, etc with LRPs on the same tuple. Check that adding/removing LRPs, services, pods that overlap, the logic consistently applies.

Why do we special case ClusterIP above? Do we restrict the CiliumLocalRedirectPolicy in a way where it can only ever overlap with ClusterIP range?

serviceMatcher LRPs only apply to clusterIP services for now. This is documented in the CRD spec that users typically reference - https://github.com/cilium/cilium/blob/master/pkg/k8s/apis/cilium.io/v2/clrp_types.go#L122
I'll add it to the gsg as well, among other things including motivations, run-time behavior, etc

The what is documented above, but I don't see the why. Definitely makes sense to document but that doesn't preclude also discussing here. Why does it matter if it's clusterIP? Is this because LRP is only intended to apply to traffic within a node?

I'm not fully sure what are the semantics & implications when there are already existing services present.

I have been operating on the assumption that LRP always takes precedence. I would assume that adding and then deleting LRP that overlaps with another service type would end up putting the maps back into the initial state.

Yes, as the note [1] in service.go indicates, as long as a local-redirect service exists for a frontend, cilium agent won't override it with clusterIP service type (e.g., during k8s service sync) that has overlapping frontend.
I want to clarify something with regard to Joe's comment -

another service type would end up putting the maps back into the initial state.

When the local-redirect service entry is deleted (either LRP is deleted or no backends exist for the frontend), we'll restore the clusterIP service entry with all the available (remote or local) backends at the time of the restore.  It's possible that the service endpoints list might have changed since the time it was deleted from the map.
[1] https://github.com/cilium/cilium/blob/master/pkg/service/service.go#L543

My other open questions are:

What happens if the LPR entry is clashing with e.g. NodePort service (or in general any other type aside from ClusterIP)? Given a NodePort service has multiple front-end svc entries in the svc map, do we clean these up, or do we continue with a half-working state? E.g. that is, direct matches go to the LPR, but wild-card matches go to the NodePort service?
Why do we special case ClusterIP above? Do we restrict the CiliumLocalRedirectPolicy in a way where it can only ever overlap with ClusterIP range?

serviceMatcher LRPs only apply to clusterIP services for now. This is documented in the CRD spec that users typically reference - https://github.com/cilium/cilium/blob/master/pkg/k8s/apis/cilium.io/v2/clrp_types.go#L122
I'll add it to the gsg as well, among other things including motivations, run-time behavior, etc
Also, this is enforced in the code - https://github.com/cilium/cilium/blob/master/pkg/k8s/watchers/service.go#L85

Okay, but what happens if the admin has a below yaml where it sets the ip and port to a NodePort one? Would the address range / port be enforced somewhere to avoid overlap?
apiVersion: "cilium.io/v2"
kind: CiliumLocalRedirectPolicy
metadata:
  name: "lrp-addr"
spec:
  redirectFrontend:
    addressMatcher:
      ip: "<IP_ADDRESS>"
      toPorts:
        - port: "8080"
          protocol: TCP
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        app: proxy
    toPorts:
      - port: "80"
        protocol: TCP

My other open questions are:

What happens if the LPR entry is clashing with e.g. NodePort service (or in general any other type aside from ClusterIP)? Given a NodePort service has multiple front-end svc entries in the svc map, do we clean these up, or do we continue with a half-working state? E.g. that is, direct matches go to the LPR, but wild-card matches go to the NodePort service?
Why do we special case ClusterIP above? Do we restrict the CiliumLocalRedirectPolicy in a way where it can only ever overlap with ClusterIP range?

serviceMatcher LRPs only apply to clusterIP services for now. This is documented in the CRD spec that users typically reference - https://github.com/cilium/cilium/blob/master/pkg/k8s/apis/cilium.io/v2/clrp_types.go#L122
I'll add it to the gsg as well, among other things including motivations, run-time behavior, etc
Also, this is enforced in the code - https://github.com/cilium/cilium/blob/master/pkg/k8s/watchers/service.go#L85

Okay, but what happens if the admin has a below yaml where it sets the ip and port to a NodePort one? Would the address range / port be enforced somewhere to avoid overlap / clashing?
apiVersion: "cilium.io/v2"
kind: CiliumLocalRedirectPolicy
metadata:
  name: "lrp-addr"
spec:
  redirectFrontend:
    addressMatcher:
      ip: "<IP_ADDRESS>"
      toPorts:
        - port: "8080"
          protocol: TCP
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        app: proxy
    toPorts:
      - port: "80"
        protocol: TCP

That's an addressMatcher LRP and not a serviceMatcher LRP. The expectation is that user would specify the latter when they want traffic destined to a service to be redirected. We do have validations in place to check for duplicates [1], but they do validations only for addressMatcher frontends and service names.
If needed, we can potentially validate that the addressMatcher frontend.ip doesn't overlap with the service-cluster-ip-range [2]?
[1] https://github.com/cilium/cilium/blob/745a26b2ead09e689f8626413467d4083a7fa00a/pkg/redirectpolicy/manager.go#L508
[2] https://kubernetes.io/docs/concepts/services-networking/service/

Why do we special case ClusterIP above? Do we restrict the CiliumLocalRedirectPolicy in a way where it can only ever overlap with ClusterIP range?

serviceMatcher LRPs only apply to clusterIP services for now. This is documented in the CRD spec that users typically reference - https://github.com/cilium/cilium/blob/master/pkg/k8s/apis/cilium.io/v2/clrp_types.go#L122
I'll add it to the gsg as well, among other things including motivations, run-time behavior, etc

The what is documented above, but I don't see the why. Definitely makes sense to document but that doesn't preclude also discussing here. Why does it matter if it's clusterIP? Is this because LRP is only intended to apply to traffic within a node?

LocalRedirectPolicy lets user specify configurations that enable traffic redirection within a node. As a result, these apply only for E-W traffic. In my mind, this is different from selecting only node-local backends for nodePort services in order to preserve the client IP address in case of N-S traffic.
In any case, since this behavior is not exposed to end users via any CRD spec (except the description note), we could always relax the restrictions in our implementation, if needed?
Thoughts?

If the goal (and target use cases) are only for allowing local traffic to be redirected, then I would feel more comfortable if we have reassurances that traffic from outside the node is not subject to the same service translation. Seems like a smaller potential attack surface.


How can we create Task bot / app in slack . Kindly help !!

The Bot shall open a modal view by using /task command.
Which opens a prompt asking what the task is (Large text box), when it is due (Date box), and the responsible party (Gives you a drop down of users in the channel, DM, or group chat) , Click Submit•
Bot responds “New Task has been created for “Responsible party” due on “Date” see all tasks by typing /task list.
Visualizing your tasks-
If any user types in /task list, it will show them the tasks that have been assigned to that specific channel, group chat, or DM
Ie: I am in a group chat with Tom and Jerry and I type /task list

Hey @SagarKNL,
We can't write the app for you, but we can point you to relevant documentation to get you on the right path. Please take a look at:

Listening and responding to commands. This is the entrypoint for your app, you would start here with your /task command.
The above command would probably need to open a modal. Here you can populate a modal with form fields and a submit button. You should wire up the submit button so that it is considered as a view_submission.
You then need to listen on view submissions to write code to process the submitted data. This view submission handler probably wants to respond to the user. You can use the built-in say method to easily do that.

Hope that's helpful. Good luck!


Implement dpkg Source

See original issue on GitLab
In GitLab by [Gitlab user @brejoc] on Apr 13, 2017, 20:40
I'm opening this ticket for discussion and to better understand the goals. What I got from the IRC conversation with [Gitlab user @tristanvb]:

We want a way to somehow mirror dpkg packages of a system.
Dumping a package list into the dpkg source should produce such a cloned system.

In GitLab by [Gitlab user @brejoc] on Apr 13, 2017, 20:50
[Gitlab user @tristanvb] I've looked around and it should be possible to use dpkg --get-selections > the_list to get a pkg list and dpkg --set-selections < the_list && apt-get -y update && apt-get dselect-upgrade to install all the required packages. This should work very well, but doesn't include package versions.
I have one additional question: Am I actually able to simply invoke dpkg commands when DpkgSource ist executed?

In GitLab by [Gitlab user @tristanvb] on Apr 14, 2017, 13:10
Ok so this is a good start! But we're going to need specific versions :)
So, I continue to think this is a breakthrough idea, and have dumped a lot of information here:

Some tools in BuildStream which will will come in handy in implementation
Some deeper explanation of BuildStreams expectations and functionality
Some additional thoughts, which include some "gotchas" which came to mind while writing up the rest of this

So here comes a large markdown formatted brain dump which I hope will be useful for moving forward :)
Utilities that BuildStream provides
The mirror directory
So a Source plugin type is "given" a directory in which to mirror sources, this directory can be obtained with Source.get_mirror_directory(). Calling that as self.get_mirror_directory() will return a directory that belongs to the 'dpkg' Source, and will also result in it's creation if it does not yet exist.
URI specific subdirectories
A specific source will usually cache/mirror it's data in a specific directory, a convenience function utils.url_directory_name() can be used to create a directory name from a url.
So a dpkg Source will mirror packages from a specific URL (either official debian, or some downstream or unrelated debian repo), in a subdirectory named with this utility, inside the dpkg mirror directory.
Project url aliases
For convenience, and also to allow one to mirror all of a projects dependencies in a single location for the sake of guaranteeing repeatability, all Sources must support project url aliases.
When reading a url which may be named foo:bar/baz, a Source uses the Project.translate_url() method to resolve a full uri from the alias read at Plugin.configure() time.
Remember that a Source is subclass of Plugin, so anything on the Plugin is expected of a Source, or can be used by a Source.
Calling Host Tools
Source plugins are the only thing allowed to call out to host tools, we strictly only depend on host tooling for injecting external data into the pipeline.
Obtaining tools at preflight time
Host tooling should be obtained at Plugin.preflight() time, using the utils.get_host_tool() utility function, like so:
  ...
  def preflight(self):

      self.dkpg = utils.get_host_tool('dpkg')

This ensures that an error is raised at preflight time right away, before hours of processing commence. The utility will raise an error so that BuildStream will print an intelligible error in the frontend.
It's worthy to note here that a Project itself brings in additional host dependencies over the base BuildStream dependencies (i.e. you dont need a host git, unless you are using a project which uses git sources).
Running tools at fetch/track/stage time
When you need to actually run the tool, use the tool you previously obtained at preflight time with the Plugin.call() and Plugin.check_output() APIs, which are similar to the corresponding APIs in the subprocess module in the standard libs.
The linked docs contain examples for calling out to host tools.
It's important to use these APIs to call out to host tools because:

They will automatically ensure process suspend/resume works
They will automatically ensure the process is terminated with the parent
They will also ensure that stdout/stderr ends up in the corresponding BuildStream log files

BuildStream expectations
So here are the expectations that BuildStream has, as to how a source should behave.
Isolated configuration
This is not a problem for sources like git or tar, as any host/user configuration pertaining to how these host tools are installed will not have any effect on how the sources are mirrored, or how they are staged into the sandbox area before entering the sandbox.
For Source types which represent distro packaging mechanisms however, this may not be the case. RPM for instance installs a lot of distortion into /etc/rpm (macros and rc files) which may or may not effect how the package is installed.
It is of vital importance here that any and all host configuration for using dpkg which might effect the result of an install into a sysroot location be overridden in such a way that the project has complete control and that there is no host contamination which may cause the staging of a package into a sandbox to have a different result on one host or another.
Network Connectivity
A Source should only require network connectivity in it's Source.track() and Source.fetch() implementations.
This is to say that, you need the network:

To run bst fetch
To run bst track
To run bst build only if the Source refs have not yet been mirrored locally

You dont need the network to run bst build if you have already fetched on the same project/pipeline.
Definitions of url, ref and track properties in a Source configuration
So these could have totally arbitrary names, but by convention we want to keep the same names across the board.
url
This is normally an aliased url (as described above) and for a dpkg source should indicate the debian repository you want to interact with.
ref
This represents an exact Source reference, which will never change, it should be bit-for-bit identical when downloaded on any host.
For dpkg, this should be a fully revisioned package or group of packages.
track
This represents a symbolic name, usually the default is to interact with a VCS so this would be a branch or tag name.
For dpkg, a valid track name, when dealing with upstream debian repository urls, should probably be something like jessie or stretch or sid.
Note that:

A Project can be fetched and built if no track was ever specified for one or more Sources
A project cannot be built if there is no ref specified for one or more Sources
A source will have a ref if it was successfully tracked
A source which lacks a track configuration, cannot be tracked, but may be built if there is already a ref

Essentially, an end user needs to specify either track or ref, otherwise they are not able to obtain sources in any way.
Fetch Activity
At fetch time, it is assumed that a Source already has a valid ref.
If a Source does not yet have a ref, then BuildStream will error out before fetch is ever called, indicating to the user that they should first call bst track to get a ref for that Source.
A plugin at this time must ensure that local copies are downloaded for the specific ref which is configured in their Source.
Track Activity
At track time, it is the Sources responsibility to check with the remote repository and find the latest version available for the configured track value.
A Source may fetch and download the remote payload at track time if this is convenient but not necessarily. For a git source for instance, it is more convenient to simply run git fetch origin on the git mirror first and then later find out what is the latest commit sha for a tracking branch. For a dpkg source, I suspect that one can obtain the latest versions for jessie without actually downloading and mirroring the said packages, deferring that to a later invocation of bst fetch.
Stage Activity
At stage time, the Source will be handed a directory in which the payload it represents should be dumped.
At this time, Sources are guaranteed to have passed the fetch stage, so the payload is present in the mirror directory and no network activity is required.
For a distro package staging activity, this means to install the extracted package into the staging directory, treating the staging directory as a sysroot (which it actually is).
Additional Thoughts
Single vs multipackage Source
While writing this, it struck me that perhaps a more obvious Source implementation for distro package based sources would be to have one Source represent exactly one package.
This has some pros and cons, though
Pros:

The Source implementation becomes much simplified when only mirroring and tracking a single package and it's version
The project which uses an import Element to bring in a bunch of sources, has a finer grained control on what exact packages it imports, this means it's easier to distinguish when we want to import:
** A runtime environment and some specific set of libraries
** An SDK, which would include a runtime plus some compilers and tooling which we may not want to include in a docker deployment
** A BSP, including kernel and drivers for a given arch, which we only need when deploying a bootable image for a VM or bare metal

Cons:

We lose the ability to introspect dependencies dictated by the package manager in question, this becomes an exercise left to the user, or perhaps some surrounding tooling, or something we havent invented yet.

Package databases
To complete the experience of being able to stage packages, build things, and create new packages for deployment inside a BuildStream pipeline (and perhaps even as a requirement just to get off the ground), it seems required that something take care of initializing a package database for the given tool set.
For instance, when staging RPMs into a sysroot, one first does: rpm --dbpath=${sysroot}/var/lib/rpm --initdb and then calls rpm -ivh --root ${sysroot} rpms-to-install to install them into that root.
Debian packages surely also have some database which needs to be created and maintained, which is a side effect of installing packages from a dpkg source and not part of the payload itself.
How should we address this ?
One idea might be to include an option on a dpkg source configuration which says to initialize and create some base layout directories and database in the staging area to be carried forward in the pipeline (cached as an artifact).
With this approach, perhaps this 'base' source would have to be staged first and would not require any ref, url or track parameters to function (as what it essentially does is just provide a base for further dkpg sources to work with, but not any payload).

In GitLab by [Gitlab user @brejoc] on Apr 15, 2017, 21:49
[Gitlab user @tristanvb] Thanks for the detailed explanations! It's now much clearer to me how BuildStream works!
dpkg indeed allows to install packages to a staging directory: dpkg -i --root=${sysroot} package.deb
I'll see if I can test this this weekend. But I wouldn't be to optimistic, since pre and post install scripts might be a problem here. I couldn't find a source that explains that in detail, so I'll do some test runs.
An other option might be a chroot!?
Regarding single vs multipackage Source. I afraid that a single source approach might get tedious very fast. Managing more than 100 pkg installations could then be a burden instead of a flexibility. And pkg dependencies might drive you over the edge, since you'd have to check this manually or you'd constantly run into build errors.
One additional quest. Where is the sysroot coming from? I assume sysroot would at that point of time be some kind of overlay filesystem with a minimal Debian installation form a previous Source?

In GitLab by [Gitlab user @tristanvb] on Apr 16, 2017, 13:29
[Gitlab user @brejoc] hi... I'll try to reply briefly, on my way to the airport :)
So three things,
pre and post scriptlets
These are indeed a bit annoying, most likely they will remain mostly unneeded for build purposes, needed if one wants to deploy a booting system, and somewhere in between in the case of deploying sdks or docker container images.
The ideal here would be to translate those into integration commands in the bst public data domain, which means "these commands should be run against this dependency in the sandbox"
Currently the puclic domain data cannot be dynamically populated, but i have a couple of other use cases for this so i could make that happen.
But this seems a bit much for an initial implementation so i hope in the first iteration we can ignore install scriptlets.
single vs multiple
Agreed, however this still raises some questions, i think the solution here is that a dpkg source needs a list of unversioned package names the user can provide in the configuration in order to determine what is desired (surely the user does not want the entire content of the distro).
sysroot
Where does the sysroot come from ?
This can be something a bit different depending on the Element, basically the core provides a Sandbox to the elements at Element.assemble() time, and the element is then free to stage dependencies and sources anywhere inside the temporary sandboxes root directory.
For the typical import element case i expect to use dpkg sources with; sources are staged directly at the root of the sandbox and then an artifact is collected directly from that root.
I think more to the point: the sysroot will be empty for any useful purposes, hence my previous comments about package databases and such side effects. It seems to me that a side effect of creating some base directory structure and/or initializing an empty package database at the beginning of the stage operation may be needed.
chroots
Just another word here, we dont want to ever require being root. So we dont want chroots, but can do some things with bubblewrap as that is a hard dependency of the Sandbox (commands in the sandbox are actually run as uid 0 in a uid namespace).
While the dpkg source could use bubblewrap for some things i think its much preferrable that we dont.
in closing
Hope this provides some pointers, while im flying around you might look at the buildstream-tests repo and see how we're using the said integration commands, and what element public data is for.
If we allow the elements to dynamically populate the public data at assemble time and make a statement that one may only read public data from their dependencies at assemble time, then solving scriplets may just be a matter of adding a dpkg specific import element implementation to populate that data, after which the regular integration rules apply.

In GitLab by [Gitlab user @brejoc] on Apr 24, 2017, 09:18
[Gitlab user @tristanvb] It seems to be harder than expected to mirror package installations in bulk with exact version numbers. I'm still evaluating options… :\

In GitLab by [Gitlab user @tristanvb] on Apr 24, 2017, 11:03
[Gitlab user @brejoc] hi!
I just got back to seoul today, nice to hear you are still investigating this :)
From my limited understanding debian packaging scripts are quite exposed (layer upon layer of seldom used apis/scripts) so i wouldnt rule this out just yet... I'll try to do some research on this tomorrow too, maybe i can squeeze the info out of the right irc channel...
The feature of staging something deterministic is however quite important, i hope we can do it this way as the alternatives i think will invariably include extra moving parts (i.e maybe extra mirroring solution either with custom interface or mirroring deterministic snapshots into ostree repos)... Not ideal.

In GitLab by [Gitlab user @tristanvb] on Apr 25, 2017, 12:22
[Gitlab user @brejoc] So I found some interesting info just asking people in #debian irc.
First there is a tool called apt-offline which is used for generating a signature of a specific debian version and generating an installer for that version for another machine that does not have network connectivity.
Then there is also apt-zip and apt-walkabout which are closely related.
Note that I don't expect these tools by themselves to satisfy the requirements out of the box, however I do suspect that examining how these tools work will give us a good idea of how to implement the dpkg source, at least for the purposes of bst track and bst fetch. For the stage() part I think we already have most of what we need with dpkg --install --root=<staging dir> and will later look into pulling the install scriptlets and turning those into BuildStream public data as discussed previously.

In GitLab by [Gitlab user @brejoc] on Apr 26, 2017, 21:59
[Gitlab user @tristanvb] Those tools are indeed very interesting! Thanks for the links.
While digging around in python-apt I had an idea. Why don't we simply mirror/download all the packages currently installed on the reference installation. Then we create a local repo (dpkg-scanpackages), which is then the only file resource used in /etc/sources.list.
That way we also wouldn't have to worry about 3rd party repos.
I haven't tested that yet, but I think that could work. What do you think?

In GitLab by [Gitlab user @brejoc] on Apr 27, 2017, 19:10
I've been investigating a bit further. I could only test with Ubuntu, since I simply haven't got old enough Debian installations. I assume Debian repositories are mostly organized the same. It seems like downloading the packages from the currently set-up repos only works with a mostly up-to-date package list. As soon as there are old enough packages installed, downloading from the repos fails. The old version simply isn't there anymore. But we might be able to pull packages from snapshot.debian.org. But I'm still not sure, if the overall approach is correct. :thinking:

In GitLab by [Gitlab user @tristanvb] on Apr 28, 2017, 08:39
So you seem to be correct that older package versions are not reliably fetchable via official debian distro repositories. And also correct that snapshot.debian.org is useful for exactly this purpose, I think the tradeoff here is that the official user facing repositories have a lot of mirrors run by contributors and not everyone wants to mirror the entire history of debian.
It seems also quite plausible for an organization to run a snapshot repository of their own in the case they want to be sure that snapshots will exist for them (I dont think debian snapshots are going away, but there is not much of a guarantee they will always be there).
So, I tried asking around in #debian on OFTC a bit more, one promising approach is the possibility of running something like debootstrap inside an unprivileged namespace (bubblewrap) for this, seems there is already a thread about this.
Something like debootstrap would work, but we would want to:

Be able to run this against a snapshot.debian.org with specific package versions in the debootstrap configuration
Be able to run it as an unprivileged user
Still be able to update the configuration to the "latest package versions of release foo" (for tracking)

Sorry if I've sent you on a bit of a wild goose chase, this seems to be very hard ! but the benefits of this would still be enormous... maybe we will have to settle for some middle ground in the short term, though.
Also, I hope you will be on IRC again one day if you have time to chat :)

In GitLab by [Gitlab user @tristanvb] on May 3, 2017, 13:34
[Gitlab user @brejoc] Ok so... I'm going to leave this bug open because I still think it would be awesome to have a Source implementation that does something like this for dpkg and then another one for rpm... however...
I have found that running debootstrap as super user, is amazingly simple and gets us really most of the way.
So I just wanted to give you a heads up and I'm not sure how else to reach you... I'm planning to experiment with this this week, and perhaps later setup a service (which seems ridiculously easy) which can do some leg work for us.
What I am thinking is to have a service which:

Has some configuration for what kinds of images you want to produce, and for what target architectures
This configuration would only include some toplevel unversioned package names
Periodically use this configuration to:
Run debootstrap to create a chrootable runtime based on these packages
Commit the result to an ostree repository under a target branch name
Commits can have the date at which the import occurred in their commit messages

This way buildstream can work with the intermediate ostree repos in the regular way, with fully deterministic checkouts (i.e. helping with repeatability and reproducibility of builds) and still be able to track the latest import of a given debian import.
Of course, this approach means there is a moving part, which kind of sucks... but it looks like implementing the dkpg Source for this is going to be a lot of work and in the short term it will be great to experiment with this approach.

In GitLab by [Gitlab user @brejoc] on May 3, 2017, 19:49
[Gitlab user @tristanvb] Sorry, for the silence! Last weekend I've been coming back to Germany from half a year of traveling through France, Spain and Portugal with a campervan. Tough switch with some challenges. Hopefully everything will be back to "normal" next week, when we also (hopefully) moved from the campervan into an apartment.
debootstrap is an amazing tool. I also came across it during my researches. Good to see, that this can be leveraged despite the downside. Seems like I have to wrap my brain also around ostree in the near future. I've not worked with it yet.
Although a dpkg source might not be the most easy task, I've got give up yet. I'll hopefully be able to dedicate a bit more time in the next weeks and also show up in IRC again.

In GitLab by [Gitlab user @knownexus] on Feb 25, 2018, 22:31
We have been discussing the concept of allowing the usage of pre-built dpkg sources to be used as sources in bst projects.
We have, so far, come up with 2 options:

Create a source plugin
This would import packges and extract the content, allowing them to be built by buildstream and used as a normal source

Modifying the produced artifact to make buildstream treat it like a bst generated artifact
This would allow the use of pre-built artifacts without re-building them,
However this seems like it would be far more limited in its usability as the artifacts would need to be generated in a known way.
This would also make it more difficult to update the dependencies, as you would need to replace the artifact whenever you wanted to update a related source

For now, we've decided on option (1), as it is more generalised and more maintainable.
Here's what i've come up with for the creation of the DPKG Source Plugin:
Have the binary fetched from an online repository - stated in the element.bst file
Using _downloadablefilesource.py
Generate a cache key for the binary
Extract the file like a tar into a set dir - either relative or stated in the element.bst file
Remove tmp dir and contents
Limit the contents to the base dir and it's contents - stated in the element.bst file

In GitLab by [Gitlab user @juergbi] on Mar 1, 2018, 08:52

[...] allowing them to be built by buildstream

By this I assume you mean creating a BuildStream artifact from the contents of a .deb, not actually building anything from sources. Do I understand this correctly?
I definitely agree on option (1) being the more maintainable choice.

Limit the contents to the base dir and it's contents - stated in the element.bst file

Do you have a use case in mind for this?
Will any metadata from the .deb be used? E.g., runtime dependencies or integration commands? If not, what's the plan to handle dependencies? Require manually maintaining the dependency list in the .bst file? Generate .bst files with external tool?

In GitLab by [Gitlab user @knownexus] on Mar 1, 2018, 11:00
[Gitlab user @juergbi]

yes, you are correct, that's what i meant

Basically meaning that there may be layers of directories inside the .deb that does not contain
information that BuildStream needs to build, and will be bypassed to prevent unneeded parsing

-i'll get back to you on that

In GitLab by [Gitlab user @knownexus] on Mar 2, 2018, 12:23
mentioned in merge request !305

In GitLab by [Gitlab user @richardmaw-codethink] on Mar 2, 2018, 13:03
mentioned in commit 0ee2bce0ae4bdf3fea81af3fccdab13264003d2b

In GitLab by [Gitlab user @tristanvb] on Mar 8, 2018, 12:13
Just noting, it would be feasible to:

Unpack and serialize the package metadata at some constant location

or make that feature optional

Have a later element which stages the metadata, use it to dynamically set public data during it's build phase
Later reuse that data as integration commands, or even to rewrap the same original debian package based on the existing binaries

Not sure this is desired functionality right now, but it should be quite possible.

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 1, 2018, 11:43
assigned to [Gitlab user @knownexus]

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jun 1, 2018, 11:45
[Gitlab user @knownexus] can we close this ticket, due to https://gitlab.com/BuildStream/buildstream/merge_requests/305?

In GitLab by [Gitlab user @knownexus] on Jun 1, 2018, 12:54
[Gitlab user @LaurenceUrhegyi] I believe so

In GitLab by [Gitlab user @knownexus] on Jun 1, 2018, 12:54
closed


Do we have a test isolation problem?

It seems like some database state is being preserved from test to test. #926 is an example of this.
Here's a way that I think we could investigate.

Un-comment this line, and change :membership to :user. https://github.com/bullet-train-co/bullet_train/blob/main/test/factories/memberships.rb#L23 (This makes the factory invalid, but it exposes the fact that we have order-dependent interaction between tests.)
Run this command to test with a clean test database:$ rails db:test:prepare; rails test test/controllers/api/v1/users_controller_test.rb test/controllers/api/open_api_controller_test.rb --seed 46583

You should see the error mentioned at the top of #926.
Comment out this line: https://github.com/bullet-train-co/bullet_train/blob/main/test/controllers/api/v1/users_controller_test.rb#L8C5-L8C35
Re-run the tests and they should pass.$ rails db:test:prepare; rails test test/controllers/api/v1/users_controller_test.rb test/controllers/api/open_api_controller_test.rb --seed 46583

Figure out which database state is carrying over between tests, and why. (We should expect that test to always pass or always fail, regardless of which tests are run before it. Theoretically we should start every test from the same state every time.)

I haven't been able to reproduce this problem again. Going to close this issue.


How should wordk relation Many To Many?

I'm trying to use this guide Many To Many
`package main
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
)
type Rl struct {
gorm.Model
Name string
}
type Accnt struct {
gorm.Model
Email string
Roles []Rl gorm:"many2many:accnt_rls"
}
func main() {
db, _ := gorm.Open("postgres", "user=postgres password=pass dbname=postgres sslmode=disable")
db.CreateTable(&Rl{}, &Accnt{})

admRole := &Rl{Name: "Admin"}
db.Create(admRole)

acc := &Accnt{Email<EMAIL_ADDRESS>db.Create(acc)
acc.Roles = []Rl{*admRole}
db.Save(acc)

account := &Accnt{}
db.First(account)
fmt.Println("first account is ", account.ID, " ", account.Email) //first account is<EMAIL_ADDRESS>
role := &Rl{}
db.First(role)
fmt.Println("first role is ", role.ID, " ", role.Name) //first role is  1 Admin

relatedRoles := []Rl{}
db.Model(account).Related(&relatedRoles)

fmt.Println(len(relatedRoles))  //0 but i expected 1

}`

What am i doing wrong?

i was able to get related data with Association(), but why do not work Related()?

It's not possible to deduce the field so you have to specify it explicitly.
db.Model(account).Related(&relatedRoles, "Roles")


The authentication input field is not prefilled for the Bearer header token

What happens?
I can't get the Bearer authentication token to prefill. The security section of the openapi file (https://api.theirstack.com/openapi.json) looks like this
    "securitySchemes": {
      "APIKeyQuery": {
        "type": "apiKey",
        "in": "query",
        "name": "token"
      },
      "APIKeyCookie": {
        "type": "apiKey",
        "in": "cookie",
        "name": "theirstack_token"
      },
      "Bearer": {
        "type": "http",
        "scheme": "bearer"
      },
      "OAuth2PasswordBearer": {
        "type": "oauth2",
        "flows": {
          "password": {
            "scopes": {

            },
            "tokenUrl": "/token"
          }
        }
      }
    }

And the config I'm passing looks like this:
        var configuration = {
                "title": "TheirStack API Reference",
                "isEditable": false,
                "spec": {
                        "url": "/openapi.json"
                },
                "proxyUrl": "https://api.scalar.com/request-proxy",
                "darkMode": false,
                "layout": "modern",
                "theme": "default",
                "showSidebar": true,
                "searchHotKey": "k",
                "metaData": {},
                "customCss": "",
                "hiddenClients": [],
                "authentication": {
                        "preferredSecurityScheme": "Bearer",
                        "Bearer": {
                                "token": "MYTOKEN"
                        }
                }
        }

What did you expect to happen?
I'd expect the Bearer Token input field to be prefilled with the value passed in the authentication section

How can we reproduce the issue?
Create an HTML with this content and open it in your browser
<!doctype html>
    <html>
    <head>
        <title>TheirStack API Reference</title>
        <meta charset="utf-8" />
        <meta
        name="viewport"
        content="width=device-width, initial-scale=1" />
    </head>
    <body>
        <script
            id="api-reference">
        </script>

        <script>
        var configuration = {"title": "TheirStack API Reference", "isEditable": false, "spec": {"url": "/openapi.json"}, "proxyUrl": "https://api.scalar.com/request-proxy", "darkMode": false, "layout": "modern", "theme": "default", "showSidebar": true, "searchHotKey": "k", "metaData": {}, "customCss": "", "hiddenClients": [], "authentication": {"preferredSecurityScheme": "Bearer", "Bearer": {"token": "MYTOKEN"}}}

        var apiReference = document.getElementById('api-reference')
        apiReference.dataset.configuration = JSON.stringify(configuration)
        </script>
        <script src="https://cdn.jsdelivr.net/npm/@scalar/api-reference"></script>
    </body>
    </html>

Swagger/OpenAPI Example
https://api.theirstack.com/openapi.json

@xoelop try this pattern:
				authentication: {
					preferredSecurityScheme: "bearerAuth",
					http: {
						bearer: {
							token: 'MYTOKEN',
						},
					},
				},

Hey @xoelop! To echo what @rivulent said the configuration so set a default Bearer token should be:
const configuration = {
  // ...rest of your configuration
  authentication: {
    preferredSecurityScheme: 'Bearer',
    http: {
      bearer: {
        token: 'MYTOKEN',
      },
    },
  },
}

Which will set the Bearer as the default authentication method and set the bearer token to MYTOKEN.

Same, if I didn't add the basic part, cURL commands wouldn't prefill with the request body properly.

fwiw, this worked for me, but I did need to add an http.basic object as well, to stop something from trying to 'read properties of undefined'. so
  authentication: {
    preferredSecurityScheme: "Bearer",
      http: {
        bearer: {
          token: "token",
        },
        basic: { username: "user", password: "password" },
      },
  }

Related: https://github.com/scalar/scalar/issues/1513#


Enable jar extraction to continue even when single file extraction fails

Extraction of other files from jar is not halted when a single file errors out, and a warning is displayed for the user.

@cesarsotovalero Comments have been addressed in PR.


Allow usage of GitHub API Token for results submission

The operator should allow the user to provide a secret containing a GitHub API Token for results submission as documented in the Operator Certification CI Pipeline Instructions.
https://github.com/redhat-openshift-ecosystem/certification-releases/blob/main/4.9/ga/ci-pipeline.md#add-a-github-api-token-for-the-repo-where-the-pr-will-be-created

/assign

Key to be used in CR should be github-api-token when creating the secrete, since this is the value the pipeline uses.


Problem with check 2.6

Running check 2.6, with TLS enabled, can give a warning that TLS is not enabled. It comes about when the TCP socket is defined in a Systemd override, but the TLS flags are defined in the daemon's config file. I think the check assumes everything is defined in either the unit file (or override), or the config file. Often, however, host definitions appear in the Systemd config, whilst the TLS config is placed in the daemon config file. It's perfectly legitimate to split configuration this way, so I guess the check should cater for it?
I haven't tested it thoroughly, but by re-using the majority of the function for check 2.6, the following seems to work:
# 2.6
check_2_6() {
  check_2_6="2.6  - Ensure TLS authentication for Docker daemon is configured"
  totalChecks=$((totalChecks + 1))

  if [ grep -i 'tcp://' "$CONFIG_FILE" 2>/dev/null 1>&2 ] || \
    [ $(get_docker_cumulative_command_line_args '-H' | grep -vE '(unix|fd)://') >/dev/null 2>&1 ]; then
    if [ $(get_docker_configuration_file_args '"tlsverify":' | grep 'true') ] || \
        [ $(get_docker_cumulative_command_line_args '--tlsverify' | grep 'tlsverify') >/dev/null 2>&1 ]; then
      pass "$check_2_6"
      logjson "2.6" "PASS"
      currentScore=$((currentScore + 1))
    elif [ $(get_docker_configuration_file_args '"tls":' | grep 'true') ] || \
      [ $(get_docker_cumulative_command_line_args '--tls' | grep 'tls') >/dev/null 2>&1 ]; then
      warn "$check_2_6"
      warn "     * Docker daemon currently listening on TCP with TLS, but no verification"
      logjson "2.6" "WARN"
      currentScore=$((currentScore - 1))
    else
      warn "$check_2_6"
      warn "     * Docker daemon currently listening on TCP without TLS"
      logjson "2.6" "WARN"
      currentScore=$((currentScore - 1))
    fi
  else
    info "$check_2_6"
    info "     * Docker daemon not listening on TCP"
    logjson "2.6" "INFO"
    currentScore=$((currentScore + 0))
  fi
}

Hi @nbrownuk and thanks for reporting this. I'll have a look at the proposed changes and run some test as soon as possible.

Hi @konstruktoid is there anything I can do to help move this issue along? Just let me know if I can help.

Hi again @nbrownuk and I'm so sorry for the way too long delay. Things stacked up and I completely forgot about this one.


Hide and mark as deprecated the --transform_amd_modules flag

I wish to delete the TransformAMDToCJSModule pass. As a first step, indicate that the open source flag is deprecated and remove it from the help display.

LGTM. I'll bring it in.


[Bug]: OAuth2 is giving error on prod while the same project works fine on release.

Is there an existing issue for this?

[X] I have searched the existing issues

Description
When I try to execute a drop box project which used OAuth2.0, it runs fine on release. However, when the same project is exported to production, its showing error.
Steps To Reproduce

Configure/ Edit the data source
Click on "Save and Authorise"

Observe - First, there is a success message and then there is an error message.
Public Sample App
https://app.appsmith.com/app/drop-box/orders-638dd78c2913ba43778b913d
Issue video log
https://www.loom.com/share/ec2459b1de564bff997d90bc14cb40a6
Version
V1.8.14

6m REST API usage: 5624
let's say 75% of them use authenticated api: 4218
of these, let's say again 50% use Oauth2.0: 2109
Stats

Stat
Values

Reach
2109

Effort (months)
0.25


cache column offsets

reduces time required to iterate over all quest records from ~43s to sub 1s on my machine.

ah forgot to do this, thanks for that


[Help] I can't finish any runs if my program is stiil running (even for none-wandb-related things).

Weights and Biases version: 0.8.31
Python version:  3.6.9
Operating System: Linux

Description
When I run many runs sequentially, I can see on the website that state of past runs keeps showing running, runtime keeps increasing too, and only when my program is stopped runs will show finished.

Is that mean I am running many many runs simultaneously during the running of my program ?

I want to shut down the run when I finish it.

I want to see the true runtime of runs.

What I Did
import time
for _ in range(100):
  wandb.init(entity=<my username>, project='test', config={'param':233})
  wandb.log({'metric':332})
  wandb.join()
  time.sleep(10)

This should work if you add reinit=True to your wandb.init() call

Or alternatively you can do:
import time
for _ in range(100):
  run = wandb.init(entity=<my username>, project='test', config={'param':233}, reinit=True)
  with run:
     run.log({'metric':332})
  time.sleep(10)


logout button doesn't appear to work

Clicking the logout button sends me back to Step 1, this is good.
However, when I click the "Google Login" button again I am not prompted to login with Google. It simply shows the previous account I logged-in with.
It doesn't seem to actually log me out.

It's worked fine when sin into multiple account.
If you have sin into only one account then it login automatically.
@andypotanin Need to show the login prompt every time or logout option in API.
Something like No prompt for re-authentication with OAUth2. Why and how to force it?

@alimuzzaman What do you mean by:

It's worked fine when sin into multiple account

User can sign in into multiple Google accounts at the same time with oAuth?
I guess I'm really not sure what the issue is.

@andypotanin

Should user ask for login for every time user click "Google Login" button or only after user click logout?

Only after the user click's the logout button.

@andypotanin add parameter max_auth_age=0 to auth url  if it's passed from user site.
Maybe https://github.com/UsabilityDynamics/node-product-api/blob/production/routes/stateless-v1/index.js#L332


PyTorch nightly version required?

In Requirements:
Pytorch version 0.4 or higher.
However, the maskrcnn_benchmark used in models/mrcnn.py requires PyTorch from a nightly release.
Am I missing something?

Yes, maskrcnn_benchmark got updated quickly that some dependencies got broken. I will adapt the code to use the trochvision maskrcnn_benchmark. Thanks for pointing this out.

Hi, @IssamLaradji , thanks for your implementation, just to make sure I understand corrently, so the prm_point_dict.json is obtained after training using PRM,  and proposal_dict.json is the coarse segmentation proposals obtain by MCG, right?
if I want to train on a new dataset, such as COCO dataset, I first need to get the MCG proposals and then train the PRM to obtain the peak points, right?

Hi all,
i installed the pytorch through anaconda by this commands:
conda install -c pytorch torchvision-nightly
conda install -c pytorch torchvision

unfortunately, maskrcnn_benchmark dependencies got broken
       from maskrcnn_benchmark.structures.bounding_box import BoxList
ModuleNotFoundError: No module named 'maskrcnn_benchmark'

How can i fix it?


 For a one
[  117s]         dimensional array, this returns those entries not returned by
[  117s]         `arr[obj]`.
[  117s]
[  117s]         Parameters
[  117s]         ----------
[  117s]         arr : array_like
[  117s]             Input array.
[  117s]         obj : slice, int or array of ints
[  117s]             Indicate indices of sub-arrays to remove along the specified axis.
[  117s]
[  117s]             .. versionchanged:: 1.19.0
[  117s]                 Boolean indices are now treated as a mask of elements to remove,
[  117s]                 rather than being cast to the integers 0 and 1.
...
[  117s]                 keep = ones(N, dtype=bool)
[  117s] >               keep[obj,] = False
[  117s] E               IndexError: arrays used as indices must be of integer (or boolean) type
[  117s]
[  117s] /usr/lib64/python3.8/site-packages/numpy/lib/function_base.py:4406: IndexError

Thanks, @bnavigator.  Looks like we just need to change the argument to a list with one element and should work with old and new numpy.

I am preparing a PR


Updated unit test instructions in README

Can one of the admins verify this patch?

Can one of the admins verify this patch?


fix: Collection pfp object fit

Thank you for your contribution to the KodaDot - One Stop Shop for Polkadot NFTs.
👇 __ Let's make a quick check before the contribution.
PR Type

[x] Bugfix
[ ] Feature
[ ] Refactoring

Needs Design check

@exezbcz please review

Context

[x] Closes #6718
[ ] Requires deployment <snek/rubick/worker>

Did your issue had any of the "$" label on it?

[x] My DOT address: Payout

Screenshot 📸

[x] My fix has changed UI

Copilot Summary

🤖 Generated by Copilot at e1eeea8
This pull request updates the image styling for the CollectionBanner and UnlockableCollectionBanner components by adding the object-fit-cover class to the img elements. This class is defined in the global.scss file, which is also modified to remove an unused class. The goal is to make the banner images fit the containers better and improve the visual appearance of the collections.

🤖 Generated by Copilot at e1eeea8

To make the banner images fit
The object-fit-cover class was a hit
But it had to be moved
To another component improved
And the global styles got a bit of a split

Two kinds of object-fit could be chosen. Which one do you prefer to? @exezbcz
object-fit: contain

object-fit: cover

pay 20 usd

😍 Perfect, I’ve sent the payout
💵 $20 @ 4.77 USD/DOT ~ 4.193 $DOT
🧗 16SjUbGKSdjCdWTy3NNT3JxbRVGGqD4mwkHpc6BD9U2Rp29Z
🔗 0x5336eeaeba15b2d958ec8f1d0e9dddf1f89e12385358beb3f770d5a5c7f1f988
🪅 Let’s grab another issue and get rewarded!
🪄 github.com/kodadot/nft-gallery/issues


add README and CONTRIBUTING guideline

Description
Add a more detailed README and CONTRIBUTING guideline.
Type of change

[ ] Bug fix
[ ] New feature
[ ] New protocol
[ ] Breaking change
[x] Documentation update

@drunkirishcoder reminder: update example readme's accordingly.


doc: add VUE_APP_API_BASE_URL in .env.example and readme

I see we are using VUE_APP_API_BASE_URL variable in the axios file but not declaring it in .env.example.
It'd be helpful for new contributors if we can add this.

Hiii @hvini ,  lemme know if you want this to be added.

Hello @Gaurav-Verma07 . Thanks for noticing.
Do you still want to work on this? Ping me if not. For now I'll have you assigned

@KarinePistili I experienced the issue mentioned here today while setting up the project locally. It's a genuine problem that could hinder new contributors. I'm keen to work on resolving this quickly to improve the onboarding experience.

@dakshsinghrathore , a PR has already been raised. #49


Multiple Conversation Threads, Prompt Customization, and Conversation Templates

Another feature that could enhance the flexibility and organization of using Aider in complex or multi-faceted projects is the support for multiple conversation threads.
Feature Request:
Please add support for multiple conversation threads in Aider, enabling users to maintain separate discussions or sessions on different aspects of the same repository without interference. This feature could be managed through a --thread command line parameter, which defaults to "main" but can be set to other names to switch between independent conversation histories and file selections.
Use Case:
In projects where different components or features require distinct but simultaneous development focus, having the ability to switch contexts within the same repository environment can be invaluable. For example, one might want to discuss database issues in one thread and UI enhancements in another, without mixing the conversations and relevant file edits.
Proposed Implementation:
Introduce a --thread parameter that specifies the active conversation thread.
Store conversation-specific data like .aider.input.history and other session files in thread-specific subdirectories or encode the thread name in the file names. For instance, using an .alder directory at the top level for global configurations, and thread-XXX subdirectories for each conversation thread's data, could help keep project directories organized and avoid clutter, while enabling future extensibility and per-thread customization.
Questions for Consideration:
Version Control for .alder Files: Is it considered good practice to check in .alder files into the repo to maintain conversation history across different development environments? What are the implications for privacy and clutter in version control, and uses of pre-configured thread templates for particular tasks like chatting about documentation, apis, porting, internationalization, etc.
Customizing Prompts per Conversation: Beyond simply including files with instructions, is there a mechanism to customize or augment the built-in AI prompts based on the conversation thread? This could tailor the assistance more closely to the context of each thread. Could there be an /instruct command like /add that adds those files as system prompts instead of user input?
Saved conversation threads could be used as Conversation Templates that are pre-configured, parameterized, and start private conversation threads that may or may not be ignored and checked into source code control.
Conversation Templates could included pre-configured sets of files, wildcards, or dynamic procedural selections (like dependencies on .h files from selections of .c files, so a conversation about documentation could include all the header files, and a conversation about user interface could include all of the header files and all of the user interface .c files, and user interface api documentation).
Selecting files per thread could be partially accomplished by threads having their own .alderignore files (including white lists) layered on top of the global repo wide alder white list. But procedure file selection by running shell commands or scripts would also be powerful. Also save and restore named sets of selected files on a per-conversation basis. So three layers of file name globbing + ignoring: repo, conversation thread, named file list.
This enhancement would not only make it easier to manage multiple development streams within a single repository but also align Aider with more complex development workflows where multitasking across different components is necessary.
I would also love a "/foreach [list of files] [instructions]" command whose behavior could be specialized in each conversation.
Thank you for considering this feature. I believe it could significantly enhance the practicality of Aider in more complex projects. I look forward to your insights and hopefully seeing this capability in a future release!
I am excited about the potential for this feature and how it could improve Aider!
System Info:
Aider version: v0.21.2-dev
Model being used: gpt-4-0613
Operating environment: Mac

Thanks for trying aider and filing this issue. I'm not sure I can address these concepts any time soon.
There's a FAQ entry that might contain helpful information:
https://aider.chat/docs/faq.html#can-i-script-aider
I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.


[v5.1] linking binary files is broken aka  asm("_binary_favicon_png_start"); (IDFGH-10339)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
release/v5.1
Operating System used.
Windows
How did you build your project?
Command line with Make
If you are using Windows, please specify command line type.
None
Development Kit.
ESP32-S3 Dev Kit C
Power Supply used.
USB
What is the expected behavior?
Linking multiple binary files in the same source file should work. They should each return a different assembly address.
What is the actual behavior?
All assembly address are the same.
{
            extern const unsigned char startAsm[] asm("_binary_jAssets_js_start"); // 0x3c1badce
            extern const unsigned char endAsm[]   asm("_binary_jAssets_js_end"); // 0x3c1bb591
}
{
            extern const unsigned char startAsm[] asm("_binary_style_css_start"); // 0x3c1badce
            extern const unsigned char endAsm[]   asm("_binary_style_css_end"); // 0x3c1bb591
}

Steps to reproduce.
void pd_api_webapp_wildcard(pd_xtrans_t* xtrans, const char* uri, pd_error_t* error)
{
    pd_err_unset(error);

    {
        const unsigned char* start = NULL;
        const unsigned char* end = NULL;

        // allow caching
        eTry(,pd_xtrans_resp_set_hdr,(xtrans, "Cache-Control", "public, max-age=31536000", error));

        if (strstr(uri, "/webapp/css/style.css") != NULL){
            extern const unsigned char startAsm[] asm("_binary_style_css_start");
            extern const unsigned char endAsm[]   asm("_binary_style_css_end");
            start = startAsm;
            end = endAsm;
            ESP_LOGI(TAG, "getting %s /webapp/css/style.css startAsm %p endAsm %p", uri, startAsm, endAsm);
        }

        if (strstr(uri, "/webapp/js/jAssets.js") != NULL){
            extern const unsigned char startAsm[] asm("_binary_jAssets_js_start");
            extern const unsigned char endAsm[]   asm("_binary_jAssets_js_end");
            start = startAsm;
            end = endAsm;
            ESP_LOGI(TAG, "getting %s /webapp/js/jAssets.js startAsm %p endAsm %p", uri, startAsm, endAsm);
        }

       ..... about ~20 more handlers just like this .....

        const size_t size = end - start;

        // found the file we wanted?
        if (!start || !end) {
            eFail("could not find %s", uri);
            return;
        }

        if (xtrans->which == PD_XTRANS_HTTP){
            pd_http_set_mime_from_uri(xtrans->to.httpReq, uri, true);
        }

        // send
        eTry(,pd_xtrans_resp_send,(xtrans, start, size, error));

        pd_err_success(error);
    }
end:
    return;
}

Debug Logs.
I (14621) pd api webapp: getting /webapp/js/jAssets.js?v=6a64ebfff1dc /webapp/js/jAssets.js startAsm 0x3c1badce endAsm 0x3c1bb591
I (14361) pd api webapp: getting /webapp/css/style.css?v=6a64ebfff1dc /webapp/css/style.css startAsm 0x3c1badce endAsm 0x3c1bb591

More Information.
No response

this worked on v4.4

I was able to fix it by giving them each a unique name. This was not required on v4.4.
        extern const unsigned char startAsm1[] asm("_binary_style_css_start");
        extern const unsigned char endAsm1[]   asm("_binary_style_css_end");

        extern const unsigned char startAsm2[] asm("_binary_style_css_start");
        extern const unsigned char endAsm2[]   asm("_binary_style_css_end");

etc

Hello @chipweinberger ,
The compiler is explicit about what is happening, when I try on my side, I get the warning:
warning: 'asm' declaration ignored due to conflict with previous rename [-Wpragmas]
   29 |         extern const unsigned char startAsm[] asm("_binary_jAssets_js_start");
      |         ^~~~~~
warning: 'asm' declaration ignored due to conflict with previous rename [-Wpragmas]
   30 |         extern const unsigned char endAsm[]   asm("_binary_jAssets_js_end");
      |         ^~~~~~

So the compiler will simply ignore the redefinition of the global symbol startAsm. Moreover, this seems like an undefined behavior to me in the sense that it is highly dependent on the compiler and linker, which would explain why it may work with other compiler versions (v4.4's) and not now.
Anyway, you don't need to use asm here, C can already reference any assembly symbol thanks to the linker, so the proper way to access your files is:
        if (strstr(uri, "/webapp/css/style.css") != NULL){
            extern uint8_t _binary_style_css_start;
            extern uint8_t _binary_style_css_end;
            start = &_binary_style_css_start;
            end = &_binary_style_css_end;
            ESP_LOGI(TAG, "getting %s /webapp/css/style.css startAsm %p endAsm %p", uri, startAsm, endAsm);
        }

        if (strstr(uri, "/webapp/js/jAssets.js") != NULL){
            extern uint8_t _binary_jAssets_js_start;
            extern uint8_t _binary_jAssets_js_end;
            start = &_binary_jAssets_js_start;
            end = &_binary_jAssets_js_end;
            ESP_LOGI(TAG, "getting %s /webapp/js/jAssets.js startAsm %p endAsm %p", uri, startAsm, endAsm);
        }

Make sure that in your assembly files, both *_start and *_end are global:
    .globl _binary_jAssets_js_start
    .globl _binary_jAssets_js_end
    .globl _binary_style_css_start
    .globl _binary_style_css_end

Thanks for the very detailed response @o-marshmallow !
I may have missed the warning somehow! As long as this is expected behavior, then that is totally fine. I just wanted to make sure this was not an unexpected regression. Thanks again.

My pleasure, I hope the previous example works for you


Update demos for Godot 3.4.3-stable

Draft for now as the Dodge the Creeps demo is crashing. I'll see if it's reproducible in 3.3-stable, ~it sounds related to #59~ (that PR didn't change Dodge the Creeps, that predates it).
Thread 1 "godot-3.4" received signal SIGSEGV, Segmentation fault.
0x00000000032ddd04 in PtrToArg<bool>::convert (p_ptr=0x0) at ./core/method_ptrcall.h:104
104     MAKE_PTRARG(bool);

(gdb) bt
#0  0x00000000032ddd04 in PtrToArg<bool>::convert (p_ptr=0x0) at ./core/method_ptrcall.h:104
#1  0x0000000006082fc6 in MethodBind2RC<float, StringName const&, bool>::ptrcall (this=0x7471e70, p_object=0x73cb750, p_args=0x7fffffffc318, r_ret=0x7fffffffc310) at ./core/method_bind.gen.inc:2112
#2  0x00000000035fe2d8 in godot_method_bind_ptrcall (p_method_bind=0x7471e70, p_instance=0x73cb750, p_args=0x7fffffffc318, p_ret=0x7fffffffc310) at modules/gdnative/gdnative/gdnative.cpp:67
#3  0x00007fff9796f818 in godot::___godot_icall_float_String (arg0=..., inst=0x8356820, mb=<optimized out>) at godot-cpp/include/gen/__icalls.hpp:6754
#4  godot::Input::get_action_strength (this=this@entry=0x8356820, action=...) at godot-cpp/src/gen/Input.cpp:95
#5  0x00007fff978ef51d in Player::_process (this=0x8150fe0, p_delta=0.15000003576278687) at src/player.cpp:15
#6  0x00007fff978efcdc in godot::_WrappedMethod<Player, void, double>::apply<0> (args=0x7fffffffc4a8, obj=<optimized out>, this=<optimized out>) at godot-cpp/include/core/Godot.hpp:295
#7  godot::__wrapped_method<Player, void, double> (method_data=<optimized out>, user_data=<optimized out>, args=0x7fffffffc4a8) at godot-cpp/include/core/Godot.hpp:310
#8  0x0000000003630637 in NativeScriptInstance::call_multilevel (this=0x830b4c0, p_method=..., p_args=0x7fffffffc4a8, p_argcount=1) at modules/gdnative/nativescript/nativescript.cpp:892
#9  0x0000000004f0f94e in Node::_notification (this=0x8369fa0, p_notification=17) at scene/main/node.cpp:56
#10 0x0000000003113a38 in Node::_notificationv (this=0x8369fa0, p_notification=17, p_reversed=false) at ./scene/main/node.h:45
#11 0x0000000003114067 in CanvasItem::_notificationv (this=0x8369fa0, p_notification=17, p_reversed=false) at ./scene/2d/canvas_item.h:163
#12 0x0000000004eb1475 in Node2D::_notificationv (this=0x8369fa0, p_notification=17, p_reversed=false) at ./scene/2d/node_2d.h:37
#13 0x00000000054045e5 in CollisionObject2D::_notificationv (this=0x8369fa0, p_notification=17, p_reversed=false) at ./scene/2d/collision_object_2d.h:38
#14 0x0000000005404cdb in Area2D::_notificationv (this=0x8369fa0, p_notification=17, p_reversed=false) at scene/2d/area_2d.h:38
#15 0x0000000005f5ca91 in Object::notification (this=0x8369fa0, p_notification=17, p_reversed=false) at core/object.cpp:927
#16 0x0000000004f454b3 in SceneTree::_notify_group_pause (this=0x80821a0, p_group=..., p_notification=17) at scene/main/scene_tree.cpp:990
#17 0x0000000004f43145 in SceneTree::idle (this=0x80821a0, p_time=0.150000036) at scene/main/scene_tree.cpp:536
#18 0x00000000030eddc1 in Main::iteration () at main/main.cpp:2201
#19 0x00000000030b1fe7 in OS_X11::run (this=0x7fffffffcd90) at platform/x11/os_x11.cpp:3643
#20 0x000000000309fc4f in main (argc=1, argv=0x7fffffffd6b8) at platform/x11/godot_x11.cpp:55

I tested Dodge the Creeps in 3.3-stable and 3.3.4-stable and both work fine.
Then I tried again with a clean build (including git clean -fxd in the godot-cpp folder) and now it's working fine with godot-3.4.3-stable. So it was just an issue of our SCons buildsystem for godot-cpp not properly rebuilding everything that needs to be on updates.


Upload LFS Objects from the browser

This is a feature request. Let's offer a way to upload Git LFS objects from the web browser. Currently you have to do it from Git a browser upload of Git LFS objects would be convenient.

Have you tried this since #5702 was merged?
If the file should be in the LFS as determined by the git attributes it will be stored in the LFS.

I'm going to close this issue as this is possible since #5702


Game Crashing When I Open The Recipe For Essentia Filter (NEW)

I looked that old game crush issue it's not looks like that i deleted that mod on my mod folder but issue is keep still here is an CRUSH REPORT FILE : crash-2019-03-02_10.48.01-client.txt


Javascript error in the main process

Runtime error: Javascript error occurred in the main process.

Hardware: Rpi 5 4gb
Os: Debian 12
6.6.28+rpt-rpi-2712

Suspect the same issue as #76


[BUG] Telegram Bot: Multiple downloads use the same progress indicator

Is your feature request related to a problem? Please describe.
 Sometimes person using the telegram bot stop their downloads because they thought that their previous dowload stop progressing.
Describe the solution you'd like
 different progress bar for multiple downloads
Describe alternatives you've considered

Additional context

Isn't there a progress indicator already?

yes there is, but the multiple downloads share the same progress indicator.

I hope that they have separate indicator

but if it is a hustle for you, you could opt to not do it.

You're right, I forgot to adjust that. It's still storing just one message to always update. Will fix that.

thank you!!!!!

what does this mean?

The headless browser couldnt find the chapter body(css selector: article) on the page within 30 seconds. It could be a server time out or there is no article element on the page (the chapter url for that chapter is somehow wrong). I would suggest checking out that chapter manually and then if that is ok, try again.

Should be fixed now!

@Flameish can you make a discord bot please!

You already opened another issue for that, please keep these separated. It makes it way easier to keep track of what needs to be done. If there is any news, I will answer there.

Ok, sorry!

downloads still share the same progress indicator

ohhh never mind I just forget to update


update to use Guard 2.0

What's the plan for releasing a new gem?

@ezekg - waiting for access
@effkay - just asking again (I guess you're still on vacation): could you add me<EMAIL_ADDRESS>and @thibaudgg<EMAIL_ADDRESS>- just in case) as owners? Thanks!

@e2 @thibaudgg You're good to go, sorry for the delay!

Guard::CoffeeScript 2.0.1 released!
Thanks @effkay!


Slower behavior using Fast API (same gunicorn config) Gunicorn with uvicorn.workers.UvicornWorker than using default one

I'm using this dependencies:
starlette==0.13.6 uvicorn==0.11.8 fastapi==0.61.1
According to every check I've worked with, using FastAPI is supposed to handle more request per second than a Flask App but I'm finding different behaviors.

Using Flask + Gunicorn, I'm getting twice the number of req/s vs using FastAPI + Gunicorn(uvicorn.workers.UvicornWorker)

The other params (besides the worker class) are the same for both apps.
What can I check to improve this ? or is this the expected behavior? If so, what could be other places I could look into to improve the performance of using FastAPI ?

How are you benchmarking them?

It's kinda hard to tell without code and how you benchmarking them (running options etc.)
Also, I did some benchmarking you might want to check it out

I am benchmarking them using locust (https://locust.io/) with 100 concurrent users and a hatch rate of 5.
The application is receiving an avro schema and running some calculations for both cases.  The only difference besides the actual api (FastAPI vs Flask) in both codes is that the FastAPI implementations need to run the request.body() in an asynchronous way
If you need a little bit more detail just let me know. Thanks in advance!

Yes, please add these,
How do you run FastAPI, and options?

Example: waitress-serve  --call "your:app"

The code that you use for testing for FastAPI

from fastapi import FastAPI, Request

app = FastAPI()

@app.get("/etc")
async def etc(request: Request):
    return await request.body()

The code that you use for testing for Flask
from flask import Flask

...

etc.

If you do not use uvicorn, it would't give you a good performance.
It's maybe a choice that Flask app + Tornado run.
    http_server = HTTPServer(WSGIContainer(app))
    http_server.listen(80)
    IOLoop.instance().start()

Here are the results using ab to benchmark it too and using Fast API is still slower than Flask:

Uvicorn   + Fast API:

`This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests
Server Software:        uvicorn
Server Hostname:        localhost
Server Port:            8080
Document Path:          /invocations
Document Length:        283 bytes
Concurrency Level:      10
Time taken for tests:   99.046 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      808000 bytes
Total body sent:        39252000
HTML transferred:       566000 bytes
Requests per second:    20.19 [#/sec] (mean)
Time per request:       495.232 [ms] (mean)
Time per request:       49.523 [ms] (mean, across all concurrent requests)
Transfer rate:          7.97 [Kbytes/sec] received
387.01 kb/s sent
394.98 kb/s total
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       6
Processing:   137  491 349.1    397    3087
Waiting:      137  490 349.0    397    3087
Total:        137  491 349.1    398    3087
Percentage of the requests served within a certain time (ms)
50%    398
66%    520
75%    590
80%    639
90%    912
95%   1136
98%   1549
99%   1843
100%   3087 (longest request)`

Gunicorn + Flask:

`This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests
Server Software:        gunicorn/20.0.4
Server Hostname:        localhost
Server Port:            8080
Document Path:          /invocations
Document Length:        283 bytes
Concurrency Level:      10
Time taken for tests:   86.898 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      862000 bytes
Total body sent:        39252000
HTML transferred:       566000 bytes
Requests per second:    23.02 [#/sec] (mean)
Time per request:       434.491 [ms] (mean)
Time per request:       43.449 [ms] (mean, across all concurrent requests)
Transfer rate:          9.69 [Kbytes/sec] received
441.11 kb/s sent
450.80 kb/s total
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:   285  432 144.4    392    1661
Waiting:      285  431 144.3    392    1659
Total:        286  432 144.4    393    1661
Percentage of the requests served within a certain time (ms)
50%    393
66%    418
75%    439
80%    457
90%    550
95%    778
98%    925
99%   1030
100%   1661 (longest request)`

@Miguelme I don't know how these benchmarks are useful for us to help you.

I posted you the benchmarks and the way I execute each of the apps using gunicorn hoping that could help.
I will paste you here the code difference too. If there is anything else you could need to help me, feel free to ask to see if I can provide it.
Flask Code:
`
app = Flask(name)
@flask_app.route("/invocations", methods=['POST'])
def invocations():
    input = list(serialize_input(request.body()))
    output = # runs_some_calculations
    response_data = serialize_output_to_avro(output)
    response = make_response(response_data)
    response.headers['Content-Type'] = 'avro/binary'

    return response

`
FastApi Code:
`
app = FastAPI()
@app.post("/invocations")
async def invocations(request: Request):
    input = list(serialize_input(await request.body()))
   output = # runs_some_calculations
    response_data = serialize_output_to_avro(output)
    response = Response(response_data, headers={'Content-Type': 'avro/binary'})

    return response

`

Thanks for the help everyone! 🍰
@Miguelme I see you are running stuff that seems like it's probably blocking code (e.g. Avro stuff) inside of an async function, so you are probably blocking the main event loop. Read more about it here: https://fastapi.tiangolo.com/async/
If you want to check general benchmarks run by a third-party, check out those done by TechEmpower.

Sorry for the long delay! 🙈  I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.


Add minting functionality to token

This changeset adds a permissioned minting function to the ArrowToken contract, which allows the contract owner to mint new tokens.
Only the address owning the token contract is permitted to mint new tokens.

Great work! All looks good to me


Missing required profiles dir on Prefix

I suspect this is a user error but on Prefix, pkgcheck scan in my git repo works, while pkgdev commit doesn't:
~/Documents/git/gentoo/app-editors/ne $ pkgcheck scan
app-editors/ne
  UnknownManifest: unknown distfile in Manifest: [ ne-3.1.1.tar.gz ]
  UnstableOnly: for arches: [ amd64, ppc, ppc64, x86 ], all versions are unstable: [ 3.3.0 ]
  UnusedLocalUse: metadata.xml unused local USE flag: [ tinfo ]

~/Documents/git/gentoo/app-editors/ne $ pkgdev commit
pkgdev commit: error: invalid repo: missing required profiles dir: '/Users/sam/.cache/pkgcore/repos/gentoo'

How should I allow it to automagically configure as normal (or recognise my ${EPREFIX}/etc/portage/* config if that's ever used, which I doubt)? I've not made any configuration changes on a normal Gentoo system.
This is with 9999 for the whole suite.

 That should currently only be needed for the github action (and similar CI setups) although it could be made to work with a prefix setup, I just don't care much about that since I've yet to use pkgcore in a prefix environment and I doubt anyone else uses pkgcore to install things atm.

Yeah, that's fine. But I didn't really want it to exist in the first place. Doesn't help:
 ~/Documents/git/gentoo/app-portage/nattka $ pkgdev commit
pkgdev commit: error: invalid repo: missing required profiles dir: '/Users/sam/.cache/pkgcore/repos/gentoo'
~/Documents/git/gentoo/app-portage/nattka $ rm -rf ~/.cache/pkgcore
~/Documents/git/gentoo/app-portage/nattka $ pkgdev commit
pkgdev commit: error: invalid repo: missing required profiles dir: '/Users/sam/.cache/pkgcore/repos/gentoo'

I'll try to reproduce this on a non-Gentoo box somewhere so I can fix it.

$ pkgdev commit --debug
Traceback (most recent call last):
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/config/central.py", line 213, in _instantiate
    self._instance = callable_obj(*pargs, **configdict)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/ebuild/repository.py", line 630, in tree
    return tree_cls(
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/ebuild/repository.py", line 302, in __init__
    raise errors.InvalidRepo(f'missing required profiles dir: {self.location!r}')
pkgcore.repository.errors.InvalidRepo: invalid repo: missing required profiles dir: '/Users/sam/.cache/pkgcore/repos/gentoo'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/config/central.py", line 139, in instantiate
    self._instance = self._instantiate()
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/config/central.py", line 218, in _instantiate
    raise errors.InstantiationError(
pkgcore.config.errors.InstantiationError: Failed instantiating section 'gentoo': exception caught from 'pkgcore.ebuild.repository.tree'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sam/Gentoo/usr/lib/python-exec/python3.9/pkgdev", line 4, in <module>
    scripts.run(basename(__file__))
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/__init__.py", line 38, in run
    sys.exit(tool())
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/snakeoil/cli/tool.py", line 79, in __call__
    ret = self.main()
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/cli.py", line 13, in main
    return super().main()
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/snakeoil/cli/tool.py", line 185, in main
    self.handle_exec_exception(e)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/snakeoil/cli/tool.py", line 170, in main
    exitstatus = func(self.options, self.out, self.err)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/pkgdev_commit.py", line 679, in _commit
    args = determine_msg_args(options, changes)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/pkgdev_commit.py", line 581, in determine_msg_args
    message.append(changes.prefix + changes.summary)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/snakeoil/klass.py", line 176, in __get__
    obj = self.function(instance)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/pkgdev_commit.py", line 459, in summary
    elif summary := PkgSummary(self._options, self.ebuild_changes).generate():
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/pkgdev_commit.py", line 220, in generate
    return self.status_funcs[statuses](self)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/pkgdev_commit.py", line 350, in modify
    if self.old_repo is None:
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/snakeoil/klass.py", line 176, in __get__
    obj = self.function(instance)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgdev/scripts/pkgdev_commit.py", line 211, in old_repo
    return self.options.domain.add_repo(
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/ebuild/domain.py", line 639, in add_repo
    repo_obj = ebuild_repo.tree(config, repo_config, **kwargs)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/ebuild/repository.py", line 620, in tree
    masters = tuple(config.objects['repo'][r] for r in repo_config.masters)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/ebuild/repository.py", line 620, in <genexpr>
    masters = tuple(config.objects['repo'][r] for r in repo_config.masters)
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/config/central.py", line 41, in __getitem__
    return conf.instantiate()
  File "/Users/sam/Gentoo/usr/lib/python3.9/site-packages/pkgcore/config/central.py", line 143, in instantiate
    raise errors.InstantiationError(self.name) from e
pkgcore.config.errors.InstantiationError: Failed instantiating section 'gentoo'

I'm unable to reproduce this on multiple non-Gentoo systems (no macos though), so it might have to wait a bit until I get into adding unit tests for historical repo queries and see if the macos envs break.

Try pulling the whole stack from git again, I might have fixed this case with some additional repo init rework.

You seem to have fixed it!

For posterity, this was fixed by reworking repo init to favor external repos in https://github.com/pkgcore/pkgcore/commit/80120597529dba6ac913b14c357f0be171f289ab.


Fix SSL support between master and node(s)

@twiest Please review
@detiber @akram FYI

@jwhonce you'll also want to add something like this: https://github.com/detiber/openshift-online-ansible/blob/enterprise2/roles/base_os/tasks/main.yaml#L14-L20
Otherwise, the cli commands are broken by default with https enabled.

@detiber Did you want me to add that code in this PR, or are you going to create a PR with it?

@jwhonce I can create an additional PR for that. Incoming shortly


Make alert word UI follow correct data flow for error handling and propagation.

Fixes: #1269.

@HarshitOnGitHub can you break this into smaller commits so that it's more reviewable?  E.g. the change to how the status is indicated could be its own small commit, etc.

One suggestion, it would be better to add one more case for an error when the entered word is already present in the alert word list. This way we don't have duplicates (considering the comparison is case sensitive).
@tejaskasetty Yeah this is a known issue to me(haven't opened an issue for it though) and I plan to work on it in a separate PR. :)

@HarshitOnGitHub Then everything else seems to be fine.

@HarshitOnGitHub I merged the first 3 commits.  Posting some comments on the rest.

@timabbott Addressed all the comments, broken up the commit you mentioned and also fixed the issue @tejaskasetty had mentioned.

Merged, thanks @HarshitOnGitHub!


referencing hypercore-streams docs

To have only one source of truth for the hypercore-streams API I figured it is a good idea to just reference the API in order to make sure that users look at the right place for the options.
Companion PR to https://github.com/hypercore-protocol/hypercore-streams/pull/8

Closing cause old and lots have changed


What's the requirement for Azure VM type for windows container?

What's the requirement for Azure VM type for windows container? Windows container should be supported by specified VM types, for example I believe Standard_B8ms VM pool cannot be used to run container workload. So could you provide the VM types info that support Windows container in Azure Batch?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 206a4592-16ac-8ef7-7a33-508535c3ea50
Version Independent ID: 24785e7d-1caf-dd48-dc26-e5d258d899b1
Content: Container workloads on Azure Batch
Content Source: articles/batch/batch-docker-container-workloads.md
Service: batch
GitHub Login: @dlepow
Microsoft Alias: danlep

@nicolaszhang
Thanks for the feedback! We are currently investigating and will update you shortly.

@nicolaszhang we have a guide already on what sizes are supported in Batch
https://docs.microsoft.com/en-us/azure/batch/batch-pool-vm-sizes
Essentially all series are supported with only a few exceptions which you can read from the above link.

But that link doesn't mentioned which VM type is supported to run CONTAINERIZED workload in batch, for example I don't think Standard_B8ms VM pool can be used to run Windows container workload in batch. So, could you please elaborate on that? Specially, in my scenario, I'd like to run Windows container with/without hyper-v in Azure Batch VM.

As per the link I shared none of the B series work in Batch.

Running a containerized VM in batch does not depend on the node size. Apart from the resources allocated.
You can pick any of the VM sizes in the link I shared and then you just use one of the following marketplace images

Windows Server 2016 Datacenter with Containers i
CentOS for Azure Batch container pools
CentOS (with RDMA drivers) for Azure Batch container pools
Ubuntu Server for Azure Batch container pools
Ubuntu Server (with RDMA drivers) for Azure Batch container pools

@nicolaszhang - Just chiming in here. In addition to the VM sizes article Micah mentioned, for Windows container workloads specifically the following statement is included in the documentation: "For Windows container workloads, it's recommended to choose a multicore VM size for your pool". For recommendations for your particular scenario, you could also reach out to the askwabatch alias (since you're within Microsoft). Hope this helps.

Thanks @dlepow!
@nicolaszhang I will close this out for now but if you feel you need more information just let us know and we can reopen and continue.

Sorry, I may not make myself clear, I mean VM TYPE ( A-Serials, B-Serials, etc). And I think A-Serials, B-Serials VM etc doesn't support Windows Container, so Azure Batch should not be able to support that VM for Windows container workload either, correct me if I'm wrong here. If so, I think it's better to clearly mention here which serials of VM of batch pool support Windows container (with and without hyper-v container) well

There is no VM series limitations I am seeing for Windows Containers. There are system requirements such as a minimum about of CPU and RAM but not about a particular series.
https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/system-requirements
https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/index