[RFD] Standardization of JWTs for user authorization

Architectural Decision Record: Implementing JWT-Based Authorization
Context
In microservices architecture, securing API access is a critical concern. We need a reliable, scalable, and efficient way to manage authentication and authorization across different services. This RFD is constrained for discussion of using a bearer token for authorization of a client to one or more microservices.  Client authentication  is out of scope for this RFD as is service to service communication.
JSON Web Tokens (JWT)s are described in IETF RFC 7519 with best practices in IETF RFC 8725.  They provide a standardized way of securely transmitting information about the entity making a request and are widely used in modern web applications.
Decision
We propose a JWT-based authorization scheme, where JWTs serve as bearer tokens in HTTP headers.  Each token will must meet the standards below for use of registered claims in addition to reasonable date claims.  The token must be signed securely by a known issuer.  For defense in depth, the token must be initially validated at any API Gateway and re-validated by each service involved in delivery of the response.  This is in addition to any policy processing that may happen within any service mesh.

API Gateway Validation:

The API Gateway will perform the initial validation of the JWT. It will check if the JWT is correctly signed by a known issuer, ensuring the token's integrity and authenticity.
It will verify that the token's date claims (like iat, nbf, exp) are within reasonable ranges, preventing expired or not-yet-valid tokens from being used.
The API Gateway will also confirm that the first downstream service intended to process the request is included in the JWT's aud (audience) claim.

Service Endpoint Validation:

Each service endpoint will perform its own validation of the JWT to ensure that the request has the necessary permissions for the requested operation. This double validation ensures an additional layer of security.

JWT Claim Usage

Registered Claims:

iss (Issuer): To identify the issuer of the token. Each authentication scope should have a separate issuer
sub (Subject): To identify the principal that is the subject of the JWT. The subject must be unique within the issuer.  In any case where an authorization decision cannot be made through data in the token, a service should be able to look up extended permissions information at the issuer based on the subject.  (API Key ID)
aud (Audience): To ensure that the JWT is intended for the service or API receiving it.  Any microservice not included in the array of strings in the aud claim should not perform any action based on the authorization in the token
exp (Expiration Time): To check token's validity period.
nbf (Not Before): To identify the time before which the JWT must not be accepted for processing.
iat (Issued At): To know when the token was issued.
jti (JWT ID): To provide a unique identifier for the JWT.

Additional Claims for Authorization:

roles/permissions: To specify the roles or permissions assigned to the token's subject, enabling fine-grained access control.
scope: To define the scope of access granted by the token.
Custom claims as per specific service requirements.

Consequences

Enhanced Security: Double validation at both the API Gateway and service endpoints adds an additional security layer.
Standardization: Using JWTs standardizes the authorization mechanism across services.
Scalability: JWTs facilitate scalability as they do not require server-side session storage.
Complexity: Implementing double validation and managing JWTs, especially in terms of claim management and token lifecycles, adds complexity to the system.
Performance Considerations: Care must be taken to ensure that the validation process does not introduce significant latency.

This specifically doesn't specify what will create or manage the JWTs.  Keycloak and AWS cognito can both create and manage JWTs that conform to the spec.  Likely lots of other tools as well.

In the scenario where each micro service is responsible for parsing JWTs, is the idea to reuse a common parser library (or module, or whatever Go calls them) across all OCHAMI micro services to avoid duplicating code?

The standard is more important than the library, but yes. I think a library is a good idea.

Regarding the list of registered claims, those are exactly the list of claims in the JSON Web Token Claims Registry in RFC7519, so if these are expected to be required then I think that is reasonable. (On a related note, would it improve clarity to borrow the MUST/SHOULD/MAY language from RFC2119?)
Regarding the "additional claims," are those expected to be required components of the JWT too?
I am curious how scope is different from roles or permissions. Intuition suggests (to me at least) that they are different words to describe a similar concept, i.e., whether the JWT permits the subject to perform a desired action at the API endpoint. Do you have an example that would distinguish them?

The goal of this RFD is to specify how the claims should be interpreted by the microservices.  The IETF RFC describes the registered claims in such general terms that we can't solely rely on it.
As for scope and roles/permissions, I'm suggesting that a role may be analogous to a RBAC group from LDAP and that a scope acts as a limiter in the case that a token is only viable for mutating the state of a subset of the resources behind an API.  As an example, we can create a specific API Key for a member of the Admin Group that we want to be able to act with admin permissions, but since we want to embed it in a script that can change the boot image associated with a job, we only want that token to be valid for a subset of nodes that are part of the job.

The goal of this RFD is to specify how the claims should be interpreted by the microservices. The IETF RFC describes the registered claims in such general terms that we can't solely rely on it.

OK. Since there is no normative or informative text in the "JWT Claim Usage" section, it was not clear to me whether you were proposing "Registered Claims" and/or "Additional Claims for Authorization" as required components of a JWT, or if they were just examples. It sounds like it is the latter?

As for scope and roles/permissions, I'm suggesting that a role may be analogous to a RBAC group from LDAP and that a scope acts as a limiter in the case that a token is only viable for mutating the state of a subset of the resources behind an API. As an example, we can create a specific API Key for a member of the Admin Group that we want to be able to act with admin permissions, but since we want to embed it in a script that can change the boot image associated with a job, we only want that token to be valid for a subset of nodes that are part of the job.

That makes sense, thanks.

In discussing multi-tenancy, we're considering adding a claim for partition-id that is an array of uuids for which the jwt is valid to act.  Microservices can use this information to limit the scope of an action to resources within the named partitions.

As part of an experimental repo, we created a standard middleware that can validate jwts to ensure they are properly signed and contain a set of required claims.
The idea here is that each microservice may choose to extend the usage of this middleware and use the claims in the business logic.
https://github.com/OpenCHAMI/node-orchestrator/blob/9d5d661eedb17044d502bc3c40a7aaccb307068a/main.go#L144C1-L173C2

(All views expressed are my own. If at all, they originate from my role as an OpenCUBE developer.)
While I see why you want to move away from SPIFFE/SPIRE, in particular when decoupling from K8s, a discussion on the shortcomings of the current approach would proof useful I think. Alongside how this integrates with SPIFFE/SPIRE if someone still wanted to maintain such a system.
For the double validation without explicitly defining an implementation to be used I think this calls for weird bugs. I see why you'd want this in a scenario where services can run as standalone containers rather than behind a service mesh and relying solely on a proxy. But keeping this in sync may become difficult. Maybe explicitly restricting the API gateway to only validate the base claims?
Also the claim that double validation increases security is something I do not understand in this context.
Lastly I am worried about this:

Scalability: JWTs facilitate scalability as they do not require server-side session storage.

While I have to admit that I lack practical experience, many blog posts and even books claim the use of JWTs as session storage as being dangerous (and often used as one of the reasons to promote JWT alternatives which avoid this by specification).
Some more nitpicks:

sub (Subject): To identify the principal that is the subject of the JWT. The subject must be unique within the issuer. In any case where an authorization decision cannot be made through data in the token, a service should be able to look up extended permissions information at the issuer based on the subject. (API Key ID? Email Address?)

Not entirely happy about this mechanism as I would fear amplification attacks (or behaviour looking like one).

exp (Expiration Time): To check token's validity period.
nbf (Not Before): To identify the time before which the JWT must not be accepted for processing.

Together (nbf,exp) are giving the validity period.

I'd be interested in seeing commentary on alternatives here. Re

I see why you want to move away from SPIFFE/SPIRE

those considerations aren't covered in much detail here (or in https://github.com/OpenCHAMI/roadmap/issues/21, which was the other place I found auth considerations mentioned), so we should record and summarize discussions from elsewhere in this RFD.

We can't have JWTs alone as an authorization or authentication system--JWTs are (often) components of such systems, but they aren't systems in themselves.
We will likely find that we need at least some of the broader system structure covered by either OAuth/OIDC or SPIFFE/SPIRE in the long run, and I'd recommend that we encourage adoption of those if possible, but with carveouts for components that can't support them for whatever reason.
Are there concerns re SPIRE primarily related to not going full Kubernetes, or something else? If there are aspects of the broader SPIFFE and OAuth frameworks that we want to omit or handle differently, what are they?

For background, https://spiffe.io/docs/latest/spire-about/comparisons/ lays out where SPIFFE/SPIRE sits in the overall landscape. https://spiffe.io/docs/latest/spiffe-about/overview/ and https://spiffe.io/docs/latest/spire-about/spire-concepts/ cover the protocol and reference implementation. Neither is coupled to Kubernetes:

SPIFFE is a general protocol and not tied to any particular environment.
SPIRE supports Kubernetes and provides Kubernetes resources for deploying it. It also has Docker Compose manifests and basic Linux binaries and source available.

I also mention OIDC because it's the de facto standard "OAuth has a lot of areas where you need to fill in the blank to get a viable system, we fill in those blanks" protocol. It uses JWTs for its tokens (OAuth kinda vaguely declares "you'll need tokens, figure out a mechanism for them") and enumerates various actors (clients, things that want to trust clients, and a source of truth for vetting clients).
There was previous mention of Keycloak in this RFD, and Keycloak is an OIDC IdP, though AFAIK it can also provide more bare bones OAuth that doesn't conform to OIDC as well (I've generally only interacted with it in an OIDC context).
There are portions of OIDC that we likely don't care about (it's partially targeted at handling human accounts, and as such has standards around personal names and such), but you can ignore those if you don't need them, while still using the rest of the framework.
SPIRE provides an OIDC discovery provider, though it's not clear if they provide a full IdP. Initial review of some blogs suggests they do.


Globals are not auto-completed

Describe the bug
Globals are not auto-completed
To reproduce
Steps to reproduce the behaviour:

Create a global (e.g. 'website')
Assign field to said global.
Try to use the autocomplete functionality

Expected behaviour
The editor should auto-complete globals.
Screenshots
Other auto-completions do work:

Global auto-completion doesn't work:

Versions

Plugin version: 1.0.5
Craft version: 3.7.20

Not experiencing that behavior here. It's possible you just need to regenerate the auto-complete classes, can you try deleting them from storage/runtime/compiled_templates and see if they regenerate with your globals in there as expected?

We might consider adding an event listener to invalidate the cache whenever:

Globals are changed via Globals::EVENT_AFTER_SAVE_GLOBAL_SET
Fields are changed via Fields::EVENT_AFTER_SAVE_FIELD

cc: @bencroker

So I was correct that we should add a listener to this event so that we invalidate the generated classes when a global set is saved, because they could be adding or renaming the injected global various name.
However, globals still weren't showing up. It's because as of Craft CMS 3.7.8, Craft conditionally loads the GlobalsExtension only if it's a frontend request. So if the autocomplete classes are regenerated via a CP request (which is most likely), the globals won't be present.
Now we manually add the globals to the mix every time, which should take care of the issue.
Addressed in: https://github.com/nystudio107/craft-autocomplete/commit/0a2169e2bd22a607f334e46d8a08b0bf6645ca9d
Released as Craft Autocomplete 1.0.6 -> https://github.com/nystudio107/craft-autocomplete/releases/tag/1.0.6

Very cool!

Very cool indeed. Thanks for the release!


20201217 - meeting

:alarm_clock: Time: 17 Dec 2020, 19:00 GMT
Just an hour hanging out and talking about DAT/Hyper* stuff. Now that Beaker has turned 1.0 🎂, we've got new opportunities to invite and involve hackers of all skill sets and levels!
:point_right: Post a comment in this github issue with any topics you'd like to cover this week
:link: https://zoom.us/j/181036238 (Native)
:link: https://zoom.us/wc/join/181036238?pwd= (Web)

Timezone
Date/Time

California
Dec 17 Thu, 11 AM

Colorado
Dec 17 Thu, noon

New York
Dec 17 Thu, 02 PM

Buenos Aires
Dec 17 Thu, 04 PM

London
Dec 17 Thu, 07 PM or 19:00

Berlin
Dec 17 Thu, 08 PM or 20:00

Or in your local time: https://www.timeanddate.com/worldclock/fixedtime.html?msg=Dat+Comm-Comm+Meeting+2020-12-17&iso=20201217T19&p1=1440&ah=1

The best meetings have lots of chat but even more code. Come and share your screen and show us what you're hacking on!

We're live, here are the notes:
https://hackmd.io/-iho-HnzTbiBZfDE1GRBRw


Create Event Card component with the design specified below

Please assign this to me.

Hi @Martijn0405 , first of all thanks for creating a descriptive issue and willing to help us here.
You could definitely go ahead and raise a PR for the issue that has been created by you

https://github.com/Real-Dev-Squad/website-calendar/issues/124#issue-1619257254

Is this available? @harshith-venkatesh

@harshith-venkatesh Could you please decide who to assign. In the meanwhile I already made a PR fixing this issue.

I think we will be going forward with @Martijn0405 's PR, since it has already been raised

Is this issue completed?


Download links are broken

Like #3409, download links don't work anymore => 404

Only the "latest" links are broken.
This is intentional, because we're no longer uploading pre-compiled binaries to dl.google.com as part of our release process.
The metrics we have show that downloads of the pre-compiled files from dl.google.com are very near zero each month.
It seems that nearly everyone uses Maven or NPM to install closure-compiler.
We're trying to streamline our external release process with the ultimate goal of being able to release more frequently.
Since no one (for a small but apparently non-zero value of "no one") seemed to use these links, and uploading to them is more tedious than you would expect, we decided to drop them from our process.
We updated the binary download page to reflect this, but failed to make an announcement.
https://github.com/google/closure-compiler/wiki/Binary-Downloads
Sorry about that.
If fetching the compiler via Maven or NPM doesn't work for you, the alternative is to build from source.
https://github.com/google/closure-compiler#building-it-yourself


💳 paid subscriptions via stripe

a person who create a xiome community will be able to customize subscription plans, which users will be able to buy, establishing recurring credit card payments for the merchant's services.

[ ] functionality in test suites

merchants can link their stripe account
community staff can customize tiered subscription plans, and the privileges associated with each each kind of subscription
community members can purchase subscriptions, and manage their subscriptions (cancellations, updating credit card info, etc)

[ ] mockmode frontend is operational

we have the components to exercise the ui

[ ] stage environment integrates with stripe testing systems

we should be able to exercise the whole system against a stripe testing account on the xiome stage

[ ] release to production connected to xiome's real stripe account

closed with the intention of decomposing this into smaller tickets


Add a screen for displaying all trending Shows

As a series enthusiast,
I want to browse a screen dedicated to trending shows displayed in a grid format with endless pagination
so that I can continuously discover and explore the most popular shows without interruptions.
Acceptance Criteria:

The trending shows screen should be easily accessible from home sections with a "More" button.
Shows should be displayed as thumbnails in a grid format.
Upon scrolling to the bottom of the list, the next set of trending shows should automatically load (endless pagination).
There should be a visual indication (e.g., loading spinner) while new shows are being fetched.
In case of a network error or no more trending shows available, an appropriate message should be displayed to the user.
Tapping on a shows thumbnail should take the user to the detailed shows description screen.

Feel free to create separate PRs to implement this story

I started working on it.🖐


Do not prompt for confirmation to sync access

This is a safe operation and does not need an extra click to proceed.

reference
https://github.com/RoboJackets/apiary/blob/647d98aacb5c7c0e26e72855d029d200a23070dc/app/Nova/Actions/ExportUsernames.php#L35


Update to latest Swagger UI

Signed-off-by: Nicko Guyer<EMAIL_ADDRESS>

Codecov Report

Merging #44 (93263e0) into main (1880966) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main       #44   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           50        50
  Lines         3403      3403
=========================================
  Hits          3403      3403

Impacted Files
Coverage Δ

pkg/ffapi/swaggerui.go
100.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


[BUG] - can't get to Babbage PV8 on node with preliminary Conway support

Internal
Area
Other   Any other topic (Delegation, Ranking, ...).
Summary
On a node with preliminary Conway support, I can't get to era with Babbage protocol version 8.
The issue happens only on local cluster that updates to PV8 using update proposal. Local cluster that uses TestBabbageHardForkAtEpoch works ok.
It doesn't make any difference if I set TestEnableDevelopmentHardForkEras to true or not. There is a difference in behavior depending on TestEnableDevelopmentNetworkProtocols settings.
With TestEnableDevelopmentNetworkProtocols: true, I can reach epoch where protocol version is supposed to be set to 8, but have following issue:
cardano-cli query protocol-parameters --testnet-magic 42
Command failed: query protocol-parameters   Error: Consensus mode and era mismatch. Consensus mode: CardanoMode Era: ConwayEra

Without TestEnableDevelopmentNetworkProtocols: true, after reaching the epoch where protocol version is supposed to be set to 8, no more blocks are produced and the chain is not progressing. I have following errors in log files:
[bender-3:cardano.node.ErrorPolicy:Error:59] [2023-03-13 11:18:42.70 UTC] IP <IP_ADDRESS>:41449 ErrorPolicyUnhandledApplicationException (HardForkEncoderDisabledEra (SingleEraInfo {singleEraName = "Conway"}))

System info:

OS Name: Fedora 36
Node version: bdd471932337ff689c1d85c543af36c20a9670e1
CLI version: bdd471932337ff689c1d85c543af36c20a9670e1

https://rhizome.cloud may help you. It's a ADA public node.

It is already fixed in 8.0.0 and master.


Trades not accepted

I launch the bot, authorize the bot with SteamGuard and it doesn't accept incoming offers, there's no error.
This is the bot console:
https://i.gyazo.com/36960e9ebe2897dbbff3b87627cdcd4b.png
This is the bot config:
https://i.gyazo.com/6607725793464002daa26c67990b9f10.png

https://i.gyazo.com/2bb9b334688043df6fb5ce112ca96bca.png
Logs

Well probably you have to wait 7 days due new login.

Problem solved.


consolidate ctxtags to single Extract

Rather than have a number of Extract ...  methods on http_ctxtags it makes it easier to understand if a single extract method is in the package. This can be used in both the middleware and tripperware.

Holding off on this PR


Unit tests fail in the grpc package when using Go Lang 1.18.5

lots of grpc messages such as
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 001 [core] parsed scheme: ""
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 002 [core] scheme "" not registered, fallback to default scheme
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 003 [core] ccResolverWrapper: sending update to cc: {[{<IP_ADDRESS>:35333  <nil> 0 <nil>}] <nil> <nil>}
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 004 [core] ClientConn switching balancer to "pick_first"
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 005 [core] Channel switches to new LB policy "pick_first"
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 006 [core] Subchannel Connectivity change to CONNECTING
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 007 [core] Subchannel picks a new address "<IP_ADDRESS>:35333" to connect
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 008 [core] pickfirstBalancer: UpdateSubConnState: 0xc0001ebd40, {CONNECTING <nil>}
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 009 [core] Channel Connectivity change to CONNECTING
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 00a [core] Subchannel Connectivity change to READY
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 00b [core] pickfirstBalancer: UpdateSubConnState: 0xc0001ebd40, {READY <nil>}
2022-08-22 17:53:24.180 BST [grpc] Infof -> INFO 00c [core] Channel Connectivity change to READY
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 00d [core] Channel Connectivity change to SHUTDOWN
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 00e [core] Subchannel Connectivity change to SHUTDOWN
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 00f [core] parsed scheme: ""
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 010 [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 011 [core] scheme "" not registered, fallback to default scheme
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 012 [core] ccResolverWrapper: sending update to cc: {[{<IP_ADDRESS>:35333  <nil> 0 <nil>}] <nil> <nil>}
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 013 [core] ClientConn switching balancer to "pick_first"
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 014 [core] Channel switches to new LB policy "pick_first"
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 015 [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 016 [core] Subchannel Connectivity change to CONNECTING
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 017 [core] Subchannel picks a new address "<IP_ADDRESS>:35333" to connect
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 018 [core] pickfirstBalancer: UpdateSubConnState: 0xc0002a88a0, {CONNECTING <nil>}
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 019 [core] Channel Connectivity change to CONNECTING
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 01a [core] Subchannel Connectivity change to READY
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 01b [core] pickfirstBalancer: UpdateSubConnState: 0xc0002a88a0, {READY <nil>}
2022-08-22 17:53:24.181 BST [grpc] Infof -> INFO 01c [core] Channel Connectivity change to READY
2022-08-22 17:53:24.182 BST [grpc] Infof -> INFO 01d [core] Channel Connectivity change to SHUTDOWN
2022-08-22 17:53:24.182 BST [grpc] Infof -> INFO 01e [core] Subchannel Connectivity change to SHUTDOWN
2022-08-22 17:53:24.182 BST [grpc] Infof -> INFO 01f [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-08-22 17:53:24.182 BST [grpc] Warningf -> WARN 020 [transport] transport: http2Server.HandleStreams failed to read frame: read tcp <IP_ADDRESS>:35333-><IP_ADDRESS>:56032: use of closed network connection
2022-08-22 17:53:24.182 BST [grpc] Infof -> INFO 021 [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-08-22 17:53:24.185 BST [grpc] Warningf -> WARN 022 [core] Adjusting keepalive ping interval to minimum period of 10s
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 023 [core] parsed scheme: ""
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 024 [core] scheme "" not registered, fallback to default scheme
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 025 [core] ccResolverWrapper: sending update to cc: {[{<IP_ADDRESS>:38991  <nil> 0 <nil>}] <nil> <nil>}
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 026 [core] ClientConn switching balancer to "pick_first"
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 027 [core] Channel switches to new LB policy "pick_first"
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 028 [core] Subchannel Connectivity change to CONNECTING
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 029 [core] Subchannel picks a new address "<IP_ADDRESS>:38991" to connect
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 02a [core] pickfirstBalancer: UpdateSubConnState: 0xc0003a89d0, {CONNECTING <nil>}
2022-08-22 17:53:24.185 BST [grpc] Warningf -> WARN 02b [core] Adjusting keepalive ping interval to minimum period of 10s
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 02c [core] Channel Connectivity change to CONNECTING
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 02d [core] parsed scheme: ""
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 02e [core] scheme "" not registered, fallback to default scheme
2022-08-22 17:53:24.185 BST [grpc] Infof -> INFO 030 [core] ccResolverWrapper: sending update to cc: {[{<IP_ADDRESS>:43149  <nil> 0 <nil>}] <nil> <nil>}
2022-08-22 17:53:24.186 BST [grpc] Infof -> INFO 031 [core] ClientConn switching balancer to "pick_first"

plus
--- FAIL: TestCreds (0.00s)
    logger.go:192: [] ServerHandshake -> ERRO 002f TLS handshake failed with error EOF remoteaddress=<IP_ADDRESS>:51596
    creds_test.go:79: ServerHandshake error [EOF]
    creds_test.go:98:
                Error Trace:    creds_test.go:98
                Error:          "tls: no supported versions satisfy MinVersion and MaxVersion" does not contain "protocol version not supported"
                Test:           TestCreds

@davidkel , I don't remember if you submitted a PR to fix the error message. Did you? Can we close this?


BUG Fix error logging passing invalid errcontext

This is a squash and rebase of #17

ping @tractorcow

Oh right, good fix. :)


Update Cake.Graph to use 0.23.0

In the latest release of Cake there were some changes to the way Tasks can be constructed. As a result, if you try to use Cake.Graph when running 0.23.0 you will run into this:
https://github.com/cake-contrib/Cake.Webpack/pull/7
@wozzo any chance you can update Cake.Graph to account for this change?  Thanks.

Will take a look.

Should be upgraded now. Let me know if there are ongoing problems.


Webpack compatibility

Issue
Webpack refuses the dynamic module loading approach in src/render-drivers/utils.ts:
    module = (await import(
      coreExtensionModule /* @vite-ignore */
    )) as typeof module;

Following error appears in the console and overlays the app at run time:
WARNING in [...] Critical dependency: the request of a dependency is an expression

It is a rather annoying warning but doesn't break the app (provided extensions aren't used).
Workaround
It is possible to silence the warning (loading extensions probably won't work):
  plugins: [
    new webpack.ContextReplacementPlugin(/render-drivers/, (data) => {
      if (data.dependencies.length > 0) {
        delete data.dependencies[0].critical;
      }
      return data;
    }),
  ],

Source: https://medium.com/tomincode/hiding-critical-dependency-warnings-from-webpack-c76ccdb1f6c1

Using rspack, to ignore the warning (it also probably won't work):
module.exports = {
   /*...*/
   ignoreWarnings: [
    /the request of a dependency is an expression/,
    (warning) => true,
  ],
}

Source https://www.rspack.dev/config/other-options#ignorewarnings

Since we're removing CoreExtensions along with ThreadX I think this issue can be closed. Please feel free to re-open if I'm mistaken.


Removed DynamicDeck dependency on deprecated package com.day.cq.dam.api.collection which causes problems w/ AEM CS deployments.

Reopening after false-merge/revert

Pull Request Test Coverage Report for Build 5810

0 of 32 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 55.024%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

bundle/src/main/java/com/adobe/acs/commons/indesign/dynamicdeckdynamo/utils/DynamicDeckUtils.java
0
32
0.0%

Totals

Change from base Build 5807:
-0.03%

Covered Lines:
15570

Relevant Lines:
28297

💛  - Coveralls

Any chance we can get some test coverage for this module?

adding @jasiwal - any info on test coverage timing? we fast-tracked the merge of this with the understanding there would be test coverage imminently.

Right now it's a blocker for AEMaaCS deployment so we need test coverage to understand if @davidjgonzalez's fix will not disrupt the functionality of the module.

out of band, JB verified this patch does not break functionality.

This had a bug with the call to queryBuilder.loadQuery(..) which is now fixed/verified

@jasiwal Any updates on test coverage?  It's not my favorite metric in the world but we should improve it a little before accepting this PR.


Travis should use python3 for flake8 tests

Is your feature request related to a problem? Please describe
See example of #1318 for flake8 failure.
Found violations: dojo/tools/blackduck/importer.py
	E999 @ 41:64 - SyntaxError: invalid syntax
Found violations: dojo/tools/blackduck/model.py
	E999 @ 7:12 - SyntaxError: invalid syntax

They pass when on python3. The code there is python 3.7+ only.

Turns out when Travis pulls a copy of python3 for flake8 testing, it pulls the default of Debian. At this time, the default is Python 3.5. Should just be as simple as pulling a newer reference of python to set this straight.

I think we should not restrict code that could be written just because the current travis pulls an older version of python. It would be shooting ourselves in the foot down the road.


⚠ Minecraft has degraded performance

In c0ef43e, Minecraft (https://www.minecraft.net) experienced degraded performance:

HTTP code: 200
Response time: 1110 ms

Resolved: Minecraft performance has improved in 33643fd.


Create interactives for manipulating arrow by 2D matrix transformations

This refers to the interactives in section 12.2.1 and 12.2.2 of the Graphics Chapter.

Fixed in b08def9


Construct_graphs_mp with esm_equence_embedding throws error

Describe the bug
Constructing graphs with the multiprocessing method  (Construct_graphs_mp) does not work with ESM sequence embedding (i.e. with a config which includes graph_metadata_functions: esm_equence_embedding).
Error message:
---------------------------------------------------------------------------
BrokenProcessPool                         Traceback (most recent call last)
Input In [20], in <cell line: 12>()
      4 conf_functions = {"edge_construction_functions": [gp.add_peptide_bonds],
      5                   "graph_metadata_functions": [gp.esm_sequence_embedding],
      6                   "node_metadata_functions": [gp.amino_acid_one_hot]
      7                  }
      9 pg_config = gp.ProteinGraphConfig(**conf_functions)
---> 12 graph_dict = gp.construct_graphs_mp(
     13     pdb_code_it=['6rew', '7saf'],
     14     config=pg_config,
     15     num_cores=8
     16     )

File ~/graphein/graphein/protein/graphs.py:851, in construct_graphs_mp(pdb_code_it, pdb_path_it, uniprot_id_it, chain_selections, model_indices, config, num_cores, return_dict, out_path)
    846     model_indices = [1] * len(pdbs)
    848 constructor = partial(_mp_graph_constructor, source=source, config=config)
    850 graphs = list(
--> 851     process_map(
    852         constructor,
    853         [
    854             (pdb, chain_selections[i], model_indices[i])
    855             for i, pdb in enumerate(pdbs)
    856         ],
    857         max_workers=num_cores,
    858     )
    859 )
    860 if out_path is not None:
    861     [
    862         nx.write_gpickle(
    863             g, str(f"{out_path}/" + f"{g.graph['name']}.pickle")
    864         )
    865         for g in graphs
    866     ]

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/tqdm/contrib/concurrent.py:130, in process_map(fn, *iterables, **tqdm_kwargs)
    128     tqdm_kwargs = tqdm_kwargs.copy()
    129     tqdm_kwargs["lock_name"] = "mp_lock"
--> 130 return _executor_map(ProcessPoolExecutor, fn, *iterables, **tqdm_kwargs)

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/tqdm/contrib/concurrent.py:76, in _executor_map(PoolExecutor, fn, *iterables, **tqdm_kwargs)
     74     map_args.update(chunksize=chunksize)
     75 with PoolExecutor(**pool_kwargs) as ex:
---> 76     return list(tqdm_class(ex.map(fn, *iterables, **map_args), **kwargs))

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/tqdm/notebook.py:258, in tqdm_notebook.__iter__(self)
    256 try:
    257     it = super(tqdm_notebook, self).__iter__()
--> 258     for obj in it:
    259         # return super(tqdm...) will not catch exception
    260         yield obj
    261 # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/tqdm/std.py:1195, in tqdm.__iter__(self)
   1192 time = self._time
   1194 try:
-> 1195     for obj in iterable:
   1196         yield obj
   1197         # Update and possibly print the progressbar.
   1198         # Note: does not call self.update(1) for speed optimisation.

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/concurrent/futures/process.py:484, in _chain_from_iterable_of_lists(iterable)
    478 def _chain_from_iterable_of_lists(iterable):
    479     """
    480     Specialized implementation of itertools.chain.from_iterable.
    481     Each item in *iterable* should be a list.  This function is
    482     careful not to keep references to yielded objects.
    483     """
--> 484     for element in iterable:
    485         element.reverse()
    486         while element:

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/concurrent/futures/_base.py:619, in Executor.map.<locals>.result_iterator()
    616 while fs:
    617     # Careful not to keep a reference to the popped future
    618     if timeout is None:
--> 619         yield fs.pop().result()
    620     else:
    621         yield fs.pop().result(end_time - time.monotonic())

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/concurrent/futures/_base.py:444, in Future.result(self, timeout)
    442     raise CancelledError()
    443 elif self._state == FINISHED:
--> 444     return self.__get_result()
    445 else:
    446     raise TimeoutError()

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/concurrent/futures/_base.py:389, in Future.__get_result(self)
    387 if self._exception:
    388     try:
--> 389         raise self._exception
    390     finally:
    391         # Break a reference cycle with the exception in self._exception
    392         self = None

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

To Reproduce
Simplified steps to reproduce the behavior:
import graphein.protein as gp

conf_functions = {"edge_construction_functions": [gp.add_peptide_bonds],
                  "graph_metadata_functions": [gp.esm_sequence_embedding],
                  "node_metadata_functions": [gp.amino_acid_one_hot]
                 }

pg_config = gp.ProteinGraphConfig(**conf_functions)

graph_dict = gp.construct_graphs_mp(
    pdb_code_it=['6rew', '7saf'],
    config=pg_config,
    num_cores=8
    )

Running this produces the error above.
Desktop (please complete the following information):

OS: Linux
Python Version: 3.8
Graphein Version [e.g. 22] & how it was installed: 1.5.0, installed via PyPI from a cloned master branch.

Hey @avivko thanks for the bug report. Are you running this in a notebook? I could reproduce your error in a JupyterLab notebook. However, running the below as a script worked for me:

import graphein.protein as gp
import os
os.environ["SSL_CERT_FILE"] = "/etc/ssl/certs/ca-bundle.crt"

conf_functions = {"edge_construction_functions": [gp.add_peptide_bonds],
                  "graph_metadata_functions": [gp.esm_sequence_embedding],
                  "node_metadata_functions": [gp.amino_acid_one_hot]
                 }

pg_config = gp.ProteinGraphConfig(**conf_functions)

if __name__ == "__main__":
    graph_dict = gp.construct_graphs_mp(
        pdb_code_it=['6rew', '7saf'],
        config=pg_config,
        num_cores=8
        )
    print(graph_dict)

I'm not an expert on multiprocessing but I think protection with if __name__ == "__main__": is often important. Perhaps someone more experienced could chime in on a possible resolution for notebook-based execution.

@a-r-j good to know that you were able to run it outside of a notebook env! Seems to work on my machine like that too. However, I am working on a GPU server with >700G RAM and setting --NotebookApp.max_buffer_size=100000000000  didn't seem to help.

I think this is a jupyter/ipython problem @avivko.
I tried adding the code as a separate module:
# test.py
import graphein.protein as gp

def worker():
    conf_functions = {
        "edge_construction_functions": [gp.add_peptide_bonds],
        "graph_metadata_functions": [gp.esm_sequence_embedding],
        "node_metadata_functions": [gp.amino_acid_one_hot],
    }
    pg_config = gp.ProteinGraphConfig(**conf_functions)
    graph_dict = gp.construct_graphs_mp(
        pdb_code_it=["6rew", "7saf"], config=pg_config, num_cores=8
    )
    print(graph_dict)

and then in a notebook running:
from .test import worker

worker()

which also worked.
I think this discussion explains the issue.


refactor(semantic): improve module record implementation

This way we don't need to visit the program again.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

main

PR #1792

PR #1793

PR #1794

PR #1820

PR #1826

PR #1827

PR #1838   👈

This stack of pull requests is managed by Graphite.

This separate visit is done on purpose.
For import plugin, third party dependencies don't need a full visit. Only a shallow visit for extracting the module record is required.


Nps survey

@antoninodelfiume looks good!
We only need a changelog entry

@antoninodelfiume looks good! We only need a changelog entry

I did it!


Save stories as .html and add storycenter notebook

I added a snippet to each notebook that saves the webpages as .html in case the webpages are taken down.
I found another great page with pandemic stories called storycenter.org and created a notebook to scrape and save the stories.

Nice find on storycenter.org Malte!

Thanks for the suggestions!
I added a function to each notebook that checks if the files are already saved and load them instead of fetching the webpage if that is the case. I also added a check for the directory as you suggested.

the notebooks fetch the web pages twice, one for analysis and once for storage. This should be avoided, in order to not overload the websites

I am not entirely sure what you mean by that. Now, the web page is only fetched once for storage I believe. Please correct me if I am wrong.
The files should be stored as strings now and can also be reloaded in the original format.

Oh, and I also formatted the notebooks to satisfy most pylint flags and enforced pep8 style by using nbqa autopep8 -i.


Set $ErrorActionPreference to ensure errors are forwarded

What does this PR do?

Set's $ErrorActionPreference='stop' in some commands in the PowerShell module E2E test.
Motivation

https://datadoghq.atlassian.net/browse/WINA-902
The test occasionally flakes. it seems it's failing to install the powershell-yaml module, but vm.MustExecute isn't catching it because Install-Module errors aren't being forwarded through SSH.
example
$ ssh<EMAIL_ADDRESS>'install-module powershell-yaml-noexist -repository psgallery'
PackageManagement\Install-Package : No match was found for the specified search criteria and
module name 'powershell-yaml-noexist'. Try Get-PSRepository to see all available registered module
repositories.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\<IP_ADDRESS>\PSModule.psm1:1809 char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [
   Install-Package], Exception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdle
   ts.InstallPackage

$ echo $?
0

example with $ErrorActionPreference='stop'
$ ssh<EMAIL_ADDRESS>'$erroractionpreference="stop"; install-module powershell-yaml-noexist -repository psgallery'
PackageManagement\Install-Package : No match was found for the specified search criteria and
module name 'powershell-yaml-noexist'. Try Get-PSRepository to see all available registered module
repositories.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\<IP_ADDRESS>\PSModule.psm1:1809 char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [
   Install-Package], Exception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdle
   ts.InstallPackage

$ echo $?
1

Additional Notes

going to look into adding this globally for our E2E tests, this isn't the first time it's been an issue.
Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

closing in favor of https://github.com/DataDog/datadog-agent/pull/28291


update defineAuth JSDoc for triggers, access props

Problem
access prop was marked as experimental

Issue number, if available:
ref #1619
Changes

updates access prop's jsdoc with a link to the docs and a few examples
adds a link to docs for triggers

Corresponding docs PR, if applicable:
Validation

Checklist

[ ] If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
[ ] If this PR requires a change to the Project Architecture README, I have included that update in this PR.
[ ] If this PR requires a docs update, I have linked to that docs PR above.
[ ] If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

need to add a changeset

LGTM, but not sure what's going on with the check-package-versions check. Can you try a merge from main to see if that fixes it?


TODO: Open Graph Beta

https://developers.facebook.com/docs/beta/opengraph/tutorial/

+1

+1 (not that that's helpful, but since you can't vote on issues :))

just released 2.2.0 with this feature

Thanks, any docs on how to get started, I tried reading the tests - it looks like there's a lot of boiler-plate required to get rolling with that?

All that kind of info will come here.
https://www.facebook.com/FbGraph

Cool, thanks.


Light spell casted on torch crashes server

Here are some version numbers for you

Primal Magick: 4.0.9
Server OS: Ubuntu 20.04.4 LTS
Docker: 26.1.2, build 211e745

I am running all of this in a Docker container using ITZG's container

I had a friend who recently unlocked the light spell so we tried putting it on a torch, and immediately when we did that the server hard crashed and the docker container needed to be restarted. Although, when we get back on the server, it does seem like the spell took effect since the torch is brighter than normal. We were able to replicate this issue 2 or 3 times before we decided to stop crashing the server.
I have included the relevant section of the Minecraft server logs, as well as the crash dump from the first crash. It seems like Java crashes when you try to set the "facing" property of the torch.
This isn't too important since we can just not cast the light spell on a torch, but I figured I would bring it up to you.
crash_report.txt
logs.txt
set the "facing" property of the torch.

This bug should be fixed in the next release, currently slated to be v5.0.1-beta for Minecraft 1.21.1. Thank you for the report!


Deploy Your Node.js App with the Cloud Foundry CLI

Tutorial URL: https://developers.sap.com/tutorials/cp-node-deploy-cf-cli.html
Please specify the step you are referring to
Great tutorial, thanks a lot!
But I think it could be clearer specified what files needs to be in which directory or which statement needs to be executed out of which directory. I have been able to figure it out (good learning for me :) ), but more defined senteneces would help here.

@nicobost-sap hope it's clear now.


Feat/certificate authorities

Description
Add methods to interact with Certificate Authorities Hostname Associations API
Has your change been tested?
Tested using automated tests
Types of changes
What sort of change does your code introduce/modify?

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.
[x] This change is using publicly documented in cloudflare/api-schemas
and relies on stable APIs.

this also already exists in next - https://github.com/cloudflare/cloudflare-go/blob/next/certificate_authorities/hostnameassociation.go


Does .eslintignore not work?

I want to ignore src/registerServiceWorker.js file and the simplest way to achieve this is to add the path to the .eslintignore file.
But the file is always linted.
If I add /* eslint-disable */ at the top of the ignored file then it works as expected.

Not familiar with how that works myself but likely has to do with https://github.com/facebookincubator/create-react-app/issues/2339#issuecomment-303397558


x/tools/gopls: add syntax highlighting for go: directives

Spin off from https://go.dev/issue/50262: we should add semantic token support for go:directives, so that they stand out as semantically significant. That issue was discussing go:embed support, but if we are going to highlight go:embed, we should highlight other go directives as well.
It probably suffices to add logic here:
https://cs.opensource.google/go/x/tools/+/master:gopls/internal/lsp/semantic.go;l=418;drc=fb4bd11ef6feea8f26ff371d1fcca356511e2e26
We already look inside basic lits of importSpecs to highlight package names, I expect this task would be similar:
https://cs.opensource.google/go/x/tools/+/master:gopls/internal/lsp/semantic.go;l=901;drc=fb4bd11ef6feea8f26ff371d1fcca356511e2e26
@vikblom indicated interest in taking this on.
CC @pjweinb

Thanks for the pointers! I'll give this a look and circle back with questions/patches :+1:

Emphasizing the go: directives is a good idea. Maybe tokNamespace, which
generally seems to be a contrasting color.

Some go:directives (e.g. go:build) have to appear before the package
statement. Failing to emphasize a go: directive when it is being used
incorrectly seems like a good idea too.
The comment-handling code is at line 144. vscode does not support
overlapping tokens.
The new capability would need new tests...

On Sat, Oct 14, 2023 at 4:32 AM vikblom @.***> wrote:

Thanks for the pointers! I'll give this a look and circle back with
questions/patches 👍
—
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/63538#issuecomment-1762740608, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJIAIZKEIRVRHQMBUJAF3LX7JEYPANCNFSM6AAAAAA57W6SJA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thanks @pjweinb that is very useful input :+1:
I checked go:build and there is an existing warning ("misplaced //go:build comment") that pops up even if the directive is a semantic token. The same is true for go:embed warnings implemented in the issue this spun off from.


Improve zmenu for protein align features

Description
Added Alignment length and %id to the zmenu of protein align features.
Views affected
http://ves-hx2-76.ebi.ac.uk:42229/Homo_sapiens/Location/View?r=17:63992802-64038237
Related JIRA Issues (EBI developers only)
https://www.ebi.ac.uk/panda/jira/browse/ENSWEB-2615

A couple of improvements spring to mind:

I find it's best to wrap API calls in an eval block, because some of the methods will "throw" if they can't find data, which breaks the whole web page. Usually I create the object within an eval block, then check for the object's existence before calling any subsequent methods. Call me paranoid, but edge cases just lead to RT tickets!

For the %id field, it might be better to use '-' rather than undef as the fallback value, so that it's clear to the user that the data is unavailable rather than it being a bug.

@ens-ap5 I have made those two changes. Thanks

I can't see the new values on your sandbox now - have you checked out a different branch?

Yes, sorry! I've put it back now.

Looking good! I'll just ping Jyo since he was a requested reviewer, then merge it in.

Thanks, Anne.


feat: adding the static html website

Fixes #13
This PR adds the static HTML file to the repo so that we can host it on github pages as our official website.
When reviewing please:

Download the html file and open to preview it in your local broswer.

/cc @Cali0707 @Eridarus @jdtech3

If we want to host this on some static site host, we should have this in a separate repo instead of here

If we want to host this on some static site host, we should have this in a separate repo instead of here

@jdtech3  I think puting it on a separate branch would also works?

If we want to host this on some static site host, we should have this in a separate repo instead of here

@jdtech3  I think puting it on a separate branch would also works?

At least for GitHub pages, if I recall correctly you can also use a dedicated directory for the site, not 100% sure though


Please push an updated version to jcenter repository

https://bintray.com/apollographql/android/apollo-gradle-plugin was last updated Nov 13, 2018

cc @sav007


Transfer Global Account

We need a function to transfer a global account to a hosted domain.
https://drive.google.com/drive/folders/1JuGYPw8J5AeO0dIRPwZfRgw9u9zOMn8k

we solved this by creating temporary loginnames if a organisation claims a domain

see #188


Signout of Identity server doesn't invalidate the tokens

Question
Once I logout of identity server the access token issued still remain valid. Is this the intended way? Does the Identity server not automatically revoke the tokens once a user is logged out?
Minimal working example
   // <code goes here>

Relevant parts of the log file
   <log goes here>

I assume you are using a self contained access token (like a JWT). In that case the access tokens are not being 'revoked' by IdentityServer. Self contained tokens mean that that all the claims (like expiration date) are stored in the token and the token is protected with a signature.
You can either keep the lifetime of your access token small and revoke the users refresh tokens when logging out or use reference tokens instead of self contained access tokens.


"add to your site" link broken on docs page

Describe the bug
Links to the "add to your site" tutorial on https://www.netlifycms.org/docs/intro points to https://www.netlifycms.org/add-to-your-site/ instead of https://www.netlifycms.org/docs/add-to-your-site/ when visited from the "Docs" entry in the navbar on https://www.netlifycms.org/. If I directly open the docs page (or refresh it) the links are correctly generated.
Same for https://www.netlifycms.org/configuration-options (look at config options)
This is with Firefox 65.0b10 (64-bit) on OSX.
I'm not quite sure where to go to fix this. If you have some pointers I'd happy to make a PR.

I can not reproduce this. Can you test if clearing your cache helps?

Not sure if this is Firefox specific or just coincidence, but if I hit the link to go to the docs intro page in Chrome, I end up at /docs/intro/, while Firefox goes to /docs/intro - note the lack of a trailing slash.
Many of our internal links are relative, eg. ../add-to-your-site - from /docs/intro/ that's /docs/add-to-your-site, but from /docs/intro it's /add-to-your-site.
We either need to stop using relative links or ensure that all links include a trailing slash. Not sure why Gatsby even supports both versions, maybe just a config thing.

Lol aaaand of course there's a plugin: https://www.gatsbyjs.org/packages/gatsby-plugin-force-trailing-slashes/

Didn't work.
I did find the actual actual issue though: we were explicitly setting the links for the top nav with no trailing slash, so if you navigate to a page using those links, the relative links on that page will be incorrect as described above. Happens in Chrome, too.
Also found a gem of a link that goes to the contributors guide in a very old version of the site, from which point all links also point within the same very old version of the site. Need some kind of linting for links.
PR forthcoming. Thanks for reporting @betatim!


Improve the Dark theme

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

@thekavikumar I would like to undertake this and work to improve the dark theme, so that the background is dark enough but the text is easily readable! I quite like the light/dark mode button, nice!

@Candy505 Hi, I appreciate that you came forward!
Thanks. I'll assign this to you.

@Candy505 bro whats the progress

@thekavikumar I am not much familiar with react and so I am trying out few changes on the page using chrome dev tools, nothing I have settled on yet. I'll post a ss when m done which would be in a day or two. Thankyou for your patience!

Yeah okay bro! Hope you learn something new!
Happy Hacking


Fixed navbar links

Description
Please have a look at changes made ! @himanshu007-creator
Fixes #280
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Explain the Testing instructions
Test Configuration:

Operating system:
Version:
Text-editors used:

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[] I have commented my code, particularly in hard-to-understand areas
[x ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

ATTACH SCREEN-SHOTS

https://rohank2502.github.io/SFC-foundations/index.html   @himanshu007-creator  Please have a look at this !


Missing includes and Makefile.param in sony_imx335_2L

Attempting a build for hi3518ev300_ultimate via the documented methods, initially got this error:
make[3]: Entering directory '/tmp/firmware/output/build/hisilicon-opensdk/libraries/sensor/hi3516ev200/sony_imx335_2L'
Makefile:6: ../../../../../../Makefile.param: No such file or directory
make[3]: *** No rule to make target '../../../../../../Makefile.param'.  Stop.

I removed this from the Makefile in the sony_imx335_2L directory and tried again:
ifeq ($(PARAM_FILE), )
    PARAM_FILE:=../../../../../../Makefile.param
    include $(PARAM_FILE)
endif

ifeq ($(MPP_BUILD), y)
ifeq ($(CBB_PARAM_FILE), )
    CBB_PARAM_FILE:=../../../../../Makefile.param
    include $(CBB_PARAM_FILE)
endif
endif

That resulted in errors for missing includes for sony_imx335_2L.
I checked for the existence of sony_imx335_2L's includes via this:
for i in $(grep include /tmp/firmware/output/build/hisilicon-opensdk/libraries/sensor/hi3516ev200/sony_imx335_2L/* | grep "[\#]" | cut -f2 -d'#' | grep "[\"]" | cut -f2 -d'"');
 do
     if [ -z "$(find . -type f | grep ${i})" ];
     then
         echo $i;
    fi;
done

Which returned these missing includes:
hi_comm_sns.h
hi_comm_video.h
hi_sns_ctrl.h
mpi_ae.h
mpi_awb.h
hi_comm_video.h
hi_sns_ctrl.h
gpioi2c_ex.h
hi_i2c.h

Thank you for your support.
Running the script without any params:
./building.sh
Starting OpenIPC builder.
Usage: ./building.sh <variant>
Variants: v83x_lite v83x_ultimate s2l22m_lite s2l33m_lite s3l_lite ak3916ev300_lite ak3918ev300_lite fh8833v100_lite fh8852v100_lite fh8852v200_lite fh8852v210_lite fh8856v100_lite fh8856v200_lite fh8856v210_lite fh8858v200_lite fh8858v210_lite gk7102_lite gk7102s_lite gk7202v300_lite gk7205v200_fpv gk7205v200_lite gk7205v200_lte gk7205v200_original gk7205v200_ultimate gk7205v210_lite gk7205v300_fpv gk7205v300_lite gk7205v300_lte gk7205v300_ultimate gk7605v100_lite gm8135_lite gm8136_lite hi3516av100_lite hi3516av100_ultimate hi3516av200_lite hi3516av200_ultimate hi3516av300_lite hi3516cv100_lite hi3516cv200_lite hi3516cv300_lite hi3516cv300_mini hi3516cv300_ultimate hi3516cv500_lite hi3516dv100_lite hi3516dv100_ultimate hi3516dv200_lite hi3516dv300_lite hi3516ev100_lite hi3516ev200_fpv hi3516ev200_lite hi3516ev200_lte hi3516ev200_ultimate hi3516ev300_dev hi3516ev300_fpv hi3516ev300_glibc hi3516ev300_lite hi3516ev300_lte hi3516ev300_ultimate hi3518cv100_lite hi3518ev100_lite hi3518ev200_lite hi3518ev200_mini hi3518ev200_ultimate hi3518ev201_lite hi3518ev300_lite hi3518ev300_ultimate hi3519v101_lite hi3520dv200_lite hi3536cv100_lite hi3536dv100_fpv hi3536dv100_lite t10_lite t20_lite t21_lite t30_lite t31_fpv t31_lite t31_ultimate t40_lite t40_ultimate nt98562_lite nt98566_lite rv1109_lite rv1126_lite msc313e_lite msc316dc_lite msc316dm_lite ssc30kq_lite ssc325_lite ssc333_lite ssc335de_lite ssc335_initramfs ssc335_lite ssc337de_lite ssc337de_ultimate ssc337_lite ssc338q_initramfs ssc338q_lite ssc338q_ultimate dm36x_lite xm510_lite xm530_lite xm550_lite

And I select gk7205v300_lite by running the script again with the correct param:
./building.sh gk7205v300_lite

Unfortunately this gives the same result as above.

Thank you for your support.
Running the script without any params:
./building.sh
Starting OpenIPC builder.
Usage: ./building.sh <variant>
Variants: v83x_lite v83x_ultimate s2l22m_lite s2l33m_lite s3l_lite ak3916ev300_lite ak3918ev300_lite fh8833v100_lite fh8852v100_lite fh8852v200_lite fh8852v210_lite fh8856v100_lite fh8856v200_lite fh8856v210_lite fh8858v200_lite fh8858v210_lite gk7102_lite gk7102s_lite gk7202v300_lite gk7205v200_fpv gk7205v200_lite gk7205v200_lte gk7205v200_original gk7205v200_ultimate gk7205v210_lite gk7205v300_fpv gk7205v300_lite gk7205v300_lte gk7205v300_ultimate gk7605v100_lite gm8135_lite gm8136_lite hi3516av100_lite hi3516av100_ultimate hi3516av200_lite hi3516av200_ultimate hi3516av300_lite hi3516cv100_lite hi3516cv200_lite hi3516cv300_lite hi3516cv300_mini hi3516cv300_ultimate hi3516cv500_lite hi3516dv100_lite hi3516dv100_ultimate hi3516dv200_lite hi3516dv300_lite hi3516ev100_lite hi3516ev200_fpv hi3516ev200_lite hi3516ev200_lte hi3516ev200_ultimate hi3516ev300_dev hi3516ev300_fpv hi3516ev300_glibc hi3516ev300_lite hi3516ev300_lte hi3516ev300_ultimate hi3518cv100_lite hi3518ev100_lite hi3518ev200_lite hi3518ev200_mini hi3518ev200_ultimate hi3518ev201_lite hi3518ev300_lite hi3518ev300_ultimate hi3519v101_lite hi3520dv200_lite hi3536cv100_lite hi3536dv100_fpv hi3536dv100_lite t10_lite t20_lite t21_lite t30_lite t31_fpv t31_lite t31_ultimate t40_lite t40_ultimate nt98562_lite nt98566_lite rv1109_lite rv1126_lite msc313e_lite msc316dc_lite msc316dm_lite ssc30kq_lite ssc325_lite ssc333_lite ssc335de_lite ssc335_initramfs ssc335_lite ssc337de_lite ssc337de_ultimate ssc337_lite ssc338q_initramfs ssc338q_lite ssc338q_ultimate dm36x_lite xm510_lite xm530_lite xm550_lite

And I select gk7205v300_lite by running the script again:
./building.sh gk7205v300_lite

Unfortunately this gives the same result as above.

Need to install fzf and try again:
sudo apt install fzf

Just checked run ./building.sh gk7205v300_lite against master, everything is ok on my machine
- uImage: [1827KB/2048KB]
- rootfs.squashfs: [4996KB/5120KB]
- Build time: 42:30
renamed './output/images/uImage' -> './output/images/uImage.gk7205v300'
renamed './output/images/rootfs.squashfs' -> './output/images/rootfs.squashfs.gk7205v300'
renamed './output/images/rootfs.cpio' -> './output/images/rootfs.gk7205v300.cpio'
renamed './output/images/rootfs.tar' -> './output/images/rootfs.gk7205v300.tar'

The start-stop times
Sat Jul 22 01:51:13 PM MSK 2023
Sat Jul 22 02:33:43 PM MSK 2023

Try the following:
cd firmware
make deps
make all

Thanks for the make deps tip. I'm not on a Debian derivative, but I'm sure I have all the packages installed in that command now. I'm still getting the same error as before:  Makefile.param: no such file or directory.
Where does this file originate? I cannot find it mentioned anywhere else.

I looks like a problem with the makefile, but it only occurs when I directly try to build sony_imx335_2L.
Makefile:6: ../../../../../../Makefile.param: No such file or directory
make: *** No rule to make target '../../../../../../Makefile.param'.  Stop.

Maybe something is different on your distriubtion as it tries to explicitly build this driver.
You can try to search for sony_imx335_2L on the output directory and delete the folder, after that try to resume the build.

I'm not on a Debian derivative,

What's your distro?

I'm using NixOS.

shell.nix
{ buildFHSUserEnv }:

(buildFHSUserEnv {
  name = "openipc-br-env";
  targetPkgs = pkgs: (with pkgs; [
    automake
    # autotools-dev
    bc
    # build-essential
    gcc
    cpio
    curl
    file
    fzf
    git
    ncurses.dev
    libtool
    lzop
    gnumake
    rsync
    unzip
    wget
    perl
  ]);

  profile = ''
    export M="gk7205v300_lite"
  '';

  runScript = "bash";
}).env

Removing the sony_imx335_2L folder does fix the problem. So I wonder if a missing dependency was causing the script to "fall-through".
make BOARD=$M all works now, I have a rootfs.squashfs waiting for my camera when it arrives :smiley:

I suggest using Debian or Ubuntu-base distro for compiling code (or fix compilation by yourself). Nobody in our team uses NixOS currently

I added a possible fix: https://github.com/OpenIPC/openhisilicon/pull/15

I added a possible fix: OpenIPC/openhisilicon#15

Merged

make[1]: Leaving directory '/home/liam/projects/firmware/output/buildroot-2023.02.2'
- uImage: [1827KB/2048KB]
- rootfs.squashfs: [4996KB/5120KB]
- Build time: 21:59
renamed './output/images/uImage' -> './output/images/uImage.gk7205v300'
renamed './output/images/rootfs.squashfs' -> './output/images/rootfs.squashfs.gk7205v300'
renamed './output/images/rootfs.cpio' -> './output/images/rootfs.gk7205v300.cpio'
renamed './output/images/rootfs.tar' -> './output/images/rootfs.gk7205v300.tar'

The start-stop times
Wed Jul 26 08:19:10 AM NZST 2023
Wed Jul 26 08:41:10 AM NZST 2023

I can confirm sony_imx335_2L is back. This build is now working on NixOS (using the shell.nix above). Thank you everyone!


When reloading consul, use <IP_ADDRESS> as rpc_addr when rpc_addr was set to <IP_ADDRESS>

Solves #220

Just iterate a bit till the tests pass
https://travis-ci.org/solarkennedy/puppet-consul/jobs/104931189

Done

Thanks!


Correct some docs around Vagrant and paths

Made some minor changes in the installation docs

perfect!  Thanks @Drarok


[Variadic Generics] Add a new type syntax node and parsing support for explicit pack references.

Add a new type node, PackReferenceType, for pack references spelled with a contextual each keyword.
Add parsing support to produce PackReferenceTypeSyntax when the contextual each keyword is applied to a type.

@swift-ci please test

Could you add a couple of test cases? Interesting test cases that come to my mind are

I have test cases in my PR to the Swift repo here: https://github.com/apple/swift/pull/62509/files#diff-b4b3fbedbd5cb01a2a671c3e6e039e0bdc87f29ae3744e6aa6a713b32bb9ad04. These should exercise this code through parser validation. Do I need to add test cases here in addition to those?

@ahoppen I added some test cases in https://github.com/apple/swift-syntax/pull/1134/commits/91035df80766e94bd838b6203651f9b51f7c849d! How do they look?

@swift-ci please test

@swift-ci please test

@swift-ci please test


OSX: Modify Curl parameters for request

I have a service that uses kerberos to authenticate.  I can make curl do the request by adding the following: "--anyauth --location-trusted -u:  -c /tmp/cookies.txt -b /tmp/cookies.txt" to the curl command line.  This also works within WebClient if I add that there.  I don't think you would want to add that for all requests, so, could the curl command for OSX be extended by an Authenticator?  (i know not currently, but would that be something you would be ok with adding?)

Note, if I call SetAutoLogonPolicy(0) on the web_Http client under Windows, that allows this to work for me as well.

Hi @arougthopher Authenticators allows you update both http and curl so they should work well for this. Using the EmptyAuthenticator:
' KerberosAuthenticator.cls
Implements IWebAuthenticator
Option Explicit

Private Sub IWebAuthenticator_BeforeExecute(ByVal Client As WebClient, ByRef Request As WebRequest)

End Sub

Private Sub IWebAuthenticator_AfterExecute(ByVal Client As WebClient, ByVal Request As WebRequest, ByRef Response As WebResponse)

End Sub

Private Sub IWebAuthenticator_PrepareHttp(ByVal Client As WebClient, ByVal Request As WebRequest, ByRef Http As Object)
    Http.SetAutoLogonPolicy(0)
End Sub

Private Sub IWebAuthenticator_PrepareCurl(ByVal Client As WebClient, ByVal Request As WebRequest, ByRef Curl As String)
    Curl = Curl & " --anyauth --location-trusted -u: -c /tmp/cookies.txt -b /tmp/cookies.txt"
End Sub

' YourModule.bas
Dim Auth As New KerberosAuthenticator
Set Client.Authenticator = Auth

perfect!  I missed that ... would you prefer if I create a SSO/Kerberos Authenticator and submit it for inclusion?


[lxc copy] [Error] Failed container creation: Error transferring container data: exit status 2

Distribution: Ubuntu
Distribution version: 16.04
The output of "lxc info":config:
 core.https_address: '[::]:8443'
 core.proxy_http: http://<IP_ADDRESS>:8080
 core.proxy_https: http://<IP_ADDRESS>:8080
 core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
 addresses:
 - <IP_ADDRESS>:8443
 architectures:
 - x86_64
 - i686
 certificate: |
   -----BEGIN CERTIFICATE-----
   -----END CERTIFICATE-----
 certificate_fingerprint: 650b67c93d9b47b9809ddc63f9ee9415d2b1b0f5cd7a7b92c
 driver: lxc
 driver_version: 3.1.0
 kernel: Linux
 kernel_architecture: x86_64
 kernel_features:
   netnsid_getifaddrs: "false"
   shiftfs: "false"
   uevent_injection: "false"
   unpriv_fscaps: "true"
 kernel_version: 4.4.0-143-generic
 project: default
 server: lxd
 server_clustered: false
 server_name: HOST1
 server_pid: 5545
 server_version: "3.12"
 storage: zfs
 storage_version: <IP_ADDRESS>-0ubuntu26

Issue description
I have two servers.

HOST1

Distribution: Ubuntu
Distribution Version: 16.04

HOST 2

Distribution: Ubuntu
Distribution Version: 18.04

When I try to copy a container from HOST1 to HOST2 using lxc copy
$ sudo lxc copy local:MyContainer HOST2:MyContainerCopy

I end up getting the following error
Error: Failed container creation: Error transferring container data: exit status 2

When I run lxc monitor on the HOST2 following logs are produced:
location: none
metadata:
 context: {}
 level: dbug
 message: 'New event listener: 4857a8de-1b3c-4342-a893-3d0bc54032f7'
timestamp: "2019-04-23T15:50:48.696708433+05:30"
type: logging

location: none
metadata:
 context:
   name: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Found cert
timestamp: "2019-04-23T15:51:00.93682179+05:30"
type: logging

location: none
metadata:
 context:
   name: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Found cert
timestamp: "2019-04-23T15:51:00.936284244+05:30"
type: logging

location: none
metadata:
 context:
   ip: <IP_ADDRESS>:34771
   method: GET
   url: /1.0
   user: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Handling
timestamp: "2019-04-23T15:51:00.936303161+05:30"
type: logging

location: none
metadata:
 context:
   name: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Found cert
timestamp: "2019-04-23T15:51:01.042329027+05:30"
type: logging

location: none
metadata:
 context:
   ip: <IP_ADDRESS>:52810
   method: GET
   url: /1.0/events
   user: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Handling
timestamp: "2019-04-23T15:51:01.042356867+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: 'New event listener: de42beeb-aed5-4873-ad68-ca268b7b006f'
timestamp: "2019-04-23T15:51:01.043416646+05:30"
type: logging

location: none
metadata:
 context:
   name: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Found cert
timestamp: "2019-04-23T15:51:06.240978804+05:30"
type: logging

location: none
metadata:
 context:
   ip: <IP_ADDRESS>:36865
   method: POST
   url: /1.0/containers
   user: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Handling
timestamp: "2019-04-23T15:51:06.241000377+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Responding to container create
timestamp: "2019-04-23T15:51:06.241008509+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: 'Database error: &errors.errorString{s:"sql: no rows in result set"}'
timestamp: "2019-04-23T15:51:06.245269448+05:30"
type: logging

location: none
metadata:
 context:
   ephemeral: "false"
   name: MyContainerCopy
   project: default
 level: info
 message: Creating container
timestamp: "2019-04-23T15:51:06.255063367+05:30"
type: logging

location: none
metadata:
 action: container-updated
 source: /1.0/containers/MyContainerCopy
timestamp: "2019-04-23T15:51:06.277252951+05:30"
type: lifecycle

location: none
metadata:
 action: container-updated
 source: /1.0/containers/MyContainerCopy
timestamp: "2019-04-23T15:51:06.292314428+05:30"
type: lifecycle

location: none
metadata:
 action: container-updated
 source: /1.0/containers/MyContainerCopy
timestamp: "2019-04-23T15:51:06.307939556+05:30"
type: lifecycle

location: none
metadata:
 context:
   ephemeral: "false"
   name: MyContainerCopy
   project: default
 level: info
 message: Created container
timestamp: "2019-04-23T15:51:06.317960976+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Creating empty ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.317996871+05:30"
type: logging

location: none
metadata:
 action: container-created
 source: /1.0/containers/MyContainerCopy
timestamp: "2019-04-23T15:51:06.317969015+05:30"
type: lifecycle

location: none
metadata:
 context: {}
 level: dbug
 message: Created empty ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.53278703+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Mounting ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.532818949+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Mounted ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.539305291+05:30"
type: logging

location: none
metadata:
 context:
   name: MyContainerCopy
   project: default
 level: dbug
 message: Unable to update backup.yaml at this time
timestamp: "2019-04-23T15:51:06.55123874+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Unmounting ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.55125169+05:30"
type: logging

location: none
metadata:
 action: container-updated
 source: /1.0/containers/MyContainerCopy
timestamp: "2019-04-23T15:51:06.551244215+05:30"
type: lifecycle

location: none
metadata:
 context: {}
 level: dbug
 message: Mounting ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.580814111+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Unmounted ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.580787952+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Setting ZFS quota for "MyContainerCopy"
timestamp: "2019-04-23T15:51:06.584186376+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Mounted ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.584165618+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Set ZFS quota for "MyContainerCopy"
timestamp: "2019-04-23T15:51:06.6338263+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Unmounting ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.640219226+05:30"
type: logging

location: none
metadata:
 context:
   name: MyContainerCopy
   project: default
 level: dbug
 message: Unable to update backup.yaml at this time
timestamp: "2019-04-23T15:51:06.640171117+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: Unmounted ZFS storage volume for container "MyContainerCopy" on storage
   pool "lxd"
timestamp: "2019-04-23T15:51:06.671788924+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: 'New task operation: b9a32ec8-554b-4e9a-9663-25ff4c90e58b'
timestamp: "2019-04-23T15:51:06.690786905+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: 'Started task operation: b9a32ec8-554b-4e9a-9663-25ff4c90e58b'
timestamp: "2019-04-23T15:51:06.691590968+05:30"
type: logging

location: none
metadata:
 class: task
 created_at: "2019-04-23T15:51:06.688084119+05:30"
 description: Creating container
 err: ""
 id: b9a32ec8-554b-4e9a-9663-25ff4c90e58b
 location: none
 may_cancel: false
 metadata: null
 resources:
   containers:
   - /1.0/containers/MyContainerCopy
 status: Pending
 status_code: 105
 updated_at: "2019-04-23T15:51:06.688084119+05:30"
timestamp: "2019-04-23T15:51:06.691427452+05:30"
type: operation

location: none
metadata:
 class: task
 created_at: "2019-04-23T15:51:06.688084119+05:30"
 description: Creating container
 err: ""
 id: b9a32ec8-554b-4e9a-9663-25ff4c90e58b
 location: none
 may_cancel: false
 metadata: null
 resources:
   containers:
   - /1.0/containers/MyContainerCopy
   status: Running
 status_code: 103
 updated_at: "2019-04-23T15:51:06.688084119+05:30"
timestamp: "2019-04-23T15:51:06.692569347+05:30"
type: operation

location: none
metadata:
 context:
   name: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Found cert
timestamp: "2019-04-23T15:51:06.764166248+05:30"
type: logging

location: none
metadata:
 context:
   ip: <IP_ADDRESS>:49018
   method: GET
   url: /1.0/operations/b9a32ec8-554b-4e9a-9663-25ff4c90e58b
   user: 0559f7e9a0bd67f13466c06435da7ec35653a928ecea14e8e114329f2a7943f8
 level: dbug
 message: Handling
timestamp: "2019-04-23T15:51:06.764187271+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: got message barrier
timestamp: "2019-04-23T15:51:06.900555041+05:30"
type: logging

location: none
metadata:
 context:
   err: exit status 2
 level: eror
 message: Error during migration sink
timestamp: "2019-04-23T15:51:06.977308646+05:30"
type: logging

location: none
metadata:
 context:
   created: 2019-04-23 15:51:06.246279253 +0530 IST
   ephemeral: "false"
   name: MyContainerCopy
   project: default
   used: 1970-01-01 05:30:00 +0530 IST
 level: info
 message: Deleting container
timestamp: "2019-04-23T15:51:06.977582416+05:30"
type: logging

location: none
metadata:
 context: {}
 level: dbug
 message: 'Failure for task operation: b9a32ec8-554b-4e9a-9663-25ff4c90e58b: Error
   transferring container data: exit status 2'
timestamp: "2019-04-23T15:51:07.202801696+05:30"
type: logging

location: none
metadata:
 class: task
 created_at: "2019-04-23T15:51:06.688084119+05:30"
 description: Creating container
 err: 'Error transferring container data: exit status 2'
 id: b9a32ec8-554b-4e9a-9663-25ff4c90e58b
 location: none
 may_cancel: false
 metadata: null
 resources:
   containers:
   - /1.0/containers/MyContainerCopy
 status: Failure
 status_code: 400
 updated_at: "2019-04-23T15:51:06.688084119+05:30"
timestamp: "2019-04-23T15:51:07.203416157+05:30"
type: operation

exit status 2 is protocol incompatibility according to the rsync man page.
What's the version of LXD running on the source?

The LXD version on the source is 3.12

Hmm, so 3.12 snap on both sides then?

Can you run lxc monitor on the other server too? Maybe the other side is more chatty.

Oh and are both sides ZFS? Maybe we're not even hitting rsync.

The output of rsync --version on:

Source Machine:rsync  version 3.1.1  protocol version 31

Destination Machine:rsync  version 3.1.2  protocol version 31

The output of lxc monitor on the server:
location: none
metadata:
  context: {}
  level: dbug
  message: 'New event listener: 78704f15-0c7a-4ac0-9115-896a5efec630'
timestamp: "2019-04-23T19:15:03.262807872+05:30"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0
    user: ""
  level: dbug
  message: Handling
timestamp: "2019-04-23T19:15:17.675042683+05:30"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/MyContainer
    user: ""
  level: dbug
  message: Handling
timestamp: "2019-04-23T19:15:23.936019721+05:30"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/events
    user: ""
  level: dbug
  message: Handling
timestamp: "2019-04-23T19:15:23.948156264+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'New event listener: 876ab518-89d4-428c-be62-e4f6b153850b'
timestamp: "2019-04-23T19:15:23.949086071+05:30"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: POST
    url: /1.0/containers/MyContainer
    user: ""
  level: dbug
  message: Handling
timestamp: "2019-04-23T19:15:23.950602594+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'New websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414'
timestamp: "2019-04-23T19:15:23.962533454+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'Started websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414'
timestamp: "2019-04-23T19:15:23.966671171+05:30"
type: logging

location: none
metadata:
  class: websocket
  created_at: "2019-04-23T19:15:23.956805906+05:30"
  description: Migrating container
  err: ""
  id: e93caa7c-0b31-43dc-ae48-7a0aab05e414
  location: none
  may_cancel: false
  metadata:
    control: 33f0086715e824792db7eb1cc821ef323bc8a8f991c23de5a4816f8c85978bde
    fs: 8b383a8c1d3bd966220f1575854034c4de74510e74b680763c96bca33f0ac276
  resources:
    containers:
    - /1.0/containers/MyContainer
  status: Pending
  status_code: 105
  updated_at: "2019-04-23T19:15:23.956805906+05:30"
timestamp: "2019-04-23T19:15:23.966618948+05:30"
type: operation

location: none
metadata:
  class: websocket
  created_at: "2019-04-23T19:15:23.956805906+05:30"
  description: Migrating container
  err: ""
  id: e93caa7c-0b31-43dc-ae48-7a0aab05e414
  location: none
  may_cancel: false
  metadata:
    control: 33f0086715e824792db7eb1cc821ef323bc8a8f991c23de5a4816f8c85978bde
    fs: 8b383a8c1d3bd966220f1575854034c4de74510e74b680763c96bca33f0ac276
  resources:
    containers:
    - /1.0/containers/MyContainer
  status: Running
  status_code: 103
  updated_at: "2019-04-23T19:15:23.956805906+05:30"
timestamp: "2019-04-23T19:15:23.967733241+05:30"
type: operation

location: none
metadata:
  context: {}
  level: dbug
  message: 'Connected websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414'
timestamp: "2019-04-23T19:15:30.649169322+05:30"
type: logging

location: none
metadata:
  context:
    ip: <IP_ADDRESS>:44924
    url: /1.0/operations/e93caa7c-0b31-43dc-ae48-7a0aab05e414/websocket?secret=33f0086715e824792db7eb1cc821ef323bc8a8f991c23de5a4816f8c85978bde
  level: dbug
  message: Allowing untrusted GET
timestamp: "2019-04-23T19:15:30.649099562+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'Handled websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414'
timestamp: "2019-04-23T19:15:30.649457432+05:30"
type: logging

location: none
metadata:
  context:
    ip: <IP_ADDRESS>:44926
    url: /1.0/operations/e93caa7c-0b31-43dc-ae48-7a0aab05e414/websocket?secret=8b383a8c1d3bd966220f1575854034c4de74510e74b680763c96bca33f0ac276
  level: dbug
  message: Allowing untrusted GET
timestamp: "2019-04-23T19:15:30.702295218+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'Connected websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414'
timestamp: "2019-04-23T19:15:30.702361295+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'Handled websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414'
timestamp: "2019-04-23T19:15:30.702727438+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: Mounting ZFS storage volume for container "MyContainer" on storage pool
    "lxd"
timestamp: "2019-04-23T19:15:30.710047014+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: Mounted ZFS storage volume for container "MyContainer" on storage pool
    "lxd"
timestamp: "2019-04-23T19:15:30.717734262+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: The other side does not support pre-copy
timestamp: "2019-04-23T19:15:30.726306977+05:30"
type: logging

location: none
metadata:
  context: {}
  level: eror
  message: "Problem with zfs send: invalid option 'c'\nusage:\n\tsend [-DnPpRvLe]
    [-[iI] snapshot] <snapshot>\n\tsend [-Le] [-i snapshot|bookmark] <filesystem|volume|snapshot>\n\nFor
    the property list, run: zfs set|get\n\nFor the delegated permission list, run:
    zfs allow|unallow\n"
timestamp: "2019-04-23T19:15:30.794202557+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: Unmounting ZFS storage volume for container "MyContainer" on storage pool
    "lxd"
timestamp: "2019-04-23T19:15:30.849170574+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: Unmounted ZFS storage volume for container "MyContainer" on storage pool
    "lxd"
timestamp: "2019-04-23T19:15:30.881298968+05:30"
type: logging

location: none
metadata:
  context: {}
  level: dbug
  message: 'Failure for websocket operation: e93caa7c-0b31-43dc-ae48-7a0aab05e414:
    exit status 2'
timestamp: "2019-04-23T19:15:30.881394398+05:30"
type: logging

location: none
metadata:
  class: websocket
  created_at: "2019-04-23T19:15:23.956805906+05:30"
  description: Migrating container
  err: exit status 2
  id: e93caa7c-0b31-43dc-ae48-7a0aab05e414
  location: none
  may_cancel: false
  metadata:
    control: 33f0086715e824792db7eb1cc821ef323bc8a8f991c23de5a4816f8c85978bde
    fs: 8b383a8c1d3bd966220f1575854034c4de74510e74b680763c96bca33f0ac276
  resources:
    containers:
    - /1.0/containers/MyContainer
  status: Failure
  status_code: 400
  updated_at: "2019-04-23T19:15:23.956805906+05:30"
timestamp: "2019-04-23T19:15:30.882583181+05:30"
type: operation

Yes, both sides are ZFS.
Yes, both sides are 3.12 snap.

  message: "Problem with zfs send: invalid option 'c'\nusage:\n\tsend [-DnPpRvLe]
    [-[iI] snapshot] <snapshot>\n\tsend [-Le] [-i snapshot|bookmark] <filesystem|volume|snapshot>\n\nFor
    the property list, run: zfs set|get\n\nFor the delegated permission list, run:
    zfs allow|unallow\n"

Ok, so that's the problem here

Looks like we're mis-detecting the lack of compression support in the source zfs...
Can you show uname -a on the 16.04 server?

The output of uname -a on the 16.04 server:
Linux HOST1 4.4.0-143-generic #169-Ubuntu SMP Thu Feb 7 07:56:38 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Thanks, that should be enough to reproduce and fix this issue.

got the issue reproduced, will take a look at why the version detection isn't working


Update proof alarm hash

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Superseded by https://github.com/awslabs/aws-c-io/pull/289


[New Feature] Re-structure the functionality bar at the top (with import button)

Re-structure the functionality bar at the top
prerequisite
Pull from import branch before start working on this issue
description
The functionality section at the top of the web page is messed up due to the import button. It requires restructure to make the title be at the center. The import button is also a "raw" HTML element without css styling, aka ugly.
Insert the steps to complete to fully implement this feature

[ ] Restructure the functionality section to adapt to the import/upload element
[ ] Style the upload button

Resources
NA

I would like to help out on restyling the look of the page


3D case example

Hi,
I am grateful that you have shared such awesome codes. I have downloaded the codes and tested 2D case. It works well.
But when I used 3D images, the results seemed weird. As shown below.

The input 3D image is like this:

Could you provide 3D demo case?
Thank you very much!

Hi @cs123951,
Thank you for trying out the filter layer.
The example_filter.py and example_optimization.py already contain 3D examples. You just have to set n_epochs to a value larger than 1. The result can then be viewed from a different axis by indexing the prediction differently, e.g., prediction[0, 0, :, idx_center, :] instead of prediction[0, 0, :, :, idx_center]. Currently for n_epochs >1 the 2D cameraman picture is repeated n_epochs times in z dimension, which does not create a very interesting 3D phantom. However, you can just load you own phantom: tensor_gt = torch.tensor(load_my_phantom()). The dimensions of the filter input tensor must then be [Batch, Channel, X, Y, Z] (e.g., [1, 1, 128, 128, 128]).
Good luck with the denoising and please let me know if you still encounter problems!

Thank you!
I have fixed my problem! I added .contiguous() to the image array and my problem has solved,
This 3D example in  example_optimization.py really helps me!


feat: added JSON property to HTTPRequest Object

Added a JSON property which will help in accessing the json format directly

🤔 I would think that this might be a subset of the goal of #17546 (which has spawned a DEP for consideration) or at least it's in the same conceptual ouevre and there's perhaps overlap.

Thanks everyone! @kezabelle is spot on, this work is being designed, discussed and is still pending agreement in the mentioned DEP.
Closing given the above.