chore: bdd-spring-1574376051 to 0.0.1

chore: Promote bdd-spring-1574376051 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Rebase with upstream

it is a step in the road to merge the fork.
after we will need to create several PR on the main project (I hope they will be accepted, but they might open some long discussions roll_eyes )
it is the same as #29 but with handresolved conflicts (I still don't know how I got them.... :roll_eyes: )

:scream:


Forward authentication response headers

Do you want to request a feature or report a bug?
Feature
What did you expect to see?
It would be useful to allow headers returned from the forward authentication HTTP endpoint to be added to the final HTTP response. This would allow scenarios where the forward authentication endpoint can refresh authentication credentials and add the new credentials as a header/cookie.
This can be accomplished with a new configuration for http forward auth of "addHeadersToResponse" or similar.
Example:
addHeadersToResponse = ["Set-Cookie"]

This would result in any cookies added from HTTP forward auth response to be returned with the final response to the requesting client.
End-to-End example.

Traefik receives request to entrypoint with forward auth.
Traefik forward request to auth HTTP URI
JWT is used as a cookie for authentication. Forward auth server validates JWT, but expiration of JWT is soon.
Forward auth server requests new JWT for user and uses "Set-Cookie" header to overwrite previous JWT. Returns 20* response telling Traefik to continue routing.
Request is routed to correct frontend/backend and response obtained.
Forward auth middleware adds "Set-Cookie" header to outgoing response with new JWT.
New JWT cookie is used for subsequent requests.

Duplicate of #3515
Fixed by #3521

@ldez I don't think this is a duplicate. That solution allows headers to be added to the request that is sent to the backend. The use case above requires the headers to be added to the outgoing response.

I could really use this.
We use a Rails server for authentication, and use forward authentication to convert the cookie to a JWT for other services. This works great, but doesn't allow the cookie to get updated.
My Kubernetes Ingress configuration looks like this
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/auth-type: forward
    ingress.kubernetes.io/auth-url: http://example-rails.default/authenticate
    ingress.kubernetes.io/auth-response-headers: Authorization
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /api
            backend:
              serviceName: example
              servicePort: 80

A possible workaround is to add Set-Cookie to ingress.kubernetes.io/auth-response-headers and then have the service copy the header to the response itself; but that requires that the service know about the cookie. In this case, I want the service to only know about JWTs.

http://example-rails.default/authenticate

ingress.kubernetes.io/auth-url: http://example-rails.default/authenticate

Is the auth-url is k8s inner service address?

@0x457 using a k8s service url does work, I have a service using this with oauth2-proxy.

@0x457 using a k8s service url does work, I have a service using this with oauth2-proxy.

How to use oauth2-proxy? Are there any examples?

@0x457 using a k8s service url does work, I have a service using this with oauth2-proxy.

Where's some example about  oauth2-proxy?  👍

Hi. I have the same issue here.
I want to use an external bot protection service which return a 200 or 401 according to the user agent / source IP etc.. It Also set a Cookie via the Set-Cookie Header which is not forwarded to the client response but only to the application backend.
Ideally Traefik should merge the cookies set by the application with the cookies set by the auth forward service.

We have the same issue, with our forwardAuth service to manipulate the cookies.
Would like to propose as a draft #4730 (cannot tag because issue is closed)

Hi, I'm gonna try to solve this soon, just wanted to clarify something before writing code:
I've just dived to the problem with overwriting headers. So, what I think is we have 3 cases:

Set-Cookie header is allowed to be set multiple times for different cookie names. rfc6265. So there should not be problem with that.
Set-Cookie sent from forwardAuth service should be overwritten by target service in case they have same cookie names.
For all the other headers values must be concatenated

Is this this the right behaviour?

Any progress on this?
I am using it on Swarm and have the same issue.
For now I am leaving my services without refreshing the cookies.

We're stumbling over the same issue right now, and need to set refreshed access tokens from our forward-auth middleware backend. We're currently running a workaround with sending a 307 redirect to the same URI to set headers which works surprisingly well, but we'd prefer to set the cookie directly, without the forward that will probably break some APIs.
My analysis of the situation so far (enumeration just for easier discussion):

in general, the HTTP RFCs do not have a requirement on "repeated" headers, unless a proxy may not change their order; adding them seems fine
there is a requirement on duplicate set-cookie headers on the same cookie key, which should not be sent following RFC 6265

we could prefer the auth server cookies
we could prefer backend cookies
we could leave the choice to the administrator/middleware configuration

a rather naïve implementation like the one in #6893 will not be able to fulfill RFC 6265, but a proper wrapping response writer would be required to handle this
traefik already has such a wrapper in the header middleware pkg/middlewares/headers/responsewriter.go (which does not (yet) conform to RFC 6265 either, though; at least I did not find any code indicating so)
this cannot be easily reused yet, as the configuration happens in unexported fields/functions (ie. newResponseModifier)

My proposal would be to:

implement RFC6265 compliant behavior for the header plugin, which following discussion in #6893 is a bug
reuse the already implemented and tested responsewriter.go by exporting required fields/methods

should the code be moved to some kind of util package in this case? I did not find an obvious place yet

add configuration to the forward auth plugin to define what cookies should be merged

probably somewhat aligned to how the add header plugin does the configuration
we need to think about, which headers should come first (backend or auth server), which also implies which system will potentially be able to override cookies from the other

If we can agree on this concept, I'd be ready to jump in with the required code.
CC @ldez @jbdoumenjou since you discussed previous approaches on this issue.

As we have the same requirements than the others here: Any news on the issue ? What about the proposal of @JensErat ?
AFAIS all posts here needs the "set-cookie" header, so only merge cookies (instead of headers) would be sufficient.
Using all configured cookies from auth response and the remaining (i.e. without the configured ones) cookies from backend would satisfy:

No duplicate cookie names according to RFC
As the cookies from auth server has higher priority, the backend can't overwrite the security relevant cookies from auth server.

@traefiker don't auto-close

Hello ! I'm also in need of such a feature!
I'm using docker with services accessible on *.maison (ex: grafana.maison), cross-domain cookies don't work, I need my middleware to be able to pass a Set-Cookie to the user (so that the middleware can control the cookies).

Closed by #8924


1.19 UPDATE?

Hi tho, great mod i didnt find another alternative like this one for the 1.19 tho, im still using it on my mod pack 1.16.5
Im wondering if u have a free time to maybe update the mod to the 1.19 please?
Thanks and i will wait for ur answer!

Thanks! if u use twitter or u have a discord server give me the links and i drop u a follow or join

still no 1.19

https://github.com/Toshimichi0915/Thymine/releases/tag/v1.7.2
I have uploaded 1.19 version! Could you reply me if that works?, then I can close this issue.

@MyNameIsF @PolloEnCubitos

Yup it appear to be working fine, i already test it 1 hour without any problem if i found a bug im just going to open a new issue


ERROR

From this day it doesn't work with error
Wed Nov 22 16:46:30 UTC 2023 Traceback (most recent call last): File "/home/ubuntu/.dashcore/sentinel/
bin/sentinel.py", line 230, in <module> main() File "/home/ubuntu/.dashcore/sentinel/bin/sentinel.py",
 line 176, in main attempt_superblock_creation(dashd) File "/home/ubuntu/.dashcore/sentinel/bin/sentin
el.py", line 66, in attempt_superblock_creation proposal_quorum=dashd.governance_quorum(), File "/home
/ubuntu/.dashcore/sentinel/lib/dashd.py", line 97, in governance_quorum hpmn_weight = masternode_count
["detailed"]["hpmn"]["enabled"] Dockerfile LICENSE README.md bin database lib requirements.txt
sentinel-cron.log sentinel.conf share test venv 4 KeyError: 'hpmn'

Do i need Sentinel to work a masternode?

Please provide all details of your system / stack when reporting any issues. This includes, dash core version, OS, OS version, etc. In this case, I think you simply need to read the v20 release notes https://github.com/dashpay/dash/blob/v20.0.0/doc/release-notes.md
Specifically:
https://github.com/dashpay/dash/blob/v20.0.0/doc/release-notes.md#sentinel-deprecation

Thanx! I have just read the Release Notes for 20.0 so excuse me for my inattentiveness. Sentinel is depricated from v20.0.0


Store the Netconf session ID and expose it using driver.SessionID()

This PR adds the ability to store the session ID returned by the server in the initial Hello message.
The driver exposes it to the user via driver.SessionID() uint64.

LGTM, thanks a bunch! 🥳


Remove static bound from type_id

Rendered

One could not do any sensible reflection on lifetimes anyways, right?

That's true of the current state of the world. type_id has a 'static bound so it doesn't have to commit to considering lifetimes or not. This is more reflection on the type, which lives for some lifetime, rather than reflection on the lifetime itself.
What this RFC would let you do is get a type id for whatever you can stick in a Box<T>, whether that's T: 'static or the more general T: 'a. Then you can safely store and fetch data for arbitrary types in a type map that can be bound by some non-static lifetime.

How can you fetch data back? I can store something that points to the stack and then retrieve it with 'static lifetimes instead.

The motivating example would work like this:

Define a marker trait like NonStaticAny, and implement for all T
Define your own NonStaticTypeId, that's basically the same as the one in std, minus the lifetime bound.
Define a type map like HashMap<NonStaticType Id, Box<NonStaticAny + 'a>>
You can then safely coerce an entry keyed by a type id of T into a *const T. And since you have a lifetime 'a that you know the data in the map lives for you can safely coerce the *const T into a &'a T.

I have an implementation of this in a sandbox repo. (Note it's just can we make this work? quality).
This is a fairly specific motivating usecase, but this RFC is an important baby step towards expanding the scope of reflection traits. It'll let us start to experiment with building APIs outside of the standard library and understand the landscape better.

No, you can't, I can put &'a T in and request &'static T and you can't tell the difference.

Note you can probably do this with less unsafe code than my example, because it returns data with a lifetime that isn't tied to the borrow of the map. It's probably not worth focussing on the exact semantics of that though.

@eddyb Ah I see what you mean! Yup that's a problem, I'll take that design back to the drawing board and see what I can do.

There is absolutely no way to use this with Any. If you try, I will break it, and I fear it will end up misused.
You need HKT/ATC, so you can have a T<'a>, require T<'static>: 'static and be invariant over 'a.
If you do any less, I can show you how it's broken. Oh and the best part? You don't need this RFC!
Why? Because for<'a> fn(T<'a>) should work with TypedId already.
Even better: you can encode ATC today, via <T as Trait<'a>>::Result`, so you can do this today.

To be clear: you can't use the TypeId of T<'static>, it has to be something with for<'a> ... T<'a>, otherwise I can store T<'a> = (&'a A, &'static B) and get U<'a> = (&'static A, &'a B) and they're identical if you substitute 'a with 'static. for<'a> is as close as you'll get to ecoding lifetime parametrism.

Hmm I see your point. The idea was to treat the type id as a unique id for the type at runtime, and rely on binding pointers to that data to concrete lifetimes as soon as possible. It's obviously much more subtle than I thought though.

I don't know about type_id itself, but I think it would be fairly straightforward to generalize Any by parameterizing it over a lifetime, with a 'static default for BC -- trait Any<'a = 'static>: 'a. This doesn't let you dynamically distinguish between different lifetimes, it just punts on the question of lifetimes entirely, so that you can at least cast between two types when both of them satisfy the same one (rather than when both are 'static).

What do you mean "satisfy"? How do you prevent me changing a lifetime from the upcast to the downcast?

Instead of
impl Any + 'static {
    fn downcast_ref<T>(&self) -> Option<&T> where T: Any;
}

as we have currently, we would have:
impl<'a> Any<'a> + 'a {
    fn downcast_ref<T>(&self) -> Option<&T> where T: Any<'a> + 'a;
}

Here both the input (the Any) and the output (T) need to be valid for the same lifetime 'a, determined by the caller.
(I think those + 'a parts are redundant, because trait Any<'a>: 'a already requires that, but that's true currently with 'static as well, yet the impl is written impl Any + 'static, which I'm guessing may be due to a typechecker deficiency where it can't figure it out?)

"valid for the same lifetime" doesn't mean much. How do you prevent a 'b (larger than 'a) from turning into 'static without encoding the difference in TypeId somehow?

Hm. We'd need to express that T must live for at most 'a, and we can only express at least?

Again, "valid for some lifetime" doesn't mean anything when you don't know the actual type.
You probably want "ensure all lifetimes are exactly this one" which doesn't have anything to do with validity or outliving of the entire type.
Lile I said, what you want is a lifetime -> type lambda you can get a TypeId from. Then you can inject whatever lifetime you want (I think Any<'a> would be invariant over 'a so at least that could work).

@eddyb in any case, this is the intrinsic; some (such as @sdleffler) would like to at least be able to play with it.

I'd like to see an example usecase that is not unsound (also, can they not use an HRTB encoding?).

@eddyb I don't know if you'd call it "sound", but I've been using TypeIds to encode existential versions of phantom-typed values. Since I don't actually hold the value any longer than its lifetime - I just make use of the TypeId - nothing's going to escape a scope, since the lifetime is "erased" when the type gets TypeId'd. A lack of lifetime restrictions would be quite useful for that, as it would remove the requirement for the user to touch the Keyed trait (I call it AsStatic in my code.) It's especially useful since I want to use types which are allocated with arenas as the phantom types, and those arena-allocated types contain references to other instances of the same type which live in the arena, and have the arena's lifetime. I can safely ensure lifetime correctness elsewhere in the program, and the phantom type still does its job.
On the other hand, I can keep AsStatic and write my own custom derive for it, but I'd rather not if there's a cleaner solution (and I think this one is.)

@sdleffler I am now convinced there is a valid usecase, but we have to put a lot of warnings, because it's unusable for actually having a lifetime-parametrized Any, which most people want.

My usecase, a scoped IoC container, is a bit of a bust since it relied on the lifetime of the container borrow being shorter than the lifetime of the reference it returned. I thought I had the scope of that reference nailed down but, as @eddyb has shown, you can't do that. So it's unsound as it stands. I'll just go back to using ref counting for that.
@sdleffler I think you have a better case, I'll update the RFC to focus on the experiment/build with tools outside std, which is what you seem to be doing with bounded-any, rather than peddling my now-defunct motivation. Anything you can contribute to move it forward would be much appreciated!

@KodrAus bounded-any isn't actually the "real use case" I have to put forward here; the real use case I have looks something like this:
#[derive(Hash, PartialEq, Eq)]
pub struct Name<T>(Binding, PhantomData<T>);

#[derive(Hash, PartialEq, Eq)]
pub struct AnyName(Binding, TypeId);

#[derive(Clone, Hash, PartialEq, Eq)]
pub enum Binding {
    Free(String, u64),
    Bound(u64, u64),
}

This is a locally nameless variable encoding for doing substitution in, say, a lambda calculus. The funny thing about this example is that lifetimes are actually totally irrelevant, and this works fine for me; I use phantom types here to ensure that I am correctly substituting say, an Expr for an Expr (given a Name<Expr> to substitute with.) I also need to be able to abstract over what things are substituted for names so I can do things like collect the set of all free variables in a term. In this particular use case, any lifetimes are actually just artifacts of the arenas I'll be using to store these objects. In order to downcast an AnyName to a Name<T>, the T has to be supplied explicitly, and as such any lifetimes in it are valid in that scope; so even if the AnyName was created from what was originally a Name<Expr<'a>>, I can downcast the AnyName into a Name<Expr<'b>>, and everything will work safely, because 'b came from a context that it's necessarily valid in, and I can now substitute an Expr<'b> for a Name<Expr<'b>> just fine.
That said I'll be coming back to bounded-any sporadically, but I just wanted to clarify it's not my most pressing concern for this feature, especially since it can't be used as a trait object like Any (at the moment I have things separated out into BoundedAnyRef<'a> and other things, which is a mess.)

@sdleffler @eddyb I've replaced the motivation. It's a bit more hand-wavy now because I didn't want to describe the exact details of @sdleffler's usecase, but I can update it if you think that should be in there.
What do you think of the new intrinsic idea, that's specifically stripping concrete lifetimes? I'm not sure what the maintenance burden would be for forking the type id if the main implementation was updated. So it may be a silly idea.

FWIW the current intrinsic strips lifetimes (we can't actually preserve them to that point) and if you copy the libcore definition but remove 'static, it works fine.

@eddyb Right, so the implementation would be really simple, I even have a branch floating around where I did this to test it.
I'm feeling like there's probably not a whole lot to gain adding a separate intrinsic that explicitly removes the 'static bound rather than just removing it from the one we've got.
Does anyone have any more thoughts on this?

I'm fine with changing the intrinsic, but anything stuck on stable needs TypeId to also change, and that is really dangerous if people try to use it for anything Any-like.

This is only proposing a change to the intrinsic that can be consumed by users on nightly who opt in to intrinsics. As far as stable users are concerned TypeId is exactly the same. But your point about attempting to build your own Any-like trait should be mentioned in the docs for type_id.

@rfcbot fcp merge
As this is only proposing an unstable-only relaxation (of core::intrinsics::type_id) that can already be obtained by re-declaring the intrinsic (as we're currently not checking the 'static bound), I'm suggesting we move ahead and accept it, separate from any other endeavors to expand TypedId or Any.

ping @pnkfelix, @BurntSushi (checkboxes)

Nominating for lang team triage, this has been sitting around for too long.

Just curious if I understand the issue with Any being 'static :  You could deal with basic data structures that do not contain non-'static references, but if you contain a non-'static reference then your structure has a lifetime parameter that cannot be seen by simple type reflection, and thus cannot be preserved.  To do reflection correctly, you might restrict Any to particular "lifetime kind", but doing so means you cannot allow any type parameters, as they might hide additional lifetimes.  Is that about right?

@burdges Yeh, I think a great example @eddyb showed before is a tuple of (A: 'a, B: 'static). We assume 'a is the shorter lifetime, so this tuple is 'a. The problem is that has the same type of as (A: 'static, B: 'a). The tuple is still 'a, but now we've extended the lifetime of A beyond what it's actually valid for. If you're relying on that type id to encode lifetimes then you're going to end up supporting unsound behaviour, because of cases like that a simple scheme can't catch.

I see. I hadn't even considered what happens if you get some 'static lifetimes in there.  There is seemingly no reasonable notion of "lifetime kind" that does this, although if you've a specific collection of types then maybe you can make it work, albeit very carefully.

I did specify a scheme that should be sound though, by using ATC (or a trait with a lifetime parameter).
Ideally you'd write a type lambda but this shouldn't be much worse (and you can wrap common cases).

Hmm, this caveat from @eddyb:

As this is only proposing an unstable-only relaxation (of core::intrinsics::type_id) that can already be obtained by re-declaring the intrinsic (as we're currently not checking the 'static bound), I'm suggesting we move ahead and accept it, separate from any other endeavors to expand TypedId or Any.

is ... important to me. I would not be comfortable accepting this RFC if it were not for the above conditions (the most important one being that accessing this relies on an unstable feature anyway and would presumably remain behind that wall)...
I am debating about whether the RFC itself is sufficiently clear about how limited its scope is ... but I guess we aren't ever planning to stabilize compiler intrinsics...?

@rfcbot reviewed

I guess we aren't ever planning to stabilize compiler intrinsics

That’s right. Compiler intrinsics are permanently unstable. The caveat is that somebody proposing a stable wrapper later on might not know about this RFC at the time. That’s the most likely scenario to get a stable access to this intrinsic without 'static lifetime requirement.

Compiler intrinsics are permanently unstable.

I do not agree with this at all. I think compiler intrinsics happen to all be unstable right now, but I expect we will stabilize them sooner or later.

I do not agree with this at all. I think compiler intrinsics happen to all be unstable right now, but I expect we will stabilize some of them sooner or later.

Sorry, this was curt. To expand: I don't see any reason to say that "for ever and ever" we will never stabilize an intrinsic. For that matter, I view transmute as effectively an intrinsic and it is stable. I agree that the current mechanism of declaring intrinsics is not very good (and I keep meaning to propose to remove it altogether, in favor of lang-items) but I think we will definitely stabilize (and indeed have already done) "special items".

I would not be comfortable accepting this RFC if it were not for the above conditions (the most important one being that accessing this relies on an unstable feature anyway and would presumably remain behind that wall)...

@pnkfelix You're right this is relying on the fact that the intrinsic is unstable and users have to go looking for it, otherwise its potential edges would be a bigger problem. I think documentation on the intrinsic should be enough to communicate those edges in this case, but there is more we could do, like add a new intrinsic without the bound and mark it unsafe.
For type_id, I see the path to stabilisation being giving users stable alternatives to its use cases, like TypeId, and some future scheme that accounts for lifetimes. At least being able to opt-in to type_ids newly introduced edges directly we can play with some more of these use cases and have something concrete to work with for future RFCs.

So now that the final comment period is complete what are the next steps for this RFC?

The RFC will need to be merged. I just wrote up a description of the steps, here, as it happens...

@glaebhoerl

Hm. We'd need to express that T must live for at most 'a, and we can only express at least?

Have you considered opening an RFC for that? The syntax 'a: T would be a good place to start

@mikeyhew I haven't, I don't even feel like I understand it.

@glaebhoerl you're right, it is more complicated than I originally thought — if the type can't outlive 'a, does that mean you can't downcast to a type like i32?
@eddyb's suggestion of an Any<'a> trait looks more promising. I tried it out in a playground, and it seems to work. It uses the *const Self receiver for the type_id method so that it can be called on trait objects:
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2015&gist=5fa7a0f75f6a2434d46e98de1542ee19


Best practice suggestions for Managed Identities

Please add some information about managed identities to this discussion about best practices for identity management.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ed24fd26-b126-c171-7edd-9f6713f23ab5
Version Independent ID: 1e240640-ff8c-8183-bd28-ebe3aa4674ff
Content: Azure identity & access security best practices
Content Source: articles/security/fundamentals/identity-management-best-practices.md
Service: security
Sub-service: security-fundamentals
GitHub Login: @TerryLanfear
Microsoft Alias: terrylan

@DeanGross - Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close


[Cohort #5] | Data Input & Web-hooks

Question
Can you please give us an update on how to input external data, apart from data from exchanges, in the Hummingbot scripts/strategies?
For instance, is it possible to integrate any signal (buy, sell, etc.) or information (like prices or oscillator value) from any platform providing web-hooks messages (like TradingView, but also smaller platforms like TRDR, etc.).
Same question, for "On-chain" signals from Glassnode, for instance. Can we integrate these data if they offer web-hooks; what about if they only offer an API instead. Do we need to develop a connector, like for an Exchange?
Thank you!
NB. I see there are 2 options: ConfluxBot or HelloRoger, which one you recommend and why?
Strategy Design Template
No response
Environment
No response

@thenkinet very good question!
You will need to integrate this API to Hummingbot by creating a new DataFeed. The code is very straight forward and there is a snippet that I created for a student that wanted to use the data from an API created by him: https://www.notion.so/hummingbot-foundation/Scripts-Snippets-3b4b454c9a1049d08497beddf52c9f30?pvs=4#5106a9817afe4692bf3afad4c4b6ab22


NIFI-10556 - Add Deltalake processor

Summary
Processor to support Deltalake tables.

Tests will be added after the concept of this processor if acceptable.
During the PR creation I realized that this processor uses so many dependencies, for example it transitively uses the aws-java-sdk which is 280mb alone. I'll try to reduce it's size.
Meanwhile please review the concept of this processor and any feedback is welcomed.

NIFI-10556
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 8
[ ] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[x] Documentation formatting appears as expected in rendered files

I'll try to review this soon.

You're also going to need to add this to nifi-assembly. I would suggest doing that with an optional profile like the other profiles because delta lake support is a niche thing (like Accumulo, graph db support, etc)

Successfully reduced the size of the processor by using the nifi-hadoop-libraries-nar, thanks @exceptionfactory, and by splitting up the aws sdk dependency.


Uncaught TypeError: Cannot read property 'split' of null

[Enter steps to reproduce below:]

Open one file in Atom editor.
Have only one Pane open with one file open.
Choose Packages -> Split Diff -> Toggle menu option.
Crash occurs.

Atom Version: 1.7.2
System: Mac OS X 10.11.4
Thrown From: split-diff package, v0.7.4
Stack Trace
Uncaught TypeError: Cannot read property 'split' of null
At /Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:833

TypeError: Cannot read property 'split' of null
    at Function.module.exports.Range.fromText (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/range.js:47:19)
    at TextBuffer.module.exports.TextBuffer.setTextInRange (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:588:24)
    at /Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:576:26
    at TextBuffer.module.exports.TextBuffer.transact (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:828:18)
    at TextBuffer.module.exports.TextBuffer.setTextInRange (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:574:21)
    at TextBuffer.module.exports.TextBuffer.setText (/Applications/Atom.app/Contents/Resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:514:19)
    at TextEditor.module.exports.TextEditor.setText (/Applications/Atom.app/Contents/Resources/app.asar/src/text-editor.js:810:26)
    at Object.module.exports.SplitDiff.getVisibleEditors (/Users/kcole/.atom/packages/split-diff/lib/split-diff.coffee:77:21)
    at Object.module.exports.SplitDiff.diffPanes (/Users/kcole/.atom/packages/split-diff/lib/split-diff.coffee:108:16)
    at Object.module.exports.SplitDiff.toggle (/Users/kcole/.atom/packages/split-diff/lib/split-diff.coffee:400:8)
    at atom-workspace.subscriptions.add.atom.commands.add.split-diff:toggle (/Users/kcole/.atom/packages/split-diff/lib/split-diff.coffee:34:32)
    at CommandRegistry.module.exports.CommandRegistry.handleCommandEvent (/Applications/Atom.app/Contents/Resources/app.asar/src/command-registry.js:260:29)
    at CommandRegistry.handleCommandEvent (/Applications/Atom.app/Contents/Resources/app.asar/src/command-registry.js:3:61)
    at CommandRegistry.module.exports.CommandRegistry.dispatch (/Applications/Atom.app/Contents/Resources/app.asar/src/command-registry.js:160:19)
    at AtomEnvironment.module.exports.AtomEnvironment.dispatchApplicationMenuCommand (/Applications/Atom.app/Contents/Resources/app.asar/src/atom-environment.js:1100:28)
    at EventEmitter.outerCallback (/Applications/Atom.app/Contents/Resources/app.asar/src/application-delegate.js:314:25)
    at emitTwo (events.js:87:13)
    at EventEmitter.emit (events.js:172:7)

Commands
     -0:00.9.0 split-diff:toggle (atom-text-editor.editor.is-focused)

Config
{
  "core": {
    "themes": [
      "atom-dark-ui",
      "atom-dark-syntax"
    ]
  }
}

Installed Packages
# User
atom-jinja2, v0.5.0
split-diff, v0.7.4

# Dev
No dev packages

I think I can fix this. Quick question though, was the file you were diffing in a git repository?

Yes the file was part of a hit repository.

Anything special about the file or repository? For instance, was the file not part of the repo / not committed yet? It looks like the latest copy of the file from the branch couldn't be opened, so it tried to set your second pane text to null.
Sorry for all the questions, just trying to figure out if there is a way to fix it so that it will show you your git changes instead of failing and opening with a blank pane.


Product profiles doc 2.2.0

Description:

Need to review steps mentions in product profiles doc.
In order to get profiles to work, I had to follow this document [1] as well.
[1] https://docs.wso2.com/display/AM220/Deploying+WSO2+API-M+in+a+Distributed+Setup#DeployingWSO2API-MinaDistributedSetup-Step6.5-ConfigureandstarttheTrafficManager
Steps to reproduce:
Content Positioning in Documentation:

Link: https://docs.wso2.com/display/AM220/Product+Profiles

Heading (& Step):

Any other reference:

Optional Fields
Related Issues:

Suggested Labels:

Suggested Assignees:

This is based on a much older version of API-M so will close this as not planned.


How to listen for Deletes on Objects from SalesForce

Currently we only get Insert & Updates from the salesforce.
How do i get the Deletes from salesforce as well. if Logic app is triggered for INSERT/UPDATE/DELETE is there a way to find out
if the record was updated/Inserted or deleted on salesforce side ?
Also what mechanism does logic app use to listen for changes ?
Salesforce support the following

CDC with replayID
SOSQL Object Query
Push Topics

say we disable logic app for 8 hrs and then reenable it, will logic app pickup all the changes that occurred after it went down till current ?
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b089c20e-6e3f-ec60-bd2d-263bf12bf286
Version Independent ID: 10423981-4d5b-f5b4-e72c-15a5ddcc34b8
Content: Connect to Salesforce from Azure Logic Apps - Azure Logic Apps
Content Source: articles/connectors/connectors-create-api-salesforce.md
Service: logic-apps
GitHub Login: @ecfan
Microsoft Alias: estfan

@nssidhu Thank you for your feedback! We will review and update as appropriate.

@nssidhu I do see that you have also create this Q&A query.
As I mentioned in Q&A you cannot track the delete item. There are 2 reasons:
•	Not every type in Salesforce supports “soft deletions” (setting IsDeleted flag to true), some data types uses a normal deletions (move record to Recycle Bin), and some records of certain types do not support deletion at all (for example, User).
•	Even if we consider a type which supports soft deletion, a connector trigger uses SOQL query to get a set of modified records.  LastModifiedDate will be updated when a record is safe deleted.
If we disable logic app for 8 hrs and then reenable it, will logic app pick up all the changes that occurred after it went down till current ?
Yes this is the expected behavior.
Hope this clears thing up. Please let me know if you have any queries or concerns.

For Deletes(soft Deletes), i can confirm they are not flowing through. I bet you have to append "ALL ROWS" to the end of SOSQL to get the Soft Deletes. that will be lot helpful if can enable that to flow through

@nssidhu I have passed your feedback to the team and I do see similar feedback for the salesforce connector to query the deleted records. You can always pass your feedback following this  for azure logic app connectors.
But unfortunately as of now it is not possible.

@nssidhu As of now this is not possible and if you have any product feedback you can always pass it in feedback forum. We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


fix: pass memory to GetRunningApp instead of initializing twice

GetRunningApp was using turn context to get instance of memory but not returning the instance.
Then immediatly after this call another instance would be created.
Changes to create instance and pass to GetRunningApp

:tada: This PR is included in version 0.325.1 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


JsonConvert.DeserializeObject will converting a value like "[]" into a new T

This seems like a bug as I would expect an empty like [ ] to convert into null or be unconvertable. If there is a setting to make this happen that would be great, and sorry if this is a dupe (searching for [] is not producing any meaningful results.)
Source/destination types
public class MyType
{
    public string FirstName {get;set;}
    public string LastName {get;set;}
    public int MyId {get;set;}
}

Source/destination JSON
Below is what displays in the Watch window when inspecting an properties of an instance of the JToken  named resultJson , the full inspection in the watch window being resultJson["ouputData"]["data"].ToString()
[]

When inspecting resultJson["ouputData"]["data"].HasValue the result is false
Expected behavior
I expect when I call  that either a null is returned or an exception is thrown.
Actual behavior
I get an instance of MyType with null string property values and a 0 for the integer.
Steps to reproduce
var value = JsonConvert.DeserializeObject<MyType>(resultJson["ouputData"]["data"].ToString());

No, attempting to deserialize the string "[]" to MyType will throw an exception.
Proof: https://dotnetfiddle.net/7NKoJb
I suggest you assign resultJson["ouputData"]["data"].ToString() to a string variable before invoking JsonConvert.DeserializeObject and inspect the string variable in the debugger. I suspect somewhere somehow you got confused during trouble-shooting about the actual result of resultJson["ouputData"]["data"].ToString()...

got confused

This particular code is kind of a mess because its handling a few different payload structures trying to Deserialize from different possible places.
The STR example is consolidated, its actually several statements with checks to make sure the payload has the output data and data elements. The problem I think in this case had two contributing components - the wrong type was being passed in as T, and some of the T types had properties with internal setters.

It would be nice to have a serializer option that would throw an exception when a property could not be set, or some other easier diagnostic than debugging a library.


feat: add operator for switching backend execution

The operator segments the execution across multiple backends.
It is based on the working CachedNode implementation.
closes #191

Closed in favor of #282


Fix --source option

This fixes the erroneous output described in https://github.com/PyCQA/pydocstyle/issues/256.
Perhaps someone else can decide if the line numbers displayed by --source should be of fixed width (in which case lines https://github.com/PyCQA/pydocstyle/blob/master/src/pydocstyle/violations.py#L56-L59 can be removed) or not (in which case line https://github.com/PyCQA/pydocstyle/blob/master/src/pydocstyle/violations.py#L60 can be removed).

Can you please add some tests to make sure this doesn't break again?

Closing - continued in #260.


Migrate to newer github action components to avoid deprecation warnings

Desired behavior
Most of the actions/checkout, actions/upload-artifact and alike components are pinned to the latest v3 version. We need to move to v4 to avoid deprecation warnings related to Node.js.
Alternatives considered
N/A
Implementation suggestion
N/A
Additional context
See https://github.com/maliput/maliput_py/actions/runs/7782058733 for example.
https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/
Due date: 2024/06/30

[ ] maliput
[ ] maliput_dragway
[ ] maliput_multilane
[ ] maliput_malidrive
[ ] maliput_sparse
[ ] maliput_py
[ ] maliput_integration
[ ] maliput_integration_test
[ ] maliput_object
[ ] maliput_object_py
[ ] maliput_osm
[ ] maliput_sim
[ ] maliput_viz
[ ] maliput_xodr
[ ] delphyne
[ ] delphyne_demos
[ ] delphyne_gui
[ ] lanelets_eval
[ ] maliput-rs

Making use of the occasion to update other actions as well:

actions/checkout@v4
actions/upload-artifact@v4
<EMAIL_ADDRESS>codecov/codecov-action@v4
docker/login-action@v3
docker/metadata-action@v5
docker/build-push-action@v5
softprops/action-gh-release@v2
<EMAIL_ADDRESS>

Curiously, all workflows using the manylinux image failed. That is related to a glibc library version installed in the image which collides with a node dependency required by actions/checkout@v4. Seems like sticking to v3 for now is the only way moving forward.
In particular, the manylinux image has glibc-2.17-326.el7_9.x86_64 and the requirement is for glibc 2.25+.

Curiously, all workflows using the manylinux image failed. That is related to a glibc library version installed in the image which collides with a node dependency required by actions/checkout@v4. Seems like sticking to v3 for now is the only way moving forward.
In particular, the manylinux image has glibc-2.17-326.el7_9.x86_64 and the requirement is for glibc 2.25+.

Until November 30, 2024 we are ok: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/


Use of aria-live inconsistent across www.gov.uk search results

Pages that display search results with filters in a left-hand column do not identify the summary information consistently to assistive tech'.
Overview
When JS is enabled, the search results on these pages are updated by AJAX.
The change in the page content needs to be announced to users of assistive tech'. This is done by setting aria-live on the results summary paragraph but the value it is set to is not the same across www.gov.uk.
The implementation of the live-region is also slightly different for the search page compared to the policies page.
What needs to be done
One value should be chosen for the aria-live attribute and make it the same everywhere.
The implementation is also slightly different on the search page. Both pages should be made the same.
Note: work done on this a while ago was documented in this excellent blog post.
Pages owned by finder-frontend
The pages I've found of this type (there may be others) are:

https://www.gov.uk/government/publications
https://www.gov.uk/government/consultations
https://www.gov.uk/government/statistics
https://www.gov.uk/government/announcements
https://www.gov.uk/government/policies
https://www.gov.uk/search

5 & 6 are owned by this app. I've raised another issue for this on 1-4.

I looked at all the linked pages but can only see one value (aria-live="assertive"). Has this been fixed? Only https://www.gov.uk/government/statistics doesn't have any aria-live at all. (At least I couldn't find it.)
But having briefly looked at the implementation now, I would improve two other things:

What is wrapped in the div with the aria-live includes the number of results (in an h2), the facet tags and on some pages the "get email alerts" and "subscribe to feed" links. At least those links should not be announced to screen readers because they don’t change. (Actually, I assume the URL changes, but that won't be apparent to anyone, including screen reader users.) The facet tags can be up for debate if it makes sense for them to be announced, but I would suggest to only wrap the h2 (or add the aria-live directly to the h2).

After filtering at least once, the h2 gets nested in another h2. That is invalid HTML.

@injms or @andysellick
I have point 2 from ⬆ as a ticket but it seems that is has been fixed.
can either of you confirm or alternatively show how to replicate (in case i'm missing something)?

@kr8n3r yeah I think I remember spotting that and fixing it.

thanks Andy

@tombye I believe we replaced all of the finders that this ticket is talking about (originally from whitehall, now rebuilt in finder-frontend) and the aria issue should have now been fixed. Can you confirm if this is the case?

Looks like all of these now share the same implementation and the issues @selfthinker raised are now fixed so I'm marking this as resolved 👍


Feature/773 update character check consent form

What's included?
Candidates are able to log into their accounts when invited to do so via email and complete/view completed consent forms.
Who should test?
✅ Product owner
✅ Developers
✅ UTG
How to test?
(Please test together with #1312 and #1327) Here is the video description of expected functionality

Apply for an exercise.
When receive character checks request/reminder email, log into the Apply account, complete and submit consent form.
Review submitted information.

Risk - how likely is this to impact other areas?
🟠 Medium risk - this does change code that is shared with other areas
Additional context

PREVIEW:DEVELOP
can be OFF, DEVELOP or STAGING

This work needs to be integrated with Staged Application feature after I've finished preparation for access to JO API.

@mbrookeswebdev Is this blocked

@londonprojects It's in code review

@joy-ade can I please have more details, as this feedback does not tell me anything useful. I understand some other testers managed to complete the form, so if these two testers could not do it, what errors did they have if any? If the first tester clicked on Submit button, I guess that means he got to the last page of consent form? What about second tester, what exactly didn't work?

@joy-ade I've made some changes and functionality should work now.

Posted on the UTG channel today after some changes were made.


Remove prohibited words and update description and version

This task is related to the name profanity task - https://github.com/overwolf/Elerium/pull/147

"boring" and "meitner" from the list of words for name generation?


AddSortableColumn, part deux

Gotta say, this as in incredibly useful add-on that has saved me a lot by not having to handle all the details! Some things to consider:
Further spelunking re: my earlier post.
It appears that the original design was for both Available and Selected to have the same characteristics (forEachGrid in addSortableColumn). This may now be limiting, particularly after adding the ability to rearrange Selected. The sortable attribute may not be appropriate at all for the Selected grid.
Consider that future flexibility to adjust Available and Selected characteristics independently may enable some functionality that is otherwise not possible or convoluted to construct.
In my case I've run into some issues that seem insurmountable (could be I'm not doing it right!) because:

I want Available to be sorted and Selected to be client managed in terms of ordering
If I use addSortableColumn, both Available and Selected are enabled as sortable.

The client can reorder Selected by moving items up or down
sort Selected using the header arrows
drag/drop an item into the now sorted Selected
then go back to their original sort (both up/down arrows visible) and that item (to them) shows up in a random spot
now requiring more work to move it to the right spot.

More important, using a Save button to persist the Selected items after a sort loses the client ordering entirely. Not intuitive. The only fix here is for the client to manually redo the ordering and save again.

It strikes me that allowing unique choices on Selected behavior will enable us to provide a more creative user experience.
A secondary issue, minor but interesting, is that there is a case where having a header for both Available and Selected is unnecessary, saving some screen real estate. For example:

Set Available as sortable, Selected as 'client managed' (or whatever, see above)
Developer delivers Available already sorted; client is never expected to need to re-sort Available, therefore no header required; adds will fall into the correct place as the column is sorted.
No header required for the Selected column
The preferences page identifies what the grid is about, e.g., "Stock Data Attributes"
The TwinColGrid ...GridCaption of "Available" and "Selected" suffices to describe what the columns are
Available and Selected would have the same header. In my case, it would be "Attribute"

There does not appear to be a way to eliminate the column headers, likely as the design relied on them for sorting.
Thanks for your consideration!

Discovered the 'createFirstHeaderRow' constructor:

solved I want Available to be sorted and Selected to be client managed in terms of ordering
solved If I use addSortableColumn, both Available and Selected are enabled as sortable.
solved More important, using a Save button to persist the Selected items after a sort loses the client ordering entirely. Not intuitive. The only fix here is for the client to manually redo the ordering and save again.

Most of the secondary issue is solved as well. What is left is that moving an item from Selected to Available puts that item at the bottom of the list rather than back in its original spot. Not intuitive.
Still think a future where each grid can be configured separately will create more flexibility ...

Hello.

It appears that the original design was for both Available and Selected to have the same characteristics.
This may now be limiting, particularly after adding the ability to rearrange Selected. The sortable attribute may not be appropriate at all for the Selected grid.

Yes, that was the original design. Then we realized that some use cases require different configurations on each side, thus we introduced public getters for each grid (getAvailableGrid / getSelectableGrid) so that they can be freely configured. TwinColGrid retains some addColumn methods for the common case of applying the same configuration to both grids.

Most of the secondary issue is solved as well. What is left is that moving an item from Selected to Available puts that item at the bottom of the list rather than back in its original spot. Not intuitive.

I understand that this issue can be closed, since what remains is already discussed in #127 (feel free to comment if you have further suggestions or questions)


Prepare documentation for Toil paper

Continued from #742
Resolves #741, resolves #752, resolves #767, resolves #738, resolves #744, resolves #745.

Just noticed that Chapter 9: The Batch System, is not building correctly.
On further notice... There is a critical error in the abstractBatchSystem class (which is preventing it from building).

lol who wrote this section in Chapter 8, section 1?  "Toil implements lots of optimizations designed to allow it to scale out. Many are not worth mentioning, but we detail some of the key ones below that we have found useful."    I'm going to submit a commit to remove it.

And guys, we don't do periods at the end of commit messages.
A friend of mine wrote this: http://chris.beams.io/posts/git-commit/
Well worth reading on a rainy day ...

I'm going to rebase and merge this. Taking off multi-author.


Add support for edge nodes

Add support for edge nodes
This creates a number of frontend edge nodes (defaults to zero)
that are meant to run user-facing services; e.g. frontend nginx
instances.
Initial support is for SoftLayer and OpenStack, with optional
DNSimple records.

What do you think about this approach? I have experimented with setting up a couple of dedicated worker nodes ("frontend_edge"), meant for receiving user-facing traffic. The organisation's user-facing DNS records would point to these nodes, or to some hardware load balancer that runs directly in front of them.
The way we create, update and destroy these nodes would have to be different, since they need to be updated without downtime. I guess that can be solved with a master/standby setup, with a floating IP address; either at the Mesos task level or at an external hardware load balancer.

@eirslett thanks for the PR. We're going to be redoing our proxy setup in a manner very similar to this. I'm not going to merge this right away, but I will cherry pick commits into the new setup. Thanks again!


does ent support graphql federation?

Hey @demobin8,
Can you elaborate on what do you mean by supporting federation? What do you expect from ent to support in this case?

Never mind about federation, it's support by gqlgen.
And, Excuse for my noob, How to elegant deal with the entity association between two databases of two microservices in ent?


[WIP] Create new type

Implements: https://github.com/OmniSharp/omnisharp-roslyn/issues/2270
Ready for intermediate review.
Next steps:

Add tests (it will take some time since no one test is passed for me when run ./build.sh

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Anakael sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks a lot for your PR and your work. However as discussed in #2270 we do not plan to have this as part of the core server experience at the moment.
In particular, there are many edge cases in such feature which you can see e.g. in the generate type service in Roslyn https://sourceroslyn.io/#Microsoft.CodeAnalysis.Features/GenerateType/AbstractGenerateTypeService.cs


fix check for ffsll and use builtin if available

This is a comment on PR #2449.
It seems that, in the conda build environment, ffsll is defined in glibc but not included in any headers. Replacing check_function_exists with check_symbol_exists allows this situation to be detected.
A Github Actions workflow for testing the conda-build process is also included. It's running on this PR here:
https://github.com/chillenb/pyscf/actions/runs/11243087685/job/31258188365

Thanks for the solution for #2449. This change has been merged into that PR.


How to endsession/logout from the server from the client?

I've been trying to logout from the oidc server using userManager.removeUser() but these method only reset the info stored in the client side. I've seen that in the userManager configuration it's posible to define an endsession endpoint, How can I logout from the server from the client?

I found what I was looking for at the oidc-client wiki. The method to sign out is: signoutRedirect -> Returns promise to trigger a redirect of the current window to the end session endpoint.


Users (14)

It does not show me a message that everything was saved correctly

Working as expected


[Feature Request]: Set quota project in beam.io.ReadFromBigQuery

What would you like to happen?
This issue relates to the Python SDK, but it is probably relevant to other SDKs as well:
We have a use case where queries initiated by beam.io.ReadFromBigQuery should be billed on a specific GCP Project ID.
As we use a custom container, the only option for now would be setting the env. var. GOOGLE_CLOUD_QUOTA_PROJECT in the Dockerfile - but it affects all other GCP services as well.
It would be best making it configurable via the connector (i.e., beam.io.ReadFromBigQuery(..., quota_project_id='some-project-id)).
When implementing, you could gain inspiration from similar feature in beam.io.WriteToBigQuery: https://github.com/apache/beam/pull/16186.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components

[X] Component: Python SDK
[X] Component: Java SDK
[X] Component: Go SDK
[X] Component: Typescript SDK
[X] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[x] Component: Google Cloud Dataflow Runner

Also, I wonder whether implimentation of this issue would help with https://github.com/apache/beam/issues/30747

@shahar1 this level of customization might make sense.
Let's explore your specific concern for a moment, I might have others, but imagine worth understanding your needs/usecase:
What Quotas are getting hit that are problematic? Or, what are the specific billing charges you are looking to attribute elsewhere? Your are running on Dataflow? Or other? [ not critical, but curious ] What read method? [ BQ Storage Read API? ] You want to run the compute in one GCP project, but use BQ from another? If this unloads, writes to GCS and then into Dataflow [ that is another way that can occur ], do you intend to specify which project [ bucket within ] that the data is written?

Thank for your response! Here are the answers for your questions:

I'd like to attribute the queries execution to another project. In my case, the BigQuery is on project A, and beam.io.ReadFromBigQuery runs on project B - I'd like to bill project B for the queries (for that matter it could also be project C).
We use Dataflow and direct runner (when implementing, it should better a be a general solution and not Dataflow specific).
I use both methods - if I'm not wrong, in both cases you could set the quota_project_id via ClientOptions.
Yup, you got the idea correctly :)

As for #30747 - it is related, but there might be some changes in implementation as GCS is project's resource rather than a service.

@shahar1 sounds like you've got a decent idea/design in mind, which could be supported.
Are you interested in contributing? Feel free to start and include me on PRs, if that's the case.

@shahar1 sounds like you've got a decent idea/design in mind, which could be supported.
Are you interested in contributing? Feel free to start and include me on PRs, if that's the case.

I'd be happy to try!
I need first to learn how development works here (I'm coming from the Airflow community)

.take-issue

This should be pretty good --> https://github.com/apache/beam/blob/master/CONTRIBUTING.md
If you find a problem [ or that is outdated ], let's overcome and fix the docs along the way.

Are there any updates on this?

Are there any updates on this?

I haven't managed to work on it yet.
If you (or anyone else) find this feature useful and want to take over, please let me know.


[Bug]: Git Clone Fails: SSH Key Authentication Error for Private Repository

Error Message and Logs
Cloning into '.'...
Permission denied, please try again.
Permission denied, please try again.
<EMAIL_ADDRESS>( changed ): Permission denied (publickey,password).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Steps to Reproduce

Generate SSH key pair in Coolify's Key & Token management.
Copy public key to GitLab repository's Deploy Keys.
Create new project in Coolify, entering Git repository URL and required details.
Load Docker Compose file for the project.
Attempt to clone repository, resulting in SSH authentication failure.

Example Repository URL
No response
Coolify Version
v<IP_ADDRESS>-beta.35o
Are you using Coolify Cloud?
No ( Self Hosted )
Operating System and Version (self-hosted)
Ubuntu 22.04
Additional Information

No matter how many times i save the URL
it changing to some random URL

I'm having the same issue. Using Gitea trying to deploy from a private repo. Works fine with Github.
-Generate SSH key pair in Coolify's Key & Token management.
-Copy public key to Gitea repository's Deploy Keys.
-Create new project in Coolify, entering Git repository URL and required details.
-Deploy
-Attempt to clone repository, resulting in SSH authentication failure.
Error message.
Starting deployment of git@gitea-fhghghf88g.<IP_ADDRESS>.sslip.io:fhhg/hhgf.git:main to localhost.
Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:1.0.2.
Oops something is not okay, are you okay? 😢
Permission denied, please try again.
Permission denied, please try again.
git@gitea-fhghghf88g.<IP_ADDRESS>.sslip.io:Permission denied (publickey,password).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Deployment failed. Removing the new version of your application.

can you check source in the Project is it also changing ?

No I can change the source in the project file.

Yes After done everything correct still failing to clone

Try This. The port Gitea was using was missing from the ssh url.
What I did.
Under "Service Stack," click "Edit Compose File" and see what port Gitea is listening on. Mine is 22222.
You'll need to add that port to your Gitea SSH URL. Something like this:
git@gitea-gccgasdds4sdsd.3.444.232.32.sslip.io:22222/user/repo

Thanks But
Not working for me

Any Solution on this ?

Any solution on this problem ?

Update: When the server is local (host.docker.internal) It seems the SSH command to fetch the docker-compose.yaml file from the git repo on the fly is executed as user "webuser" instead of user "root". Therefore ssh is looking for /home/webuser/.ssh/id_rsa (does not exist) instead instead of /root/.ssh/id_rsa (exists)
Code: app/Models/Application.php 1199
        ['commands' => $cloneCommand] = $this->generateGitImportCommands(deployment_uuid: $uuid, only_checkout: true, exec_in_docker: false, custom_base_dir: '.');
        $workdir = rtrim($this->base_directory, '/');
        $composeFile = $this->docker_compose_location;
        $fileList = collect([".$workdir$composeFile"]);
        $commands = collect([
            "rm -rf /tmp/{$uuid}",
            "mkdir -p /tmp/{$uuid}",
            "cd /tmp/{$uuid}",
            $cloneCommand,
            'git sparse-checkout init --cone',
            "git sparse-checkout set {$fileList->implode(' ')}",
            'git read-tree -mu HEAD',
            "cat .$workdir$composeFile",
        ]);

+1 having the same issue. Hope that we can fix it or we have to move it to Github.

I've the same issue. It's affecting my ability to use Coolify. Might have to drop it's usage altogether without the ability to use private repositories effectively.


iPhone 6s iOS 10.2 nonce setting fails

So since nonce enabler is installed by default with Yalu I thought I could just set my generator in case of failures and I need to restore however when I set the boot nonce in nvram it gets error setting variable iokit/common general error

get root first

#162 not code related specifically to this jailbreak.

I am root
And it kind of is code related because Yalu is meant to enable this but for some reason it fails

lol, what do u mean root, type "su" and your root pwd then add your nonce, pls google before post here

Exactly what I have said I am root i even tried your way of 'su' even though I already am and it still gives the same error

with root

without root

okay lets show you on my device
https://gyazo.com/f8a7a47b9c30d3a53aea75377245229a
as you see from the whoami command i am infact root

why not use MTerminal

I have tried on mterminal also same result just easier to do it via putty

dude you con not com.apple

in your second photo maybe you need to update to beta 7 yalu102 if you need help you can dm me on twitter @modz2014

That was other guy
And I got it working now I'll close this for some reason the ipa off his site didn't work so I just downloaded and compiled it and tried again and worked just fine


illegal instruction on an i7

 Test of subprogram number 10            IZAMAX
                                    ----- PASS -----
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./dblat2 < ./dblat2.dat
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./dblat3 < ./dblat3.dat
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./cblat3 < ./cblat3.dat
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./cblat2 < ./cblat2.dat
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./zblat3 < ./zblat3.dat

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:
#0  0x7f6292772e9d in ???
#1  0x7f62927720e3 in ???
#2  0x7f6291ec43af in ???
#3  0x46989d in ???
/bin/sh: line 1:  7826 Illegal instruction     OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./zblat3 < ./zblat3.dat
Makefile:164: recipe for target 'level3' failed
make[1]: *** [level3] Error 132
make[1]: *** Waiting for unfinished jobs....
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./zblat2 < ./zblat2.dat
rm -f ?BLAT2.SUMM
OPENBLAS_NUM_THREADS=2 ./sblat2 < ./sblat2.dat
OPENBLAS_NUM_THREADS=2 ./dblat2 < ./dblat2.dat
OPENBLAS_NUM_THREADS=2 ./cblat2 < ./cblat2.dat
OPENBLAS_NUM_THREADS=2 ./zblat2 < ./zblat2.dat
make[1]: Leaving directory '/mnt/quadrupole/sources/math/OpenBLAS-0.3.15/test'
Makefile:145: recipe for target 'tests' failed
make: *** [tests] Error 2

cpuinfo:
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
stepping        : 9
microcode       : 0x15
cpu MHz         : 3400.117
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 6800.23
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Is there anything that needs to be added? Other LTS distribution with similar perks? It is best worked around here so that next users just type make and it works.
https://github.com/xianyi/OpenBLAS/wiki/faq#binutils
Could you help with as --version (2.34-5 or something like that) cc --version gfortran --version  (4.3.2-1 or similar, preferably both of same version) , and in general what is that system - say RHEL7 would work correctly with AVX1 CPUs without patching,maybe some Debian/Ubuntu comeercial LTS or what is it. If there are still supported/-able vendor patches we will try to do so that even CD pre-release of same OS works error-free. Sometimes even go deeper in archaeology if setup seems relevant/ widely used.

git bisect puts the blame on #2890 (which would make sense if the older gcc mishandled -mssse3 somehow), but reverting the changes at that point does not make it compile again with gcc4.

If that question was for me,
as --version
GNU assembler (GNU Binutils) 2.36.1
cc --version
cc (GCC) 7.3.0
gfortran --version
GNU Fortran (GCC) 7.3.0
You will notice that I upgraded binutils (before the one successful build using TARGET=NEHALEM [whatever that means])

Looks like it is the (global) addition of -mavx in Makefile.x86_64 (prompted by the introduction of intrinsics in the SUM kernels) that causes miscompilation with the older gcc. It works when that compiler flag is selectively applied to the kernel code however.
I'll fix it later today.

@themaddoctor thanks. Nehalem is CPU series, next after Core series that bridged from 32bit to 64bit world. Your CPU is 'ivy bridge' , which in OpenBLAS world is handled as "SANDYBRIDGE"
Using AVX instructions makes bulk computation like GEMM faster, see intel's marketing papers on AVX vs SSE speedup.


Using dart functions as callbacks in native javascript

I'm trying to follow the example on the documentation about this, but for some reason I'm having the following:
NoSuchMethodError: tried to call a non-function, such as null: 'dart.global.createDecoder'

This is probably a mistake of mine rather than a bug, and I know from reading previous issues here that this is not the ideal channel to ask for help, however I already asked weeks ago in stackoverflow and both Flutter and Dart Reddit communities and got no answer at all.
my native script is
import { OpusStreamDecoder } from 'opus-stream-decoder';

const decoder = null;

export function createDecoder(onRawDataHandler) {
    decoder = new OpusStreamDecoder({onRawDataHandler});
}

export async function decode(input) {
    try {
        decoder.ready.then(_ => decoder.decode(input));
    } catch (e) {
        decoder.ready.then(_ => decoder.free());
    }
}

export async function freeDecoder() {
    decoder.ready.then(_ => decoder.free());
}

My interop class is
@JS()
library opus_decoder.js;

import 'dart:typed_data';

import 'package:js/js.dart';

/// Allows assigning a function to be callable from `window.functionName()`
@JS('onRawDataHandler')
external set onDataHandler(
    Future<void> Function(Float32List left, Float32List right,
            int samplesDecoded, int sampleRate)
        f);

@JS('createDecoder')
external void createDecoder(
    Future<void> Function(Float32List left, Float32List right,
            int samplesDecoded, int sampleRate)
        onRawDataHandler);

@JS('decode')
external Future<void> decode(Uint8List input);

@JS('freeDecoder')
external Future<void> freeDecoder();

when calling the following function, the error occurs
  void callJsCreateDecoder(
      Future<void> Function(Float32List left, Float32List right,
              int samplesDecoded, int sampleRate)
          onRawDataHandler) {
    onDataHandler = js.allowInterop(onRawDataHandler);
    createDecoder(js.allowInterop(onRawDataHandler));
  }

What am I doing wrong?

You'll need to manually add the function to globalThis to expose top-levels versus export (we'll document this on dart.dev soon with the new interop). Related discussion here: https://github.com/dart-lang/sdk/issues/39995#issuecomment-832297698

Thanks for the link. Looking forward to that documentation.
I'm searching how to add a function to globalThis but haven't found anything. Could you point me in the right direction please?

Oh.. I see. Thank you so much for everything

Sorry to bother with this again, I added to the .js file
globalThis.createDecoder = createDecoder;
globalThis.decode = decode;
globalThis.freeDecoder = freeDecoder;

like you said, but the errors remains the same. Is there anything else I could do? I also tried with window

Hmm, nothing pops out at me unfortunately, so I think this might require a little bit of debugging. Obvious question, but is the native script being executed before main is executed? Is it possible to debug the generated JS and see if window.createDecoder exists at the beginning of main?

what is obvious to you might not be to me, lol. It seems you are right, it appears to be a problem with executing the script
I found this
Uncaught TypeError: Failed to resolve module specifier "opus-stream-decoder". Relative references must start with either "/", "./", or "../".

I'm doing the following
<body>
    <script src="./opus_decoder.js" type="module" defer></script>
.
.

opus_decoder.js" is in the web folder as well as index.html
"is it possible to debug the generated JS and see if window.createDecoder exists at the beginning of main?"
How can I do this?

Most of what I find on internet says that I can either use node which I don't think is possible or download the source code of the library and put it in the web folder. Issues like the following are appearing
Uncaught TypeError: Failed to resolve module specifier "opus-stream-decoder". Relative references must start with either "/", "./", or "../".
opus-stream-decoder.js:1 wasm streaming compile failed: TypeError: Failed to execute 'compile' on 'WebAssembly': Incorrect response MIME type. Expected 'application/wasm'.
failed to asynchronously prepare wasm: CompileError: WebAssembly.instantiate(): expected magic word 00 61 73 6d, found 3c 21 44 4f @+0
there is probably something that I'm missing like I was missing the part of executing the script before main. Is there any reference I could read related to this kind of thing?

what is obvious to you might not be to me, lol. It seems you are right, It was a problem with executing the script before the main script. I didn't know how to do it, none of the pages/blogs I saw as a guide mentioned it.
the functions are now found, the current problem is related to the module I need to use in my native script.
Is there anything to read about how to integrate a node module and running web assembly? I haven't tried with other modules so I don't know if the errors I'm seeing are from the module itself or mine


add Promise allSettled static methods

Hello! Promise add a static methods 'allSettled', This is the new ES2020 standard, Tested without problems in IE,Chorme,Firefox,Edge.

If anyone is interested in this API, please feel free to pick up this submission and submit another PR.


Add zinzy.website

Heya, thank you for keeping nostalgia and small web beauty alive and kicking with your work. I would love to join the xxiivv webring with my website, zinzy.website (source code). I'm a designer, researcher, and perpetual web tinkerer, and use my personal website as a central space for posting on the web.
The webring link is located in the footer of my website.
Thank you for considering me, and a happy 2024 to you and yours!

Hi Zinzy,
I love your website, I spent all morning going through your posts, it's nice to see stuff going way back. Welcome to the webring :) Two things:

The webring icon would look much nicer without a white background, I think the white stroke svg version would be perfect, it's a vector file.
Is there anywhere on the website where I can see your design work? I got curious reading the posts but never found any link to something like a project's page?

Happy new year :)

Hey Devine,
Thank you for adding me! And thanks for your kind comments about my site. It's always a work in progress, and I hope to turn it into a bit more of a wiki. Your website is inspiration for sure.
I was being a bit of a brat with the webring icon 😏 . I'll pay it a bit more respect in my next update.
I don't have any design work available to the public, since the vast majority of it is sealed by strict NDAs. If there's anything specific you're looking for, I'm happy to see what I can do!


heroic-games-launcher: Update to 2.9.0

@Calinou this is an important PR.
Updates heroic-games-launcher to 2.9.0.

[x] I have read the Contributing Guide.

/verify

/verify

/verify

/verify

/verify

/verify

Relates to https://github.com/Heroic-Games-Launcher/HeroicGamesLauncher/issues/2902
Relates to https://github.com/Heroic-Games-Launcher/HeroicGamesLauncher/issues/2905

@Calinou do you mind manully rolling heroic games launcher back to 2.8.0, and disable autoupdate until the issues listed above are fixed.

I'm going to close this PR until the latter issues are resolved.

I'm going to close this PR until the latter issues are resolved.

It looks like 2.9.1 was just released. It apparently resolved some of the issues: https://github.com/Heroic-Games-Launcher/HeroicGamesLauncher/issues/2902#issuecomment-1652185413

They also removed arm64 support for Windows, do you mind manually removing it from the heroic-games-launcher manifest?

They also removed arm64 support for Windows, do you mind manually removing it from the heroic-games-launcher manifest?

Done: https://github.com/Calinou/scoop-games/commit/8f627911d03c94f594419867cd7bdc3364955ddc

They also removed arm64 support for Windows, do you mind manually removing it from the heroic-games-launcher manifest?

Done: 8f62791

Thank you very much.


pull request for e2e test at 11/23/2016 05:56:09

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Handle errors when trying to get config

Added error checking when retrieving the config and also handle the promise reject in dbConfig() to avoid bubbling up to browser.

Ignore


adding min delay to the read

What type of PR is this? (check all applicable)

[x] ✨ Feature
[ ] 🐞 Bug Correction
[ ] 📝 Documentation Update
[ ] 🚩 Other

Changes:

Adicionando limite inferior no delay de leitura
Testing

O cara só pensa na Hacktoberfest


[BUG] panic on an old mac

Describe the bug
> macchina
thread 'main' panicked at 'No entry found for key 0x7fe6c5600ba0', /Users/fdncred/.cargo/registry/src/github.com-1ecc6299db9ec823/core-foundation-0.9.1/src/dictionary.rs:233:43
stack backtrace:
   0:        0x1080a8d29 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h34f76b2ca42fab41
   1:        0x108080fad - core::fmt::write::h8f13d857463c51c1
   2:        0x1080a8476 - std::io::Write::write_fmt::h0a6214ccb1a8468b
   3:        0x1080a7ba4 - std::panicking::rust_panic_with_hook::h28bf9d6c3916221e
   4:        0x1080cb54a - std::panicking::begin_panic_handler::{{closure}}::h99569328be4adae9
   5:        0x1080cb4b8 - std::sys_common::backtrace::__rust_end_short_backtrace::h9a7e5cc6475b496b
   6:        0x1080cb460 - _rust_begin_unwind
   7:        0x1080cb42f - std::panicking::begin_panic_fmt::hc364d6defd6a88b8
   8:        0x108095862 - core_foundation::dictionary::CFMutableDictionary<K,V>::get::{{closure}}::h1ef14b3201e41ad6
   9:        0x108026032 - macchina::data::get_all_readouts::h822a64d8592cec98
  10:        0x1080281dc - macchina::main::hbb20e545ea1f7921
  11:        0x10800a84a - std::sys_common::backtrace::__rust_begin_short_backtrace::h41354b2dac9becba
  12:        0x10800ac46 - std::rt::lang_start::{{closure}}::hb43fe7bf99502281
  13:        0x108033dfb - _main

To Reproduce
Steps to reproduce the behavior:

cargo install macchina
macchina

Expected behavior
not panic
Screenshots
If applicable, add screenshots to help explain your problem.
macchina --doctor output
macchina --doctor
thread 'main' panicked at 'No entry found for key 0x7f800a501240', /Users/fdncred/.cargo/registry/src/github.com-1ecc6299db9ec823/core-foundation-0.9.1/src/dictionary.rs:233:43
stack backtrace:
   0:        0x10ff35d29 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h34f76b2ca42fab41
   1:        0x10ff0dfad - core::fmt::write::h8f13d857463c51c1
   2:        0x10ff35476 - std::io::Write::write_fmt::h0a6214ccb1a8468b
   3:        0x10ff34ba4 - std::panicking::rust_panic_with_hook::h28bf9d6c3916221e
   4:        0x10ff5854a - std::panicking::begin_panic_handler::{{closure}}::h99569328be4adae9
   5:        0x10ff584b8 - std::sys_common::backtrace::__rust_end_short_backtrace::h9a7e5cc6475b496b
   6:        0x10ff58460 - _rust_begin_unwind
   7:        0x10ff5842f - std::panicking::begin_panic_fmt::hc364d6defd6a88b8
   8:        0x10ff22862 - core_foundation::dictionary::CFMutableDictionary<K,V>::get::{{closure}}::h1ef14b3201e41ad6
   9:        0x10feb3032 - macchina::data::get_all_readouts::h822a64d8592cec98
  10:        0x10feb51dc - macchina::main::hbb20e545ea1f7921
  11:        0x10fe9784a - std::sys_common::backtrace::__rust_begin_short_backtrace::h41354b2dac9becba
  12:        0x10fe97c46 - std::rt::lang_start::{{closure}}::hb43fe7bf99502281
  13:        0x10fec0dfb - _main

System Information
You don't have to provide this information if you're not comfortable doing so, but it'll help us solve the issue a lot faster.

Desktop Environment : Aqua
Operating System: macOS High Sierra 10.13.6
Terminal : iterm2
Macchina's version: 0.6.9

Thanks for reporting this, this is a libmacchina issue.
It's not clear to me whether our implementation is lacking is at fault for this panic or there's a bug within core-foundation. @123marvin123 should be able to provide you with more information regarding this bug.

It's a MacBook Pro 15"

It's a MacBook Pro 15"

We're looking into this :)

@fdncred You mentioned it was an old MacBook, did you replace the battery?
We're thinking this is the reason we can't fetch your battery statistics.

Interesting. Yes, I replaced the battery long ago.

Is it an original Apple battery or a third-party replacement? I think the problem is that there is no information about the battery in apples internal system, so macchina fails to query the needed information.
When you execute macchina and it crashes, are you plugged in into wall power or are you using battery power?

It's a 3rd party battery that I bought off Amazon and installed myself. What you're saying makes sense, other than having an unhandled exception.
It fails on battery or power. It makes no difference.

Hey there!
We have fixed this and it'll be available in the next release :)

Great! Is there any ETA on that release?

Great! Is there any ETA on that release?

It should be out as soon as #78 is merged!

No worries. I'll have to give this a try later this evening. Appreciate the help!

No worries. I'll have to give this a try later this evening. Appreciate the help!

Hey, how are you doing?
Just came to tell you that the new release is out :smiley:

Looks like it's working. Thanks again!


fix(menu): Open KMS in same window, instead of new tab

:tada: This PR is included in version 1.34.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Implement concept of stream admin

This PR is for implementing concept of stream admin,

[x] Add a role field to subscription model. (merged)
[x] Allow stream admins to update stream properties and to deactivate a stream.(merged)
[x] Allow stream admin to remove others from stream.(merged)
[x] Add is_stream_admin field to Subscription objects sent to clients.(merged)
[x] Allow stream admin to change stream properties from UI.
[x] Prevent unsubscribing of last stream admin.
[x] Change role from stream admin to role if stream admin unsubscribes.
[x] Make creator of stream a stream admin.
[x] Add API endpoint to change the role of user in stream.
[x] Add stream_admins field to Subscription objects passed to clients.
[ ] Add indicator for stream admin subscribers list.
[ ] UI for changing subscription role.

@zulipbot remove "has conflicts"

This has not been decided and needs to be discussed.

@timabbott this is ready for review.

@sahil839 thanks for your patience with this getting cut from the 3.0 release plan.  Looking at it now.

This is looking good!  I think the first commit may by mergable; I'll do so if it is.  Once that's done:

I posted I think a full set of comments on the commit I'd like to merge next, namely the one to ensure stream administrators (in that commit an empty set except in tests that create one) have the administrative access to the stream they have.  We can merge that into master as soon as it's ready without other changes.
Next, we'll want to integrate "streams: Add is_stream_admin field to sub objects passed to clients.".  I think we should change that to pass role in the API rather than is_stream_admin; I hope we've learned our lesson from the user-level role field.  And you'll need to bump API_FEATURE_LEVEL; may be more or less a rewrite of that commit.

Let me know when you have those commits ready for a next review.  I think we'll be able to merge the bulk of this PR this week, though we may not be able to finish the "Stream administrators" feature until we resolve questions like "Must a stream have an administrator?" and figure out how to do UI for changing roles in streams.

@timabbott Updated the PR with following changes-

Added check for unsubscribed user trying to access private stream. I was confused where to add the tests for this becuase we allow realm admins to update unsubscribed private streams anyway and also currently we have restriction that stream admin is always the user who is subscribed for public as well as private streams. So I have added the tests for this as a separate test as test_non_admins_cannot_access_unsub_private_stream for now.
Added StreamAdministratorRequired exception class.
And some other changes in tests that you suggested.

Regarding the is_stream_admin field passed to clients, we currently pass boolean fields only ('is_admin, is_owner) to clients, while clients send roleto API for changing it. So, wanted to confirm whether we want to passis_stream_adminorrole` to clients.

I merged the first two commits as the series ending with ca1a8ac78f5863e5fd1241b0259871b5a2849607.  In theory, that's most of the complexity on the backend.  Can you proceed with making the API send role?

Currently, we do not change role of stream admin to member of unsubscribing in the original Subscription object, we only change the role to member in stream_dict sent to API.
I will update this to change the role itself on unsubscribing, after seeing how other products handle this.

Merged the feature as f046c9c58a24a8ad5cae8c79a71100648e29ec05 after a bit of polish on the API documentation and changelog.

@timabbott I have made the following changes-

Added commit for preventing last stream admin from unsubscribing.
Added commit for changing role to member when stream admin unsubscribes.
Renamed sub.is_admin to sub.is_stream_admin
Made the suggested changes in frontend commit.

Next, I will be working on adding endpoint for changing role and adding an indication for stream admin in subscribers list.
Regarding showing stream admin in subscribers list, I am thinking of two options- adding a separate role column or adding stream admin besides the name and nothing for members like Iago (stream administrator). Any suggestions on this or some other ideas?

@zulipbot remove "has conflicts"

@timabbott This PR is ready for the next review. Updated the PR with following -

Added the commit for preventing last stream admin to unsubscribe. Currently, this restriction is for all the streams. We can change it if we want to keep it only for private streams.
Changing role from stream admin to member on unsubscribing.
Added API endpoint for changing role.

@timabbott Rebased the PR. Please  review.

I merged a couple more prep commits as the series ending with ab90abfa4c316197b85d0d55b4b63d0af0a1e2e6.  In the future, it'd be great if you can put independent refactors like those at the start of the branch so they're easy to find :).

@timabbott rebased the PR with the following changes-

Moved a prep commit to extract check_subscription_property_update to the top.
Squashed the tweak commit you added (also fixed the typo in that commit).

@timabbott Added a commit to allow stream admins to post to stream irrespective of stream_post_policy. Please review this PR.

Can you rebase?  I'll also merge what I can now.

@timabbott rebased.

@timabbott I have rebased the PR. Please review.

@zulipbot remove "has conflicts"

From the discussion on CZO, this was blocked on p1c2u/openapi-core#265 somehow, so FYI we pulled in the fix for that with #17266.

From the discussion on CZO, this was blocked on p1c2u/openapi-core#265 somehow, so FYI we pulled in the fix for that with #17266.

This PR was not blocked for that as I had already made a workaround commit for changing the order of those endpoints which fixed that error and that commit is already merged to master.

For get_stream_admin_dict_for_streams, I think we'll want to add a database index including the role property; the most natural would be an index on recipient, role, active.  We should test what query plan that generates for get_stream_admin_dict_for_streams; it's possible we'll want to change the query to be interacting with recipient_ids instead.

@timabbott this is ready for review. Have done the following updates -

Added the index (recipient_id, role, active)
Changed get_stream_admin_dict to accept recipient_ids. So this also led to change in the return type of this and we send dict of the form {'recipient_id': list of stream admins} now, we can use dict of form stream_id: list of stream admins now also but will involve and extra query.
I cannot understand the point here of incorrect logic here, it handles the case of streams with no stream admins also.
Added user_profile__is_active=True in get_stream_admin_dict query.
Extracted the check_last_admin_cannot_unsubscribe function.
Added reason: unsubscribe in the realmaudit log entries. Do we also need to add reason: deactivate for deactvating stream cases. Also the current models forRealmAuditLog has constant values for OLD_VALUE, etc. fields in extra data. Do we also want to add reason as a constant in the RealmAuditLog table and then use it or using string "reason" is fine.
Added RealmAuditLog entries also where it was required as mentioned by you.
Reordered the commits and also added a prep commit for frontend change.
Moved the logic of checking role in access_send_stream_message (cannot remember the reason why I did it in check_message before)

One change is left which is to allow the stream admin to unsubscribe if there is only one subscriber of the stream which is stream admin.

Last push was to just move the prep commit for refactoring at the top.

@timabbott this is ready for review. Have done the following updates -

Added the index (recipient_id, role, active)
Changed get_stream_admin_dict to accept recipient_ids. So this also led to change in the return type of this and we send dict of the form {'recipient_id': list of stream admins} now, we can use dict of form stream_id: list of stream admins now also but will involve and extra query.
I cannot understand the point here of incorrect logic here, it handles the case of streams with no stream admins also.
Added user_profile__is_active=True in get_stream_admin_dict query.
Extracted the check_last_admin_cannot_unsubscribe function.
Added reason: unsubscribe in the realmaudit log entries. Do we also need to add reason: deactivate for deactvating stream cases. Also the current models forRealmAuditLog has constant values for OLD_VALUE, etc. fields in extra data. Do we also want to add reason as a constant in the RealmAuditLog table and then use it or using string "reason" is fine.
Added RealmAuditLog entries also where it was required as mentioned by you.
Reordered the commits and also added a prep commit for frontend change.
Moved the logic of checking role in access_send_stream_message (cannot remember the reason why I did it in check_message before)

One change is left which is to allow the stream admin to unsubscribe if there is only one subscriber of the stream which is stream admin.

Last push was to just move the prep commit for refactoring at the top.

This push was for fixing a variable name, now this is ready for review. WIll add the code to allow stream admins to unsubscribe if all the users are being unsubscribed in a couple of days.

@timabbott this is ready for review. Have done the following updates -

Added the index (recipient_id, role, active)
Changed get_stream_admin_dict to accept recipient_ids. So this also led to change in the return type of this and we send dict of the form {'recipient_id': list of stream admins} now, we can use dict of form stream_id: list of stream admins now also but will involve and extra query.
I cannot understand the point here of incorrect logic here, it handles the case of streams with no stream admins also.
Added user_profile__is_active=True in get_stream_admin_dict query.
Extracted the check_last_admin_cannot_unsubscribe function.
Added reason: unsubscribe in the realmaudit log entries. Do we also need to add reason: deactivate for deactvating stream cases. Also the current models forRealmAuditLog has constant values for OLD_VALUE, etc. fields in extra data. Do we also want to add reason as a constant in the RealmAuditLog table and then use it or using string "reason" is fine.
Added RealmAuditLog entries also where it was required as mentioned by you.
Reordered the commits and also added a prep commit for frontend change.
Moved the logic of checking role in access_send_stream_message (cannot remember the reason why I did it in check_message before)
One change is left which is to allow the stream admin to unsubscribe if there is only one subscriber of the stream which is stream admin.
Will try to get it done ASAP, but would be nice to get a review here till then.  (Posting the list again here as faced some issue in GitHub mobile app which edits the comment on selecting Quote and reply option)

I think we can probably close this, since our user groups plans in https://github.com/zulip/zulip/issues/19525 replace it.
There's probably cleanup to do, in the form of deleting Subscription.role?

Close, since we've now removed subscription.role.